Mesa-Optimization

A mesa-optimizer is a system produced by training that itself searches for good outputs, plans or policies according to an internally represented objective. The training process selects the system; the system then performs its own optimization. Those two processes can favor different things, creating an Inner Alignment problem. Risks from Learned Optimization in Advanced Machine Learning Systems introduced this framework.^[1]

Training optimization and internal search

The base optimizer is the learning algorithm, such as gradient descent, that searches over model parameters. Its base objective is the criterion used to select or improve the learned system. A mesa-objective is the criterion represented within a learned optimizer that guides its own search. Not every learned algorithm performs such search, so not every model has a mesa-objective in this sense.^[1]

This is a distinction about computation, not just success. The paper contrasts a classifier optimized for low error with a network that searches possible plans and evaluates their predicted outcomes. A system’s behavior can score highly without the system internally searching to maximize that score. Conversely, an optimizer need not be a separate agent hidden inside a network: the network itself can implement the search.^[1]

The authors discuss built-in tree search in game-playing systems as a contrast. Search deliberately supplied by the designer is not enough to establish mesa-optimization; the optimizing algorithm must itself be produced by learning. These examples motivate a possibility, rather than demonstrate that ordinary trained networks acquire internal search.^[1]

Why matching behavior is insufficient

A learned optimizer’s criterion can agree with the base objective on training inputs and diverge elsewhere. The paper’s red-exit maze thought experiment makes the mismatch concrete: reaching exits and reaching red objects both earn success during training, but separate when exits are blue and unrelated objects are red. Inner Alignment develops this example. A change in behavior would reveal a generalization failure, but would not by itself establish internal search or identify its represented objective.^[1]

Reward Hacking concerns a rewarded measure coming apart from the intended outcome. Mesa-optimization concerns how a learned system computes its actions. A mesa-optimizer could be aligned, misaligned without deception, or strategically cooperative during training. Deceptive Alignment adds assumptions about modeling training pressure and preserving an objective for later; deception is not part of the definition of mesa-optimization.^[1]

The practical question is therefore both what the learned system does and what computation produces that behavior. Key Papers: A Reading Guide connects objective specification, learned optimization and experiments on training-dependent cooperation.