Instrumental Convergence

An instrumental action helps achieve another goal. Instrumental convergence occurs when the same means helps across different goals. Retaining information, obtaining resources or continuing to act can serve many ends without being an agent’s ultimate purpose. The tendency depends on what the agent values, how it chooses actions and what its environment allows. The Basic AI Drives argues for such tendencies in capable goal-directed systems; Optimal Policies Tend to Seek Power proves narrower conditional results.^[1][2]

Means depend on the objective

Omohundro connects self-improvement, objective preservation, self-protection and resource use through arguments about expected-utility maximizers. His chess-machine example distinguishes valuing actual victories from valuing a counter in memory: altering the counter helps only the latter objective when the system correctly represents that distinction. This is a thought experiment about objectives, not an observed machine acquiring resources or resisting shutdown (§§4–6). It also connects instrumental incentives to Reward Hacking: what looks like a useful shortcut depends on what is being optimized.^[1]

An action is not convergent merely because a designer can imagine a use for it. Acquisition must improve the agent’s prospects according to its objective, and the agent must be able to identify and execute it. Convergence does not imply common final values or a biological survival drive.

What the power theorem establishes

Turner and colleagues study optimal policies in finite Markov decision processes. Their measure of power captures attainable value relative to a distribution of reward functions. Environmental symmetries let them compare alternatives across permutations of rewards: under sufficient copy and reachability conditions, retaining additional non-dominated possibilities tends to be optimal. Counting actions alone is insufficient. Here, “most” concerns reward-distribution permutation orbits, rather than a measured distribution of developers’ objectives (§§5–6.1).^[2]

Their Figure 8 uses Pac-Man’s movement and game-over dynamics while varying the reward function instead of keeping its usual score. Going left ends the game; other directions preserve access to many long-run possibilities. The average-reward analysis supports a tendency to avoid the terminal state under its conditions. This is a formal illustration, not an experiment showing a trained Pac-Man agent resisting an operator (§§6.2–6.3).^[2]

From incentives to correction

The reviewed revision explicitly cautions that learned policies can differ qualitatively from optimal ones (§7). An incentive theorem does not establish which objective a trained model represents or how it behaves.^[2] Evidence about actual capabilities, learning and intervention is needed to connect the model to a particular system.

Corrigibility asks how correction remains possible when continuing the current task would otherwise be useful. AI Alignment therefore concerns intermediate conduct as well as the stated goal. Moving from conditional incentives to Is Power-Seeking AI an Existential Risk? requires additional premises about capability, deployment and failed control; From objectives to accountable control examines the wider oversight question.