Is Power-Seeking AI an Existential Risk?

Joseph Carlsmith examines a particular route from advanced AI to existential catastrophe. The report separates the technical possibility of building powerful agents from incentives to deploy them, difficulties controlling their objectives, and failures of correction. Its contribution is a structured argument with subjective conditional estimates, rather than a measured rate of AI catastrophe.^[1]

Which systems does the argument concern?

Carlsmith calls the relevant systems APS: advanced, planning and strategically aware. They outperform the best humans on some tasks that confer substantial power, make and execute plans using models of the world, and understand the consequences of gaining or retaining power over humans. This does not require superiority in every domain, unlimited competence, consciousness, or a single system taking over everything (§§1–2).^[1]

His definition of misalignment is narrower than a harmful mistake: it concerns unintended behavior arising from problems with a system’s objectives. A financial agent making a mistaken investment while genuinely trying to earn money illustrates a capability failure; competently pursuing an unwanted objective illustrates the alignment concern. Carlsmith acknowledges that the boundary can be difficult to draw. He also distinguishes behaving as a designer intends from the much broader question of whether AI benefits society (§4.1).^[1]

Six links in the pathway

The argument asks whether, by 2070, the following sequence will hold. Each premise after the first is assessed conditional on the preceding premises (§1):

1. Feasibility: APS systems become technically possible and financially feasible.
2. Incentives: actors have strong incentives to build and deploy them.
3. Alignment difficulty: systems that avoid unintended power-seeking throughout their actual deployment inputs are much harder to build than systems that remain superficially attractive to deploy but would fail on some such inputs.
4. Harmful deployment: some deployed systems encounter inputs that trigger unintended power-seeking with high impact. Carlsmith illustrates the scale with collective damage above $1 trillion in 2021 dollars; this is a scenario threshold, not an observed loss.
5. Failed correction: some of that power-seeking scales in aggregate to the permanent disempowerment of approximately all humanity.
6. Catastrophe: that disempowerment destroys humanity’s long-term potential.^[1]

The sequence makes several gaps visible. A system can be misaligned without being deployed in the conditions that reveal it. An attempt to gain power need not succeed. A severe failure can prompt correction before permanent disempowerment. And the final premise concerns the value of humanity’s possible future, rather than simply whether people lose a particular contest.^[1]

Power, deployment and correction

Power matters instrumentally because control of resources or freedom from interference can help realize many objectives. But the report distinguishes having an incentive, acting on it, and successfully gaining power. It considers three broad ways to prevent problematic behavior: control objectives, restrict capabilities, or constrain the circumstances a system encounters. Their practical difficulty depends partly on whether safeguards remain competitive with less constrained systems (§§4.2–4.3).^[1]

Deployment decisions add an institutional layer. Competition, risks imposed on others, the number of actors able to deploy systems, and deceptive behavior could make individually attractive deployments dangerous collectively. Conversely, non-APS systems or human supervision might supply useful alternatives; the report does not establish that every valuable application requires an APS agent (§§3, 5).^[1]

Correction remains a separate uncertainty. A capable agent may still face limits in computing resources, physical infrastructure, cybersecurity or coordination. Carlsmith considers warning failures, containment, industry responses and state action. An adequate response would have to address underlying incentives and recurring alignment failures, rather than merely stop one episode. These are speculative pathways and constraints, not validated intervention effects (§§6.2–6.4).^[1]

Estimates and limits

The original roughly 5% estimate for this six-premise route by 2070 becomes above 10% for power-seeking AI catastrophe in a May 2022 author note. These are dated subjective judgments, not measured frequencies or a current consensus. Carlsmith stresses that the premises are imprecisely operationalized. Conditional estimates should incorporate what preceding premises imply; treating them as independent questions would misrepresent the exercise. He also discusses sensitivity to how an argument is divided into premises, and the possibility of catastrophe through routes that do not satisfy his full sequence (§8).^[1]

The framework helps locate disagreements: a technical claim about Corrigibility addresses only part of a pathway that also depends on deployment choices and institutional correction. Current and Near-Term AI as a Potential Existential Risk Factor examines a different route through social systems. Neither a local alignment failure nor a successful shutdown test settles the full argument.

Historical context

The report identifies an April 2021 public release; arXiv submission followed on June 16, 2022. The linked text is arXiv v2, revised August 13, 2024, and retains the dated May 2022 probability update.^[1]