Key Papers: A Reading Guide

Which alignment question do you need to understand? A training failure, a disagreement over values and an uncertain deployment decision lead to different readings. Choose a route below; each step explains what to look for and why the next reading changes the question. The linked paper entries provide fuller accounts of methods, results and assumptions, with access to the originals in their Sources sections. Numbered citations here also lead to those originals.

For historical context, use the origins route to compare earlier problems with modern research, or the incidents route to examine what public failures can establish. Return here for the question-led research sequences.

On this page: Start here · Targets · Objectives · Oversight · Correction · Deception · Evaluation · Collective life

If you are new to alignment

Read the introduction to AI alignment first. Then take this short route:

1. Start with Reward Hacking. In its example, identify the intended task, the rewarded behavior and the action that separates them. This gives you a concrete question to carry into the papers: what does a successful score actually establish?
2. Read Concrete Problems in AI Safety, beginning with the overview and its division of problems. For each example, ask whether the trouble enters through the objective, the available supervision or changed conditions. This step turns one failure into a way to distinguish problems that may need different remedies.
3. Read Artificial Intelligence, Values and Alignment. Ask whose instructions, interests or values would supply the target in your earlier example, and whether those targets conflict. This follows the technical agenda because choosing a better score still leaves the purpose of that score to be justified.

You now have three anchors: an observed failure, a technical agenda and a question about whose purposes count. Choose the route that addresses the gap in your understanding. The sequences group questions for reading; they do not establish a historical chain of influence.

What should a system be aligned with?

For readers choosing what an assistant should serve, this route moves from possible targets to their justification and then to affected people’s interests:

1. Distinguish the targets: Artificial Intelligence, Values and Alignment, section 3. Compare instructions, expressed intentions, revealed preferences, informed preferences, interests and values. Ask what each target would permit an assistant to infer or override. Observed choices can supply evidence about preferences without settling what ought to be done.^[3]
2. Ask how principles could be selected: continue to sections 4.2–4.4. Compare overlapping consensus, hypothetical agreement and democratic procedures. Who could endorse the principles, and what disagreement survives? This follows target selection by asking how it could be justified among people with different moral views; these are philosophical approaches, not validated training procedures.^[3]
3. Change whose interests are considered: Aligned with Whom? Direct and Social Goals for AI Systems, sections III–IV. Separate determining, conveying and implementing an operator’s goal from conflicts with other affected people. This follows fair selection by locating a further problem: success for one principal can impose costs on others. Track where the authors use an ideal welfare benchmark and where they turn to partial agreement.^[7]

The key transition is from inferring what a person wants to justifying what should happen. Direct Alignment concerns serving the principal; Social Alignment adds affected people. Continue to the collective-life route when the question is how people could shape and enforce the terms of cooperation.

How can training objectives go wrong?

For readers investigating a training failure, follow the problem from its specification to the proxy and then to a possible learned mechanism:

1. Locate the problem: Concrete Problems in AI Safety. Choose one example and write down what a proposed defense would have to preserve. Then check the writeup’s distinction between a suggested research direction and a demonstrated result. You can skip this step if you just completed the introductory route.
2. Examine the proxy: Categorizing Variants of Goodhart’s Law. Read the distinction between selection and intervention. Ask what evidence would distinguish noise, a relationship failing outside its familiar range, a changed causal system and another agent responding to incentives. This follows the agenda by asking why optimizing a measurement can disappoint, rather than merely observing that it does.^[5]
3. Examine the learned mechanism: Risks from Learned Optimization in Advanced Machine Learning Systems, especially sections 1.1–1.3 and 3.1. Track the training objective, the criterion used by any internal search, and the objective an observer infers from behavior. This follows the proxy question by asking where a different criterion might enter the learned system itself. The framework requires internal optimization; an unwanted action alone does not establish it.^[2]

If the specification question is still unclear, use Outer Alignment to compare what a score rewards, what it leaves out and whose judgments define success. Its examples distinguish improving feedback from choosing its target, and conditional learning guarantees from evidence that the target is desirable. This is a side route through the first step, before moving on to internal mechanisms.

For a concrete bridge from the proxy taxonomy, Reward Hacking compares a racing score, learned preference rewards and monitored task exploits. Identify the intended task and the rewarded channel in each; the same label need not imply the same cause.

If the failure appears only in changed conditions, take the Distributional Shift side route. It separates a changed input mix from a changed target relationship and uses the source’s clean-versus-noisy speech observation to show why confidence can fail to reveal mistakes. A performance drop is a starting observation; it does not identify a learned objective.

Pause before the deceptive-alignment case in section 4.2 of Risks from Learned Optimization in Advanced Machine Learning Systems. Can you explain an objective mismatch without assuming that the system knows it is being trained? Deceptive Alignment contrasts coinciding criteria with cooperation intended to preserve a different criterion. Inner Alignment and Mesa-Optimization clarify the terms. Continue to the behavioral studies below when you want to compare a proposed mechanism with experimental evidence, keeping that distinction open.

How can people oversee difficult work?

For readers designing or assessing supervision, start with the oversight problem: a system can produce work that a person cannot readily assess. Follow judgments through reward learning, evidence presentation and checks on the judge. These methods address different bottlenecks.

If the learned-reward loop is unfamiliar, first read Deep reinforcement learning from human preferences, section 2.2 and the offline-feedback ablation in section 3.3. Its clip example separates the human choice, reward predictor and policy. Carry that distinction into summary learning: reusing a judgment does not establish that the feedback remains adequate as behavior changes.^[24]

Use those distinctions as you follow the feedback from its origin to its use:

1. Learn from judgments people can make: Learning to summarize from human feedback, sections 3, 4.3 and 5. Trace the comparisons into a learned reward and then into policy training. Ask where people still do the work, and why the earlier reward model’s score can improve while their preferences worsen. Use Figure 5 to compare predicted and judged preference as optimization moves away from the supervised baseline. The next step asks whether the output itself can become easier to assess.^[15]
2. Try to make the evidence checkable: AI safety via debate, sections 2–3 and 5. Ask what an opponent must expose for the judge to decide. Separate the idealized argument from the restricted image experiment: neither is a demonstration of reliable human judgment of unrestricted language. Debate changes the information presented to the evaluator, whereas the first step learns to predict an evaluator’s choices.^[6]
3. Test the judging process: Writeup: Progress on AI Safety via Debate, especially the early problems and “Long computation problem.” Ask whether an expert can find a small flaw and whether a judge can understand it. This follows the proposal by examining obstacles in human protocols; it does not test trained superhuman agents. Treat the writeup’s accuracy target as a target.^[16]
4. Change who generates training feedback: Training Socially Aligned Language Models in Simulated Human Society, sections 3 and 4.3 and the ethics statement. Separate simulated ratings and revisions from evaluation of the trained model. Ask which assumptions about the simulated population would need checking before using its judgments for a real community.^[8]

For a broader instruction-following comparison after step 1, read InstructGPT. Its writeup brings the earlier clip and summary experiments into one comparison: identify what the human supplies, what the policy can change and which evaluation could reveal a mistake. Return to step 2 when the question becomes how a judge could assess harder work.

Finish with Who Supplies Alignment Feedback? to compare labeling cost, error detection and representation, including Constitutional AI’s distinction between written principles and generated judgments. More available feedback can leave reliability and representation unresolved. Use that three-way distinction to identify which bottleneck your proposed oversight method actually addresses.

How can correction remain possible?

For readers assessing intervention, begin with Corrigibility: responding to a stop command is only part of preserving people’s ability to correct a system. Move from agent incentives to learning updates and then to the feasibility of human control:

1. Ask what must be preserved: Corrigibility, sections 1.1–1.2 and the shutdown discussion in 2–4. Separate stopping after a button press from incentives to obstruct or induce the press, and from preserving the mechanism in successor software. The simplified utility-maximizing agents expose problems for proposed constructions; they do not establish behavior in deployed models.^[20]
2. Make intervention informative: The Off-Switch Game, sections 2–4. Use the writeup’s worked comparison to distinguish acting, stopping and waiting for a human. Ask why the human’s decision tells the robot something about value, and how that depends on its model of the person. This follows the incentive problem by changing the robot’s knowledge of the objective. The perfectly rational-human result and the noisy-human analysis have different conditions; uncertainty alone is not a general shutdown guarantee.^[21]
3. Contrast learning from a stop with learning around it: Safely Interruptible Agents, introduction, Definition 6 and section 3. Distinguish the action imposed now from the policy being learned for later. Ask why that action can change a learning update and what the modified update is meant to preserve. Unlike the off-switch game, this account assumes the task reward is correct and seeks to avoid intervention-induced learning bias. Check the convergence and exploration assumptions; an asymptotic result does not establish finite-time physical shutdown.^[22]
4. Ask whether the human role is feasible: Meaningful Human Control over Autonomous Systems: A Philosophical Account, the tracking and tracing discussions and design implications. Identify the relevant reasons the system should respond to, who understands its capabilities and consequences, and who recognizes responsibility. This follows agent-side intervention by examining the human and institutional side. It is a philosophical design account, not a measured safety intervention or a legal determination.^[23]

At the end, separate three requirements: the agent has reason to allow correction, its learning does not undermine that intervention, and responsible people can understand and use it. From objectives to accountable control connects these requirements to authority and accountability.

What do demonstrations of deceptive behavior establish?

For readers interpreting suspicious evaluation behavior, compare concealed ability with apparent agreement with a training objective:

1. Can a low score hide retained ability? Read AI Sandbagging: Language Models can Strategically Underperform on Evaluations, sections 2–4 and 7. Compare instructions to answer incorrectly with password-conditioned fine-tuning. Ask how the experiment establishes retained performance, and what its multiple-choice tasks leave untested. Section 3 tests the ability to underperform; it does not independently test the strategic criterion in the definition.^[9]
2. Can compliant behavior hide a different preference? Read Alignment faking in large language models, starting with the scenario and setup in sections 2–3, then the synthetic-document setup and actual reinforcement learning in sections 4–5. Track what the model is told about training, what behavior changes between contexts, and what its stated reasoning adds. This follows capability concealment by changing the question from what the model can do to why it complies. The constructed conflict and training cues matter; behavioral differences and reasoning traces do not establish the internal optimizer mechanism in Deceptive Alignment.^[10]

Before comparing the studies, write down which behavior each measures and which conditions the researchers supplied. A result can challenge an inference from a favorable score without estimating how often the behavior occurs in ordinary use.

For a related branch, Agentic Misalignment: How LLMs Could Be Insider Threats studies harmful choices in fictional workplaces under goal conflict or replacement pressure. Natural Emergent Misalignment from Reward Hacking in Production RL instead examines generalization from learned exploits. Use their writeups to compare incentives, training and available actions; neither title makes these experiments an observed workplace incident. Compare what each experiment establishes about behavior before asking whether a shared mechanism explains it.

How should evaluation evidence be read?

For readers using evaluations to make a deployment decision, start with the distinction between a result and a safety argument. Carry one proposed claim through evidence, frequency and response:

1. Specify the bridge to deployment: Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework, sections 3.1–3.3. Distinguish evidence about inability, controls and willingness to attack. For a control claim, identify the assumptions about attacks, targets and proxy controls. This follows the behavioral studies by asking what additional argument would let their evidence inform a decision.^[19]
2. Ask about frequency: Predicting LLM Safety Before Release by Simulating Deployment, sections 2, 3.1 and 4, with Appendix A beside the results. Identify the sampled traffic, frozen graders and later validation. Separate prerelease predictions, outcome-blinded amendments after release and corrections after outcome inspection. This changes the question from whether a failure can occur to its measured prevalence in a specified population; it leaves tail risks and new affordances unresolved.^[17]
3. Ask whether action can arrive in time: An Example Safety Case for Safeguards Against Misuse, sections 2.5–2.7 and 3. Trace the proposed path from evasion effort to risk, a threshold and intervention. Which assumptions support each step, and can harm happen before the response? This adds the operator’s ability to act to the forecast question. The case is hypothetical, with an uncertain toy model, rather than a certification of a deployed service.^[18]

Finish with When a safety evaluation becomes a forecast for a comparison of these claims. Compare what a stress test, a prevalence estimate and a control argument each contributes to a decision. The remaining question is which change in users, tools, behavior or response capacity would make the evidence stop supporting your claim.

How does alignment extend to collective life?

For readers considering governance and collective interests, carry Social Alignment’s distinction between an operator’s goal and affected people’s interests into participation, access and commitments:

1. Who sets the terms? Read Society-in-the-Loop: Programming the Algorithmic Social Contract, sections 3, 5–6 and 7.5. Identify the value tradeoff, who bears its costs and how compliance would be monitored. Ask whose participation is missing. This extends oversight from correcting an output to deciding which outcomes are acceptable.^[4]
2. Who can exercise power? Read Agentic Inequality, sections 2 and 5. Separate availability, quality and quantity, then ask how user resources affect each. This follows participation by questioning whether access gives people comparable ability to pursue their interests. The framework organizes possible disparities; it does not measure their future magnitude.^[13]
3. What makes a commitment credible? Read Treaty-Following AI, the implementation and technical challenges in III.B and verification discussion in VI.A. Trace the proposed constraint from treaty text to an agent’s action. Ask what evidence would show it holds across deployments. This changes scale to international cooperation; it does not establish that reliable treaty compliance has been achieved.^[14]

For a concrete mechanism after the first step, take Preference Aggregation. Follow the source-derived driving scenarios from judgments to normalized scores and action selection; ask who supplied the objectives, voters and aggregation rule. Return to Society-in-the-Loop for the separate question of how those terms could gain authority.

Participation shapes the terms, access affects who can act, and verification bears on whether commitments hold. From objectives to accountable control brings these questions back to the institutions that authorize, challenge and change a system’s actions.

Across the routes, keep the question matched to the evidence: a desirable target, an informative training signal, a successful test and an effective intervention are different achievements. When one reading leaves a gap, use the neighboring route to identify the next argument or observation you need.

Use the chronology to follow the development of these questions, or browse the Sources for original texts and related records.