Safety Case

A safety case is a structured argument that a particular system, used under specified conditions, poses an acceptable level of risk. It connects a claim to evidence, explains the assumptions connecting them, and makes gaps and objections inspectable. Its purpose is to support a decision about use and safeguards; assembling a case does not itself guarantee safety.^[1]

Specify the claim and its scope

Start with the system, operating environment, harmful outcome, people exposed, time horizon and acceptable risk threshold. Break the main claim into smaller claims that can be tested. A statement that a monitor detects suspicious actions needs different support from a statement that the whole system prevents their consequences.

An Example Safety Case for Safeguards Against Misuse supplies a concrete proposed argument. For a hypothetical assistant API, the authors estimate the effort needed to evade safeguards, then model the additional risk compared with a world without the assistant. The case concerns novice actors pursuing a specified pathway to large-scale harm. A jailbreak success fraction alone does not establish that risk, and the claim does not cover every adversary or autonomous model action.^[1]

Different arguments need different evidence. Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework distinguishes inability (the model cannot carry out the attack), control (defenses prevent unacceptable outcomes even if it tries), and trustworthiness (it will not try despite being capable). Low observed success can have several explanations; it does not identify which argument holds.^[2]

Connect evidence to deployment assumptions

An evaluation measures behavior in its test setting. A deployment claim needs an explanation of why the result applies to the actual users, tools, permissions and opportunities for harm.

For manipulation control, the framework proposes conservative comparisons: proxy targets must be at least as susceptible as real employees, red-team attacks must bound the model’s attack capability, and the tested controls must be no stronger than the deployed ones. Transfer also needs to account for distribution changes, adversarial adaptation and accumulated risk. These are assumptions requiring support, rather than consequences of a favorable score.^[2]

SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors illustrates the measurement problem with human-designed synthetic coding-agent transcripts. Detection thresholds depend on the benign workload used to measure false alarms. Taking the highest score from repeated judgments changes benign scores too, so that procedure requires its own calibration. The benchmark tests detection of depicted harmful behavior; it does not execute attacks end to end on production systems. Permissions, network restrictions, state across sessions and intervention timing require additional evidence before detection can support prevention.^[3]

The outcome measure matters as well. Safe Learning Under Irreversible Dynamics via Asking for Help compares learner and mentor trajectories from the same starting state. Its Figure 2 shows why judging both only after the learner has entered a permanently bad state can erase the cost of that mistake. The paper’s conditional asymptotic expected-reward result is distinct from zero harmful actions during a fixed deployment interval.^[5]

Address objections and contrary evidence

A useful case states what would undermine each claim and how the evidence addresses it. For an inability argument, strategic underperformance can hide capability. For trustworthiness, recognizing a honeypot can explain why a model refrains from attempting an attack; failed execution can still reveal an attempt. The manipulation framework identifies strong trustworthiness evidence for highly capable models as an unresolved research need.^[2]

For a control argument, the attacker tested may be too weak or too indiscriminate. Attack selection against trusted monitoring finds that choosing when to submit a coding attack can undermine defenses that look effective against an always-attacking baseline. Its game assumes perfect human audits and counts a valid backdoor as success at insertion. A deployment argument must justify its attacker search, information access, opportunities, audit budget and actual intervention behavior.^[4]

Information access, audit artifacts and feedback to the agent also belong in the assessment. For detailed studies, see How does information access affect LLM monitors’ ability to detect sabotage?, ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D, Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute and Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback. A capability test needs its own containment argument: who could be exposed, what actions are allowed and how the test is stopped. AISI detects unsanctioned agent actions during cyber evaluation provides a documented case to examine.

Connect the argument to decisions and response

The misuse example requires pre-release testing, continued evaluation and changes to safeguards or access before estimated risk crosses the threshold. Its risk model has highly uncertain parameters. The illustrative one-month response is arbitrary and too slow for pathways where large-scale harm can occur within days. The paper proposes commitments for a hypothetical developer; it does not demonstrate that a real operator fulfills them.^[1]

A case therefore needs a responsible decision-maker, an explicit basis for proceeding or restricting use, and evidence that the required response can happen in time. When the system, tools, users or response capacity change, revisit the affected assumptions. Predicting LLM Safety Before Release by Simulating Deployment and When a safety evaluation becomes a forecast develop the distinction between forecasting measured failure prevalence and supporting deployment decisions. A safety case makes that evidential bridge and the remaining uncertainty available for scrutiny.