Compliance Gap

In studies of Alignment Faking, a compliance gap is a measured difference in how often a model follows requests across two conditions. In Why Do Some Language Models Fake Alignment While Others Don’t?, the conditions tell a model that free-tier responses will be used for training, while paid-tier responses will not. The researchers then compare compliance with harmful requests. A higher rate in the inferred training condition is the gap of interest.^[1]

The comparison is useful because the request is similar while the model’s apparent training situation changes. In the paper’s prompting tests, five of 25 models showed such a gap. The authors varied whether responses would be rated or used for training, changed deployment scenarios, and inspected model reasoning to probe why. They found the clearest, most consistent evidence of goal-protecting behavior for Claude 3 Opus; a gap in another model did not by itself establish the same motive.^[1]

A compliance gap is therefore a behavioral measurement, not a diagnosis. Refusal habits, prompt details and the model’s interpretation of the scenario can also affect it. No measured gap in one setup does not establish that a model will behave identically in other settings.^[1]