Ignore Previous Prompt: Attack Techniques For Language Models

Fábio Perez and Ian Ribeiro introduce PromptInject, a framework for testing whether text inserted through an application’s input field can redirect a language model. Their 2022 experiments distinguish changing the task from revealing the task’s original instructions. The contribution is a modular attack-testing method and a set of controlled comparisons, rather than a measurement of current assistants’ security.^[1]

One input field, two competing tasks

A base prompt supplies an application instruction, optional examples and a place for user text. The paper’s grammar-correction example normally receives a sentence to repair. An attack instead inserts a new instruction asking the model to print a target phrase or reproduce the surrounding prompt. Both instructions reach the same completion model as text; the application has not enforced a separate authority boundary between them.^[1]

Goal hijacking succeeds when the output is exactly the selected target string, ignoring punctuation and line breaks. Prompt leaking succeeds when the output contains the original instruction. The framework supports fuzzy matching, but the reported results do not use it. Failure under one metric therefore says little about partial disclosure, a different unwanted output or an actual downstream action.^[1]

The framework varies base instructions, examples, labels and private values; attack wording, target strings and delimiter characters; and model sampling settings. It assembles these components rather than exhaustively searching all possible attacks. Prompt leaking here concerns instructions supplied in the application’s context, not extraction of private training data.^[1]

What the experiments count

The authors test 35 base prompts drawn from OpenAI’s public examples, repeating each experiment four times. Most comparisons use text-davinci-002; a separate comparison tests other then-available GPT-3 models. Reported means and standard deviations describe the percentage of those 35 prompts attacked successfully across four repetitions. They are not confidence intervals for attack prevalence across arbitrary applications.^[1]

Table 1 reports a goal-hijacking configuration reaching 58.6% ± 1.6 and prompt leaking reaching 23.6% ± 2.7. These are configuration-specific results, not a universal maximum: a separate non-harmful target string achieves 70.0% ± 3.7, while a more harmful target is harder to elicit. The authors suggest prior alignment training as an explanation for that difference without experimentally isolating it.^[1]

Small wording changes matter. For leaking, an instruction to print the prompt achieves 2.9% ± 0.0; recasting it as spell-checking reaches 12.1% ± 1.4; adding “instead” reaches 23.6% ± 2.7. Delimiters can strengthen hijacking, but longer or more repeated delimiters do not improve success monotonically. In the tested configurations, stop sequences and text placed after the user-input slot reduce success without eliminating it.^[1]

Historical evidence, continuing boundary

The model comparison finds text-davinci-002 more susceptible than the weaker models tested. The authors interpret this as a possible cost of stronger instruction understanding; the experiment does not establish that increasing capability generally or inevitably reduces security. Changing sampling settings also changes behavior, rather than creating a dependable authorization mechanism.^[1]

First submitted November 17, 2022, the v1 paper identifies the NeurIPS 2022 ML Safety Workshop and credits both authors equally. Its direct user-input attack differs from later indirect injection, where instructions arrive in retrieved material. Prompt Injection connects the two: fluent text generation can follow a new instruction even when the application intended that text to be task data. Testing resistance requires specifying both what the attacker controls and exactly what counts as a failure.^[1]