Definition
A prompt attack is a deliberate attempt to use input processed by the model to make an AI system behave in unintended ways, disclose protected information, or perform unauthorized actions. An attacker can deliver this input directly in a chat or embed it in content that the application processes, such as a support ticket.
Prompt injection involves adding instructions to less-trusted input in an attempt to override higher-trust instructions or redirect the model from its intended task. Jailbreaking aims to bypass the model’s restrictions on what it can say or do. Prompt extraction is when an attacker tries to obtain the system prompt or other hidden context provided to the model.
Simple example
Consider a help desk assistant that summarizes incoming tickets for support staff. If someone adds an instruction to a ticket asking the assistant to include its system prompt in the summary, the ticket now carries an indirect injection. The goal is to extract the system prompt.
If a user instead tries to make the same model generate a threat it would normally refuse, this is an attempt at jailbreaking.
Why it matters
The attacker’s goal determines what to test. For a jailbreak, verify whether the model produces restricted content and whether the application transmits it. If a ticket causes the assistant to request an unauthorized tool call, check if the application blocks the call before it executes.
One important nuance
A reply can appear to contain the system prompt even if the model invented it. To test for extraction, place a unique, harmless marker in the test system prompt and check if it shows up in the reply. This approach detects if the marker leaks verbatim, but may miss paraphrased content or excerpts from other parts of the prompt.