Safety & Security

Concepts for adversarial inputs, unsafe behavior, and controls around AI systems.

Guardrails are controls around an AI system that constrain, validate, block, or mediate inputs, outputs, and actions.

Indirect prompt injection reaches a model through untrusted external content rather than the user's direct instruction.

Jailbreaking attempts to bypass restrictions on a model's outputs or behavior through crafted prompts or interactions.

Prompt injection is an attempt to make untrusted input influence model behavior beyond that input's authority.

Reverse prompt engineering attempts to reconstruct an unknown prompt from observable model outputs.