Definition
Prompt caching, also called prefix caching in some systems, reuses model-serving computation associated with a prompt prefix that has already been processed. When a later request begins with a compatible repeated prefix, the serving layer can reuse cached intermediate state instead of performing all prefix computation again.
This is an inference optimization. The prompt content still participates in the request, and the model’s weights are not changed.
Simple example
An application sends the same long system instructions and tool definitions with many requests. On a cache hit, the serving system reuses cached computation for the matching prefix and processes the uncached remainder before generation begins.
If one token near the start changes, the reusable portion may shrink or the request may miss the cache entirely, depending on the implementation’s matching rules.
Why it matters
Repeated prompt prefixes can make prefill work and input-token processing a significant part of latency and cost. Caching may reduce Time to First Token or billed processing for eligible requests and can make large shared instructions less expensive to reuse.
When the serving system exposes cache metrics, applications can track them and keep shared prefix content stable.
One important nuance
Prompt caching is not response caching. It reuses intermediate prompt computation, while response caching returns a previously stored application result. Cache scope, lifetime, privacy boundaries, minimum prefix size, matching rules, pricing, and automatic behavior vary by provider and runtime. Never assume a cache hit or use caching as a reason to include unnecessary context.