Definition
A key-value cache, usually called a KV cache, stores the attention key and value representations already computed for earlier tokens. During autoregressive generation, the model produces one new token at a time. Reusing cached keys and values avoids recomputing those representations for the full prefix at every decoding step.
The cache is maintained per active sequence and typically per transformer layer. A dynamic full-attention cache grows with the sequence. Sliding-window or chunked attention can cap how much state it retains, while a static cache can reserve storage in advance.
Simple example
After a model processes a 2,000-token prompt, the cache contains the prompt’s keys and values. The model then selects the first output token. On the next decode step, it processes that token, adds its keys and values to the cache, and attends over the combined history to produce the second output token. It does not recompute the prompt’s keys and values.
The same process continues until generation stops or the sequence reaches its limit.
Why it matters
KV caching removes repeated work during decoding. How a serving system allocates and moves the cache affects token latency, batching, and how many sequences of different lengths it can keep active. Paging can reduce wasted cache memory when request lengths differ.
For a new, uncached prompt, prefill still processes the prompt before the first output token. A runtime with a cached matching prefix can reuse that part instead of computing it again.
One important nuance
The KV cache trades computation for memory. For full attention, cache memory depends on cached sequence length, the number of layers storing KV states, KV head count and dimension, the number of concurrently cached sequences, and cache element precision. Quantizing model weights does not necessarily reduce cache memory, so long or concurrent requests can still consume substantial memory.