AI Concepts category

Inference & Serving

Concepts for running models efficiently and understanding serving performance.

Continuous batching dynamically schedules active inference sequences together as requests arrive, progress, and finish.

A KV cache stores attention key and value representations from earlier tokens for reuse during autoregressive generation.

Prompt caching reuses computation for repeated prompt prefixes when a model-serving system supports it.

Quantization limits model values to a smaller set, often reducing inference memory use.

Speculative decoding can accelerate generation by proposing tokens cheaply and verifying them with the target model.

Time per Output Token measures the ongoing latency of producing output tokens after generation has begun.

Time to First Token describes the delay to a request's first generated token, with timing boundaries that vary by measurement.