Inference & Serving

Concepts for running models efficiently and understanding serving performance.

Batch inference processes a collected set of model inputs without requiring immediate results for individual inputs.

Continuous batching dynamically schedules active inference sequences together as requests arrive, progress, and finish.

Decode is the phase of autoregressive inference that extends a response one output token at a time after prefill.

Dynamic batching groups incoming inference requests into model batches at serving time, sometimes waiting briefly for more requests.

A KV cache stores attention key and value representations from earlier tokens for reuse during autoregressive generation.

Model inference runs a trained model on an input to produce a prediction or generated output.

Online inference serves incoming model requests with latency targets appropriate to the caller's needs.

Prefill processes a model's input tokens and prepares the state used to generate a response.

Prompt caching reuses computation for repeated prompt prefixes when a model-serving system supports it.

Quantization limits model values to a smaller set, often reducing inference memory use.

Speculative decoding can accelerate generation by proposing tokens cheaply and verifying them with the target model.

Time per Output Token measures the ongoing latency of producing output tokens after generation has begun.

Time to First Token describes the delay to a request's first generated token, with timing boundaries that vary by measurement.