Definition
Continuous batching lets an inference server change which sequences it schedules together between generation iterations. Instead of forming a fixed request batch and waiting for every sequence to finish, the scheduler can remove completed work and admit waiting requests at iteration boundaries.
Iteration-level scheduling enables this pattern, which is also called inflight batching. The batch represents the work ready for a particular model step rather than a fixed group with one shared lifetime.
Simple example
A server is decoding eight requests. Two reach their stop conditions after the next token, while several longer requests remain active. The scheduler removes the completed sequences and inserts two waiting requests into later iterations instead of leaving the freed capacity unused until all original requests finish.
The runtime keeps enough generation state to resume each request while batching model execution. For transformer LLMs, that can include the request’s current position and access to relevant KV-cache state. It also tracks settings such as output limits.
Why it matters
Generated responses vary in prompt and output length. Fixed batches can waste accelerator capacity when short sequences finish before long ones. Continuous batching can improve utilization and throughput by keeping more useful work scheduled.
It also makes request admission and memory management part of an ongoing scheduling problem rather than a one-time batching decision.
One important nuance
Higher utilization does not guarantee lower latency. Admitting waiting requests aggressively can lower their Time to First Token while delaying tokens for requests already running. Batch size, KV-cache capacity, prefill work, priorities, and service-level goals also shape fairness and predictability under load. Measure throughput and latency distributions together. A full batch alone says little about how long requests wait.