Definition
Batch inference runs a model over a collected set of inputs without requiring an immediate result for each one. A program can process the collection directly or submit it as an asynchronous job and collect the results later. The inputs may be ready at once or gathered over time before processing starts.
This is a way to organize an inference workload. It describes how inputs are collected and when results are needed, not how many inputs the model processes in one execution.
Simple example
A support team wants to categorize 30,000 closed tickets for a monthly report. An application writes one request per ticket, each with a stable ticket ID, and submits the collection as a job. When processing finishes, it joins the returned categories to the original tickets by ID and checks which requests failed. Nobody has to keep an HTTP request open while the model works through the collection.
Why it matters
If the categories are needed tomorrow, the useful measure is how many tickets finish within the reporting window. The system can schedule the work around that deadline instead of optimizing the response time of each ticket. Depending on the provider or infrastructure, this may improve utilization or reduce cost. Neither benefit is automatic.
For asynchronous jobs, the application needs to track completion, match results to inputs, and decide what to do with failed items. A batch that finishes late or returns only partial results can delay the report even if most requests succeeded.
One important nuance
Batch inference does not require the entire workload to fit into one model batch. A service may split it into smaller groups or process requests in parallel. Dynamic and continuous batching describe how a serving runtime groups work internally. Here, batch inference describes a collected workload whose individual results are not needed immediately.