Definition
Inference-time compute is the computation used while a trained model answers a request. The term usually comes up when a system spends more than it would on a single, direct response. It might generate more intermediate steps, produce several candidate answers, or use another model pass to compare and revise them. This is also called test-time compute.
The model’s weights stay the same. The application or inference runtime allocates the extra work to this request.
Simple example
An application asks a model to fix a parser that accepts whitespace-only input when it should reject it. A single pass produces one patch. With a larger inference budget, the application samples three patches separately, then uses another model pass to compare them with the requirement.
That extra work may surface a better patch, but the comparison is still a model judgment. A test that covers whitespace-only input would provide separate evidence about the result.
Why it matters
Extra generation and comparison can improve results on some tasks, especially when the first answer is likely to miss a constraint. They also consume more tokens or model calls, which increases cost and often delays the response. For a simple extraction task, several extra passes may just add cost. A tricky code change may be worth the additional candidates.
Measure the effect on the tasks your application handles. Compare answer quality, latency, and cost at each budget instead of assuming that a longer response means a better one.
One important nuance
More compute does not guarantee a correct answer. Several candidates can share the same false assumption, and a model asked to choose among them can select the wrong one. When a system generates multiple candidates, how it assesses them matters. For code, a relevant test can provide stronger evidence than another unsupported model judgment.