Definition
Model inference is running a trained model on an input to produce an output. The model uses parameters learned during training. An inference run does not update those parameters. Depending on the model, the output might be a class score, an embedding, or generated tokens.
Simple example
An application sends an incident report to a language model and asks for a short summary. The serving runtime processes the input and repeatedly uses the model to score possible next tokens. It selects tokens until a stopping condition is reached, then returns the generated text. The model’s learned parameters are the same before and after the request.
Why it matters
Each model call uses compute. Input and output lengths, model choice, hardware, and request batching can affect latency and compute demand. Monetary cost also depends on the deployment and pricing model. Engineers need to measure this runtime work when setting request budgets or capacity limits.
An application can change the prompt or supply retrieved context for a particular request. That may change the answer, but it does not train the model. Fine-tuning is a separate process that updates learned parameters for later use.
One important nuance
One inference request does not always mean one pass through the model. A classifier may produce scores in a single pass. An autoregressive language model makes successive next-token predictions, often reusing intermediate state between them. Both are inference. In autoregressive generation, longer outputs generally require more decoding work.