Definition
An autoregressive language model defines a probability distribution over possible next tokens, conditioned on the tokens before them. During generation, the model produces next-token scores. A decoding procedure selects or samples a token, extends the prefix, and continues with the next prediction conditioned on that prefix.
This factorizes the probability of a complete sequence into a series of conditional next-token probabilities. The generated tokens become part of the context for later predictions, so an early choice can affect every token that follows.
Simple example
Given the prefix The deployment failed because, possible next tokens might decode to the, it, or a. The exact token boundaries depend on the tokenizer. If decoding picks a token that decodes to the, the next prefix is The deployment failed because the. Generation continues until a stop condition or token limit is reached.
Temperature is a decoding control: it adjusts the model’s scores before sampling. Each prediction still depends on the preceding tokens.
Why it matters
Output length adds latency because each new token depends on an earlier selection. Output positions cannot all be decoded independently. Early selections can also cause a response to drift because they become part of the prefix for later predictions.
In transformer inference, KV caching can avoid recomputing key and value representations for earlier tokens, while speculative decoding can reduce sequential decoding latency.
One important nuance
Autoregressive describes a modeling and generation process. Transformer names a neural-network architecture. Decoder-only transformers often generate autoregressively, and encoder-decoder transformers can do so as well. Encoder-only transformers serve other tasks, while architectures beyond transformers can also model sequences autoregressively.