Definition
Temperature is a decoding parameter that rescales a model’s token scores before they are converted into a probability distribution for sampling. Lower positive values sharpen the distribution, increasing the relative probability of already likely tokens. Higher values flatten it, giving less likely candidates more probability mass.
Temperature changes how the next token is selected from the model’s scores. It does not retrain the model or add knowledge to the prompt.
Simple example
Suppose the leading candidates for one position have probabilities concentrated around approved, accepted, and several less likely alternatives. With a lower temperature, sampling is more likely to choose the leading candidate repeatedly across runs. With a higher temperature, the alternatives receive a greater chance of selection.
This adjustment happens at each generation step, so a different early token can produce a different continuation.
Why it matters
Temperature lets applications trade repeatability against variation when sampling is enabled. A narrow extraction task may benefit from stable decoding, while generating several candidate phrasings may benefit from more variation.
It should be evaluated with other decoding controls and the actual task. Changing temperature can affect correctness, format adherence, diversity, and the reproducibility of tests.
One important nuance
Temperature is not a creativity setting. It changes distribution sharpness, while the available candidates still come from the model and current context. APIs may treat zero, unsupported ranges, seeds, and interactions with top-p or top-k differently. Some serving systems are not fully deterministic even with a low or zero setting, so do not promise byte-identical output without measuring the complete stack.