Definition
Top-k sampling limits the next-token candidate set to the k tokens with the highest probabilities. All other candidates are removed, the remaining probabilities are normalized, and one token is sampled from that fixed-size set.
At each generation step, the model produces new next-token scores based on the tokens so far. The decoder uses those scores to keep the top k, so the candidates can change from step to step even though the limit stays fixed.
Simple example
With k = 5, the decoder retains only the five most likely tokens for the current position. If the model assigns most probability to two tokens, three much less likely candidates still remain eligible. At a more ambiguous position, only five candidates remain even if 20 tokens have similar probabilities.
Temperature can also change the relative probabilities used for sampling, independently of the top-k limit.
Why it matters
Top-k provides a direct bound on how many token candidates can be sampled. It can prevent selections from the long low-probability tail and can make generation behavior easier to reason about than sampling from the full vocabulary.
Its fixed size is also a limitation. The same value may be too broad for a confident step and too narrow for an uncertain one, so task-level evaluation matters more than a universal default.
One important nuance
Top-k sampling is unrelated to retrieval top-k. Sampling top-k filters candidate tokens during generation. Retrieval top-k controls how many documents or passages a retrieval stage returns. Both use a ranked list and a count named k, but they operate at different stages and changing one does not change the other.