Definition
Top-p sampling, also called nucleus sampling, limits token sampling to a probability mass rather than a fixed number of candidates. Tokens are ordered from highest to lowest probability, then the smallest prefix whose cumulative probability reaches the configured value p is retained. The next token is sampled from that set after normalization.
With p = 0.9, the retained set contains at least 90 percent of the probability mass presented to top-p filtering at that step.
Simple example
At one position, the first two tokens may already have probabilities of 0.55 and 0.36. A top-p value of 0.9 needs only those candidates. At another position, probability may be spread across 40 plausible tokens, so many more must remain to reach the same threshold.
The decoder recalculates the candidate set for each new token using the next-token distribution for the updated sequence.
Why it matters
Top-p cuts off the low-probability tail while allowing the candidate count to change between generation steps. A spread-out next-token distribution leaves more choices than a concentrated one.
Use p to tune output variation, then test the result with your prompts, temperature setting, and provider defaults.
One important nuance
The number of retained tokens is dynamic. Top-p does not mean “keep the top 90 percent of vocabulary entries”, and a value of 0.9 does not guarantee a fixed level of quality or diversity. Providers can also differ in the order and exact handling of decoding controls, so the same numeric configuration is not necessarily portable across models or APIs.