Sampling, temperature and top-p: how a model picks its next token
Language models assign a probability to every possible next token. Greedy search always takes the most likely token, but sampling introduces random choices. Temperature alters the probability distribution, and top-p (nucleus sampling) restricts the pool to a core set of likely tokens. The chosen decoding strategy shapes coherence, diversity, and creativity.
The model's probability distribution and temperature
According to IBM's article on LLM temperature, a language model assigns a numerical score, called a logit, to every token in its vocabulary, and then converts these scores into a softmax probability distribution. The temperature parameter modifies this distribution: lower temperatures make high-probability tokens even more dominant, while higher temperatures increase the chance of selecting less probable tokens.
As a result, temperature is a key setting for adjusting output determinism. Setting it low makes the model behave more like a deterministic expert, while setting it high introduces more randomness, which can be useful for brainstorming or for generating multiple variations of a phrase.
Greedy and beam search
The simplest decoding strategy is greedy search, which always selects the token with the highest probability. According to the Hugging Face blog on decoding methods, this approach often leads to repetitive output, and it can miss a better sequence that starts with a slightly less probable token. Beam search addresses this by keeping several candidate sequences, called beams, and then choosing the sequence with the highest overall probability.
Beam search is particularly useful for tasks where the output should align with an expected structure, such as image captioning or speech transcription. Greedy search remains a fast baseline but is rarely chosen for open-ended generation because of its tendency to repeat itself.
Sampling from the full vocabulary
Instead of taking the most likely token, sampling randomly draws a token according to the entire probability distribution. The Hugging Face generation strategies documentation explains that multinomial sampling gives every token with non-zero probability a chance to be selected, which reduces repetition and can produce more creative output.
However, the full distribution includes many extremely unlikely tokens. Selecting one of those can lead to incoherent text, so practical implementations often impose additional constraints to keep the sample within a reasonable set of candidates.
Nucleus sampling and top-p
To limit the sampling pool without fixing an arbitrary number, the paper The Curious Case of Neural Text Degeneration proposes nucleus sampling. This method considers only the smallest set of tokens whose cumulative probability exceeds a chosen threshold, and then samples from that set.
This approach, sometimes called top-p sampling, preserves diversity while trimming the long tail of unlikely words. Because the nucleus is dynamic, it adapts to each step's probability distribution, keeping the output fluent and meaningful.
Choosing a strategy for creative and factual tasks
The right decoding method depends on the goal. For short, factual answers, greedy search or a low temperature can provide reliable responses. For open-ended creative writing, sampling with a moderate temperature and a top-p threshold can yield varied and engaging text. Research on hierarchical neural story generation shows that language models can produce coherent stories when given a prompt, but the decoding strategy must allow for enough randomness to avoid dull repetition.
In practice, users can experiment by adjusting temperature from low to high while keeping top-p in a range that blocks the very worst tokens. The optimal settings are often found by testing a few combinations on a sample prompt and inspecting the outputs.
| Strategy | How it selects | Output quality | Typical use |
|---|---|---|---|
| Greedy | Always highest probability | Tends to repeat | Short, factual answers |
| Sampling | Random from full distribution | More diverse, can be incoherent | Creative writing with moderation |
| Top-p sampling | Random from smallest set with cumulative probability above threshold | Diverse yet coherent | Balanced creative and factual |
| Beam search | Keeps multiple sequences, picks highest overall probability | Less repetitive, may still loop | Structured tasks like captioning |
Frequently asked questions
What does temperature do in language generation?
Temperature changes how sharply the probability distribution is focused. Low temperature makes the most likely tokens even more likely, while high temperature spreads the probability among less likely tokens, introducing more randomness.
What is top-p sampling?
Top-p sampling, also called nucleus sampling, restricts random sampling to the smallest set of tokens whose cumulative probability reaches a chosen threshold, and then samples from that set.
How do temperature and top-p work together?
Temperature reshapes the probability distribution, and top-p then limits the sample to a subset of that distribution. They control different aspects of the sampling pool and are often used together.
Why does greedy search sometimes repeat itself?
Greedy search always picks the highest-probability token, which can lock the model into a loop of repeated phrases because it never explores less likely alternatives.
When should beam search be used instead of sampling?
Beam search is suited for tasks where the output should follow a known structure, such as image captioning or speech recognition, because it keeps multiple candidate sequences and selects the one with the highest overall probability.