How text generation works: next-token prediction one step at a time

Updated October 21, 2026

Large language models generate text by repeatedly predicting the next token given the previous tokens. They use probability distributions over a vocabulary of subword tokens. Decoding strategies like greedy search, beam search, and sampling determine which token to choose at each step.

The core idea: predicting one token at a time

A language model estimates the probability of a token or a sequence of tokens occurring within a longer sequence. For text generation, the model receives a prompt and produces a probability distribution over the next possible token. It then selects one token, appends it to the prompt, and repeats the process until an end-of-sequence token is generated or a maximum length is reached, as explained in the Hugging Face text generation guide.

This autoregressive approach assumes that the probability of an entire sequence can be decomposed into the product of conditional next-token probabilities, given all previous tokens. The Hugging Face decoding methods blog describes this decomposition as the foundation for most contemporary generation systems.

Each step of generation uses only the tokens produced so far, including the original prompt, to determine what comes next. This means the model's output is built incrementally, one token at a time, rather than all at once.

Tokens and vocabulary

The basic units that language models process are called tokens. A token can be a word, a subword, or even a single character. Most modern models use subword tokenization, where prefixes, roots, and suffixes become separate tokens. For example, the word 'unwatched' may be split into 'un', 'watch', and 'ed', as noted in the Google ML Crash Course on large language models.

Tokenization is language specific, and the number of characters per token varies. In English, one token corresponds to roughly four characters or about three quarters of a word. The model's vocabulary is fixed during training, and the generation loop selects from this finite set of tokens at every step.

Tokens are the smallest unit of language modeling, so the model does not see characters or words directly. Instead, it works with token identifiers and maps them to learned embeddings that carry semantic meaning.

Decoding strategies

Once the model produces a probability distribution for the next token, a decoding strategy decides which token to select. Greedy search simply picks the token with the highest probability at each step. This is the simplest method, but it can miss better sequences that start with a slightly less probable token, as demonstrated in the Hugging Face decoding methods blog.

Beam search keeps track of several top hypotheses, usually called beams, and explores multiple possible continuations in parallel. At each step, it retains the `num_beams` most likely sequences, then chooses the one with the overall highest probability at the end. This reduces the risk of missing good sequences hidden behind a low probability first choice.

Sampling selects a token randomly according to the probability distribution. This can produce more varied and creative text, but it also introduces nondeterminism. Temperature scaling adjusts how uniform the distribution is, so a higher temperature yields more diverse outputs.

Training and scaling

Language models learn to predict tokens from large text corpora. The Dive into Deep Learning language models chapter explains that probabilities are estimated from word frequencies in the training data, and that smoothing techniques can handle rare word combinations.

Scaling up models has been shown to improve their ability to perform tasks from just a few examples. The arXiv paper on GPT-3 describes a 175 billion parameter autoregressive model that achieves strong performance on many NLP tasks without any gradient updates or fine-tuning, relying only on text interaction for few-shot demonstrations.

The transformer architecture, which uses self-attention mechanisms, enables models to consider the entire preceding context at once. As explained in the AWS article on GPT, this allows the model to focus on different parts of the input dynamically and generate long, coherent responses.

Limitations and challenges

Greedy and beam search often produce repetitive text. The Hugging Face decoding methods blog notes that this repetition is a common problem in greedy and beam search, and that sampling can offer some relief.

Estimating probabilities for rare word sequences is difficult. The Dive into Deep Learning language models chapter points out that many plausible three-word combinations may never appear in a training corpus, so without smoothing their estimated probability is zero.

Generated text can be fluent and plausible while containing factual errors. The arXiv paper on GPT-3 also acknowledges that the model has trouble with some tasks and datasets, and that its outputs may not always be reliable.

Frequently asked questions

What is a token in language modeling?

A token is the smallest unit of language that a model processes, typically a word, subword, or single character. Most modern models use subwords to balance vocabulary size and meaning.

Why does generated text sometimes repeat itself?

Greedy search and beam search tend to select high-probability tokens that can form loops. Sampling and other randomization methods can reduce this tendency.

How does beam search work?

Beam search maintains multiple candidate sequences, called beams, and explores them in parallel. At each step, it keeps the most likely beams and finally selects the sequence with the highest overall probability.

What is temperature in sampling?

Temperature is a parameter that scales the probability distribution before sampling. A higher temperature makes the distribution more uniform, producing more diverse outputs; a lower temperature makes it more concentrated on the top tokens.

How does a model know when to stop generating?

Models are trained to produce an end-of-sequence token. When that token is selected, generation stops. Alternatively, a maximum token limit can be set by the application.