Perplexity: how language models are scored on predicting the next word
Perplexity (PPL) is a common evaluation metric for autoregressive language models. It is defined as the exponentiated average negative log-likelihood of a sequence. Lower perplexity means the model is more confident in its predictions, and the metric is sensitive to tokenization and context length.
What perplexity measures
Perplexity is a metric used to evaluate how well a language model predicts a sequence of tokens. It is especially relevant for autoregressive models, which generate text token by token, because it measures the model's confidence in the next token given the preceding context.
The metric is defined as the exponentiated average negative log-likelihood of a sequence. In simple terms, a lower perplexity means the model assigns higher probability to the actual next tokens, indicating better predictive performance.
Calculating perplexity for fixed-length models
Modern language models have a maximum context size, so they cannot condition on an entire long document. The Hugging Face documentation explains that a fixed-length model's perplexity should be evaluated with a sliding-window strategy, where the context window moves over the sequence so each prediction uses as much preceding context as possible. Hugging Face, Perplexity of fixed-length models
A naive approach splits the text into disjoint chunks and computes the log-likelihood of each chunk independently. This is faster but gives a higher, less accurate perplexity because the model has less context at many steps. A strided sliding window, which moves the context by larger strides, offers a practical compromise between speed and accuracy.
Perplexity among other evaluation metrics
Perplexity is not the only metric used to assess language models. The Google for Developers Machine Learning Glossary defines classification metrics such as accuracy, precision, and recall, which measure correctness of categorical predictions rather than likelihood of a sequence. Google Machine Learning Glossary
The Holistic Evaluation of Language Models (HELM) project argues that a single metric is insufficient, and it measures seven metrics including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across many scenarios. This multi-metric approach highlights that perplexity, while useful for intrinsic language modeling, does not capture all aspects of model quality. arXiv: Holistic Evaluation of Language Models
Foundational textbooks, such as the open Speech and Language Processing book by Jurafsky and Martin, cover perplexity within the chapter on n-gram language models, establishing it as a standard measure in natural language processing. Stanford Speech and Language Processing
Tools for computing perplexity
Several software frameworks support perplexity evaluation. For example, the EleutherAI evaluation framework provides a unified way to test generative language models on many benchmarks, and it can be used to compute perplexity along with other metrics. EleutherAI's evaluation framework
Hugging Face's Transformers library provides code examples for computing perplexity with models such as GPT-2, demonstrating the sliding-window technique and how to load tokenizers and datasets.
| Metric | What it measures | Typical use case |
|---|---|---|
| Perplexity | How well a model predicts the next token in a sequence | Autoregressive language models |
| Accuracy | Number of correct predictions divided by total predictions | Classification tasks |
| Precision | Among positive predictions, the proportion that are correct | Classification, especially imbalanced datasets |
| Recall | Among actual positive instances, the proportion that are predicted correctly | Classification, especially imbalanced datasets |
Frequently asked questions
What does a perplexity score indicate?
Lower perplexity indicates the model is more confident in predicting the next token, meaning better predictive performance.
Why does tokenization affect perplexity?
Because perplexity is calculated on tokenized sequences, and different tokenizers may split text differently, which changes the number of tokens and the conditioning contexts.
Is perplexity comparable between models with different context lengths?
Not directly, because models with larger context windows have more information, so comparing raw perplexity scores requires careful consideration of context size and tokenizer.
What is the relationship between perplexity and cross-entropy?
Perplexity is the exponentiation of cross-entropy, as stated in the Hugging Face documentation.