BLEU, ROUGE and Other Overlap Metrics for Generated Text
BLEU and ROUGE are classic metrics that measure the overlap of n-grams between generated text and reference text. BERTScore uses contextual embeddings to compute token similarity, correlating better with human judgment. These metrics help evaluate machine translation and summarization systems.
What Are Overlap Metrics?
Overlap metrics compare a candidate text, such as a machine translation or summary, against one or more reference texts written by humans. They count how many words or short sequences of words appear in both the candidate and the reference. The more overlap, the higher the score, under the assumption that good output resembles human-written text.
The BLEU paper by Papineni et al. introduced BLEU for machine translation evaluation. It computes the precision of n-grams, meaning the proportion of n-grams in the candidate that also appear in the reference, and applies a brevity penalty to discourage overly short outputs.
ROUGE, described in Lin's ROUGE package paper, was designed for summarization. It measures recall, the proportion of reference n-grams that appear in the candidate, which suits tasks where the reference contains the key information that should be covered.
How BLEU Works
BLEU calculates a modified n-gram precision for n-grams of lengths one through four. Each n-gram in the candidate is counted only up to the maximum number of times it appears in any single reference, preventing a candidate from inflating its score by repeating common phrases.
The precision scores are combined using a geometric mean, and a brevity penalty is applied if the candidate is shorter than the reference. This penalty reduces the score of outputs that are too concise, as they may omit important content.
The ACL Anthology entry for the BLEU paper provides the full details of the method, including the formula and the experimental setup. The paper reports that BLEU correlates well with human judgments of translation quality.
How ROUGE Works
ROUGE includes several variants, such as ROUGE-N, which measures n-gram recall, and ROUGE-L, which uses the longest common subsequence to account for sentence-level structure. ROUGE-N is commonly reported as ROUGE-1 and ROUGE-2, referring to unigrams and bigrams.
The ROUGE paper by Lin describes the package and its application to summarization evaluation. It compares automatically generated summaries against reference summaries, and the paper shows that ROUGE scores correlate with human ratings of summary quality.
Because ROUGE emphasizes recall, it is sensitive to whether the candidate covers the key points in the reference. This makes it suitable for tasks like summarization, where the goal is to include the most important information rather than to match the reference exactly.
Beyond Exact Overlap: BERTScore
Exact overlap metrics fail when the candidate uses synonyms or paraphrases that differ from the reference. BERTScore addresses this by using contextual embeddings from BERT to compute token similarity, as described in the BERTScore paper on arXiv. Each token in the candidate is matched to the most similar token in the reference based on cosine similarity.
The paper reports that BERTScore correlates better with human judgments than existing metrics and provides stronger model selection performance when evaluated on the outputs of 363 machine translation and image captioning systems. It also shows resilience on an adversarial paraphrase detection task.
BERTScore is not a pure overlap metric, but it is often discussed alongside BLEU and ROUGE because it serves the same purpose: automatic evaluation of generated text. It requires a pretrained language model, which adds computational cost compared to simple n-gram counting.
Using Evaluation Libraries
Implementing these metrics from scratch is possible, but libraries provide convenient and consistent implementations. The Hugging Face Evaluate documentation describes a library that gives access to dozens of evaluation methods for different domains, including NLP, with a single line of code.
The Evaluate library allows users to load metrics, compute scores, and save results in a reproducible way. It also offers interactive demos and documentation cards for each metric, explaining limitations and usage.
For a deeper understanding of the underlying concepts, the Stanford SLP textbook covers language models and evaluation in detail. It is a standard reference for natural language processing and provides context for why these metrics are used.
| Metric | Primary Use | Core Idea | Strengths | Limitations |
|---|---|---|---|---|
| BLEU | Machine translation | N-gram precision with brevity penalty | Fast, correlates with human judgment | Penalizes synonyms, favors exact matches |
| ROUGE | Summarization | N-gram recall and longest common subsequence | Good for coverage of key points | May over-reward repetitive text |
| BERTScore | General text generation | Token similarity via contextual embeddings | Handles paraphrases, better correlation | Requires pretrained model, slower |
Frequently asked questions
What is the main difference between BLEU and ROUGE?
BLEU measures precision, the fraction of n-grams in the candidate that appear in the reference, while ROUGE measures recall, the fraction of reference n-grams that appear in the candidate. BLEU is common for machine translation, ROUGE for summarization.
Why does BLEU apply a brevity penalty?
The brevity penalty reduces the score of candidates that are shorter than the reference, because short outputs can achieve high precision by including only common n-grams while omitting important content.
How does BERTScore differ from BLEU and ROUGE?
BERTScore uses contextual embeddings to compute token similarity, so it can recognize synonyms and paraphrases, whereas BLEU and ROUGE rely on exact n-gram matches. This makes BERTScore more flexible but computationally heavier.
Can these metrics be used for tasks other than translation and summarization?
Yes, they are applied to any text generation task where reference texts exist, such as image captioning, dialogue response generation, and question answering. The choice of metric depends on the task's goals.
Where can one find implementations of these metrics?
Libraries like the Hugging Face Evaluate library provide implementations of many metrics, including BLEU, ROUGE, and BERTScore. The documentation explains how to load and compute them.