Human evaluation of AI output: ratings, preferences and agreement
Human evaluation of AI output typically uses direct ratings, pairwise preference judgments, and agreement metrics. Research shows that evaluation practices vary widely across studies, which makes comparison difficult. Standardized definitions and detailed reporting are essential for reliable and reproducible results.
The role of human evaluation
Human assessment remains the most trusted form of evaluation for AI-generated text, but the field lacks uniform practices. A study of 165 natural language generation papers found more than 200 different terms used to describe evaluated quality aspects, which makes it difficult to compare findings across research projects. the ACL Anthology paper documents this diversity and calls for standardized definitions.
Three common approaches can be distinguished: direct ratings, pairwise preference judgments, and agreement metrics. Ratings ask a rater to assign a score to a single output on a defined scale. Preference judgments present two outputs and ask which is better. Agreement metrics quantify how consistently different raters judge the same material.
Each approach has strengths and weaknesses. Ratings can capture fine-grained differences but are sensitive to the wording of criteria and scale. Preferences are easier for raters but provide only binary information. Agreement metrics help gauge reliability but depend on the number and background of raters.
Rating scales and quality criteria
Rating studies typically ask humans to judge outputs on qualities such as fluency, coherence, helpfulness, or truthfulness. The variety of terms used in the literature is a major obstacle, since the same concept can be named differently across papers. Clear evaluation sheets with explicit definitions and instructions are needed to support reproducibility.
A rating scale should be defined precisely: the number of points, the meaning of each point, and the instructions given to raters. Without such details, results are hard to interpret or compare. Many papers omit these details, which contributes to the inconsistency noted in the field.
Raters may also interpret scales differently depending on their background and experience. Pilot testing and rater training can reduce variability. Reporting the number of raters and their level of expertise helps readers assess the strength of the evidence.
Preference-based evaluation
Pairwise comparison is a popular alternative to numeric ratings. The Chatbot Arena platform lets users see two model responses and vote for the one they prefer. It has collected over 240K votes since its launch, and the Chatbot Arena paper confirms that the crowdsourced votes agree well with judgments from expert raters.
This approach reduces the cognitive burden on raters because they only need to make a relative choice rather than assign an absolute score. The platform also uses statistical methods to rank models from the pairwise votes, which yields a leaderboard that many developers cite.
Human feedback can also guide model training. In the InstructGPT work, labelers ranked outputs from different model sizes, and these rankings were used to fine-tune the model through reinforcement learning from human feedback. the InstructGPT paper reports that a 1.3 billion parameter model was preferred over a 175 billion parameter model, despite having far fewer parameters.
Agreement and reliability
Measuring agreement between raters is essential for understanding the reliability of human evaluations. Common statistics include Cohen's kappa for two raters and Fleiss' kappa for more than two. These metrics range from chance-level agreement to perfect agreement, and they help readers know whether the ratings are consistent.
The Chatbot Arena research explicitly checks the agreement between non-expert crowdsourced voters and expert raters, finding that the two groups align closely. This agreement lends credibility to crowdsourced evaluation platforms.
A comprehensive evaluation framework, HELM, emphasizes measuring multiple metrics beyond accuracy, such as calibration, robustness, fairness, and toxicity. the HELM paper highlights that standardized conditions and transparent reporting are necessary for meaningful comparison across models and studies.
Challenges in implementation
Building a reliable human evaluation is difficult in practice. Anthropic's guide discusses several obstacles, such as contamination in popular multiple-choice benchmarks and sensitivity to formatting changes, which can shift accuracy by a noticeable amount. Anthropic's evaluation guide also describes the complexity of implementing bias benchmarks like BBQ.
Even simple evaluations require careful judgment calls about instructions, scale design, and data collection. A benchmark that seems straightforward may hide subtle flaws that undermine its validity. For example, the guide notes that a benchmark like MMLU can be contaminated if the model sees the questions during training.
To improve the field, researchers recommend standardizing definitions, using evaluation sheets, and reporting all relevant details. These measures make human evaluations more transparent, reproducible, and comparable across different studies and systems.
| Method | Typical setup | Reported finding or concern |
|---|---|---|
| Ratings | Score a single output on a numeric scale | Many different quality terms are used, making comparison difficult |
| Pairwise preference | Choose the better of two outputs | Crowdsourced votes agree with expert ratings in Chatbot Arena |
| Agreement metrics | Calculate inter-rater reliability | Consistency is often reported but methods vary across studies |
Frequently asked questions
What is a pairwise preference evaluation?
It is a method where a human rater sees two AI outputs, such as two chat responses, and chooses which one is better. The Chatbot Arena platform uses this approach to rank models based on many pairwise votes.
How is rater agreement measured?
Agreement is typically measured with statistics like Cohen's kappa or Fleiss' kappa, which compare the observed agreement to chance-level agreement. The Chatbot Arena study also checks agreement between crowdworkers and expert raters.
Why is standardization important in human evaluation?
Without standard definitions and reporting practices, results from different studies cannot be compared reliably. The ACL study found over 200 different quality terms and calls for evaluation sheets and standardized terminology.
What challenges do practitioners face when building evaluations?
Anthropic's guide notes that benchmark contamination, formatting sensitivity, and incomplete implementations can distort results. Bias benchmarks like BBQ require careful implementation and interpretation.
What is a common mistake in reporting human evaluations?
Many papers omit details about the rating scale, instructions, number of raters, and how criteria were defined. This makes it impossible to replicate the evaluation or compare it with other studies.