Why language models hallucinate and the methods used to reduce it
Language models hallucinate because they predict plausible next tokens from training patterns rather than check facts. Imperfect training data, a lack of grounding, and design limits all contribute. Researchers and vendors have developed benchmarks to measure truthfulness and techniques such as explicit uncertainty prompts, retrieval grounding, and citation verification to reduce hallucinations.
What an AI hallucination is
An AI hallucination is an output that sounds plausible but is factually wrong, irrelevant, or fabricated. The IBM explanation of AI hallucinations notes that a large language model can present fake facts, invented studies, or nonexistent URLs as if they were accurate.
Text models are trained to predict which word or symbol comes next based on statistical regularities found in their training text. When information is sparse or ambiguous, the model may generate details that fit its observed patterns instead of reflecting actual events.
Hallucinations are a known barrier to reliable deployment, especially in healthcare, finance, and legal work. They can be reduced but not fully eliminated, as they are partly inherent to how generation works.
Why models fabricate content
Insufficient or biased training data is a common cause. Google's discussion of AI hallucinations says models learn patterns from data, so if the data has gaps or errors, the model may learn incorrect patterns and make inaccurate predictions.
A lack of grounding also contributes. Models may struggle to incorporate real-world knowledge or factual information, and this can lead to outputs that are plausible but incorrect, such as fabricating links to web pages that never existed.
Additionally, models are designed to select the most statistically likely continuation rather than to verify truth. Subtle criteria like legal validity or scientific correctness are not directly encoded, so the model prioritizes pattern match over factual accuracy.
Benchmarks and detection methods
The TruthfulQA benchmark paper on arXiv created a set of questions to measure whether models mimic human falsehoods. It contains 817 questions spanning 38 categories, including health, law, finance, and politics.
In that benchmark, the best model tested was truthful on 58% of questions, while human performance was 94%. The paper also found that larger models were generally less truthful, suggesting that scaling alone does not improve truthfulness.
The arXiv survey on hallucination in large language models proposes a taxonomy of hallucination factors and reviews detection methods and benchmarks. It also discusses challenges and open questions, including hallucinations in vision-language models and knowledge boundaries.
Techniques to reduce hallucinations
Prompting techniques can help. Anthropic's documentation on reducing hallucinations recommends explicitly giving the model permission to admit uncertainty, which can drastically reduce false information.
For long documents, the guide suggests extracting direct quotes before performing a task, and then using those quotes as the only basis for analysis. Verification with citations is another technique: the model must find a supporting quote for each claim or retract it.
Restricting the model to provided documents instead of general knowledge also reduces fabrication. Beyond prompting, the NIST Generative AI Profile provides a cross-sectoral risk management framework that helps organizations incorporate trustworthiness considerations into design and evaluation.
These methods reduce but do not eliminate hallucinations. The NIST profile is intended for voluntary use and supports improving the ability of organizations to address trustworthiness throughout the AI lifecycle.
| Technique | How it works |
|---|---|
| Permit uncertainty statements | Explicitly allow the model to say it does not know. |
| Use direct quotes | Extract exact text from documents before reasoning. |
| Verify with citations | Require a supporting quote for each claim. |
| Restrict to provided documents | Forbid use of general knowledge. |
| Limit outcomes with regularization | Penalize extreme predictions to reduce overfitting. |
| Train on relevant sources | Use data specific to the task. |
Frequently asked questions
What causes AI hallucinations?
Insufficient training data, lack of grounding, and the model's tendency to predict statistically likely text rather than verify facts are common causes.
How is hallucination measured?
Benchmarks like TruthfulQA present questions that humans often answer falsely, then check if the model avoids mimicking those falsehoods.
Can hallucinations be completely eliminated?
No, they can be reduced but not eliminated. Techniques like prompting for uncertainty and grounding in documents lower the frequency.
What role does grounding play?
Grounding connects the model to external, verifiable information, which helps prevent it from inventing details like nonexistent citations.