Evaluation & Safety
Hallucination
When an AI model generates information that sounds plausible but is factually incorrect, fabricated, or unsupported by its training data or provided context.
Why it matters
Hallucination is the primary barrier to deploying AI in high-stakes domains like healthcare, legal, and finance. Understanding and mitigating it is essential for production AI.
Why models hallucinate
LLMs are fundamentally next-token predictors. They generate text that is statistically likely, not necessarily true. When the model lacks sufficient knowledge or context, it fills gaps with plausible-sounding fabrications rather than admitting uncertainty.
Types of hallucination
- Factual fabrication — inventing facts, citations, or statistics that don't exist.
- Context contradiction — generating output that contradicts the provided context or documents.
- Instruction drift — gradually departing from the user's instructions over long outputs.
Mitigation strategies
- RAG — ground responses in retrieved documents.
- Citations — require the model to cite specific sources for claims.
- Confidence calibration — train models to express uncertainty.
- Guardrails — post-generation fact-checking and consistency validation.
Related Terms
Retrieval-Augmented Generation(RAG)— A technique that grounds a language model's output in external data by retrieving relevant documents before generating a response.Large Language Model(LLM)— A neural network trained on massive text corpora that can understand and generate human language, typically with billions of parameters.Guardrails— Programmatic constraints placed around AI model inputs and outputs to prevent harmful, off-topic, or policy-violating behavior.
On the AI Radar
Agentic RAG— Embeds autonomous AI agents into the RAG pipeline, replacing static retrieve-then-generate workflows with systems that plan retrieval strategies, use multiple tools, reflect on evidence, and iterate across multiple search steps for complex multi-hop queries.Contextual Retrieval— A technique from Anthropic that prepends chunk-specific explanatory context to each document chunk before embedding and BM25 indexing. Uses an LLM to generate a short context snippet situating each chunk within its source document, improving retrieval accuracy significantly.