The Real Reason AI Invents Facts (And How to Make It Stop)

In June 2023, a federal judge in New York did something unusual: he ordered two lawyers to explain why they shouldn't be sanctioned for submitting fake legal cases.

Steven Schwartz had been practicing law for over 30 years. He needed cases to support his client's argument against Avianca Airlines. So he did what millions of professionals now do — he asked ChatGPT.

The AI gave him six cases. Full citations. Court names. Dates. Judicial opinions with compelling legal reasoning. One case, Varghese v. China Southern Airlines, seemed perfect for his argument.

Schwartz, being careful, asked ChatGPT to confirm: "Is Varghese a real case?"

"Yes," ChatGPT replied. It even claimed the case could be found on Westlaw and LexisNexis.

None of the six cases existed. ChatGPT had invented them — complete with fake judges, fake courts, and fake legal reasoning. The result: a $5,000 fine, court sanctions, and two lawyers required to send apology letters to every judge falsely named as an author of the fabricated opinions.

This wasn't a bug. It wasn't a glitch. This is what AI does. And if you're using ChatGPT, Claude, or any AI for research, it's probably happening to you too — you just haven't caught it yet.

Why AI Makes Things Up

Here's the uncomfortable truth: AI doesn't know anything. It guesses.

Think of it like a student who read every textbook ever written but never actually visited a library. When you ask a question, they don't look it up. They guess what a good answer would sound like based on all the patterns they've memorized.

Most of the time, they're right. The patterns are good. But sometimes — especially when you ask about something specific, recent, or obscure — they confidently invent things that don't exist.

This isn't a flaw in the design. It's the design itself.

Large language models are trained to predict the next word. That's it. Their entire goal is to produce text that sounds plausible — not text that's true. They compress the internet into patterns, and compression loses details. Specific facts, exact citations, precise numbers — these get fuzzy.

When an AI doesn't know something, it doesn't say "I don't know." It guesses. And it guesses with the same confident tone it uses when it's right.

The numbers are sobering. According to Vectara's Hallucination Leaderboard, even the best models — GPT-4o, for instance — hallucinate at least 1.5% of the time. Some models hit 25% or higher. That's not a bug to fix. That's baked into the architecture.

The Library Card Solution

So what actually works?

The answer is a technique called Retrieval-Augmented Generation — or RAG. The name is technical, but the concept is simple.

Remember our student who guesses based on patterns? Now imagine giving that student a library card. Before answering your question, they walk to the library, find the actual source, read it, and then respond based on what they found.

That's RAG.

Instead of relying solely on its compressed "memory," the AI first retrieves relevant documents from a knowledge base — your documents, verified sources, or curated databases. It reads those documents. Then it generates an answer grounded in what it actually found.

The difference is fundamental:

• Without RAG: AI with amnesia trying to remember facts
• With RAG: AI with a research assistant handing it the right documents before it speaks

Here's what that looks like in practice. You ask a question. The system searches a database of verified documents. It pulls the most relevant passages. It hands those passages to the AI as context. The AI generates an answer based on that context — and can cite exactly where the information came from.

The proof is in the research. Studies have found that combining RAG with other safeguards can lead to hallucination reductions of 40% to 96%, depending on implementation. A cancer information study published in JMIR Cancer saw hallucination rates drop from 40% to essentially zero when the AI was grounded in verified medical sources.

That's not incremental improvement. That's a different category of reliability.

What This Actually Means for Your Work

If you're using AI for anything where accuracy matters — research, client work, documentation, analysis — this distinction is everything.

Think of it as a reliability spectrum:

Tier 1: Raw AI (ChatGPT, Claude without context)
Good for brainstorming. Good for first drafts. Good for editing suggestions. Terrible for facts. Never trust a citation without verifying it yourself. The Avianca case is what happens when you skip this step.

Tier 2: AI with RAG
The AI is grounded in your documents or verified sources. Hallucination rates drop dramatically. You're not asking "What does the AI know?" — you're asking "What can the AI look up?" Much safer for research and synthesis.

Tier 3: AI with RAG plus verification
The gold standard. The AI retrieves sources, generates answers, and shows you exactly where each claim came from. You can verify claims against the original documents. This is what you want for high-stakes work.

The mindset shift is subtle but critical. Stop treating AI like an oracle that knows things. Start treating it like a research assistant that can look things up — if you give it the right sources to search.

If accuracy matters, you need AI that retrieves before it generates.

The Beginning of the Journey

AI hallucinations aren't bugs. They're features of how large language models work. These systems predict plausible text, not truthful text. When they don't know, they guess — and they guess confidently.

RAG changes the equation by giving AI access to actual sources instead of just pattern-matching from memory. The research is clear: grounding AI in verified documents dramatically reduces hallucinations.

The lawyer in the Avianca case learned this lesson the expensive way. You don't have to.

This article explained the why — why AI makes things up and why retrieval is the solution. But how does RAG actually work under the hood? In the next article, we'll break down the four components that make RAG tick: how documents get ingested, how they're converted to searchable embeddings, how retrieval finds the right context, and how generation produces grounded answers.