AI Hallucinations Explained: Technical Causes and Legal Consequences
If you want to use AI tools responsibly, you need to understand why they invent things. Not at PhD level — but enough to recognize that this is an architectural feature, not a fixable bug.
I have been building software systems since the early 2000s. I have watched a lot of technology get oversold. But AI hallucinations represent something different from the usual gap between marketing and reality. They are not a limitation that better engineering will eliminate. They are a fundamental property of how large language models work.
Understanding this matters because it changes how you should use these tools. The difference between "this tool occasionally makes mistakes" and "this tool generates statistically plausible text without any mechanism for verifying its own accuracy" is the difference between a tool you can trust with verification and a tool you should never trust without it.
What Actually Happens Inside the Model
Large language models are, at their core, next-token prediction engines. Given a sequence of text, the model predicts the most probable next word (technically, next token) based on patterns in its training data.
When you ask an LLM a legal question, it does not look up the answer in a database. It does not consult a case reporter. It generates a sequence of words that is statistically probable given the question you asked and the patterns it learned during training.
This means the model can generate a citation that looks perfect — correct format, plausible case name, reasonable-sounding court and year — without any mechanism to verify whether that case exists. It generates the citation because "Smith v. Jones, 482 F.3d 1127 (9th Cir. 2007)" is the kind of character pattern that commonly follows the kinds of legal statements it has been generating. The format is right because the model has seen millions of citations. The content may be entirely fabricated because the model has no way to check.
This is not a bug. It is the architecture.
The Two Failure Modes
The Stanford study on legal AI hallucinations identified two distinct types, and the distinction matters for how you protect yourself.
Incorrect information. The AI describes the law incorrectly. It states a wrong holding, mischaracterizes a procedural posture, or fabricates a statutory provision. This is the obvious failure mode — the kind that a careful reader might catch because the legal statement itself seems wrong.
Misgrounding. The AI describes the law correctly but cites a source that does not support the claim — or does not exist at all. This is the insidious failure mode. The legal analysis reads well. The proposition is accurate. But the citation attached to it is disconnected from the proposition. A reviewer who confirms that the legal statement is correct may assume the citation must also be correct. It is not.
In my experience evaluating AI systems, misgrounding is the more dangerous failure mode precisely because it passes the "does this seem right?" test that most human review applies as a first filter.
The Stanford Numbers
The Stanford Legal RAG Hallucinations study tested the major legal AI platforms:
- Lexis+ AI hallucinated on 17% of queries
- Westlaw AI-Assisted Research hallucinated on 33% of queries
- General-purpose models (ChatGPT) hallucinated on 58-82% of legal queries
Before this study, vendors made aggressive claims. LexisNexis advertised "100% hallucination-free linked legal citations." Thomson Reuters claimed RAG "dramatically reduces hallucinations to nearly zero."
The researchers' conclusion: "The providers' claims are overstated."
I want to be fair to the vendors. These are purpose-built legal AI tools with retrieval-augmented generation — they do search real databases before generating responses. They are substantially better than general-purpose ChatGPT. But "better than ChatGPT" is a low bar for professional legal research. A 17-33% hallucination rate means that roughly one in four to one in six AI-assisted research queries will contain an error.
Why RAG Does Not Solve It
Retrieval-augmented generation is the industry's primary response to the hallucination problem. The idea: instead of generating purely from the model's internal patterns, first retrieve relevant documents from an authoritative database, then generate a response grounded in those documents.
In theory, this should eliminate hallucinations. In practice, it reduces them but does not eliminate them. Here is why, from an engineering perspective:
Retrieval can fail. The search query generated by the model may not retrieve the most relevant documents. Or it may retrieve documents that are tangentially related but not directly on point.
The model can misinterpret retrieved content. Even when the right documents are retrieved, the model may mischaracterize their content during response generation. The generation step is still probabilistic text production — having the right source document in the context window does not guarantee the model will accurately summarize it.
Citation linking can fail. The model may generate a correct legal statement from one retrieved document and then attach a citation from a different retrieved document. The statement is right. The source is right. The connection between them is wrong.
RAG is a significant improvement over raw generation. It is not a complete solution. If someone tells you their RAG system has "eliminated" hallucinations, ask them to define their testing methodology and show their numbers.
The Real-World Consequence Trajectory
The Mata v. Avianca case (2023) was the opening act. Steven Schwartz submitted six fabricated citations from ChatGPT. He then asked ChatGPT to confirm they were real — it did — and submitted that confirmation to the court. Judge Castel imposed sanctions and found "subjective bad faith."
The case count has grown substantially. A database maintained by Damien Charlotin now documents over 300 instances worldwide. The trend in judicial response is clear:
Early cases (2023-2024): warnings, modest fines, CLE requirements. Courts treated these as errors of ignorance.
Middle period (2024-2025): increasing fines, fee disgorgement, mandatory AI policies. Courts treated these as errors of negligence.
Recent cases (2025-2026): the court in Johnson v. Dunn explicitly declared that "monetary sanctions are proving ineffective at deterring false, AI-generated statements of law in legal pleadings." The next phase involves bar referrals and potential license suspension.
The trajectory is unmistakable. Courts are running out of patience.
Why Better Models Will Not Fix This
This is the point where optimists usually argue that next year's model will be much better. And they are right — next year's model will hallucinate less frequently. But "less frequently" is not "never."
The fundamental architecture of autoregressive language models — predicting the next token based on probability distributions — does not include a verification step. The model does not check its output against ground truth before producing it. It cannot, because ground truth checking requires a different kind of system than token prediction.
Will we eventually have AI systems that reliably verify their own output? Possibly. Current architectures do not support it. Multi-agent systems where one model generates and another verifies are promising but add latency and cost, and the verifier is itself a language model subject to the same limitations.
For the foreseeable future — and I am talking years, not months — verification by humans against authoritative sources remains the only reliable quality assurance mechanism.
What This Means for Practice
If you understand the technical reality, the practical implications are clear:
Every AI-generated citation requires independent verification. Not because the tool is bad, but because the architecture cannot guarantee accuracy. This is not a best practice. It is a professional obligation.
Different tools hallucinate at different rates and in different ways. The Stanford data provides baselines. Your own evaluation in your specific practice areas will be more informative. Test your tools. Know their failure patterns.
RAG is helpful but not sufficient. The 17% hallucination rate for Lexis+ AI demonstrates that even purpose-built legal AI with proprietary databases still fails one time in six. Build your workflow for that reality, not for the marketing claim.
Documentation matters. Record when you use AI, what tool you used, and what verification you performed. This documentation may be the difference between a malpractice claim and a defensible practice record.
Stay current. Models change. Tools update. Hallucination patterns shift. The duty of competence includes understanding the current state of the technology you use. This is a continuing obligation, not a one-time education.
The Bottom Line
AI hallucinations are not a temporary problem that better technology will solve in the next release cycle. They are an inherent property of probabilistic text generation that current architectures cannot eliminate, only reduce.
Understanding this does not mean avoiding AI tools. It means using them with clear-eyed awareness of their limitations and building verification into every workflow that produces output for external use.
The tools are genuinely useful. They are not genuinely reliable. Those two truths coexist, and responsible practice requires holding both simultaneously.
Key Takeaways
- AI hallucinations are an architectural feature of large language models, not a bug — they occur because LLMs generate statistically probable text without internal verification
- Stanford research found 17-33% hallucination rates for specialized legal AI tools; vendor claims of "hallucination-free" are empirically disproven
- Two failure modes: incorrect information (wrong law) and misgrounding (right law, wrong citation) — misgrounding is more dangerous because it passes superficial review
- RAG reduces but does not eliminate hallucinations — retrieval failures, misinterpretation of retrieved content, and citation-linking errors persist
- Verification by humans against authoritative sources is the only reliable quality assurance mechanism for the foreseeable future