TwinLadder Weekly

Issue #16 | September 2025

---

Editor's Note

I keep a folder on my desktop called "near misses." It contains drafts, memos, and research outputs where AI produced plausible but wrong content that I caught before it went anywhere. A fabricated case citation in a research memo. A real case with a misattributed holding. A statutory provision quoted accurately but from a superseded version of the statute.

I catch these because I have twenty years of practice to draw on. I know when something does not sound right, even when the citation checks out superficially. The question that keeps me awake is whether the next generation of lawyers — trained on AI from their first day — will develop that instinct.

Damien Charlotin's AI hallucination database has grown to 486 cases globally, with 324 in the United States alone. The growth rate has accelerated from two cases per week to two or three per day. These are not edge cases any more. They are a pattern. And the pattern reveals something more troubling than lawyers being careless. It reveals a profession increasingly unable to verify what it does not independently know.

For European practitioners, the hallucination problem carries an additional dimension. Under the EU AI Act, Article 4 requires documented AI literacy for anyone deploying AI systems. A firm whose lawyers cannot identify hallucinated output is not merely at risk of sanctions. It is non-compliant with European law that has been in force since February 2025.

---

[HIGH CONFIDENCE]

486 Cases and Counting: What the Hallucination Database Reveals

Charlotin, an independent practitioner at Pelekan Data Consulting and Research Fellow at HEC's Smart Law Hub, has built what has become the definitive global tracker of AI hallucination cases in legal proceedings. The data tells a clear story, and it is not the one you would expect.

Hallucination Data	Scale of the Problem
486 cases globally (Charlotin database)	Growth from 2/week to 2-3/day
324 in the United States	90% involve solo practitioners or small firms
17% hallucination rate (Lexis+ AI, best-in-class)	34% hallucination rate (Westlaw AI-Assisted Research)
58-88% hallucination rate (general-purpose LLMs)	No purpose-built legal AI tool is hallucination-free
$3,000-$31,000+ sanctions range	Escalating — courts are losing patience

The failure modes are more varied than "fake citations." Charlotin identifies three distinct categories:

Fabricated cases — the AI invents non-existent citations. These are the easiest to catch. The case simply does not exist. A basic Westlaw or Lexis search reveals the fabrication in seconds. This is the Mata v. Avianca scenario, and it is the one that gets media attention.

Fake quotes — a real case with a fabricated quotation. Harder to catch, because the case exists and the citation is valid. The lawyer checks the case name, sees it is real, and does not verify the specific quotation. This requires reading the actual decision, not just confirming the citation.

Misattributed reasoning — the citation is correct, the case is real, but the legal principle attributed to it is not supported by the decision. This is the most dangerous category because it requires substantive legal analysis to detect. The very skill the lawyer was hoping AI would substitute for is the skill needed to catch the error.

Who gets caught? The database reveals a stark pattern. 90% of sanctioned lawyers are solo practitioners or small firms. 56% represent plaintiffs; 31% represent defendants. Sanctions range from $3,000 to $31,000+. The concentration in small firms is not mysterious — larger firms have developed governance policies, mandatory verification workflows, and training programmes. Solo practitioners often lack these structures.

This concentration should concern European regulators. The EU AI Act applies equally to solo practitioners and Magic Circle firms. But the solo practitioner in Vilnius using ChatGPT for research has far fewer resources to build verification workflows than Freshfields. The regulatory obligation is uniform. The capacity to comply is not.

The cases keep coming. In July 2025, Judge Nina Wang in Denver ordered MyPillow attorneys Christopher Kachouroff and Jennifer DeMaster to pay $3,000 each for filing a motion containing more than two dozen mistakes, including hallucinated cases. The attorneys claimed the filing was an accidental draft. The "final" version was still riddled with errors. The court found their explanations not credible. Commentators noted the sanction was light — a $31,000 sanction in another case better reflects the current direction. Courts are losing patience. Sanctions are escalating.

In Canada, Ko v. Li (2025 ONSC 2766) established a template for court responses. A Toronto lawyer submitted a factum in matrimonial proceedings with case citations that could not be found on any legal database. Her admission was frank: "These authorities were drafted using a legal research tool powered by artificial intelligence. I made the grave mistake of failing to verify the case law." The court initiated contempt proceedings, ultimately dismissing them on conditions including CPD courses and refunding the client. Ontario now requires lawyers to certify the authenticity of every authority cited in factums under Rule 4.06.1(2.1).

In Australia, a solicitor was prohibited from unsupervised practice for two years for AI-related failings. The pattern is global: warnings, then sanctions, then professional consequences, then — inevitably — civil liability.

A parallel development deserves attention. In Walters v. OpenAI, a Georgia court ruled that OpenAI is not liable when ChatGPT fabricates defamatory content about a real person, finding that the platform's disclaimers about potential inaccuracies meant no reasonable reader would interpret the output as factual assertion. The ruling effectively says: AI companies warned you. If you relied on unverified AI output, that is your problem.

That framing transfers directly to legal practice. AI vendors include disclaimers. Ethics rules require verification. Courts impose sanctions on lawyers, not on tools. The entire accountability structure assumes lawyers will exercise independent judgment — which brings us to the fundamental problem.

Even purpose-built legal AI tools hallucinate at significant rates. The Stanford study (Magesh et al., 2025) tested dedicated legal research platforms. General-purpose LLMs hallucinated at rates of 58-88%. But even Lexis+ AI, the best performer, showed a roughly 17% error rate. One in six queries. For a profession that demands perfection in citations, that rate is extraordinary. No lawyer would accept a 17% error rate from a human research assistant.

The solution is not finding a "hallucination-free" tool. No such tool exists. The solution is building verification into every workflow — not as an afterthought, but as a structural element of how AI-assisted work is produced.

---

The Competence Question

Here is the paradox at the heart of the hallucination problem: the skill required to verify AI output is the same skill that AI use gradually erodes.

Consider a fourth-year associate who has used AI for legal research since her first day of practice. She has never spent a full day in a law library. She has never built a research trail from scratch — starting with a secondary source, identifying key cases, reading them in full, tracing citations forward, and constructing an argument from primary materials. Her research process begins with a prompt and ends with a verification check.

Now give her an AI memo with a correct citation to a real case, but with a legal proposition that the case does not actually support. The third category of hallucination — misattributed reasoning. To catch this, she would need to read the cited decision, understand its holding, and assess whether the proposition attributed to it is genuinely supported. That is not a verification skill. That is a legal analysis skill. It develops through years of doing the research that AI now handles.

The firms that avoid sanctions are not just the ones with better checklists. They are the ones where lawyers have the substantive knowledge to recognise when AI output is wrong — which means they are the ones where lawyers developed that knowledge before AI made it optional.

I spoke with a mid-sized US litigation practice that has maintained zero AI-related sanctions. Their protocol is straightforward: every citation is pulled from original sources before use, quotations are verified verbatim, and a second attorney signs off on AI-assisted filings. But the partner in charge told me the protocol is not what actually prevents errors. "The protocol catches the easy stuff. What catches the hard stuff is that our lawyers know the law well enough to notice when something does not smell right. You cannot protocol your way to that."

Article 4 of the EU AI Act implicitly recognises this. "Sufficient AI literacy" is not just knowing how to use a tool. It is knowing enough about the domain to evaluate what the tool produces. The hallucination database is, in a very real sense, a catalogue of insufficient AI literacy — of lawyers who knew how to use a tool but not how to verify its output.

---

What To Do

1. Pull up the Charlotin database and review it. The hallucination cases tracker is free. Spend thirty minutes reviewing recent cases in your jurisdiction and practice area. Understanding how others have been sanctioned is the most effective prevention.

2. Implement a three-layer verification protocol. For any AI-assisted filing: (a) confirm the citation exists in an official reporter, (b) verify that direct quotations appear verbatim in the source, (c) read the cited decision's actual holding and confirm it supports the proposition. Layer three is where most failures occur and most sanctions result.

3. Never let AI verify its own output. If you ask an LLM whether its citations are accurate, it will confidently confirm them — even when they are fabricated. The court in Ko v. Li was explicit: "The attorney cannot outsource professional responsibility to a chatbot." Independent source confirmation is non-negotiable.

4. Invest in developing judgment, not just process. Verification checklists catch known error types. Professional judgment catches everything else. Ensure your junior lawyers are building substantive legal knowledge — through close reading, from-scratch research exercises, and mentoring — not just learning to operate AI tools.

5. Document your verification procedures for Article 4 compliance. European firms must demonstrate that staff deploying AI have "sufficient AI literacy." A documented verification protocol — showing that lawyers are trained to identify all three categories of hallucination — serves dual purpose: quality assurance and regulatory compliance.

---

Quick Reads

• Charlotin's AI Hallucination Cases Database — the definitive tracker, free and regularly updated, now covering 486+ cases globally.

• NPR: MyPillow lawyers sanctioned for AI hallucinations — the July 2025 sanctions with useful detail on how courts assess AI-related misconduct.

• Stanford HAI: AI on trial — the benchmark study showing 17%+ hallucination rates even in purpose-built legal AI tools, essential reading for anyone relying on these platforms.

• McCarthy Tetrault: Ko v. Li analysis — thorough analysis of the Canadian case that produced Ontario's new certification requirement.

• Eric Goldman: Walters v. OpenAI — the ruling that effectively shields AI companies from liability for hallucinations, leaving accountability entirely with the human users.

---

One Question

If verification requires the same legal judgment that AI use is designed to supplement, and if that judgment develops through the very practice AI now handles, who will be competent to verify AI output in ten years?

---

TwinLadder Weekly | Issue #16 | September 2025

Compliance is the floor. Competence is the mission.