Branchoria AI Sense AI Sense

Clear, practical explanations of what AI does, where it helps, and where it fails.

Within Avianca case

Why AI Cannot Verify Its Own Fake Cases

The Avianca filing shows why asking an AI system to confirm its own citations can make a false source look even more convincing.

On this page

  • How the fake authorities were challenged
  • Why the same tool repeated the error
  • Safer ways to verify legal citations
Preview for Why AI Cannot Verify Its Own Fake Cases

Introduction

The most revealing mistake in the Mata v. Avianca affair was not that attorney Steven Schwartz received fake case citations from ChatGPT. It was what happened after those citations were questioned. Rather than independently checking the authorities in established legal databases, Schwartz returned to the same AI system and asked whether the cases were genuine. ChatGPT assured him that they were. Those assurances were false. The episode became a textbook example of a self-check trap: using a system that generated an error to verify the accuracy of that same error. The result showed why large language models can appear persuasive even when they are wrong, and why apparent confidence is not the same as verification. [Scottish Legal News]scottishlegal.comScottish Legal NewsLawyer led astray by ChatGPT apologises to court | Scottish Legal NewsMay 30, 2023…Published: May 30, 2023

Self Check Trap illustration 1

How the Fake Authorities Were Challenged

The problem emerged when opposing counsel and the court attempted to locate several authorities cited in the plaintiff’s filing. The cited decisions could not be found in recognised legal research systems, prompting questions about whether the cases existed at all. Judge P. Kevin Castel eventually ordered the lawyers to provide copies of the authorities they had relied upon. [SDNY Blog]sdnyblog.comSDNY BlogJudge Castel Sanctions Lawyers Who Submitted Fake Cases Generated By ChatGPT | SDNY BlogJune 28, 2023…Published: June 28, 2023

At that stage, the issue was no longer simply a research mistake. The disputed cases had already been flagged as potentially non-existent. This was the moment when an independent verification process was needed. Instead, Schwartz relied on ChatGPT again.

Court filings and later reporting showed that Schwartz asked the chatbot whether the cases were real. According to screenshots he submitted, ChatGPT responded that the authorities existed and could be found in legal databases. The system even provided additional details that made the citations appear authentic. Those confirmations were themselves fabricated. [Scottish Legal News]scottishlegal.comScottish Legal NewsLawyer led astray by ChatGPT apologises to court | Scottish Legal NewsMay 30, 2023…Published: May 30, 2023

The significance of this step is easy to overlook. The original hallucination might have been detected if it had been checked against a trusted source. Instead, the challenge itself was routed back through the same mechanism that had created the error.

The ChatGPT Fake Cases Sanctions Decision (Mata v. Avianca)

Channel: Lawful Masses with Leonard French

Open on YouTube

Why the Same Tool Repeated the Error

Large language models are designed to generate plausible text, not to serve as authoritative databases. When ChatGPT produced the fictional cases, it was not retrieving verified court opinions. It was generating language that resembled legal research.

When Schwartz later asked whether the cases were genuine, the model faced a similar task. Rather than consulting an external legal database and proving the authorities existed, it generated another plausible-sounding response. Because the earlier fabricated cases already existed within the conversation context, the model effectively reinforced its own invention. [arXiv]arxiv.orgAssessing the Reasoning Abilities of ChatGPT in the Context of Claim VerificationFebruary 16, 2024…Published: February 16, 2024

This illustrates a common misunderstanding about AI systems. Many users assume that asking a model to “double-check” its work creates an independent review. In reality, a self-check performed by the same model is often not independent at all. The model may reproduce the same underlying mistake while expressing equal or greater confidence.

Several factors made the error especially convincing:

  • Consistency masqueraded as accuracy. The second answer matched the first answer, creating the impression of confirmation.
  • Detailed explanations increased credibility. ChatGPT supplied procedural descriptions and database references that sounded authoritative.
  • Natural-language fluency concealed uncertainty. The responses were written in the same confident style used for genuine information.
  • No external validation occurred. The model’s claims were not tested against a source capable of proving whether the cases existed. [Scottish Legal News+2SDNY Blog]scottishlegal.comScottish Legal NewsLawyer led astray by ChatGPT apologises to court | Scottish Legal NewsMay 30, 2023…Published: May 30, 2023

The result was a feedback loop in which an invented authority became more believable because the system that invented it later endorsed it.

Self Check Trap illustration 2

Why Confidence Is Not Verification

The Schwartz episode highlights a broader lesson about artificial intelligence: confidence and correctness are separate issues.

A legal database verifies a citation by locating an actual judicial opinion. A court clerk verifies a filing by checking records. A lawyer verifies authority by consulting primary sources. These processes depend on evidence external to the original claim.

ChatGPT’s responses, by contrast, were internally coherent but not externally validated. The model could explain why a case supposedly existed without possessing reliable evidence that it did. The convincing explanation therefore became part of the problem. Judge Castel later described the submitted opinions as containing obvious defects, including nonsensical legal analysis and procedural inconsistencies that should have raised concerns. [SDNY Blog]sdnyblog.comSDNY BlogJudge Castel Sanctions Lawyers Who Submitted Fake Cases Generated By ChatGPT | SDNY BlogJune 28, 2023…Published: June 28, 2023

The incident demonstrated that an AI system can generate not only a false claim but also a persuasive justification for that false claim.

Lawyer files ChatGPT DISASTER in COURT (Mata v. Avianca, Inc.)

Channel: Lawful Masses with Leonard French

Open on YouTube

The lesson from Schwartz’s mistake is not that AI tools are unusable. It is that verification must come from a source independent of the system that generated the information.

For legal citations, safer practice includes:

  1. Check every cited authority in recognised legal research databases.
  2. Read the original opinion rather than relying on an AI summary.
  3. Treat AI-generated citations as unverified leads until independently confirmed.
  4. Use multiple sources when a citation is challenged or difficult to locate.
  5. Assume that a model’s confidence does not constitute evidence.

These principles are now frequently cited in legal guidance surrounding generative AI because the Mata case exposed a failure mode that extends beyond law. Any field that relies on sources, references or factual verification can encounter the same problem if an AI system is asked to validate its own output. [Steptoe+2acc.com]steptoe.comJudge Castel Sanctions Lawyers Who Submitted Fake Cases Generated by Chat GPT | SteptoeSteptoeJudge Castel Sanctions Lawyers Who Submitted Fake Cases Generated by ChatGPT | Steptoe

Self Check Trap illustration 3

The Lasting Lesson of the Self-Check Trap

The enduring importance of Steven Schwartz’s verification mistake lies in its simplicity. The critical failure was not merely accepting an AI-generated answer. It was mistaking repeated agreement for independent confirmation.

When ChatGPT produced fake cases and then later vouched for those same cases, it created the appearance of verification without performing verification. The incident remains one of the clearest real-world demonstrations that AI systems can amplify their own errors when users rely on them as both the source of information and the judge of its accuracy. [Scottish Legal News+2Steptoe]scottishlegal.comScottish Legal NewsLawyer led astray by ChatGPT apologises to court | Scottish Legal NewsMay 30, 2023…Published: May 30, 2023

Federal Ruling against Steven Schwartz in Chatbot case. June 27, 2023 story by Mark Miller

Channel: It's 5:05

Open on YouTube

Amazon book picks

Further Reading

Books and field guides related to Why AI Cannot Verify Its Own Fake Cases. Use these as the next step if you want deeper reading beyond the article.

Browse more on Amazon: The Alignment Problem Human Compatible Artificial Intelligence

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Using USA
Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

  1. Source: steptoe.com
    Link: https://www.steptoe.com/en/news-publications/sdny-blog/judge-castel-[sanctions
    Source snippet

    Judge Castel Sanctions Lawyers Who Submitted Fake Cases Generated by ChatGPT | Steptoe...

  2. Source: arxiv.org
    Link: https://arxiv.org/abs/2402.10735
    Source snippet

    Assessing the Reasoning Abilities of ChatGPT in the Context of Claim VerificationFebruary 16, 2024...

    Published: February 16, 2024

  3. Source: acc.com
    Link: https://www.acc.com/resource-library/practical-lessons-attorney-ai-missteps-mata-v-avianca

  4. Source: scottishlegal.com
    Link: https://www.scottishlegal.com/articles/lawyer-led-astray-by-chatgpt-apologises-to-court
    Source snippet

    Scottish Legal NewsLawyer led astray by ChatGPT apologises to court | Scottish Legal NewsMay 30, 2023...

    Published: May 30, 2023

  5. Source: sdnyblog.com
    Link: https://www.sdnyblog.com/judge-castel-sanctions-lawyers-who-submitted-fake-cases-generated-by-chatgpt/
    Source snippet

    SDNY BlogJudge Castel Sanctions Lawyers Who Submitted Fake Cases Generated By ChatGPT | SDNY BlogJune 28, 2023...

    Published: June 28, 2023

Additional References

  1. Source: youtube.com
    Title: Federal Ruling against Steven Schwartz in Chatbot case
    Link: http://www.youtube.com/watch?v=Q0X8VKmkQcc
    Source snippet

    Steven Schwartz ChatGPT lawyer fake cases verification mistake lawyer tries using chatgpt and looses case #chatgpt Ai Telly Tube...

    Published: June 27, 2023

  2. Source: legalcheek.com
    Title: www.legalcheek.com U S lawyer apologises after using fake cases made by Chat GPT
    Link: https://www.legalcheek.com/2023/05/us-lawyer-apologises-after-using-fake-cases-made-by-chatgpt/
    Source snippet

    lawyer apologises after using fake cases made by ChatGPT - Legal CheekMay 31, 2023...

    Published: May 31, 2023

  3. Source: youtube.com
    Title: www.youtube.com The Chat GPT Fake Cases Sanctions Decision (Mata v. Avianca)
    Link: https://www.youtube.com/watch?v=jPtOcsx7ATw
    Source snippet

    ChatGPT Fake Cases Sanctions Decision (Mata v. Avianca) - YouTubeJune 27, 2023...

    Published: June 27, 2023

  4. Source: youtube.com
    Title: Chat GPT Lawyers Who Cited Fake Cases Sanctioned by Court
    Link: http://www.youtube.com/watch?v=sEOapG7-kro
    Source snippet

    Federal Ruling against Steven Schwartz in Chatbot case. June 27, 2023 story by Mark Miller...

    Published: June 27, 2023

  5. Source: youtube.com
    Title: Lawyer files Chat GPT DISASTER in COURT (Mata v. Avianca, Inc.)
    Link: http://www.youtube.com/watch?v=GExSDY6Wz9Y
    Source snippet

    The ChatGPT Fake Cases Sanctions Decision (Mata v. Avianca)...

  6. Source: youtube.com
    Title: How to Use Chat GPT to Ruin Your Legal Career
    Link: http://www.youtube.com/watch?v=oqSYljRYDEM
    Source snippet

    Lawyer files ChatGPT DISASTER in COURT (Mata v. Avianca, Inc.)...

Topic Tree

Follow this branch

Parent topic

Avianca case The court case that exposed fake AI law

Related pages 2