Why AI Can Be Confidently Wrong

Introduction

AI systems can be useful without being reliable in the way people often assume. A large language model can produce fluent, well-structured answers that are partly wrong, unsupported, internally inconsistent or entirely invented. This problem is commonly called “hallucination”, though “confabulation” is often a better term because the system is not seeing things; it is generating plausible text from patterns. NIST defines the risk as generative AI producing confidently stated but false or erroneous content that may mislead users.[NIST Publications]nvlpubs.nist.govNIST PublicationsArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile…

The core danger is not merely that AI makes mistakes. People already expect software, search engines and humans to make mistakes. The distinctive risk is that modern AI can make errors with the surface features of expertise: citations, reasoning steps, legal language, medical tone, statistical detail or institutional confidence. When statistical fluency is mistaken for verified knowledge, the user may treat a guess as a fact.

How plausible details get invented

A language model generates text by predicting likely continuations, not by consulting a built-in ledger of verified truths for every statement. That design can produce accurate summaries and explanations, but it can also produce names, dates, quotations, citations and causal explanations that merely fit the pattern of the answer. NIST notes that generative systems approximate the statistical distribution of their training data, which helps explain why outputs can be coherent while still being false or inconsistent.[NIST Publications]nvlpubs.nist.govNIST PublicationsArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile…

A simple example is a model asked for a source. It may know the pattern of an academic citation: author names, paper title, journal, year, page range and digital object identifier. If it lacks reliable access to the actual source, it may still assemble a citation-shaped answer. The result can look more trustworthy than a vague answer because it contains the very details readers associate with verification.

This is why hallucinations often cluster around:

OpenAI researchers have argued that hallucinations persist partly because many training and evaluation systems reward guessing more than admitting uncertainty. If a test gives points for a correct answer but no credit for saying “I do not know”, models learn behaviour that resembles a student guessing under exam pressure.[arXiv]arxiv.orgarXiv Why Language Models HallucinateWhy Language Models HallucinateSeptember 4, 2025…Published: September 4, 2025

Why confidence is not the same as knowledge

Human readers often use fluency as a shortcut for competence. A confident paragraph with clean structure, careful caveats and technical vocabulary feels more credible than a hesitant one. Large language models exploit that expectation unintentionally: they are optimised to produce helpful-looking language, and helpful-looking language often sounds certain.

That is why an AI answer can be unreliable even when it contains sensible reasoning steps. NIST warns that generative systems may produce confabulated logic or citations that appear to justify an answer, further encouraging users to trust it.[NIST Publications]nvlpubs.nist.govNIST PublicationsArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile… A chain of reasoning is not proof that the model checked the world; it may be another generated artefact.

The problem is especially awkward because the same behaviour that makes AI useful can make it risky. Users want direct answers, not endless hedging. Developers therefore tune systems to be responsive, polite and decisive. But the more an answer resembles a finished expert response, the easier it is to miss the places where the system has silently substituted likelihood for evidence.

The legal cases made the failure visible

Court filings have become one of the clearest public examples because legal citations are highly checkable. In Mata v. Avianca, lawyers submitted material that included non-existent judicial opinions with fake quotes and fake citations generated by an AI tool, leading to sanctions.[UC Berkeley Law]law.berkeley.eduUC Berkeley Law Mata vAvianca, Inc., 678 F.Supp.3d 443 (2023)December 1, 2025 — After air carrier moved to dismiss, passengers' attorneys filed affirmation in…Published: December 1, 2025 The case became a warning because the hallucinations were not vague mistakes: they had the form of real legal authority.

The pattern did not stop there. Stanford researchers studying AI legal research tools reported that general-purpose chatbots hallucinated between 58% and 82% of the time on legal queries in an earlier study, while specialised legal tools still produced hallucinations in benchmarking despite being marketed as more grounded.[Stanford HAI]hai.stanford.eduai trial legal models hallucinate 1 out 6 or more benchmarking queriesStanford HAIAI on Trial: Legal Models Hallucinate in 1 out of 6 (or More…23 May 2024 — Our previous study of general-purpose chatbots…Published: May 2024

The UK has seen similar concerns. In 2025, the High Court warned legal professionals after fake case-law citations appeared in proceedings, including one case in which 18 of 45 cited cases were reportedly fictitious.[The Guardian]theguardian.comIn response, Dame Victoria Sharp, president of the King's Bench Division, emphasized the threat to the justice system and stated that leg… These incidents show why “the user should check” is not a complete safeguard. Busy professionals can be fooled by outputs that look like the kind of material they already handle every day.

Why evaluation can miss failure modes

AI systems are often tested on benchmarks: sets of questions with expected answers. Benchmarks are useful, but they can understate real-world unreliability. A model may do well on short, closed questions yet fail when asked to summarise a messy document, interpret conflicting sources, cite authorities or answer a question whose correct response is “there is not enough evidence”.

Google DeepMind’s FACTS Grounding benchmark was designed to test whether models can produce long-form responses grounded only in a provided document, using 1,719 examples.[Google DeepMind]deepmind.googleOpen source on deepmind.google. That kind of evaluation matters because many real-world tasks are not trivia questions. They require the model to stay within evidence, resist adding outside assumptions and avoid overstating what a document says.

However, even better benchmarks face hard limits. A test set cannot cover every possible prompt, domain, language, adversarial phrasing or edge case. It may also reward an answer that matches a known label while missing whether the model expressed the right level of uncertainty. This is why factuality cannot be reduced to a single leaderboard score.

A further problem is that AI failures are often context-dependent. A model might answer correctly when asked directly, then fail when the same issue appears inside a longer task. It might retrieve the right document but misread it. It might cite a real source but attribute the wrong claim to it. These are not always captured by simple accuracy scores.

Retrieval helps, but it does not make answers automatically safe

One common response to hallucination is retrieval-augmented generation, often shortened to RAG. In a RAG system, the model retrieves documents from a search index, database or knowledge base before answering. This can reduce reliance on the model’s internal memory and can make answers easier to audit.

Retrieval is a real improvement, but it is not a magic fix. Anthropic has described “Contextual Retrieval” methods that reduced failed retrievals by 49%, and by 67% when combined with reranking, showing that much of the reliability problem depends on whether the right evidence is found in the first place.[Anthropic]anthropic.comcontextual retrievalcontextual retrieval

Even when the right documents are retrieved, errors can remain. A 2025 review of hallucination mitigation in retrieval-augmented systems identified risks from low-quality, obsolete or narrow retrieval sources.[MDPI]mdpi.comOpen source on mdpi.com. Other research argues that retrieval-augmented medical systems can still mislead users by decontextualising accurate facts, omitting important sources or reinforcing a user’s misconception.[arXiv]arxiv.orgarXiv Retrieval-augmented systems can be dangerous medical communicatorsarXiv Retrieval-augmented systems can be dangerous medical communicators

The practical lesson is that grounding changes the problem rather than removing it. Instead of asking only “Did the model invent this?”, users must also ask “Did it retrieve the right material?”, “Did it interpret the material correctly?” and “Did it leave out something essential?”

The most common unreliable-answer patterns

Hallucinations are not all the same. Some are obvious fabrications, but many are subtler forms of unreliability. A useful way to read AI output is to watch for distinct failure patterns.

Fabricated evidence is the most visible form. The model invents a source, case, quote, statistic, person, organisation or event. Fake legal citations and invented academic references fall into this category.

Misattributed evidence is harder to detect. The source may be real, but the model claims it says something it does not say. This can be more dangerous than a fake citation because a quick search confirms that the source exists.

Over-compression happens when the model summarises a complicated issue so neatly that important exceptions disappear. This is common in legal, medical, policy and scientific topics where the “short answer” depends on jurisdiction, patient context, methodology or date.

Stale knowledge occurs when a model answers from older patterns. This matters for prices, laws, product specifications, software behaviour, office-holders, scientific consensus and safety guidance.

False certainty appears when the answer gives a confident conclusion despite weak evidence. This can happen even when every individual sentence sounds reasonable.

Reasoning drift happens when a model starts with the right facts but moves toward a conclusion not supported by them. NIST specifically notes that confabulations can include outputs that contradict previous statements in the same context.[NIST Publications]nvlpubs.nist.govNIST PublicationsArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile…

Why humans still over-trust flawed answers

AI errors are partly technical, but the risk is also social. People tend to defer to systems that are fast, polished and consistent, especially when they are under time pressure. NIST identifies human-AI configuration risks such as automation bias and over-reliance, where users may unjustifiably treat AI content as higher quality than other sources.[NIST Publications]nvlpubs.nist.govNIST PublicationsArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile…

This makes hallucination a workflow problem, not just a model problem. A lawyer pasting AI-generated citations into a brief, a student relying on an invented source, or a manager using an AI summary of a policy document may all believe they are saving time. The hidden cost appears later, when verification becomes harder than doing the original research properly.

The risk increases when organisations reward speed without building verification into the process. If AI output moves directly from prompt to publication, filing, email, diagnosis support or business decision, the system’s fluency becomes a substitute for review.

Practical checks before trusting outputs

AI answers should be treated differently depending on what is at stake. For brainstorming, drafting, translation, formatting or generating alternative phrasings, some imperfection may be acceptable. For factual claims, advice, compliance, legal reasoning, medical interpretation or public-facing material, the burden of verification is much higher.

A practical checking routine should include:

1. Separate style from substance. A well-written answer is not necessarily a true answer. Check the claims, not the tone.
2. Verify citations directly. Open the source and confirm that it exists, that it says what the AI claims, and that the cited passage supports the point.
3. Look for missing uncertainty. Reliable answers often say what is unknown, contested, jurisdiction-dependent or date-sensitive.
4. Check recency-sensitive claims. Laws, prices, product features, standards, software behaviour and public roles can change quickly.
5. Ask for evidence boundaries. A grounded answer should distinguish between what appears in the provided material and what is inferred.
6. Use independent sources for high-stakes decisions. Do not rely on a single model output when the consequence of being wrong is serious.
7. Prefer systems with traceable retrieval for factual work. Even then, inspect the retrieved material rather than trusting the generated summary.

The most important habit is to ask: “Could I defend this claim without mentioning the AI?” If the answer is no, the claim is not ready to trust.

What reliability requires

The strongest AI systems will not be the ones that never make mistakes; they will be the ones designed so mistakes are easier to detect, contain and correct. That means better model training, better evaluation, better retrieval, clearer uncertainty, careful user interfaces and human review where consequences are meaningful.

For users, the key mental shift is simple: AI is not a truth machine. It is a powerful language and pattern tool that can assist reasoning, research and drafting, but it does not remove the need for evidence. The more authoritative an answer sounds, the more important it is to ask what supports it.

Amazon book picks

Further Reading

Books and field guides related to Why AI Can Be Confidently Wrong. Use these as the next step if you want deeper reading beyond the article.

Book

Artificial Intelligence

By Melanie Mitchell

Explains why fluent AI systems can still fail at reasoning and reliability.

See on Amazon

Book

Co-Intelligence

By Ethan Mollick

Covers strengths, weaknesses, and verification practices for generative AI.

See on Amazon

Book

The Alignment Problem

By Brian Christian

Explains why AI systems can produce convincing but unreliable outputs and how alignment challenges arise.

See on Amazon

Book

The Coming Wave

By Mustafa Suleyman

Discusses systemic risks created by increasingly capable AI systems.

See on Amazon

Browse more on Amazon: Artificial Intelligence Co Intelligence The Alignment Problem

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

A.I. - Artificial Intelligence Movie/Film Poster Art PICTURE / PRINT 9" x 8"

SearcheBay.co.uk: artificial intelligence poster

Browse similar oneBay.co.uk

Example eBay listing

A.i. ARTIFICIAL INTELLIGENCE (2001) UK cinema poster Spielberg Kubrick Jude Law

SearcheBay.co.uk: artificial intelligence poster

Browse similar oneBay.co.uk

Example eBay listing

A.I. Artificial Intelligence Movie Film Poster Art Print

SearcheBay.co.uk: artificial intelligence poster

Browse similar oneBay.co.uk

Example eBay listing

Artificial intelligence is no a mat Framed Wall Art Poster Canvas Print Picture

SearcheBay.co.uk: artificial intelligence poster

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: nvlpubs.nist.gov
Link:https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

2. Source: arxiv.org
Title: arXiv Why Language Models Hallucinate
Link:https://arxiv.org/abs/2509.04664

Published: September 4, 2025

3. Source: law.berkeley.edu
Title: UC Berkeley Law Mata v
Link:https://www.law.berkeley.edu/wp-content/uploads/archive/2025/12/Mata-v-Avianca-Inc.pdf

Published: December 1, 2025

4. Source: hai.stanford.edu
Title: ai trial legal models hallucinate 1 out 6 or more benchmarking queries
Link:https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries

Published: May 2024

5. Source: arxiv.org
Link:https://arxiv.org/abs/2405.20362

6. Source: deepmind.google
Link:https://deepmind.google/blog/facts-grounding-a-new-benchmark-for-evaluating-the-factuality-of-large-language-models/

7. Source: anthropic.com
Title: contextual retrieval
Link:https://www.anthropic.com/engineering/contextual-retrieval

8. Source: mdpi.com
Link:https://www.mdpi.com/2227-7390/13/5/856

9. Source: arxiv.org
Title: arXiv Retrieval-augmented systems can be dangerous medical communicators
Link:https://arxiv.org/abs/2502.14898

10. Source: nist.gov
Title: hallucination detection large language models using diversion [decoding]({{ ‘decoding/’ | relative_url }})
Link:https://www.nist.gov/publications/hallucination-detection-large-language-models-using-diversion-decoding

11. Source: arxiv.org
Link:https://arxiv.org/html/2602.06176v1

12. Source: arxiv.org
Link:https://arxiv.org/html/2508.01781v1

13. Source: arxiv.org
Link:https://arxiv.org/html/2508.03860

14. Source: arxiv.org
Link:https://arxiv.org/html/2509.04664v1

15. Source: arxiv.org
Link:https://arxiv.org/html/2604.03173v1

16. Source: OpenAI
Title: why language models hallucinate
Link:https://openai.com/index/why-language-models-hallucinate/

17. Source: cdn.openai.com
Title: why language models hallucinate
Link:https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4aaa5/why-language-models-hallucinate.pdf

18. Source: community.openai.com
Title: hallucination in retrieval augmented chatbot rag
Link:https://community.openai.com/t/hallucination-in-retrieval-augmented-chatbot-rag/408275

19. Source: deepmind.google
Link:https://deepmind.google/blog/facts-benchmark-suite-systematically-evaluating-the-factuality-of-large-language-models/

20. Source: cloud.google.com
Title: what are ai hallucinations
Link:https://cloud.google.com/discover/what-are-ai-hallucinations

21. Source: theguardian.com
Link:https://www.theguardian.com/technology/2025/jun/06/high-court-tells-uk-lawyers-to-urgently-stop-misuse-of-ai-in-legal-work

22. Source: theguardian.com
Title: utah lawyer chatgpt ai court brief
Link:https://www.theguardian.com/us-news/2025/may/31/utah-lawyer-chatgpt-ai-court-brief

23. Source: fedcourt.gov.au
Title: needham j 20250627
Link:https://www.fedcourt.gov.au/digital-law-library/judges-speeches/justice-needham/needham-j-20250627

24. Source: layerxsecurity.com
Link:https://layerxsecurity.com/generative-ai/hallucinations/

Additional References

25. Source: youtube.com
Title: NIST AI 600-1 Explained | Generative AI Risk Management (AI RMF Gen AI Profile)
Link:https://www.youtube.com/watch?v=NSlXQ2bwFF0

26. Source: youtube.com
Title: Why LLMs Hallucinate: A Theory of Errors
Link:https://www.youtube.com/watch?v=XticsxlSlTM

27. Source: youtube.com
Title: LLM Hallucinations: 146,000 Fake Citations Found
Link:https://www.youtube.com/watch?v=0E3mHRT-g8I

28. Source: naturalandartificiallaw.com
Link:https://naturalandartificiallaw.com/false-citations-ai-hallucinations-uk/

29. Source: researchgate.net
Link:https://www.researchgate.net/publication/391876200AI_Hallucinations_and_Misinformation_Navigating[Synthetic

30. Source: researchgate.net
Link:https://www.researchgate.net/publication/399075968_Mitigating_Hallucinations_in_Large_Language_Models_via_Retrieval-Augmented_Generation_RAG

31. Source: researchgate.net
Link:https://www.researchgate.net/publication/404018526_Assessing_Llm_Hallucinations_And_The_Reliability_Of_Using_LLms_For_Automated_Hallucination_Detection

32. Source: medium.com
Link:https://medium.com/%40adnanmasood/a-field-guide-to-llm-failure-modes-5ffaeeb08e80

33. Source: damiencharlotin.com
Link:https://www.damiencharlotin.com/hallucinations/

34. Source: cntxt.tech
Link:https://www.cntxt.tech/insights/ai-hallucination-causes-examples-and-mitigation-strategies