Branchoria AI Sense AI Sense

Clear, practical explanations of what AI does, where it helps, and where it fails.

Within Predictions

What happens when an AI prediction is wrong?

False positives and false negatives matter because model mistakes can turn into delays, denials, missed care or extra scrutiny.

On this page

  • False positives and false negatives in everyday systems
  • How errors propagate through organisational processes
  • Why accuracy alone does not measure real harm
Preview for What happens when an AI prediction is wrong?

Introduction

AI predictions are never perfect. A model that estimates fraud risk, disease risk or the likelihood of loan default will inevitably make mistakes. The crucial question is not whether errors occur, but what happens when organisations act on those errors. A false alarm can delay a payment, trigger an investigation or deny access to a service. A missed warning can allow fraud to proceed, leave a medical condition undetected or expose people to preventable risks. The practical harm comes from the decisions and processes attached to a prediction, not from the prediction alone. Research and governance frameworks such as the NIST AI Risk Management Framework emphasise that AI risks arise from the interaction between technical performance, human judgement and organisational systems rather than from model accuracy in isolation. [NIST Publications+2NIST]nvlpubs.nist.govai.100 1Comments on the AI RMF Playbook may be sent via email to AIframework@nist.gov at any time…

Error harms illustration 1 Understanding why prediction errors become real-world harms is essential for understanding artificial intelligence. The same statistical error can be relatively harmless in one setting and highly consequential in another depending on how institutions respond to it.

False positives and false negatives in everyday systems

The two most common prediction errors are false positives and false negatives.

A false positive occurs when a system predicts a problem that is not actually present. A false negative occurs when a system fails to detect a problem that does exist. Every predictive system faces a trade-off between these error types. Reducing one often increases the other. NIST guidance therefore encourages organisations to measure and document both kinds of error rather than relying on a single accuracy score. [EPIC]epic.orgFraming the Risk Management Framework: Actionable…Apr 13, 2023 — Measure and document performance criteria such as accuracy (false…

In practice, the harms are often very different:

  • A fraud detection system may incorrectly flag a legitimate transaction, delaying access to money or creating inconvenience for customers.
  • A lending model may wrongly classify a reliable applicant as high risk, leading to rejection or additional scrutiny.
  • A healthcare model may incorrectly identify a patient as high risk, resulting in unnecessary tests and anxiety.
  • A medical prediction system may miss signs of disease, delaying treatment that could have improved outcomes.
  • A fraud system may fail to detect genuine fraud, allowing financial losses to continue. [LSEG+2The Payments Association]lseg.comUnderstanding False Positives in ScreeningUnderstanding False Positives in Screening - GlossaryIn financial risk management, a false positive refers to an erroneous system ale…

The significance of an error depends on its consequences. In some applications, a false positive is mainly an inconvenience. In others, it can affect employment, healthcare, financial access or public services. Likewise, some false negatives merely reduce efficiency, while others can create serious safety risks. Studies of fraud detection and cybersecurity repeatedly show that organisations often assign different costs to different error types because the consequences are not symmetrical. [MDPI]mdpi.comReducing False Negatives in Ransomware Detectionby R Bold · 2022 · Cited by 52 — The risk of a false negative in this type of system…

Smarter AI Fails in Worse Ways (New Research)

Channel: Claudius Papirus · Views: 5.1K · Uploaded: March 2026 · Length: 14 minutes

Open on YouTube

How errors spread through organisational processes

Prediction errors rarely remain isolated events. Once a prediction enters an organisational workflow, it can influence multiple decisions.

Consider a fraud score that incorrectly identifies a customer as suspicious. The score may trigger an automatic transaction freeze. The freeze may then require manual review. During review, staff may begin from the assumption that the system identified a genuine problem. Additional documentation may be requested, causing delays and frustration. Even if the final decision is eventually corrected, the person affected has already experienced consequences.

This chain illustrates an important point: harms often arise from the process surrounding an error rather than from the error itself.

A similar pattern appears in healthcare. A risk model may trigger extra testing, referrals or monitoring. If the prediction is wrong, the patient may face unnecessary procedures, additional costs or emotional stress. Conversely, a missed risk prediction can prevent a patient from receiving attention when it is most needed.

Researchers increasingly describe these outcomes as sociotechnical harms because they emerge from interactions between technology, people and institutions rather than from software alone. AI governance literature emphasises that assessing impacts requires examining the full decision environment, including workflows, incentives and oversight mechanisms. [Trilateral Research+2arXiv]trilateralresearch.comTrilateral Research A survey of artificial intelligence risk assessmentTrilateral ResearchA survey of artificial intelligence risk assessment…December 16, 2021 — While risk assessment is about identificati…Published: December 16, 2021

Error harms illustration 2

When people trust the score too much

Prediction errors can become more damaging when people place excessive confidence in AI outputs.

Studies of automation bias show that decision-makers may treat algorithmic recommendations as more reliable than they really are. When a risk score appears objective or mathematically precise, staff may be less likely to challenge it. As a result, incorrect predictions can gain influence beyond their actual reliability. [EIMT]eimt.edu.euwhat is automation bias in ai securityWhat is Automation Bias in AI Security9 Feb 2026 — Automation bias is exactly that risk. It happens when people stop questioning AI d…

This does not mean that humans simply follow AI blindly. Rather, predictions can shape attention and expectations. A reviewer presented with a “high-risk” label may examine information differently from a reviewer who sees no warning. The error therefore influences subsequent human judgement even when a person retains formal authority over the final decision.

Why accuracy alone does not measure real harm

A common misunderstanding is that a highly accurate model automatically produces acceptable outcomes.

Accuracy measures how often a system is correct overall. Harm depends on who experiences mistakes, how serious those mistakes are and what actions follow them. Two systems can have identical accuracy while producing very different social consequences.

Imagine two models that are each 95% accurate. One is used to recommend films. The other is used to prioritise medical reviews. The same error rate carries very different implications because the consequences of mistakes differ dramatically.

Even within the same application, overall accuracy can hide important details. A model may perform well for most people but make substantially more mistakes for particular groups because of data limitations, representation problems or historical biases. NIST and OECD guidance therefore stress evaluating fairness, validity, reliability and impact alongside raw predictive performance. [NIST+3Scrut+3OECD]scrut.ioFairness and bias (NIST AI RMF)Learn how the NIST AI RMF addresses unfair or discriminatory outcomes through data quality, testing…

For this reason, responsible AI evaluation often asks questions such as:

  • Who bears the cost of errors?
  • Which error type is considered most harmful?
  • Are some groups affected more often than others?
  • Can decisions be reviewed or appealed?
  • Are there safeguards when predictions are uncertain?

These questions focus on consequences rather than statistics alone.

Why Most Predictive Models Fail in the Real World (And How to Fix Them)

Channel: Shorful

Open on YouTube

Why institutions determine the severity of harm

Prediction models create estimates, but organisations determine what those estimates mean in practice.

A system that merely flags cases for additional review creates a different level of risk from a system that automatically denies access to services. Likewise, providing an appeal process can reduce harm even when prediction errors still occur. Human oversight, transparency and opportunities for correction can all influence whether mistakes become lasting consequences. NIST’s risk-management approach treats these governance decisions as central elements of AI safety and trustworthiness. [NIST+2NIST Publications]nist.govAI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society a…

This means that harmful outcomes are not solely technical failures. They are often the result of institutional choices about thresholds, automation, review procedures and accountability. Predictive models will always make some mistakes. The real challenge is deciding how much power those mistakes are allowed to have over people’s lives.

YouTubeResponsible AI Risk Management | NIST AI Risk Framework Explained

Responsible AI Risk Management | NIST AI Risk Framework Explained

Open on YouTube

The key takeaway

AI prediction errors become practical harms when organisations attach actions to predictions. False positives can lead to delays, scrutiny and denial of opportunities. False negatives can allow risks, fraud or medical problems to go unnoticed. The severity of these harms depends not only on model performance but also on organisational design, human oversight and the consequences attached to each prediction. Understanding AI therefore requires looking beyond whether a prediction is correct and examining what happens when it is wrong.

Error harms illustration 3

Amazon book picks

Further Reading

Books and field guides related to What happens when an AI prediction is wrong?. Use these as the next step if you want deeper reading beyond the article.

Browse more on Amazon: The Alignment Problem Prediction Machines The Signal and the Noise

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Using USA
Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

  1. Source: nvlpubs.nist.gov
    Title: ai.100 1
    Link: https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
    Source snippet

    Comments on the AI RMF Playbook may be sent via email to AIframework@nist.gov at any time...

  2. Source: nist.gov
    Link: https://www.nist.gov/itl/ai-risk-management-framework
    Source snippet

    AI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society a...

  3. Source: arxiv.org
    Link: https://arxiv.org/abs/2401.15229

  4. Source: epic.org
    Link: https://epic.org/framing-the-risk-management-framework-actionable-instructions-by-nist-in-the-measure-section-of-the-ai-rmf/
    Source snippet

    Framing the Risk Management Framework: Actionable...Apr 13, 2023 — Measure and document performance criteria such as accuracy (false...

  5. Source: nvlpubs.nist.gov
    Title: AI.600 1
    Link: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
    Source snippet

    NIST PublicationsArtificial Intelligence Risk Management Frameworkby N AI · 2024 · Cited by 118 — Evaluate the rate of false positives an...

  6. Source: lseg.com
    Title: Understanding False Positives in Screening
    Link: https://www.lseg.com/en/risk-intelligence/glossary/risk-management/false-positive
    Source snippet

    Understanding False Positives in Screening - GlossaryIn financial risk management, a false positive refers to an erroneous system ale...

  7. Source: mdpi.com
    Link: https://www.mdpi.com/2076-3417/12/24/12941
    Source snippet

    Reducing False Negatives in Ransomware Detectionby R Bold · 2022 · Cited by 52 — The risk of a false negative in this type of system...

  8. Source: arxiv.org
    Link: https://arxiv.org/pdf/2403.15646
    Source snippet

    Application of the NIST AI Risk Management Framework to...by N Swaminathan · 2024 · Cited by 20 — The framework can be used to iden...

  9. Source: eimt.edu.eu
    Title: what is automation bias in ai security
    Link: https://www.eimt.edu.eu/what-is-automation-bias-in-ai-security
    Source snippet

    What is Automation Bias in AI Security9 Feb 2026 — Automation bias is exactly that risk. It happens when people stop questioning AI d...

  10. Source: scrut.io
    Link: https://www.scrut.io/glossary/fairness-and-bias-nist-ai-rmf
    Source snippet

    Fairness and bias (NIST AI RMF)Learn how the NIST AI RMF addresses unfair or discriminatory outcomes through data quality, testing...

  11. Source: oecd.org
    Link: https://www.oecd.org/content/dam/oecd/en/publications/reports/2022/02/oecd-framework-for-the-classification-of-ai-systems_336a8b57/cb6d9eca-en.pdf
    Source snippet

    2019), the first set of AI standards that governments pledged to incorporate...Read more...

  12. Source: oecd.org
    Title: component 4
    Link: https://www.oecd.org/en/publications/2026/02/oecd-due-diligence-guidance-for-responsible-ai_7831bb49/full-report/component-4.html
    Source snippet

    OECD Due Diligence Guidance for Responsible AI19 Feb 2026 — Identify risks of the AI system facilitating or advocating for outcomes that...

  13. Source: nist.gov
    Link: https://www.nist.gov/cyberframework
    Source snippet

    Cybersecurity Framework | NISTby N Keller · 2013 · Cited by 11 — Cybersecurity Framework helping organizations to better understand and i...

  14. Source: nvlpubs.nist.gov
    Title: NIST.AI.100 2e2025
    Link: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf
    Source snippet

    [Machine Learning]({{ 'machine-learning/' | relative_url }}): A Taxonomy and Terminology...by A Vassilev · 2025 · Cited by 39 — This NIST Trustworthy and Responsible AI report prov...

  15. Source: epic.org
    Title: comments to nist on managing the risks of misuse with ai foundation models
    Link: https://epic.org/documents/epic-comments-to-nist-on-managing-the-risks-of-misuse-with-ai-foundation-models/
    Source snippet

    EPIC Comments to NIST on Managing the Risks of Misuse...Sep 9, 2024 — And model bias can hinder misuse risk mitigation directly as well...

  16. Source: mdpi.com
    Link: https://www.mdpi.com/2227-9091/13/10/198
    Source snippet

    Application of Standard Machine Learning Models for...by D Farahmandazad · 2025 · Cited by 2 — In this paper, we present a comprehensive...

  17. Source: oecd.ai
    Title: risk thresholds consultation
    Link: https://oecd.ai/en/site/ai-futures/discussions/risk-thresholds-consultation
    Source snippet

    Public consultation on AI risk thresholds10 Sept 2024 — However, compute thresholds alone should generally not determine which mitigation...

  18. Source: thepaymentsassociation.org
    Link: https://thepaymentsassociation.org/article/ai-and-fraud-prevention-the-hidden-risks-of-false-positives-and-black-box-models/
    Source snippet

    The Payments AssociationAI and fraud prevention: The hidden risks of false positives...4 Sept 2025 — Poor explainability and excessive f...

  19. Source: trilateralresearch.com
    Title: Trilateral Research A survey of artificial intelligence risk assessment
    Link: https://www.trilateralresearch.com/wp-content/uploads/2022/01/A-survey-of-AI-Risk-Assessment-Methodologies-full-report.pdf
    Source snippet

    Trilateral ResearchA survey of artificial intelligence risk assessment...December 16, 2021 — While risk assessment is about identificati...

    Published: December 16, 2021

Additional References

  1. Source: linkedin.com
    Link: https://www.linkedin.com/posts/joseph-kamara_aigovernance-riskassessment-activity-7440139482458329088-dOzb
    Source snippet

    AI Governance: Separate Risk, Impact, and Algorithmic...Your AI risk assessment and your AI impact assessment are not the same thing. Th...

  2. Source: pwc.co.uk
    Link: https://www.pwc.co.uk/forensic-services/assets/impact-of-ai-on-fraud-and-scams.pdf
    Source snippet

    Impact of Artificial Intelligence on Fraud and ScamsMachine learning techniques are generally already embedded in our fraud detection sys...

  3. Source: hyperproof.io
    Link: https://hyperproof.io/navigating-the-nist-ai-risk-management-framework/
    Source snippet

    Navigating the NIST AI Risk Management FrameworkWhen it comes to reliability, NIST says it is a “goal for overall correctness” when an AI...

  4. Source: linkedin.com
    Link: https://www.linkedin.com/pulse/understanding-nist-ai-rmf-managing-risk-beyond-tom-wechsler-vh2se
    Source snippet

    Understanding the NIST AI RMF: Managing AI Risk Beyond...NIST AI RMF focuses on AI risks, trustworthiness, fairness, transparency, valid...

  5. Source: thoropass.com
    Link: https://www.thoropass.com/blog/nist-ai-rmf
    Source snippet

    Understanding the NIST AI Risk Management FrameworkFor example, these KPIs might include the accuracy of AI models, the frequency of fals...

  6. Source: site.unibo.it
    Link: https://site.unibo.it/hypermodelex/en/publications/2024-04-01-nist-ai-risk-management-genai.pdf/%40%40download/file/2024-04-01-NIST-AI-RISK-MANAGEMENT-GENAI.pdf
    Source snippet

    Intelligence Risk Management FrameworkEvaluate the rate of false positives and false negatives in content [provenance]({{ 'provenance/' | relative_url }}), as well as true pos...

  7. Source: bundeswirtschaftsministerium.de
    Link: https://www.bundeswirtschaftsministerium.de/Redaktion/DE/Downloads/M-O/oecd-due-diligence-guidance-for-responsible-ai.pdf?__blob=publicationFile&v=1
    Source snippet

    AI systems by establishing principles and policy guidelines that foster innovation while addressing risks.Read more...

  8. Source: researchgate.net
    Link: https://www.researchgate.net/publication/401252625_National_Institute_of_Standards_and_Technology_NIST_2023_AI_risk_management_framework_AI_RMF_10_US_Department_of_Commerce
    Source snippet

    U.S. Department of... data privacy and algorithmic bias, stressing the need for transparent AI...Read more...

  9. Source: ijpsjournal.com
    Link: https://www.ijpsjournal.com/article/healthcare-predictive-modeling-for-identifying-fraud-in-medical-insurance-claims
    Source snippet

    d fraud detection towards improved accuracy, explainability, and real-time fraud...Read more...

  10. Source: pmc.ncbi.nlm.nih.gov
    Title: PMCHealthcare insurance fraud detection using data mining
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11046758/
    Source snippet

    Anomaly detection, Clustering, and classification can detect fraudulent insurance claims [21]. After...Read mor...

Topic Tree

Follow this branch

Parent topic

Predictions When an AI prediction changes what happens next

Related pages 2