Branchoria AI Sense AI Sense

Clear, practical explanations of what AI does, where it helps, and where it fails.

Within AI Sense

Who Is Responsible When AI Fails?

Responsible AI requires attention to biased data, human accountability, testing limits, and what happens when systems fail.

On this page

  • Bias in data and system outputs
  • Human responsibility and escalation points
  • Standards, testing, and safeguards
Preview for Who Is Responsible When AI Fails?

Introduction

When an AI system fails, responsibility does not belong to the software. It belongs to the people and organisations that choose the data, define the goal, buy or build the system, deploy it in a real setting, monitor its effects and decide what happens when someone is harmed. That is the core idea behind responsible AI: bias must be anticipated, accountability must be assigned before deployment, and safeguards must be tested under realistic failure conditions rather than assumed from good intentions.

Overview image for Responsible AI This matters because AI systems can appear neutral while reproducing unequal treatment. A model may learn from biased historical data, use a poor proxy for the real target, work well for majority groups but badly for minorities, or encourage humans to over-trust an automated recommendation. Responsible AI is therefore not a slogan about being ethical. It is a governance discipline: map the risks, test the system, document decisions, keep humans meaningfully empowered, and create clear escalation routes when the system is wrong.

Bias starts before the model gives an answer

AI bias is often described as if it appears at the final output, but it usually enters much earlier. The training data may under-represent some groups, encode past discrimination, or contain measurements that look objective but are actually shaped by unequal access to services. A model can also become biased because engineers optimise for the wrong target: what is easy to measure may not be what is fair or useful.

A striking healthcare example came from research published in Science in 2019. The study found racial bias in a widely used health-risk algorithm because the system used healthcare spending as a proxy for medical need. Since Black patients in the dataset had historically received less spending than equally ill White patients, the algorithm underestimated their need for extra care. The researchers estimated that correcting the bias would raise the share of Black patients receiving additional help from 17.7% to 46.5%. The lesson is not simply “bad data causes bad AI”; it is that a seemingly practical measurement choice can quietly turn structural inequality into an automated decision rule. [Federal Trade Commission]ftc.govFederal Trade CommissionDissecting racial bias in an algorithm used to manage the…July 2, 2020 — by Z Obermeyer · Cited by 9554 — Thus…Published: July 2, 2020

Bias can also show up as uneven technical performance. The Gender Shades study by Joy Buolamwini and Timnit Gebru tested three commercial gender classification systems and found that darker-skinned women were the most misclassified group, with error rates of up to 34.7%, while the maximum error rate for lighter-skinned men was 0.8%. The study also noted that common benchmark datasets were heavily composed of lighter-skinned subjects, showing how “average accuracy” can hide serious failures for specific groups. [Proceedings of Machine Learning Research]proceedings.mlr.pressProceedings of Machine Learning ResearchGender Shades: Intersectional Accuracy Disparities in…January 21, 2018 — by J Buolamwini · 201…Published: January 21, 2018

This is why responsible AI testing has to look beyond a single headline accuracy score. A system used in hiring, policing, lending, education, healthcare or welfare can be unacceptable even if it performs well on average. The relevant question is: who gets the errors, how severe are they, and can affected people challenge the result?

Responsible AI illustration 1

Biased outputs become governance failures when nobody owns them

Bias becomes more dangerous when an organisation treats an AI output as if it were an independent authority. In reality, AI systems do not remove human judgement; they relocate it. Someone decides the system’s purpose, approves its data sources, signs the procurement contract, sets the confidence threshold, trains staff, and chooses whether users can appeal.

The Amazon recruiting-tool case is a useful warning. Reuters reported in 2018 that Amazon abandoned an experimental machine-learning recruitment system after it showed bias against women. The tool had reportedly learned from patterns in past applications in a male-dominated technical workforce, penalising signals associated with women’s résumés. The important point is not that all hiring AI will behave the same way, but that historical success data can teach a model to reproduce the inequalities of the past. [Reuters]reuters.comOpen source on reuters.com.

The same accountability problem appears in criminal justice and policing. ProPublica’s 2016 investigation of the COMPAS recidivism risk tool triggered a major debate about fairness metrics in risk prediction, including whether different definitions of fairness can conflict with one another. Later analysis and critique have shown how difficult it is to separate technical accuracy, legal accountability and social consequences when a score influences bail, sentencing or parole decisions. [ProPublica]propublica.orgPro Publica Machine BiasPro Publica Machine Bias

Facial recognition has produced even clearer examples of harm. The Innocence Project reported in 2024 that there were at least seven confirmed cases of misidentification linked to facial recognition technology, six involving Black people wrongfully accused. The ACLU reported in April 2026 that more than a dozen people were then known to have been wrongfully arrested after police reliance on facial recognition. These cases show why a “possible match” must never become a substitute for investigation, disclosure and independent evidence. [Innocence Project]innocenceproject.orgartificial intelligence is putting innocent people at risk of being incarceratedartificial intelligence is putting innocent people at risk of being incarcerated

Wrongfully Arrested Because of Flawed Face Recognition Technology

Channel: ACLU

Open on YouTube

Human oversight only works when humans can actually intervene

“Human in the loop” is one of the most common phrases in responsible AI, but it can be misleading. A human who rubber-stamps a recommendation, lacks time to investigate it, cannot see the evidence behind it, or faces pressure to follow the system is not a meaningful safeguard. They are part of the interface, not a source of accountability.

Research on human-AI decision-making shows why this matters. A 2024 study found that human judgement was affected when people received incorrect algorithmic support, particularly when the algorithmic advice came before their own judgement. In other words, an AI suggestion can anchor the human reviewer, making oversight weaker rather than stronger. [PMC]pmc.ncbi.nlm.nih.govPMCThe impact of AI errors in a human-in-the-loop processPMCThe impact of AI errors in a human-in-the-loop process

Regulators increasingly recognise this problem. The EU AI Act requires human oversight for high-risk AI systems and says that oversight should aim to prevent or minimise risks to health, safety and fundamental rights, including risks that remain even when other safeguards are in place. [Artificial Intelligence Act]artificialintelligenceact.euOpen source on artificialintelligenceact.eu. The European Data Protection Supervisor has also stressed that effective human oversight should have a tangible positive impact on outcomes, such as preventing harm, improving fairness and strengthening accountability. [European Data Protection Supervisor]edps.europa.euEuropean Data Protection Supervisor Tech DispatchEuropean Data Protection Supervisor Tech Dispatch

Useful escalation points are therefore practical, not symbolic. They include:

  • Pre-deployment gates: no high-impact system goes live until bias, safety, privacy and security risks have been reviewed.
  • Confidence thresholds: uncertain or low-quality cases are routed to trained humans rather than forced through automation.
  • Override rights: staff can challenge or reverse AI outputs without being punished for slowing the process down.
  • Appeal routes: affected people can understand, contest and correct decisions that affect them.
  • Incident reporting: harmful errors are logged, investigated and used to change the system or withdraw it.

The goal is not to pretend humans are flawless. It is to design a workflow where human judgement has enough information, authority and time to matter.

Bias, Fairness & Responsible AI Explained | AI Ethics in Data Science

Channel: The Learning Studio

Open on YouTube

Standards turn responsibility into repeatable work

Responsible AI is easier to discuss in principles than to implement in a busy organisation. That is why standards and risk frameworks matter. They translate broad values such as fairness and accountability into recurring tasks: inventory systems, classify risk, test models, document decisions, monitor performance, assign owners and review failures.

The NIST AI Risk Management Framework identifies trustworthy AI characteristics including validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. NIST also emphasises that these qualities must be balanced according to the system’s context of use, because an AI chatbot for drafting marketing copy does not carry the same risk as a model used in healthcare or criminal justice. [NIST AI Resource Center]airc.nist.govAI Resource Center AI Risks and TrustworthinessAI Resource Center AI Risks and Trustworthiness

The OECD AI Principles similarly frame trustworthy AI around human rights, democratic values, transparency, robustness, safety and accountability. Their importance lies partly in setting a common policy language: organisations cannot simply claim innovation as a reason to avoid responsibility for foreseeable harm. [OECD]oecd.orgOpen source on oecd.org.

ISO/IEC 42001 takes a management-system approach. ISO describes it as the first global standard for AI management systems, giving organisations requirements and guidance to establish, maintain and continually improve how they govern AI. In plain terms, it asks organisations to make responsible AI part of management practice rather than leaving it to ad hoc ethics reviews after a system is already built. [ISO]iso.orgiso 42001 explained what it isiso 42001 explained what it is

For generative AI and large language model applications, safeguards also need to cover security failure modes. OWASP’s Top 10 for Large Language Model Applications lists prompt injection as a leading risk, where crafted inputs manipulate a model into unauthorised or unsafe behaviour. It also highlights insecure output handling, where unvalidated model outputs can trigger downstream security problems. These risks show that responsible AI is not only about social bias; it is also about resilience when systems are connected to tools, databases, code or business processes. [OWASP Foundation]owasp.orgOpen source on owasp.org.

Responsible AI illustration 2

Testing must match the real world, not the demo

A responsible AI safeguard is only as strong as the test behind it. Many failures happen because systems are tested in clean conditions, then deployed into messy environments with poor-quality inputs, distribution shifts, rushed staff, adversarial users or people unlike those represented in the training data.

Good testing asks several questions at once. Does the system work for different demographic groups? Does it fail more often when image quality is poor, language is non-standard, forms are incomplete or users have disabilities? Does performance degrade after deployment? Can users manipulate it? Can staff detect when it is wrong? Does the system create a feedback loop, where earlier AI decisions shape the data used to train the next version?

The US Federal Trade Commission’s Rite Aid case shows what happens when safeguards are treated as optional. In 2023, the FTC announced that Rite Aid would be banned from using facial recognition technology for surveillance purposes for five years after allegations that the retailer failed to implement reasonable procedures to prevent consumer harm. The FTC specifically pointed to failures around accuracy testing, image-quality controls and risks of false positives, including risks linked to race and gender. [Federal Trade Commission]ftc.govFederal Trade Commission Rite Aid Banned from Using AI Facial Recognition AfterFederal Trade Commission Rite Aid Banned from Using AI Facial Recognition After

A practical responsible AI programme therefore needs layered controls:

Data checks: examine representativeness, missingness, labelling quality, proxy variables and historical bias.

Model evaluation: test subgroup performance, false positives, false negatives, calibration, robustness and uncertainty.

Human factors testing: check whether reviewers over-trust the system, understand its limits and have time to investigate.

Security testing: probe for prompt injection, data leakage, unsafe tool use, model poisoning and misuse.

Post-deployment monitoring: track drift, complaints, overrides, appeals, incidents and unexpected patterns of harm.

Withdrawal criteria: define in advance when a system must be paused, rolled back or replaced.

These are governance choices as much as technical ones. An organisation that cannot monitor or explain a high-impact AI system may not be ready to deploy it.

The Double-Edged Sword of AI, with Dr. Ziad Obermeyer

Channel: NEJM Group

Open on YouTube

Accountability is clearest when failure paths are designed in advance

The hardest question is not “can AI be biased?” but “what happens when it is?” Responsible AI requires named owners, documented decisions and real consequences. Without those, accountability dissolves across vendors, data providers, internal teams, frontline staff and executives.

A useful responsibility map separates roles. The developer is responsible for design choices, documentation, testing and known limitations. The deploying organisation is responsible for deciding whether the system is appropriate in its setting. Managers are responsible for training, monitoring and escalation. Human reviewers are responsible for decisions only when they have genuine agency. Senior leaders are responsible for risk appetite, procurement standards and whether the organisation continues using a harmful system.

This is why “the vendor said it was accurate” is not enough. Accuracy depends on context: the population, input quality, decision threshold, workflow, user training and consequences of error. A facial recognition model, for instance, may perform differently on a controlled benchmark than on a blurry shop camera image or police still. A healthcare model may seem predictive while optimising for cost rather than care need. A hiring tool may look efficient while filtering out candidates who do not resemble past employees.

Responsible AI safeguards work best when they are boringly explicit: who signs off, who monitors, who can stop the system, who informs affected people, who investigates incidents, and who pays attention after launch. The central rule is simple: AI can support decisions, but it should not become a place where responsibility disappears.

1/2

Responsible AI illustration 3

Amazon book picks

Further Reading

Books and field guides related to Who Is Responsible When AI Fails?. Use these as the next step if you want deeper reading beyond the article.

Browse more on Amazon: The Alignment Problem Atlas of AI Weapons of Math Destruction

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Using USA
Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

  1. Source: ftc.gov
    Link: https://www.ftc.gov/system/files/documents/public_events/1548288/privacycon-2020-ziad_obermeyer.pdf
    Source snippet

    Federal Trade CommissionDissecting racial bias in an algorithm used to manage the...July 2, 2020 — by Z Obermeyer · Cited by 9554 — Thus...

    Published: July 2, 2020

  2. Source: reuters.com
    Link: https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK0AG/

  3. Source: propublica.org
    Title: Pro Publica Machine Bias
    Link: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

  4. Source: aclu.org
    Link: https://www.aclu.org/news/privacy-technology/more-than-a-dozen-wrongful-arrests-due-to-police-reliance-on-facial-recognition-technology

  5. Source: pmc.ncbi.nlm.nih.gov
    Title: PMCThe impact of [AI errors]({{ ‘ai-errors/’ | relative_url }}) in a human-in-the-loop process
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC10772030/

  6. Source: airc.nist.gov
    Title: AI Resource Center AI Risks and Trustworthiness
    Link: https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/

  7. Source: nvlpubs.nist.gov
    Title: ai.100 1
    Link: https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf

  8. Source: oecd.org
    Link: https://www.oecd.org/en/topics/sub-issues/ai-principles.html

  9. Source: legalinstruments.oecd.org
    Link: https://legalinstruments.oecd.org/en/instruments/oecd-legal-0449

  10. Source: iso.org
    Title: iso 42001 explained what it is
    Link: https://www.iso.org/home/insights-news/resources/iso-42001-explained-what-it-is.html

  11. Source: owasp.org
    Link: https://owasp.org/www-project-top-10-for-large-language-model-applications/

  12. Source: genai.owasp.org
    Title: Gen AI Security Project LLMRisks Archive
    Link: https://genai.owasp.org/llm-top-10/

  13. Source: ftc.gov
    Title: Federal Trade Commission Rite Aid Banned from Using AI Facial Recognition After
    Link: https://www.ftc.gov/news-events/news/press-releases/2023/12/rite-aid-banned-using-ai-facial-recognition-after-ftc-says-retailer-deployed-technology-without

  14. Source: nist.gov
    Link: https://www.nist.gov/itl/ai-risk-management-framework

  15. Source: nist.gov
    Link: https://www.nist.gov/

  16. Source: oecd.org
    Link: https://www.oecd.org/en.html

  17. Source: oecd.org
    Title: 2448f04b en
    Link: https://www.oecd.org/content/dam/oecd/en/publications/reports/2023/02/advancing-accountability-in-ai_753bf8c8/2448f04b-en.pdf

  18. Source: oecd.org
    Title: ba602d18 en
    Link: https://www.oecd.org/content/dam/oecd/en/publications/reports/2023/11/common-guideposts-to-promote-interoperability-in-ai-risk-management_9629ed36/ba602d18-en.pdf

  19. Source: genai.owasp.org
    Link: https://genai.owasp.org/

  20. Source: propublica.org
    Title: how we analyzed the compas recidivism algorithm
    Link: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

  21. Source: oecd.ai
    Link: https://oecd.ai/en/ai-principles

  22. Source: oecd.ai
    Link: https://oecd.ai/en/dashboards/ai-principles/P7

  23. Source: oecd.ai
    Link: https://oecd.ai/en/dashboards/ai-principles/P8

  24. Source: aws.amazon.com
    Title: ai lifecycle risk management iso iec 420012023 for ai governance
    Link: https://aws.amazon.com/blogs/security/ai-lifecycle-risk-management-iso-iec-420012023-for-ai-governance/

  25. Source: ftc.gov
    Title: 2023190 rite aid corporation ftc v
    Link: https://www.ftc.gov/legal-library/browse/cases-proceedings/2023190-rite-aid-corporation-ftc-v

  26. Source: proceedings.mlr.press
    Link: https://proceedings.mlr.press/v81/buolamwini18a
    Source snippet

    Proceedings of Machine Learning ResearchGender Shades: Intersectional Accuracy Disparities in...January 21, 2018 — by J Buolamwini · 201...

    Published: January 21, 2018

  27. Source: innocenceproject.org
    Title: artificial intelligence is putting innocent people at risk of being incarcerated
    Link: https://innocenceproject.org/news/artificial-intelligence-is-putting-innocent-people-at-risk-of-being-incarcerated/

  28. Source: artificialintelligenceact.eu
    Link: https://artificialintelligenceact.eu/article/14/

  29. Source: digital-strategy.ec.europa.eu
    Title: Digital Strategy AI Act | Shaping Europe’s digital future
    Link: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai

  30. Source: edps.europa.eu
    Title: European Data Protection Supervisor Tech Dispatch
    Link: https://www.edps.europa.eu/system/files/2025-09/25-09-15_techdispatch-human-oversight_en.pdf

  31. Source: iso-library.com
    Link: https://iso-library.com/standard/23894/

  32. Source: pubmed.ncbi.nlm.nih.gov
    Link: https://pubmed.ncbi.nlm.nih.gov/31649194/

  33. Source: sentinelone.com
    Title: nist ai risk management framework
    Link: https://www.sentinelone.com/cybersecurity-101/cybersecurity/nist-ai-risk-management-framework/

  34. Source: isms.online
    Title: iso 42001
    Link: https://www.isms.online/iso-42001/

  35. Source: edps.europa.eu
    Title: 2025 11 11 ai risks management guidance en
    Link: https://www.edps.europa.eu/system/files/2025-11/2025-11-11_ai_risks_management_guidance_en.pdf

  36. Source: kpmg.com
    Title: iso iec 42001
    Link: https://kpmg.com/ch/en/insights/artificial-intelligence/iso-iec-42001.html

  37. Source: proceedings.mlr.press
    Link: https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf

  38. Source: GOV.UK
    Title: ethics transparency and accountability framework for automated decision making
    Link: https://www.gov.uk/government/publications/ethics-transparency-and-accountability-framework-for-automated-decision-making/ethics-transparency-and-accountability-framework-for-automated-decision-making

  39. Source: academy.evalcommunity.com
    Title: oecd ai principles
    Link: https://academy.evalcommunity.com/oecd-ai-principles/

  40. Source: scrut.io
    Title: nist ai risk management framework
    Link: https://www.scrut.io/post/nist-ai-risk-management-framework

  41. Source: openpracticelibrary.com
    Title: human in the loop
    Link: https://openpracticelibrary.com/practice/human-in-the-loop/

  42. Source: gotrust.nl
    Title: iso 42001
    Link: https://www.gotrust.nl/iso-42001

  43. Source: autoriteitpersoonsgegevens.nl
    Title: eu ai act
    Link: https://www.autoriteitpersoonsgegevens.nl/en/themes/algorithms-ai/eu-ai-act

  44. Source: artificialintelligenceact.eu
    Link: https://artificialintelligenceact.eu/high-level-summary/

  45. Source: artificialintelligenceact.eu
    Link: https://artificialintelligenceact.eu/section/3-2/

  46. Source: aiethicslab.rutgers.edu
    Title: oecd ai principles
    Link: https://aiethicslab.rutgers.edu/glossary/oecd-ai-principles/

  47. Source: diligent.com
    Title: nist ai risk management framework
    Link: https://www.diligent.com/resources/blog/nist-ai-risk-management-framework

  48. Source: witness.ai
    Title: ai risk management
    Link: https://witness.ai/blog/ai-risk-management/

Additional References

  1. Source: chicagobooth.edu
    Link: https://www.chicagobooth.edu/research/tolan/research/2019/dissecting-racial-bias-in-an-algorithm-used-to-manage-the-health-of-populations

  2. Source: youtube.com
    Title: Bias, Fairness & Responsible AI Explained | AI Ethics in Data Science
    Link: https://www.youtube.com/watch?v=C78FeCxVCaY
    Source snippet

    The Double-Edged Sword of AI, with Dr. Ziad Obermeyer...

  3. Source: youtube.com
    Title: The Double-Edged Sword of AI, with Dr. Ziad Obermeyer
    Link: https://www.youtube.com/watch?v=c5KcSNeGXOM
    Source snippet

    AI governance: bias audits for public sector AI...

  4. Source: researchgate.net
    Link: https://www.researchgate.net/publication/373896468_Gender_Bias_in_Hiring_An_Analysis_of_the_Impact_of_Amazon%27s_Recruiting_Algorithm

  5. Source: wilmerhale.com
    Link: https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20240717-what-are-highrisk-ai-systems-within-the-meaning-of-the-eus-ai-act-and-what-requirements-apply-to-them

  6. Source: marcellodibello.com
    Link: https://www.marcellodibello.com/algorithmicfairness/handout/ProPublica-Northpointe.html

  7. Source: allendowney.github.io
    Link: https://allendowney.github.io/RecidivismCaseStudy/02_calibration.html

  8. Source: aiactblog.nl
    Link: https://www.aiactblog.nl/en/ai-act/artikel/14

  9. Source: medium.com
    Link: https://medium.com/%40lamdaa/compas-unfair-algorithm-812702ed6a6a

  10. Source: cycoresecure.com
    Link: https://www.cycoresecure.com/frameworks/eu-ai-act

Topic Tree

Follow this branch

Parent topic

AI Sense

Related pages 11

More on this topic 5