Who Is Responsible When AI Fails?

Introduction

When an AI system fails, responsibility does not belong to the software. It belongs to the people and organisations that choose the data, define the goal, buy or build the system, deploy it in a real setting, monitor its effects and decide what happens when someone is harmed. That is the core idea behind responsible AI: bias must be anticipated, accountability must be assigned before deployment, and safeguards must be tested under realistic failure conditions rather than assumed from good intentions.

This matters because AI systems can appear neutral while reproducing unequal treatment. A model may learn from biased historical data, use a poor proxy for the real target, work well for majority groups but badly for minorities, or encourage humans to over-trust an automated recommendation. Responsible AI is therefore not a slogan about being ethical. It is a governance discipline: map the risks, test the system, document decisions, keep humans meaningfully empowered, and create clear escalation routes when the system is wrong.

Bias starts before the model gives an answer

AI bias is often described as if it appears at the final output, but it usually enters much earlier. The training data may under-represent some groups, encode past discrimination, or contain measurements that look objective but are actually shaped by unequal access to services. A model can also become biased because engineers optimise for the wrong target: what is easy to measure may not be what is fair or useful.

A striking healthcare example came from research published in Science in 2019. The study found racial bias in a widely used health-risk algorithm because the system used healthcare spending as a proxy for medical need. Since Black patients in the dataset had historically received less spending than equally ill White patients, the algorithm underestimated their need for extra care. The researchers estimated that correcting the bias would raise the share of Black patients receiving additional help from 17.7% to 46.5%. The lesson is not simply “bad data causes bad AI”; it is that a seemingly practical measurement choice can quietly turn structural inequality into an automated decision rule.[Federal Trade Commission]ftc.govFederal Trade CommissionDissecting racial bias in an algorithm used to manage the…July 2, 2020 — by Z Obermeyer · Cited by 9554 — Thus…Published: July 2, 2020

Bias can also show up as uneven technical performance. The Gender Shades study by Joy Buolamwini and Timnit Gebru tested three commercial gender classification systems and found that darker-skinned women were the most misclassified group, with error rates of up to 34.7%, while the maximum error rate for lighter-skinned men was 0.8%. The study also noted that common benchmark datasets were heavily composed of lighter-skinned subjects, showing how “average accuracy” can hide serious failures for specific groups.[Proceedings of Machine Learning Research]proceedings.mlr.pressProceedings of Machine Learning ResearchGender Shades: Intersectional Accuracy Disparities in…January 21, 2018 — by J Buolamwini · 201…Published: January 21, 2018

This is why responsible AI testing has to look beyond a single headline accuracy score. A system used in hiring, policing, lending, education, healthcare or welfare can be unacceptable even if it performs well on average. The relevant question is: who gets the errors, how severe are they, and can affected people challenge the result?

Biased outputs become governance failures when nobody owns them

Bias becomes more dangerous when an organisation treats an AI output as if it were an independent authority. In reality, AI systems do not remove human judgement; they relocate it. Someone decides the system’s purpose, approves its data sources, signs the procurement contract, sets the confidence threshold, trains staff, and chooses whether users can appeal.

The Amazon recruiting-tool case is a useful warning. Reuters reported in 2018 that Amazon abandoned an experimental machine-learning recruitment system after it showed bias against women. The tool had reportedly learned from patterns in past applications in a male-dominated technical workforce, penalising signals associated with women’s résumés. The important point is not that all hiring AI will behave the same way, but that historical success data can teach a model to reproduce the inequalities of the past.[Reuters]reuters.comOpen source on reuters.com.

The same accountability problem appears in criminal justice and policing. ProPublica’s 2016 investigation of the COMPAS recidivism risk tool triggered a major debate about fairness metrics in risk prediction, including whether different definitions of fairness can conflict with one another. Later analysis and critique have shown how difficult it is to separate technical accuracy, legal accountability and social consequences when a score influences bail, sentencing or parole decisions.[ProPublica]propublica.orgPro Publica Machine BiasPro Publica Machine Bias

Facial recognition has produced even clearer examples of harm. The Innocence Project reported in 2024 that there were at least seven confirmed cases of misidentification linked to facial recognition technology, six involving Black people wrongfully accused. The ACLU reported in April 2026 that more than a dozen people were then known to have been wrongfully arrested after police reliance on facial recognition. These cases show why a “possible match” must never become a substitute for investigation, disclosure and independent evidence.[Innocence Project]innocenceproject.orgartificial intelligence is putting innocent people at risk of being incarceratedartificial intelligence is putting innocent people at risk of being incarcerated

Human oversight only works when humans can actually intervene

“Human in the loop” is one of the most common phrases in responsible AI, but it can be misleading. A human who rubber-stamps a recommendation, lacks time to investigate it, cannot see the evidence behind it, or faces pressure to follow the system is not a meaningful safeguard. They are part of the interface, not a source of accountability.

Research on human-AI decision-making shows why this matters. A 2024 study found that human judgement was affected when people received incorrect algorithmic support, particularly when the algorithmic advice came before their own judgement. In other words, an AI suggestion can anchor the human reviewer, making oversight weaker rather than stronger.[PMC]pmc.ncbi.nlm.nih.govPMCThe impact of AI errors in a human-in-the-loop processPMCThe impact of AI errors in a human-in-the-loop process

Regulators increasingly recognise this problem. The EU AI Act requires human oversight for high-risk AI systems and says that oversight should aim to prevent or minimise risks to health, safety and fundamental rights, including risks that remain even when other safeguards are in place.[Artificial Intelligence Act]artificialintelligenceact.euOpen source on artificialintelligenceact.eu. The European Data Protection Supervisor has also stressed that effective human oversight should have a tangible positive impact on outcomes, such as preventing harm, improving fairness and strengthening accountability.[European Data Protection Supervisor]edps.europa.euEuropean Data Protection Supervisor Tech DispatchEuropean Data Protection Supervisor Tech Dispatch

Useful escalation points are therefore practical, not symbolic. They include:

The goal is not to pretend humans are flawless. It is to design a workflow where human judgement has enough information, authority and time to matter.

Standards turn responsibility into repeatable work

Responsible AI is easier to discuss in principles than to implement in a busy organisation. That is why standards and risk frameworks matter. They translate broad values such as fairness and accountability into recurring tasks: inventory systems, classify risk, test models, document decisions, monitor performance, assign owners and review failures.

The NIST AI Risk Management Framework identifies trustworthy AI characteristics including validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. NIST also emphasises that these qualities must be balanced according to the system’s context of use, because an AI chatbot for drafting marketing copy does not carry the same risk as a model used in healthcare or criminal justice.[NIST AI Resource Center]airc.nist.govAI Resource Center AI Risks and TrustworthinessAI Resource Center AI Risks and Trustworthiness

The OECD AI Principles similarly frame trustworthy AI around human rights, democratic values, transparency, robustness, safety and accountability. Their importance lies partly in setting a common policy language: organisations cannot simply claim innovation as a reason to avoid responsibility for foreseeable harm.[OECD]oecd.orgOpen source on oecd.org.

ISO/IEC 42001 takes a management-system approach. ISO describes it as the first global standard for AI management systems, giving organisations requirements and guidance to establish, maintain and continually improve how they govern AI. In plain terms, it asks organisations to make responsible AI part of management practice rather than leaving it to ad hoc ethics reviews after a system is already built.[ISO]iso.orgiso 42001 explained what it isiso 42001 explained what it is

For generative AI and large language model applications, safeguards also need to cover security failure modes. OWASP’s Top 10 for Large Language Model Applications lists prompt injection as a leading risk, where crafted inputs manipulate a model into unauthorised or unsafe behaviour. It also highlights insecure output handling, where unvalidated model outputs can trigger downstream security problems. These risks show that responsible AI is not only about social bias; it is also about resilience when systems are connected to tools, databases, code or business processes.[OWASP Foundation]owasp.orgOpen source on owasp.org.

Testing must match the real world, not the demo

A responsible AI safeguard is only as strong as the test behind it. Many failures happen because systems are tested in clean conditions, then deployed into messy environments with poor-quality inputs, distribution shifts, rushed staff, adversarial users or people unlike those represented in the training data.

Good testing asks several questions at once. Does the system work for different demographic groups? Does it fail more often when image quality is poor, language is non-standard, forms are incomplete or users have disabilities? Does performance degrade after deployment? Can users manipulate it? Can staff detect when it is wrong? Does the system create a feedback loop, where earlier AI decisions shape the data used to train the next version?

The US Federal Trade Commission’s Rite Aid case shows what happens when safeguards are treated as optional. In 2023, the FTC announced that Rite Aid would be banned from using facial recognition technology for surveillance purposes for five years after allegations that the retailer failed to implement reasonable procedures to prevent consumer harm. The FTC specifically pointed to failures around accuracy testing, image-quality controls and risks of false positives, including risks linked to race and gender.[Federal Trade Commission]ftc.govFederal Trade Commission Rite Aid Banned from Using AI Facial Recognition AfterFederal Trade Commission Rite Aid Banned from Using AI Facial Recognition After

A practical responsible AI programme therefore needs layered controls:

Data checks: examine representativeness, missingness, labelling quality, proxy variables and historical bias.

Model evaluation: test subgroup performance, false positives, false negatives, calibration, robustness and uncertainty.

Human factors testing: check whether reviewers over-trust the system, understand its limits and have time to investigate.

Security testing: probe for prompt injection, data leakage, unsafe tool use, model poisoning and misuse.

Post-deployment monitoring: track drift, complaints, overrides, appeals, incidents and unexpected patterns of harm.

Withdrawal criteria: define in advance when a system must be paused, rolled back or replaced.

These are governance choices as much as technical ones. An organisation that cannot monitor or explain a high-impact AI system may not be ready to deploy it.

Accountability is clearest when failure paths are designed in advance

The hardest question is not “can AI be biased?” but “what happens when it is?” Responsible AI requires named owners, documented decisions and real consequences. Without those, accountability dissolves across vendors, data providers, internal teams, frontline staff and executives.

A useful responsibility map separates roles. The developer is responsible for design choices, documentation, testing and known limitations. The deploying organisation is responsible for deciding whether the system is appropriate in its setting. Managers are responsible for training, monitoring and escalation. Human reviewers are responsible for decisions only when they have genuine agency. Senior leaders are responsible for risk appetite, procurement standards and whether the organisation continues using a harmful system.

This is why “the vendor said it was accurate” is not enough. Accuracy depends on context: the population, input quality, decision threshold, workflow, user training and consequences of error. A facial recognition model, for instance, may perform differently on a controlled benchmark than on a blurry shop camera image or police still. A healthcare model may seem predictive while optimising for cost rather than care need. A hiring tool may look efficient while filtering out candidates who do not resemble past employees.

Responsible AI safeguards work best when they are boringly explicit: who signs off, who monitors, who can stop the system, who informs affected people, who investigates incidents, and who pays attention after launch. The central rule is simple: AI can support decisions, but it should not become a place where responsibility disappears.

1/2

Amazon book picks

Further Reading

Books and field guides related to Who Is Responsible When AI Fails?. Use these as the next step if you want deeper reading beyond the article.

Book

The Alignment Problem

By Brian Christian

Examines fairness, accountability, and value alignment challenges.

See on Amazon

Book

Atlas of AI

By Kate Crawford

Explores governance, power, bias, and societal impacts of AI.

See on Amazon

Book

Human Compatible

By Stuart Russell

Discusses safeguards and human control over AI systems.

See on Amazon

Book

Weapons of Math Destruction

By Cathy O'Neil

Directly addresses responsibility, bias, and consequences of automated decisions.

See on Amazon

Browse more on Amazon: The Alignment Problem Atlas of AI Human Compatible

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

I WAS AI before IT WAS COOL Enamel Pin Quotes Brooch Lapel Pins Clothing

SearcheBay.co.uk: AI enamel pin

Browse similar oneBay.co.uk

Example eBay listing

New Red Crystal Enamel Pin Brooches for Women UK Ladies Dress Art Deco Gold Plt

SearcheBay.co.uk: AI enamel pin

Browse similar oneBay.co.uk

Example eBay listing

Terminator Movie Enamel Pin Badge Cyberdyne Systems AI Skynet Metal Alloy Brooch

SearcheBay.co.uk: AI enamel pin

Browse similar oneBay.co.uk

Example eBay listing

Hold On Let Me ChatGPT This Enamel Pin Badge – Funny Tech Gift for AI Fans

SearcheBay.co.uk: AI enamel pin

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: ftc.gov
Link:https://www.ftc.gov/system/files/documents/public_events/1548288/privacycon-2020-ziad_obermeyer.pdf

Published: July 2, 2020

2. Source: reuters.com
Link:https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK0AG/

3. Source: propublica.org
Title: Pro Publica Machine Bias
Link:https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

4. Source: aclu.org
Link:https://www.aclu.org/news/privacy-technology/more-than-a-dozen-wrongful-arrests-due-to-police-reliance-on-facial-recognition-technology

5. Source: pmc.ncbi.nlm.nih.gov
Title: PMCThe impact of [AI errors]({{ ‘ai-errors/’ | relative_url }}) in a human-in-the-loop process
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10772030/

6. Source: airc.nist.gov
Title: AI Resource Center AI Risks and Trustworthiness
Link:https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/

7. Source: nvlpubs.nist.gov
Title: ai.100 1
Link:https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf

8. Source: oecd.org
Link:https://www.oecd.org/en/topics/sub-issues/ai-principles.html

9. Source: legalinstruments.oecd.org
Link:https://legalinstruments.oecd.org/en/instruments/oecd-legal-0449

10. Source: iso.org
Title: iso 42001 explained what it is
Link:https://www.iso.org/home/insights-news/resources/iso-42001-explained-what-it-is.html

11. Source: owasp.org
Link:https://owasp.org/www-project-top-10-for-large-language-model-applications/

12. Source: genai.owasp.org
Title: Gen AI Security Project LLMRisks Archive
Link:https://genai.owasp.org/llm-top-10/

13. Source: ftc.gov
Title: Federal Trade Commission Rite Aid Banned from Using AI Facial Recognition After
Link:https://www.ftc.gov/news-events/news/press-releases/2023/12/rite-aid-banned-using-ai-facial-recognition-after-ftc-says-retailer-deployed-technology-without

14. Source: nist.gov
Link:https://www.nist.gov/itl/ai-risk-management-framework

15. Source: nist.gov
Link:https://www.nist.gov/

16. Source: oecd.org
Link:https://www.oecd.org/en.html

17. Source: oecd.org
Title: 2448f04b en
Link:https://www.oecd.org/content/dam/oecd/en/publications/reports/2023/02/advancing-accountability-in-ai_753bf8c8/2448f04b-en.pdf

18. Source: oecd.org
Title: ba602d18 en
Link:https://www.oecd.org/content/dam/oecd/en/publications/reports/2023/11/common-guideposts-to-promote-interoperability-in-ai-risk-management_9629ed36/ba602d18-en.pdf

19. Source: genai.owasp.org
Link:https://genai.owasp.org/

20. Source: propublica.org
Title: how we analyzed the compas recidivism algorithm
Link:https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

21. Source: oecd.ai
Link:https://oecd.ai/en/ai-principles

22. Source: oecd.ai
Link:https://oecd.ai/en/dashboards/ai-principles/P7

23. Source: oecd.ai
Link:https://oecd.ai/en/dashboards/ai-principles/P8

24. Source: aws.amazon.com
Title: ai lifecycle risk management iso iec 420012023 for ai governance
Link:https://aws.amazon.com/blogs/security/ai-lifecycle-risk-management-iso-iec-420012023-for-ai-governance/

25. Source: ftc.gov
Title: 2023190 rite aid corporation ftc v
Link:https://www.ftc.gov/legal-library/browse/cases-proceedings/2023190-rite-aid-corporation-ftc-v

26. Source: proceedings.mlr.press
Link:https://proceedings.mlr.press/v81/buolamwini18a

Published: January 21, 2018

27. Source: innocenceproject.org
Title: artificial intelligence is putting innocent people at risk of being incarcerated
Link:https://innocenceproject.org/news/artificial-intelligence-is-putting-innocent-people-at-risk-of-being-incarcerated/

28. Source: artificialintelligenceact.eu
Link:https://artificialintelligenceact.eu/article/14/

29. Source: digital-strategy.ec.europa.eu
Title: Digital Strategy AI Act | Shaping Europe’s digital future
Link:https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai

30. Source: edps.europa.eu
Title: European Data Protection Supervisor Tech Dispatch
Link:https://www.edps.europa.eu/system/files/2025-09/25-09-15_techdispatch-human-oversight_en.pdf

31. Source: iso-library.com
Link:https://iso-library.com/standard/23894/

32. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/31649194/

33. Source: sentinelone.com
Title: nist ai risk management framework
Link:https://www.sentinelone.com/cybersecurity-101/cybersecurity/nist-ai-risk-management-framework/

34. Source: isms.online
Title: iso 42001
Link:https://www.isms.online/iso-42001/

35. Source: edps.europa.eu
Title: 2025 11 11 ai risks management guidance en
Link:https://www.edps.europa.eu/system/files/2025-11/2025-11-11_ai_risks_management_guidance_en.pdf

36. Source: kpmg.com
Title: iso iec 42001
Link:https://kpmg.com/ch/en/insights/artificial-intelligence/iso-iec-42001.html

37. Source: proceedings.mlr.press
Link:https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf

38. Source: GOV.UK
Title: ethics transparency and accountability framework for automated decision making
Link:https://www.gov.uk/government/publications/ethics-transparency-and-accountability-framework-for-automated-decision-making/ethics-transparency-and-accountability-framework-for-automated-decision-making

39. Source: academy.evalcommunity.com
Title: oecd ai principles
Link:https://academy.evalcommunity.com/oecd-ai-principles/

40. Source: scrut.io
Title: nist ai risk management framework
Link:https://www.scrut.io/post/nist-ai-risk-management-framework

41. Source: openpracticelibrary.com
Title: human in the loop
Link:https://openpracticelibrary.com/practice/human-in-the-loop/

42. Source: gotrust.nl
Title: iso 42001
Link:https://www.gotrust.nl/iso-42001

43. Source: autoriteitpersoonsgegevens.nl
Title: eu ai act
Link:https://www.autoriteitpersoonsgegevens.nl/en/themes/algorithms-ai/eu-ai-act

44. Source: artificialintelligenceact.eu
Link:https://artificialintelligenceact.eu/high-level-summary/

45. Source: artificialintelligenceact.eu
Link:https://artificialintelligenceact.eu/section/3-2/

46. Source: aiethicslab.rutgers.edu
Title: oecd ai principles
Link:https://aiethicslab.rutgers.edu/glossary/oecd-ai-principles/

47. Source: diligent.com
Title: nist ai risk management framework
Link:https://www.diligent.com/resources/blog/nist-ai-risk-management-framework

48. Source: witness.ai
Title: ai risk management
Link:https://witness.ai/blog/ai-risk-management/

Additional References

49. Source: chicagobooth.edu
Link:https://www.chicagobooth.edu/research/tolan/research/2019/dissecting-racial-bias-in-an-algorithm-used-to-manage-the-health-of-populations

50. Source: youtube.com
Title: Bias, Fairness & Responsible AI Explained | AI Ethics in Data Science
Link:https://www.youtube.com/watch?v=C78FeCxVCaY

51. Source: youtube.com
Title: The Double-Edged Sword of AI, with Dr. Ziad Obermeyer
Link:https://www.youtube.com/watch?v=c5KcSNeGXOM

52. Source: researchgate.net
Link:https://www.researchgate.net/publication/373896468_Gender_Bias_in_Hiring_An_Analysis_of_the_Impact_of_Amazon%27s_Recruiting_Algorithm

53. Source: wilmerhale.com
Link:https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20240717-what-are-highrisk-ai-systems-within-the-meaning-of-the-eus-ai-act-and-what-requirements-apply-to-them

54. Source: marcellodibello.com
Link:https://www.marcellodibello.com/algorithmicfairness/handout/ProPublica-Northpointe.html

55. Source: allendowney.github.io
Link:https://allendowney.github.io/RecidivismCaseStudy/02_calibration.html

56. Source: aiactblog.nl
Link:https://www.aiactblog.nl/en/ai-act/artikel/14

57. Source: medium.com
Link:https://medium.com/%40lamdaa/compas-unfair-algorithm-812702ed6a6a

58. Source: cycoresecure.com
Link:https://www.cycoresecure.com/frameworks/eu-ai-act