Within Machine Learning
When learned patterns become unfair
A model can reproduce unfair patterns when the examples, labels, or surrounding systems reflect unequal treatment.
On this page
- How bias enters examples and labels
- Why the model may copy old decisions
- How evaluation can look for uneven harm
Page outline Jump by section
Introduction
Artificial intelligence systems learn from examples. That ability makes them useful, but it also creates a risk: if the examples reflect unequal treatment, incomplete measurement, or historical discrimination, the model may learn patterns that reproduce those problems. In machine learning, a model is not given a moral understanding of fairness. It is rewarded for finding statistical relationships in data. As a result, unfair outcomes can emerge even when developers did not intend them. Research and policy bodies such as the National Institute of Standards and Technology (NIST) describe AI bias as a socio-technical problem that can arise from data, human decisions, institutional practices, and the way systems are deployed. [NIST Publications+2NIST]nvlpubs.nist.govNIST PublicationsTowards a Standard for Identifying and Managing Bias in…by R Schwartz · 2022 · Cited by 808 — Systemic and implicit b…
Understanding biased data is important because AI systems increasingly influence hiring, lending, healthcare, education, policing, and online services. When a model learns an unfair pattern, it can scale that pattern across thousands or millions of decisions.
How bias enters examples and labels
Bias can enter a machine-learning system long before any model is trained. The most obvious route is through the examples used for learning.
A dataset may fail to represent the population on which the model will be used. If certain groups appear less often in the training data, the model may learn weaker patterns for them and perform less accurately. This problem has been documented in facial analysis and recognition systems, where underrepresentation of women and people with darker skin tones contributed to uneven performance across demographic groups. [NIST+2MIT Sloan]nist.govstudy evaluates effects race age sex face recognition softwareNIST Study Evaluates Effects of Race, Age, Sex on Face…Dec 19, 2019 — A new NIST study examines how accurately face recognition so…
Labels can also introduce bias. In supervised learning, models learn from examples paired with a “correct” answer. However, labels are often created by people or derived from historical decisions. If past decisions were themselves influenced by prejudice, unequal access, or flawed assumptions, the labels may encode those patterns. The model then learns not only the intended task but also traces of the historical behaviour embedded in the data. NIST notes that harmful bias can emerge from systemic and institutional conditions reflected in training data, not merely from technical errors. [NIST Publications+2NIST]nvlpubs.nist.govNIST PublicationsTowards a Standard for Identifying and Managing Bias in…by R Schwartz · 2022 · Cited by 808 — Systemic and implicit b…
Another source of bias is incomplete measurement. Suppose a system is trained to predict “job success” using historical promotion records. Promotions may reflect workplace inequalities rather than pure job performance. The model cannot distinguish between genuine ability and distortions hidden within the data. It learns whatever statistical relationship best predicts the recorded outcomes.
Why the model may copy old decisions
Machine-learning models are designed to identify patterns that help them make accurate predictions. They do not automatically know which patterns are socially acceptable and which are problematic.
If historical decisions contain unequal treatment, a model may discover that those patterns improve predictive accuracy on its training data. From the model’s perspective, reproducing the pattern is a successful strategy because it reduces error according to the objective it was given.
A widely discussed example comes from risk assessment tools used in criminal justice. Analyses of the COMPAS recidivism prediction system found evidence of racial disparities in its predictions, raising concerns that historical data and statistical relationships could contribute to unequal outcomes for different groups. The debate around COMPAS became influential because it illustrated how an apparently objective algorithm can inherit controversial patterns from past records. [arXiv]arxiv.orgarXiv Algorithmic Bias in Recidivism Prediction: A Causal PerspectiveAlgorithmic Bias in Recidivism Prediction: A Causal PerspectiveNovember 24, 2019…
The same mechanism can occur in many settings:
- A hiring model may learn preferences that mirror past recruitment decisions.
- A lending model may reflect historical differences in access to credit.
- A healthcare model may perform worse for groups that were underrepresented in medical data.
- A recommendation system may reinforce existing social stereotypes if those stereotypes appear frequently in its training material.
These outcomes do not necessarily require explicit information about race, gender, or other protected characteristics. Correlated variables can sometimes act as indirect signals, allowing patterns of unequal treatment to persist even when sensitive attributes are removed. This is one reason why fairness cannot be solved simply by deleting a single column from a dataset. [NIST Publications]nvlpubs.nist.govNIST PublicationsTowards a Standard for Identifying and Managing Bias in…by R Schwartz · 2022 · Cited by 808 — Systemic and implicit b…
Real-world examples of uneven performance
The consequences of biased learning become most visible when systems are tested on different populations.
One of the most influential studies came from NIST’s evaluation of facial recognition technologies. Researchers found that the majority of tested algorithms exhibited demographic performance differences, with error rates varying across race, age, and sex. In several identification tasks, some algorithms were far more likely to produce false matches for certain demographic groups than for others. [NIST]nist.govstudy evaluates effects race age sex face recognition softwareNIST Study Evaluates Effects of Race, Age, Sex on Face…Dec 19, 2019 — A new NIST study examines how accurately face recognition so…
Subsequent analyses and reviews highlighted similar concerns. Researchers and commentators noted that some facial recognition systems misidentified Black and East Asian faces at substantially higher rates than white faces, while women often experienced higher error rates than men. These findings helped transform AI fairness from a niche technical topic into a broader public policy issue. Scientific American+2Harvard Journal of Law & Technology [scientificamerican.com]scientificamerican.comhow nist tested facial recognition algorithms for racial biasScientific AmericanHow NIST Tested Facial-Recognition Algorithms for Racial…Dec 27, 2019 — NIST's tests revealed that many of these al…
The lesson extends beyond facial recognition. Bias can appear whenever training data, labels, or deployment conditions differ across groups. A model can achieve impressive average accuracy while still imposing disproportionate errors on specific populations.
How evaluation can look for uneven harm
Detecting unfair learned patterns requires more than measuring overall accuracy.
A model that is 95% accurate on average may still perform poorly for particular groups. For that reason, fairness evaluation often examines outcomes separately across demographic categories and compares error rates, access to benefits, or exposure to harms. [OECD.AI]oecd.aiEqual outcomesThis metric addresses Fairness and Human Agency… demographic parity or equalized odds and is used to detect and mitigate…
Several fairness metrics have been proposed: [geeksforgeeks.org]geeksforgeeks.orgFairness MetricsDemographic Parity, Equalized OddsJul 23, 2025 — Fairness metrics like Demographic Parity and Equalized Odds help check if an AI system t…
- Demographic parity asks whether positive outcomes occur at similar rates across groups.
- Equal opportunity examines whether qualified individuals receive favourable outcomes at similar rates.
- Equalized odds compares error rates across groups to identify systematic disparities.
- Outcome-based measures evaluate whether benefits and harms are distributed unevenly across populations. [OECD.AI+2IEIE SPC]oecd.aiEqual outcomesThis metric addresses Fairness and Human Agency… demographic parity or equalized odds and is used to detect and mitigate…
However, fairness assessment is not straightforward. Different fairness metrics can conflict with one another, meaning that improving one measure may worsen another. Researchers therefore emphasise that fairness is partly a governance question: organisations must decide which harms matter most in a particular application and justify the trade-offs they make. [PMC]pmc.ncbi.nlm.nih.govBias in AI systems: integrating formal and socio-technical…by A Ahmad · 2026 — The technical fairness literature proposes various m…
Evaluation also requires ongoing monitoring. A model that appears fair during development may produce different outcomes when deployed in a changing real-world environment. NIST and other governance frameworks therefore recommend continuous testing, documentation, and review rather than treating fairness as a one-time certification exercise. [NIST+2Scrut]nist.govAI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society a…
Why biased data remains a governance challenge
A common misconception is that biased AI is simply a technical bug that can be fixed by improving code. In practice, many fairness problems originate in social systems, organisational processes, and historical data. NIST’s work on AI bias stresses that harmful outcomes can arise from systemic, human, and institutional sources as well as computational ones. [NIST+2NIST Publications]nist.govtheres more ai bias biased data nist report highlightsThere's More to AI Bias Than Biased Data, NIST Report…Mar 16, 2022 — The NIST report acknowledges that a great deal of AI bias ste…
This makes governance essential. Organisations must ask where data came from, whose experiences are missing, how labels were created, and which groups bear the greatest risks if the system makes mistakes. International frameworks and AI governance initiatives increasingly emphasise fairness, accountability, and human rights because technical performance alone cannot guarantee equitable outcomes. [OECD+2OECD Legal Instruments]oecd.orgAI principlesThe OECD AI Principles promote use of AI that is innovative and trustworthy and that respects human rights and democrati…
Biased data matters because machine-learning systems learn from the world as it is recorded, not necessarily as it ought to be. Without careful evaluation and oversight, AI can transform historical patterns of unequal treatment into automated decisions that appear objective while continuing to distribute benefits and harms unevenly. [ibm.com+2ibm.com]ibm.comtory outcomes…
Amazon book picks
Further Reading
Books and field guides related to When learned patterns become unfair. Use these as the next step if you want deeper reading beyond the article.
Weapons of Math Destruction
Shows how biased data and models can create unfair outcomes.
Algorithms of Oppression
Examines how data and algorithms can reproduce social inequalities.
Race After Technology
Explores bias and discrimination embedded in technological systems.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: nvlpubs.nist.gov
Link: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdfSource snippet
NIST PublicationsTowards a Standard for Identifying and Managing Bias in...by R Schwartz · 2022 · Cited by 808 — Systemic and implicit b...
-
Source: nist.gov
Title: theres more ai bias biased data nist report highlights
Link: https://www.nist.gov/news-events/news/2022/03/theres-more-ai-bias-biased-data-nist-report-highlightsSource snippet
There's More to AI Bias Than Biased Data, NIST Report...Mar 16, 2022 — The NIST report acknowledges that a great deal of AI bias ste...
-
Source: nist.gov
Link: https://www.nist.gov/itl/ai-risk-management-frameworkSource snippet
AI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society a...
-
Source: nist.gov
Title: study evaluates effects race age sex face recognition software
Link: https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-softwareSource snippet
NIST Study Evaluates Effects of Race, Age, Sex on Face...Dec 19, 2019 — A new NIST study examines how accurately face recognition so...
-
Source: mitsloan.mit.edu
Title: unmasking bias facial recognition algorithms
Link: https://mitsloan.mit.edu/ideas-made-to-matter/unmasking-bias-facial-recognition-algorithmsSource snippet
MIT SloanUnmasking the bias in facial recognition algorithmsDec 13, 2023 — In this excerpt, Buolamwini discusses how datasets used to tra...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCBeating the bias in facial recognition technology
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7575263/Source snippet
the bias in facial recognition technology - PMC - NIHby J Lunter · 2020 · Cited by 38 — NIST figures highlight the continuing bias proble...
-
Source: arxiv.org
Title: arXiv Algorithmic Bias in Recidivism Prediction: A Causal Perspective
Link: https://arxiv.org/abs/1911.10640Source snippet
Algorithmic Bias in Recidivism Prediction: A Causal PerspectiveNovember 24, 2019...
Published: November 24, 2019
-
Source: oecd.org
Title: algorithmic bias the state of the situation and policy recommendations a0b7cec1
Link: https://www.oecd.org/en/publications/oecd-digital-education-outlook-2023_c74f03de-en/full-report/algorithmic-bias-the-state-of-the-situation-and-policy-recommendations_a0b7cec1.htmlSource snippet
Algorithmic bias: the state of the situation and policy...Dec 13, 2023 — The term algorithmic bias has been used to describe many proble...
-
Source: nvlpubs.nist.gov
Link: https://nvlpubs.nist.gov/nistpubs/ir/2019/NIST.IR.8280.pdfSource snippet
NIST PublicationsFace Recognition Vendor Test (FRVT), Part 3: Demographic...by P Grother · 2019 · Cited by 93 — NIST has conducted tests...
-
Source: jolt.law.harvard.edu
Title: why racial bias is prevalent in facial recognition technology
Link: https://jolt.law.harvard.edu/digest/why-racial-bias-is-prevalent-in-facial-recognition-technologySource snippet
Harvard Journal of Law & TechnologyWhy Racial Bias is Prevalent in Facial Recognition...Nov 3, 2020 — Many of these algorithms were foun...
-
Source: oecd.ai
Link: https://oecd.ai/en/catalogue/metrics/equal-outcomesSource snippet
Equal outcomesThis metric addresses Fairness and Human Agency... demographic parity or equalized odds and is used to detect and mitigate...
-
Source: scrut.io
Link: https://www.scrut.io/glossary/fairness-and-bias-nist-ai-rmfSource snippet
lgorithmic checks...
-
Source: ieiespc.org
Link: https://ieiespc.org/ieiespc/XmlViewer/f437628Source snippet
aphic parity and equalized odds. is a meta-algorithm that modifies the...Read more...
-
Source: arxiv.org
Link: https://arxiv.org/html/2403.16101v1Source snippet
Evaluating Fairness Metrics Across Borders from Human...Mar 24, 2024 — We use four fairness metrics; quantitative parity demographi...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12823528/Source snippet
Bias in AI systems: integrating formal and socio-technical...by A Ahmad · 2026 — The technical fairness literature proposes various m...
-
Source: oecd.org
Link: https://www.oecd.org/en/topics/sub-issues/ai-principles.htmlSource snippet
AI principlesThe OECD AI Principles promote use of AI that is innovative and trustworthy and that respects human rights and democrati...
-
Source: legalinstruments.oecd.org
Link: https://legalinstruments.oecd.org/en/instruments/oecd-legal-0449Source snippet
OECD Legal InstrumentsRecommendation of the Council on Artificial IntelligenceOECD.AI is an inclusive hub for public policy on AI that ai...
-
Source: oecd.ai
Link: https://oecd.ai/en/catalogue/metrics/safe-sustainable%2C-accurate%2C-fair-and-explainableSource snippet
SAFE (Sustainable, Accurate, Fair and Explainable)These tools and metrics are designed to help AI actors develop and use trustworthy AI s...
-
Source: ibm.com
Link: https://www.ibm.com/think/topics/algorithmic-biasSource snippet
tory outcomes...
-
Source: ibm.com
Link: https://www.ibm.com/think/topics/ai-biasSource snippet
istorted and potentially harmful outputs...
-
Source: nist.gov
Link: https://www.nist.gov/Source snippet
National Institute of Standards and TechnologyNIST promotes U.S. innovation and industrial competitiveness by advancing measurement scien...
-
Source: nist.gov
Link: https://www.nist.gov/video/bias-aiSource snippet
bias can be purposeful...
-
Source: nvlpubs.nist.gov
Title: AI.600 1
Link: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdfSource snippet
Intelligence Risk Management Frameworkby N AI · 2024 · Cited by 115 — Review, document, and measure sources of bias in GAI training and T...
-
Source: pages.nist.gov
Title: frvt demographics
Link: https://pages.nist.gov/frvt/html/frvt_demographics.htmlSource snippet
Effects in Face Recognition - NIST PagesThis page summarizes and links to all FRTE data and reports related to demographic effects in fac...
-
Source: oecd.org
Link: https://www.oecd.org/content/dam/oecd/en/publications/reports/2019/06/artificial-intelligence-in-society_c0054fa1/eedfee77-en.pdfSource snippet
irness represents the notion of equality before the law.Read more...
-
Source: oecd.ai
Link: https://oecd.ai/en/catalogue/tools?page=1&terms=biasSource snippet
ctive(s). FairnessData Governance &...Read more...
-
Source: oecd.ai
Link: https://oecd.ai/en/catalogue/metrics/equal-performanceSource snippet
Equal performanceFairness: Equal performance directly measures parity in outcomes (or error rates/performance metrics) across defined gro...
-
Source: scientificamerican.com
Title: how nist tested facial recognition algorithms for racial bias
Link: https://www.scientificamerican.com/article/how-nist-tested-facial-recognition-algorithms-for-racial-bias/Source snippet
Scientific AmericanHow NIST Tested Facial-Recognition Algorithms for Racial...Dec 27, 2019 — NIST's tests revealed that many of these al...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8475322/Source snippet
Face Recognition Technology Used by Governmentby M Gentzel · 2021 · Cited by 61 — This paper presents a novel philosophical analysis of t...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7879975/Source snippet
comparison across face recognition algorithms - PMCby JG Cavazos · 2020 · Cited by 304 — We conclude that race bias needs to be measured...
-
Source: geeksforgeeks.org
Title: Fairness Metrics
Link: https://www.geeksforgeeks.org/artificial-intelligence/fairness-metrics-demographic-parity-equalized-odds/Source snippet
Demographic Parity, Equalized OddsJul 23, 2025 — Fairness metrics like Demographic Parity and Equalized Odds help check if an AI system t...
Additional References
-
Source: linkedin.com
Link: https://www.linkedin.com/pulse/fairness-ai-intent-impact-perspective-igor-perisicSource snippet
Fairness in AI: An Intent and Impact PerspectiveFor example, you can't always be “fair” according to both Equalized Odds and Predictive R...
-
Source: cogentinfo.com
Link: https://cogentinfo.com/resources/addressing-gender-bias-in-facial-recognition-technology-an-urgent-need-for-fairness-and-inclusionSource snippet
Addressing Gender Bias in Facial Recognition TechnologyApr 14, 2025 — The NIST report (2019) noted that most commercial facial recognitio...
-
Source: eng.kwdi.re.kr
Link: https://eng.kwdi.re.kr/publications/researchReportDetail.do?idx=104361&p=2Source snippet
Bias in the Use of Artificial Intelligence Deep...In these cases, artificial intelligence working through algorithms brought consequence...
-
Source: advancingjustice-aajc.org
Link: https://www.advancingjustice-aajc.org/sites/default/files/2020-02/Algorithmic%20Bias%20Backgrounder_0.pdf -
Source: ansi.org
Link: https://www.ansi.org/standards-news/all-news/3-21-22-theres-more-to-ai-bias-than-biased-data-nist-report-findsSource snippet
3/21/2022. Join NIST's March Workshop on AI Bias. As part of an ongoing effort to support the...Read more...
-
Source: facialsuite.com
Link: https://facialsuite.com/Source snippet
Explore personalized facials, skin rejuvenation, aesthetic treatments, wellness services, and advanced...
-
Source: hyperproof.io
Link: https://hyperproof.io/navigating-the-nist-ai-risk-management-framework/Source snippet
Navigating the NIST AI Risk Management FrameworkIn this guide, we will provide a comprehensive overview of AI risk, the challenges you mi...
-
Source: ama.drwhy.ai
Link: https://ama.drwhy.ai/nist-ai-risk-management-framework-ai-rmf.htmlSource snippet
NIST has identified three major categories of AI bias to be considered and managed...Read more...
-
Source: wired.com
Title: best algorithms struggle recognize black faces equally
Link: https://www.wired.com/story/best-algorithms-struggle-recognize-black-faces-equally/Source snippet
The Best Algorithms Still Struggle to Recognize Black FacesJul 22, 2019 — US government tests find even top-performing facial recognition...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/384496830_UNESCO%27s_AI_Ethics_Principles_Challenges_and_OpportunitiesSource snippet
ce, which outlines key principles for ensuring [responsible AI]({{ 'responsible-ai/' | relative_url }}) development.Read more...
Topic Tree