Within Biased data
When old decisions become AI training data
Historical decisions can become training labels that make unfair treatment look like a prediction target rather than a warning sign.
On this page
- How labels inherit past human choices
- Why prediction rewards can preserve unfair patterns
- Examples from hiring, lending, and promotion records
Page outline Jump by section
Introduction
One of the most important ways unfairness enters artificial intelligence is through the labels used during training. In supervised machine learning, a model learns by studying examples paired with an answer or outcome. Those answers are often treated as “ground truth” — the correct result the system should learn to reproduce. The problem is that many labels are not objective facts. They are records of past human decisions, institutional practices, or resource allocations that may already contain unequal treatment.
When historical decisions become training labels, an AI system can learn that those decisions are the target to imitate. Instead of recognising past discrimination as a warning sign, the model interprets it as evidence of what successful prediction looks like. As a result, unfair patterns can be preserved or even scaled across future decisions. Researchers increasingly describe this as a label problem rather than simply a data problem: the model may be learning exactly what it was asked to learn, but the labels themselves reflect a biased history. [arXiv]arxiv.orgarXiv Identifying and Correcting Label Bias in Machine LearningIdentifying and Correcting Label Bias in Machine LearningJanuary 15, 2019…
How labels inherit past human choices
A supervised learning system does not understand why a label exists. It only sees a relationship between inputs and outcomes.
Suppose a company trains a hiring model using ten years of recruitment records. The model receives résumés as inputs and past hiring decisions as labels. From the system’s perspective, people who were hired become examples of success and people who were rejected become examples of failure.
The crucial issue is that hiring decisions are not purely measures of talent. They may also reflect organisational preferences, historical inequalities, workplace culture, or conscious and unconscious bias. If those influences affected previous decisions, they become embedded in the labels themselves. The AI is then rewarded for finding patterns that reproduce those outcomes. [PMC]pmc.ncbi.nlm.nih.govAI bias: exploring discriminatory algorithmic decision-making…by L Belenguer · 2022 · Cited by 359 — The data used in training is i…
This mechanism differs from simple underrepresentation. Even if a dataset contains many examples from different groups, the labels attached to those examples may still encode unequal treatment. The model is not merely learning from people; it is learning from decisions made about people.
Researchers studying label bias have shown that classifiers can inherit unfairness when observed labels differ from the outcomes that would have existed under fair treatment. In such cases, the model faithfully reproduces biased labels because those labels define the training objective. [arXiv]arxiv.orgarXiv Identifying and Correcting Label Bias in Machine LearningIdentifying and Correcting Label Bias in Machine LearningJanuary 15, 2019…
Why prediction rewards can preserve unfair patterns
Machine-learning systems are usually optimised to maximise predictive accuracy. If historical decisions contain systematic patterns, reproducing those patterns often improves performance on the training data.
This creates a difficult paradox. The model may appear successful because it accurately predicts past decisions, yet those decisions may not represent the outcome society actually wants.
A hiring model trained on previous recruitment choices is rewarded when it predicts whom recruiters selected in the past. A lending model is rewarded when it predicts which applicants previously received loans. A promotion model is rewarded when it predicts who was promoted before. The optimisation process encourages imitation of historical outcomes, not evaluation of whether those outcomes were fair. [FRA]fra.europa.eufra 2022 bias in algorithms enBias in algorithms – Artificial intelligence and discrimination7 Dec 2022 — Algorithms used for predictive policing and for offensive…
Because machine-learning systems are designed to discover statistical regularities, they can identify subtle correlations that humans may not notice. If a protected group historically received fewer opportunities, the model may learn indirect indicators associated with that group’s lower historical success rates. From the model’s perspective, these correlations are useful predictive signals. From a fairness perspective, they may simply be traces of past exclusion.
Hiring records: when recruitment history becomes the target
A widely discussed example emerged from Amazon’s experimental recruiting system. The company trained a machine-learning tool using historical résumés and hiring patterns collected over many years. Because the technology industry and Amazon’s technical workforce were heavily male-dominated during that period, the training data reflected a history in which men were more frequently hired for technical roles. [Reuters]reuters.comInsight - Amazon scraps secret AI recruiting tool that…Oct 9, 2018 — SAN FRANCISCO (Reuters) - Amazon.com Inc's machine-learnin…
According to reporting by Reuters, the system learned patterns that disadvantaged female applicants. It reportedly downgraded certain résumé features associated with women because those features were statistically linked to candidates who had not been hired in the historical data. [Reuters]reuters.comInsight - Amazon scraps secret AI recruiting tool that…Oct 9, 2018 — SAN FRANCISCO (Reuters) - Amazon.com Inc's machine-learnin…
The important lesson is that the model did not invent a preference against women on its own. It learned from labels generated by past hiring outcomes. Historical hiring decisions functioned as examples of what the system should emulate. Because the labels reflected a male-dominated recruitment history, the model treated that history as predictive guidance. [Reuters+2Reddit]reuters.comInsight - Amazon scraps secret AI recruiting tool that…Oct 9, 2018 — SAN FRANCISCO (Reuters) - Amazon.com Inc's machine-learnin…
This case became influential because it demonstrated how an apparently objective prediction system can transform organisational history into future decision-making rules.
Lending records: approval decisions are not neutral labels
Credit and lending systems face a similar challenge. Historical loan approvals and rejections are often used to train predictive models.
At first glance, approval records appear useful because they capture previous lending decisions. However, they may also reflect differences in access to financial services, historical redlining practices, unequal wealth accumulation, or institutional risk assessments shaped by past discrimination.
If a model learns primarily from previous approval decisions, it may conclude that applicants who resemble historically favoured groups are safer choices. The system can therefore reproduce patterns embedded in earlier lending behaviour, even when those patterns originated outside the model itself. [FRA]fra.europa.eufra 2022 bias in algorithms enBias in algorithms – Artificial intelligence and discrimination7 Dec 2022 — Algorithms used for predictive policing and for offensive…
The challenge is that a loan approval label often represents a human decision rather than a direct measurement of a person’s true creditworthiness. Treating the decision as unquestionable ground truth risks turning past institutional choices into future automated choices.
Promotion records: success labels can reflect unequal opportunity
Promotion data is frequently used as a proxy for employee performance. Yet promotion is not identical to merit.
A promotion depends on many factors beyond individual ability: access to influential projects, visibility to managers, mentoring opportunities, workplace culture, networking, and evaluation practices. If these opportunities have historically been distributed unevenly, promotion records may capture those inequalities alongside genuine performance differences.
When promotion outcomes become labels, a model may learn which employees historically advanced rather than which employees demonstrated the strongest potential. The system can then reinforce existing organisational patterns by identifying candidates who resemble previous promotees. In effect, the AI treats historical advancement as evidence of future worthiness. [PMC]pmc.ncbi.nlm.nih.govAI bias: exploring discriminatory algorithmic decision-making…by L Belenguer · 2022 · Cited by 359 — The data used in training is i…
This is one reason researchers often distinguish between observed outcomes and the underlying qualities organisations actually wish to measure.
The healthcare example: when the label itself measures the wrong thing
One of the clearest demonstrations of label-driven bias came from healthcare risk prediction.
Researchers led by Ziad Obermeyer examined a widely used health-management algorithm that helped determine which patients should receive additional care. The system predicted future healthcare costs and used those cost predictions as a proxy for medical need. Because Black patients historically received less healthcare spending than White patients with similar health conditions, healthcare costs were an imperfect label for illness. [Science+2PubMed]science.orgLess money is spent on Black patients who have the same…Read more…
The algorithm therefore learned a misleading lesson. Lower spending appeared to indicate lower need, even when patients were equally or more ill. As a result, many Black patients were assigned lower risk scores than their health status warranted. Researchers found that changing the target variable substantially reduced the racial disparity. [Booth School of Business]chicagobooth.eduBooth School of BusinessDissecting racial bias in an algorithm used to manage the…We show that a widely used algorithm, typical of thi…
This case illustrates a broader principle: biased labels do not always arise from prejudice in individual decisions. Sometimes the label is a convenient proxy that reflects unequal social conditions. The model faithfully predicts the label, but the label does not faithfully represent the concept that decision-makers actually care about. [Booth School of Business]chicagobooth.eduhow racial bias infected major health care algorithmBooth School of BusinessHow Racial Bias Infected a Major Health-Care Algorithm25 Oct 2019 — The algorithm uses health-care costs as a pro…
Why fixing the data is not always enough
A common misconception is that bias can be removed simply by collecting more data. Additional data may help in some situations, but it does not automatically solve label bias.
If the labels themselves encode unfair decisions, a larger dataset may merely provide more examples of the same pattern. The model can become even better at reproducing historical behaviour.
For this reason, many fairness researchers focus on examining how labels were created. Questions include:
- What real-world process generated the label?
- Does the label represent a fact, a judgement, or a historical decision?
- Could unequal treatment have influenced the outcome?
- Is the label a proxy for something that cannot be directly observed?
- Would a different target better represent the goal of the system?
Research on algorithmic fairness increasingly treats label quality as a central concern because inaccurate or biased labels can distort both model training and fairness audits. [arXiv]arxiv.orgarXiv Who Decides if AI is Fair? The Labels Problem in Algorithmic AuditingarXiv Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing
The key lesson: AI often learns decisions, not just facts
The phrase “biased data” can make the problem sound as though the issue lies only in the information fed into a model. In many high-stakes systems, however, the deeper issue is that training labels are records of human choices.
When hiring decisions, loan approvals, promotion outcomes, or resource allocations become labels, AI systems may learn to treat those past choices as the correct answers. The model is rewarded for reproducing historical patterns, even when those patterns emerged from unequal institutions or opportunities. As a result, unfair treatment can be transformed from a historical problem into a predictive target.
Understanding this mechanism is essential for understanding AI fairness. A model that copies the past may be performing exactly as designed. The harder question is whether the labels defining success deserve to be copied in the first place. [arXiv+2PMC]arxiv.orgarXiv Identifying and Correcting Label Bias in Machine LearningIdentifying and Correcting Label Bias in Machine LearningJanuary 15, 2019…
Amazon book picks
Further Reading
Books and field guides related to When old decisions become AI training data. Use these as the next step if you want deeper reading beyond the article.
Weapons of Math Destruction
Demonstrates how historical decisions encoded in data can perpetuate unfairness.
Algorithms of Oppression
Shows how historical and social biases become embedded in algorithmic systems.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: arxiv.org
Title: arXiv Identifying and Correcting Label Bias in Machine Learning
Link: https://arxiv.org/abs/1901.04966Source snippet
Identifying and Correcting Label Bias in Machine LearningJanuary 15, 2019...
Published: January 15, 2019
-
Source: arxiv.org
Title: arXiv Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing
Link: https://arxiv.org/abs/2111.08723 -
Source: arxiv.org
Title: arXiv Removing biased data to improve fairness and accuracy
Link: https://arxiv.org/abs/2102.03054 -
Source: fra.europa.eu
Title: fra 2022 bias in algorithms en
Link: https://fra.europa.eu/sites/default/files/fra_uploads/fra-2022-bias-in-algorithms_en.pdfSource snippet
Bias in algorithms – Artificial intelligence and discrimination7 Dec 2022 — Algorithms used for predictive policing and for offensive...
-
Source: reuters.com
Link: https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK0AG/Source snippet
Insight - Amazon scraps secret AI recruiting tool that...Oct 9, 2018 — SAN FRANCISCO (Reuters) - Amazon.com Inc's machine-learnin...
-
Source: reddit.com
Link: https://www.reddit.com/r/womenintech/comments/1gh7nvg/insight_amazon_scraps_secret_ai_recruiting_tool/Source snippet
in resumes submitted to the company over a 10-year period...
-
Source: reddit.com
Link: https://www.reddit.com/r/worldnews/comments/9mzonc/amazon_built_an_ai_tool_to_hire_people_but_had_to/Source snippet
s discriminating against women.Read more...
-
Source: reddit.com
Title: [N] Algorithm used to identify patients for extra care is
Link: https://www.reddit.com/r/MachineLearning/comments/dmyibw/n_algorithm_used_to_identify_patients_for_extra/Source snippet
Bias occurs because the algorithm uses health costs as a proxy for health needs. [...] Reformulating the algorithm so that it no longer u...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8830968/Source snippet
AI bias: exploring discriminatory algorithmic decision-making...by L Belenguer · 2022 · Cited by 359 — The data used in training is i...
-
Source: science.org
Link: https://www.science.org/doi/10.1126/science.aax2342Source snippet
Less money is spent on Black patients who have the same...Read more...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/31649194/Source snippet
racial bias in an algorithm used to manage the...by Z Obermeyer · 2019 · Cited by 9003 — Thus, despite health care cost appearing to be...
-
Source: chicagobooth.edu
Link: https://www.chicagobooth.edu/research/tolan/research/2019/dissecting-racial-bias-in-an-algorithm-used-to-manage-the-health-of-populationsSource snippet
Booth School of BusinessDissecting racial bias in an algorithm used to manage the...We show that a widely used algorithm, typical of thi...
-
Source: chicagobooth.edu
Title: how racial bias infected major health care algorithm
Link: https://www.chicagobooth.edu/review/how-racial-bias-infected-major-health-care-algorithmSource snippet
Booth School of BusinessHow Racial Bias Infected a Major Health-Care Algorithm25 Oct 2019 — The algorithm uses health-care costs as a pro...
-
Source: dictionary.cambridge.org
Link: https://dictionary.cambridge.org/dictionary/english/algorithmicSource snippet
English meaning - Cambridge Dictionaryconnected with or using algorithms that control what someone is shown on a computer application s...
-
Source: chicagobooth.edu
Title: racial bias in health care algorithm
Link: https://www.chicagobooth.edu/media-relations-and-communications/press-releases/racial-bias-in-health-care-algorithmSource snippet
Widely used health care prediction algorithm biased...Oct 25, 2025 — “The algorithms encode racial bias by using health care costs to de...
Additional References
-
Source: medium.com
Link: https://medium.com/%40Andrew_D./amazons-ai-recruiting-tool-that-showed-bias-against-women-d5cffc38c2c4Source snippet
Amazon's AI recruiting tool that showed bias against womenAccording to a Reuters article, “Amazon scraps secret AI recruiting tool that s...
-
Source: incidentdatabase.ai
Link: https://incidentdatabase.ai/cite/37/Source snippet
Incident 37: Amazon's Experimental Hiring Tool Allegedly...Amazon reportedly developed an AI-powered recruiting tool to score job applic...
-
Source: rrapp.spia.princeton.edu
Link: https://rrapp.spia.princeton.edu/an-algorithm-designed-to-predict-health-care-costs-as-a-proxy-for-health-needs-critically-underestimates-the-needs-of-black-patients-with-life-threatening-consequences/Source snippet
risk, automating racism - RRAPP11 Oct 2020 — An algorithm designed to predict health care costs as a proxy for health needs critically un...
-
Source: cl.uni-heidelberg.de
Link: https://www.cl.uni-heidelberg.de/courses/ws19/bias/material/obermeyer.pdfSource snippet
uni-heidelberg.dePS/HS Bias: Bias TypesObermeyer et al: Dissecting racial bias in an algorithm used to manage the... • costs not a good...
-
Source: privacyinternational.org
Link: https://privacyinternational.org/examples/3085/after-three-years-amazon-stopped-using-ai-based-hiring-tool-discriminated-againstSource snippet
After three years, Amazon stopped using an AI-based...Oct 10, 2018 — The hiring tool learned to discriminate against resumes that includ...
-
Source: mediawell.ssrc.org
Title: amazon scraps secret ai recruiting tool that showed bias against women reuters
Link: https://mediawell.ssrc.org/news-items/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-reuters/Source snippet
Scraps Secret AI Recruiting Tool that Showed...Amazon.com Inc's (AMZN.O) machine-learning specialists uncovered a big problem: their new...
-
Source: taylorfrancis.com
Title: amazon scraps secret ai recruiting tool showed bias women jeffrey dastin
Link: https://www.taylorfrancis.com/chapters/edit/10.1201/9781003278290-44/amazon-scraps-secret-ai-recruiting-tool-showed-bias-women-jeffrey-dastinSource snippet
Amazon Scraps Secret AI Recruiting Tool that Showed Bias...by J Dastin · 2022 · Cited by 4031 — Amazon Scraps Secret AI Recruiting Tool...
-
Source: aft.org
Link: https://www.aft.org/hc/spring2025/bosco_chin_parkerSource snippet
increase the profits of healthcare companies can exacerbate existing biases and increase health...Read more...
-
Source: open.edu
Title: Module 2: Real-Life Examples of Bias | OLCreate Reuters web-page
Link: https://www.open.edu/openlearncreate/mod/book/tool/print/index.php?chapterid=35310&id=227372Source snippet
Link. Dastin, J. (2022). Amazon scraps secret AI recruiting tool that showed bias against women. In Ethics of data and analytics (pp. 296...
-
Source: cut-the-saas.com
Link: https://cut-the-saas.com/ai/case-study-how-amazons-ai-recruiting-tool-learnt-gender-biasSource snippet
mber of female candidates selected for the next stage - All due to a lack of...Read more...
Topic Tree