Within Health Proxy
The simple target change that reduced bias
The strongest fix was not abandoning prediction, but asking the model to predict health status more directly instead of spending.
On this page
- Why reformulating the target mattered
- Which health status signals were closer to need
- How teams can test proxy targets before launch
Page outline Jump by section
Introduction
One of the most important lessons in artificial intelligence is that fairness problems often begin before any model is trained. In the healthcare risk-scoring case that exposed large racial disparities in care allocation, the most effective remedy was not a more complex algorithm, additional demographic variables, or a sophisticated fairness adjustment. It was a change in what the model was asked to predict. Researchers found that the system had been trained to predict future healthcare spending, even though decision-makers actually wanted to identify patients with the greatest health needs. By reformulating the prediction target to better reflect illness and medical need, much of the disparity in patient selection was reduced. This case has become a leading example of why target selection is one of the most consequential design decisions in AI systems. [Chicago Booth]chicagobooth.eduChicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth…
Why reformulating the target mattered
The original healthcare algorithm was successful at predicting future costs. The problem was that future costs were not the same thing as future need. Historical spending patterns reflected many factors beyond illness, including access to care, insurance coverage, provider behaviour, and longstanding inequalities in healthcare utilisation. As a result, patients with similar levels of illness could generate very different spending records. [Chicago Booth]chicagobooth.eduChicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth…
From a machine-learning perspective, the model was doing exactly what it had been asked to do. It learned patterns that forecast expenditure. The mismatch arose because healthcare organisations were using those expenditure predictions as a stand-in for a different goal: deciding who needed additional medical support. This is often described as a problem formulation or target-specification error, where the operational target differs from the real-world objective. [Montreal AI Ethics Institute]montrealethics.aiMontreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et…
Researchers showed that when patients received the same risk score, Black patients were typically sicker than White patients. The algorithm had effectively learned the healthcare system’s spending patterns rather than the underlying burden of disease. After the target was changed to reflect health needs more directly, the estimated share of Black patients identified for additional care rose from 17.7% to 46.5%, demonstrating that the choice of target variable was a major source of the bias. [Chicago Booth]chicagobooth.eduChicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth…
This finding shifted the discussion from “How do we fix a biased model?” to “Are we predicting the right thing in the first place?” That change in perspective has influenced responsible-AI practices far beyond healthcare. [Montreal AI Ethics Institute]montrealethics.aiMontreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et…
Which health-status signals were closer to need
The key insight was that direct measures of illness are usually better indicators of care needs than financial measures. Instead of treating spending as the target, researchers examined health-related outcomes that more closely tracked a patient’s disease burden.
Examples of signals that are generally closer to medical need include:
- The number and severity of chronic conditions.
- Clinical indicators showing whether diseases are controlled or uncontrolled.
- Future complications associated with existing illnesses.
- Hospitalisations and serious health events.
- Measures of overall disease burden rather than financial expenditure.
- Composite health-risk scores built from diagnoses and clinical records. [Chicago Booth+2eScholarship]chicagobooth.eduChicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth…
These measures are not perfect. Every health indicator is itself a simplification of a patient’s condition. However, they are conceptually closer to the decision-makers’ true goal: identifying patients who would benefit most from intervention. The closer a target aligns with the intended outcome, the less opportunity there is for social inequalities to enter through an indirect proxy. [Montreal AI Ethics Institute]montrealethics.aiMontreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et…
An important practical lesson emerges here. Developers often prefer targets that are easy to measure because they are readily available in databases. Costs, clicks, purchases, and engagement metrics are attractive because they already exist. Yet convenience can hide important distortions. A target that is easy to collect may be a poor representation of the outcome that organisations actually care about. [Montreal AI Ethics Institute]montrealethics.aiMontreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et…
How teams can test proxy targets before launch
The healthcare case illustrates a broader implementation challenge: organisations frequently use proxy targets because direct measurements of the desired outcome are difficult to obtain. Responsible AI therefore requires testing whether a proxy truly represents the intended objective.
Several practical checks can help:
Ask what decision-makers really want
Before model development begins, teams should explicitly define the real-world outcome that motivates the system. If a hospital wants to identify patients needing support, then the target should reflect health status rather than expenditure. If the target and objective differ, developers should document why. [Montreal AI Ethics Institute]montrealethics.aiMontreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et…
Compare multiple candidate targets
Rather than assuming a single proxy is correct, teams can evaluate several alternatives. For example, healthcare developers might compare models predicting spending, disease burden, hospital admissions, or future complications. Differences in outcomes across groups may reveal hidden weaknesses in a proxy target. [PubMed]pubmed.ncbi.nlm.nih.govPrediction Models for Future High-Need High-Cost Healthcare Use: a Systematic Review - PubMedJanuary 11, 2022…
Examine subgroup performance
A proxy may appear accurate overall while failing for particular populations. Testing whether people with the same predicted score have similar underlying conditions across demographic groups can expose target-related bias before deployment. The healthcare case became visible precisely because researchers examined patients with equivalent risk scores and found substantial differences in illness burden. [Chicago Booth]chicagobooth.eduChicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth…
Test for alignment, not only accuracy
Traditional machine-learning metrics measure agreement between predictions and labels. However, if the labels themselves are imperfect proxies, high accuracy can be misleading. Teams should evaluate whether the target reflects the outcome decision-makers genuinely care about, not merely whether predictions match historical records. [Montreal AI Ethics Institute]montrealethics.aiMontreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et…
The broader AI lesson from the target change
The healthcare example is often remembered as a case of algorithmic bias, but its deeper significance lies in how the bias was reduced. The strongest intervention did not involve abandoning prediction or removing machine learning from the process. Instead, it involved redefining the prediction target so that it corresponded more closely to the underlying objective.
For AI practitioners, this illustrates a foundational principle: a model cannot learn the outcome people care about if it is trained to optimise something else. Fairness, usefulness, and accuracy are all influenced by how a problem is formulated. Choosing the right target is therefore not a minor technical detail. It is one of the most important design decisions in the entire AI development process. [ResearchTrend.AI+3Chicago Booth+3eScholarship]chicagobooth.eduChicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth…
Amazon book picks
Further Reading
Books and field guides related to The simple target change that reduced bias. Use these as the next step if you want deeper reading beyond the article.
Weapons of Math Destruction
Strong illustration of how choosing the wrong target creates unfair outcomes.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: escholarship.org
Link: https://escholarship.org/uc/item/6h92v832 -
Source: researchtrend.ai
Link: https://researchtrend.ai/papers/2308.02081Source snippet
Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare | ResearchTrend.AI...
-
Source: chicagobooth.edu
Link: https://www.chicagobooth.edu/research/tolan/research/2019/dissecting-racial-bias-in-an-algorithm-used-to-manage-the-health-of-populationsSource snippet
Chicago BoothDissecting racial bias in an algorithm used to manage the health of populations - Tolan Center | Chicago Booth...
-
Source: montrealethics.ai
Link: https://montrealethics.ai/target-specification-bias-counterfactual-prediction-and-algorithmic-fairness-in-healthcare/Source snippet
Montreal AI Ethics InstituteTarget specification bias, counterfactual prediction, and algorithmic fairness in healthcare | Montreal AI Et...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/35018571/Source snippet
Prediction Models for Future High-Need High-Cost Healthcare Use: a Systematic Review - PubMedJanuary 11, 2022...
Published: January 11, 2022
-
Source: pmc.ncbi.nlm.nih.gov
Title: Predictive cues reduce but do not eliminate intrinsic response bias
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6732222/Source snippet
June 21, 2019...
Published: June 21, 2019
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/30845187/Source snippet
March 7, 2019...
Published: March 7, 2019
-
Source: Wikipedia
Title: Algorithmic bias
Link: https://en.wikipedia.org/wiki/Algorithmic_bias
Additional References
-
Source: academic.oup.com
Link: https://academic.oup.com/ije/article-abstract/49/6/1763/5814327Source snippet
OUP AcademicIntersections of machine learning and epidemiological methods for health services research | International Journal of Epidemi...
-
Source: youtube.com
Title: [Understanding]({{ ‘understanding/’ | relative_url }}) bias in health AI
Link: https://www.youtube.com/watch?v=zks3pdPN7A4Source snippet
Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions - YouTube Dissecting Racial Bias in an Algorithm that Gu...
-
Source: diagnprognres.biomedcentral.com
Link: https://diagnprognres.biomedcentral.com/articles/10.1186/s41512-022-00119-9Source snippet
ostic Research | Full TextMarch 24, 2022...
Published: March 24, 2022
-
Source: youtube.com
Title: Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions
Link: https://www.youtube.com/watch?v=y6eo0FZIqjkSource snippet
Ziad Obermeyer on tackling 'data bottleneck' to combat AI bias...
-
Source: youtube.com
Title: Ziad Obermeyer on tackling ‘data bottleneck’ to combat AI bias
Link: https://www.youtube.com/watch?v=xWLGch3g3mASource snippet
Understanding Bias and Fairness in AI-enabled Healthcare Software...
-
Source: youtube.com
Title: Understanding Bias and Fairness in AI-enabled Healthcare Software
Link: https://www.youtube.com/watch?v=bcqofACB-SkSource snippet
Understanding bias in health AI - Dr. Stephen Pfohl...
Topic Tree