Within Responsible AI
When spending is mistaken for medical need
A practical shortcut in health-risk scoring can turn unequal access to care into automated underestimation of patient need.
On this page
- How the proxy choice created racial bias
- What the correction changed for care allocation
- How to test proxy targets before deployment
Page outline Jump by section
Introduction
A common lesson in artificial intelligence is that a model can be mathematically accurate while still making unfair decisions. One of the clearest examples comes from healthcare risk scoring, where some systems used healthcare spending as a shortcut for estimating how much medical care a patient needed. On the surface, this seemed sensible: patients with greater health needs often generate higher medical costs. In practice, however, spending reflected not only illness but also unequal access to treatment. As a result, the algorithm learned patterns of healthcare consumption rather than patterns of medical need. The outcome was a systematic underestimation of the needs of many Black patients, despite similar or greater levels of illness. This case has become a landmark example of how a poorly chosen proxy target can embed social inequalities into AI-driven decisions. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
How the proxy choice created racial bias
The mechanism behind the bias was surprisingly simple. Healthcare organisations wanted to identify patients who would benefit from additional support programmes, such as intensive care management and preventive interventions. Instead of directly predicting illness burden, future complications, or unmet medical needs, a widely used commercial algorithm was trained to predict future healthcare costs. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
From a machine-learning perspective, this choice looked attractive. Costs were easy to measure, available in administrative records, and strongly correlated with many health outcomes. The problem was that healthcare spending is not determined solely by how sick a person is. Spending is also shaped by access to healthcare, insurance coverage, provider behaviour, geography, historical inequalities, and patterns of treatment. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
Because Black patients in the United States have historically received less healthcare spending than White patients with comparable levels of illness, the algorithm learned a distorted relationship. When it saw two patients with similar medical conditions, it often assigned lower risk to the Black patient because historical spending records suggested lower future costs. The model was accurately predicting expenditures while inaccurately representing health needs. [DOI+2PubMed]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
This illustrates an important AI principle: bias can emerge even when sensitive attributes are not explicitly used. The system did not need to contain a rule saying that one racial group should receive less support. Instead, the bias arose through the target variable itself. By treating spending as a stand-in for need, the model inherited inequalities already present in the healthcare system. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
Why the model appeared successful
One reason this problem remained hidden is that the algorithm performed well according to conventional evaluation metrics. If the goal was predicting future costs, it was highly effective. The difficulty was that stakeholders actually cared about a different outcome: identifying patients with substantial health needs.
This distinction matters because machine-learning systems optimise whatever target they are given. If developers ask a model to predict costs, it will find patterns that predict costs. It will not automatically infer that decision-makers really wanted a measure of illness, unmet need, or future health deterioration. The case therefore became a textbook example of a “problem formulation” error, where the chosen target differs from the real-world objective. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
What the correction changed for care allocation
Researchers examining the algorithm found that Black patients assigned the same risk score as White patients were generally sicker, showing signs of more severe and less controlled illness. The model was therefore systematically under-ranking many patients who could have benefited from additional support programmes. [DOI+2PubMed]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
The impact on care allocation was substantial. The researchers estimated that removing the bias would increase the proportion of Black patients identified for extra care from 17.7% to 46.5%. In practical terms, many patients who had previously been overlooked would become eligible for programmes designed to reduce complications and improve long-term health outcomes. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
The correction did not require abandoning predictive analytics. Instead, it involved changing what the model predicted. When researchers reformulated the system to focus on indicators more closely connected to health status rather than expenditure, the racial disparity was dramatically reduced while maintaining useful predictive performance. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
This outcome is significant for responsible AI because it shows that fairness improvements do not always require complex mathematical adjustments after deployment. Sometimes the most effective intervention is earlier in the design process: selecting a target variable that genuinely reflects the decision being made. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
How to test proxy targets before deployment
The healthcare spending case offers a practical framework for evaluating AI systems before they affect real people.
Ask whether the target measures the thing that matters
Developers should identify the real-world objective and then examine whether the training target genuinely captures it. If the goal is identifying patients with serious health needs, expenditure may be only an indirect and potentially misleading measure. Similar problems can occur whenever organisations optimise for what is easy to count rather than what they actually value. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
Compare proxy outcomes with direct indicators
A useful safeguard is to compare the chosen proxy with independent measures of the underlying concept. In the healthcare case, researchers looked at indicators of illness and disease burden rather than relying solely on spending records. That comparison revealed that patients with identical scores often had very different levels of actual sickness. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
Examine performance across affected groups
Aggregate accuracy can conceal important disparities. Models should be evaluated separately for different populations to determine whether the proxy behaves differently across groups. The healthcare algorithm appeared effective overall, yet produced systematically different consequences for Black and White patients because the proxy variable itself reflected unequal treatment patterns. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
Investigate the social processes behind the data
Responsible AI requires understanding how data are generated. Healthcare spending was not merely a neutral measurement; it was the outcome of a healthcare system with documented disparities in access and treatment. Any proxy influenced by social, institutional, or economic inequalities deserves additional scrutiny before it becomes the target of an automated decision system. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
What this mechanism teaches about responsible AI
The healthcare spending example remains influential because it demonstrates a subtle but powerful source of algorithmic bias. The problem was not primarily a coding error, a lack of accuracy, or the explicit use of race. It was the decision to substitute a convenient measurement—healthcare costs—for the concept that decision-makers actually cared about: medical need.
For organisations deploying AI in healthcare, insurance, or public health programmes, the central lesson is that proxy targets deserve the same scrutiny as algorithms themselves. A model can faithfully optimise the wrong objective and still produce harmful outcomes. Responsible AI therefore begins not with the choice of model, but with the question of what exactly the system has been asked to predict. [DOI]doi.orgDissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019…
Amazon book picks
Further Reading
Books and field guides related to When spending is mistaken for medical need. Use these as the next step if you want deeper reading beyond the article.
The AI Revolution in Medicine
Provides healthcare-specific context for AI deployment decisions.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: doi.org
Link: https://doi.org/10.1126/science.aax2342Source snippet
Dissecting racial bias in an algorithm used to manage the health of populations | ScienceOctober 25, 2019...
Published: October 25, 2019
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/31649194/Source snippet
Dissecting racial bias in an algorithm used to manage the health of populations - PubMed...
Additional References
-
Source: ischool.berkeley.edu
Link: https://www.ischool.berkeley.edu/events/2019/dissecting-racial-bias-algorithm-guides-health-decisions-millionsSource snippet
UC Berkeley School of InformationDissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions | UC Berkeley School of...
-
Source: youtube.com
Title: AI for the Fair-Minded: Bias in AI in Under 9 Minutes
Link: https://www.youtube.com/watch?v=dfYwjBpxg3cSource snippet
DEI Special Seminar: Algorithmic bias and data platforms (Ziad Obermeyer)...
-
Source: youtube.com
Title: Keynote Presentation: Dissecting Algorithmic Bias
Link: https://www.youtube.com/watch?v=JfKYO1W4uuASource snippet
Dissecting Algorithmic Bias | Ziad Obermeyer | AI FOR GOOD DISCOVERY...
-
Source: onwork.edu.au
Title: Dissecting racial bias in an algorithm used to manage the health of populations
Link: https://onwork.edu.au//bibitem/2019-Obermeyer%2CZiad-Powers%2CBrian-etal-Dissecting%2Bracial%2Bbias%2Bin%2Ban%2Balgorithm%2Bused%2Bto%2Bmanage%2Bthe%2Bhealth%2Bof%2Bpopulations/ -
Source: youtube.com
Title: Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions
Link: https://www.youtube.com/watch?v=y6eo0FZIqjk -
Source: youtube.com
Title: Dissecting Algorithmic Bias | Ziad Obermeyer | AI FOR GOOD DISCOVERY
Link: https://www.youtube.com/watch?v=U5MlyFsMi-ESource snippet
London Speaker Bureau...
-
Source: youtube.com
Title: DEI Special Seminar: Algorithmic bias and data platforms (Ziad Obermeyer)
Link: https://www.youtube.com/watch?v=6xo2tzuOyY4 -
Source: techxplore.com
Title: Tech Xplore How computer algorithms help spread racial
Link: https://techxplore.com/news/2019-10-algorithms-racial-bias-healthcare.pdf
Topic Tree