Within Training data
When AI learns the wrong shortcut
Models can learn shortcuts that look useful in training data but collapse when the real-world setting changes.
On this page
- How accidental patterns enter training data
- Why shortcuts can pass ordinary tests
- Examples of failures when settings change
Page outline Jump by section
Introduction
Artificial intelligence systems learn from patterns in data, not from an inherent understanding of the world. This creates a recurring problem: a model may discover a pattern that predicts the correct answer in its training data without learning the underlying reason the answer is correct. Researchers often call these misleading patterns spurious correlations or shortcuts. They can make an AI system look highly accurate during development while causing it to fail when conditions change. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
The danger is not that the model learns nothing useful. Rather, it learns a mixture of meaningful signals and accidental clues. As long as those clues remain present, performance may appear excellent. When the environment changes and the clues disappear, the model’s apparent intelligence can collapse surprisingly quickly. This is one of the main reasons AI systems that perform well in laboratories sometimes struggle in real-world deployment. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
How accidental patterns enter training data
Training datasets are created from real-world observations, and real-world data often contains hidden regularities. Some of those regularities reflect genuine relationships. Others are accidents of how the data was collected.
Imagine a dataset used to identify animals. If photographs of cows are usually taken in grassy fields, a model may discover that green textures are a strong predictor of the “cow” label. The grass is correlated with cows in the dataset, but grass is not what makes an animal a cow. A model that relies heavily on background scenery has learned a shortcut rather than the intended concept. [DeepLearning.ai]deeplearning.aiThe Causes of Shortcut Learning in Neural NetworksJun 3, 2020 — Dataset bias can cause models to focus on spurious correlations rather th…
The same mechanism appears across many domains:
- Medical images may contain hospital-specific markings, scanner characteristics, or demographic patterns that happen to correlate with diagnoses.
- Language models may learn that certain words frequently appear alongside particular labels, even when those words are not the true reason for the label.
- Computer vision systems may rely on image backgrounds, lighting conditions, or camera artefacts instead of the objects they are supposed to recognise. [PMC+2IJCAI]pmc.ncbi.nlm.nih.govPMC“Shortcuts” Causing Bias in Radiology Artificial Intelligenceby I Banerjee · 2023 · Cited by 150 — In this review, the authors discuss…
A key insight from research on shortcut learning is that machine-learning systems are rewarded for finding whatever signal most efficiently reduces prediction errors. They are not automatically rewarded for finding the signal humans consider meaningful. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
Why shortcuts can pass ordinary tests
One reason spurious correlations are difficult to detect is that standard testing often uses data that resembles the training data.
If both training and testing datasets contain the same accidental pattern, a shortcut can continue producing correct answers. The model therefore receives a high score and appears reliable. Developers may conclude that the system has learned the intended concept when it has actually learned an easier proxy. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
Researchers describe this as a generalisation problem. The model has learned a rule that works within a particular data distribution but not outside it. A shortcut can therefore survive multiple rounds of evaluation because the evaluation environment contains the same hidden bias. [PMC]pmc.ncbi.nlm.nih.govPMCBeyond Distribution Shift: Spurious Features Throughby N Murali · 2023 · Cited by 18 — Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label d…
This explains why accuracy numbers alone can be misleading. A model may achieve excellent benchmark performance while still relying on fragile signals. Studies of shortcut learning have shown that monitoring accuracy does not necessarily reveal whether a model has learned robust features or merely convenient correlations. [OpenReview]openreview.netShortcut Learning Through the Lens of Early Training…by N Murali · Cited by 3 — Summary Of The Paper: The paper studies the…
Why new settings expose the problem
The failure occurs when the environment changes.
Machine-learning researchers often refer to this as a distribution shift: the statistical patterns present during deployment differ from those seen during training. When that happens, shortcut-based predictions lose their value. [PMC]pmc.ncbi.nlm.nih.govPMCBeyond Distribution Shift: Spurious Features Throughby N Murali · 2023 · Cited by 18 — Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label d…
Consider the difference between two types of features:
- Core features are directly related to the task and remain useful across many situations.
- Spurious features happen to correlate with the answer in one setting but may disappear in another. [PMC]pmc.ncbi.nlm.nih.govPMCBeyond Distribution Shift: Spurious Features Throughby N Murali · 2023 · Cited by 18 — Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label d…
A model that has learned core features can adapt better to new circumstances because those features remain relevant. A model that depends on spurious features effectively has part of its reasoning removed when conditions change. The result may be sudden performance drops, unexpected mistakes, or behaviour that appears irrational to human observers. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
Researchers studying deep neural networks describe shortcuts as decision rules that work on standard benchmarks but fail under more challenging or realistic conditions. In other words, the model succeeds because the shortcut exists, not because it has learned the deeper concept developers intended. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
Examples of failures when settings change
Medical imaging and hospital-specific clues
Medical AI provides some of the clearest examples. Researchers have repeatedly found that diagnostic models can exploit non-medical signals present in training data.
In studies of chest X-rays, systems have sometimes learned information related to image acquisition, hospital identity, scanner characteristics, or other contextual cues instead of focusing solely on disease indicators. Such shortcuts can produce impressive results on familiar datasets yet perform poorly when applied to patients from different hospitals or imaging environments. [PMC+2PMC]pmc.ncbi.nlm.nih.govPMC“Shortcuts” Causing Bias in Radiology Artificial Intelligenceby I Banerjee · 2023 · Cited by 150 — In this review, the authors discuss…
During the COVID-19 pandemic, researchers showed that some AI systems for detecting COVID-19 from chest radiographs relied on shortcuts unrelated to the disease itself. These findings raised concerns that models appearing highly accurate during development might not generalise safely to new clinical settings. [MedRxiv]medrxiv.org2020.09.13.20193565v2.full textAI for radiographic COVID-19 detection selects shortcuts…by AJ DeGrave · 2020 · Cited by 893 — Using state-of-the-art technique…
Image recognition and background dependence
Vision systems often learn environmental cues because those cues are easier to detect than the object itself.
A model trained on photographs where wolves frequently appear in snowy environments may partly associate snow with the wolf label. If later presented with wolves in different surroundings, the system may become less reliable. Similar effects have been observed across many image-recognition tasks involving backgrounds, textures, and scene context. [DeepLearning.ai]deeplearning.aiThe Causes of Shortcut Learning in Neural NetworksJun 3, 2020 — Dataset bias can cause models to focus on spurious correlations rather th…
Research has also shown that image classifiers can become unusually sensitive to texture patterns. Instead of recognising the overall structure of an object, they may rely on surface characteristics that happen to correlate with the label in the training data. [DeepLearning.ai]deeplearning.aiThe Causes of Shortcut Learning in Neural NetworksJun 3, 2020 — Dataset bias can cause models to focus on spurious correlations rather th…
Language models and misleading word associations
Shortcut learning is not limited to images. In language tasks, models can learn that certain words strongly co-occur with particular labels and then over-rely on those words.
For example, a sentiment-analysis system may learn that specific terms frequently appear in positive reviews and use those terms as shortcuts for prediction. When the wording changes, even though the underlying meaning remains similar, performance can deteriorate because the shortcut no longer works. [arXiv]arxiv.orgOpen source on arxiv.org.
Why shortcuts are so attractive to AI systems
An important lesson from the research literature is that shortcuts are often the easiest patterns to learn.
If a simple feature predicts the correct label most of the time, optimisation algorithms naturally gravitate towards it. The model is rewarded for reducing errors quickly, and the shortcut may provide a fast route to high training accuracy. [OpenReview]openreview.netShortcut Learning Through the Lens of Early Training…by N Murali · Cited by 3 — Summary Of The Paper: The paper studies the…
This means that increasing model size or training time does not automatically solve the problem. More powerful models can sometimes become even better at discovering subtle accidental correlations hidden within large datasets. The issue originates in the relationship between the data and the learning objective, not merely in a lack of computational power. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
What this reveals about AI learning
Spurious correlations highlight a fundamental property of modern AI: models learn statistical relationships, not human intentions.
When a dataset contains accidental clues that consistently predict the correct answer, the model has little reason to ignore them. As a result, impressive performance can coexist with fragile understanding. The system may appear intelligent until it encounters a setting where the shortcut no longer applies. [arXiv]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
Understanding shortcut learning therefore helps explain a central challenge in artificial intelligence. The question is not only whether a model can find patterns, but whether it learns patterns that remain valid when the world changes. Spurious correlations matter because they allow a model to succeed for the wrong reason—and success for the wrong reason is often revealed only when the model faces a new environment. [arXiv+2PMC]arxiv.orgarXiv Shortcut Learning in Deep Neural NetworksShortcut Learning in Deep Neural NetworksApril 16, 2020…
Amazon book picks
Further Reading
Books and field guides related to When AI learns the wrong shortcut. Use these as the next step if you want deeper reading beyond the article.
Hands-on Machine Learning with Scikit-Learn, Keras, and Tenso...
Covers overfitting, generalization, data leakage, and robustness against misleading patterns.
The Alignment Problem
Discusses bias, shortcuts, dataset issues, and failures caused by models learning the wrong signals.
Artificial Intelligence
Uses examples showing how AI systems exploit shortcuts instead of understanding concepts.
Data Science for Business
Explains correlation versus causation and common modeling mistakes in real-world datasets.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: arxiv.org
Title: arXiv Shortcut Learning in Deep Neural Networks
Link: https://arxiv.org/abs/2004.07780Source snippet
Shortcut Learning in Deep Neural NetworksApril 16, 2020...
Published: April 16, 2020
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCBeyond Distribution Shift: Spurious Features Through
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11029547/Source snippet
by N Murali · 2023 · Cited by 18 — Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label d...
-
Source: deeplearning.ai
Link: https://www.deeplearning.ai/the-batch/when-models-take-shortcuts/Source snippet
The Causes of Shortcut Learning in Neural NetworksJun 3, 2020 — Dataset bias can cause models to focus on spurious correlations rather th...
-
Source: ijcai.org
Link: https://www.ijcai.org/proceedings/2023/0560.pdfSource snippet
s often take dataset biases as a shortcut to make [decisions]({{ 'decisions/' | relative_url }}) rather than understand tasks, lead- ing to...Read more...
-
Source: medrxiv.org
Title: 2020.09.13.20193565v2.full text
Link: https://www.medrxiv.org/content/10.1101/2020.09.13.20193565v2.full-textSource snippet
AI for radiographic COVID-19 detection selects shortcuts...by AJ DeGrave · 2020 · Cited by 893 — Using state-of-the-art technique...
-
Source: arxiv.org
Link: https://arxiv.org/html/2412.05152v1Source snippet
Navigating Shortcuts, Spurious Correlations, and...6 Dec 2024 — Shortcuts, also described as Clever Hans behavior, spurious correlations...
-
Source: openreview.net
Link: https://openreview.net/forum?id=5wa-ueGGI33Source snippet
Shortcut Learning Through the Lens of Early Training...by N Murali · Cited by 3 — Summary Of The Paper: The paper studies the...
-
Source: arxiv.org
Link: https://arxiv.org/abs/2205.12593 -
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12344020/Source snippet
and Mitigating the Clever Hans Effect in Medical...by C Vásquez-Venegas · 2024 · Cited by 23 — The Clever Hans effect occurs when machin...
-
Source: arxiv.org
Link: https://arxiv.org/pdf/2004.07780Source snippet
[Deep learning]({{ 'deep-learning/' | relative_url }}) has triggered the current rise of artificial intelligence and is the workhorse of today's machine intelligence.Read more...
-
Source: openreview.net
Link: https://openreview.net/forum?id=3UMzxqDcpYSource snippet
ible to detect shortcut learning and locate where the shortcut is happening in a neural network.Read more...
-
Source: youtube.com
Title: Shortcut Learning in Deep Neural Networks
Link: http://www.youtube.com/watch?v=D-eg7k8YSfsSource snippet
Shortcut Learning - A generalization problem in deep neural networks...
-
Source: youtube.com
Title: Shortcut Learning
Link: http://www.youtube.com/watch?v=rntdFo9y3L4Source snippet
in AI (Deep Learning) - An Overview...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11192466/Source snippet
PMC“Shortcuts” Causing Bias in Radiology Artificial Intelligenceby I Banerjee · 2023 · Cited by 150 — In this review, the authors discuss...
-
Source: dictionary.cambridge.org
Link: https://dictionary.cambridge.org/dictionary/english-chinese-traditional/spuriousSource snippet
in Traditional Chinese - Cambridge Dictionaryfalse and not what it appears to be, or (of reasons and judgments) based on something that h...
Additional References
-
Source: dictionary.com
Link: https://www.dictionary.com/browse/spuriousSource snippet
SPURIOUS Definition & Meaningnot genuine, authentic, or true; not from the claimed, pretended, or proper source; counterfeit. Synonyms: d...
-
Source: en.wiktionary.org
Link: https://en.wiktionary.org/wiki/spuriousSource snippet
spurious (comparative more spurious, superlative most spurious) False, not authentic, not genuine. Extraneous, stray; not relevant or wanted...
-
Source: cds.ismrm.org
Link: https://cds.ismrm.org/protected/23MProceedings/PDFfiles/0154_M9imKVQZg.htmlSource snippet
Nature Machine Intelligence... (B) Proposed: By interrupting the shortcut, learning a spurious-correlation is avoided.Read more...
-
Source: github.com
Link: https://github.com/rgeirhos/shortcut-perspectiveSource snippet
rted in the perspective article "Shortcut Learning in Deep Neural Networks"Read more...
-
Source: youtube.com
Title: Spurious Correlations and the Rise of Causal AI: A Guide for Investors
Link: http://www.youtube.com/watch?v=OOUzdEN_DewSource snippet
This Shortcut Learning in Deep Neural Networks video outlines how artificial intelligence models pick up on accidental text or background...
-
Source: linkedin.com
Link: https://www.linkedin.com/pulse/unveiling-pitfalls-shortcut-learning-deep-neural-networks-tuiamat-icoaeSource snippet
nding factors that are not representative of the true task at...Read more...
-
Source: dl.acm.org
Link: https://dl.acm.org/doi/10.1145/3626772.3657729Source snippet
acm.orgCOMI: COrrect and MItigate Shortcut Learning Behavior in...Jul 11, 2024 — In this paper, we propose COMI, a novel method to COrre...
-
Source: techxplore.com
Title: 2020 12 exploring notion shortcut deep neural
Link: https://techxplore.com/news/2020-12-exploring-notion-shortcut-deep-neural.htmlSource snippet
Exploring the notion of shortcut learning in deep neural...23 Dec 2020 — Geirhos and Michaelis believed that shortcut learning, the phen...
-
Source: reddit.com
Link: https://www.reddit.com/r/MachineLearning/comments/hxt85n/r_understanding_neural_network_shortcut_learning/Source snippet
t discriminating consistent clues and inconsistent clues.Read more...
-
Source: scispace.com
Title: Shortcut learning in deep neural networks
Link: https://scispace.com/papers/shortcut-learning-in-deep-neural-networks-1sq5hzzuwt?citations_page=63Source snippet
Robert GeirhosIn this Perspective we seek to distil how many of deep learning's failures can be seen as different symptoms of the same un...
Topic Tree