Within Attention
Why does attention need more than one head?
Multiple attention heads let a transformer track several kinds of word relationships at once instead of forcing one attention pattern to do everything.
On this page
- What an attention head learns
- Why parallel heads create richer context
- Limits of interpreting head roles too literally
Page outline Jump by section
Introduction
Multi-head attention exists because a single attention pattern is usually not enough to capture all the relationships that matter in language at the same time. When a transformer reads a sentence, it may need to track grammar, word meaning, pronoun references, sentence structure, and nearby context simultaneously. Multi-head attention solves this by running several attention calculations in parallel, each with its own learned parameters. During training, different heads often specialise in different kinds of relationships because that division of labour helps the model make better predictions. The result is a richer understanding of language than a single attention mechanism can typically provide. [NeurIPS Papers+2arXiv]papers.neurips.cc7181 attention is all you needWith a single…Read more…
What an attention head learns
An attention head is not given a predefined role. Engineers do not label one head as “grammar” and another as “pronoun resolution”. Instead, each head begins with different learned projection matrices and is trained through the same prediction objective as the rest of the model. Over time, the heads discover patterns that improve performance. [NeurIPS Papers+2Sebastian Raschka, PhD]papers.neurips.cc7181 attention is all you needWith a single…Read more…
The key idea is that every head views the same sentence through a different learned representation space. Because each head uses its own query, key, and value projections, the words are transformed differently before attention is calculated. This creates multiple perspectives on the same text. The original Transformer paper described this as attending to information from different representation subspaces at different positions. [NeurIPS Papers]papers.neurips.cc7181 attention is all you needWith a single…Read more…
Research analysing trained language models has found evidence that some heads become associated with recognisable linguistic relationships. In BERT, researchers identified heads that strongly tracked relationships such as:
- Direct objects linked to verbs.
- Determiners linked to nouns.
- Objects linked to prepositions.
- References between pronouns and the entities they refer to. [arXiv+2ACL Anthology]arxiv.orgarXiv What Does BERT Look At? An Analysis of BERT's AttentionWhat Does BERT Look At? An Analysis of BERT's AttentionJune 11, 2019…
These findings suggest that at least some heads develop specialised behaviours because those behaviours are useful for predicting language accurately.
Why parallel heads create richer context
The advantage of multiple heads becomes clearer when considering how many relationships can exist in a single sentence.
Take the sentence:
“The scientist thanked the assistant because she solved the problem.”
To understand the word “she”, the model may need to identify a likely antecedent. At the same time, it may need to understand the grammatical structure of the sentence, recognise the relationship between “solved” and “problem”, and track nearby words that influence meaning.
If a model relied on a single attention pattern, all these relationships would compete for the same set of attention weights. Multi-head attention allows several patterns to coexist. One head can focus on local grammatical links while another tracks longer-distance references and another captures broader sentence-level context. Their outputs are then combined into a single representation. [NeurIPS Papers+2Sebastian Raschka, PhD]papers.neurips.cc7181 attention is all you needWith a single…Read more…
This arrangement also helps reduce a problem sometimes described as representational interference. Different linguistic signals do not all have to be encoded through the same attention map. By distributing work across heads, the model gains additional capacity to represent complex language structures. Recent theoretical work argues that splitting attention into multiple heads can increase the model’s effective representational capacity by reducing interference between competing patterns. [arXiv]arxiv.orgA Capacity-Based Rationale for Multi-Head Attentionby M Adler · 2025 — This analysis yields a new, capacity-based rationale for mult…
An intuitive way to think about this is to imagine several specialists examining the same document. One notices grammatical structure, another follows references between people, another tracks topic changes, and another looks for important nearby context. The final understanding combines all of their observations rather than relying on a single viewpoint.
Why different heads naturally diverge
The reason heads often learn different relationships is not that diversity is explicitly programmed into them. It emerges from the optimisation process.
Each head starts with different parameters. During training, gradient descent rewards parameter configurations that improve prediction accuracy. If two heads perform exactly the same function, one of them may contribute little additional value. The training process therefore often pushes heads towards complementary behaviours because complementary information improves the model more than perfect duplication. [NeurIPS Papers]papers.neurips.cc7181 attention is all you needWith a single…Read more…
Researchers have gone further and experimented with techniques that explicitly encourage heads to become more different from one another. Studies introducing disagreement regularisation found that encouraging diversity among heads can improve translation performance, supporting the idea that varied attention patterns are useful rather than accidental. [arXiv]arxiv.orgarXiv Multi-Head Attention with Disagreement RegularizationMulti-Head Attention with Disagreement RegularizationOctober 24, 2018…
Evidence from machine translation provides another example. Researchers observed that different attention heads in transformer translation systems could align with different translation possibilities, suggesting that separate heads can capture distinct aspects of linguistic correspondence between languages. [arXiv]arxiv.orgarXiv Generating Diverse Translation by Manipulating Multi-Head AttentionGenerating Diverse Translation by Manipulating Multi-Head AttentionNovember 21, 2019…
Limits of interpreting head roles too literally
Although attention-head specialisation is real, it is easy to overstate how neatly heads divide language into separate tasks.
Studies of BERT found some highly interpretable heads, but many heads exhibited broader or less obvious behaviour. Some focused heavily on punctuation, special tokens, or fixed positional patterns rather than clear linguistic concepts. Heads within the same layer can also display similar behaviour. [arXiv+2ACL Anthology]arxiv.orgarXiv What Does BERT Look At? An Analysis of BERT's AttentionWhat Does BERT Look At? An Analysis of BERT's AttentionJune 11, 2019…
Researchers have also shown that many attention heads can sometimes be removed with surprisingly small performance losses. This suggests that transformers often contain redundancy, and that language understanding is distributed across many components rather than residing entirely within a few specialised heads. [ResearchGate]researchgate.netResearch Gate Multi-Head Attention: Collaborate Instead of ConcatenateMotivated by the observation that trained attention heads share common key/query…Read more…
As a result, it is safest to think of attention heads as contributors to a larger system rather than isolated linguistic modules. A head may strongly correlate with a particular relationship, but meaning in a transformer emerges from interactions among many heads, layers, and neural computations working together. [arXiv+2ACL Anthology]arxiv.orgarXiv What Does BERT Look At? An Analysis of BERT's AttentionWhat Does BERT Look At? An Analysis of BERT's AttentionJune 11, 2019…
Why more than one head matters
Multi-head attention changed transformer models because it allowed several language relationships to be tracked at once. Different heads can learn different representation spaces, attend to different positions, and contribute complementary information to the final token representation. Evidence from language-model analysis shows that some heads become associated with syntax, reference tracking, positional patterns, and other useful linguistic structures. At the same time, these roles are neither perfectly separated nor always easy to interpret. The strength of multi-head attention comes less from any single head and more from the collective ability of many heads to build a richer picture of language than a single attention pattern could provide. [Sebastian Raschka, PhD+4NeurIPS Papers+4arXiv]papers.neurips.cc7181 attention is all you needWith a single…Read more…
Amazon book picks
Further Reading
Books and field guides related to Why does attention need more than one head?. Use these as the next step if you want deeper reading beyond the article.
Natural Language Processing with Transformers
Explains attention, multi-head attention, transformers, and representation learning in an accessible way.
Hands-On Large Language Models
Covers transformer internals, attention mechanisms, and modern language models.
Deep Learning
Rating: 3.5/5 from 6 Google Books ratings
Provides foundational concepts behind representation learning and neural architectures.
Transformers for Natural Language Processing
Focuses on transformer architecture, attention heads, and NLP applications.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: papers.neurips.cc
Title: 7181 attention is all you need
Link: https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdfSource snippet
With a single...Read more...
-
Source: arxiv.org
Link: https://arxiv.org/html/1706.03762v7Source snippet
Attention Is All You NeedMulti-head attention allows the model to jointly attend to information from different representation subspaces a...
-
Source: arxiv.org
Title: arXiv What Does BERT Look At? An Analysis of BERT’s Attention
Link: https://arxiv.org/abs/1906.04341Source snippet
What Does BERT Look At? An Analysis of BERT's AttentionJune 11, 2019...
Published: June 11, 2019
-
Source: arxiv.org
Link: https://arxiv.org/pdf/2509.22840Source snippet
A Capacity-Based Rationale for Multi-Head Attentionby M Adler · 2025 — This analysis yields a new, capacity-based rationale for mult...
-
Source: arxiv.org
Title: arXiv Multi-Head Attention with Disagreement Regularization
Link: https://arxiv.org/abs/1810.10183Source snippet
Multi-Head Attention with Disagreement RegularizationOctober 24, 2018...
Published: October 24, 2018
-
Source: arxiv.org
Title: arXiv Generating Diverse Translation by Manipulating Multi-Head Attention
Link: https://arxiv.org/abs/1911.09333Source snippet
Generating Diverse Translation by Manipulating Multi-Head AttentionNovember 21, 2019...
Published: November 21, 2019
-
Source: researchgate.net
Title: Research Gate Multi-Head Attention: Collaborate Instead of Concatenate
Link: https://www.researchgate.net/publication/342587809_Multi-Head_Attention_Collaborate_Instead_of_ConcatenateSource snippet
Motivated by the observation that trained attention heads share common key/query...Read more...
-
Source: arxiv.org
Link: https://arxiv.org/abs/1706.03762Source snippet
[1706.03762] Attention Is All You Need12 Jun 2017 — We propose a new simple network architecture, the Transformer, based solely on attent...
-
Source: arxiv.org
Link: https://arxiv.org/pdf/1906.04341Source snippet
1906.04341v1 [cs.CL] 11 Jun 2019by K Clark · 2019 · Cited by 2741 — Having shown BERT attention heads reflect cer- tain aspects of...
-
Source: attention.com
Title: Our AI agents that learn from that data and automates busywork
Link: https://www.attention.com/Source snippet
AI agents that learn from your best sales...Attention records your sales touchpoints across meetings, emails, calls, CRMs an...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/335778955_What_Does_BERT_Look_at_An_Analysis_of_BERT%27s_AttentionSource snippet
An Analysis of BERT's AttentionAttention heads in different layers attend to different things (Clark et al. 2019). For example, some att...
-
Source: sebastianraschka.com
Link: https://sebastianraschka.com/faq/docs/multi-head-attention.htmlSource snippet
Sebastian Raschka, PhDWhy do transformer-based LLMs use multi-head attention...In short, transformers use multi-head attention instead o...
-
Source: aclanthology.org
Link: https://aclanthology.org/W19-4828/Source snippet
An Analysis of BERT's Attentionby K Clark · 2019 · Cited by 2666 — BERT's attention heads exhibit patterns such as attending to delimiter...
-
Source: aclanthology.org
Link: https://aclanthology.org/W19-4828.pdfSource snippet
An Analysis of BERT's Attentionby K Clark · 2019 · Cited by 2741 — We use the “base” sized BERT model, which has 12 layers containing 12...
Additional References
-
Source: merriam-webster.com
Link: https://www.merriam-webster.com/dictionary/attentionSource snippet
ATTENTION Definition & Meaning5 days ago — 1. a: the act or state of applying the mind to something Our attention was on the game. You s...
-
Source: medium.com
Link: https://medium.com/dair-ai/aspects-of-language-captured-by-bert-32bc3c54016fSource snippet
Aspects of language captured by BERT | by elvis | DAIR.AIThe proposed analysis method probes the attention heads at a broader level (anal...
-
Source: apxml.com
Link: https://apxml.com/courses/introduction-to-transformer-models/chapter-2-[self-attentionSource snippet
Benefits of Multiple Attention HeadsMulti-Head Attention provides a mechanism for the model to look at the input sequence from multiple v...
-
Source: codesignal.com
Link: https://codesignal.com/learn/courses/deconstructing-the-transformer-architecture/lessons/multi-head-attention-mechanismSource snippet
Multi-Head Attention Mechanism | CodeSignal LearnYour implementation demonstrates how parallel attention heads can attend to different re...
-
Source: medium.com
Link: https://medium.com/%40kavierim/transformers-from-scratch-part-3-multi-head-attention-d1a3a061ba89Source snippet
Transformers From Scratch: Part 3 — Multi-Head AttentionImplements Multi-Head Attention, allowing the model to focus on different represe...
-
Source: merriam-webster.com
Link: https://www.merriam-webster.com/dictionary/whatSource snippet
WHAT Definition & MeaningThe meaning of WHAT is —used as an interrogative expressing inquiry about the identity, nature, or value of an o...
-
Source: github.com
Link: https://github.com/clarkkev/attention-analysisSource snippet
clarkkev/attention-analysisThis repository contains code for What Does BERT Look At? An Analysis of BERT's Attention. It includes code fo...
-
Source: peratham.medium.com
Link: https://peratham.medium.com/a-paper-a-day-2-what-does-bert-look-at-an-analysis-of-berts-attention-2f24d855302Source snippet
An Analysis of...BERT's attention heads might have different patterns such as attending to delimiter tokens, specific positional offsets...
-
Source: datascience.stackexchange.com
Title: in transformers multi headed attention how attending different representation
Link: https://datascience.stackexchange.com/questions/94886/in-transformers-multi-headed-attention-how-attending-different-representationSource snippet
Transformer's multi-headed attention, how attending "...25 May 2021 — The multi-headed model can capture richer interpretations because...
Published: May 2021
-
Source: aryanupadhyay.com
Title: multi head attention in transformers explained concepts math mechanics
Link: https://www.aryanupadhyay.com/post/multi-head-attention-in-transformers-explained-concepts-math-mechanicsSource snippet
Multi-Head Attention in Transformers Explained2 Mar 2026 — Learn how multi-head attention works in Transformers, why single self-attentio...
Topic Tree