AI Research Scientist · Attention Mechanisms & Kernels

Multi-head attention design

Real lesson card · Page 1 of 4

Multi-head attention

Multi-head attention splits the input into

h

heads, each with its own learned query, key, and value projection matrices, so every head can compute attention independently before the results are combined.

Example

With

h=8

heads and

d_{\text{model}}=512

, each head works in a 64-dimensional subspace with its own

W_Q

W_K

W_V

matrices, learned separately from every other head.

Split, Attend, Concatenate, Project

1
SplitThe input is projected $h$ times with separate $W_Q, W_K, W_V$ matrices, giving each head its own slice of the same tokens.
2
AttendEach head runs scaled dot-product attention independently on its own Q, K, V, yielding one output vector per token.
3
ConcatenateAll $h$ heads’ per-token outputs are concatenated into a single long vector.
4
ProjectAn output matrix $W_O$ projects that vector back to $d_{\text{model}}$ , merging the heads into one representation.

Setup

A 2-head layer processes a 6-token sentence. Head A’s projections favor nearby tokens; Head B’s favor the first token, regardless of query position.

1
Head A’s $QK^T$ scores peak 1-2 positions away.
WhyIts own $W_Q, W_K$ bias dot products toward small gaps.
2
Head B’s $QK^T$ scores peak on token 1 for every query.
WhyDifferent $W_Q, W_K$ let it learn a long-range pattern instead.
3
Both heads’ outputs are concatenated, then projected by $W_O$ .
WhyThis merges the local and long-range views into one vector.

Takeaway

Independent QKV parameters let heads specialize instead of duplicating one pattern.

Myth

Multi-head attention just runs the same attention computation h times in parallel purely to speed up training, with every head learning the identical pattern.

Reality

Each head owns its own

W_Q, W_K, W_V

, so heads specialize into different relational patterns; concatenating and projecting merges these distinct views into one representation, not a redundant copy.

Recall check from the same lesson

In a multi-head attention layer with d_model = 512 split into 8 heads, each producing a 64-dimensional output, the per-head outputs are concatenated into a 512-dimensional vector before being multiplied by the output projection matrix $W_O$.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

Attention Is All You Need (Vaswani et al., 2017)

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Attention Mechanisms & Kernels questions

All Attention Mechanisms & Kernels Attention head specialization and interpretability signals Attention mechanism computational complexity Self-attention mechanism (QKV)