AI Research Scientist · Attention Mechanisms & Kernels
Attention head specialization and interpretability signals
Real lesson card · Page 1 of 4
Attention head specialization and interpretability signals
Attention head specialization
The empirical finding that individual heads in a trained transformer consistently attend in distinctive, function-like ways rather than uniformly.Example
One head in many transformers reliably attends from a verb to its subject, e.g. from ran to dog in ‘The dog ran fast.’How to probe a head
- 1Extract weightsRun many inputs through the model and pull each head’s attention matrix.
- 2Find recurring targetsCheck if the head consistently attends from one token type to another, like verb to subject.
- 3Confirm across layersRepeat across layers and inputs; a stable pattern signals specialization.
Setup
A head attends from each verb to its subject across 50 sentences.- 1Observe the repeated pattern.WhyMany examples rule out coincidence.
- 2Call it correlational evidence.WhyIt shows association, not causation.
Takeaway
This supports a descriptive claim about attention, not a causal claim about computation.Myth
If a head’s attention weight from a verb to its subject is high, that head is causally driving the model’s subject-verb agreement computation.Reality
High attention weight only shows correlation; causal claims need interventions like activation patching or circuit analysis, which lie beyond this pattern-based evidence.Recall check from the same lesson
If a head reliably attends from adjectives to the nouns they modify across many inputs, this alone proves the head is the mechanism the model uses to combine adjective meaning into the noun's representation.
Review the explanation
Answer: False. The recurring pattern only shows a correlational regularity in attention weights; confirming it is the actual causal mechanism would require additional techniques such as activation patching, which go beyond attention-pattern analysis.
Sources
One sitting · 20–30 minutes
A focused session on your AI Research Scientist interview
LearnBench starts from what you already know — skip what you have, master what you’re missing.
Start now