AI Research Scientist · Attention Mechanisms & Kernels

Self-attention mechanism (QKV)

Real lesson card · Page 1 of 4

Query, Key, and Value vectors

For each token, three separate learned weight matrices

W_Q

W_K

W_V

project its embedding into a query vector

q

, a key vector

k

, and a value vector

v

— three distinct roles computed from the same input.

Example

A token’s embedding

x

times

W_Q

gives

q=W_Q x

, different from

k=W_K x

using that same

x

: one token, three separate vectors.

Computing Raw Attention Scores

1
Take the queryStart with the query vector $q_i$ for the position asking what it should attend to.
2
Dot against every keyCompute $q_i \cdot k_j$ for every key vector $k_j$ in the sequence, including $j=i$ itself.
3
Stack into a score vectorCollect the results into $s_i=[q_i\cdot k_1,\dots,q_i\cdot k_n]$ , one raw score per key position.
4
Read scores as relevanceLarger dot products mean higher raw relevance between that key and the query, before any scaling or normalization is applied.

Myth

Dividing the scores by

\sqrt{d_k}

before softmax is just a cosmetic tweak that barely matters.

Reality

d_k

grows, dot-product variance grows too, spreading raw scores wider and pushing softmax toward a near one-hot output with near-zero gradients. Dividing by

\sqrt{d_k}

keeps score variance stable across dimensions, so softmax stays in its sensitive range and gradients keep flowing during training.

Setup

Query

q=[1,0]

, keys

k_1=[2,0]

k_2=[0,2]

d_k=2

. Find the attention weights.

1
Dot product: $q\cdot k_1=2$ , $q\cdot k_2=0$ .
WhyMeasures raw alignment between the query and each key vector.
2
Scale by $\sqrt{2}\approx1.41$ , giving scores $\approx1.41$ and $0$ .
WhyKeeps score variance stable so softmax doesn’t saturate as dimension grows.
3
Softmax: $e^{1.41}\approx4.09$ , $e^{0}=1$ , weights $\approx0.80$ and $\approx0.20$ .
WhyConverts the scaled scores into a normalized probability distribution usable as attention weights.

Takeaway

Scaling before softmax keeps attention weights smoothly spread instead of collapsing onto one key, preserving gradient signal as dimensionality changes.

Recall check from the same lesson

If the sqrt(d_k) scaling step is skipped, larger key dimensions make raw dot-product scores spread out more, pushing softmax toward a near one-hot output and shrinking the gradients that flow back through the attention weights.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

Attention Is All You Need (Vaswani et al., 2017)

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Attention Mechanisms & Kernels questions

All Attention Mechanisms & Kernels Attention head specialization and interpretability signals Attention mechanism computational complexity Multi-head attention design