AI Research Scientist · Transformer Architecture Components

Feed-forward and normalization sublayers

Real lesson card · Page 1 of 3

Feed-forward sublayer (FFN)

The feed-forward sublayer is the second sublayer in a transformer block, sitting right after attention. Unlike attention, which mixes information across positions, the FFN processes each position’s vector entirely on its own.

Example

After attention lets token 5 gather context from tokens 1-4, the FFN then transforms token 5’s vector alone, with no reference to any other token’s vector.

How the FFN transforms one position

1
Project upA linear layer projects $x$ into a wider space: $h = W_1 x + b_1$
2
Apply nonlinearityAn elementwise nonlinearity (ReLU or GELU) is applied to $h$ , adding curvature attention alone can’t produce.
3
Project downA second linear layer projects back to the model’s width: $y = W_2\,\sigma(h) + b_2$
4
Repeat identically per positionThese same weights apply identically and independently at every position — no position influences another here.

Myth

It’s tempting to see the FFN as a minor add-on after the ‘real work’ of attention — extra parameters bolted on for size, not function.

Reality

The FFN gives the model nonlinear representational capacity. Attention only computes weighted averages of value vectors — a linear mixing operation. The FFN’s nonlinearity reshapes each position’s representation in curved, non-linear ways; removing it cuts capacity, not just parameter count.

Recall check from the same lesson

Because the FFN sublayer's weight matrices are shared across positions, two different tokens at two different sequence positions in the same block are transformed using the exact same $W_1$ and $W_2$, even though each token still gets its own distinct output vector.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

Vaswani et al., Attention Is All You Need (arXiv 1706.03762)

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

Feed-forward sublayer (FFN)

How the FFN transforms one position

Sources

A focused session on your AI Research Scientist interview

More Transformer Architecture Components questions