AI Research Scientist · Transformer Architecture Components

Layer normalization mechanics

Real lesson card · Page 1 of 4

LayerNorm sublayer

LayerNorm is applied around each attention block and each feed-forward block — normalizing either the block’s input before it runs (pre-norm) or the residual sum just after the block’s output rejoins the stream (post-norm).

Example

One transformer layer has two LayerNorms: one wrapping self-attention, one wrapping the feed-forward block.

Computing LayerNorm Per Sample

1
MeanCompute $\mu=\frac{1}{d}\sum_i x_i$ across this one sample’s own features.
2
VarianceCompute $\sigma^2=\frac{1}{d}\sum_i(x_i-\mu)^2$ the same way — per sample, not per batch, since batch statistics are unreliable in transformers.
3
Normalize $\hat{x}_i=\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}}$ gives this sample zero mean, unit variance.
4
Scale & shift $y_i=\gamma\hat{x}_i+\beta$ lets the network relearn a useful scale.

Setup

One feature’s value across 4 stacked blocks grows unchecked: 1, 4, 16, 64.

1
Without LayerNorm, the scale compounds roughly 4× per block.
WhyUnnormalized growth multiplies across depth, risking overflow.
2
With LayerNorm, each block’s output is reset to zero mean, unit variance.
WhyPer-sample rescaling erases the prior growth every single layer.

Takeaway

LayerNorm caps runaway activation growth, which is what stabilizes deep transformer training.

Pre-norm

LayerNorm sits inside the residual branch, before the sublayer: $x+\text{Sublayer}(\text{LN}(x))$ .
The raw input skips into the sum unnormalized.

Post-norm

LayerNorm sits after the residual sum $\text{LN}(x+\text{Sublayer}(x))$ .
The normalization applies once, after summing the branch.

Recall check from the same lesson

In a pre-norm transformer block, LayerNorm is applied to the input before it enters the attention or feed-forward sublayer, and the raw unnormalized input is what gets added into the residual sum.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

LayerNorm sublayer

Computing LayerNorm Per Sample

Sources

A focused session on your AI Research Scientist interview

More Transformer Architecture Components questions