AI Research Scientist · Deep Network Training Basics

Weight initialization strategies

Real lesson card · Page 1 of 4

Variance-preserving initialization

A choice of initial weight scale such that the variance of activations (forward pass) and gradients (backward pass) stays roughly constant across layers at the start of training, instead of shrinking toward zero or exploding.

Example

In a 50-layer network, weights drawn too large make activations grow layer over layer until they saturate; weights drawn too small make activations shrink toward zero before reaching the output.

Xavier/Glorot — for tanh/sigmoid

Assumes a roughly linear region near zero, so variance scales as $1/\text{fan\_in}$ (or an average of fan-in and fan-out).
Keeps signal balanced through symmetric, zero-centered nonlinearities.

Kaiming/He — for ReLU-family

Accounts for ReLU zeroing out about half the pre-activations, so variance scales as $2/\text{fan\_in}$ to compensate for that lost signal.
Matches the asymmetric, non-zero-centered shape of ReLU and its variants.

Setup

A transformer has 24 stacked residual blocks. Each block’s output is added directly onto the residual stream, so by block 24 the stream has summed 24 independent contributions.

1
Initialize each block’s output-projection weights with the standard Kaiming/He scale (variance proportional to $2/\text{fan\_in}$ ).
WhyThis keeps a single block’s own output at the right scale in isolation.
2
Shrink that per-block variance further by a depth-dependent factor, e.g. scale weights by $1/\sqrt{2L}$ for $L$ blocks.
WhyWithout shrinking, the residual stream’s variance would grow roughly with the number of added blocks, since each addition contributes independent variance.
3
Check that the residual stream’s variance after all 24 additions stays close to its value at block 1.
WhyThis confirms the scaled-down initialization offsets the accumulation from residual depth.

Takeaway

Deeper residual networks need extra shrinkage in the initial weight scale beyond the activation-based rule alone, because every residual addition stacks more variance onto the stream.

Setup

A learner initializing a network with a newer activation function, like GELU, wants the exact variance formula to use, the way Xavier’s derivation follows from tanh’s slope near zero.

Expected

They expect one precise formula can be derived from the activation’s exact mathematical shape, just as with the classic derivations.

What breaks

For many modern activations no clean closed-form variance formula is derived or used in practice; practitioners instead match the nearest scheme by shape (e.g., Kaiming for GELU, since it behaves like ReLU) and verify empirically.

Lesson

Choosing an initialization scheme is about matching the activation’s general shape and the network’s depth/residual structure, not deriving an exact formula for every function.

Recall check from the same lesson

Since GELU is a smooth, non-ReLU activation, you must derive its exact analytic variance formula before you can pick a working initialization scheme for a network that uses it.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

Variance-preserving initialization

Sources

A focused session on your AI Research Scientist interview

More Deep Network Training Basics questions