In a very long recurrent sequence model, adding LayerNorm at each timestep is enough to fully prevent gradient decay across all time steps, without needing a gating mechanism.

False. LayerNorm rescales the activations and gradients at each individual step, but it doesn't control how much gradient signal is allowed to persist backward across many steps of the same reused weight matrix. Gating mechanisms address exactly that long-range decay by learning, per step, how much gradient to let flow backward — a job normalization alone doesn't do.

Vanishing and exploding gradients Interview Q&A

Vanishing and exploding gradients

During backprop, the gradient reaching an early layer is the product of many per-layer (or per-timestep) local gradient factors. If those factors are consistently below 1, the product shrinks exponentially toward zero — vanishing. If consistently above 1, it grows exponentially — exploding.

Example

In a 50-layer network with each factor averaging 0.9, the gradient reaching layer 1 scales by

0.9^{50}\approx 0.005

: the signal is nearly gone.

Setup

A 4-layer net has each layer’s local factor at 0.5; another 4-layer net has each factor at 2.0. What reaches layer 1 in each case?

1
Multiply the four 0.5 factors: $0.5^4=0.0625$ .
WhyLocal gradients multiply across layers, so shrinking factors compound geometrically with depth.
2
Multiply the four 2.0 factors: $2.0^4=16$ .
WhyGrowing factors compound the same way, but toward large magnitudes instead of small ones.
3
Compare the resulting layer-1 updates: near-zero vs. huge.
WhyNear-zero gradients stall learning (vanishing); huge gradients cause erratic, diverging updates (exploding).

Takeaway

The same multiplicative chain produces opposite training symptoms — stalling or divergence — depending on whether per-step factors sit below or above 1.

Normalization layers (BatchNorm, LayerNorm) rescale each layer's activations — and therefore the gradients flowing through them — to a controlled range at every layer. This resets the multiplicative product's scale before it can compound toward zero or infinity across depth. That's distinct from rerouting gradient flow around layers: normalization keeps the same backward path but keeps its scale stable at each step.

Myth

Adding normalization layers fully solves vanishing gradients, even across long recurrent sequences.

Reality

Normalization rescales activations per layer, but recurrent networks reuse the same weight matrix across many time steps, so gradients can still decay over long ranges even with normalized activations. Gating (as in LSTM cells) instead learns, per time step, how much gradient signal to pass backward — creating paths that bypass full multiplicative decay over long sequences.

Vanishing and exploding gradients

Vanishing and exploding gradients

Sources

A focused session on your AI Research Scientist interview

More Deep Network Training Basics questions