If you strip the skip connections out of a well-trained 50-layer residual network and retrain the resulting plain stack from scratch, it will typically be just as easy to train as the original residual version, since the individual sublayer transformations are unchanged.

False. Removing the skip connections removes the identity gradient path entirely. The resulting plain 50-layer stack must rely on gradients propagated purely through 50 multiplied sublayer Jacobians, and empirically such deep plain stacks train far worse, or fail to train at all, even though each individual sublayer transformation is unchanged.

Residual connections and deep network trainability Interview Q&A

Residual connection

A residual (skip) connection adds a sublayer’s input directly to its output, so the block computes

y = x + F(x)

instead of just

y = F(x)

. This creates an identity shortcut path running parallel to the transformation

F

Example

In a transformer block, the sublayer might be self-attention; the block’s output is the attention result plus the original input

x

, not the attention result alone.

Backpropagating through $y = x + F(x)$ gives $\frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x}$ . The identity term's derivative is exactly 1, so a full-strength gradient signal passes straight through the shortcut, however small or unstable $\frac{\partial F}{\partial x}$ is.

Stack many blocks and each re-injects that additive 1. Gradients stop being repeatedly multiplied through sublayer Jacobians and shrunk toward zero — they survive the trip back to the earliest layers.

Plain 50-layer stack

Gradients pass only through each sublayer’s Jacobian, layer after layer.
The backward signal shrinks or explodes over 50 multiplications.
Training stalls badly once depth passes roughly 20-30 layers.

Residual 50-layer stack

Each block adds an identity path alongside its Jacobian.
Gradient magnitude is largely preserved through every shortcut.
Training converges smoothly even well past 50 layers.

Myth

Residual connections mathematically guarantee that gradient norms stay perfectly preserved no matter how deep the network gets.

Reality

Residual connections only add an identity term to the backward pass; that’s the rationale, and empirically it makes very deep stacks reliably trainable where plain stacks fail. Proving exact gradient-norm preservation and designing initialization schemes to complement it are separate topics beyond this rationale.

Residual connections and deep network trainability

Residual connection

Sources

A focused session on your AI Research Scientist interview

More Deep Network Training Basics questions