In epoch-wise double descent, holding model size fixed while training for more epochs, test error only ever decreases monotonically as training loss falls, since the interpolation-threshold spike seen in model-wise double descent cannot occur along the training-time axis.

False. Epoch-wise double descent shows the same decrease-increase-decrease shape as model-wise, but along the training-time axis instead of the capacity axis: test error can spike near the epoch where training loss first nears zero, then decrease again with further training. The axis that varies differs between the two variants, but the qualitative shape - and its violation of monotonic classical expectations - is the same.

Double descent phenomenon Interview Q&A

Double descent

Double descent is the pattern where test error first decreases, then increases near the interpolation threshold (the point where a model can just barely fit the training data exactly), then decreases again as capacity or training time keeps growing beyond that point.

Example

As a network’s width grows, test error drops, then spikes right where training error first hits zero, then drops again as width keeps increasing past that point.

Myth

Once a model exceeds its optimal capacity, test error should climb monotonically forever - classical bias-variance intuition says more capacity just means more variance and steadily worse generalization.

Reality

Test error does rise near the interpolation threshold, but growing capacity further past that point brings a second, unexpected decrease - a shape classical theory never predicted.

Model-wise double descent

Axismodel size grows while training length stays fixed.
Peakerror spikes near the size that first fits training data exactly.
Second dipfurther growth lowers test error again.

Epoch-wise double descent

Axistraining time grows for one fixed-size model.
Peakerror spikes near the epoch where training loss nears zero.
Second dipmore training past that epoch lowers test error again.

Why the second dip happens

Effective model complexity, not raw parameter count, drives the shape. Near the interpolation threshold there is usually only one way to fit the data exactly, so that fit is fragile and noise-sensitive, causing the spike. Past that point, many fitting solutions exist, and training tends to settle on smoother ones among them, keeping effective complexity controlled even as parameters keep growing. This explains the shape qualitatively; deriving the exact transition point is a separate question.

Double descent phenomenon

Double descent

Why the second dip happens

Sources

A focused session on your AI Research Scientist interview

More Generalization Theory & Phenomena questions