Because an SGD-trained model reached low-norm weights without an explicit penalty, we can conclude that SGD's loss function must contain an implicit L2 term the optimizer discovered during training.

False. SGD's bias toward flat minima comes from the noise in its updates, and its bias toward low-norm solutions from the gradient dynamics themselves — neither comes from a penalty term appearing in the loss. Implicit regularization is a dynamics effect, not a hidden objective term — the loss function itself never changes.

Implicit regularization of SGD Interview Q&A

implicit regularization

Implicit regularization is a generalization-improving bias that comes purely from an optimizer’s own dynamics, with no penalty term added to the loss. Explicit regularization, like an L2 term

\lambda \|w\|^2

, instead bakes that bias directly into the objective being minimized.

Example

Train a network with plain SGD and no weight decay anywhere in the code: the trajectory itself still favors flatter, lower-norm solutions, with zero penalty term present.

Noise as a steering force

Each SGD step follows a noisy mini-batch gradient, not the true gradient. Near sharp minima that noise causes big fluctuations, effectively kicking the iterate back out; near flat minima fluctuations are gentle, so the iterate settles. Averaged over many steps, this asymmetry steers the trajectory toward flatter regions — purely from the noise structure, before any penalty term enters the picture.

Plain SGD, no penalty

Weight normsettles near $\|w\| \approx 4.1$ after training
Mechanismgradient-noise fluctuations disfavor sharp minima, letting the iterate settle in flat ones
No $\lambda\|w\|^2$ term ever appears in the loss

SGD plus explicit L2

Weight normsettles near $\|w\| \approx 3.9$ after training
Mechanismadded $\lambda\|w\|^2$ term directly penalizes large weights
Final norm and test accuracy end up close to the plain-SGD run

Myth

If a model trained with plain SGD ends up with small weights and good generalization, there must be a hidden L2 term or weight decay somewhere in its setup.

Reality

No penalty is needed: the optimizer’s own training dynamics alone bias it toward flat, low-norm solutions - noise steers toward flat minima, while the gradient dynamics themselves supply the norm bias - so plain SGD can come close to an explicitly regularized model’s outcome without any added term in its loss.

Implicit regularization of SGD

implicit regularization

Noise as a steering force

Sources

A focused session on your AI Research Scientist interview

More Scaling Laws & Stability questions