AI Research Scientist · Scaling Laws & Stability

Sharpness and flat minima

Real lesson card · Page 1 of 4

Sharpness

Sharpness describes how steeply the loss increases as parameters move away from a minimum: a sharp minimum sits in a narrow, high-curvature bowl, while a flat minimum sits in a wide, low-curvature basin.

Example

Picture two valleys at the same loss depth: one is a narrow V-shaped notch, the other a broad, gently sloped bowl. The notch is sharp; the bowl is flat.

Flatter minima empirically show a smaller generalization gap — the gap between training and test loss — than sharper minima at the same training loss. Why: small parameter shifts, from noise or from train-test mismatch, barely raise loss inside a wide flat basin, but sharply raise loss inside a narrow sharp one. This is a robust empirical pattern, not a proven law, yet strong enough to guide design choices.

When
You must pick between two optimizers with similar training loss.
Choose the flatness-seeking one, such as SAM.
Why
It steers training into wide basins, which the sharpness-generalization link predicts will generalize better.
When
You’re choosing between a very large batch size and a moderate one for the same training budget.
Prefer the moderate batch, since very large batches tend to converge to sharp minima.
Why
Gradient noise from smaller batches steers training toward wide, flat basins, matching the flat-minima generalization pattern.

Setup

Two teammates want to rank exactly how sharp two saved checkpoints are.

Expected

Reasoning qualitatively, they expect to eyeball a single loss-curve slice and call one sharper.

What breaks

One 2D slice can look flat while the true curvature, spanning many directions, is sharp elsewhere — the full picture needs the Hessian eigenvalue spectrum, not a glance.

Lesson

Qualitative sharpness reasoning explains why flat basins generalize better and justifies flatness-seeking methods; it cannot certify one point’s exact sharpness — that needs formal curvature metrics beyond this scope.

Recall check from the same lesson

To justify preferring a visibly flatter minimum for deployment over a visibly sharper one, you must first compute both minima's Hessian eigenvalues and show the flatter one's eigenvalues are numerically smaller.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Scaling Laws & Stability questions

All Scaling Laws & Stability Depth vs width tradeoffs in transformer scaling Emergent capabilities and scaling discontinuities Implicit regularization of SGD Loss landscape geometry Neural scaling law breakdowns and irreducible loss Numerical precision and mixed-precision failure modes Scaling laws and compute-optimal allocation Sharpness-aware minimization (SAM) algorithm Training instability diagnosis