AI Research Scientist · Scaling Laws & Stability

Training instability diagnosis

Real lesson card · Page 1 of 4

Loss spike

A loss spike is a sudden, sharp jump in training loss over just one or a few steps, followed by the loss returning toward its prior trend. It’s transient: the curve bounces, not breaks.

Example

Loss tracks near 2.1 for hundreds of steps, jumps to 9.4 at step 5,000, then falls back to about 2.2 within the next 20 steps.

Loss spike

Stays finitethe loss value is still a real, readable number, just large.
Transientusually recovers within tens of steps as gradients settle.

NaN/Inf failure

Non-finiteloss or gradients become NaN or Inf, no longer a usable number.
Permanentonce weights update with NaN gradients, the model state is corrupted and won’t self-recover.

Setup

Loss holds near 1.8 for 3,000 straight steps, no spike upward, no NaN or Inf in the logs.

Expected

The usual checklist expects a visible jump or a broken value explaining the unhealthy run.

What breaks

Neither shows up: loss stays finite and steady, simply stopped decreasing — stalled, not spiked or corrupted.

Lesson

Spot a plateau by absence: no jump, no NaN, just loss refusing to fall.

Myth

Any stretch where the loss line is climbing counts as ‘a spike’.

Reality

A true spike is a brief jump that recovers within tens of steps; sustained upward drift over hundreds of steps with no recovery instead signals divergence — a separate, more serious failure mode.

Recall check from the same lesson

In a training log, loss is 2.9 at step 800, becomes NaN at step 801, and stays NaN for the rest of the run. This should be diagnosed as a loss spike because the value jumped sharply at step 801.

Finish the teach card to unlock

Real lesson card · Page 1 of 4

Training instability diagnosis

Learning rate as root cause

An inappropriately high or unstable learning rate produces instability because each update step overshoots the loss landscape’s curvature, pushing weights past a good minimum into a region of higher loss. If the overshoot compounds step after step, the result is a spike or outright divergence rather than steady descent.

Example

If a run is stable at LR = 1e-4 but starts spiking right after the schedule bumps to LR = 3e-3, the learning rate increase is the natural first suspect.

Setup

A transformer’s output projection is initialized with std = 1.0 instead of the usual

1/\sqrt{d}

scaling. Loss explodes within the first 40 steps, before the learning-rate warmup has ramped past its near-zero floor.

1
Note when the blowup starts.
WhyInit-driven failures appear almost immediately, in the first few dozen steps, because oversized weights inflate activations before LR has warmed up enough to be doing meaningful damage on its own.
2
Check whether the instability persists even with LR near zero.
WhyIf lowering LR further doesn’t help, LR isn’t the driver — the fault must be in the starting weight magnitudes themselves.
3
Inspect activation and gradient norms at step 1.
WhyPoor initialization inflates forward activations and backward gradients before any optimizer step runs, confirming the problem originates at initialization, not from the update rule.

Takeaway

Instability that appears at the very start of training, while LR is still near zero from warmup, points to initialization scale rather than the learning rate.

Setup

LR is small and stable, weights use standard

1/\sqrt{d}

initialization, and training runs fine for 3,000 steps in fp16 — then loss suddenly becomes NaN and never recovers.

Expected

With both LR and initialization already ruled out, the usual checklist expects the run to keep behaving, since those are the two hypotheses normally checked first.

What breaks

fp16 has a limited exponent range; an activation or gradient value overflows past roughly 65,504, becomes Inf, and that Inf propagates into NaN on the next operation — a failure mode LR/init reasoning can’t explain.

Lesson

When LR and init both check out yet NaNs still appear, suspect numeric precision (e.g. fp16 overflow) rather than re-litigating the learning rate or initialization.

Myth

Since learning rate, initialization, and precision cover most instability cases, any loss spike or NaN must trace back to one of those three hyperparameter or numeric settings.

Reality

A single corrupted or mis-scaled batch — an extreme outlier value, wrong unit scaling, or a stray Inf already sitting in the input — can trigger an identical-looking spike or NaN even with perfect LR, init, and precision, so the offending batch itself deserves a check before re-tuning the model.

Recall check from the same lesson

A loss spike that occurs during the very first 20 steps of training, while warmup is still holding the learning rate near zero, is most likely explained by a learning-rate spike rather than by initialization scale.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Scaling Laws & Stability questions

All Scaling Laws & Stability Depth vs width tradeoffs in transformer scaling Emergent capabilities and scaling discontinuities Implicit regularization of SGD Loss landscape geometry Neural scaling law breakdowns and irreducible loss Numerical precision and mixed-precision failure modes Scaling laws and compute-optimal allocation Sharpness and flat minima Sharpness-aware minimization (SAM) algorithm