AI Research Scientist · Scaling Laws & Stability

Numerical precision and mixed-precision failure modes

Real lesson card · Page 1 of 4

fp16 overflow

fp16 uses only 5 exponent bits, capping representable magnitude near

65504

. When a gradient or activation’s magnitude exceeds that ceiling in a forward or backward pass, the value clips to infinity or NaN instead of a large finite number.

Example

An attention score climbing to

70000

during training becomes inf in fp16; the next multiply turns it into NaN, corrupting the loss and halting learning.

Myth

Since overflow produces visible NaNs, the only real danger in fp16 training is gradients growing too large.

Reality

FP16’s smallest positive normal value is about

6.10\times10^{-5}

, but subnormal values extend to about

5.96\times10^{-8}

. Values smaller than that round to zero, while subnormal values retain reduced precision. Loss scaling moves small gradients into a safer representable range before they are stored in fp16.

Setup

A backward pass yields a gradient of magnitude

2^{-20}

, just below fp16’s smallest value.

1
Scale the loss by 1024 before backward().
WhyChain rule scales every gradient by the same factor.
2
Gradients now sit near $2^{-10}$ .
WhyThat falls inside fp16’s representable range.
3
Unscale gradients by dividing by 1024 before the optimizer step.
WhyRestores true magnitude for the fp32 update.

Takeaway

Loss scaling shifts small gradients into fp16’s range before clipping, then undoes it — fixing underflow at the root.

fp16 (5-bit exponent)

Rangenarrow ceiling near $65504$ causes overflow to inf or NaN.
Underflowgradients below the minimum normal round silently to zero.

bf16 (8-bit exponent)

Rangematches fp32’s full exponent span, so clipping rarely occurs.
Precisiononly 7 mantissa bits, so stored values round more coarsely.

Recall check from the same lesson

If an fp16 run instead shows gradient norms silently plateauing near zero rather than NaN losses, scaling the loss upward before backward() would help because it shifts those small gradients into fp16's representable range, and then dividing the gradients back down by the same factor before the optimizer step restores their true magnitude.

Finish the teach card to unlock

Sources

Checked August 25, 2026 · Editorial policy

NVIDIA Mixed Precision Training Guide

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Scaling Laws & Stability questions

All Scaling Laws & Stability Depth vs width tradeoffs in transformer scaling Emergent capabilities and scaling discontinuities Implicit regularization of SGD Loss landscape geometry Neural scaling law breakdowns and irreducible loss Scaling laws and compute-optimal allocation Sharpness and flat minima Sharpness-aware minimization (SAM) algorithm Training instability diagnosis