AI Research Scientist · Scaling Laws & Stability

Scaling laws and compute-optimal allocation

Real lesson card · Page 1 of 4

power-law scaling relationship

A power-law scaling relationship says loss falls as a constant power of a scale variable (model size, data size, or compute):

L(C) = a \cdot C^{-\alpha}

. Each fixed multiplicative jump in scale yields the same proportional drop in loss.

Example

With

\alpha=0.05

: going from

10^{20}

10^{21}

FLOPs cuts loss by the same fraction as going from

10^{21}

10^{22}

— constant ratio, not constant amount.

Fitting a Power-Law Curve

1
Log both axesConvert each $(C,L)$ data point to $(\log C, \log L)$ , turning the power law into a line: $\log L = -\alpha \log C + \log a$ .
2
Fit the lineLeast-squares regression on the log-log points yields the best-fit slope $-\alpha$ and intercept $\log a$ .
3
Recover and verifyExponentiate the intercept to get $a$ , then check residuals hug the line within the sampled range.

Setup

A curve fit on runs from

10^{19}

–

10^{22}

FLOPs:

L(C) = 3.1 \cdot C^{-0.05}

. Predict loss at

C=5\times10^{21}

1
Plug $C=5\times10^{21}$ into $L(C)$
WhyDirect evaluation within the fitted range.
2
Compute $C^{-0.05}\approx0.082$
WhyGives the proportional scaling factor.
3
Multiply by $a=3.1$
WhyConverts the scaling factor to absolute loss, $\approx 0.25$ .

Takeaway

Reading a curve means evaluating it at the target scale — trustworthy only when that scale lies inside the fitted range.

Setup

The same curve, fit on

10^{19}

–

10^{22}

FLOPs, is pushed to

C=10^{27}

— five orders past the fitted data.

Expected

Naively evaluating the formula predicts one smooth, specific loss, as if the decay continues unchanged far beyond the fit.

What breaks

At that scale, data exhaustion, architecture limits, or optimizer shifts absent from the fit can pull real loss well off the curve.

Lesson

Trust the fit near its sampled range only; treat far extrapolations as guesses, not reliable predictions.

Recall check from the same lesson

If a power-law curve is fit using compute values between $10^{19}$ and $10^{22}$ FLOPs, plugging $C=10^{30}$ FLOPs into that same fitted formula gives a prediction just as reliable as one at $C=10^{21}$ FLOPs, since it's a single continuous equation valid at any scale.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Scaling Laws & Stability questions

All Scaling Laws & Stability Depth vs width tradeoffs in transformer scaling Emergent capabilities and scaling discontinuities Implicit regularization of SGD Loss landscape geometry Neural scaling law breakdowns and irreducible loss Numerical precision and mixed-precision failure modes Sharpness and flat minima Sharpness-aware minimization (SAM) algorithm Training instability diagnosis