AI Research Scientist · Optimization Algorithms & Schedules

Learning rate schedules (warmup and decay)

Real lesson card · Page 1 of 4

Decay shape

After warmup ends, the learning rate follows a decay shape that lowers it for the rest of training: cosine glides smoothly toward near-zero, linear falls at a constant rate, and step cuts it by a fixed factor at set points.

Example

Peak rate 1e-3 over 100k steps: cosine reaches about 1e-6 by the end, linear falls in a straight line to 0, and step drops 10x at steps 50k and 80k.

How the Warmup Ramp Works

1
Start near zeroThe learning rate begins at a small fraction of its eventual peak, so the very first weight updates are gentle.
2
Ramp up over N stepsOver the first few hundred to few thousand steps, the rate climbs steadily toward the target peak value.
3
Peak reached, decay takes overBy the time the ramp finishes, gradients have already nudged weights away from their unstable random start, so full-size steps are now safe and the decay shape begins.

Setup

Train for 10,000 steps. Run A holds a constant rate of 3e-4 throughout. Run B warms up linearly from 0 to 3e-4 over the first 500 steps, then cosine-decays to 3e-6 by the end.

1
Run A applies 3e-4 immediately from step 1.
WhyRandomly initialized weights meet a large step size right away, causing loss spikes early on.
2
Run B ramps from 0 to 3e-4 across 500 steps.
WhySmall early updates let the model settle before taking full-size steps, avoiding the spike Run A suffers.
3
Both runs sit near 3e-4 around step 500; Run B then cosine-decays toward 3e-6.
WhyShrinking the rate late lets the model settle into a sharper minimum instead of oscillating around it.
4
Compare final loss at step 10,000.
WhyRun B lands at a lower, steadier loss; Run A either spiked early or plateaued higher since its rate never dropped.

Takeaway

Warmup prevents the early blowup and decay sharpens the late-training minimum — together they beat holding one constant rate the whole run.

Myth

As long as the base learning rate is small enough, warmup is unnecessary, and which decay shape you pick barely matters.

Reality

Even a modest rate can destabilize randomly-initialized weights before any steps are taken, so warmup helps regardless of the peak value. And decay shape controls how fast the rate falls late in training: skip it, or pick a mismatched shape, and the rate stays too high near the end, blocking the fine-grained convergence a shrinking rate enables.

Recall check from the same lesson

Because a run used a small constant learning rate of 1e-5 for its entire 20,000 steps instead of warmup+cosine-decay, it would reach just as strong a final result, since the rate never got large enough to cause early instability.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Optimization Algorithms & Schedules questions

All Optimization Algorithms & Schedules Adam optimizer Batch size effects on optimization Cosine annealing and other LR decay shapes Gradient descent optimization