If a training loss curve plateaus and then shows sudden sharp drops at exactly epochs 30, 60, and 90, with smooth loss in between, the schedule most likely used step decay with milestones at those epochs rather than cosine annealing.

True. Step decay's abrupt multiplicative LR cuts at fixed milestones shrink the step size and gradient-noise floor right at those points, producing the classic sharp loss drops onto a new plateau; cosine's continuous taper has no discontinuities, so its loss improves smoothly with no milestone-aligned events.

Cosine annealing and other LR decay shapes Interview Q&A

Cosine annealing

Cosine annealing decays the learning rate smoothly along a cosine curve from an initial value

\eta_0

toward a minimum

\eta_{\text{min}}

\eta_t = \eta_{\text{min}} + \tfrac{1}{2}(\eta_0-\eta_{\text{min}})\left(1+\cos\left(\tfrac{t}{T}\pi\right)\right)

No abrupt jumps — just a smooth taper.

Example

Over 100 epochs, LR eases from 0.1, barely moves early, drops steepest mid-training, then flattens near 0.001 by epoch 100 — a smooth S-curve.

Cosine decay

Rate of change varies — barely moves near the start and end, steepest at the midpoint of training.
Plots as a smooth S-curve with no kinks or corners.

Linear decay

Rate of change is constant — the LR drops by the same fixed amount every step, from $\eta_0$ to a final value.
Plots as a straight line the whole way, with a single slope.

Setup

LR starts at 0.1 and is halved every 10 epochs. What happens to the LR and loss curve at each 10-epoch milestone?

1
LR stays flat at 0.1, then cuts to 0.05 exactly at epoch 10.
WhyStep decay holds the rate constant within an interval, then applies a fixed multiplicative drop only at milestones.
2
The loss curve, which had flattened, drops sharply right after epoch 10 before settling onto a new plateau.
WhyThe abrupt LR cut shrinks the step size and gradient-noise floor all at once, so loss falls quickly - a gradual taper spreads that same improvement out smoothly.

Takeaway

Step decay is a staircase — flat plateaus, sudden cliffs — unlike cosine or linear’s smooth taper.

Shape shapes dynamics: cosine's smooth taper lets loss improve smoothly throughout training. Step decay's abrupt multiplicative cuts shrink the step size all at once, producing a visible sharp drop in loss right at each milestone before it plateaus again. Linear sits between — no discontinuity, but its constant slope doesn't linger at low LR near the end the way cosine does. Spotting the shape — S-curve, straight line, or staircase — tells you what the loss curve will do.

Cosine annealing and other LR decay shapes

Cosine annealing

Sources

A focused session on your AI Research Scientist interview

More Optimization Algorithms & Schedules questions