AI Research Scientist · Deep Network Training Basics
Gradient clipping
Real lesson card · Page 1 of 4
Gradient clipping
Gradient clipping
Gradient clipping caps the norm or magnitude of a model’s gradients before the optimizer applies its update, so a single bad batch can’t push parameters too far.Example
If a batch’s gradient norm spikes to 50 while the clip threshold is 5, clipping rescales every gradient down so the overall norm becomes exactly 5 before the update happens.Global-Norm Clipping Procedure
- 1Compute the global normTreat every parameter gradient as one long vector and compute a single scalar norm:
- 2Compare to thresholdCompare to a chosen threshold (say, 1.0). If , do nothing — the update proceeds unchanged.
- 3Rescale proportionallyOtherwise, multiply every gradient by , shrinking all of them by the same factor — direction stays the same, only magnitude shrinks.
Plain SGD + clipping
- SGD applies the clipped gradient directly — clipping is the sole guard against a destabilizing step.
- A norm-5 clip means every gradient shrinks by the same factor before the update lands.
Adaptive optimizer + clipping
- Adam-style optimizers already rescale each gradient by a per-parameter variance estimate.
- Clipping still runs first, bounding the raw pre-update gradient norm before that per-parameter rescaling happens.
- WhenEarly training, or right after a loss spike from a bad batchKeep clipping enabled, with a modest threshold like 1.0-5.0WhyGradients are often erratic before the model settles, so one bad batch can otherwise wreck the parameters.
- WhenStable, late-stage training with smoothly decreasing lossClipping matters less, though many pipelines leave it on as cheap insuranceWhyGradients are already well-behaved, so the threshold is rarely triggered and the update goes through unchanged.
Recall check from the same lesson
Once a model's loss curve has been smoothly decreasing for a long stretch and gradients are well-behaved, keeping gradient clipping enabled provides little additional protection because the clip threshold is rarely triggered.
Review the explanation
Answer: True. In stable late-training regimes, gradient norms rarely exceed the threshold, so clipping mostly sits idle. Its main protective value shows up during early or unstable training phases where large gradient spikes are common.
Sources
One sitting · 20–30 minutes
A focused session on your AI Research Scientist interview
LearnBench starts from what you already know — skip what you have, master what you’re missing.
Start now