AI Research Scientist · Scaling Laws & Stability

Sharpness-aware minimization (SAM) algorithm

Real lesson card · Page 1 of 3

Perturb-then-descend

is like

Scouting worst footing

Before stepping onto uneven ground, you probe the worst nearby footing, then choose your step using what that probe revealed. SAM works similarly: it approximates the highest-loss point near the current weights, evaluates the gradient there, then applies that gradient update to the original weights. The perturbed weights are a temporary probe, not the point from which the optimizer takes its actual parameter step.

Computing the ascent-step perturbation

1
Find the worst-case directionCompute $g=\nabla_w L(w)$ and normalize it: $\hat g = g/\|g\|$ - the unit direction that increases loss fastest from the current weights.
2
Bound it to the neighborhoodScale by the radius $\rho$ : $\epsilon=\rho\hat g$ , the farthest step allowed within the $\rho$ -ball around $w$ .
3
Form the perturbed weightsAdd it to $w$ : $w_{\text{adv}}=w+\epsilon$ . This is the ascent step - an approximation of the worst-case point near $w$ , used only for the next step.

Setup

Given weight

w=1.0

, loss

L(w)=w^2

, radius

\rho=0.1

, and learning rate

\eta=0.1

, compute one SAM update and compare it to plain SGD.

1
Compute the ascent gradient: $g=2w=2.0$ at $w=1.0$ .
WhyThis is the loss-increasing direction the perturbation is built from.
2
Perturb the weight: $\epsilon=\rho=0.1$ , so $w_{\text{adv}}=1.1$ .
WhyThis places the search at the worst-case point inside the neighborhood, not at $w$ itself.
3
Evaluate the descent gradient at $w_{\text{adv}}$ , not at $w$ : $\nabla L(w_{\text{adv}})=2(1.1)=2.2$ , then update $w \leftarrow 1.0-0.1(2.2)=0.78$ , versus SGD’s $1.0-0.1(2.0)=0.80$ .
WhyUsing the perturbed-point gradient makes the update react to how loss behaves nearby, not just at one point.

Takeaway

Because the descent step always uses the perturbed-point gradient, SAM’s update implicitly penalizes regions where loss rises steeply nearby - steering weights toward minima that stay low across their whole neighborhood, not just at one point.

Recall check from the same lesson

Since SAM's ascent step already finds the highest-loss point near the current weights, the descent step could equivalently just run ordinary gradient descent from the original weights $w$ and still bias training toward flatter minima.

Finish the teach card to unlock

Sources

Checked August 25, 2026 · Editorial policy

Sharpness-Aware Minimization for Efficiently Improving Generalization

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Scaling Laws & Stability questions

All Scaling Laws & Stability Depth vs width tradeoffs in transformer scaling Emergent capabilities and scaling discontinuities Implicit regularization of SGD Loss landscape geometry Neural scaling law breakdowns and irreducible loss Numerical precision and mixed-precision failure modes Scaling laws and compute-optimal allocation Sharpness and flat minima Training instability diagnosis