AI Research Scientist · Generalization Theory & Phenomena

Grokking mechanistic explanations

Real lesson card · Page 1 of 4

Weight-norm growth dynamics hypothesis

Grokking’s mechanistic explanations are competing causal hypotheses, not settled facts. This one claims the weight norm’s trajectory itself drives delayed generalization: it grows during memorization, plateaus, then its shrinkage triggers the switch to generalizing.

Example

In a modular-arithmetic transformer, weight norm climbs while train loss drops, plateaus for thousands of steps, then dips sharply as test accuracy jumps — the dip is treated as the cause.

Weight decay: delay then trigger

1
Memorization dominates earlyWeight decay is weak early, so the large memorizing circuit fits training data fast while the smaller generalizing circuit stays underdeveloped.
2
Decay erodes itOnce training loss is near zero, decay keeps shrinking the memorizing circuit’s weights every step, slowly eroding its dominance.
3
Threshold flips controlWhen decay pushes memorizing weights below the generalizing circuit’s effective size, the cheaper circuit wins — test accuracy jumps abruptly.

Slingshot mechanism

Signatureweight-norm and loss oscillate in spikes before the transition, rather than declining smoothly.
Triggergeneralization follows a spike-and-recovery cycle in the training dynamics, not a steady descent.

Weight-decay account

Signatureweight norm falls in one smooth, monotonic decline once decay dominates the loss landscape.
Triggergeneralization appears once the shrinking norm crosses a threshold, with no spikes in the curve.

Setup

A training run shows the generalizing circuit is already more weight-efficient than the memorizing circuit early on.

Expected

Efficiency alone predicts generalization should appear as soon as the efficient circuit exists, since decay should favor it immediately.

What breaks

Test accuracy stays flat for tens of thousands more steps — efficiency existed long before the transition fired.

Lesson

Circuit efficiency explains which circuit wins eventually, but only decay’s pressure explains when — the hypotheses combine, not substitute.

Recall check from the same lesson

If a generalizing circuit is already measurably more weight-efficient than the memorizing circuit from early in training, the circuit-efficiency hypothesis alone fully explains why the generalization transition doesn't fire until much later.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Generalization Theory & Phenomena questions

All Generalization Theory & Phenomena Bias-variance tradeoff and diagnosis Double descent phenomenon Double descent risk-curve explanation Generalization theory Grokking and delayed generalization Grokking phase-transition mechanistic account Information bottleneck and generalization bounds Neural tangent kernel linearization regime Neural tangent kernel perspective