Because weight decay and simplified internal representations reliably appear around the time a model groks, researchers have proven that weight decay is the mechanistic cause of grokking's abrupt generalization jump.

False. Weight decay and representation simplification are well-documented correlates of grokking across studied tasks, but correlation is not proof of a mechanism. The field has not converged on a settled causal explanation for why the abrupt transition happens, so stating one factor as THE proven cause overstates current knowledge.

Grokking and delayed generalization Interview Q&A

Setup

A transformer trained on modular addition (mod 97) hits 99% train accuracy by step 1,000, while test accuracy stays stuck below 10% (chance is about 1%). It jumps from 8% to 96% around step 9,000.

1
Compare the train and test curves.
WhyThey split early: memorization finishes long before test accuracy moves.
2
Notice the test-accuracy rise is a sharp jump, not a slope.
WhyThat abruptness after a long flat stretch is grokking’s signature.

Takeaway

Grokking is near-perfect train accuracy held for many steps while test accuracy stays flat, then jumps abruptly rather than gradually.

Across grokking runs, the abrupt jump coincides with two things: weight decay pressure pushing weights toward smaller norms, and internal representations reorganizing into simpler, structured form - e.g. modular-arithmetic networks develop clean circular embeddings. Removing weight decay often delays or blocks grokking; probing activations shows the structured circuit is amplified gradually before the jump, which coincides instead with cleanup of the memorized components. These are documented correlations, not proof of causation.

Double descent

Capacity axistest error can rise then fall as size or training time grows past an interpolation point.
Grokking’s late jump echoes this second descent, but comes from extra training time, not scaled capacity.

Implicit regularization

Optimization axisgradient descent and weight decay bias training toward simpler, low-norm solutions.
Grokking’s jump looks like implicit regularization finally winning over a memorized solution, after delay.

Myth

Since weight decay and representation simplification both show up right before grokking’s jump, it’s tempting to think researchers have proven weight decay’s pressure toward simpler representations is THE cause of grokking.

Reality

Those are documented correlations across specific tasks, not an agreed-upon mechanism - the field still debates whether weight decay, representation geometry, or other factors actually drive the abrupt transition, and no explanation is settled yet.

Grokking and delayed generalization

Sources

A focused session on your AI Research Scientist interview

More Generalization Theory & Phenomena questions