AI Research Scientist · Scaling Laws & Stability

Depth vs width tradeoffs in transformer scaling

Real lesson card · Page 1 of 4

Depth vs. width allocation

A fixed parameter budget can be spent on more layers (depth) or on wider layers (width) — same total size, different shape.

Example

A 1B-parameter budget could become 48 thin layers or 12 wide ones; both fit the budget but train and perform differently.

When
You need more sequential reasoning steps per token
lean slightly deeper
Why
extra layers add cheap composition depth
When
You need richer per-token representations with faster parallel training
lean slightly wider
Why
width adds capacity without lengthening the sequential path

Setup

Stretch the same budget into 200 razor-thin layers instead of a balanced split.

Expected

More layers should mean more reasoning steps and better performance overall.

What breaks

Instead, signal degrades across so many thin sequential steps, training destabilizes, and accuracy falls below the balanced split.

Lesson

Depth helps only up to a point — past it, extra layers hurt more than they help.

Myth

Making layers wider is a safe way to add capacity since it avoids depth’s training instability.

Reality

Extremely wide, shallow layers waste parameters on redundant features and limit sequential composition, underperforming a balanced split just as extreme depth does.

Recall check from the same lesson

If a very wide, shallow transformer underperforms a balanced depth/width model at the same parameter budget, the most likely cause is that the wide model has too few layers to build up sequential composition, not that it has too few total parameters.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Scaling Laws & Stability questions

All Scaling Laws & Stability Emergent capabilities and scaling discontinuities Implicit regularization of SGD Loss landscape geometry Neural scaling law breakdowns and irreducible loss Numerical precision and mixed-precision failure modes Scaling laws and compute-optimal allocation Sharpness and flat minima Sharpness-aware minimization (SAM) algorithm Training instability diagnosis