AI Research Scientist · Scaling Laws & Stability
Loss landscape geometry
Real lesson card · Page 1 of 4
Loss landscape geometry
loss landscape
The loss landscape is the surface traced out by training loss as a function of every parameter in a network. Because networks have many parameters and nonlinear layers, this surface is generally non-convex: it holds many local minima and saddle points, not one single global bowl.Example
A network with millions of weights doesn’t sit in one smooth valley — its landscape has countless dips, ridges, and flat stretches spread across that parameter space.Setup
Take : is a local minimum?- 1Compute the gradient; at , .WhyZero gradient marks any critical point.
- 2Compare loss at () to and ().WhyHigher loss both sides confirms a true dip.
Takeaway
Zero gradient plus higher loss nearby confirms a minimum — but only locally; other minima and saddles sit elsewhere on the surface.Local minimum
- Gradientzero in every direction at this point.
- Curvaturethe loss rises no matter which way you step.
- Nearby losshigher on all sides, so no direction lowers loss further.
Saddle point
- Gradientalso zero in every direction at this point.
- Curvatureloss rises along some directions and falls along others.
- Nearby losslower along at least one direction, so descent can keep going.
Myth
If training stalls with a near-zero gradient, the model must be trapped in a bad local minimum.Reality
In high-dimensional networks, most zero-gradient points are saddle points, not minima — curving upward in every direction at once becomes rare as dimensions grow, so a stall is more often escapable.Recall check from the same lesson
A point where the gradient is exactly zero and the loss increases in every direction around it is called a saddle point.
Review the explanation
Answer: False. That description is a local minimum, not a saddle point. A saddle point requires the loss to increase in some directions and decrease in others; increasing in every direction around a zero-gradient point is precisely what makes it a minimum, not a saddle.
Sources
One sitting · 20–30 minutes
A focused session on your AI Research Scientist interview
LearnBench starts from what you already know — skip what you have, master what you’re missing.
Start nowMore Scaling Laws & Stability questions
All Scaling Laws & StabilityDepth vs width tradeoffs in transformer scalingEmergent capabilities and scaling discontinuitiesImplicit regularization of SGDNeural scaling law breakdowns and irreducible lossNumerical precision and mixed-precision failure modesScaling laws and compute-optimal allocationSharpness and flat minimaSharpness-aware minimization (SAM) algorithmTraining instability diagnosis