AI Research Scientist · Generalization Theory & Phenomena
Bias-variance tradeoff and diagnosis
Real lesson card · Page 1 of 4
Bias-variance tradeoff and diagnosis
Bias vs. variance
Bias is systematic error from an overly simple model missing the true pattern. Variance is error from sensitivity to the training set, causing predictions to swing across samples.Example
A straight line on curved data has high bias; a deep tree on noisy data has high variance.Reading train vs. val curves
- 1Check absolute lossIf both losses stay high, the model can’t fit training data — bias-dominated underfitting.
- 2Check the gapIf train loss is low but val loss is much higher, the model overfit — variance-dominated.
- 3ClassifySmall gap with high loss means bias; large gap with low train loss means variance.
Setup
After training, train loss is 0.05 and validation loss is 0.42, both curves flat.- 1Read absolute levelsWhy0.05 train loss shows the model fits training data well.
- 2Read the gapWhyThe 0.37 gap means it doesn’t generalize to new data.
- 3ClassifyWhyLow train loss plus large gap signals variance-dominated overfitting.
Takeaway
A big gap with low train loss marks overfitting, not underfitting.Myth
A large train/val gap always means overfitting.Reality
A noisy or small validation set, or an unrepresentative split, can create a big gap even when train loss is also high — check absolute train loss too, not the gap alone.Recall check from the same lesson
If validation loss is much higher than training loss purely because the validation set is small and noisy, that always means the model is overfitting and needs more regularization.
Review the explanation
Answer: False. A big train/val gap can come from a noisy or unrepresentative validation split rather than genuine overfitting; you have to check the absolute training loss too before concluding the regime is variance-dominated.
Sources
One sitting · 20–30 minutes
A focused session on your AI Research Scientist interview
LearnBench starts from what you already know — skip what you have, master what you’re missing.
Start nowMore Generalization Theory & Phenomena questions
All Generalization Theory & PhenomenaDouble descent phenomenonDouble descent risk-curve explanationGeneralization theoryGrokking and delayed generalizationGrokking mechanistic explanationsGrokking phase-transition mechanistic accountInformation bottleneck and generalization boundsNeural tangent kernel linearization regimeNeural tangent kernel perspective