AI Research Scientist · Deep Network Training Basics

Dropout and stochastic regularization

Real lesson card · Page 1 of 4

Dropout

Dropout is a training-time regularization technique: each pass, every unit is independently zeroed out with probability

p

, so gradients update a different random subnetwork every step instead of the full network.

Example

With

p=0.5

on 10 units, about 5 fire on one pass, a different 5 the next.

Because any unit might vanish on the next pass, no unit can safely rely on a specific partner unit to fix its mistakes. Each unit is pushed to compute something useful on its own, so units stop co-adapting into fragile, mutually-dependent teams.

Myth

Dropout should stay active at test time, since that’s what made training work.

Reality

At test time, dropout is turned off and outputs are rescaled (multiplying by the keep probability, or equivalently scaling during training) so the expected activation matches the value the downstream layers were trained to expect.

Dropout ensemble effect

is like

Overlapping team roster

Each dropout pass trains one of exponentially many weight-sharing subnetworks, like a company drawing overlapping project teams from one talent pool—no team can lean on a specific teammate always showing up. At test time you don’t literally vote across sub-teams; you approximate that average by running the full network with rescaled weights.

Recall check from the same lesson

Dropout reduces overfitting mainly because randomly removing units during training makes the network smaller and thus less prone to memorizing; this shrinking effect, not any ensemble or co-adaptation reasoning, is the real explanation for why it works.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

Dropout

Sources

A focused session on your AI Research Scientist interview

More Deep Network Training Basics questions