AI Research Scientist · Optimization Algorithms & Schedules

Adam optimizer

Real lesson card · Page 1 of 4

Adam optimizer

First and second moment estimates

For each parameter, Adam keeps two running averages of its gradients:

m_t

, an exponentially-decayed mean (first moment), and

v_t

, an exponentially-decayed mean of squared gradients (second moment, uncentered variance). Both update every step:

m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t

v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2

Example

If a parameter’s gradient is consistently 0.4,

m_t

drifts toward 0.4 while

v_t

drifts toward 0.16, tracking direction and typical magnitude separately.

Bias-Correcting the Moment Estimates

1
Zero-initialized state $m_0=v_0=0$ , so raw estimates start biased low, especially with typical $\beta_1{=}0.9$ , $\beta_2{=}0.999$ .
2
Divide by (1 − β^t) $\hat m_t=m_t/(1-\beta_1^t)$ and $\hat v_t=v_t/(1-\beta_2^t)$ rescale the raw estimates upward, most strongly at small $t$ .
3
Correction fades over timeAs $t$ grows, $\beta_1^t,\beta_2^t\to0$ , so $\hat m_t\to m_t$ , $\hat v_t\to v_t$ — the fix matters only early in training.

Setup

t=1

: gradient

g_1=0.6

\beta_1=0.9

\beta_2=0.999

\epsilon=10^{-8}

. Compute Adam’s update.

1
Raw moments: $m_1=0.06$ , $v_1=0.00036$ .
WhyAfter one step, $m_t=(1-\beta_1)g$ and $v_t=(1-\beta_2)g^2$ .
2
Bias-corrected: $\hat m_1=0.6$ , $\hat v_1=0.36$ .
WhyDividing by $(1-\beta^t)$ removes the early zero-init bias.
3
Update ratio: $\hat m_1/(\sqrt{\hat v_1}+\epsilon)\approx1.0$ , then scaled by $\alpha$ .
WhyRoot-second-moment normalization keeps the step near unit scale.

Takeaway

Correction plus normalization turns a raw gradient into a self-scaled step.

Myth

Adam just speeds up every parameter’s update by the same amount, like one universal accelerator on the raw gradient.

Reality

Each parameter divides by its own

\sqrt{\hat v_t}+\epsilon

. Large or noisy gradients get a bigger divisor, shrinking the step; small, sparse gradients get a tiny divisor, so weak signals aren’t drowned out — scaling adapts per parameter, not uniformly.

Recall check from the same lesson

Because Adam divides by the bias-corrected second moment, a parameter whose gradient is mostly zero (sparse) gets a proportionally larger effective step on the rare occasions it does receive a nonzero gradient, compared to a parameter with the same-magnitude but constantly-present gradients.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Optimization Algorithms & Schedules questions

All Optimization Algorithms & Schedules Batch size effects on optimization Cosine annealing and other LR decay shapes Gradient descent optimization Learning rate schedules (warmup and decay)