AI Research Scientist · Optimization Algorithms & Schedules

Gradient descent optimization

Real lesson card · Page 1 of 4

Gradient descent update rule

Gradient descent updates a parameter by stepping opposite the gradient of the loss:

\theta \leftarrow \theta - \eta \nabla_\theta L(\theta)

Here

\eta

is the learning rate controlling step size.

Example

With

\theta=5

, gradient

\nabla_\theta L=2

, and

\eta=0.1

\theta \leftarrow 5-0.1(2)=4.8

How the learning rate shapes the trajectory

1
Too small $\eta=0.01$ : steps are tiny, progress toward the minimum is slow but stable.
2
Too large $\eta=1.5$ : steps overshoot, loss oscillates or diverges instead of shrinking.
3
Well-tuned $\eta=0.1$ : steps shrink as the gradient flattens, converging steadily.

Full-batch gradient

Averages all 10,000 examples’ gradients each step: accurate but costly.
Smooth convergence path since noise is minimal.

Mini-batch & stochastic

Mini-batch of 32 examples costs far less per step but adds noise.
Stochastic (batch=1) is cheapest yet noisiest, needing more steps to settle.

Myth

Vanilla gradient descent already gives each parameter its own adaptive learning rate based on past gradients.

Reality

It uses one global

\eta

applied identically to every parameter each step; per-parameter adaptive scaling (like Adam) is a separate, more advanced optimizer built on top of this update rule.

Recall check from the same lesson

For a training set with 50 million examples, computing the full-batch gradient every step is usually preferred over mini-batch or stochastic updates because it gives the most accurate gradient direction each step.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

An overview of gradient descent optimization algorithms (Ruder, 2016)

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Optimization Algorithms & Schedules questions

All Optimization Algorithms & Schedules Adam optimizer Batch size effects on optimization Cosine annealing and other LR decay shapes Learning rate schedules (warmup and decay)