AI Research Scientist · Deep Network Training Basics

Backpropagation through deep networks

Real lesson card · Page 1 of 4

Backpropagation

Backpropagation is the chain-rule algorithm that computes the gradient of the loss

L

with respect to every parameter in a multi-layer network by propagating that gradient backward through the computation graph.

Example

To get

\partial L/\partial w_1

for a weight three layers deep, backprop multiplies the local derivatives along the graph path from

L

back to

w_1

Tracing the backward pass

1
Seed the loss gradientSet $\partial L/\partial L = 1$ at the loss node—the starting upstream gradient for the whole backward sweep.
2
Multiply by each local derivativeAt every node, multiply the upstream gradient by that operation’s local derivative, using the forward pass’s cached values.
3
Propagate backward until parametersPass the product to each input node, moving edge by edge until every parameter holds its chain-rule gradient, summing shared paths.

Setup

Compute the forward pass for

u = w \cdot x

v = u + b

L = v^2

, given

x=2

w=3

b=1

1
Compute $u = w \cdot x = 6$ and cache it.
WhyBackprop will reuse this node’s cached inputs: $\partial u/\partial w = x$ and $\partial u/\partial x = w$ .
2
Compute $v = u + b = 7$ and cache it.
Why $v$ is needed to compute $\partial L/\partial v = 2v$ later.
3
Compute $L = v^2 = 49$ .
WhyThis is where the backward sweep seeds its gradient at 1.

Takeaway

Values are cached because local derivatives will need them:

v

feeds

\partial L/\partial v = 2v

, and the multiply node’s inputs

x

and

w

feed

\partial u/\partial w

and

\partial u/\partial x

Computation graph

is like

Assembly-line flowchart

Both are directed graphs where each node consumes outputs from its predecessors, and both passes just walk the same edges in opposite directions. The analogy breaks where a flowchart could loop or dead-end: a computation graph must stay acyclic, and every node needs a path forward to the loss, or its gradient never arrives.

Recall check from the same lesson

During the backward pass, if you discard the cached intermediate values from the forward pass, you can still compute correct gradients as long as you know the final loss value.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

CS231n: Backpropagation, Intuitions (Stanford CS231n course notes)

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

Backpropagation

Tracing the backward pass

Sources

A focused session on your AI Research Scientist interview

More Deep Network Training Basics questions