AI Research Scientist · Softmax & Cross-Entropy

Softmax function and logits

Real lesson card · Page 1 of 4

Softmax

Softmax converts logits

z_1,\dots,z_n

into a distribution:

p_i=\dfrac{e^{z_i}}{\sum_j e^{z_j}}

. Every

p_i>0

and they sum to 1, giving valid class probabilities.

Example

Logits $[2,1,0]$ give exponentials $[7.39,2.72,1]$, sum

11.11

, so probabilities

\approx[0.665,0.245,0.090]

Numerically Stable Softmax

1
Find the max logitTake $m=\max_i z_i$ from the vector — e.g. $m=2$ for $[2,1,0]$.
2
Subtract before exponentiatingCompute $z_i-m$ for each entry, giving $[0,-1,-2]$ so no exponent can overflow.
3
Exponentiate and normalizeExponentiate the shifted values and divide by their sum. The result is mathematically identical to the raw formula, just overflow-safe.

Setup

A model outputs logits $[4,1,0]$ for three classes. How does dividing by a temperature

T

before softmax change the resulting distribution?

1
Compute softmax at $T=1$ : probabilities $\approx[0.936,0.047,0.017]$ .
WhyThis is the baseline — already peaked toward the largest logit.
2
Set $T=0.5$ : logits become $[8,2,0]$, softmax $\approx[0.997,0.002,0.001]$ .
WhyA small $T$ stretches the gaps between logits, pushing the distribution toward one-hot.
3
Set $T=5$ : logits become $[0.8,0.2,0]$, softmax $\approx[0.50,0.27,0.22]$ .
WhyA large $T$ shrinks the gaps, flattening the distribution toward uniform.

Takeaway

Temperature rescales the gaps between logits before normalization — it never changes which class is most likely, only how confidently the distribution commits.

Myth

Since softmax looks like several independent squashing functions applied side by side, its gradient must be diagonal, like a sigmoid’s elementwise derivative.

Reality

Every output depends on every logit through the shared normalization, so the Jacobian is

\text{diag}(p)-pp^\top

— the off-diagonal terms are nonzero whenever

i\neq j

Recall check from the same lesson

Because softmax squashes each logit into a probability, its Jacobian is a diagonal matrix where entry $i$ only depends on $p_i(1-p_i)$, just like the sigmoid derivative.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Softmax & Cross-Entropy questions

All Softmax & Cross-Entropy Cross-entropy loss and maximum likelihood training