AI Research Scientist · Softmax & Cross-Entropy

Cross-entropy loss and maximum likelihood training

Real lesson card · Page 1 of 4

Cross-entropy loss

Cross-entropy loss measures how far a model’s predicted probability distribution over classes is from the true target distribution; it’s the default training objective for classification and next-token prediction.

Example

For a 3-class model predicting $[0.7, 0.2, 0.1]$ when the true class is class 0, cross-entropy is

-\log(0.7) \approx 0.36

Cross-entropy $H(p,q)=-\sum_i p_i\log q_i$ . With one-hot true labels this reduces to $-\log q_y$ , the negative log-likelihood — so minimizing CE over the dataset is exactly maximizing likelihood. Also $H(p,q)=H(p)+D_{KL}(p\|q)$ : since $H(p)$ is fixed, minimizing CE also minimizes KL divergence from the true distribution to the prediction.

Setup

Logits are $[2.0, 1.0, 0.1]$, true class 0. Softmax gives

q

. Find

\partial L/\partial z_i

for cross-entropy loss.

1
Compute softmax: $q \approx [0.66, 0.24, 0.10]$ .
WhyCross-entropy needs probabilities, not raw logits.
2
Gradient: $q - y = [-0.34, 0.24, 0.10]$ .
WhySoftmax+CE gradient reduces to predicted minus true, skipping the Jacobian.

Takeaway

Predicted probability minus true label is the whole gradient backprop receives at the logits.

Myth

Cross-entropy training always compares predictions to a hard one-hot target.

Reality

Label smoothing softens that target, spreading a little mass onto other classes — raising loss slightly but reducing overconfidence and improving calibration.

Recall check from the same lesson

Label smoothing keeps the target for the true class at exactly 1 and every other class at exactly 0, so it trains identically to standard one-hot cross-entropy.

Finish the teach card to unlock

Sources

Checked August 27, 2026 · Editorial policy

One sitting · 20–30 minutes

A focused session on your AI Research Scientist interview

LearnBench starts from what you already know — skip what you have, master what you’re missing.

Start now

More Softmax & Cross-Entropy questions

All Softmax & Cross-Entropy Softmax function and logits