Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

Mon, 14 Sep 2026 00:00:00 +0000

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

So far, every classification in these notes has been binary — spam or not, XOR’s 0 or 1 — and the output has been single sigmoid feeding binary cross-entropy, whose gradient collapsed to the beautiful $\delta = a - y$ .

Real classifiers rarely answer two-way questions. “Which of 10 digits is this image?” “Which of 50,000 tokens comes next?” “Is this a cat, a dog, or a bird?” This note generalizes the output of a neural network to $k$ classes, and it turns out almost everything we know carries over — with soft-max doing the job sigmoid did.

Classification on Sanketh's Blog

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities