Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities
Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities So far, every classification in these notes has been binary — spam or not, XOR’s 0 or 1 — and the output has been single sigmoid feeding binary cross-entropy, whose gradient collapsed to the beautiful $\delta = a - y$ . Real classifiers rarely answer two-way questions. “Which of 10 digits is this image?” “Which of 50,000 tokens comes next?” “Is this a cat, a dog, or a bird?” This note generalizes the output of a neural network to $k$ classes, and it turns out almost everything we know carries over — with soft-max doing the job sigmoid did. ...