Softmax on Sanketh's Blog

Attention: Turning Token Vectors into Context Vectors

Sun, 20 Sep 2026 00:00:00 +0530

Attention: Turning Token Vectors into Context Vectors

The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time.

But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first.

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

Mon, 14 Sep 2026 00:00:00 +0000

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

So far, every classification in these notes has been binary — spam or not, XOR’s 0 or 1 — and the output has been single sigmoid feeding binary cross-entropy, whose gradient collapsed to the beautiful $\delta = a - y$ .

Real classifiers rarely answer two-way questions. “Which of 10 digits is this image?” “Which of 50,000 tokens comes next?” “Is this a cat, a dog, or a bird?” This note generalizes the output of a neural network to $k$ classes, and it turns out almost everything we know carries over — with soft-max doing the job sigmoid did.