Attention: Turning Token Vectors into Context Vectors

Sun, 20 Sep 2026 00:00:00 +0530

Attention: Turning Token Vectors into Context Vectors

The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time.

But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first.

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

Sun, 20 Sep 2026 00:00:00 +0530

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order.

This note turns attention into the actual Transformer block. The block is attention plus four supporting acts — multi-head projection, a residual connection, layer normalization, and a small feedforward network — and then positional encoding bolted onto the input. None of them are exotic. Each one exists to patch a specific failure of the bare mechanism, and by the end you should be able to name the patch for each failure.

Attention on Sanketh's Blog

Attention: Turning Token Vectors into Context Vectors

Attention: Turning Token Vectors into Context Vectors

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position