https://sankethbk.github.io/blog/tags/positional-encoding/Recent content in Positional-Encoding on Sanketh's BlogHugo -- 0.166.0en-usSun, 20 Sep 2026 00:00:00 +0530https://sankethbk.github.io/blog/posts/ml/2026-09-20-2-the-transformer-block/Sun, 20 Sep 2026 00:00:00 +0530https://sankethbk.github.io/blog/posts/ml/2026-09-20-2-the-transformer-block/<h1 id="the-transformer-block-multi-head-attention-residuals-norms-and-position">The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position</h1> <p>The <a href="https://sankethbk.github.io/blog/posts/ml/2026-09-20-1-attention-turning-token-vectors-into-context">attention note</a> derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only <em>one</em> relevance pattern at a time, and the whole operation is blind to token order.</p> <p>This note turns attention into the actual Transformer block. The block is attention plus four supporting acts — multi-head projection, a residual connection, layer normalization, and a small feedforward network — and then positional encoding bolted onto the input. None of them are exotic. Each one exists to patch a specific failure of the bare mechanism, and by the end you should be able to name the patch for each failure.</p>