The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position
The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order. This note turns attention into the actual Transformer block. The block is attention plus four supporting acts — multi-head projection, a residual connection, layer normalization, and a small feedforward network — and then positional encoding bolted onto the input. None of them are exotic. Each one exists to patch a specific failure of the bare mechanism, and by the end you should be able to name the patch for each failure. ...