Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search

Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search The KV-cache note made the generation loop practical: prefill the prompt, keep each layer’s keys and values, and run only the newest token through the model at every decode step. But it deliberately skipped the final decision in that loop. After the final Transformer layer and language-model head, we have one logit per vocabulary token. Which token do we actually put into the answer? ...

November 21, 2026 · 16 min

Speculative Decoding: Verifying Several Tokens in Parallel

Speculative Decoding: Verifying Several Tokens in Parallel The decoding-strategies note answered which token to choose from a next-token distribution. It did not remove the central inefficiency of generation: every accepted token normally requires another serial step through the large target model. KV caching makes each of those steps cheaper by reusing old keys and values. But the loop is still serial: target model → token 1 → target model → token 2 → target model → token 3 Speculative decoding adds a second, cheaper model called the draft model. The draft proposes several tokens serially. The large target model then examines the whole proposed suffix in one parallel forward pass and decides what to accept. ...

November 21, 2026 · 17 min

KV Cache: Making Autoregressive Generation Practical

KV Cache: Making Autoregressive Generation Practical The language-model training note ended with an asymmetry: training processes every position in parallel, but generation produces one token at a time. Each new token is appended to the context, and the model is run again to predict the next one. If we implement that loop literally, we repeat an enormous amount of work. To generate token 100, we recompute the embeddings, queries, keys, values, attention outputs, and feedforward outputs for tokens 1 through 99 — even though we already computed all of them in earlier iterations. ...

November 10, 2026 · 20 min

Attention: Turning Token Vectors into Context Vectors

Attention: Turning Token Vectors into Context Vectors The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time. But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first. ...

September 20, 2026 · 24 min

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order. Attention is blind to token order This was highlighted in previous note here 4. Sets, not sequences. Nothing in the formula knows which token came first. Permute the input rows of $X$ and the outputs permute identically — attention is permutation equivariant: it treats the sequence as a bag of tokens with affinities. That’s great for set-like inputs and a true bug for language, whose meaning is bound up in order (“dog bites man” ≠ “man bites dog”). Fixing that is positional encoding, coming in the next note. One attention head can learn only relevance pattern at a time This idea was only mentioned at the end of the previous note. It was never justified or derived. So let’s try to make sense of it. Short answer: The answer lies in what happens when we compute $QK^\top$ . From 3b1b’s video if this token is useful to another token, what information should it send? If we think about it, a token can be useful to the next token in multiple different ways—for a given token A, its predecessor token B might be more useful when considering grammatical context, and token C might be more useful when considering semantic context. We are forcing one attention head to somehow capture all those relationships and turn them into one set of weights ( $W_Q$ , $W_K$ , and $W_V$ ). Even if it somehow ends up capturing these different spatial relationships, at best they fight with each other and average them out. (Technically, $W_V$ (the Value weights) doesn’t decide who to look at; it only decides what information to extract once the connection is made. The “fighting and averaging out” happens specifically inside $W_Q$ and $W_K$ , which calculate the attention map.) Long answer: One attention head can only learn one relevance pattern at a time because the query-key interaction is governed by a single pair of linear projection matrices ( $W_Q$ and $W_K$ ). When we compute $QK^\top$ , we perform a single matrix multiplication that collapses the high-dimensional token representations into one scalar affinity score for every pair of tokens. Mathematically and conceptually, here is why this limits a single head to one “type” of similarity: The Rank and Geometry of the Attention Matrix The similarity matrix before softmax is computed as: ...

September 20, 2026 · 35 min

Training a Language Model End to End: From Text to Loss to Generation

Training a Language Model End to End: From Text to Loss to Generation We now have every moving part: token → embedding (the lookup note), tokens mix via masked multi-head attention (previous two notes), blocks stack, and the whole thing is just a computation graph built from matrix multiplies and softmax — the exact family the general $L$ -layer loop handles. What remains is boring-sounding but is actually the point of the whole series: how does a stack of transformer blocks become a language model that predicts text? The answer has four pieces: a final linear layer to vocabulary-sized logits, cross-entropy per position against the next token, teacher forcing during training, and autoregressive sampling at inference. The pieces are individually simple; seeing them end-to-end is what makes “GPT” stop being magical. ...

September 20, 2026 · 15 min