Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search

Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search The KV-cache note made the generation loop practical: prefill the prompt, keep each layer’s keys and values, and run only the newest token through the model at every decode step. But it deliberately skipped the final decision in that loop. After the final Transformer layer and language-model head, we have one logit per vocabulary token. Which token do we actually put into the answer? ...

November 21, 2026 · 16 min

Lab 1.4: Weight Sharing — From Position-Anchored Parts to Convolutional Features

Lab 1.4: Weight Sharing — From Position-Anchored Parts to Convolutional Features MNIST internals series. Lab 1.1 gave us class templates. Lab 1.2 watched them reorganize. Lab 1.3 added a hidden layer and the templates dissolved into shared, position-anchored stroke fragments. That last phrase is the loose thread. A fully connected hidden unit has a different weight for every pixel, so detecting the same short stroke at two locations costs two different sets of weights. Nothing in the architecture says that a useful pattern in the top-left corner should also be useful in the bottom-right. ...

November 21, 2026 · 22 min

Speculative Decoding: Verifying Several Tokens in Parallel

Speculative Decoding: Verifying Several Tokens in Parallel The decoding-strategies note answered which token to choose from a next-token distribution. It did not remove the central inefficiency of generation: every accepted token normally requires another serial step through the large target model. KV caching makes each of those steps cheaper by reusing old keys and values. But the loop is still serial: target model → token 1 → target model → token 2 → target model → token 3 Speculative decoding adds a second, cheaper model called the draft model. The draft proposes several tokens serially. The large target model then examines the whole proposed suffix in one parallel forward pass and decides what to accept. ...

November 21, 2026 · 17 min

KV Cache: Making Autoregressive Generation Practical

KV Cache: Making Autoregressive Generation Practical The language-model training note ended with an asymmetry: training processes every position in parallel, but generation produces one token at a time. Each new token is appended to the context, and the model is run again to predict the next one. If we implement that loop literally, we repeat an enormous amount of work. To generate token 100, we recompute the embeddings, queries, keys, values, attention outputs, and feedforward outputs for tokens 1 through 99 — even though we already computed all of them in earlier iterations. ...

November 10, 2026 · 20 min

Lab 1.3: From Templates to Features — What One Hidden Layer Does to the Weights

Lab 1.3: From Templates to Features — What One Hidden Layer Does to the Weights MNIST internals series. Lab 1.1 gave us the clean answer: in a linear softmax model, a weight row is a matched-filter photograph of its class. Lab 1.2 showed those photographs being painted, re-carved, and out-bid. Now the question this series was really building toward: add one hidden layer. What happens to the weight picture? If you’ve watched 3Blue1Brown’s neural nets series, you might expect: “layers learn edges, then curves, then parts; visualizing raw weights won’t tell you much here.” We can now test that ourselves instead of taking it on faith, and the truth lands in a more interesting place: ...

October 27, 2026 · 37 min

Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute

Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute Language-model internals series. In the MNIST series we took apart what weights represent for vision. New series, same ritual — build it from scratch, check gradients numerically, claim nothing without a baseline or a measured number. The question for this lab, in the plainest possible form: A word becomes a row of numbers. How can those numbers possibly mean anything? ...

October 27, 2026 · 24 min

Lab 2.2: A Language Model Without Attention — RNN, BPTT, and the Measured Death of Gradient Signal

Lab 2.2: A Language Model Without Attention — char-level RNN, backprop through time, and the measured death of gradient signal Language-model internals series. Lab 2.1 gave words geometry without any sequence at all — windows don’t know order. This lab puts order back, the pre-2017 way: a recurrent network that reads one character at a time and carries everything it remembers in one vector. The organizing question of this series, sharpened: ...

October 27, 2026 · 15 min

Lab 2.3: Gates as Memory Engineering — LSTM From Scratch on a Task That Was Supposed to Kill the RNN

Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN Language-model internals series. Lab 2.2 measured the diagnosis instead of reciting it: on a trained char-RNN, the forward echo of a character and the backward BPTT blame both halve every ~2.3 steps (per-step retention 0.74), because the transported term per step is (roughly) $W_{hh}^{\top}\mathrm{diag}(1-h^2)$ — every step multiplies your memory by something whose average magnitude is far below 1. ...

October 27, 2026 · 21 min

Lab 1.1: MNIST From Pixels to a Neural Network — What Do the Weights Actually Represent?

Lab 1.1: What Do the Weights of a Neural Network Actually Represent? MNIST internals series — the practical companion to the ML notes (softmax & cross-entropy, backprop, training a 2-layer network). This time we stop deriving and start looking. Anyone who has trained a model eventually asks: The model is just a pile of numbers. In this lab’s case, exactly 7,860 of them (7,850 weights + 10 biases). What do those numbers actually mean? ...

October 19, 2026 · 20 min

Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize

Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize MNIST internals series. In Lab 1.1 we established what a weight row is: a learned photograph of a class, contrast-carved against its rivals, painted by the update rule $W_c \leftarrow W_c - \eta\,\tfrac1N\sum (\hat p_c - y_c)\,x^\top$ . Now the experiment that motivated the series: start the model in an impoverished world — only 1s and 2s exist — then let it discover 3. This is continual learning in miniature, and it lets us watch, directly and quantitatively: ...

October 19, 2026 · 19 min