Ml

Lab 1.3: From Templates to Features — What One Hidden Layer Does to the Weights MNIST internals series. Lab 1.1 gave us the clean answer: in a linear softmax model, a weight row is a matched-filter photograph of its class. Lab 1.2 showed those photographs being painted, re-carved, and out-bid. Now the question this series was really building toward: add one hidden layer. What happens to the weight picture? If you’ve watched 3Blue1Brown’s neural nets series, you might expect: “layers learn edges, then curves, then parts; visualizing raw weights won’t tell you much here.” We can now test that ourselves instead of taking it on faith, and the truth lands in a more interesting place: ...

Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute Language-model internals series. In the MNIST series we took apart what weights represent for vision. New series, same ritual — build it from scratch, check gradients numerically, claim nothing without a baseline or a measured number. The question for this lab, in the plainest possible form: A word becomes a row of numbers. How can those numbers possibly mean anything? ...

Lab 2.2: A Language Model Without Attention — char-level RNN, backprop through time, and the measured death of gradient signal Language-model internals series. Lab 2.1 gave words geometry without any sequence at all — windows don’t know order. This lab puts order back, the pre-2017 way: a recurrent network that reads one character at a time and carries everything it remembers in one vector. The organizing question of this series, sharpened: ...

Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN Language-model internals series. Lab 2.2 measured the diagnosis instead of reciting it: on a trained char-RNN, the forward echo of a character and the backward BPTT blame both halve every ~2.3 steps (per-step retention 0.74), because the transported term per step is (roughly) $W_{hh}^{\top}\mathrm{diag}(1-h^2)$ — every step multiplies your memory by something whose average magnitude is far below 1. ...

Lab 1.1: What Do the Weights of a Neural Network Actually Represent? MNIST internals series — the practical companion to the ML notes (softmax & cross-entropy, backprop, training a 2-layer network). This time we stop deriving and start looking. Anyone who has trained a model eventually asks: The model is just a pile of numbers. In this lab’s case, exactly 7,860 of them (7,850 weights + 10 biases). What do those numbers actually mean? ...

Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize MNIST internals series. In Lab 1.1 we established what a weight row is: a learned photograph of a class, contrast-carved against its rivals, painted by the update rule $W_c \leftarrow W_c - \eta\,\tfrac1N\sum (\hat p_c - y_c)\,x^\top$ . Now the experiment that motivated the series: start the model in an impoverished world — only 1s and 2s exist — then let it discover 3. This is continual learning in miniature, and it lets us watch, directly and quantitatively: ...

Attention: Turning Token Vectors into Context Vectors The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time. But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first. ...

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order. This note turns attention into the actual Transformer block. The block is attention plus four supporting acts — multi-head projection, a residual connection, layer normalization, and a small feedforward network — and then positional encoding bolted onto the input. None of them are exotic. Each one exists to patch a specific failure of the bare mechanism, and by the end you should be able to name the patch for each failure. ...

Training a Language Model End to End: From Text to Loss to Generation We now have every moving part: token → embedding (the lookup note), tokens mix via masked multi-head attention (previous two notes), blocks stack, and the whole thing is just a computation graph built from matrix multiplies and softmax — the exact family the general $L$ -layer loop handles. What remains is boring-sounding but is actually the point of the whole series: how does a stack of transformer blocks become a language model that predicts text? The answer has four pieces: a final linear layer to vocabulary-sized logits, cross-entropy per position against the next token, teacher forcing during training, and autoregressive sampling at inference. The pieces are individually simple; seeing them end-to-end is what makes “GPT” stop being magical. ...

Activation Functions: Why Sigmoid Fades and ReLU Won The previous note finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention: dZ[l-1] = dA[l-1] * activation_derivative(A[l-1]) Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network’s non-linearity touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way. ...