Lab 1.3: From Templates to Features — What One Hidden Layer Does to the Weights MNIST internals series. Lab 1.1 gave us the clean answer: in a linear softmax model, a weight row is a matched-filter photograph of its class. Lab 1.2 showed those photographs being painted, re-carved, and out-bid. Now the question this series was really building toward: add one hidden layer. What happens to the weight picture? If you’ve watched 3Blue1Brown’s neural nets series, you might expect: “layers learn edges, then curves, then parts; visualizing raw weights won’t tell you much here.” We can now test that ourselves instead of taking it on faith, and the truth lands in a more interesting place: ...
Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute
Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute Language-model internals series. In the MNIST series we took apart what weights represent for vision. New series, same ritual — build it from scratch, check gradients numerically, claim nothing without a baseline or a measured number. The question for this lab, in the plainest possible form: A word becomes a row of numbers. How can those numbers possibly mean anything? ...
Lab 2.2: A Language Model Without Attention — RNN, BPTT, and the Measured Death of Gradient Signal
Lab 2.2: A Language Model Without Attention — char-level RNN, backprop through time, and the measured death of gradient signal Language-model internals series. Lab 2.1 gave words geometry without any sequence at all — windows don’t know order. This lab puts order back, the pre-2017 way: a recurrent network that reads one character at a time and carries everything it remembers in one vector. The organizing question of this series, sharpened: ...
Lab 2.3: Gates as Memory Engineering — LSTM From Scratch on a Task That Was Supposed to Kill the RNN
Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN Language-model internals series. Lab 2.2 measured the diagnosis instead of reciting it: on a trained char-RNN, the forward echo of a character and the backward BPTT blame both halve every ~2.3 steps (per-step retention 0.74), because the transported term per step is (roughly) $W_{hh}^{\top}\mathrm{diag}(1-h^2)$ — every step multiplies your memory by something whose average magnitude is far below 1. ...
Lab 1.1: MNIST From Pixels to a Neural Network — What Do the Weights Actually Represent?
Lab 1.1: What Do the Weights of a Neural Network Actually Represent? MNIST internals series — the practical companion to the ML notes (softmax & cross-entropy, backprop, training a 2-layer network). This time we stop deriving and start looking. Anyone who has trained a model eventually asks: The model is just a pile of numbers. In this lab’s case, exactly 7,860 of them (7,850 weights + 10 biases). What do those numbers actually mean? ...
Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize
Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize MNIST internals series. In Lab 1.1 we established what a weight row is: a learned photograph of a class, contrast-carved against its rivals, painted by the update rule $W_c \leftarrow W_c - \eta\,\tfrac1N\sum (\hat p_c - y_c)\,x^\top$ . Now the experiment that motivated the series: start the model in an impoverished world — only 1s and 2s exist — then let it discover 3. This is continual learning in miniature, and it lets us watch, directly and quantitatively: ...
Lab 2: Memory Hierarchy and Caches
Lab 2: Memory Hierarchy and Caches This is the hands-on report for the lab in /home/sanketh/mem-lab on the Linux x86 laptop. It is the practical companion to content/posts/performance-engineering/2026-10-11-memory-and-caches.markdown — every experiment measures something the theory note claims, and a few go past it. Same machine caveat as Lab 1: this is an Intel Core i5-8250U (15 W mobile part, base 1.6 GHz, max 3.4 GHz). It throttles under sustained load, so chase ratios, not absolute nanoseconds. All single-threaded timings below are pinned to one core with taskset -c 2. ...
Performance Engineering: Memory Hierarchy, Caches, and Data Layout
Performance Engineering: Memory Hierarchy, Caches, and Data Layout This note is the natural follow-up to the CPU microarchitecture note. There we saw that the CPU retires instructions as fast as it can and stalls when it cannot overlap work. For most programs, the dominant reason it cannot overlap work is memory: data is not where the CPU needs it, when the CPU needs it. The goal here is to understand the cache hierarchy as a programmer sees it, how data layout controls cache behavior, and how to measure that behavior with the right tools. ...
Lab 1: CPU Internals
Lab 1: CPU Internals This is a hands-on report for the lab in /home/sanketh/cpu-lab on your Linux x86 laptop. It ties each experiment back to the ideas in content/posts/performance-engineering/2026-10-04-cpu-microarchitecture.markdown. Important note about this laptop: the CPU is an Intel Core i5-8250U (15 W mobile part, base 1.6 GHz, max 3.4 GHz). Under sustained load it thermally throttles, so absolute nanoseconds vary between runs. The conclusions come from ratios and TMA buckets, not from chasing the exact ns. ...
Performance Engineering: CPU Microarchitecture for the Practitioner
Performance Engineering: CPU Microarchitecture for the Practitioner This note explains what a modern CPU actually does while your program runs, and how to read that behavior through hardware performance counters. The goal is not to memorize every unit in a specific core design, but to build a mental model that makes perf output meaningful. If the previous note asked “where does the time go,” this note answers “what is the CPU doing with the cycles it consumes.” ...