Interpretability on Sanketh's Blog

Lab 1.3: From Templates to Features — What One Hidden Layer Does to the Weights

Tue, 27 Oct 2026 00:00:00 +0530

Lab 1.3: From Templates to Features — What One Hidden Layer Does to the Weights

MNIST internals series. Lab 1.1 gave us the clean answer: in a linear softmax model, a weight row is a matched-filter photograph of its class. Lab 1.2 showed those photographs being painted, re-carved, and out-bid.

Now the question this series was really building toward: add one hidden layer. What happens to the weight picture?

If you’ve watched 3Blue1Brown’s neural nets series, you might expect: “layers learn edges, then curves, then parts; visualizing raw weights won’t tell you much here.” We can now test that ourselves instead of taking it on faith, and the truth lands in a more interesting place:

Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute

Tue, 27 Oct 2026 00:00:00 +0530

Lab 2.1: How Word Vectors Learn to Mean — Embeddings From Zero, and What They Secretly Compute

Language-model internals series. In the MNIST series we took apart what weights represent for vision. New series, same ritual — build it from scratch, check gradients numerically, claim nothing without a baseline or a measured number.

The question for this lab, in the plainest possible form:

A word becomes a row of numbers. How can those numbers possibly mean anything?

Lab 1.1: MNIST From Pixels to a Neural Network — What Do the Weights Actually Represent?

Mon, 19 Oct 2026 00:00:00 +0530

Lab 1.1: What Do the Weights of a Neural Network Actually Represent?

MNIST internals series — the practical companion to the ML notes (softmax & cross-entropy, backprop, training a 2-layer network). This time we stop deriving and start looking.

Anyone who has trained a model eventually asks:

The model is just a pile of numbers. In this lab’s case, exactly 7,860 of them (7,850 weights + 10 biases). What do those numbers actually mean?

Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize

Mon, 19 Oct 2026 00:00:00 +0530

Lab 1.2: Teach It 1s and 2s First, Then Introduce 3 — Watching Weights Reorganize

MNIST internals series. In Lab 1.1 we established what a weight row is: a learned photograph of a class, contrast-carved against its rivals, painted by the update rule $W_c \leftarrow W_c - \eta\,\tfrac1N\sum (\hat p_c - y_c)\,x^\top$ .

Now the experiment that motivated the series: start the model in an impoverished world — only 1s and 2s exist — then let it discover 3. This is continual learning in miniature, and it lets us watch, directly and quantitatively: