Lab 1.4: Weight Sharing — From Position-Anchored Parts to Convolutional Features

Lab 1.4: Weight Sharing — From Position-Anchored Parts to Convolutional Features MNIST internals series. Lab 1.1 gave us class templates. Lab 1.2 watched them reorganize. Lab 1.3 added a hidden layer and the templates dissolved into shared, position-anchored stroke fragments. That last phrase is the loose thread. A fully connected hidden unit has a different weight for every pixel, so detecting the same short stroke at two locations costs two different sets of weights. Nothing in the architecture says that a useful pattern in the top-left corner should also be useful in the bottom-right. ...

November 21, 2026 · 22 min

Embeddings: From One-Hot Vectors to Learned Representations

Embeddings: From One-Hot Vectors to Learned Representations Everything in the previous notes assumed the network’s input was already a list of numbers — x1 = 1, x2 = 2, pixel intensities, whatever. But most interesting data is not numeric. “cat”, “dog”, “bank”, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string "cat" by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the embedding layer — turns out to be one of the most consequential ideas in modern deep learning. ...

September 14, 2026 · 31 min