Neural-Networks on Sanketh's Blog

Attention: Turning Token Vectors into Context Vectors

Sun, 20 Sep 2026 00:00:00 +0530

Attention: Turning Token Vectors into Context Vectors

The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time.

But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first.

Activation Functions: Why Sigmoid Fades and ReLU Won

Mon, 14 Sep 2026 00:00:00 +0000

Activation Functions: Why Sigmoid Fades and ReLU Won

The previous note finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention:

dZ[l-1] = dA[l-1] * activation_derivative(A[l-1])

Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network’s non-linearity touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way.

Embeddings: From One-Hot Vectors to Learned Representations

Mon, 14 Sep 2026 00:00:00 +0000

Embeddings: From One-Hot Vectors to Learned Representations

Everything in the previous notes assumed the network’s input was already a list of numbers — x1 = 1, x2 = 2, pixel intensities, whatever. But most interesting data is not numeric. “cat”, “dog”, “bank”, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string "cat" by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the embedding layer — turns out to be one of the most consequential ideas in modern deep learning.

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

Mon, 14 Sep 2026 00:00:00 +0000

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

So far, every classification in these notes has been binary — spam or not, XOR’s 0 or 1 — and the output has been single sigmoid feeding binary cross-entropy, whose gradient collapsed to the beautiful $\delta = a - y$ .

Real classifiers rarely answer two-way questions. “Which of 10 digits is this image?” “Which of 50,000 tokens comes next?” “Is this a cat, a dog, or a bird?” This note generalizes the output of a neural network to $k$ classes, and it turns out almost everything we know carries over — with soft-max doing the job sigmoid did.

Training a 2-Layer Network in NumPy: From Scalar to Vectorized Backprop

Mon, 14 Sep 2026 00:00:00 +0000

Training a 2-Layer Network in NumPy: From Scalar to Vectorized Backprop

The previous note did the backward pass for a small 2-layer network by hand, one scalar at a time. That is the best way to understand what backprop actually does.

This note takes the next step: turn that scalar walk into compact, vectorized NumPy code. The math is unchanged; the only thing that changes is notation. I will introduce every matrix slowly — what its rows and columns mean, where the division by the batch size comes from, and why it is exactly the same algorithm you already did by hand.

Backpropagation in a Fully-Connected Network, From Scratch

Sat, 12 Sep 2026 00:00:00 +0000

Why is there no deadlock in the order of corrections?
Why is this cheap enough to do for billions of parameters?
What is PyTorch’s autograd doing when you call loss.backward()?

1. What you will learn

The shape of a fully-connected (dense) network and what “fully-connected” means.
How to forward a single training example through every operation, by hand.
The backward pass as a message-passing process, with the exact algebra at each edge.
The recursion that lets you go from 2 layers to 100 layers.
Why nothing breaks due to ordering — the backward pass computes gradients; it does not apply updates.
Why backprop costs about one extra forward pass, not one forward pass per parameter.
A pseudocode implementation of the whole algorithm.
What an autograd engine records, and how loss.backward() / optimizer.step() / optimizer.zero_grad() map onto what we do by hand.

2. The network we are going to train

Logistic regression is a single layer: input → weighted sum → sigmoid → probability. Its decision boundary is a line (or hyperplane). There is a famous class of problems it cannot solve — XOR is the classic example — where no single line separates the two classes.