Ml on Sanketh's Blog

Attention: Turning Token Vectors into Context Vectors

Sun, 20 Sep 2026 00:00:00 +0530

Attention: Turning Token Vectors into Context Vectors

The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time.

But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first.

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

Sun, 20 Sep 2026 00:00:00 +0530

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order.

This note turns attention into the actual Transformer block. The block is attention plus four supporting acts — multi-head projection, a residual connection, layer normalization, and a small feedforward network — and then positional encoding bolted onto the input. None of them are exotic. Each one exists to patch a specific failure of the bare mechanism, and by the end you should be able to name the patch for each failure.

Training a Language Model End to End: From Text to Loss to Generation

Sun, 20 Sep 2026 00:00:00 +0530

Training a Language Model End to End: From Text to Loss to Generation

We now have every moving part: token → embedding (the lookup note), tokens mix via masked multi-head attention (previous two notes), blocks stack, and the whole thing is just a computation graph built from matrix multiplies and softmax — the exact family the general $L$ -layer loop handles.

What remains is boring-sounding but is actually the point of the whole series: how does a stack of transformer blocks become a language model that predicts text? The answer has four pieces: a final linear layer to vocabulary-sized logits, cross-entropy per position against the next token, teacher forcing during training, and autoregressive sampling at inference. The pieces are individually simple; seeing them end-to-end is what makes “GPT” stop being magical.

Activation Functions: Why Sigmoid Fades and ReLU Won

Mon, 14 Sep 2026 00:00:00 +0000

Activation Functions: Why Sigmoid Fades and ReLU Won

The previous note finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention:

dZ[l-1] = dA[l-1] * activation_derivative(A[l-1])

Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network’s non-linearity touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way.

Embeddings: From One-Hot Vectors to Learned Representations

Mon, 14 Sep 2026 00:00:00 +0000

Embeddings: From One-Hot Vectors to Learned Representations

Everything in the previous notes assumed the network’s input was already a list of numbers — x1 = 1, x2 = 2, pixel intensities, whatever. But most interesting data is not numeric. “cat”, “dog”, “bank”, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string "cat" by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the embedding layer — turns out to be one of the most consequential ideas in modern deep learning.

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

Mon, 14 Sep 2026 00:00:00 +0000

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities

So far, every classification in these notes has been binary — spam or not, XOR’s 0 or 1 — and the output has been single sigmoid feeding binary cross-entropy, whose gradient collapsed to the beautiful $\delta = a - y$ .

Real classifiers rarely answer two-way questions. “Which of 10 digits is this image?” “Which of 50,000 tokens comes next?” “Is this a cat, a dog, or a bird?” This note generalizes the output of a neural network to $k$ classes, and it turns out almost everything we know carries over — with soft-max doing the job sigmoid did.

Training a 2-Layer Network in NumPy: From Scalar to Vectorized Backprop

Mon, 14 Sep 2026 00:00:00 +0000

Training a 2-Layer Network in NumPy: From Scalar to Vectorized Backprop

The previous note did the backward pass for a small 2-layer network by hand, one scalar at a time. That is the best way to understand what backprop actually does.

This note takes the next step: turn that scalar walk into compact, vectorized NumPy code. The math is unchanged; the only thing that changes is notation. I will introduce every matrix slowly — what its rows and columns mean, where the division by the batch size comes from, and why it is exactly the same algorithm you already did by hand.

Backpropagation in a Fully-Connected Network, From Scratch

Sat, 12 Sep 2026 00:00:00 +0000

Why is there no deadlock in the order of corrections?
Why is this cheap enough to do for billions of parameters?
What is PyTorch’s autograd doing when you call loss.backward()?

1. What you will learn

The shape of a fully-connected (dense) network and what “fully-connected” means.
How to forward a single training example through every operation, by hand.
The backward pass as a message-passing process, with the exact algebra at each edge.
The recursion that lets you go from 2 layers to 100 layers.
Why nothing breaks due to ordering — the backward pass computes gradients; it does not apply updates.
Why backprop costs about one extra forward pass, not one forward pass per parameter.
A pseudocode implementation of the whole algorithm.
What an autograd engine records, and how loss.backward() / optimizer.step() / optimizer.zero_grad() map onto what we do by hand.

2. The network we are going to train

Logistic regression is a single layer: input → weighted sum → sigmoid → probability. Its decision boundary is a line (or hyperplane). There is a famous class of problems it cannot solve — XOR is the classic example — where no single line separates the two classes.

Computational Graphs, Part 2: Branching — Why Gradients Add

Sat, 12 Sep 2026 00:00:00 +0000

Computational Graphs, Part 2: Branching — Why Gradients Add

The previous note covered the forward pass, the chain rule, and the backward pass on a graph where every input had exactly one path to the output. This note adds the one remaining piece: what happens when an input feeds into more than one operation.

When that happens, there are multiple paths from the input to the output. The chain rule tells us to add the contributions from those paths.

Computational Graphs, Part 3: A Single Neuron and Logistic Regression

Sat, 12 Sep 2026 00:00:00 +0000

Computational Graphs, Part 3: A Single Neuron and Logistic Regression

The previous note showed how gradients add when one input feeds multiple operations. With that in place, we can now look at a real model: a single neuron. We will draw it as a graph, run the forward pass and backward pass by hand, and then connect it back to the logistic regression from the first note.

1. What you will learn

How a single neuron is a small computational graph.
The forward pass through a weighted sum and an activation function.
The backward pass through the same graph.
Why logistic regression is exactly a one-neuron network with sigmoid activation.
How the cross-entropy loss fits into the graph as an extra node.
Why the gradient formula from logistic regression matches the chain-rule result.

2. A single neuron

A neuron with two inputs has three steps:

Computational Graphs and Backpropagation

Wed, 09 Sep 2026 00:00:00 +0000

Computational Graphs and Backpropagation

This note explains how to compute gradients for any function by breaking it into a graph of simple operations. It is the bridge between the gradient-descent picture from the linear and logistic regression note and the layered functions we will later call neural networks.

The ideas are:

Draw the function as a graph of operations.
Evaluate the graph from inputs to output: the forward pass.
Use the chain rule to carry sensitivities from the output back to the inputs: the backward pass.

We build this on one tiny example and walk through every step.

ML Refresher: Linear and Logistic Regression

Mon, 07 Sep 2026 00:00:00 +0000

ML Refresher: Linear and Logistic Regression

This is the first note in the ML → Deep Learning → Transformers → LLMs series. The goal is to rebuild working memory of the basics before we get to neural networks: what a model is, how a loss function measures error, and how gradient descent tunes parameters.

We will implement linear regression and logistic regression from scratch in NumPy, then compare with scikit-learn. If the code and gradients feel obvious, you are ready for the next note (computational graphs and backprop). If not, this is exactly the foundation to lock down first.