Backpropagation

Training a 2-Layer Network in NumPy: From Scalar to Vectorized Backprop The previous note did the backward pass for a small 2-layer network by hand, one scalar at a time. That is the best way to understand what backprop actually does. This note takes the next step: turn that scalar walk into compact, vectorized NumPy code. The math is unchanged; the only thing that changes is notation. I will introduce every matrix slowly — what its rows and columns mean, where the division by the batch size comes from, and why it is exactly the same algorithm you already did by hand. ...

Why is there no deadlock in the order of corrections? Why is this cheap enough to do for billions of parameters? What is PyTorch’s autograd doing when you call loss.backward()? 1. What you will learn The shape of a fully-connected (dense) network and what “fully-connected” means. How to forward a single training example through every operation, by hand. The backward pass as a message-passing process, with the exact algebra at each edge. The recursion that lets you go from 2 layers to 100 layers. Why nothing breaks due to ordering — the backward pass computes gradients; it does not apply updates. Why backprop costs about one extra forward pass, not one forward pass per parameter. A pseudocode implementation of the whole algorithm. What an autograd engine records, and how loss.backward() / optimizer.step() / optimizer.zero_grad() map onto what we do by hand. 2. The network we are going to train Logistic regression is a single layer: input → weighted sum → sigmoid → probability. Its decision boundary is a line (or hyperplane). There is a famous class of problems it cannot solve — XOR is the classic example — where no single line separates the two classes. ...

Computational Graphs, Part 2: Branching — Why Gradients Add The previous note covered the forward pass, the chain rule, and the backward pass on a graph where every input had exactly one path to the output. This note adds the one remaining piece: what happens when an input feeds into more than one operation. When that happens, there are multiple paths from the input to the output. The chain rule tells us to add the contributions from those paths. ...

Computational Graphs, Part 3: A Single Neuron and Logistic Regression The previous note showed how gradients add when one input feeds multiple operations. With that in place, we can now look at a real model: a single neuron. We will draw it as a graph, run the forward pass and backward pass by hand, and then connect it back to the logistic regression from the first note. 1. What you will learn How a single neuron is a small computational graph. The forward pass through a weighted sum and an activation function. The backward pass through the same graph. Why logistic regression is exactly a one-neuron network with sigmoid activation. How the cross-entropy loss fits into the graph as an extra node. Why the gradient formula from logistic regression matches the chain-rule result. 2. A single neuron A neuron with two inputs has three steps: ...

Computational Graphs and Backpropagation This note explains how to compute gradients for any function by breaking it into a graph of simple operations. It is the bridge between the gradient-descent picture from the linear and logistic regression note and the layered functions we will later call neural networks. The ideas are: Draw the function as a graph of operations. Evaluate the graph from inputs to output: the forward pass. Use the chain rule to carry sensitivities from the output back to the inputs: the backward pass. We build this on one tiny example and walk through every step. ...