Chain-Rule

Why is there no deadlock in the order of corrections? Why is this cheap enough to do for billions of parameters? What is PyTorch’s autograd doing when you call loss.backward()? 1. What you will learn The shape of a fully-connected (dense) network and what “fully-connected” means. How to forward a single training example through every operation, by hand. The backward pass as a message-passing process, with the exact algebra at each edge. The recursion that lets you go from 2 layers to 100 layers. Why nothing breaks due to ordering — the backward pass computes gradients; it does not apply updates. Why backprop costs about one extra forward pass, not one forward pass per parameter. A pseudocode implementation of the whole algorithm. What an autograd engine records, and how loss.backward() / optimizer.step() / optimizer.zero_grad() map onto what we do by hand. 2. The network we are going to train Logistic regression is a single layer: input → weighted sum → sigmoid → probability. Its decision boundary is a line (or hyperplane). There is a famous class of problems it cannot solve — XOR is the classic example — where no single line separates the two classes. ...

Computational Graphs, Part 2: Branching — Why Gradients Add The previous note covered the forward pass, the chain rule, and the backward pass on a graph where every input had exactly one path to the output. This note adds the one remaining piece: what happens when an input feeds into more than one operation. When that happens, there are multiple paths from the input to the output. The chain rule tells us to add the contributions from those paths. ...

Computational Graphs and Backpropagation This note explains how to compute gradients for any function by breaking it into a graph of simple operations. It is the bridge between the gradient-descent picture from the linear and logistic regression note and the layered functions we will later call neural networks. The ideas are: Draw the function as a graph of operations. Evaluate the graph from inputs to output: the forward pass. Use the chain rule to carry sensitivities from the output back to the inputs: the backward pass. We build this on one tiny example and walk through every step. ...