Computational-Graphs

Computational Graphs, Part 2: Branching — Why Gradients Add The previous note covered the forward pass, the chain rule, and the backward pass on a graph where every input had exactly one path to the output. This note adds the one remaining piece: what happens when an input feeds into more than one operation. When that happens, there are multiple paths from the input to the output. The chain rule tells us to add the contributions from those paths. ...

Computational Graphs, Part 3: A Single Neuron and Logistic Regression The previous note showed how gradients add when one input feeds multiple operations. With that in place, we can now look at a real model: a single neuron. We will draw it as a graph, run the forward pass and backward pass by hand, and then connect it back to the logistic regression from the first note. 1. What you will learn How a single neuron is a small computational graph. The forward pass through a weighted sum and an activation function. The backward pass through the same graph. Why logistic regression is exactly a one-neuron network with sigmoid activation. How the cross-entropy loss fits into the graph as an extra node. Why the gradient formula from logistic regression matches the chain-rule result. 2. A single neuron A neuron with two inputs has three steps: ...

Computational Graphs and Backpropagation This note explains how to compute gradients for any function by breaking it into a graph of simple operations. It is the bridge between the gradient-descent picture from the linear and logistic regression note and the layered functions we will later call neural networks. The ideas are: Draw the function as a graph of operations. Evaluate the graph from inputs to output: the forward pass. Use the chain rule to carry sensitivities from the output back to the inputs: the backward pass. We build this on one tiny example and walk through every step. ...