https://sankethbk.github.io/blog/tags/autograd/Recent content in Autograd on Sanketh's BlogHugo -- 0.166.0en-usSat, 12 Sep 2026 00:00:00 +0000https://sankethbk.github.io/blog/posts/ml/2026-09-12-3-backpropagation-in-a-fully-connected-network/Sat, 12 Sep 2026 00:00:00 +0000https://sankethbk.github.io/blog/posts/ml/2026-09-12-3-backpropagation-in-a-fully-connected-network/<ul> <li>Why is there no deadlock in the order of corrections?</li> <li>Why is this cheap enough to do for billions of parameters?</li> <li>What is PyTorch&rsquo;s autograd doing when you call <code>loss.backward()</code>?</li> </ul> <hr> <h2 id="1-what-you-will-learn">1. What you will learn</h2> <ul> <li>The shape of a fully-connected (dense) network and what &ldquo;fully-connected&rdquo; means.</li> <li>How to forward a single training example through every operation, by hand.</li> <li>The backward pass as a message-passing process, with the exact algebra at each edge.</li> <li>The recursion that lets you go from 2 layers to 100 layers.</li> <li>Why nothing breaks due to ordering — the backward pass <em>computes</em> gradients; it does not <em>apply</em> updates.</li> <li>Why backprop costs about one extra forward pass, not one forward pass per parameter.</li> <li>A pseudocode implementation of the whole algorithm.</li> <li>What an autograd engine records, and how <code>loss.backward()</code> / <code>optimizer.step()</code> / <code>optimizer.zero_grad()</code> map onto what we do by hand.</li> </ul> <hr> <h2 id="2-the-network-we-are-going-to-train">2. The network we are going to train</h2> <p>Logistic regression is a single layer: input → weighted sum → sigmoid → probability. Its decision boundary is a line (or hyperplane). There is a famous class of problems it cannot solve — XOR is the classic example — where no single line separates the two classes.</p>