https://sankethbk.github.io/blog/tags/relu/Recent content in Relu on Sanketh's BlogHugo -- 0.166.0en-usMon, 14 Sep 2026 00:00:00 +0000https://sankethbk.github.io/blog/posts/ml/2026-09-14-2-activation-functions-sigmoid-relu/Mon, 14 Sep 2026 00:00:00 +0000https://sankethbk.github.io/blog/posts/ml/2026-09-14-2-activation-functions-sigmoid-relu/<h1 id="activation-functions-why-sigmoid-fades-and-relu-won">Activation Functions: Why Sigmoid Fades and ReLU Won</h1> <p>The <a href="https://sankethbk.github.io/blog/posts/ml/2026-09-14-1-training-a-2-layer-network-in-numpy">previous note</a> finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention:</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>dZ[l-1] = dA[l-1] * activation_derivative(A[l-1]) </span></span></code></pre></div><p>Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network&rsquo;s <em>non-linearity</em> touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way.</p>