Activation Functions: Why Sigmoid Fades and ReLU Won
Activation Functions: Why Sigmoid Fades and ReLU Won The previous note finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention: dZ[l-1] = dA[l-1] * activation_derivative(A[l-1]) Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network’s non-linearity touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way. ...