Training a 2-Layer Network in NumPy: From Scalar to Vectorized Backprop The previous note did the backward pass for a small 2-layer network by hand, one scalar at a time. That is the best way to understand what backprop actually does. This note takes the next step: turn that scalar walk into compact, vectorized NumPy code. The math is unchanged; the only thing that changes is notation. I will introduce every matrix slowly — what its rows and columns mean, where the division by the batch size comes from, and why it is exactly the same algorithm you already did by hand. ...

Why is there no deadlock in the order of corrections? Why is this cheap enough to do for billions of parameters? What is PyTorch’s autograd doing when you call loss.backward()? 1. What you will learn The shape of a fully-connected (dense) network and what “fully-connected” means. How to forward a single training example through every operation, by hand. The backward pass as a message-passing process, with the exact algebra at each edge. The recursion that lets you go from 2 layers to 100 layers. Why nothing breaks due to ordering — the backward pass computes gradients; it does not apply updates. Why backprop costs about one extra forward pass, not one forward pass per parameter. A pseudocode implementation of the whole algorithm. What an autograd engine records, and how loss.backward() / optimizer.step() / optimizer.zero_grad() map onto what we do by hand. 2. The network we are going to train Logistic regression is a single layer: input → weighted sum → sigmoid → probability. Its decision boundary is a line (or hyperplane). There is a famous class of problems it cannot solve — XOR is the classic example — where no single line separates the two classes. ...

Computational Graphs, Part 2: Branching — Why Gradients Add The previous note covered the forward pass, the chain rule, and the backward pass on a graph where every input had exactly one path to the output. This note adds the one remaining piece: what happens when an input feeds into more than one operation. When that happens, there are multiple paths from the input to the output. The chain rule tells us to add the contributions from those paths. ...

Computational Graphs, Part 3: A Single Neuron and Logistic Regression The previous note showed how gradients add when one input feeds multiple operations. With that in place, we can now look at a real model: a single neuron. We will draw it as a graph, run the forward pass and backward pass by hand, and then connect it back to the logistic regression from the first note. 1. What you will learn How a single neuron is a small computational graph. The forward pass through a weighted sum and an activation function. The backward pass through the same graph. Why logistic regression is exactly a one-neuron network with sigmoid activation. How the cross-entropy loss fits into the graph as an extra node. Why the gradient formula from logistic regression matches the chain-rule result. 2. A single neuron A neuron with two inputs has three steps: ...

Computational Graphs and Backpropagation This note explains how to compute gradients for any function by breaking it into a graph of simple operations. It is the bridge between the gradient-descent picture from the linear and logistic regression note and the layered functions we will later call neural networks. The ideas are: Draw the function as a graph of operations. Evaluate the graph from inputs to output: the forward pass. Use the chain rule to carry sensitivities from the output back to the inputs: the backward pass. We build this on one tiny example and walk through every step. ...

ML Refresher: Linear and Logistic Regression This is the first note in the ML → Deep Learning → Transformers → LLMs series. The goal is to rebuild working memory of the basics before we get to neural networks: what a model is, how a loss function measures error, and how gradient descent tunes parameters. We will implement linear regression and logistic regression from scratch in NumPy, then compare with scikit-learn. If the code and gradients feel obvious, you are ready for the next note (computational graphs and backprop). If not, this is exactly the foundation to lock down first. ...

CUDA Profiling and Performance Measurement The notes up to this point covered what makes CUDA kernels fast in theory: coalescing, tiling, occupancy, and asynchronous execution. This note covers how to measure whether a kernel is actually fast and how to find the bottleneck when it is not. Profiling is what separates guessing from optimization. Without it, you can apply every best practice and still be slower than a simpler implementation. ...

CUDA Unified Memory: One Pointer, Two Processors The previous note covered explicit CUDA streams and how to overlap host-device transfers with computation. This note covers Unified Memory, an alternative to explicit cudaMalloc / cudaMemcpy / cudaFree. It lets both CPU and GPU use a single pointer, at the cost of some performance complexity under the hood. Unified Memory is seductive: it removes the need to think about host and device pointers separately. But it does not remove the physical reality that CPU and GPU memory are separate. Understanding when it helps and when it hurts is essential. ...

CUDA Reduction: Parallel Sum, Max, and Other Tree Algorithms The previous note covered occupancy. This note covers reduction: taking a large array and producing a single value (a sum, max, min, product, or any associative binary operation). Reduction is one of the most important parallel algorithms on a GPU, and it teaches several key CUDA ideas at once: shared memory, thread cooperation, warp-level execution, and multi-kernel launches. This note builds the algorithm step by step, from a naive atomic version to a tree-based shared-memory reduction. ...

CUDA Streams and Host/Device Overlap The previous note covered reduction, the first parallel algorithm on the GPU side. This note moves to the host-device boundary: how to keep the GPU busy while the CPU prepares the next batch of data, and how to overlap data movement with computation. So far, every kernel launch and every cudaMemcpy has been synchronous. The host waits, the GPU works, the host waits again. CUDA streams let you break that pattern. ...