Performance Engineering: CPU Microarchitecture for the Practitioner This note explains what a modern CPU actually does while your program runs, and how to read that behavior through hardware performance counters. The goal is not to memorize every unit in a specific core design, but to build a mental model that makes perf output meaningful. If the previous note asked “where does the time go,” this note answers “what is the CPU doing with the cycles it consumes.” ...

Performance Engineering: A Practical Starting Map This is the first note in a performance-engineering sequence. It is not a summary of an SRE handbook, a cloud-observability guide, or a treatise on dashboard design. The goal is practical skill at the level of a single machine: understand why a program is slow, decide what to change, and verify that the change worked. The contexts that matter here are things like a chess engine, the microbenchmark problems on HFT University, a hot loop in the Linux kernel, or any single-node program where you own the source and the hardware. The operating system is the boundary we press against; the hardware is the ground truth. ...

Lab: Word2Vec’s Static Embedding Problem Word2Vec and GloVe produce one vector per word. That is their strength — compact, reusable — and their fundamental weakness: a word with multiple meanings is forced into a single point in space. This notebook uses tiny synthetic vectors to show the problem before we move to contextual models. import numpy as np def cosine(a, b): return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))) # Hand-designed 4-dim vectors. 'bank' is forced to sit between two meanings. embeddings = { "river": np.array([1.0, 0.0, 0.0, 0.0]), "stream": np.array([0.9, 0.1, 0.0, 0.0]), "money": np.array([0.0, 1.0, 0.0, 0.0]), "cash": np.array([0.1, 0.9, 0.0, 0.0]), "bank": np.array([0.5, 0.5, 0.0, 0.0]), # one static compromise "cat": np.array([0.0, 0.0, 1.0, 0.1]), "dog": np.array([0.0, 0.0, 0.9, 0.2]), } 1. Polysemy: one vector cannot be in two places at once In a real corpus, bank appears near river and near money. Word2Vec averages the pressure and lands somewhere in between. ...

Attention: Turning Token Vectors into Context Vectors The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time. But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first. ...

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order. This note turns attention into the actual Transformer block. The block is attention plus four supporting acts — multi-head projection, a residual connection, layer normalization, and a small feedforward network — and then positional encoding bolted onto the input. None of them are exotic. Each one exists to patch a specific failure of the bare mechanism, and by the end you should be able to name the patch for each failure. ...

Training a Language Model End to End: From Text to Loss to Generation We now have every moving part: token → embedding (the lookup note), tokens mix via masked multi-head attention (previous two notes), blocks stack, and the whole thing is just a computation graph built from matrix multiplies and softmax — the exact family the general $L$ -layer loop handles. What remains is boring-sounding but is actually the point of the whole series: how does a stack of transformer blocks become a language model that predicts text? The answer has four pieces: a final linear layer to vocabulary-sized logits, cross-entropy per position against the next token, teacher forcing during training, and autoregressive sampling at inference. The pieces are individually simple; seeing them end-to-end is what makes “GPT” stop being magical. ...

Activation Functions: Why Sigmoid Fades and ReLU Won The previous note finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention: dZ[l-1] = dA[l-1] * activation_derivative(A[l-1]) Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network’s non-linearity touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way. ...

Embeddings: From One-Hot Vectors to Learned Representations Everything in the previous notes assumed the network’s input was already a list of numbers — x1 = 1, x2 = 2, pixel intensities, whatever. But most interesting data is not numeric. “cat”, “dog”, “bank”, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string "cat" by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the embedding layer — turns out to be one of the most consequential ideas in modern deep learning. ...

Softmax and Multiclass Cross-Entropy: Turning Raw Scores Into Probabilities So far, every classification in these notes has been binary — spam or not, XOR’s 0 or 1 — and the output has been single sigmoid feeding binary cross-entropy, whose gradient collapsed to the beautiful $\delta = a - y$ . Real classifiers rarely answer two-way questions. “Which of 10 digits is this image?” “Which of 50,000 tokens comes next?” “Is this a cat, a dog, or a bird?” This note generalizes the output of a neural network to $k$ classes, and it turns out almost everything we know carries over — with soft-max doing the job sigmoid did. ...