Lab 2: Memory Hierarchy and Caches

Lab 2: Memory Hierarchy and Caches This is the hands-on report for the lab in /home/sanketh/mem-lab on the Linux x86 laptop. It is the practical companion to content/posts/performance-engineering/2026-10-11-memory-and-caches.markdown — every experiment measures something the theory note claims, and a few go past it. Same machine caveat as Lab 1: this is an Intel Core i5-8250U (15 W mobile part, base 1.6 GHz, max 3.4 GHz). It throttles under sustained load, so chase ratios, not absolute nanoseconds. All single-threaded timings below are pinned to one core with taskset -c 2. ...

October 17, 2026 · 39 min

Performance Engineering: Memory Hierarchy, Caches, and Data Layout

Performance Engineering: Memory Hierarchy, Caches, and Data Layout This note is the natural follow-up to the CPU microarchitecture note. There we saw that the CPU retires instructions as fast as it can and stalls when it cannot overlap work. For most programs, the dominant reason it cannot overlap work is memory: data is not where the CPU needs it, when the CPU needs it. The goal here is to understand the cache hierarchy as a programmer sees it, how data layout controls cache behavior, and how to measure that behavior with the right tools. ...

October 11, 2026 · 18 min

Lab 1: CPU Internals

Lab 1: CPU Internals This is a hands-on report for the lab in /home/sanketh/cpu-lab on your Linux x86 laptop. It ties each experiment back to the ideas in content/posts/performance-engineering/2026-10-04-cpu-microarchitecture.markdown. Important note about this laptop: the CPU is an Intel Core i5-8250U (15 W mobile part, base 1.6 GHz, max 3.4 GHz). Under sustained load it thermally throttles, so absolute nanoseconds vary between runs. The conclusions come from ratios and TMA buckets, not from chasing the exact ns. ...

October 6, 2026 · 33 min

Performance Engineering: CPU Microarchitecture for the Practitioner

Performance Engineering: CPU Microarchitecture for the Practitioner This note explains what a modern CPU actually does while your program runs, and how to read that behavior through hardware performance counters. The goal is not to memorize every unit in a specific core design, but to build a mental model that makes perf output meaningful. If the previous note asked “where does the time go,” this note answers “what is the CPU doing with the cycles it consumes.” ...

October 4, 2026 · 25 min

Performance Engineering: A Practical Starting Map

Performance Engineering: A Practical Starting Map This is the first note in a performance-engineering sequence. It is not a summary of an SRE handbook, a cloud-observability guide, or a treatise on dashboard design. The goal is practical skill at the level of a single machine: understand why a program is slow, decide what to change, and verify that the change worked. The contexts that matter here are things like a chess engine, the microbenchmark problems on HFT University, a hot loop in the Linux kernel, or any single-node program where you own the source and the hardware. The operating system is the boundary we press against; the hardware is the ground truth. ...

October 3, 2026 · 12 min

Attention: Turning Token Vectors into Context Vectors

Attention: Turning Token Vectors into Context Vectors The embeddings note ended with an honest limitation: a token’s embedding is one fixed row of a learned matrix. The word “bank” gets the same vector in “river bank” and “investment bank”. Everything the model can possibly know about “bank itself” is frozen into that row at training time. But meaning is contextual. When a model processes a sentence, what it needs at the position of “bank” is not “the generic bank vector” but “the vector of bank as it appears in this sentence”. Attention is the mechanism that builds that second thing from the first. ...

September 20, 2026 · 24 min

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position

The Transformer Block: Multi-Head Attention, Residuals, Norms, and Position The attention note derived the single mechanism — softmax over query-key scores, times values — and ended with its two obvious weaknesses: one head can track only one relevance pattern at a time, and the whole operation is blind to token order. Attention is blind to token order This was highlighted in previous note here 4. Sets, not sequences. Nothing in the formula knows which token came first. Permute the input rows of $X$ and the outputs permute identically — attention is permutation equivariant: it treats the sequence as a bag of tokens with affinities. That’s great for set-like inputs and a true bug for language, whose meaning is bound up in order (“dog bites man” ≠ “man bites dog”). Fixing that is positional encoding, coming in the next note. One attention head can learn only relevance pattern at a time This idea was only mentioned at the end of the previous note. It was never justified or derived. So let’s try to make sense of it. Short answer: The answer lies in what happens when we compute $QK^\top$ . From 3b1b’s video if this token is useful to another token, what information should it send? If we think about it, a token can be useful to the next token in multiple different ways—for a given token A, its predecessor token B might be more useful when considering grammatical context, and token C might be more useful when considering semantic context. We are forcing one attention head to somehow capture all those relationships and turn them into one set of weights ( $W_Q$ , $W_K$ , and $W_V$ ). Even if it somehow ends up capturing these different spatial relationships, at best they fight with each other and average them out. (Technically, $W_V$ (the Value weights) doesn’t decide who to look at; it only decides what information to extract once the connection is made. The “fighting and averaging out” happens specifically inside $W_Q$ and $W_K$ , which calculate the attention map.) Long answer: One attention head can only learn one relevance pattern at a time because the query-key interaction is governed by a single pair of linear projection matrices ( $W_Q$ and $W_K$ ). When we compute $QK^\top$ , we perform a single matrix multiplication that collapses the high-dimensional token representations into one scalar affinity score for every pair of tokens. Mathematically and conceptually, here is why this limits a single head to one “type” of similarity: The Rank and Geometry of the Attention Matrix The similarity matrix before softmax is computed as: ...

September 20, 2026 · 35 min

Training a Language Model End to End: From Text to Loss to Generation

Training a Language Model End to End: From Text to Loss to Generation We now have every moving part: token → embedding (the lookup note), tokens mix via masked multi-head attention (previous two notes), blocks stack, and the whole thing is just a computation graph built from matrix multiplies and softmax — the exact family the general $L$ -layer loop handles. What remains is boring-sounding but is actually the point of the whole series: how does a stack of transformer blocks become a language model that predicts text? The answer has four pieces: a final linear layer to vocabulary-sized logits, cross-entropy per position against the next token, teacher forcing during training, and autoregressive sampling at inference. The pieces are individually simple; seeing them end-to-end is what makes “GPT” stop being magical. ...

September 20, 2026 · 15 min

Activation Functions: Why Sigmoid Fades and ReLU Won

Activation Functions: Why Sigmoid Fades and ReLU Won The previous note finished with pseudocode for a network of any depth $L$ . One line in that pseudocode deserved more attention: dZ[l-1] = dA[l-1] * activation_derivative(A[l-1]) Everything else in the backward pass is matrix multiplication — copying, scaling, and adding error signals. This one line is different: it is the only place where the network’s non-linearity touches the gradients. The choice of activation function decides whether error signals survive the trip from output back to input, or die on the way. ...

September 14, 2026 · 18 min

Embeddings: From One-Hot Vectors to Learned Representations

Embeddings: From One-Hot Vectors to Learned Representations Everything in the previous notes assumed the network’s input was already a list of numbers — x1 = 1, x2 = 2, pixel intensities, whatever. But most interesting data is not numeric. “cat”, “dog”, “bank”, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string "cat" by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the embedding layer — turns out to be one of the most consequential ideas in modern deep learning. ...

September 14, 2026 · 31 min