Parallel-Computing

CUDA Programming Model: Writing and Launching Kernels The previous note covered the hierarchy: thread, warp, block, grid, and how those map to SMs. This note turns that hierarchy into actual code. By the end, you will have seen a complete CUDA program, understood every line, and know how a host program hands work to the GPU. CUDA is an extension of C/C++. Most of the code you write is ordinary C++. A small number of CUDA-specific pieces — kernel functions, the launch syntax, a few memory APIs, and built-in thread indices — turn a sequential program into a massively parallel one. ...

CUDA Thread Hierarchy: Grids, Blocks, Warps, and Threads The previous note mapped the hardware: SM, warp scheduler, register file, shared memory, L2, device memory. This note maps the software abstraction CUDA exposes on top of that hardware. The two maps fit together almost one-to-one, and once you see how, most of CUDA stops being arbitrary syntax and becomes named hardware concepts. CUDA organizes parallel work into four nested levels: Grid └── Block (threads in a block can share fast on-chip memory, and sync) └── Warp (32 threads, executed in lockstep on the hardware) └── Thread (your kernel code, from one thread's point of view) This note works from the bottom up: start with the thread (the thing you actually program), then warp, then block, then grid, then show how the whole tower maps onto the GPU die. ...

CPU vs GPU Architecture Why GPUs exist For decades, single-thread CPU performance improved “for free” — you write the same sequential code, and it runs faster on the next generation of chips, because clock speeds kept climbing. Around the mid-2000s, that stopped. Clock speeds hit a power wall — you can’t keep cranking frequency without the chip melting. So the industry pivoted from “make one core faster” to “put more cores on the chip.” This is the multicore/manycore shift, and it’s the reason parallel programming stopped being a niche HPC skill and became something every programmer eventually runs into. ...