CUDA Thread Hierarchy: Grids, Blocks, Warps, and Threads
CUDA Thread Hierarchy: Grids, Blocks, Warps, and Threads The previous note mapped the hardware: SM, warp scheduler, register file, shared memory, L2, device memory. This note maps the software abstraction CUDA exposes on top of that hardware. The two maps fit together almost one-to-one, and once you see how, most of CUDA stops being arbitrary syntax and becomes named hardware concepts. CUDA organizes parallel work into four nested levels: Grid └── Block (threads in a block can share fast on-chip memory, and sync) └── Warp (32 threads, executed in lockstep on the hardware) └── Thread (your kernel code, from one thread's point of view) This note works from the bottom up: start with the thread (the thing you actually program), then warp, then block, then grid, then show how the whole tower maps onto the GPU die. ...