A beginner's mental model
A GPU is a factory of tiny parallel workers.
Work arrives from the host, gets split into blocks, runs inside Streaming Multiprocessors, and is constantly fed by a memory hierarchy built for bandwidth.
The whole GPU die
Many SMs share one cache and many memory channels.
control & compute
memory
data movement
▣
CPU / host memorylaunches kernels • owns the program
GPU silicon
GigaThread Engineassigns blocks to SMs
PCIe / NVLinkhost ↔ device bridge
Memory controllerspeaks GDDR / HBM
GPCs — clusters that tile the chip
SM 0many warps in flight
SM 1many warps in flight
SM 2many warps in flight
SM 3many warps in flight
SM 4many warps in flight
SM 5many warps in flight
SM 6many warps in flight
SM 7many warps in flight
SM 8many warps in flight
L2 cache · chip-widethe common staging area between all SMs and device memory
Memory controllerparallel channel
Memory controllerparallel channel
HBM / GDDRlarge, bandwidth-first device memory
The GigaThread Engine chooses where a block goes. Inside each SM, warp schedulers choose what runs next.
Zoom in: one Streaming Multiprocessor
A block lives on one SM. The SM keeps several warps ready so one can run while another waits on memory.
Warp schedulerpicks a ready warp
Dispatch unitsends the instruction
Instruction cachekeeps code close
Warps: 32 threads in lockstep
One instruction • many lanes • same moment
CUDA coresFP32 / INT32 arithmetic
Tensor coresmatrix multiply-accumulate
Load / Storemove data in and out
SFUsin, exp, reciprocal…
Register fileprivate per-thread working space; using more registers can reduce how many warps fit
Shared memory + L1 cachefast per-SM SRAM — partly automatic, partly programmer-managed scratchpad
The memory ladder
Closer to the arithmetic usually means smaller and faster. The farther down you go, the more bandwidth and latency matter.
inside the SM · fastest
Registersprivate to each thread
Shared memory / L1shared by a block / cached automatically
chip and board · biggest
L2 cacheshared by every SM
Device memoryGDDR or HBM
PCIe / NVLink → CPU memorycrossing this boundary is expensive
Remember this:
More threads do not make one thread faster. They give the schedulers enough spare work to hide the waiting.