CUDA Reduction: Parallel Sum, Max, and Other Tree Algorithms
CUDA Reduction: Parallel Sum, Max, and Other Tree Algorithms The previous note covered occupancy. This note covers reduction: taking a large array and producing a single value (a sum, max, min, product, or any associative binary operation). Reduction is one of the most important parallel algorithms on a GPU, and it teaches several key CUDA ideas at once: shared memory, thread cooperation, warp-level execution, and multi-kernel launches. This note builds the algorithm step by step, from a naive atomic version to a tree-based shared-memory reduction. ...