SM, CUDA Cores, and GPU Architecture

SM, CUDA Cores, and GPU Architecture

NVIDIA's GPU architecture is built around a hierarchical structure. The smallest unit is the CUDA core, which is a simple arithmetic unit that can do one floating point or integer operation per clock cycle. CUDA cores are grouped into larger units called Streaming Multiprocessors, or SMs. Each SM contains a fixed number of CUDA cores, along with shared memory, registers, and scheduling logic.

The number of CUDA cores per SM has increased with each generation. In the Turing architecture, each SM had 64 CUDA cores. In Ampere, it went up to 128. In Ada Lovelace, each SM has 128 CUDA cores, but the architecture is more efficient so each core does more work per clock. A high-end RTX 4090 has 128 SMs for a total of 16,384 CUDA cores.

CUDA cores are not all identical. Modern NVIDIA GPUs have separate cores for integer and floating point math, and they can execute both types of instructions simultaneously. This is called FP32 and INT32 concurrency, and it improves utilization. There are also special function units for operations like square roots and trigonometric functions, and tensor cores for matrix operations.

The SM also contains a scheduler that decides which threads to run next. GPUs use massive parallelism, running thousands of threads simultaneously. The scheduler groups threads into warps of 32 threads each. All 32 threads in a warp execute the same instruction, but on different data. If threads in a warp take different code paths, some have to wait, which reduces efficiency.

Memory hierarchy is critical in GPU architecture. Each SM has its own register file and shared memory, which is extremely fast but limited in size. All SMs share a larger L2 cache, and beyond that is the global VRAM. Understanding this hierarchy is essential for writing efficient GPU code. Data that stays in registers or shared memory is hundreds of times faster to access than data in global memory.