In this lesson, you’ll learn about:
The CUDA programming model (Grids, Blocks, Warps, and Threads)
The hardware components of a GPU
How a program runs on a GPU
How threads, warps, and blocks map to GPU hardware and which memory each one uses
GPUs are powerful computing devices designed to perform massive numbers of parallel computations.
They contain thousands of cores that are individually slower than CPU cores but together deliver massive throughput for parallel computations, such as the matrix and tensor operations essential to today's LLM training and inference.
This lesson will help you better understand GPUs.
Before we move forward, I want to introduce you to my book, ‘LLMs In 100 Images’.
It is a collection of 100 easy-to-follow visuals that describe the most important concepts you need to master to understand LLMs today.
I’m offering a limited-time 30% discount on the book, which you can claim using the button below.
Understanding the CUDA programming model
CUDA is NVIDIA’s parallel computing platform and programming model for writing programs that run on NVIDIA GPUs.
In CUDA terminology, the CPU is called the host, and its memory is called the host memory. Similarly, the GPU is called the device, and its memory is called the device memory.
A CUDA program has two parts:
Host code that runs on the CPU (the host)
Device code that runs on the GPU (the device). Kernels are functions in the device code that the CPU launches to run in parallel on the GPU. Starting a kernel is called “launching” the kernel.
When the CPU (host) launches a kernel, it creates thousands to millions of threads. A Thread is the smallest unit of program execution.
All threads created by a kernel launch are called a Grid, and each grid contains one or more equally-sized Blocks (also called Thread Blocks). Blocks are groups of threads in a grid that can cooperate and share the GPU's fast on-chip shared memory.
Each block is further divided into Warps. A warp is the GPU's actual unit of execution and scheduling.
A warp contains 32 threads and functions based on the SIMT (Single-Instruction, Multiple-Thread) model. This means that all threads in a warp run the same code/instruction at the same time.
However, each thread may follow different branches of the code and does not need to follow the same execution path.
Understanding GPU hardware
The computational unit of a GPU is called a Streaming Multiprocessor (SM).
The main components of an SM are:
CUDA cores (INT32, FP32, FP64): For basic arithmetic calculations
Tensor Cores: For matrix multiplication calculations
Warp schedulers and Dispatch units: For choosing which warps to run next
Load/ Store (LD/ST) units: For moving data between memory and registers
Registers/ Register files: For storing each thread’s data temporarily
L1 cache: For temporarily storing data to be used across the components of an SM
Shared memory: Partition of the same SRAM as the L1 cache that can be used to store data. While L1 caches data automatically, shared memory can be used programmatically.
Alongside this, in an SM of a GPU like the H100, one can find many other components such as:
SFUs (Special Function Units): For performing exponential, logarithmic, and trigonometric functions.
Texture (Tex) units: For processing surface texture data efficiently
Tensor Memory Accelerator: For efficiently moving multi-dimensional data between global memory and shared memory
L0 and L1 instruction cache: Small and fast cache memory that stores frequently used instructions close to the compute cores
How are instructions executed on a GPU
When a program runs, the operating system loads its executable from the file system into CPU memory (RAM). This executable contains the code to be run on the CPU (host code) and the GPU (device code with kernels).
The CPU runs the host code. It copies the device code (using the CUDA driver) and input data to GPU memory, then launches the kernel.
The kernel launch creates a grid of one or more blocks, and each block is assigned to an SM.
The SM divides each block into warps of 32 threads.
A warp scheduler picks a warp that is ready to run and issues its instructions, which all 32 threads in the warp execute together.
The tensor and CUDA cores perform the arithmetic operations for a warp, and the load/store units handle memory access.
When the kernel finishes, the results are copied back from GPU memory to CPU memory. These can then be used for further computations or saved to the file system.
The GPU memory hierarchy
The GPU memory comes in different sizes and speeds of data access. Arranged in ascending order of speed and descending order of capacity, this is as follows:
Global memory/ HBM/ VRAM
L2 cache
L1 cache and Shared memory
Registers
HBM is called off-chip memory because it is mounted alongside the GPU die and works with the other memory components (L2, L1/ shared memory, and registers). These other memory components are called on-chip memory because they are etched directly on the die.
Each level of the CUDA execution hierarchy uses its own memory.
The grid uses the GPU’s HBM and the L2 cache.
Different blocks in a grid can be allocated to different SMs and share data through HBM and the L2 cache.
On Hopper and newer GPUs (e.g., H100, B200), blocks can also be grouped into a thread block cluster, within which blocks can directly access each other’s shared memory using distributed shared memory.
A block is assigned to an SM, and its threads share data using the SM’s shared memory.
Each thread has its own registers, and register data is private to each thread.
Further Reading
Join the paid tier today to get access to all posts in this newsletter, including:
and so many more!