Arithmetic Intensity, Simply Explained
A no-jargon breakdown of Arithmetic Intensity and the Roofline model, and how they are used to optimize LLM inference.
Arithmetic Intensity is a concept that will instantly help you understand what the performance bottlenecks in LLM training and serving are. Unfortunately, it is not discussed as much as it should be. So here is a no-jargon lesson to fix that.
What is Arithmetic Intensity?
Arithmetic Intensity (AI) is the ratio of arithmetic operations/computations performed to the data moved to and from memory.
It indicates how much computation is performed per unit of data transferred from memory. If a workload/kernel performs very few arithmetic operations but requires massive data reading and writing from memory, it has low arithmetic intensity, and vice versa.
Note that:
FLOP is a floating-point operation (a single arithmetic operation on floating-point numbers).
FLOPs is the plural form of FLOP or the total number of floating-point operations.
FLOPS is floating-point operations per second.
How is it used in the Roofline model?
In a GPU, HBM is the slowest but the largest, while registers are extremely fast but the smallest units of memory.
