How to compute the peak performance of GPU tensor cores?

Not sure if this is the right place to ask, but I am trying to understand the reported theoretical performance of tensor cores on NVIDIA GPUs.

Taking the H100 SXM5 as example, the reported peak performance for non-tensor cores is quite straightforward: the number of CUDA cores multiplied with the clock speed gives the number of MACs and doubling this figure gives the reported FLOP count. In our example: $16\,896 \,\text{cores} * 1\,980 \,\text{MHz} * 2 \,\frac{\text{FLOPs}}{\text{core}} = 66.908 \,\frac{\text{TFLOPs}}{\text{s}}$

Now, tensor cores enable matrix multiplications of two fixed sized matrices (e.g. a 16x8 with a 8x16 matrix for FP32 according to the CUDA docs) with a single operation. Using this example, this means this operation corresponds to $2 * 16 * 16 * 8 = 4\,096\,\text{FLOPs}$. Multiplying the number of tensor cores with the tensor core clock speed and this multiplier, we arrive at a theoretical peak performance of $528\,\text{cores} * 1\,830\,\text{MHz} * 4\,096 \,\frac{\text{FLOPs}}{\text{core}} = 3\,957.719 \,\frac{\text{TFLOPs}}{\text{s}},$ which is about 8 times more than what is reported.

One possible explanation for this factor 8 could be that it actually requires 8 tensor cores for one operation. Alternatively, it could be that each tensor core requires 8 cycles to finish the computations. I have been trying to find more information on tensor cores and how they work, but couldn't find anything that could explain where this factor 8 comes from.

This leaves me with the question(s): How to compute the peak performance of tensor cores? Is there some documentation (that I missed) explaining how tensor cores approximately work?

posted 2 months ago

CC BY-SA 4.0

mr Tsjolder‭

594 reputation 7 19 68 6