The AI FLOPS figure on a GPU datasheet is the theoretical peak rate of floating-point operations the chip could run in one numeric format, at its maximum clock, with its matrix units fully busy. It is a ceiling, not the throughput your cluster will deliver. In a 2025 study of the AxoNN framework, large-scale training runs sustained between about 22% and 50% or more of advertised peak, depending on machine and scale. Meta’s Llama 3 report (2024) gives 38-43% model FLOPS utilisation on up to 16,000 H100s. To size a cluster, start from measured utilisation, not from the spec sheet.
Reading a GPU FLOPS Figure: Peak Rate Versus What You Sustain
FLOPS is a rate: floating-point operations per second. FLOP, without the S, is a total count of operations. A training run has a FLOP budget and a GPU has a FLOP rate, so dividing one by the other gives you time.
A deep learning model is effectively a set of matrix multiplications, each made of floating-point multiplies and adds. GPU peak figures count a fused multiply-add as two operations, and Epoch AI follows NVIDIA’s own definition of peak as cores times clock times two. The marketed tensor-core figure is built the same way: tensor cores on the GPU, times maximum clock speed, times FLOP per tensor-core instruction.
That formula rests on three assumptions. The work has to be matrix multiplication running on tensor cores. Without matrix multiplication, an A100 achieves only 19.5 TFLOPS instead of the stated 312. The clock has to stay at its maximum. And the precision has to be the one the figure was quoted in, sometimes with structured sparsity on top. The datasheet value assumes full utilisation and an optimal workload, which Epoch AI calls unrealistic.
What Separates Peak FLOPS from Sustained FLOPS on a GPU Cluster?
Peak FLOPS is the theoretical ceiling; sustained FLOPS is what a real workload actually delivers. The TOP500 list reports both, Rpeak beside Rmax (the maximal LINPACK performance achieved), and ranks machines by Rmax. In June 2016, Sunway TaihuLight topped the list with 93 petaFLOPS on LINPACK against 125 peak petaFLOPS. GPU AI figures deserve the same treatment, and on a cluster the gap opens in two stages.
The Single-GPU Gap: What One Device Can Sustain
Even one large, well-shaped matrix multiplication falls short of peak. The AxoNN authors benchmarked bf16 GEMMs with cuBLAS and rocBLAS. An A100 reached 280 TFLOPS of its 312 (90%), one MI250X GCD reached 125 of 191.5 (65%), and an NVIDIA GH200 guide reports 813 TFLOPS (82%). Google DeepMind’s JAX scaling book says H100s and B200s usually reach only about 80-85% of claimed peak.
Power draw scales roughly with clock speed times the number of transistor flips per clock, and the GPU throttles once it passes its power limit. The Thonking.ai analysis of matmul behaviour calls the inability to hold peak clock under power throttling one of the primary factors separating real matmul performance from marketed specs. The same analysis argues that the H100’s roughly 3x theoretical advantage over the A100 has usually been closer to 2x in practice. Meta even reports a diurnal 1-2% throughput swing on Llama 3 405B, caused by mid-day temperatures affecting GPU voltage and frequency scaling.
The Scale-Out Gap: What a Cluster Sustains
The second loss comes from running thousands of GPUs as one job. On A100s, AxoNN sustained 50% or more of advertised peak. On MI250X it fell from about 40% to 22% at 32,768 GCDs, and on H100s it held about 30% before dropping to 23.42% at 6,144 GPUs. Megatron-LM (Narayanan et al., 2021) reported 52% of theoretical peak per GPU across 3,072 GPUs, and ByteDance’s MegaScale (2024) reported 55.2% MFU on 12,288 GPUs.
Hardware outliers cut into this too. MegaScale found that about 0.5% of machines ran substantially slower, and that the slowest machine largely set training efficiency for the whole job. Meta likewise notes that a single slow straggler can slow down thousands of other GPUs.
How Does Precision (FP64, FP32, FP16, FP8) Change the FLOPS a GPU Reports?
Every peak figure belongs to one numeric format, and the gap between formats is large. NVIDIA’s A100 datasheet makes this plain:
| Precision | Dense peak | With sparsity |
|---|---|---|
| FP64 | 9.7 TFLOPS | n/a |
| FP64 Tensor Core | 19.5 TFLOPS | n/a |
| FP32 | 19.5 TFLOPS | n/a |
| TF32 Tensor Core | 156 TFLOPS | 312 TFLOPS |
| BF16 / FP16 Tensor Core | 312 TFLOPS | 624 TFLOPS |
| INT8 Tensor Core | 624 TOPS | 1,248 TOPS |
The sparsity column applies to sparse models, for which NVIDIA says its Tensor Cores give up to 2x higher performance. If your model is not sparse in that pattern, the right-hand column does not apply to you. The H100 spec sheet is easy to misread in the same way. Its bf16 figure of 1.979e15 FLOPs/s carries a “with sparsity” asterisk, and the dense value is half, 9.89e14. Epoch AI gives about 2×10^15 FLOP/s as the H100 SXM’s dense 8-bit peak, twice the dense bf16 value above, and the SAKURAONE team (2026) computed its MFU against a dense peak of 1,979 TFLOPS per GPU. One number on a datasheet can stand for two different things, so any utilisation figure needs its denominator stated before you compare it with another.
Tensor cores have given lower-precision formats disproportionately more throughput than FP32 and FP64, starting especially with the Volta generation. Vendors keep widening that gap for FP16, BF16, FP8 and INT8, sometimes at the expense of FP64. The SAKURAONE system shows how far apart the numbers can sit on one machine: 33.95 PFLOP/s on HPL and 339.86 PFLOP/s on HPL-MxP with FP8.
Lower precision only buys the full ratio when the workload is compute-bound. A 2024 survey of mixed-precision numerics points out that for a memory-bandwidth-limited application, the gain comes only from moving more numbers per byte, so going from FP64 to FP32 caps the speedup at 2x. On HPL-MxP, Summit sped up 9.50x against a V100 FP16-to-FP64 ratio of 16.02x, while Frontier reached 8.31x against 7.99x for the MI250X. Part of the reason is that vendor peaks are not what BLAS libraries actually reach. The mechanics belong to HPL-MxP explained, which treats mixed-precision LINPACK as a tensor-core throughput signal.
Memory Bandwidth Decides Whether the Tensor Cores Are Ever Busy
The roofline model measures a kernel by arithmetic intensity: the FLOPs it performs divided by the bytes it has to move. Epoch AI puts the H100 SXM’s 8-bit dense peak against 3.35 TB/s of DRAM bandwidth, which gives a balance point of about 299 FLOP per byte. A square matrix multiplication needs dimensions around 896 to reach that balance. Below it, the tensor cores wait on memory. Splitting a problem across more GPUs shrinks at least one dimension and makes arithmetic intensity worse. For bf16 on an H100, the JAX scaling book puts the critical batch at about 295 tokens.
Delivered throughput therefore depends heavily on matrix shape. On the MI300X, a 2025 AMD training study found a GEMM needed about 200 GFLOPs of work to reach peak throughput. The same study notes that even IO-aware attention stays bound by HBM bandwidth until sequence lengths of roughly 4-16k, depending on the GPU’s roofline. A 2026 study of MFU as a power proxy measured MFU rising from around 1% at batch 1 to 28-44% at batch 128. If the operators are too small, framework overhead takes over and the GPU spends most of its time idle.
Decode-style inference sits at the far end of this curve, because each generated token reloads the weights. One MFU explainer’s roofline estimate puts batch-32 decode on an H100 at about 32 ÷ 295 ≈ 11% of peak, no matter how good the kernel. The inference side, including KV-cache reuse for inference, has its own page.
Why Does Interconnect Bandwidth Bound Realised Cluster FLOPS at Scale?
Every data-parallel or hybrid-parallel step synchronises through collective operations such as all-reduce, so scaling LLM training is fundamentally limited by inter-GPU communication. Compute per GPU has grown much faster than inter-node bandwidth per node, which is why clusters need high-bandwidth links inside the node, such as NVLink or AMD Infinity Fabric.
The bandwidth tiers are far apart. NVIDIA quotes 600 GB/s NVLink against 64 GB/s PCIe Gen4 for the A100, and 900 GB/s of GPU-to-GPU NVLink for the H100. Between nodes, SAKURAONE’s servers each carry eight 400 GbE ports for the GPU fabric. That is about 50 GB/s per port at line rate (400 ÷ 8). One representative job drove about 22.6 GB/s per port, and another showed two of eight rails stuck near 8.0 GB/s while the other six reached 18.9 GB/s. Meta orders Llama 3’s parallelism as tensor, context, pipeline, then data, so that the most bandwidth-hungry dimension stays inside one server.
Scale makes the ratio worse. When you grow a cluster at fixed model and batch size, compute and network bandwidth grow together, but data movement per GPU keeps rising, so communication eventually becomes the bottleneck. MegaScale’s MFU fell from 59.1% to 55.2% as GPU count rose at a fixed batch size, because the computation-to-communication ratio dropped. When SAKURAONE’s 64-node run spanned two pods joined by spine switches, communication share rose from 16.4% to 19.3% and compute-communication overlap fell from 72.3% to 67.2%. At 96 nodes, its MFU dropped to 35.9%.
Message size matters as well. A 2019 characterisation of NVLink found that its unidirectional bandwidth saturates only at about 4 MB per transfer. On its own MI300X fabric, the AMD team set its gradient fusion buffer exactly at its measured bandwidth saturation point and no larger, so that it could still overlap communication with compute. In MegaScale, overlapping communication with computation was worth 6.2 points of MFU.
The physical layer counts too. MegaScale found that the root cause of stalls from network-interface flapping was bad link quality between the network card, AOC cable and switch. The trade-offs in that layer, distance, rate and failure behaviour, are covered in cluster interconnect: DAC vs AOC. The wider point, that GPUs are part of a larger system, applies here directly: a cluster’s realised FLOPS depends on how fast tensors move, not just on silicon.
What Is Model FLOPS Utilisation (MFU), and What Range Is Realistic?
The PaLM paper (Chowdhery et al., 2022) defined MFU as observed throughput in tokens per second, divided by the theoretical maximum throughput of the system running at peak FLOPs. That maximum counts only the operations the forward and backward passes require, and leaves out rematerialisation. Because the denominator depends only on the model architecture and the published peak, MFU is implementation-independent. PaLM 540B reached 46.2% MFU but 57.8% hardware FLOPS utilisation, which includes rematerialisation FLOPs.
In formula form, MFU equals the model FLOPs per training step divided by (peak FLOPs per GPU × step time × number of GPUs). For a dense transformer, model FLOPs are approximately 6 × parameters × tokens.
The GPT-3, Megatron-Turing, Gopher and PaLM rows come from the PaLM paper’s comparison table.
| Training run | Hardware | Reported MFU |
|---|---|---|
| GPT-3 175B | V100 | 21.3% |
| Megatron-Turing NLG 530B | 2,240 A100 | 30.2% |
| Gopher 280B | 4,096 TPU v3 | 32.5% |
| SAKURAONE GPT-3 175B | 256-768 H100 | 35.9-41.2% |
| Llama 3 405B (BF16) | up to 16K H100 | 38-43% |
| PaLM 540B | 6,144 TPU v4 | 46.2% |
| MegaScale 175B | 12,288 GPUs | 55.2% |
The recent large runs above, from SAKURAONE, Meta and MegaScale, sit between about 36% and 55%. Meta and MegaScale both describe reaching their figures through deliberate tuning of parallelism and software. Epoch AI suggests planners assume 0.3 for large language models and 0.4 for other networks. We would treat the upper end of the published range as something you earn through tuning, not a starting assumption, and the decode figures in the roofline section as a separate regime altogether.
Do not confuse model FLOPS utilisation with the GPU utilisation counter. GPU utilisation is a time-based activity metric, so it can read high during memory-bound or stalled execution. The 2026 power study saw it at 40-55% at batch 1, while MFU sat around 1%. That is how low productive throughput hides behind a busy dashboard, as covered in the hidden cost of GPU underutilisation.
How Should I Use FLOPS Numbers When Sizing a GPU Cluster?
Epoch AI’s estimate of training compute starts from datasheet peak throughput and corrects it with a utilisation rate below 100%. To size a cluster, multiply training time, GPU count, peak and that utilisation rate, solve for GPU count or time, and put most of your effort into the utilisation term.
- Fix the workload’s FLOP budget, from 6 × parameters × tokens for a dense transformer or from a profiled step.
- Take the dense peak in the precision you will actually train in, without sparsity.
- Apply a utilisation measured on your model, at your batch size and parallelism layout. Without a measurement, use the published MFU range as a bracket, not a single point.
- Check how that utilisation holds at the target node count, including cross-pod traffic, since communication share grows with scale.
- Add headroom for repeated runs. SAKURAONE sized about 800 H100s so that a 70B model could train on 300B tokens in four months while leaving room for iterative and exploratory runs.
Worked Example: The Same Budget at Peak and at Realistic MFU
Assumptions: a compute budget of 3.8×10^25 FLOPs (the budget Meta’s Llama 3 scaling law was extrapolated to), 16,000 H100s, the dense bf16 peak of 989 TFLOPS, and 40% MFU, inside Meta’s reported 38-43%. This shows the arithmetic, not Meta’s actual schedule.
3.8e25 ÷ (16,000 × 989e12) ≈ 2.4 million seconds, about 28 days at peak; at 40% MFU the same budget takes about 70 days. The naive plan is off by 2.5x (1 ÷ 0.4). Using the sparsity figure as the peak would double the error again.
Measure Your Own Utilisation Before Optimising
The right way to speed up a system depends on its bottleneck, so understand that bottleneck first. Horace He notes that people often try speed-ups blindly, without knowing which regime their model is in. The AxoNN team found one matmul that PyTorch ran by default in TN mode reaching only 6% of theoretical peak, while other matmuls reached 55%. They fixed it by timing all modes in the first batch. Our position is to profile first and treat best practice as context-dependent, which applies directly to the utilisation figure in a sizing plan. Our guide to profiling GPU kernels to find the real bottleneck covers that step. How to read published hardware numbers fairly is covered in why spec-sheet benchmarking fails for AI.
A FLOPS Figure Without an Interconnect Assumption Is Not a Plan
A peak figure on its own leaves out the precision, the denominator, the batch regime and the fabric that sit between the datasheet and your step time. Every gap described above compounds into the final number: power-limited clocks, memory-bound shapes, communication share and stragglers. The question to settle before you procure is not how many FLOPS the GPUs have. It is what MFU you have measured or can defend at the target node count, and with which peak in the denominator. If you cannot answer that yet, a short measurement run at realistic scale will tell you more than any spec sheet.