Model FLOPS utilization (MFU) is the share of a system’s peak floating-point throughput that a workload turns into the arithmetic its model actually needs. You calculate it as observed model FLOP/s divided by one device’s peak FLOP/s multiplied by the number of devices. PaLM’s comparison table listed large-model training runs at 21.3% (GPT-3) to 46.2% (PaLM 540B), a practitioner write-up from Trainy puts most LLM training at 35% to 45%, and decode-style inference counts as well optimised at 8% to 10%. MFU is a good ratio for tracking one setup against itself. Its denominator is a datasheet number, though, so it is a weak tool for comparing one GPU with another.

Calculating model FLOPS utilization: the formula and a worked example

The metric comes from Google’s 2022 PaLM paper, which defined it as observed throughput in tokens per second relative to the theoretical maximum throughput of a system running at peak FLOPs. The detail that matters is in the denominator: the “theoretical maximum” counts only the operations the forward and backward passes require, not any recomputation the implementation chooses to do. That leaves a denominator that depends only on the model architecture and the published maximum FLOPs of the system.

For a transformer, the 2024 COLM paper on parallelisation layouts by Hagemann et al. follows the PaLM calculation: take the peak matmul throughput P, then the theoretical peak token rate is R = P / (6N + 12LHQT), where N is the parameter count, L the number of layers, H the attention heads, Q the head size and T the sequence length. MFU is achieved tokens per second divided by R.

Worked example: LLaMA 65B on A100s

The same paper works the numbers through for a LLaMA 65B run, with an A100 peak of 312 matmul TFLOP/s. The paper multiplies both the achieved throughput and the peak by 2,048, so the ratio is unaffected by how that factor is read.

Step	Value
Achieved throughput	380 × 2,048 = 778,240 tokens/s
Peak matmul throughput	312 × 10¹² × 2,048 ≈ 6.39 × 10¹⁷ FLOP/s
Attention FLOPs per token	12 × 80 × 8,192 × 2,048 ≈ 1.61 × 10¹⁰
Model FLOPs per token	6 × 65 × 10⁹ + 1.61 × 10¹⁰ ≈ 4.06 × 10¹¹
Theoretical peak tokens/s	6.39 × 10¹⁷ ÷ 4.06 × 10¹¹ ≈ 1.57 × 10⁶
MFU	778,240 ÷ 1.57 × 10⁶ ≈ 49.5% (the paper reports 49.46%)

The authors leave out the first of ten training steps because warm-up distorts it, and they report the mean of the remaining nine. For instrumenting the metric across a full run, see our companion piece on measuring and interpreting MFU in AI training.

Where the missing FLOPs go

MFU stays below 100% because its denominator is a peak that cannot be achieved in practice. A datasheet peak is computed as clock speed × FLOPs per clock per compute unit × number of compute units. That aggregate is reached only when every SM is busy, so a matmul too small to fill them all cannot hit it. Stas Bekman’s ML Engineering Open Book puts it bluntly: the advertised peak cannot be achieved in practice even under perfect conditions.

The first loss is the clock assumption. The advertised peak assumes the boost clock: about 1,965 MHz for the B200, giving 2,250 BF16 TFLOPS. A dense matmul that really fills the SMs draws the full TDP (1,000 W on a B200), and the SM clock then settles well below boost. Scaling the spec by the sustained clock gives a rough ceiling of 2,250 × 1,410 / 1,965 ≈ 1,615 TFLOPS. The measured sustained matmul rate was 1,439, and kernel overhead accounts for the rest. None of that is a broken GPU. It is how a power-limited chip behaves under real load, and an MFU calculated against 2,250 counts the whole difference as “unused”.

Memory bandwidth is the second loss, and the gap behind it has widened over two decades. In the AI and Memory Wall paper (Gholami et al., 2024), peak server FLOPS rose 60,000× over twenty years while DRAM bandwidth rose 100× and interconnect bandwidth 30×. Any phase with low arithmetic intensity is limited by bytes rather than FLOPs, and MFU normalises by peak FLOPS, so a memory-bound workload shows low MFU even while power draw and GPU utilisation stay high. Autoregressive decode is the standard case: matrix-vector work with low arithmetic intensity, which is why GPT-2 inference is far slower than BERT despite similar end-to-end FLOPs.

Communication is the third. In the 2026 CCL-Bench study, GPU MFU on the training workloads was held down by the higher communication overhead of the scale-out network. Synchronous training adds a further loss: every step waits for the slowest GPU. On one 8× B200 node, sustained matmul throughput ranged from 1,421 to 1,460 TFLOPS between GPUs, so the node trained at 1,421. Sinha et al. found 8% average performance variation across GPUs of identical architecture and SKU, with outliers up to 1.5× slower.

Why the same code reports lower MFU on an H100 than on an A100

The denominator roughly tripled, so the same code reports lower MFU unless its achieved throughput rises by the same factor. The A100 matmul peak used in MFU papers is 312 TFLOPS, and the H100’s dense FP16 datasheet figure is 989 TFLOPS, about 3.2× higher (989 ÷ 312).

In the 26Q3 results, the NVIDIA H100 80GB HBM3 scores 1,315.1 on BF16 dense matmul against 477.4 for the NVIDIA A100-SXM4-80GB, but its lead on BF16 DistilBERT training is narrower: 1,292.0 against 690.9. The H100 scores higher in both. In these results the gap shrinks from a pure matmul to a real model.

How is MFU different from the GPU utilization percentage in nvidia-smi?

The two numbers measure different things. NVIDIA’s nvidia-smi manual describes its utilisation figures (SM, memory, encoder, decoder) as average utilisation since the last monitoring cycle. As Trainy’s analysis explains, GPU utilisation in this sense only records whether a kernel is executing at a given moment. It says nothing about whether that kernel uses all the cores, and a GPU can show 100% while only reading and writing memory at zero FLOPS.

The worked example in the Trainy article makes this concrete: a kernel that runs for ten seconds on one of an H100’s 132 SMs registers 100% utilisation and 0.7% SM efficiency. Trainy describes a customer job sitting at 100% GPU utilisation and 20% MFU, which reached 38% MFU and a 4× faster training time after kernel fusion and retuned model parallelism. A 2026 study of MFU as a power proxy found the same divergence from the other side: on the AMD MI210, vendor-reported GPU utilisation behaves as a binary activity flag, while MFU tracks power linearly.

So nvidia-smi answers “is the GPU busy?” and MFU answers “is the busy GPU doing model arithmetic at a rate close to its rated peak?”. Neither tells you what causes the shortfall. That is the territory of utilisation, bottlenecks and the illusion of idle GPUs, and of the wider question of which ML metrics explain GPU bottlenecks and which mislead.

MFU or HFU: which number should a team report?

Every FLOP the chip actually executes, including activation recomputation and redundant tensor-parallel work, counts towards hardware FLOPS utilization (HFU). MFU counts only the analytic forward and backward FLOPs the model needs. Without activation recomputation the two are equal. With it, MFU comes out lower, because the recomputed forward pass counts towards HFU but not MFU. PaLM 540B is the reference pair: 46.2% MFU and 57.8% HFU.

PaLM’s authors chose MFU as the headline because recomputation trades extra hardware FLOPs for memory, while the goal of a training system is tokens per second, not using as many FLOPs as possible. They also rejected HFU as inconsistent, since observed hardware FLOPs have been reported both from analytical accounting and from hardware counters. MFU is the number for comparing models and systems, HFU the chip-utilisation number, and either needs its FLOP-counting method stated beside it.

The counting method decides whether published comparisons hold. Bekman’s checklist asks two questions before any MFU or HFU comparison: were total FLOPs calculated the same way, and was iteration time measured back to back (DataLoader and logging included) or over the forward and backward passes only? If either differs, a fair comparison is not possible.

Framework counters can also simply be wrong. In a 2026 fleet study, a 288-GPU DeepSeek-style mixture-of-experts job reported 54.27% MFU, which made it look like one of the fleet’s best workloads. Hardware counters measured 25.58%, because the framework’s FLOPs formula left out the model’s latent-space projections and overstated FLOPs by about 3×. The authors note that these formulas have to be updated for every new architecture and can go wrong silently. Any MFU figure for a novel architecture deserves a hardware-counter cross-check before anyone believes it.

Because MFU divides by a datasheet peak, what does that mean when comparing MFU across GPUs?

Only loosely, and it can mislead. MFU divides by a published peak, which the device cannot reach in practice. Three things weaken cross-device comparison.

The first is that equal MFU means different amounts of work. The 2026 power-proxy study notes that equal MFU values correspond to different raw arithmetic on different chips, so each device needed its own power-model calibration. The second is that the denominator moves with precision. The H100’s dense FP8 datasheet figure is twice its FP16 figure, and on the B200 each step down from FP16 to FP8 to FP4 doubles it again. In the fleet study, switching a run from mixed precision to BF16 left throughput roughly unchanged, yet reported MFU rose, because BF16 has a lower theoretical peak. The third is that the peak itself can be approximate. Bekman points out that a peak quoted without its SM count is approximate for at least one of two SKUs sold under one name with different SM counts.

The CCL-Bench results show why this matters for decisions. TPU v6e posted higher MFU than A100 on all five workloads, but on one of them it also had the longer average step time: 1.09 s against 0.12 s. On that workload, higher MFU did not mean a faster job. Bekman’s own guidance is to compare your setup’s MFU before and after an optimisation, not against another team’s cluster. LynxBenchAI goes further and treats datasheet metrics as structurally insufficient predictors of AI performance, because memory locality, data reuse, execution strategy and software behaviour depend on the workload and cannot be read off a spec sheet.

Is a higher MFU always better, and when is a low MFU an acceptable result rather than a problem to fix?

A low MFU is a problem when it sits well below the ceiling for that workload’s shape. Higher MFU does generally go with lower step time on a fixed workload and fixed hardware, which is the relationship CCL-Bench confirmed across its training and inference entries. Across different hardware that relationship did not always hold.

Decode is the clearest acceptable case. Decode loads the full weight tensor for every token, so its arithmetic intensity scales with batch size. On an H100 at batch 32 the roofline ceiling is about 11% of peak, and no kernel rewrite breaks that. At 8% to 10% MFU, decode is well optimised, and the 40% to 60% headline figures belong to training and prefill. In the Hagemann et al. study, skipping activation checkpointing and compensating with smaller batches or more model parallelism gave the best training throughputs.

The diagnostic value is in the gap between the measured figure and the ceiling. Prefill at 25% on a 70B FP16 H100 run points to missing FlashAttention or fusion. Decode at 4% on batch 32 points to batching or KV-cache fragmentation. Training at 18% on several nodes points to parallelism and topology rather than the chip. Finding which of these applies is profiling work, covered in what inference profiling numbers mean in practice.

At what batch size does an MFU number mean anything?

For diagnosis, the meaningful batch size is the one the workload really runs at; for any statement about the device, it is saturation. Batch size moves MFU by large factors. In the 2026 power-proxy sweep across six GPUs, MFU grew 36× to 72× between batch 1 and batch 128, and every device ran below 1.3% MFU at batch 1. A small model on a big GPU gets single-digit MFU because it is too small to saturate the device.

The micro-batch is a separate knob. Hagemann et al. found a micro-batch size of 1 gave the highest MFU for every model type they tested. State both micro-batch and global batch next to any MFU figure.

For device comparison, LynxBenchAI keeps the model and precision fixed and raises only the batch size until throughput stops improving within a defined noise band. A laptop GPU and a data-centre accelerator are each measured where they actually saturate, instead of at one fixed size that happens to suit one of them.

What FP8 infrastructure do you need before flipping it on to raise MFU?

FP8 does not raise MFU on its own, because it also raises the denominator. The H100’s dense FP8 peak is 1,979 TFLOPS against 989 at FP16. If throughput goes up by less than 2×, MFU against the FP8 peak falls. According to vLLM’s documentation, FP8 gives up to a 1.6× throughput improvement, not the 2.0× the datasheet implies. Judge FP8 by tokens per second, and quote the precision of the denominator whenever you quote MFU.

Before switching it on, check the following:

• Hardware that executes FP8. In the 26Q3 results, FP8 inference produced no result (0.0) on the NVIDIA A100-SXM4-80GB, the NVIDIA GeForce RTX 3090 and the NVIDIA A40, while the H100 80GB HBM3 scored 1,658.9 on FP8 Flan-T5 Small inference.
• A software path that uses it. Kernels, framework support and serving engine all need an FP8 path; the hardware alone does nothing. Which formats a device can use at all is set by its architecture, as covered in precision choices are constrained by hardware architecture.
• Accuracy validation on your model. A study cited by gpuperhour.com found FP8 weights and activations “effectively lossless across all model scales”. That is a study result, not a guarantee for your model, so validate it on your own evaluation set.
• A reason to want the memory. At FP8, 70B-parameter weights take about 70 GB and fit on one 80 GB H100 instead of two at 16-bit. The capacity side is covered in precision as a memory-capacity lever.

Making an FP8 path production-ready (quantising, validating and profiling it) is engineering work rather than measurement, and it is the kind of work TechnoLynx’s GPU performance engineering addresses.

What can a single sustained-throughput measurement tell me that an MFU percentage cannot?

One sustained-throughput figure at saturation tells you what the machine holds under heavy load, in completed work. MFU cannot tell you that, because half of its definition is a vendor’s number. LynxBenchAI treats sustained practical peak as the operationally relevant figure and does not treat a transient burst as deployable performance.

Bekman distinguishes Maximum Achievable Matmul FLOPS, a short boost-clock burst, from Maximum Sustainable Matmul FLOPS, what the chip holds once it is saturated at its power limit. On a B300, one shape bursts to 1,892 TFLOPS but holds only 1,487 once saturated. The same split shows up at service level: peak MFU is what one clean run achieves, while sustained MFU is what the service holds over hours of real traffic.

The LynxBenchAI methodology declares its measurement condition rather than claiming more than it observes. A warm-up phase is discarded, then completed iterations are counted inside one continuous timed window, at a batch size already raised until throughput plateaus. Every device in a release runs the same model catalogue at the same precisions, and only the batch size adapts. The integrity comes from declaring the window. The results are scores (higher is better), not utilisation percentages, and they are comparable only within one release such as 26Q3.

Measure	What it answers	What it cannot tell you
nvidia-smi utilisation	Was a kernel running during the sampling interval?	Whether the kernel used the cores or did any FLOPs
MFU	What share of the rated peak became model arithmetic	Whether the rated peak was reachable; cross-device rank
HFU	What share of the rated peak the chip executed, recomputation included	A consistent figure, since it depends on counting method
Sustained throughput at saturation	What the executor holds under heavy load after warm-up	Your exact production workload, unless that is what you run

If the argument is about one specific card that looks idle, settle it by measuring. The LynxBenchAI Personal Edition installs with pip install lynxbench-ai on Python 3.11 or later, on Linux or on Windows via WSL2, needs about 15 GB of free disk, downloads roughly 2 GB of model data once, and runs in 15 to 30 minutes. The result is a 26Q3 number for the machine on your desk, which you can set beside the published device pages. The question an MFU figure leaves open is still open, though: if the card holds its expected throughput and the job is still slow, look at the pipeline around it. Finding it is the work described in profiling tools for AI inference and how to read them, and its cost shows up as the hidden cost of GPU underutilisation.