Read the FLOPS figure off a datasheet, multiply by the number of GPUs, and you have a capacity plan. That is the move almost every team makes when they first size a cluster, and it is wrong in a way that costs real money. The peak FLOPS number on a spec sheet is a theoretical ceiling measured under ideal precision and near-perfect utilization — a condition your training or inference workload will essentially never reach. Treating it as the cluster’s effective compute leads directly to over-provisioning, because you buy GPU count against throughput the fabric between those GPUs can never actually feed. The honest version of the number is smaller and harder to find. A cluster’s realized FLOPS is bounded not just by silicon, but by how fast tensors move between GPUs. A FLOPS figure quoted without an interconnect assumption behind it is not a capacity plan — it is a marketing headline. What does AI FLOPS actually measure? FLOPS — floating-point operations per second — counts how many multiply-add and related arithmetic operations a device can push through per second. For AI workloads the relevant operations are dense matrix multiplications and the accumulation steps inside attention and feed-forward layers, which is why vendors publish separate tensor-core figures for these fused operations rather than general-purpose FP32 rates. The trap is that the published figure is a peak: what the arithmetic units could sustain if data arrived at exactly the rate they consume it, at the lowest precision the hardware supports, with no stalls. Real workloads stall constantly. They wait on high-bandwidth memory (HBM) reads, on kernel launch overhead, on the tail of a collective operation completing across every GPU in the job. Each stall is time the arithmetic units sit idle, and idle units produce zero FLOPS. So the same device honestly reports two very different numbers depending on what you ask. Peak FLOPS answers “what is the silicon capable of under ideal conditions?” Sustained FLOPS answers “what did this workload actually get?” The gap between them is where cluster sizing lives or dies. This is the same peak-versus-achieved distinction that applies on the CPU side of the pipeline, which we cover in CPU GFLOPS: reading peak vs achieved throughput on an inference path — the mechanism differs, the framing does not. Peak FLOPS vs sustained FLOPS: the gap that decides your GPU count Here is the distinction in the form that matters for procurement. Property Peak FLOPS Sustained FLOPS What it measures Theoretical arithmetic ceiling Throughput a real workload achieves Precision assumed Lowest the hardware supports (often FP8/FP4) Whatever precision your model runs in Utilization assumed Near 100%, no stalls Bounded by memory + interconnect + launch overhead Where it comes from Vendor datasheet Measurement of your workload on your fabric Valid use Comparing silicon generations Sizing capacity to a throughput target Failure mode if misused Over-provisioning GPU count — The single most useful number a sizing exercise produces is the ratio between them: model FLOPS utilization, or MFU. MFU is the fraction of peak FLOPS your workload actually converts into useful model computation. For large transformer training and inference, MFU commonly lands in the 30–60% range (observed pattern across published training reports and our own engagement measurements; not a single benchmarked rate — it depends heavily on model shape, batch size, and interconnect). A well-tuned dense training run on a tightly coupled node can push toward the top of that band; a communication-heavy or poorly batched job can sit well below it. If you plan against peak and your workload runs at 40% MFU, you have quietly over-bought by roughly a factor of two — you provisioned GPU-hours the job structurally cannot use. The correction is not to guess a discount factor. It is to measure MFU for a representative slice of the workload and size against the sustained figure it implies. How does numeric precision change the FLOPS a GPU reports? Precision format is the first thing that inflates a peak number, and it is worth being explicit about because vendors almost always quote peak at the lowest precision the tensor cores support. A modern GPU reports dramatically different FLOPS depending on the format. Dropping from FP16 to FP8 roughly doubles the arithmetic throughput the tensor cores can advertise, and FP4 doubles it again, because narrower operands let more operations pack into the same silicon and memory bandwidth per cycle (per the architectural rationale NVIDIA publishes for its tensor-core generations). That is why a headline “petaFLOPS” figure is often an FP8 or FP4 number that has nothing to do with an FP32 or FP64 workload. The catch is that precision is a first-class accuracy trade-off, not a free throughput multiplier. Running attention in FP8 or weights in FP4 changes numerical behavior, and whether that is acceptable is a per-model question. We walk through where the low-precision formats actually pay off — and where they quietly degrade accuracy — in 4-bit floating point (FP4): how it works and what it means for GPU utilisation. For sizing, the rule is simple: the peak FLOPS you plan against must be quoted at the precision your model will actually run in. An FP8 peak is irrelevant to an FP16 workload. Why interconnect bandwidth bounds realized cluster FLOPS at scale Single-GPU sustained FLOPS is gated mostly by HBM bandwidth and kernel efficiency. The moment you go multi-GPU, a second and often dominant ceiling appears: the fabric between the GPUs. Distributed training and large-model inference depend on collective operations — all-reduce, all-gather, reduce-scatter — that synchronize tensors across every GPU in the job. These collectives, typically implemented through NCCL, force the arithmetic units to wait while gradients or activations move across NVLink, across PCIe, or across the network fabric between nodes. During that wait, the GPUs produce no useful FLOPS. As the job scales out across more nodes, the collective’s completion time grows, and the fraction of wall-clock spent moving tensors rather than computing them rises with it. This is the point the datasheet cannot capture. Realized cluster FLOPS is a function of the interconnect topology, not just the GPU count. Two clusters with identical silicon and identical GPU counts will deliver materially different sustained throughput if one uses NVLink and high-bandwidth optical links between nodes while the other is stuck on lower-rate copper over longer distances. The same interconnect distance and rate constraints that decide whether a link should be direct-attach copper (DAC) or active optical cable (AOC) are the same constraints that decide how much of your peak FLOPS survives the collective. If you are pushing toward high-radix, high-bandwidth fabric, the 800G ConnectX-8 NIC and when interconnect matters is where that trade-off becomes a line item. A concrete way to see it: if you double GPU count but the collective bandwidth per GPU does not scale with it, your marginal GPUs spend a growing share of their time idle in communication. You paid for FLOPS the fabric will not feed. Right-sizing means matching GPU count to the throughput the interconnect can actually sustain — not to the arithmetic ceiling. A worked sizing example Assume you need sustained training throughput equivalent to 500 useful petaFLOPS for a large transformer, and each GPU advertises 2 petaFLOPS peak at your target precision. Naive plan: 500 ÷ 2 = 250 GPUs. This is the datasheet-multiplication trap. MFU-aware plan: if a representative run measures 45% MFU on the intended fabric, each GPU delivers roughly 0.9 useful petaFLOPS. You need about 500 ÷ 0.9 ≈ 556 GPUs to hit the target. The decision point: the naive plan under-provisions by more than 2×; the MFU-aware plan is honest but expensive. The lever that moves the 45% is the interconnect. Improving collective bandwidth so MFU climbs to 55% drops the requirement to about 455 GPUs — a fabric investment paying for itself in reduced GPU count. (Illustrative figures with explicit assumptions; MFU must be measured for your actual model and fabric, not assumed.) The point of the example is not the arithmetic. It is that the fabric and the GPU count are the same sizing decision, and you cannot make one without the other. FAQ How does AI FLOPS actually work? FLOPS counts floating-point operations per second — for AI, mostly the matrix multiplications inside attention and feed-forward layers. In practice the published figure is a theoretical peak measured under ideal conditions, so the number that matters for planning is sustained FLOPS: what your workload actually achieves once memory, precision, and interconnect stalls are accounted for. What is the difference between peak FLOPS and sustained FLOPS on a GPU cluster? Peak FLOPS is the arithmetic ceiling the silicon could hit with no stalls and lowest precision; sustained FLOPS is what a real workload converts into useful computation. The gap is large — often a 2× difference — and sizing against peak leads directly to over-provisioning GPU count against throughput the cluster cannot deliver. How does numeric precision (FP64, FP32, FP16, FP8) change the FLOPS a GPU reports? Narrower operands pack more operations into the same silicon and bandwidth, so dropping from FP16 to FP8 roughly doubles advertised throughput and FP4 doubles it again. Vendors quote peak at the lowest supported precision, so a headline petaFLOPS number is usually irrelevant unless your model actually runs in that format — and precision is an accuracy trade-off, not a free multiplier. Why does interconnect bandwidth bound realized cluster FLOPS at scale? Multi-GPU jobs depend on collective operations (all-reduce, all-gather) that force arithmetic units to wait while tensors move across NVLink, PCIe, or the network fabric. During that wait the GPUs produce zero useful FLOPS, and the wait grows as the job scales out — so realized cluster FLOPS is a function of interconnect topology, not GPU count alone. What is model FLOPS utilization (MFU), and what range is realistic? MFU is the fraction of peak FLOPS a workload converts into useful model computation. For large transformer training and inference it commonly lands in the 30–60% range (observed across published training reports and our engagements; not a single benchmarked rate), depending on model shape, batch size, and fabric. A tightly coupled, well-batched run sits near the top; a communication-heavy job sits well below. How should I use FLOPS numbers when sizing a GPU cluster instead of reading peak off a datasheet? Measure MFU for a representative slice of your workload on the intended fabric, then size against the sustained throughput that implies — not against peak. Quote peak only at the precision your model will run in, and treat GPU count and interconnect as one decision, since fabric investment that raises MFU can lower the GPU count you need. Where this leaves a sizing decision The uncomfortable part is that the number you can size against does not exist until you measure it. Peak FLOPS is available on day one; MFU requires running a representative workload on the fabric you are considering. That measurement dependency is exactly why fabric design precedes procurement — the interconnect layer decides how much of the silicon you will ever use. Our broader view on how realized throughput depends on the layers below the GPU sits on the GPU engineering practice page. The failure class to name plainly: a FLOPS number without an interconnect assumption is not a plan. If a capacity estimate cannot state the precision it assumes and the MFU it expects on a named fabric, it is a datasheet multiplication wearing a plan’s clothes — and it will over-provision. The remaining question worth pushing on is not “how many GPUs deliver 500 petaFLOPS?” but “what fabric makes the GPUs I can afford spend their time computing instead of waiting?”
