Two machines sit on a spec sheet. One quotes 1.2 TFLOPS, the other 900 GFLOPS. A team picks the higher number, ports the inference path toward it, and discovers the faster machine is no faster at all. The gigaflops figure told them nothing about where their latency actually lived. This is the trap in reading a GFLOPS number as a speed rating. A gigaflops figure — whether it comes from a CPU vendor’s datasheet or a GPU spec sheet — is a theoretical compute ceiling. It describes what the arithmetic units could sustain if every cycle were fed a perfectly aligned multiply-add and nothing else got in the way. Real inference paths rarely look like that. Most of the latency in a deployed model lives in places where FLOPS never apply at all: runtime scheduling, memory movement between host and device, and the Python glue holding the pipeline together. The practical claim is narrow and worth stating plainly. A GFLOPS figure tells you what the compute layer could do at its best; it says nothing about whether compute is your bottleneck. If it is not — and for a large share of production inference paths it is not — then the higher gigaflops number is buying you headroom you will never touch. What does a GFLOPS figure actually measure? GFLOPS stands for billions of floating-point operations per second. GFLOPS is the same unit at a different scale: a trillion of them. On a CPU, the peak figure comes from a simple product — core count, clock frequency, floating-point operations retired per cycle per core (which SIMD width and fused multiply-add both inflate), and the number of vector lanes. Multiply those together and you get the number printed on the slide. That number is almost always a peak, not a sustained figure, and the distinction matters more than the magnitude. Peak assumes: Every core runs at its advertised clock the entire time, with no thermal or power throttling. Every arithmetic unit issues a fused multiply-add every cycle. Operands arrive from cache with no stalls waiting on memory. No time is lost to scheduling, branch misprediction, or synchronisation. Under a real AVX-512 or AVX2 kernel doing dense matrix multiply, a well-tuned CPU can approach a meaningful fraction of peak — in the configurations we have profiled, a carefully vectorised GEMM might reach roughly 60–80% of the theoretical figure (observed pattern across inference-porting engagements; not a published benchmark). Wrap that same kernel in a real model with data loading, tokenisation, and Python orchestration around it, and the achieved-to-peak ratio collapses. The distinction between peak and achieved GFLOPS on a CPU inference path is where most of the disappointment lives. The gap between achieved and peak is not noise. It is the arithmetic-intensity ratio — how many FLOPs you perform per byte you move — telling you whether your workload is compute-limited or bandwidth-limited. A low ratio means you are waiting on memory, and no amount of extra peak GFLOPS will help. Why does a CPU rarely reach its peak gigaflops during real inference? The short answer is that inference is rarely one big matrix multiply. It is a sequence of operations with different shapes, and the small or memory-bound ones dominate wall-clock time even when they contribute little to the FLOP count. Consider a transformer decode step generating one token at a time. The matrix multiplies degenerate into matrix-vector products — the weight matrix is large, but you are multiplying it by a single column. Arithmetic intensity is terrible: you read the entire weight matrix from memory to produce one output vector. The bottleneck is memory bandwidth, full stop. A CPU rated at 1.2 TFLOPS and one rated at 900 GFLOPS will perform almost identically here if their memory subsystems are comparable, because neither is compute-bound. This is the same reason DGX Spark memory bandwidth, not peak FLOPS, sets the ceiling on inference throughput for many model shapes. Then there is the host layer. In a typical PyTorch inference loop, the Python interpreter dispatches each operation, the framework builds and launches the kernel, and control returns to Python before the next op. When individual kernels are small — as they are during autoregressive decode — the per-op dispatch overhead can rival or exceed the compute time itself. We see this pattern regularly: a path that looks compute-heavy on paper turns out to spend most of its time in the interpreter and the framework’s scheduling machinery. Profiling the actual path, rather than reasoning from the spec sheet, is the only way to know. Our note on what “computationally expensive” really means in an inference path walks through where that cost concentrates. How do you tell whether your path is compute-bound or not? This is the question the GFLOPS figure cannot answer for you, and the one you must answer before it becomes relevant. The tool is the roofline model: plot achievable performance against arithmetic intensity, and every workload lands either under the sloped memory-bandwidth roof or under the flat compute-bound ceiling. Here is a decision rubric we use as a first pass before profiling in depth. Compute-bound vs memory- or host-bound: a diagnostic table Signal Points toward compute-bound (GFLOPS matters) Points toward memory-bound (bandwidth matters) Points toward host-bound (glue matters) Operation shape Large, dense GEMMs; big batch sizes Matrix-vector products; single-token decode Many tiny ops per second Arithmetic intensity High FLOPs per byte moved Low FLOPs per byte moved Irrelevant — ops too small to measure What a profiler shows Kernels dominate wall time, near peak Kernels dominate wall time, far below peak Time spent in interpreter / dispatch Effect of faster clock/more FLOPS Latency drops roughly proportionally Little to no change No change Effect of faster memory Little change Latency drops No change Effect of removing Python overhead No change No change Latency drops sharply The value of this table is that it turns a hardware question into a measurement question. You do not guess whether your path is compute-bound — you profile it, attribute the latency to a layer, and then decide whether the GFLOPS number is even in play. A tool like PyTorch’s profiler, or a native profiler such as perf or VTune for the compute kernels, will show you the split directly. If you are planning a port anyway, profiling the Python inference path before a C++ or WASM rewrite is the step that makes the GFLOPS reading meaningful rather than decorative. How do CPU gigaflops compare to GPU compute figures? They are the same kind of number with the same caveat, scaled up. A GPU’s peak FP16 or FP8 figure — often quoted in tens or hundreds of TFLOPS via Tensor Cores — is still a ceiling that only large, well-batched, compute-dense workloads approach. The mechanism that keeps a GPU from its peak is the same one that keeps a CPU from its peak: memory movement and scheduling. The difference is in the arithmetic-intensity threshold. A GPU has enormous compute relative to its memory bandwidth, so the crossover point on the roofline — the intensity above which you become compute-bound — sits much higher. A workload that is comfortably compute-bound on a modest CPU can be firmly memory-bound on a high-end GPU, because the GPU’s arithmetic units are so hungry that the same bytes-per-FLOP ratio no longer keeps them fed. This is precisely why reading DGX Spark benchmarks in terms of utilisation rather than peak FLOPS gives a truer picture of what the hardware will deliver. For a cluster-sizing decision, the same logic scales again: aggregate GFLOPS across nodes tells you the ceiling, and interconnect plus memory bandwidth tell you what fraction of it you will hold. We cover the sizing side in what CPU GFLOPS means for GPU cluster sizing, and the browser-target variant in reading compute throughput for WASM inference. The consistent thread across CPU, GPU, and WASM targets: the compute figure is real and useful as a bound, but it is the ceiling, and your workload’s arithmetic intensity decides how close you get. You can read more about how the compute layer fits into a broader hardware plan on our GPU engineering page. How should GFLOPS inform — not decide — a hardware or port decision? Treat the gigaflops figure as one input among several, and never as the tiebreaker on its own. The sequence that avoids buying compute you cannot use: Profile the current path first. Attribute latency to compute, memory, and host glue. This is the baseline that everything else references. Read the arithmetic-intensity ratio. If it is low, your path is memory- or host-bound, and GFLOPS is not your lever. Only then compare compute figures — and only for the fraction of latency that profiling attributed to the compute layer. Recheck after any change. Removing a Python bottleneck or fusing kernels can shift a path from host-bound to compute-bound, at which point GFLOPS suddenly becomes relevant. The measurable payoff is avoiding a hardware purchase or a port justified by a headline number the workload never exercises. If 70% of your latency is in Python dispatch and memory movement, a machine with 30% more peak GFLOPS improves your latency by roughly the compute share it touches — a small single-digit percentage — not by 30%. That gap is the cost of reading the spec sheet instead of the profiler. FAQ What should you know about gigaflops cpu in practice? A CPU’s gigaflops figure is the product of core count, clock frequency, floating-point operations per cycle per core, and SIMD vector width. In practice it is a theoretical peak — the arithmetic ceiling if every cycle issued a fused multiply-add with no stalls. Real inference rarely runs anywhere near it, because memory movement and scheduling get in the way. What does a GFLOPS figure actually measure, and is it a peak or a sustained number? It measures the maximum rate of floating-point operations the compute units could theoretically sustain, and it is almost always a peak figure, not a sustained one. Peak assumes full clocks, no throttling, perfect operand supply from cache, and no scheduling loss. A sustained figure under a real workload is what actually matters, and it is typically a fraction of peak. Why does a CPU rarely reach its peak gigaflops during real inference? Inference is a sequence of differently shaped operations, and the small or memory-bound ones dominate wall-clock time. Autoregressive decode degenerates into matrix-vector products with low arithmetic intensity, so the bottleneck is memory bandwidth rather than compute. Per-operation dispatch overhead in the Python and framework layers adds further time that FLOPS cannot capture. How do you tell whether your inference path is compute-bound or memory- or host-bound? Profile the path and attribute latency to compute kernels, memory movement, and interpreter/dispatch overhead. Use the roofline model: high arithmetic intensity with kernels near peak means compute-bound; kernels far below peak means memory-bound; time spent in the interpreter means host-bound. The test is which resource, when made faster, actually reduces latency. How does the gigaflops of the compute layer relate to runtime scheduling and Python host overhead? The GFLOPS figure only describes the compute layer, which is often a minority of total latency. Runtime scheduling and Python host glue can dominate when individual kernels are small, as during single-token decode. In those paths, improving GFLOPS changes nothing because the compute units are already waiting on the host and memory. How should GFLOPS inform — rather than decide — a hardware choice or a port decision? Use it as a ceiling for the fraction of latency that profiling attributes to compute, never as a standalone tiebreaker. Profile first, read the arithmetic-intensity ratio, then compare compute figures only for the compute-bound portion of the path. Recheck after any change, since removing a host bottleneck can make GFLOPS relevant where it was not before. How do CPU gigaflops compare to GPU compute figures when attributing where inference latency actually lives? They are the same kind of ceiling with the same caveat, scaled up, and both are limited by memory movement and scheduling rather than the raw figure. A GPU has far more compute relative to its bandwidth, so its compute-bound threshold sits higher — a workload compute-bound on a CPU can be memory-bound on a GPU. In both cases, latency attribution by profiling, not the spec figure, tells you where the time goes. Whether a compute figure actually translates into delivered latency on a ported or accelerated path is exactly what a release-readiness check is meant to validate — and it is a fair test to run before committing to the number on the slide. If you are weighing a port against a headline gigaflops figure, an inference cost-cut sprint starts from the profiling baseline this explainer describes, so the compute layer is only credited for the latency it actually owns.
