A team looks at a spec sheet, sees a CPU rated at some hundreds of GFLOPS, runs their Python inference loop, and measures a small fraction of that. The reflex is to blame the model or the chip. Both are usually wrong. The number on the spec sheet is a ceiling that assumes every floating-point unit is fed a fused multiply-add on every clock cycle — a condition your inference path almost never meets. The gap between that theoretical peak and what you actually achieve is the most useful diagnostic you have, and most teams never read it. Peak GFLOPS is not a target. It is an upper bound derived from arithmetic that has nothing to do with your workload. Treating it as an achievable throughput leads to the wrong conclusion at the worst moment — right before someone signs off on new hardware or a full rewrite to fix a problem that was never in the hardware to begin with. How does CPU GFLOPS work, and what does it actually mean? GFLOPS — billions of floating-point operations per second — is a rate. Peak (theoretical) GFLOPS is what the silicon can do if every core’s vector units are saturated with fused multiply-add operations continuously. Achieved GFLOPS is what your program measurably extracts from those units over a real run. The peak figure comes from a clean formula: Peak GFLOPS = cores × SIMD lanes × 2 (FMA counts as two FLOPs) × clock (GHz) Work an example with explicit assumptions. Take an 8-core CPU at 3.5 GHz with AV2-256 (AVX2) delivering 8 single-precision lanes per FMA unit, two FMA units per core: 8 cores × 8 lanes × 2 FMA units × 2 (mul+add) × 3.5 GHz = 8 × 8 × 2 × 2 × 3.5 = 896 peak single-precision GFLOPS That 896 is a hardware fact under the stated assumptions — the kind of figure you can attribute to a vendor’s published SIMD width and clock specs. It is also a number your Python inference loop will essentially never approach. The whole point of reading GFLOPS as a diagnostic is understanding why not, and what the shortfall tells you. Why the gap between peak and achieved matters more than either number Latency and throughput tell you how slow the path is. The achieved-to-peak ratio tells you why, and that is the difference between a fixable problem and a hardware wall. Consider three CPUs all measuring the same 40 ms per inference. One is hitting 60% of peak GFLOPS and is memory-bandwidth bound. One is hitting 8% of peak because a hot loop lives in interpreted Python and the floating-point units are idle most of the time. One is hitting 55% and is genuinely compute-bound. Latency alone makes them look identical. The ratio makes them three completely different engineering decisions. This is the same reasoning that applies when reading GPU throughput as a compute-fit signal rather than a peak-FLOPs promise — the spec number is a ceiling, and the operationally relevant question is what fraction of it your real workload sustains under load. On the CPU side the story is the same but the failure modes differ, because the interpreter itself can stand between your code and the FPU. A low achieved-to-peak ratio has three broad causes, and telling them apart is the entire job: Interpreter / glue overhead — the FLOPs are queued behind Python object dispatch, reference counting, and per-element loop overhead. The FPU sits idle. This is a software-shaped problem. Memory-bound stalls — the arithmetic units are starved because operands cannot arrive fast enough; you are bounded by cache misses or DRAM bandwidth, not compute. Roofline reasoning applies here. Genuine compute ceiling — vector units are near-saturated with useful FMAs and you are close to what the chip can do. Only here is a hardware upgrade or a fundamentally different algorithm the honest answer. How do you measure achieved GFLOPS on a real Python inference path? You need two things: a count of floating-point operations the workload should perform, and a wall-clock time for the region doing them. Achieved GFLOPS is the first divided by the second (in nanoseconds-to-seconds terms). The count usually comes from the model’s known FLOP budget — for a dense layer it is roughly 2 × input × output per token; convolution and attention have their own well-documented formulas. In practice, hardware performance counters give the more honest reading. The tools we reach for regularly: Tool What it reveals Reads toward Linux perf stat Retired FP/SIMD instructions, cache-miss rate, instructions-per-cycle Whether FPU is busy or stalled Intel VTune / toplev Top-down breakdown: front-end bound, memory bound, core bound, retiring Which of the three causes dominates cProfile + line_profiler Where wall-clock time is spent in Python Interpreter-overhead share py-spy (sampling) Native vs Python stack time with no instrumentation cost Time trapped in the interpreter Roofline plot (arithmetic intensity vs bandwidth) Memory-bound vs compute-bound boundary Whether you are under the bandwidth roof The pattern we see across profiling passes is diagnostic, not benchmarked as a fixed rate: when py-spy shows most samples in Python frames rather than in a native BLAS or NumPy call, the FPU is not your problem — the interpreter is. When perf stat shows a high cache-miss rate and low instructions-per-cycle while retiring few FP instructions, you are memory bound. When you retire near the theoretical FP-instruction rate, you are close to the ceiling. This is the same profiling discipline we describe in profiling the Python inference path before a C++ or WASM port, where the achieved-vs-peak read is one input into the port decision, not the whole decision. When is a low ratio a Cython-shaped problem — and when is it a real ceiling? This is the divergence point, and it decides whether you write a compiler annotation or cost a hardware refresh. If the achieved-to-peak ratio is very low — single-digit percent — and a sampling profiler shows the hot loop trapped in interpreted Python (per-element iteration, dynamic dispatch, boxing), the FPU never sees most of the work. That is not a hardware limit. Rewriting the hot loop as a typed Cython extension compiled with the right optimization flags — -O2, an appropriate -march, and vectorization enabled — can move the ratio up dramatically because it lets the same arithmetic actually reach the vector units. The chip was always capable; the glue was in the way. If the ratio is moderate and the top-down breakdown says memory bound, Cython will not save you. You are bandwidth-limited, and the fix is data-layout work: blocking for cache, improving locality, reducing precision to shrink the working set, or restructuring the algorithm to raise arithmetic intensity. A faster loop that still stalls on memory buys nothing. If the ratio is already high and you are retiring FMAs near peak, you have found a genuine compute ceiling. Now — and only now — is new hardware or a different accelerator a defensible line item. Decision rubric: reading the achieved-to-peak ratio Achieved-to-peak ratio Profiler signal Diagnosis Action < ~10% Time dominated by Python frames (py-spy) Interpreter / glue overhead Cython/native the hot loop first; re-measure ~10–40% High cache-miss rate, low IPC, memory-bound in top-down Bandwidth / cache starvation Cache blocking, data layout, lower precision ~40–70% Retiring FMAs, some memory bound Mixed; partly optimized Targeted fusion, SIMD width, minor layout work > ~70% Near FP-instruction ceiling, retiring dominant Genuine compute ceiling Consider hardware/algorithm change — justified now Read these as diagnostic bands from profiling practice, not published benchmark thresholds; the exact boundaries shift with workload and microarchitecture. The value is the ordering: rule out interpreter overhead before you conclude the hardware is slow. What GFLOPS tells you that latency and throughput hide Latency answers “how long does one inference take.” Throughput answers “how many per second.” Neither tells you where the time went. Two systems can post identical latency while one wastes 90% of its floating-point capability on interpreter overhead and the other is genuinely maxed out. Only the achieved-to-peak ratio separates them, and that separation is what turns “the CPU is too slow” into a measurable, actionable ratio. GFLOPS also misleads if you read it naively. A high achieved-GFLOPS number is not automatically good — a memory-bound kernel can post respectable GFLOPS while leaving obvious algorithmic savings on the table, and a workload with low arithmetic intensity has a low useful GFLOPS ceiling no matter how fast the chip is. The number is a diagnostic in context, not a scoreboard. This is why we treat it alongside the sizing question rather than in isolation — the companion piece on CPU GFLOPS and GPU cluster sizing covers where the same figure feeds capacity planning rather than a port decision, and the WASM-inference view of CPU throughput covers what changes when the runtime is a browser sandbox. FAQ How does cpu gflops actually work? GFLOPS is a rate of billions of floating-point operations per second. In practice there are two figures that matter: peak (theoretical) GFLOPS, the ceiling the silicon can reach if its vector units are fully fed, and achieved GFLOPS, what your actual workload extracts from those units. The spec sheet quotes peak; your inference path lives at achieved, and the ratio between them is the diagnostic. What is the difference between peak (theoretical) and achieved GFLOPS, and why does the gap matter for inference? Peak is an arithmetic upper bound assuming continuous fused multiply-adds on every lane; achieved is the measured rate under a real workload. The gap matters because it tells you why the path is slow — interpreter overhead, memory-bound stalls, or a genuine compute ceiling — where latency and throughput alone cannot distinguish those cases. That distinction decides whether you write software or buy hardware. How is peak CPU GFLOPS calculated from cores, SIMD width, clock, and FMA? Multiply cores × SIMD lanes × FMA units per core × 2 (an FMA counts as a multiply and an add) × clock in GHz. For example, an 8-core 3.5 GHz chip with AVX2 (8 single-precision lanes) and two FMA units per core computes to roughly 896 peak single-precision GFLOPS. This is a hardware fact under stated assumptions, not a throughput you should expect a Python loop to reach. How do we measure achieved GFLOPS on a real Python inference path, and what tools reveal the achieved-to-peak ratio? Divide the workload’s known FLOP count by the wall-clock time of the region performing it, and corroborate with hardware counters. perf stat exposes retired FP instructions and cache-miss rate, Intel VTune’s top-down view classifies the bottleneck, and py-spy or cProfile reveal how much time is trapped in the interpreter versus native code. Together they give you the ratio and its cause. When is a low GFLOPS ratio caused by Python interpreter overhead versus a memory-bandwidth or compute ceiling? If the ratio is very low (single-digit percent) and a sampling profiler shows time dominated by Python frames, the FPU is idle behind glue — a Cython-shaped problem. If a top-down profile says memory bound with high cache misses, it is bandwidth or cache starvation, fixed by data layout not by a faster loop. If you are retiring FMAs near peak, it is a genuine compute ceiling where hardware or algorithm change is finally justified. What does the GFLOPS number tell us that latency or throughput alone does not, and where does it mislead? It tells you where the time went — two systems with identical latency can have completely different achieved-to-peak ratios and therefore completely different fixes. It misleads when read as a scoreboard: a high achieved-GFLOPS kernel can still be memory bound and wasteful, and a low-arithmetic-intensity workload has a low useful ceiling regardless of chip speed. Read it as a diagnostic in context, never as a standalone quality score. Reading achieved-vs-peak GFLOPS is one profiling step inside a larger port decision. When the ratio says interpreter overhead, the recoverable headroom often justifies a Cython annotation long before any hardware upgrade or full rewrite gets costed — which is exactly the gap the [inference cost-cut pack](Inference Cost-Cut Pack) is built to close, and the porting-and-performance assessment behind it is what turns a measured ratio into a Cython-versus-full-port call. The question worth carrying forward: before anyone concludes the CPU is the bottleneck, has the interpreter been ruled out with a number?
