The CPU GFLOPS figure on a spec sheet is a theoretical ceiling. It multiplies cores by SIMD lanes, by two for a fused multiply-add, by the number of FMA units and by the clock, and it assumes every floating-point unit finishes an FMA on every cycle. A Python inference loop almost never meets that condition, so the figure you measure will be a fraction of the peak, and that fraction is the useful number. To read CPU GFLOPS well, work out the ceiling honestly, measure achieved throughput with hardware counters and a profiler, and then decide whether the gap comes from interpreter overhead, memory bandwidth or a real compute limit. Do that before you change any code or hardware.
From Paper Peak to Measured Rate on a CPU
GFLOPS means billions of floating-point operations per second. TOP500 says theoretical peak “is based not on an actual performance from a benchmark run, but on a paper computation”. It is the number manufacturers usually quote, and it is an upper bound that programs will not exceed. In practice you deal with two numbers: the theoretical peak and the measured rate your workload reaches. Divide the second by the first and you have the figure worth discussing.
When the ratio is low, the reflex is to blame the model or the chip, and both are usually wrong. TOP500 itself lists what achieved performance depends on: the application, the algorithm, the problem size, the language, the implementation, how much effort went into optimisation, the compiler, the operating system and the hardware. The spec-sheet figure is a paper computation from the hardware alone.
How Is Peak CPU GFLOPS Calculated from Cores, SIMD Width, Clock and FMA?
The Berkeley Roofline report defines peak in-core performance as unrolled, vectorised FMAs running back to back. The bound is 2 × cores × frequency × SIMD width ÷ SIMD throughput. The factor of two is there because an FMA (d = a × b + c) counts as two floating-point operations per instruction. SIMD width ranges from 1 operation per instruction for scalar code to 16 for single precision in a 64-byte AVX-512 vector. Romain Dolbeau’s analysis of Intel CPUs finds a factor of 16 between non-vectorised SSE code (which has no FMA) and full-width single-precision AVX code with two FMAs per cycle. So “the peak” depends on the instruction set before you even consider the clock.
Worked Example: One Skylake Xeon Socket
The NASA HECC documentation for the Electra cluster gives enough detail to check the formula end to end:
- A 512-bit register holds 8 doubles. Two FMA units, each doing 2 operations per lane, give 8 × 2 × 2 = 32 double-precision operations per cycle per core. NASA lists the same figure, against 16 for Broadwell’s 256-bit units.
- The Xeon Gold 6148 has 20 cores at a base clock of 2.4 GHz.
- Peak per socket at base clock is 20 × 32 × 2.4 = 1,536 double-precision GFLOPS.
- As a check, NASA’s 89,280 Skylake cores × 32 × 2.4 GHz comes to about 6,857 TFLOPS, the total NASA lists.
- With the 3.70 GHz maximum turbo frequency, the same formula gives 20 × 32 × 3.7 = 2,368 GFLOPS.
The Clock in the Formula Is Where Peak Figures Go Wrong
Dolbeau notes that recent Intel CPUs have a different base frequency when running AVX instructions, “the very instructions we are trying to count”. On an E5-2695 v3 he measured 256-bit floating-point AVX at 3.0 GHz on a single thread, while other instructions ran at 3.3 GHz. From six active cores upward, the figures were 2.6 and 2.8 GHz. Travis Downs documents three licence tiers on an AVX-512 part: 3.2 GHz for non-AVX code, 2.8 GHz for AVX and 2.4 GHz for AVX-512. He also found that a transition into wide instructions cuts the dispatch rate to a quarter for a voltage ramp of typically 8 to 20 μs. When a frequency change is also needed, the core then halts for 8 to 10 μs. Turbo has the same problem: compute-bound kernels push the CPU to maximum turbo, but that frequency cannot be held indefinitely and eventually drops to stay within power and thermal limits.
A peak worked out at base or turbo clock overstates what wide-vector code sustains, so use the frequency your instruction mix actually runs at. Dolbeau concludes that theoretical peak “is no longer a single number”. It depends on instruction set, encoding, parallelism, frequency scaling and even the compiler options.
What Is the Difference Between Peak and Achieved GFLOPS, and Why Does the Gap Matter for Inference?
Achieved GFLOPS is the number of floating-point operations the workload actually completes, divided by wall-clock time. Across the kernels cited below it ranges from 0.35% to about 90% of peak. A well-tuned GotoBLAS reaches up to around 90% of peak for Level 3 kernels. Czaja et al. measured direct convolution at 86.72% of peak on one core with a blocked NCHW16C layout and 78.01% when multi-threaded. With a plain NCHW layout the figures were 48.73% and 45.68%. Average pooling in the same study reached 0.35% with NCHW and about 14.8% with NCHW16C. Williams, Waterman and Patterson report that conventional sparse matrix implementations often run below 10% of peak on uniprocessors.
This matters for inference because “the CPU is too slow” is a claim about hardware, while the ratio shows what kind of shortfall you actually have. A kernel at 85% of peak has little headroom left on this chip, so faster arithmetic would need different hardware or a different precision. Before anyone prices a hardware upgrade or a full port, the ratio on the hot path is the cheapest evidence available.
How Do We Measure Achieved GFLOPS on a Real Python Inference Path?
You need two instruments: a Python-level profiler, which tells you where the time goes, and hardware counters, which tell you how many floating-point operations actually ran in that time. Run them in this order:
- Find the hot path with a sampling profiler. Python’s profiling documentation recommends the statistical
profiling.samplingprofiler for most analysis, noting its minimal overhead. Its--mode gilshows GIL contention,--mode cpuexcludes I/O waits, and--nativeshows time spent in C extensions, “helping distinguish Python overhead from library performance”. High direct time in a function means the expensive work happens inside it rather than in the functions it calls. - Stabilise the machine before timing anything. Fix the processor clock so repeated runs are comparable. Isolate cores from the scheduler, pin the process to them and turn Hyper-Threading off to avoid interference and known counter errata. Leave out the first call, because optimised BLAS libraries do initialisation work on first invocation that inflates its time.
- Count floating-point instructions with the PMU. Intel’s
FP_ARITH_INST_RETIREDevents count one computation per scalar instruction and eight per 256-bit packed single-precision instruction, and FMA instructions count twice. Czaja et al. read these events withperfin two runs, one with the kernel and one that only initialises data, and subtracted the second to remove framework overhead.likwid-perfctrreads counters on x86, ARM and POWER and pins threads with-C. For repeatable workloads, group the counters rather than multiplexing them to get self-consistent results. - Use a measured peak as the denominator. Czaja et al. measured peak with runtime-generated assembly: a stream of AVX FMA instructions built to avoid dependency chains. A peak measured on the machine, at the frequency it really runs, is a fairer denominator than the spec-sheet figure.
- On a PyTorch path, use operator estimates carefully.
torch.profilercan estimate FLOPs from a formula, but only for matrix multiplication and 2D convolution.
The counters have one limit you need to keep in mind. FLOP events do not count data movement or max operations, so Czaja et al. call PMU-based work counting unsuitable for ReLU, max-pooling and reorders. On those operators a low GFLOPS reading reflects uncounted work, not a broken kernel.
When Is a Low GFLOPS Ratio Python Overhead, and When Is It a Memory or Compute Ceiling?
Two hardware limits bound a kernel’s performance, and the Roofline model combines them. The formula is attainable GFLOPS = min(peak floating-point performance, peak memory bandwidth × operational intensity), where operational intensity is floating-point operations per byte of DRAM traffic. If a kernel’s intensity puts it under the flat part of that roof it is compute-bound. If it falls under the sloped part it is memory-bound. The ridge point where the two lines meet is the minimum intensity needed to reach peak. You can measure the bandwidth term with McCalpin’s STREAM benchmark, which measures sustainable memory bandwidth and publishes machine balance.
On the Skylake socket above, NASA’s spec bandwidth of about 128 GB/s per socket puts the ridge at 1,536 ÷ 128 = 12 flops per byte at base clock. A kernel needs that much arithmetic per DRAM byte before the FPUs become the limit. For comparison, Williams et al. found x86 ridge points of 4.4 and 6.7 while their 16 kernel-machine pairs had intensities from 0.25 to 1.64, with a median of 0.60. On the Xeon and Opteron, 15 of the 16 limiting ceilings were memory-bound.
Horace He defines overhead as time spent on anything other than moving tensors or computing, including time in the Python interpreter and the framework. In his local benchmark Python managed 32 million additions per second. That is 0.032 GFLOPS, against a CPU rated at some hundreds of GFLOPS. His test for this regime is to increase the data size: overhead does not grow with problem size, so if doubling the batch adds only a little runtime, the path is overhead-bound. Pair that test with the sampling profiler’s --native and --mode gil views to see whether the time is in Python frames or in compiled code.
For interpreter overhead, Numba and Cython both significantly improved on CPython in one study. In a study by Milla et al. of an N-body code on two 28-core Xeon Platinum 8276s, explicit Cython typing alone improved performance by 547.7× on average. Their final Numba and Cython versions averaged 1018× and 1050× over the best CPython version, peaking at 1524 and 536 GFLOPS in single and double precision. PyPy did not improve on CPython plus NumPy, because the GIL stopped it parallelising. The Cython documentation names the two costs that typing removes, Python-level array lookups and C/Python type conversion, and reaches about 6300× over pure Python in its tutorial. Our article on when a Cython extension closes the inference gap without a full port takes that decision further.
Diagnostic Table: Reading the Gap
| What the measurement shows | Likely regime | Where the recoverable headroom is |
|---|---|---|
| Runtime barely grows when data size doubles; sampling profiler shows direct time in Python frames or GIL contention | Interpreter or framework overhead | Typed Cython or Numba for the hot loop; tracing to remove dispatch flexibility |
| Achieved GFLOPS tracks bandwidth × intensity; top-down analysis puts most cycles in the backend (cache and DRAM access) | Memory-bound | Data layout, cache blocking, fewer intermediate copies |
| Achieved GFLOPS close to a measured FMA peak | Compute-bound | Lower precision (float32 gave up to 2.8× in Milla et al.) |
Our position is to profile before optimising. On a CPU inference path we would start with the scale-the-data test and a sampling profile, because they are cheap and they decide which row of the table you are in. If the gap turns out to be a genuine compute or memory ceiling rather than overhead, the next question is whether a port pays off. Porting AI inference to another runtime or hardware without a model swap covers that question, and TechnoLynx’s performance and porting assessment sizes that decision before you commit.
What Does GFLOPS Tell You That Latency or Throughput Alone Do Not, and Where Does It Mislead?
Latency and throughput tell you how fast the work completes. GFLOPS against peak tells you how much of the machine that work uses, and that is what separates “this can go faster on the same chip” from “this chip is the limit”. Horace He treats achieved FLOPS as a percentage of peak as a direct measure of compute-boundness: at 80% you are at least 80% compute-bound.
The same number misleads in three ways. First, a higher ratio does not guarantee faster code. In Czaja et al., Winograd convolution ran at only 31% of peak yet was the fastest of the three convolutions they compared. Instructions per cycle has the same flaw: a program can get faster while its IPC drops, simply by executing fewer instructions. Second, FLOP counters ignore data movement and max operations, so they understate the work in layers such as ReLU, max-pooling and reorders. Third, a spec-sheet peak is not an inference speed: TOP500 warns that benchmark figures are reference points, not measures of total system performance, unless their correlation to the workload has been shown. For how gigaflops relate to response time specifically, see what gigaflops mean for inference latency. For the wider spec sheet, what each CPU spec number means for AI workloads covers the rest.
Which Metrics Should Drive GPU Cluster Compute-Versus-Interconnect Sizing?
Size GPU clusters on achieved per-device throughput for the target workload and on the measured share of time spent communicating as node count grows. CPU GFLOPS should not drive that decision. CPU GFLOPS is a poor proxy at scale because it describes neither the accelerators doing the arithmetic nor the interconnect that increasingly sets the pace. A study of a hybrid CPU-GPU N-body code on Tesla M2070 nodes shows the shift. The force calculation fell from about 100% of execution time on one node to 75% on 32 nodes, while MPI communication rose from 0.2% to 10%. Efficiency held at about 0.80 on 128 nodes for 8M particles. When each GPU had too little work, though, it could not hide memory latency and communication. The study’s own peak formula is described as “a pure ideal value” that ignores memory latency and communication. The code reached about 400 GFLOPS per GPU, around 40% of the claimed peak, which the authors considered very good.
What CPU GFLOPS means for GPU cluster sizing goes through that sizing decision in depth. For measured AI hardware benchmark results rather than vendor peaks, see the LynxBenchAI AI hardware directory. Once the bottleneck has moved to the GPU, GPU performance engineering covers kernels, memory and profiling on the target hardware.