GPU CPU GFLOPS: What Gigaflops Mean for Inference Latency

A gigaflops CPU figure is a compute ceiling, not a latency forecast: how to calculate peak GFLOPS, what LINPACK measures, and why inference rarely gets…

GPU CPU GFLOPS: What Gigaflops Mean for Inference Latency
Written by TechnoLynx Published on 11 Jul 2026

A CPU’s GFLOPS figure is the number of billions of floating-point operations it can complete per second. The peak figure on a spec sheet is theoretical: floating-point operations per cycle, times cores, times clock frequency. That makes a gigaflops CPU rating a ceiling for the compute layer, not a forecast of inference latency. On many inference paths, latency is set by how many bytes move through memory, or by fixed framework and kernel-launch costs on the host, and peak FLOPS does nothing for either. How latency splits across the layers of a serving stack is covered in what an inference engine is and where latency lives.

What Does a GFLOPS Figure Actually Measure, and Is It a Peak or a Sustained Number?

Two terms get mixed up here. FLOPs is a count of floating-point operations, and FLOP/s is a rate (throughput). A gigaflop is 10^9 of them, a teraflop 10^12. In the LINPACK tradition, the count includes both additions and multiplications, and the operands are assumed to be 64-bit.

Manufacturers quote peak performance, worked out by counting the floating-point additions and multiplications the hardware can complete per cycle and multiplying by the clock rate. The LINPACK authors call that peak “a sort of speed of light”: a limit the manufacturer guarantees no program will exceed.

On a roofline chart, a kernel’s sustained throughput is simply its FLOPs divided by its runtime. LINPACK is the classic measurement: it solves a dense system of linear equations, and because that problem is very regular it reaches a high fraction of peak (its authors say possibly too high for many real applications to match). The measured LINPACK result is usually reported as Rmax, beside the theoretical Rpeak.

In the LINPACK paper’s own table, a 2,530 MHz Pentium 4 had a peak of 5,060 Mflop/s but reached 69% of it on DGEMM and 46% on LINPACK 1000. STREAM TRIAD (for memory bandwidth) and LINPACK (for floating point) are used to measure a platform’s practical limits.

How to Calculate a CPU’s Peak Gigaflops

The working formula is FLOPs per cycle × cores × frequency, where some sources use the maximum processor frequency. WikiChip’s fuller version breaks FLOPs per cycle into instructions per cycle, operations per instruction and FLOPs per operation, then scales up by cores per node and nodes per system. The FLOPs-per-cycle term depends on the microarchitecture. It comes from vector width and the number of fused multiply-add (FMA) units, and a single FMA counts as two floating-point operations.

Microarchitecture (examples) Vector units FP64 FLOPs/cycle per core FP32 FLOPs/cycle per core
Intel Sandy Bridge, Ivy Bridge 1 × 256-bit mul + 1 × 256-bit add (AVX) 8 16
Intel Haswell to Coffee Lake 2 × 256-bit FMA (AVX2) 16 32
Intel Ice Lake, Tiger Lake, Rocket Lake 2 × 512-bit FMA (AVX-512) 32 64
AMD Zen 2, Zen 3 2 × 256-bit FMA (AVX2) 16 32
Arm Neoverse N1, Cortex-A76 2 × 128-bit FMA (NEON) 8 16

All rows in this table come from WikiChip’s per-microarchitecture listing. Skylake server parts carry AVX-512, but the number of FMA units varies by SKU, so check the specific part before using the 32-per-cycle figure.

Romain Dolbeau’s analysis of Intel CPUs finds a factor of 16 between non-vectorised code encoded in SSE (no FMA) and full-vector single-precision AVX code doing two FMAs per cycle. His conclusion is that theoretical peak is no longer a single number: it depends on the instruction set and its encoding, the parallelism, the frequency behaviour and even the compiler options.

Worked Example: One Core, Two Clocks

A study of task-parallel H-LU factorisation reports a single-core AVX2 peak of 59.2 GFLOPS at a 3.7 GHz turbo frequency, and 33.6 GFLOPS at the 2.1 GHz base frequency. The arithmetic matches the table: 16 FP64 FLOPs per cycle × 3.7 GHz = 59.2 GFLOPS, and 16 × 2.1 GHz = 33.6 GFLOPS. At FP32 the same core would have twice the FLOPs per cycle, so 32 × 3.7 = 118.4 GFLOPS peak.

Here is what a well-tuned kernel actually gets. A sequential DGEMM on that core delivered 57.0 GFLOPS at matrix order 150 and 58.9 GFLOPS at order 1,000, which is 96.2% and 99.4% of the turbo peak. That is close to the ceiling, but for a sequential DGEMM on one core. The same authors note that the multi-threaded DGEMM needed much larger problems to reach a useful fraction of peak as cores were added.

Base, Turbo or AVX Clock: Which Frequency Belongs in the Formula?

Use the frequency the cores sustain at your workload’s vector width and core count. Intel specifies four frequencies, two of them new in server-class Haswell: the marked TDP frequency, the single-core turbo frequency, the AVX base frequency and the AVX maximum all-core turbo. Dolbeau concludes that the base frequency is no longer a reliable basis for estimating peak.

Travis Downs measured an Ice Lake i5-1035 with an all-core 512-bit turbo of 3.3 GHz, 89% of its 3.7 GHz single-core scalar frequency, and a Rocket Lake i9-11900K holding 4.8 GHz on all eight cores, 94% of its 5.1 GHz single-core speed. He also points out that licence-based downclocking is only one limit: power, thermal and current limits can cut frequency too, and on his poorly cooled laptop thermal limits arrived within seconds of heavy all-core work. The frequency also shifts with core count and with the arithmetic intensity of the operations, and it can change during execution. GPUs have the same caveat: NVIDIA bases A100 peak rates on the GPU boost clock, and AMD notes that its GPUs lower the clock below boost in dense compute phases, sometimes within a single kernel.

Why GFLOPS Beats Gigahertz, and Where It Stops

Clock speed is only one term in the peak FLOPS product, beside FLOPs per cycle and cores. At the same clock, Intel peak differs by a factor of 16 between non-vectorised SSE code and full-vector single-precision AVX code with two FMAs per cycle. The LINPACK authors also note that the peak race, sold to the public as ever-rising clock rates, left behind more practical metrics such as LINPACK numbers.

GFLOPS has the same weakness as gigahertz, though: neither says whether the arithmetic units can be fed. The original roofline paper by Williams, Waterman and Patterson found that the ridge point (the minimum operational intensity needed to reach peak) predicted performance better than either clock rate or peak performance. Peak GFLOPS improves on the clock figure. It does not replace profiling your workload.

How Do CPU Gigaflops Compare to GPU Compute Figures When Attributing Where Inference Latency Actually Lives?

GFLOPS and TFLOPS differ only by a factor of 1,000. The harder part of comparing them is precision, not units. NVIDIA’s A100 whitepaper lists 19.5 TFLOPS FP32, 312 TFLOPS FP16 on Tensor Cores (624 with structured sparsity) and 624 TOPS for INT8. Those numbers span a factor of 32 on one chip, from FP32 to INT8.

One preprint on CPU inference describes a modern desktop or server CPU as executing roughly 1 TFLOP/s of arithmetic.

For attributing latency, compare compute against bandwidth on each device. The same preprint pairs that roughly 1 TFLOP/s of CPU arithmetic with roughly 50 GB/s from main memory, a machine balance of 20 FLOP per byte. The A100 pairs 19.5 TFLOPS FP32 with 1,555 GB/s of HBM2 bandwidth, about 12.5 FLOP per byte at FP32 (19,500 / 1,555). Both machines need many operations per byte before compute becomes the limit, so a bigger peak number on either one is wasted on a path that streams weights.

Higher peak does not even guarantee lower latency on a GPU. The Framework Tax study found that single-example BERT inference ran slower on an A100 than on a V100, despite the A100’s 2.75× higher peak computational throughput. The same caution applies when reading a cross-generation GPU comparison correctly.

Is My Inference Path Compute-Bound, Memory-Bound or Host-Bound?

A path left of the machine’s ridge point is memory-bound, and one whose latency stays constant regardless of batch size is framework-bound. Those two tests separate the three regimes. Arithmetic intensity is the ratio of FLOPs performed to bytes moved. The roofline model bounds attainable throughput as the lower of peak compute and peak bandwidth × intensity. Below the ridge point a kernel is bandwidth-bound: limited by how fast data moves through memory, not by how fast the cores or GPU SMs can calculate. On a dual-socket Intel Xeon, one analysis puts the minimum intensity for compute-boundness, derived from achievable peak rates, at 9.3 FLOPs per byte.

Single-token decode falls below that line. With 4-bit quantised weights at half a byte each and two operations per weight, a matrix-vector product has an intensity of 4 FLOP per byte, five times below the 20 FLOP per byte CPU machine balance. CPU decode latency is therefore set almost entirely by how many bytes move, not how many operations run. GPUs show the same pattern: Recasens and colleagues (2025) found that decode attention and matmul kernels stay memory-bound across all batch sizes, with more than half of attention kernel cycles stalled on DRAM access. AMD’s ROCm team draws the phase distinction: prefill works on large matrices and is sensitive to achievable FLOPs, while autoregressive decode is sensitive to memory bandwidth. Splitting a workload into those phases is easier when you start by counting tokens to split prefill and decode.

So why does a CPU rarely reach its peak gigaflops during real inference? Single-token decode sits below the ridge point, so memory bandwidth caps it first. Measured bandwidth falls well short of the theoretical figure too: the FBGEMM authors measured a stream triad of 15.6 GB/s per core on a Broadwell machine whose theoretical all-core bandwidth is 76.8 GB/s. Wide vector instructions can pull the clock below the frequency the peak was computed at. The operations around the core matrix multiply cost time as well: FBGEMM’s authors note that quantisation and offset steps, done carelessly, can cancel the gains of lower precision. On the same Ice Lake CPU and the same 4-bit model, llama.cpp decoded 2.9× faster than vLLM’s CPU backend, because vLLM lacked a suitable oneDNN primitive and fell back to dequantise-then-matmul.

A Quick Diagnostic for the Bound Regime

  • Latency stays flat as batch size or sequence length grows: the path is framework-bound or host-bound, and fixed CPU overhead dominates.
  • Arithmetic intensity is below the device’s ridge point: expect bandwidth, not compute, to cap throughput.
  • Kernel launch and queuing time stays constant at small batches, with the GPU under-used: the path is CPU-bound in the launch sense.
  • Latency follows kernel type, kernel size and the device’s FLOP rate: the path is compute-bound, and peak GFLOPS becomes a relevant ceiling.

How Do Runtime Scheduling and Python Host Overhead Add up Against Compute in the Overall Latency?

At low batch sizes they often dominate, so the GFLOPS of the compute layer barely enters the total. Eager frameworks run each layer as a separate operation with its own CPU kernel dispatch, and that fixed “framework tax” is explained in depth in what an inference engine is and where latency lives. The tax covers all non-GPU compute time, including Python interpretation and graph building.

In the Framework Tax study’s measurements of one model on one GPU, latency fell from 10.54 ms in eager PyTorch to 2.82 ms with CUDA Graphs and 2.11 ms with ONNX Runtime plus CUDA Graphs, while the share of SMs active rose from 2.6% to 59%. The hardware did not change; only the runtime did. For small batches, the authors recommend static or ahead-of-time inference runtimes.

Vellaisamy and colleagues (2025) found that for latency-sensitive, low-batch workloads the CPU’s single-thread performance remains critical. A GH200 system stayed CPU-bound up to batch size 32, against batch size 8 for the loosely coupled x86 systems they tested. Its Grace CPU showed a higher time to first token than the x86 hosts, which they attribute to lower CPU performance or a less mature software stack. Kernel fusion and graph capture (torch.compile building CUDA graphs) help in that CPU-bound region and offer little once the GPU is saturated. Even at large batch, host work does not vanish: CPU computation reached up to 30% of execution at batch size 512 for OPT-1.3B in the Recasens study.

How Should GFLOPS Inform, Rather than Decide, a Hardware Choice or a Port?

Use the peak figure to rule hardware out, not to pick a winner. Profile the path first so you know which layer owns the latency, then ask whether more FLOPS would touch that layer at all. Even vendor peak specifications, NERSC notes, may not reflect the power constraints and programming models real applications run under.

When the profile does point at compute, TechnoLynx’s GPU Performance Engineering work covers kernels, memory traffic and profiling on the target hardware.

Question you are settling What the GFLOPS figure tells you What decides it instead
Can this CPU handle the compute layer at all? An upper bound; if required FLOPs ÷ peak already exceeds the latency budget, rule it out Nothing more is needed to rule it out
Will the faster-peak part cut latency? Nothing, unless the path is compute-bound Bound regime from the profile; bandwidth and host overhead
Is a port to a GPU or a new runtime worth it? The ceiling the target could offer the compute share Share of latency owned by compute versus runtime and host glue
How much headroom is left in our kernels? Peak is the wrong reference Achieved versus achievable (DGEMM, LINPACK, STREAM) on the same device
Which device runs our model fastest? Little; FLOPs are a poor latency predictor even when compute-bound Measured latency on the target framework and hardware

The port question deserves particular care. If the profile shows that runtime and host overhead own most of the latency, porting the runtime rather than the hardware may buy more than any jump in peak compute. For measured device behaviour rather than spec-sheet peaks, LynxBenchAI benchmarks throughput, latency and memory across accelerators, from laptop CPUs to data-centre GPUs.

A compute figure only becomes a latency claim once measured on the target framework and hardware. That step belongs in a release-readiness decision, not in procurement. The numbers to take from the profile are the gap between peak and achieved GFLOPS on your own path and the share of latency compute actually owns.

FAQ

What Is TFLOPS vs TOPS?

TFLOPS counts trillions of floating-point operations per second, while NVIDIA quotes the A100’s INT8 and INT4 peaks in TOPS. NVIDIA, for example, quotes the A100’s FP16 Tensor Core peak as 312 TFLOPS and its INT8 peak as 624 TOPS.

What Does exaFLOPS Mean?

An exaFLOPS is 10^18 floating-point operations per second, a million teraflops.

Does LLM Inference Happen on CPUs or GPUs?

Both. On a CPU, single-token decode is bound by memory bandwidth rather than arithmetic, so throughput tracks how fast weights stream from RAM. On a GPU-hosted path, the CPU still drives kernel launches, and at low batch sizes its single-thread speed can limit latency.