A vendor spec sheet lists 3.5 TFLOPS peak for a CPU, the incumbent GPU lists more, and someone on the team draws the obvious conclusion: the GPU is faster, so the CPU port isn’t worth doing. The conclusion is wrong often enough that it deserves a closer look. Peak GigaFLOPS is a theoretical ceiling — the number the silicon can hit only when every multiply-accumulate unit is fed on every cycle. Real inference almost never runs there. Achieved throughput on a CPU depends on memory bandwidth, cache residency, whether the runtime’s kernels are vectorised for the target, and how the workload scales across threads. The spec-sheet FLOPS figure tells you almost nothing about which of those will bite until you profile the ported model on the real hardware. That gap is not a rounding error. We regularly see ported inference workloads land at a small fraction of the peak FLOPS the chip advertises — and the reasons are structural, not tuning oversights. If you decide a port on nominal GigaFLOPS, you can move to hardware the spec sheet promised was faster and end up slower than where you started. If you decide it on measured before/after throughput against the bottleneck a profiler already named, you are working with the number that actually predicts cost-per-request. What GigaFLOPS on a CPU actually measures A FLOP is one floating-point operation. GigaFLOPS is a billion of them per second; peak GigaFLOPS is the arithmetic ceiling derived from clock speed, core count, SIMD width, and how many fused multiply-add operations each vector unit retires per cycle. For a modern x86 core with AVX-512 and two FMA units, that ceiling is genuinely high — on paper. The word doing the heavy lifting is peak. It assumes the arithmetic units never wait. In practice they wait constantly: for a cache line to arrive, for a dependency to resolve, for a thread to synchronise. That is why the term you want is achieved GigaFLOPS — the operations per second the workload actually sustains — and the ratio of achieved to peak is the FLOPS utilisation. Transformer inference and many convolutional workloads spend a large share of their time moving weights and activations rather than multiplying them, so their utilisation is often well below what a compute-bound microbenchmark would suggest (an observed pattern across the porting assessments we run, not a fixed number that transfers between models). The companion explainer, what GFLOPS on a CPU measures and when it predicts inference speed, walks the definition in more detail; this article is about why the peak figure misleads a porting decision. Why higher headline GigaFLOPS doesn’t predict faster inference The intuitive model — more FLOPS means faster — assumes inference is compute-bound. Much of it is not. When a model’s throughput is capped by how fast weights stream from DRAM into the arithmetic units, adding FLOPS capacity does nothing; the multipliers were already idle waiting on memory. This is the single most common reason a spec-sheet upgrade fails to deliver, and it is why memory-bandwidth-bound inference workloads need to be diagnosed before hardware is chosen, not after. There are four caps that separate peak from achieved on CPU inference, and any one of them can dominate: Cap What it limits How it shows up in a profile Memory bandwidth How fast weights/activations reach the cores High stall cycles on memory; achieved FLOPS flat as you add threads Cache behaviour Whether the working set fits in L2/L3 Throughput cliff when model or batch size crosses a cache boundary Vectorisation (AVX / AVX-512) Whether kernels use the wide SIMD units at all Scalar or narrow-vector hot loops; utilisation stuck near a low fraction of peak Thread scaling Whether extra cores add throughput Speedup curve flattens or regresses past N threads (NUMA, contention) A chip can have a magnificent peak FLOPS number and still lose on every one of these. Conversely, a CPU with a lower peak can win if its memory subsystem keeps the cores fed and the runtime is compiled to use its vector units. The order of these caps matters too: fixing vectorisation on a workload that is already memory-bound buys nothing, because the bottleneck simply moves back to DRAM. This is the same peak-versus-achieved distinction that GPU profiling makes visible — the GPU profiling methodology for measuring achieved versus peak FLOPS is the discipline that separates a real throughput gain from a spec-sheet illusion on any target, CPU or accelerator. Does the runtime’s compilation for the target matter this much? Yes — often more than the hardware choice itself. A model exported to ONNX Runtime, oneDNN, or llama.cpp will only touch AVX-512 if the kernels were compiled with the right instruction set enabled and the runtime detects the target at load time. A generic build compiled for a baseline instruction set can leave half the silicon’s arithmetic width unused, and no amount of headline GigaFLOPS on the spec sheet recovers that. The compiler flags decide this: what -O2, -O3, and -march actually mean for a port is not incidental to CPU inference throughput — a -march=native recompile is sometimes the entire difference between a losing port and a winning one. The same logic applies to the preprocessing path, not just the model. SIMD porting of preprocessing kernels — image decode, resize, tokenisation — can be the hidden throughput drag once the model itself is vectorised, because a serving path is only as fast as its slowest stage. Measuring the model in isolation and declaring victory is a classic way to be surprised in production. How to measure effective GigaFLOPS instead of trusting the spec sheet The measurement that matters is the one taken on the target hardware, running the actual ported model, under a load pattern that resembles production. A synthetic FLOPS benchmark tells you the ceiling; it does not tell you your workload’s achieved fraction of it. Here is the sequence we use when a CPU target or a runtime recompile is on the table. Establish the baseline. Measure throughput, p95 latency, and cost-per-request on the current target before touching anything. Without a before, there is no after. Profile the ported model on the real CPU. Use a profiler that reports FLOPS utilisation and memory-stall behaviour — Intel VTune, Linux perf with the right counters, or the runtime’s own instrumentation. You are looking for which of the four caps dominates. Read the utilisation ratio, not the raw number. Achieved-over-peak tells you the headroom. A workload at a low fraction of peak has room to gain from vectorisation or memory tuning; one already near its achievable ceiling does not. Attribute the bottleneck. Memory-bound, cache-bound, under-vectorised, or thread-limited — each points to a different fix, and fixing the wrong one wastes the porting budget. Recompile and remeasure against the same load. Enable the target instruction set, re-profile, and compare like-for-like. The delta is the real throughput gain, expressed in the metrics the audit baselines. That achieved-FLOPS-utilisation reading is exactly what the profiler stage of our [inference cost-cut assessment](Inference Cost-Cut Pack) produces — the finding that tells you whether a CPU target’s headline GigaFLOPS is even reachable before any porting work begins. It is the difference between committing engineering effort on a number a spec sheet promised and committing it on a number profiling confirmed. The broader engagement design behind that assessment lives on our consulting services page. When a lower-peak CPU still wins on cost-per-request Peak FLOPS is a per-second capacity figure; cost-per-request is what a finance owner signs. They diverge whenever the workload is memory-bound, the batch size is small, or latency budgets prevent the batching that would raise utilisation. A CPU target with modest peak GigaFLOPS but strong memory bandwidth and a well-vectorised runtime can serve a memory-bound model at a lower cost-per-request than a higher-spec alternative that is idling its arithmetic units. The unit economics decide it: achieved GigaFLOPS only matters insofar as it moves the cost-per-request KPI, and that KPI is set upstream — the unit-economics framework for AI infrastructure is where the cost target a port must beat gets defined. Achieved throughput is a means; cost-per-request is the end. FAQ How does gigaflops cpu work? GigaFLOPS on a CPU is a billion floating-point operations per second, and the peak figure is the arithmetic ceiling derived from clock speed, core count, SIMD width, and FMA units per cycle. In practice a workload rarely reaches that ceiling because the arithmetic units spend time waiting on memory, cache, and synchronisation. The number that matters for inference is achieved GigaFLOPS — what the workload actually sustains on the real hardware. What is the difference between peak (theoretical) GigaFLOPS and the throughput a CPU actually achieves during inference? Peak is the theoretical maximum assuming every arithmetic unit is fed on every cycle; achieved is what the workload sustains once memory, cache, and threading realities intervene. The ratio between them — FLOPS utilisation — is often well below peak for transformer and convolutional inference, which spend much of their time moving data rather than computing. Peak tells you the ceiling; only achieved throughput predicts cost-per-request. Why does higher headline GigaFLOPS not reliably predict faster inference when porting a model to a CPU target? Because much inference is memory-bound rather than compute-bound: when throughput is capped by how fast weights stream from DRAM, adding FLOPS capacity does nothing since the multipliers are already idle waiting. A chip with higher peak GigaFLOPS can still lose to one with better memory bandwidth and a vectorised runtime. Only profiling on the target reveals which cap dominates. Which factors — memory bandwidth, cache, AVX/AVX-512 vectorisation, thread scaling — cap achieved FLOPS on CPU inference? All four can dominate, and any one is enough to keep achieved throughput far below peak. Memory bandwidth limits how fast data reaches the cores; cache behaviour causes throughput cliffs when the working set overflows L2/L3; vectorisation determines whether the wide SIMD units are used at all; and thread scaling breaks down under NUMA effects or contention. The fixes differ per cap, so attributing the right one before acting is essential. How do we measure effective GigaFLOPS utilisation for a ported model instead of trusting the spec sheet? Baseline the current target’s throughput, p95 latency, and cost-per-request first, then profile the ported model on the real CPU with a tool that reports FLOPS utilisation and memory-stall behaviour, such as Intel VTune or Linux perf. Read the achieved-over-peak ratio to find headroom, attribute the dominant bottleneck, then recompile for the target and remeasure against the same load. The like-for-like delta is the real gain. When does a CPU target with lower peak GigaFLOPS still win on cost-per-request versus a GPU or higher-spec CPU? When the workload is memory-bound, the batch size is small, or latency budgets prevent the batching that would raise utilisation on a higher-spec device. A CPU with modest peak but strong memory bandwidth and a well-vectorised runtime can serve such a model at lower cost-per-request than a higher-spec alternative idling its arithmetic units. Cost-per-request, not peak FLOPS, is the deciding metric. How do we use profiler findings to decide whether a CPU port will reach the throughput the GigaFLOPS figure implies? Profiler findings show the achieved FLOPS utilisation and whether the workload is memory-bound before any porting begins — telling you whether the spec sheet’s headline number is even reachable on that workload. If profiling shows the workload will stay memory-bound on the target, the peak GigaFLOPS is irrelevant and the port likely won’t deliver. If it shows headroom the target can unlock through vectorisation or bandwidth, the port is worth scoping. The question worth asking before the port The useful question is never “which target has more GigaFLOPS.” It is “what is my workload’s achieved fraction of peak, and which cap is holding it there.” Answer that with a profile on the real hardware and the porting decision becomes an engineering call instead of a spec-sheet guess. Get it wrong and you inherit a familiar failure class — a hardware move justified by a theoretical ceiling the workload never touches, which the inference cost-cut assessment’s profiler findings would have flagged before a line of porting code was written.
