When an AI feature is too slow or too expensive to run, CPU vs GPU for AI is rarely the right first question. Start by finding out what limits the workload: arithmetic throughput, memory bandwidth, or the CPU-side pipeline around the model (tokenisation, pre-processing, scheduling and kernel launches). Profile the workload first. Once you know what limits it, you have the context the hardware choice needs.
How much faster is a GPU than a CPU for AI inference?
In the DEEP-GAP study (2026), the same ResNet models ran between 1.92x and 58x faster on NVIDIA T4 and L4 GPUs than on a CPU baseline. The multiple depended on which GPU was used and at what numerical precision.
| Measured GPU-over-CPU result | Condition | Source |
|---|---|---|
| 1.92x | NVIDIA T4, FP32, ResNet-18 at batch 384 | DEEP-GAP, 2026 |
| 5.20x / 4.64x | NVIDIA L4, FP32, ResNet-18 / ResNet-50 | DEEP-GAP, 2026 |
| 8.84x / 8.38x | NVIDIA L4, FP16, ResNet-18 / ResNet-50 | DEEP-GAP, 2026 |
| 58.11x / 58.21x | NVIDIA L4, INT8 with TensorRT, ResNet-18 / ResNet-50 | DEEP-GAP, 2026 |
| 3.6x to 47.4x | V100 across object detection, GAN, text and image classification models | InferBench, 2020 |
Three variables move the number. The first is precision: in the DEEP-GAP results FP16 consistently beat FP32, and INT8 through TensorRT gave the largest gains because narrower data raised Tensor Core utilisation. The second is batch size: speedup rose quickly between batch sizes 1 and 32 and then levelled off as the GPU’s compute resources approached saturation. The third is the model. InferBench found lightweight MobileNet models to be memory-bound on a V100, while heavier CNNs were compute-bound, so the small models could not use the GPU’s arithmetic fully.
A GPU has thousands of cores built for parallel multiply-accumulate work and uses high-bandwidth memory such as HBM to keep them fed. CPUs, by contrast, struggle with the volume of multiply-accumulate operations in neural networks because they have narrower vector units and lower memory bandwidth. Our explainer on why GPUs are needed in AI covers that mechanism in more depth.
A GPU’s peak specification overstates what it delivers on real models. A 2018 study measured 83 Tflops/s from V100 Tensor Cores using cuBLAS GEMM, about 74% of the theoretical 125 Tflops/s, and that was on a large matrix multiply that suits the hardware well. Across six DNNs, a separate benchmark found average Tensor Core utilisation of only 3.9% to 9.2%. This is why GPU performance is not a single number, and why a spec-sheet ratio between two processors predicts very little about your model.
Why a GPU can run your model slower than the CPU
One cause is that the CPU must process metadata such as tensor shapes before launching GPU kernels, and that is a fixed cost. GPU work, on the other hand, grows with batch size, so at small batches the CPU overhead can exceed the GPU runtime and the GPU sits idle between kernel calls. A 2025 study of on-device LLM inference makes the same point: a CPU can run small matrix multiplies directly, while a GPU pays a real cost to launch a kernel for a small workload.
Some operations also ran slower on a GPU than on a CPU. On an Apple M1 running MLX, a 2025 benchmark found the GPU slower than the CPU for Gather (-16%), LeakyReLU (-17%) and ReLU (-29%) in the measured configuration, while it was much faster on Conv2d and Sort. With a discrete GPU, data must also be copied from host memory into accelerator memory, which adds significant latency.
Implementation matters too. The same 2018 Tensor Core study found that a naive CUDA WMMA kernel was no faster than ordinary single-precision GEMM on the CUDA cores. Adding shared memory made it about five times faster. If a GPU result looks wrong, check batch size, launch overhead and transfer cost before blaming the hardware.
When does running inference on a CPU make more sense than paying for a GPU?
In the sources below, CPU inference either beat the GPU or was a viable choice: when requests arrive one at a time and must be answered immediately, when a model of about 1B parameters runs on a device, when latency targets are loose and cost matters more, or when a modern server CPU meets the latency target at a usable batch size. The sources behind each condition:
- Unbatched real-time serving. In one production case, Roblox reported in 2020 that GPUs scale best with batched inputs, while its text classifier ran in a real-time request-response setting where batching only added overhead. After optimisation, its BERT-based service handled over 3,000 inferences per second on a 36-core Xeon server, compared with 500 on a cost-equivalent Tesla V100, which is 6x higher throughput per dollar.
- Small models on device. On an iPhone 15 Pro running llama.cpp, a 1-billion-parameter model reached 17 tokens per second on two CPU threads, compared with 12.8 tokens per second with GPU acceleration. The same 2025 study found that GPUs generally kept higher throughput for models above 1.5B parameters.
- Modern CPUs with matrix extensions. A 2026 CPU-only study measured a Granite Rapids CPU, which has AMX matrix hardware, at about 35 ms median latency for ResNet-50 at batch 8, which is viable for near-real-time inference. On that platform, batching from 1 to 8 gave a 2.9x throughput gain.
- Cost-led deployments with flexible latency. A 2024 survey of LLM infrastructure concludes that multi-core CPUs are a viable option for distributed inference when real-time constraints are loose and cost control is the priority.
There are limits. The same Roblox team found that fine-tuning took over 10x longer on CPU than on GPU. Older CPUs also hit their throughput ceiling early: a legacy four-core Xeon was effectively limited to batch 1 under a sub-200 ms latency target. Thread count matters as well. Throughput was best when threads matched physical or performance cores, and adding threads beyond that point degraded tail latency. For how CPU inference is programmed and tuned, see how CPU programming works for ML inference.
Compute-bound, memory-bound or pipeline-bound: what to check before choosing hardware
The core test is arithmetic intensity: the number of operations a kernel performs for each byte it moves to or from memory. NVIDIA’s performance guide defines memory time as the bytes accessed divided by memory bandwidth, and math time as the operations divided by math bandwidth. A kernel is math-limited on a given processor when its arithmetic intensity is higher than that processor’s ops:byte ratio (math bandwidth divided by memory bandwidth). It is memory-limited when its intensity is lower. A third case also exists: if the workload is too small or not parallel enough to saturate either pipeline, performance is limited by latency.
A fully connected layer applied to a single input vector (batch 1) is memory-limited, even though the same operation becomes math-limited when the matrices are large enough. The roofline model makes the test visual: you plot each layer’s arithmetic intensity against the hardware’s compute and bandwidth ceilings to see which ceiling it hits. An LLM inference survey that applies the roofline model shows why the fix depends on the answer. For a memory-bound layer, quantisation, kernel fusion and larger batches help. For a layer that is already compute-bound, further quantisation of the weights gives no improvement.
Arithmetic intensity is a per-layer property of the model. Three measurements cover the rest of the system:
- Sweep the batch size. Raise the batch size step by step and note where throughput stops rising or latency starts to climb. That saturation point helps tell the compute-bound and memory-bound regimes apart on each device.
- Scale CPU cores. Run the service under load while changing the number of CPU cores. If latency falls as cores are added, the CPU was the bottleneck. If it does not change, the workload is GPU-bound.
- Time each pipeline stage. Put timing probes at the boundary of each stage: pre-processing, transmission, batching, inference and post-processing. This shows which stage limits the request.
| What the profile shows | What it suggests | What it means for hardware |
|---|---|---|
| Arithmetic intensity above the roofline turning point | Compute-bound | Compute is the limit, not memory bandwidth |
| Compute units sit idle while memory access is saturated | Memory-bound | Try quantisation, kernel fusion or larger batches |
| GPU busy only briefly between kernels at small batch | Latency- or launch-limited | Try CUDA graphs or batching |
| Latency falls as CPU cores are added | CPU- or pipeline-bound | Add CPU cores or move pre-processing before adding GPUs |
| End-to-end throughput tracks the pre-processing stage, not inference | Pipeline-bound | Optimise the surrounding stages, not only inference |
We hold that profiling to tell these regimes apart should come before any hardware decision.
When the CPU becomes the bottleneck of a GPU inference service
The CPU becomes the bottleneck when it cannot tokenise, schedule and launch kernels fast enough to keep the GPU busy. This happens under concurrent load, with long contexts, with small batches and in tool-heavy agent workloads. Adding GPUs does not solve it. In a 2026 characterisation of multi-GPU LLM serving on vLLM, giving CPU-starved configurations enough cores reduced time-to-first-token by 1.36x to 5.40x, with no extra GPUs. The authors conclude that fewer than 4 to 8 CPU cores per GPU puts LLM inference at risk of measurable slowdown. They also found that on the A100 and H100 nodes of the production HPC clusters they analysed, the median CPU-to-GPU ratio requested was only about 1 to 2. Tokenisation alone accounted for up to 50% of total latency in long-sequence configurations.
The same pattern appears elsewhere. When vLLM’s developers profiled Llama 3 8B on one H100 in 2024, they found the HTTP API server took 33% of execution time and scheduling 29%, leaving only 38% for GPU execution. Their multi-step scheduling, which spreads CPU overhead across several steps, improved Llama 70B throughput on 4xH100 by 28%. CUDA graphs, suited to CPU-limited networks, gave speedups of up to about 1.7x on two MLPerf workloads.
In computer vision serving, a 2024 study found that non-DNN data processing took up to 56% of request time for a medium-sized image, even in a highly optimised inference server. When pre-processing ran on the CPU, adding more GPUs barely changed performance, because the extra GPUs waited for input. Common image and video pre-processing functions are also being accelerated on GPUs with libraries such as NVIDIA DALI.
Agent workloads push this further. A 2025 CPU-centric study found that tool processing on the CPU took up to 88% of end-to-end latency in tool-dominated agentic workloads. It also found that a faster GPU can move the bottleneck further onto the CPU when tool execution takes about as long as LLM inference. CPU stalls show up as low GPU utilisation. How to read utilisation correctly, and what idle capacity costs, is covered in our piece on maximising efficiency with AI acceleration. The wider point that GPUs are part of a larger system applies directly here.
Where TPUs and NPUs fit in a CPU vs GPU vs TPU decision
TPUs suit large workloads dominated by matrix operations. NPUs suit low-power, mostly quantised inference on client devices.
Google’s Cloud TPU documentation recommends TPUs for models with no custom PyTorch or JAX operations inside the main training loop, for training runs lasting weeks or months, and for large models with large effective batch sizes. The same documentation warns that programs dominated by non-matrix operations such as add, reshape or concatenate will probably not achieve high utilisation of the matrix units. It recommends GPUs for models with many custom operations that must run at least partly on CPUs. Google’s 2017 in-datacenter TPU study reported the TPU as about 15x to 30x faster than its contemporary GPU or CPU, with 30x to 80x better TOPS per watt. TPUs are also subject to memory-bandwidth limits. A 2019 benchmark found TPU v3-8 only 1.5x to 1.7x faster than TPU v2-8 despite about 2.3x higher peak FLOPS, and attributed part of the gap to a smaller 1.5x gain in memory bandwidth.
Microsoft’s Copilot+ PC guidance lists an NPU of 40+ TOPS as a requirement for many Windows AI features. It also notes that many NPUs support only lower-bit integer formats such as INT8. When the preferred execution provider is unavailable, Windows ML falls back to the GPU or CPU. Which processor wins depends on the operation. On an Intel AI PC, a benchmark found the GPU 2.7x faster on LSTM networks and the CPU fastest on simple dot products. In video classification on that platform, the NPU’s latency stayed constant across batch sizes, while the GPU’s improved as the batch grew, so the NPU suited small, memory-bound batches and the GPU large, compute-bound ones.
What published CPU vs GPU benchmarks leave out
MLPerf-style benchmarks measure defined tasks under prescribed rules. MLPerf Inference exists because the number of possible hardware and software combinations makes architecture-neutral, reproducible comparison hard. Its rules are what make results comparable across very different systems. InferBench’s authors note that results from simple, isolated models cannot easily be generalised to identify whether a system is compute-bound or memory-bound.
Four gaps matter most when you apply a published result to your own service. The first is pipeline stages. For most models studied smaller than 5 GFLOPs, non-DNN overheads dominate the request, and even models above 10 GFLOPs spent 16% to 49% of latency on non-DNN functions. The second is batch assumptions. GPUs led CPUs on latency at batch sizes under eight in InferBench, but latency on two GPUs grew much longer at large batches, so the authors advise checking the latency SLO before choosing hardware and batch size. The third is the software stack. The vLLM scheduling and output-processing changes above show how much the serving software alone moves results. The fourth is single-model coverage. A result for one ResNet or one BERT does not tell you how a model with a different arithmetic intensity will behave.
LynxBenchAI measures AI hardware. For the methodological difference between CPU and GPU metrics, see the CPU GPU comparison for system benchmarking. To check your own machine, the practical protocol for benchmarking a PC for AI sets out the steps.
The cheapest fix is often not a different processor
The 2026 multi-GPU serving study priced the fix for CPU starvation directly. Adding 16 vCPUs to a p5.48xlarge instance raised its hourly cost by roughly 1.5%, and for CPU-bound workloads performance scaled almost linearly with the added cores. Results like that are why we hold that a batch sweep, a core-scaling run and stage timings should precede a hardware purchase. Each of those can point to a fix that costs far less than new silicon.
When the profile does point at the hardware, one route is to move the runtime or the target device rather than change the model. Porting AI inference across runtimes and hardware covers that route. For profiling and kernel, memory and parallelism work on the serving stack, GPU performance engineering describes how TechnoLynx approaches it.
FAQ
Is prefill memory-bound or compute-bound?
In LLM inference, the prefill stage is mostly compute-bound and the decode stage is memory-bound, which is why decode usually runs far below the GPU’s compute capacity. With short sequences, however, prefill can itself be memory-bound, and quantisation helps it there.
What are memory-bound operations?
Memory-bound operations are those that perform few operations for each byte they access. Matrix-vector multiplication is one, because it reuses little data, which is why a 2024 benchmark found an NPU with DMA about 2x faster than a GPU on it.
