CUDA beats a CPU implementation when the workload has enough independent arithmetic per byte moved to hide transfer and launch latency. It loses when the work is branch-heavy, serial, or dominated by host-device copies. That is the whole comparison — and it has very little to do with the FLOPS numbers on either datasheet.
The naive reading of “CUDA vs CPU” is a speed contest in which the GPU delivers the headline multiple quoted in vendor benchmarks. In practice the divergence point is arithmetic intensity and data residency. A kernel that shuttles its working set across PCIe every iteration can be measurably slower than a well-vectorised, multi-threaded CPU implementation of the same maths, even on hardware with an order of magnitude more theoretical throughput.
Which workloads actually favour CUDA?
The useful test is per-workload, not per-hardware. We run it as an entry check before any API or kernel-tuning conversation starts, because the answer decides whether the rest of the work is worth doing at all.
| Signal | Favours CUDA | Stays on CPU |
|---|---|---|
| Arithmetic per byte moved | High — many independent FLOPs per element loaded | Low — one pass, little reuse |
| Data residency | Working set lives on device across many kernels | Data returns to host every iteration |
| Control flow | Uniform across threads | Branch-heavy, divergent, data-dependent |
| Parallel width | Thousands of independent work items | Serial chains, small batches |
| Per-call work | Milliseconds of kernel time | Microseconds — launch overhead dominates |
Two mechanisms erase apparent speedups. The first is host-device transfer: if a copy costs more than the kernel it feeds, the GPU is paying to receive work it finishes instantly. The second is kernel launch overhead, which is fixed per call — a loop launching thousands of tiny kernels spends most of its wall-clock time in dispatch, not arithmetic. Both are invisible in a micro-benchmark that times only the kernel body, which is exactly why so many offload decisions look better on paper than in production.
What a fair CPU baseline looks like
Most claimed GPU wins are measured against a baseline nobody would ship: single-threaded, scalar, cache-oblivious. A defensible comparison uses a CPU implementation that is multi-threaded across available cores, vectorised through AVX or the platform’s SIMD equivalent, and laid out so the hot loop respects cache lines. Compare against that, and the honest speedup is often a fraction of the advertised one — sometimes below 1.
To decide whether a passing kernel is then compute-bound or memory-bound on the device, measure achieved memory bandwidth against device peak. Kernels close to peak bandwidth are memory-bound and will not improve from more arithmetic optimisation; kernels far below it with high occupancy are compute-bound and worth tuning. Nsight Compute reports both, and profiling the transfer timeline alongside the kernel timeline is what turns an assumption into a record.
The practical output is a documented offload decision per workload: measured kernel time versus transfer time, achieved versus peak bandwidth, and end-to-end throughput against the tuned CPU baseline. Workloads that fail stay on CPU and avoid accelerator instance cost. Workloads that pass carry a target speedup the implementation can be held to. Our GPU acceleration and optimisation work treats that record as the entry test rather than a retrospective justification.
There is a downstream consequence teams rarely anticipate. The memory-movement pattern chosen at this stage — whether data stays resident on the device or round-trips to the host — is the same pattern that later resists porting between CUDA, OpenCL, and SYCL. Deciding offload on measurement rather than assumption is therefore also a portability decision, and the cheapest time to make it is before the infrastructure bill starts.
What would your workload’s throughput look like against a CPU baseline someone actually optimised?
