The CPU is not the slow default you escape from. It is a compute device with an execution model — cores, SIMD lanes, a cache hierarchy, and finite memory bandwidth — and until you understand that model you cannot say whether your profiled bottleneck is one a CPU already handles well. Teams weighing an inference target routinely reach for a GPU offload or a portable runtime like WASM before they can answer a simpler question: where does my inference path actually spend its time on the hardware I already have?

That gap is where money leaks. A port to WASM or an accelerator moves cost around, and if you have not attributed the bottleneck first, it can move that cost without moving the thing that was actually slow. We see this pattern regularly on platform teams: a latency number gets blamed on “the CPU being slow,” a runtime port ships, and the profiled hotspot is still the profiled hotspot — now with a rewrite to maintain.

What’s worth understanding about CPU programming first?

“CPU programming” for inference is not really about writing assembly. It is about arranging your computation so that four physical properties of the processor work for you instead of against you: how many cores you use, how wide each core’s arithmetic is, how well your data fits the cache hierarchy, and whether you are feeding the arithmetic units fast enough over the memory bus.

Modern CPUs execute one instruction across many data elements at once through SIMD — Single Instruction, Multiple Data. On x86 that means SSE (128-bit registers), then AVX2 (256-bit), then AVX-512 (512-bit); on Arm it is NEON and SVE. A single AVX2 fused-multiply-add can process eight single-precision floats per instruction per cycle per port. If your matrix multiply runs scalar — one float at a time — you are leaving most of that width unused, and the difference between the two paths is not a tuning nicety. It is often the whole story of why a CPU “feels slow.” We walk through exactly what widening buys in what SIMD width buys a ported inference path.

Above the single core sits threading. A well-parallelized inference kernel spreads across physical cores with something like OpenMP, oneTBB, or the thread pool inside a runtime such as ONNX Runtime or OpenVINO. But threading has a ceiling set by memory bandwidth and by the parallel fraction of the workload — adding cores to a bandwidth-bound kernel buys almost nothing, a point worth understanding before you count cores as free speedup. The relationship between core count and real throughput is not linear, and what single- versus multi-core means for AI workloads unpacks where the scaling stops.

Underneath both is the cache hierarchy — L1, L2, L3 — and main memory. A weight tensor that fits in L2 is served in a few cycles; the same tensor streamed from DRAM costs an order of magnitude more per access and is limited by bandwidth, not by how fast the ALUs can multiply. This is why data layout (contiguous, cache-blocked, aligned) frequently matters more than instruction selection. A vectorized kernel that thrashes cache can be slower than a scalar one that respects it.

What determines CPU inference performance — cores, SIMD, cache, or bandwidth?

All four, but they do not contribute equally to any given workload, and that is the point most naive comparisons miss. The honest answer is that performance is determined by whichever of these your specific model saturates first. A small convolutional model with a hot weight set that fits in cache is usually compute-bound and rewards SIMD and threading. A large language model doing token-by-token decode is dominated by streaming weights from memory — it is bandwidth-bound, and no amount of extra vector width helps because the arithmetic units are already idle waiting on data.

This is the same roofline reasoning that governs GPUs, applied to a CPU. Peak arithmetic throughput (GFLOPS) is a ceiling you only reach when arithmetic intensity — FLOPs per byte moved — is high enough. Below that ridge point you are bandwidth-limited and the FLOPS number is irrelevant. We explain why peak and achieved throughput diverge on a real path in reading peak versus achieved CPU throughput, and it is the single most useful mental model to carry into a port decision.

Regime	Dominant limiter	Symptom in a profile	What actually helps
Compute-bound	ALU / SIMD throughput	High cycles in the GEMM/conv kernel, cores near 100%	Vectorization (AVX2/AVX-512), better kernels, more cores
Bandwidth-bound	DRAM bandwidth	Cores stalled waiting on memory, low IPC, high LLC misses	Cache blocking, quantization, layout — not more cores
Cache-bound	Working-set spill	High L2/L3 miss rate, sensitivity to batch/tile size	Tiling, smaller working set, reordering loops
IO / dispatch-bound	Data feed, framework overhead	Time outside the kernel, gaps between ops	Batching, pinning, reducing Python-side dispatch

The table is the decision surface. Classify the regime first; only then does an optimization — or a port — have a defensible target.

How do I tell whether my path is compute-bound, memory-bound, or IO-bound?

You measure, and the tools are ordinary. Under Linux, perf stat gives you instructions-per-cycle, cache-miss rates, and stall cycles for the whole process; a low IPC with high last-level-cache misses is the signature of a bandwidth-bound kernel. Intel VTune and the open-source likwid go further and attribute cycles to memory versus compute directly through the top-down microarchitecture method. For the framework layer, the profilers built into PyTorch and ONNX Runtime tell you how much wall-clock time is spent inside kernels versus in dispatch and data movement between ops.

A concrete worked example, with the assumptions stated. Suppose you profile a ResNet-50 inference path at batch 1 and see the process pinned near 100% on a single core, most cycles inside the convolution kernels, and a modest cache-miss rate. That is a compute-bound, under-threaded profile: it is not using the other cores and quite possibly not using AVX. Before you conclude the CPU is the problem, you would enable the vectorized, multi-threaded kernels (build with the right -march, let the runtime use all physical cores) and re-measure. In configurations like this it is common for the recovered speedup to be large — because the baseline was leaving both SIMD width and cores on the table (observed pattern across porting engagements; not a published benchmark). The failure to make is porting that baseline to WASM and attributing the resulting slowdown to “the browser.”

Before you profile at the C++ or WASM boundary at all, it is worth profiling the Python path first, because a surprising share of “CPU is slow” latency lives in framework dispatch rather than in arithmetic — a diagnostic sequence we lay out in profiling the Python inference path before a C++ or WASM port.

When does a well-threaded, vectorized CPU implementation match or beat a port?

More often than the reflex to reach for a GPU suggests. A CPU implementation that is fully vectorized, threaded across physical cores, and cache-blocked is a genuinely fast machine for many inference shapes: small-to-medium models, low batch sizes, latency-sensitive single-request paths, and anything where the data already lives in host memory and a PCIe round-trip to a GPU would dominate the compute it saves.

The GPU wins decisively when arithmetic intensity is high and batch size is large — big transformer prefill, large convolution stacks at high throughput — because that is where thousands of parallel lanes and HBM bandwidth pay for the data-transfer overhead. A WASM target trades native SIMD and threading for portability and sandboxed distribution; it can approach native speed with WASM SIMD and threads enabled, but it will not exceed a good native CPU path, and it inherits the same regime classification you should have done first. How the CPU model maps onto what a WASM runtime actually executes is worth understanding on its own terms — a browser or edge WASM engine still runs on CPU cores, still has a cache hierarchy, and its SIMD support (128-bit fixed-width) is narrower than AVX2 or AVX-512, so a bandwidth-bound kernel stays bandwidth-bound after the port. Where inference work divides across CPU, GPU, and WASM targets is a mapping problem, not a hierarchy of “faster” and “slower.”

The reframe that matters: the comparison should never be naive CPU baseline versus port. It should be optimized CPU baseline versus port. A SIMD-and-threading fix that recovers your latency budget costs a rebuild and a re-measure; a runtime port costs a rewrite, a new toolchain, a new set of numerical and layout surprises, and ongoing maintenance. If both land the same latency, the CPU fix is almost always the cheaper engineering. Understanding what “computationally expensive” actually means in a path — and where that cost lives — is the prerequisite for making that call, which is why we treat where the cost actually lives in an inference path as a companion to this one.

When is moving off the CPU justified, and how do I estimate the gain?

Moving off the CPU is justified when the optimized CPU path is genuinely compute-bound, near its roofline ceiling, and still short of your latency or throughput target — or when portability itself is the requirement (WASM for browser distribution) rather than raw speed. The estimate is a roofline calculation against the optimized baseline, not the naive one: take the model’s FLOPs and byte movement, compute its arithmetic intensity, place it on the target device’s roofline, and compare the projected ceiling to what your tuned CPU already delivers. If the projected GPU ceiling is 2× your optimized CPU number and your budget needs 3×, the GPU does not solve the problem either.

This is exactly the analysis our inference cost-cut work is built around: clarify the CPU baseline and its real optimization headroom before the port-or-don’t-port call, so the decision is measured against a tuned path rather than a straw man. The [GPU acceleration and performance engineering practice](GPU engineering) exists to do that attribution rigorously, because the expensive mistake is not choosing the wrong target — it is choosing any target before you have classified the regime.

FAQ

How does cpu programming work in practice?

In practice, CPU programming for inference means arranging computation so four physical properties work for you: how many cores you use, how wide each core’s SIMD arithmetic is, how well data fits the cache hierarchy, and whether memory bandwidth can feed the arithmetic units. It is less about assembly and more about vectorization, threading, and cache-friendly data layout on the hardware you already have.

What determines CPU inference performance — cores, SIMD/vectorization, cache hierarchy, or memory bandwidth?

All four, but not equally for any given model — performance is set by whichever your workload saturates first. A cache-resident convolutional model is usually compute-bound and rewards SIMD and threading; a large-model token decode is bandwidth-bound and gains nothing from extra vector width because the arithmetic units are already idle waiting on memory.

How do I tell from a profiling baseline whether my inference path is CPU compute-bound, memory-bound, or IO-bound?

You measure with ordinary tools: perf stat for instructions-per-cycle and cache-miss rates, Intel VTune or likwid for top-down memory-versus-compute attribution, and the PyTorch or ONNX Runtime profilers for kernel-versus-dispatch time. Low IPC with high last-level-cache misses signals bandwidth-bound; high cycles inside the kernel with cores near 100% signals compute-bound; time spent outside the kernel signals IO or dispatch overhead.

When does a well-threaded, vectorized CPU implementation match or beat a runtime port or GPU offload?

For small-to-medium models, low batch sizes, latency-sensitive single-request paths, and workloads where data already lives in host memory, a fully vectorized and threaded CPU path often matches or beats a port. The GPU wins decisively only at high arithmetic intensity and large batch, where parallel lanes and HBM bandwidth pay for the data-transfer overhead.

How does the CPU execution model relate to what a WASM target actually executes for inference?

A WASM runtime still executes on CPU cores with the same cache hierarchy, so the regime classification carries over — a bandwidth-bound kernel stays bandwidth-bound after the port. WASM SIMD is fixed at 128-bit, narrower than AVX2 or AVX-512, so a WASM target trades native vector width and threading for portability and sandboxed distribution rather than raw speed.

What CPU-side optimizations recover latency before I consider porting?

Enabling SIMD (building with the right -march so AVX2/AVX-512 kernels are used), threading across all physical cores, and cache-friendly data layout (contiguous, blocked, aligned tensors) recover the most latency on an under-optimized path. These are a rebuild and re-measure rather than a rewrite, which is why they should be exhausted before a runtime port.

When is moving off the CPU justified, and how do I estimate that gain against an optimized CPU baseline?

Moving off is justified when the optimized CPU path is genuinely compute-bound, near its roofline ceiling, and still short of target — or when portability is the actual requirement. Estimate the gain with a roofline calculation against the tuned baseline: compute the model’s arithmetic intensity, place it on the target device’s roofline, and compare the projected ceiling to what the optimized CPU already delivers.

The discipline is not “CPU good, GPU bad” or the reverse. It is refusing to commit to a target until the regime is named and the CPU baseline is the optimized one. When an optimized CPU inference path does reach deployment, the same attribution has to hold its latency budget under production load — the release-readiness considerations for AI infrastructure apply directly the moment that tuned path leaves the profiler and meets real traffic.