A model is missing its latency target, so the team asks for a bigger GPU with more FLOPS. The upgrade lands, the benchmark barely moves, and everyone is confused. This is the single most common misread in production inference: treating every latency problem as a compute problem. On a large fraction of real serving workloads, the accelerator is not short on arithmetic — it is sitting idle, waiting on memory. Those are memory-intensive applications, and they respond to an entirely different set of levers than a compute-bound model does. The distinction matters because spending on raw compute for a memory-bound model buys you almost nothing. You can double the FLOPS on paper and watch p95 latency move by a rounding error, because the bottleneck was never the multiply-accumulate units. It was the rate at which the device could pull activations, weights, or KV-cache entries out of HBM and into the compute pipeline. Getting this classification right — before any porting work begins — is what separates a targeted, cheap fix from an expensive upgrade that changes nothing. What does “memory-intensive” actually mean for an inference workload? Every kernel that runs on a GPU has an arithmetic intensity: the ratio of floating-point operations it performs to the bytes it moves in and out of memory. When that ratio is high, the device spends most of its time computing and the compute units are the constraint — the workload is compute-bound. When the ratio is low, the device finishes the maths long before the next chunk of data arrives, so it stalls waiting on memory — the workload is memory-bound, or memory-intensive. The roofline model is the standard way to reason about this. Below a hardware-specific arithmetic-intensity threshold, your achievable throughput is capped by memory bandwidth, not by peak FLOPS. Above it, the reverse. The threshold is a property of the specific device: on an accelerator with very high peak FLOPS relative to its HBM bandwidth, more kernels fall on the memory-bound side of the ridge than most teams expect. This is why a spec sheet is a poor predictor here. Two accelerators can advertise similar peak FLOPS while differing substantially in memory bandwidth per NVIDIA’s published specifications for their respective HBM configurations, and a memory-bound workload will track the bandwidth number far more closely than the FLOPS number. If you are choosing hardware from datasheets alone, you can pick the wrong device for your workload class entirely. Which parts of AI inference are typically memory-intensive? Not all of a model behaves the same way. In practice, four structures dominate the memory-bound category: Large activations. Wide intermediate tensors — common in high-resolution vision models and long-sequence transformers — move enormous amounts of data between layers relative to the arithmetic performed on them. The KV cache. In autoregressive LLM decoding, every generated token reads back the accumulated key/value tensors for all prior tokens. The maths per step is small; the memory traffic grows with sequence length. Embedding tables. Recommendation and retrieval systems carry embedding tables that can reach tens of gigabytes. A lookup is almost pure memory access with negligible compute. Weight streaming. When a model is too large to keep resident, or when batch sizes are small, the cost of loading weights from HBM dominates the tiny amount of compute done per weight. The single sharpest boundary in the list is prefill versus decode in an LLM. Prefill processes the whole prompt at once — large matmuls, high arithmetic intensity, usually compute-bound. Decode generates one token at a time against a growing cache — low arithmetic intensity, memory-bound. The same model, on the same GPU, is compute-bound in one phase and memory-bound in the next, which is precisely why prefill/decode splitting exists as a runtime technique. We cover the KV-cache growth pattern in more depth in our discussion of what long LLM context windows cost at inference. How do we tell whether a workload is memory-bound or compute-bound? You profile it. This is the divergence point between the naive and the expert approach: the naive path assumes the bottleneck, the expert path measures it. A GPU profiler — Nsight Compute, the PyTorch profiler with CUDA events, or the metrics exposed by your serving runtime — reports the numbers you actually need: achieved memory bandwidth against the device peak, SM occupancy, and the fraction of cycles the compute units stalled on memory dependencies. The reading is usually unambiguous. If achieved HBM bandwidth is pinned near the device maximum while the compute units show high stall rates, the workload is memory-bound. If the compute units are saturated and bandwidth utilisation is moderate, it is compute-bound. A workload that shows neither near saturation has a different problem — kernel launch overhead, host-device transfer, or a serialisation bug — and no hardware change will help it. Distinguishing these cases is the whole point of the GPU profiling methodology that maps the serving path before you touch the model. Quick diagnostic: memory-bound vs compute-bound Signal Memory-bound Compute-bound Achieved HBM bandwidth Near device peak Moderate Compute-unit stall on memory High Low SM / tensor-core occupancy Low despite work queued High Effect of a bigger, higher-FLOPS GPU Minimal Meaningful Effect of higher-bandwidth memory Meaningful Minimal Dominant workload shape KV-cache decode, embedding lookup, small-batch weight streaming Prefill, large-batch matmul, dense conv This table is the fast triage. It is not a substitute for a profiler run — occupancy and stall counters can be muddied by mixed kernels — but it tells you which measurement to trust and which upgrade to stop asking for. Why does adding compute often fail to speed up a memory-bound model? Because the compute was never the limit. If a kernel spends 80% of its cycles stalled waiting for the next tensor to arrive from HBM, then doubling the number of tensor cores leaves those cores idle for a slightly longer fraction of the time. Throughput is governed by bandwidth, and bandwidth did not change. This is the mechanism behind the frustrating “we upgraded the GPU and nothing happened” experience — the upgrade delivered more of the resource the workload was not short on. The corollary is the useful part. On a memory-bound workload, the levers that do move latency all reduce or reschedule memory traffic rather than adding arithmetic capacity. That reframes the whole porting decision, and it is where the money is: recovering latency headroom without buying hardware, or buying the right hardware instead of the biggest. Which porting levers help memory-intensive workloads? There are three broad classes, and the profiler tells you which is worth pursuing. Quantisation. Moving activations, weights, and KV-cache entries from FP16 to INT8 or FP8 roughly halves the bytes moved per element. On a bandwidth-bound decode loop, that translates fairly directly into throughput, because the workload was memory-limited, not precision-limited. The trade-off is accuracy: quantisation is a numerical-precision decision, not a free lunch, and it needs validation against your task metrics before it ships. A memory-efficient runtime. Some of the largest wins come from software that schedules memory better without touching the model. PagedAttention-style KV-cache management (as in vLLM), FlashAttention’s fused, IO-aware attention kernels, and prefill/decode disaggregation all attack memory traffic and fragmentation directly. We walk through the last of these in how prefill/decode splitting cuts inference latency — it works precisely because the two phases have opposite bottleneck profiles. A machine-learning compiler such as TensorRT, XLA, or torch.compile can also fuse kernels to cut intermediate-activation round-trips to HBM; see what machine-learning compilers do in the serving path. Higher-bandwidth hardware. When quantisation and runtime tuning are exhausted and the workload is still bandwidth-pinned, the right hardware move is toward higher HBM bandwidth — not higher FLOPS. That is a different procurement conversation than “get the biggest GPU,” and it is the conversation the profiler earns you. Worked example (illustrative assumptions) Suppose a serving team runs an LLM chat endpoint where responses are long and prompts are short, so decode dominates. A profiler run shows HBM bandwidth pinned near the device peak during decode, with tensor-core occupancy low. For example, if such a workload measured 85% bandwidth utilisation and 30% tensor-core occupancy, the classification is unambiguously memory-bound. The predicted outcomes: A higher-FLOPS GPU at similar bandwidth: negligible change. FP8 KV-cache quantisation: meaningful decode throughput gain, subject to accuracy validation. vLLM PagedAttention plus prefill/decode disaggregation: latency headroom from reduced cache fragmentation and phase-appropriate scheduling. The numbers here are illustrative framing, not a benchmark. The point is the structure of the reasoning: measurement first, then the lever the measurement implies. How do we know the port actually worked? A memory-targeted port has to justify itself against the same serving metrics that any inference change is held to: p95 latency, throughput per dollar, and GPU utilisation. Quantisation that halves memory traffic but silently degrades output quality has not improved the system; it has moved the failure somewhere the latency dashboard cannot see. So the measurement loop closes on two axes at once — the performance number and the task-quality number — and both are baselined before the port so the delta is real. This is where the diagnostic distinction routes into practical work. Classifying a workload as memory- or compute-bound is the first profiling step in our [inference cost-cut engagement](Inference Cost-Cut Pack), because it determines whether memory-targeted levers are the right ones at all before anyone spends a sprint on them. The broader unit-economics framing — the cost-per-request KPI a port has to improve to be worth doing — sits alongside our R&D consulting services. FAQ How does memory-intensive applications actually work? A memory-intensive application is one whose inference kernels move a lot of data relative to the arithmetic they perform — a low arithmetic intensity. In practice it means the accelerator spends much of its time stalled waiting on HBM rather than computing, so achievable throughput is capped by memory bandwidth, not peak FLOPS. How do we tell whether an inference workload is memory-bound or compute-bound? Profile it. A GPU profiler reports achieved HBM bandwidth against the device peak, compute-unit occupancy, and memory-stall cycles. High bandwidth utilisation with low occupancy and high memory stalls means memory-bound; saturated compute units with moderate bandwidth means compute-bound. Assuming the answer instead of measuring it is the core mistake. Which parts of AI inference are typically memory-intensive — activations, KV cache, embedding tables, or weights? All four can be. Large activations in wide or long-sequence models, the KV cache in autoregressive LLM decoding, multi-gigabyte embedding tables in recommendation systems, and weight streaming under small batches are the common memory-bound structures. LLM decode and embedding lookups are the most reliably memory-bound. Why does adding compute or a bigger GPU often fail to speed up a memory-bound model? Because compute was never the limit. If a kernel spends most of its cycles stalled on memory, adding tensor cores leaves them idle a slightly larger fraction of the time — throughput is governed by bandwidth, which the upgrade did not change. That is why “we bought a bigger GPU and nothing happened” is such a common outcome. Which porting levers help memory-intensive workloads? Three classes: quantisation (fewer bytes per element, e.g. FP16 to INT8/FP8, validated against accuracy), a memory-efficient runtime (PagedAttention, FlashAttention, prefill/decode disaggregation, kernel fusion via a compiler), and, when those are exhausted, hardware with higher HBM bandwidth rather than higher FLOPS. How do we measure whether a memory-targeted port actually improved p95 latency and throughput per dollar? Baseline p95 latency, throughput per dollar, and GPU utilisation before the port, then measure the same metrics after — alongside a task-quality metric, since quantisation can trade accuracy for speed. A port that improves latency but degrades output quality has moved the failure, not fixed it. How do memory constraints show up differently in LLM inference, where the KV cache grows with sequence length? Prefill is usually compute-bound — large matmuls over the whole prompt — while decode is memory-bound, because each token reads back the entire accumulated KV cache for little arithmetic. As sequence length grows, decode memory traffic grows with it, which is why long-context serving is dominated by KV-cache bandwidth and why prefill/decode splitting helps. If your inference workload is missing its latency target and the instinct in the room is “buy a bigger GPU,” the question worth asking first is narrower and cheaper to answer: is this model starved on maths, or starved on bandwidth? The profiler settles it in an afternoon, and the answer decides whether the next dollar goes to compute, to quantisation, or to memory — three levers that are not interchangeable.
