A processor rated at 500 TFLOPS on the datasheet does not serve you 500 TFLOPS worth of inferences. The number on the spec sheet is a theoretical ceiling — the peak operations-per-second the silicon can execute under conditions your serving path never reproduces. What you actually get is delivered throughput: the requests per second your serving path sustains once batching, memory bandwidth, data movement, and queueing have taken their cut. The two numbers are related, but treating them as interchangeable is where most inference capacity planning goes wrong. This matters because the gap between theoretical and delivered throughput is usually where the recoverable performance lives. When a team decides they need faster hardware, the honest first question is whether they are anywhere near the ceiling of the hardware they already own. Often they are not — and the fix is a batching or scheduling change, not a purchase order. How does processor throughput work in practice? Processor throughput, in the datasheet sense, is a rate: how many arithmetic operations the compute units can retire per second at their advertised clock, assuming the pipelines stay full. On a GPU that number is derived from the count of multiply-accumulate units, their width, and the boost clock. It is a real property of the silicon and it is measured honestly — but it is measured under a synthetic workload designed to keep every execution unit busy with data that is already resident in registers. Inference does not look like that. A production request arrives, its input has to be tokenised or preprocessed, tensors have to move from host memory across PCIe or NVLink into device memory, kernels launch, intermediate activations spill to and from HBM, and the result travels back. The compute units are idle for large stretches of that sequence, waiting on data. Delivered throughput is therefore a property of the whole serving path, not the processor in isolation. In practice, “processor throughput” as a useful operational metric means requests per second at a fixed cost and a fixed latency budget — not FLOPS. We see this confusion routinely when a team benchmarks a model, sees single-digit-percent compute utilisation, and concludes the hardware is underpowered. The hardware is fine. The path feeding it is starved. What is the difference between theoretical peak throughput and delivered inference throughput? Theoretical peak (FLOPS or OPS) answers “how fast can this chip compute if nothing else is in the way.” Delivered inference throughput answers “how many inferences per second does my system actually complete under real request patterns.” The first is a hardware constant; the second is an emergent property of hardware, software stack, model shape, and traffic. The divergence has three recurring causes: Memory bandwidth. Many inference workloads, especially autoregressive LLM decoding, are memory-bound: the compute units finish their work and sit idle waiting for weights and KV-cache to stream out of HBM. Adding FLOPS to a bandwidth-bound workload changes nothing. We cover when memory bandwidth becomes the binding constraint in AI inference in more detail, because it is the single most common reason a “fast” processor delivers disappointing throughput. Data movement and launch overhead. Small, un-fused kernels spend more time launching and copying than computing. This is why graph compilation and kernel fusion — via torch.compile, TensorRT, or XLA — often produce a bigger throughput gain than a hardware upgrade. Batching and queueing. A processor serving one request at a time leaves most of its parallel width unused. Throughput scales with effective batch size until memory or latency limits are hit. For a grounding on why raw operation counts often fail to predict serving speed, our note on what GFLOPS actually measures and when it predicts inference speed makes the same point from the arithmetic side. Quick-answer block: peak vs delivered throughput Question Theoretical peak (FLOPS/OPS) Delivered throughput (req/sec) What does it measure? Max operations the silicon can retire Inferences the serving path completes Where does it come from? Datasheet, synthetic full-pipeline test Load test against real request patterns What limits it? Clock × unit count Whichever of compute / bandwidth / movement / queueing saturates first Does adding FLOPS help? By definition Only if the workload is compute-bound Evidence class Vendor published specification observed-pattern — measured per deployment The right-hand column is the one that maps to a business number. It is also the one a spec sheet will never tell you. Why does high processor throughput not always translate into more requests served per second? Because throughput is set by the slowest stage of the path, not the fastest. This is the standard bottleneck argument, and it holds without exception on serving paths. If your decode step is waiting on HBM bandwidth, doubling the compute throughput of the processor moves the needle by roughly zero — the compute units were already idle. If your request queue is serialising traffic because the batch scheduler is conservative, the processor spends its time under-fed regardless of how many FLOPS it can theoretically retire. In configurations we have profiled, it is common to find a modern accelerator running at well under 20% of its theoretical compute utilisation on an LLM decode workload, with HBM bandwidth as the true limiter (observed across TechnoLynx engagements; not a published benchmark). That number is not a defect — it is the expected shape of a memory-bound workload. The mistake is reading it as “we need more FLOPS.” The correct reading is “compute is not the constraint here, so a faster-compute chip is the wrong purchase.” Mapping the stages so you can see which one saturates is the whole game. Our walkthrough on mapping the serving path in a machine learning architecture diagram lays out how to draw the path stage by stage so the bottleneck stops being a guess. How do batching and request patterns change the throughput a processor actually delivers? Batching is the lever that most directly closes the gap between theoretical and delivered throughput on parallel hardware. A GPU has thousands of execution lanes; a single small request uses a fraction of them. Grouping requests into a batch lets one kernel launch amortise across many inputs, so the compute you already paid for does useful work instead of sitting idle. But batching trades latency for throughput, and real traffic is bursty. Static batching — waiting for a fixed batch to fill — starves the processor during quiet periods and adds tail latency during busy ones. Continuous (in-flight) batching, as implemented in serving runtimes like vLLM and SGLang, admits new requests into the running batch as slots free up, which is why it typically recovers a large share of the theoretical ceiling on LLM serving without wrecking the latency distribution. Request patterns matter as much as batch policy. Variable input lengths, mixed prefill and decode phases, and unpredictable arrival rates all change the effective batch the scheduler can assemble at any instant. This is why a throughput number measured under a uniform synthetic load is almost useless for capacity planning — the real distribution determines what you actually deliver. How do I measure delivered throughput and utilisation in a production serving path? Measure under the traffic you actually serve, not a benchmark harness. The minimum instrumentation is a load generator that replays a representative request distribution — input-length mix, arrival-rate pattern, and concurrency — against the live serving stack. Diagnostic checklist: is utilisation or hardware the constraint? Work through these in order. Each answer narrows where the recoverable throughput lives. Sustained requests/sec at target latency. Ramp concurrency until the p95 latency SLO breaks. The throughput just below that point is your delivered number — not the burst peak. Compute utilisation (SM occupancy / tensor-core active %). Profile with Nsight Systems or the vendor equivalent. Low occupancy under load means compute is not the constraint. Memory bandwidth utilisation. If HBM bandwidth sits near its ceiling while compute occupancy is low, you are memory-bound. A faster-compute processor will not help. Kernel launch and copy overhead. A profile dominated by many tiny kernels and host-device copies points at fusion and graph compilation, not hardware. Queue depth and scheduler idle time. If the processor is idle while requests wait, the batch scheduler — not the silicon — is the limiter. Cost-per-request. Divide the hourly cost of the instance by sustained requests/sec. This is the number that ties every measurement above to a decision. Steps 2 through 5 tell you why delivered throughput falls short of the theoretical ceiling. That “why” is what decides whether the next move is a software change or a hardware change. GPU profiling is the discipline that surfaces this, and our teams treat it as the prerequisite before any capacity recommendation. When is throughput limited by the processor versus memory bandwidth or data movement? Use the roofline framing: a workload is compute-bound when its arithmetic intensity (operations per byte moved) is high enough that the compute units saturate before the memory system does, and memory-bound when the reverse holds. Dense matrix multiply on large batches with high reuse tends toward compute-bound. Autoregressive decode with a large KV-cache and batch size of one tends sharply memory-bound. The practical test is the one in the checklist above: profile compute occupancy and memory bandwidth simultaneously under load. Whichever sits near its ceiling while the other has headroom is your constraint. Data-movement-bound workloads show a third signature — both compute and bandwidth have headroom, but the timeline is dominated by copies and launches, which points at the software stack. The role of machine learning compilers in the inference serving path is largely about collapsing that overhead through fusion and scheduling. How do I decide whether to improve utilisation on existing hardware or move to a faster processor? Rank the options by expected return, and put the hardware change last on the list until the utilisation levers are exhausted. The logic is simple: a utilisation gain applies to hardware you have already paid for, so its cost-per-request improvement is close to free, whereas a processor upgrade adds capital or hourly cost that the throughput gain has to justify. Decision rubric: utilisation lever vs hardware change Signal from profiling Likely constraint First move (before buying hardware) Compute occupancy low, HBM bandwidth near ceiling Memory-bound Quantise weights, shrink KV-cache, or a higher-bandwidth chip — not a higher-FLOPS one Compute occupancy low, bandwidth has headroom, many tiny kernels Data movement / launch overhead Graph compilation and kernel fusion (torch.compile, TensorRT) Compute occupancy low, processor idle, requests queuing Scheduling / batching Continuous batching; tune max batch size and admission policy Compute occupancy high, bandwidth headroom, latency SLO met Genuinely compute-bound A faster-compute processor is now a defensible purchase Cost-per-request acceptable, SLO met None binding Do nothing — the system is tuned In our experience, utilisation and scheduling work recovers on the order of 2–4× sustained throughput on hardware already deployed before a processor change is justified (observed pattern across cost-optimisation engagements; not a benchmarked rate). The exact multiple depends entirely on how far from the ceiling the untuned path started — a well-tuned path has little left to recover, and honesty about that is part of the analysis. The decision also has an economics layer. Throughput only becomes a business metric when it is expressed as cost-per-request, and defining that KPI cleanly is its own discipline — the unit-economics framework for AI infrastructure ties the throughput measurement to the number a finance owner actually tracks. Choosing where to spend that improvement — utilisation versus hardware — is exactly the kind of question our engineering services engagements exist to answer with measurement rather than instinct. FAQ What’s worth understanding about processor throughput first? Datasheet processor throughput is a rate — the peak operations per second the compute units can retire at full clock with the pipelines saturated. In practice, the operationally useful meaning is different: requests per second your serving path sustains at a fixed cost and latency budget. That delivered number is a property of the whole path, not the chip alone. What is the difference between theoretical peak throughput (FLOPS/OPS) and delivered inference throughput? Theoretical peak measures how fast the silicon can compute with nothing in the way; it is a hardware constant from the datasheet. Delivered inference throughput measures how many inferences per second the system actually completes under real request patterns, and it emerges from hardware, software stack, model shape, and traffic together. The gap between them is set by memory bandwidth, data movement, and batching. Why does high processor throughput not always translate into more requests served per second? Because throughput is limited by the slowest stage of the serving path, not the fastest. If decode is waiting on HBM bandwidth or requests are queuing behind a conservative scheduler, the compute units are already idle — so adding FLOPS moves the needle by roughly nothing. High peak compute only helps when compute is the actual binding constraint. How do batching and request patterns change the throughput a processor actually delivers? Batching amortises one kernel launch across many inputs, letting parallel hardware do useful work instead of sitting idle, which is the main lever for closing the peak-to-delivered gap. But batching trades latency for throughput, and bursty, variable-length traffic changes the batch a scheduler can assemble at any moment. Continuous batching admits requests into a running batch to recover throughput without wrecking tail latency. How do I measure delivered throughput and utilisation in a production serving path? Replay a representative request distribution against the live stack and ramp concurrency until the p95 latency SLO breaks — the throughput just below that is your delivered number. Then profile compute occupancy and memory bandwidth together to see which saturates, inspect kernel-launch overhead and queue depth, and divide instance cost by sustained requests/sec for cost-per-request. Those measurements tell you why delivered throughput falls short of the ceiling. When is throughput limited by the processor versus memory bandwidth or data movement? Use the roofline view: a workload is compute-bound when arithmetic intensity is high enough that compute units saturate first, and memory-bound when bandwidth saturates first. Profile occupancy and bandwidth simultaneously under load — whichever is near its ceiling while the other has headroom is the constraint. If both have headroom but copies and launches dominate the timeline, the limit is data movement in the software stack. How do I decide whether to improve utilisation on existing hardware or move to a faster processor? Rank options by return and put the hardware change last: utilisation gains apply to hardware you already own, so their cost-per-request improvement is nearly free. Only when profiling shows genuinely high compute occupancy with the latency SLO met is a faster-compute processor a defensible purchase. Until then, quantisation, kernel fusion, and continuous batching usually recover more throughput per dollar. When throughput falls short, the reflex is to read the spec sheet and blame the silicon. The discipline is to measure the path — occupancy, bandwidth, launch overhead, queue depth — and let the profile name the constraint before anyone signs a purchase order. That measurement-first stance is what the inference cost-cut audit exists to formalise: it prices delivered throughput and utilisation against the theoretical ceiling, and the recoverable performance almost always turns up somewhere other than the processor.
