A spec sheet says one accelerator delivers more FLOPS per watt than another, so the energy-constrained deployment picks it — and the deployed efficiency barely moves. This is the most common way a headline efficiency number leads a port decision astray. FLOPS per watt is a real, useful metric, but the figure printed on a data sheet describes a theoretical ceiling under ideal conditions. The number that actually governs your energy and footprint targets is the one you measure on your profiled inference path, at your utilisation, with your runtime. The distinction matters because energy-per-inference and power-envelope headroom are increasingly hard constraints, not soft preferences. Edge boxes have a thermal budget. Footprint-constrained racks have a power draw ceiling. When those limits are the reason for a hardware change, quoting peak FLOPS-per-watt spec figures across candidate accelerators tells you almost nothing about whether the change will hit the target — because peak efficiency and attained efficiency diverge by exactly the amount your workload fails to saturate the device. What does working with FLOPS per watt involve in practice? FLOPS per watt is throughput divided by power: how many floating-point operations a device completes per joule of energy, expressed as operations per second per watt. It is an efficiency ratio, not a speed. A device can be fast and inefficient, or slow and efficient — the ratio isolates energy cost from raw performance. The peak figure on a spec sheet is computed from the device’s theoretical maximum throughput (all compute units active, all cycles doing useful work) divided by its rated power at that state. It assumes perfect utilisation. Real inference almost never runs at perfect utilisation, so the achieved ratio — the operations you actually complete per watt you actually draw — is what determines whether a deployment fits its energy envelope. The gap between the two is not noise. It is a direct readout of how much of the device your workload leaves idle. Put concretely: if an accelerator advertises, say, a peak efficiency figure per its published specifications, but your model only keeps the compute units busy a fraction of the time because it stalls waiting on memory, your achieved FLOPS per watt lands well below the headline. You are still paying much of the static power for silicon that is waiting, not computing. The advertised ratio described a machine running flat out; your machine is idling between memory fetches. What is the difference between peak and achieved FLOPS per watt on a real inference path? Peak is a property of the hardware. Achieved is a property of the executor — the hardware plus the software stack plus the specific workload running on it. Those are different objects, and conflating them is the root error behind most efficiency-driven procurement mistakes. The same argument runs through what GPU throughput actually means for cluster sizing: the headline FLOPS number and the delivered FLOPS number are separated by everything the runtime and the workload do to the hardware. Three things drive the divergence, and a profiling pass attributes each one: Utilisation. If your kernels do not keep the compute units occupied — small batch sizes, launch overhead, serialised stages — you draw power for idle silicon. Low utilisation is the single largest source of the peak-to-achieved gap in inference workloads. Memory-boundedness. When a kernel spends most of its time moving data rather than computing on it, the compute units stall. You pay for HBM bandwidth and static device power while the FLOP counter barely moves. This is why the memory subsystem, not the compute peak, dominates real inference efficiency on many devices. Runtime overhead. Kernel launch latency, host-device synchronisation, unfused operations, and framework dispatch all consume time and power without producing useful FLOPS. A path running through eager-mode PyTorch will show a different achieved ratio than the same path compiled with torch.compile, TensorRT, or a fused kernel graph. The CPU-side version of this same peak-versus-attained story is worth reading alongside, because the peak-vs-achieved throughput gap on a CPU inference path is driven by the same three forces — it is not a GPU-specific quirk. Quick-answer: peak vs achieved at a glance Peak FLOPS per watt Achieved FLOPS per watt What it describes The hardware in isolation The executor (hardware × runtime × workload) Source Vendor spec sheet Profiling pass on your inference path Assumes Perfect utilisation, all units active Your real utilisation and stalls Moves the Purchasing decision Deployed energy/footprint outcome Blind to Your memory-boundedness and runtime Nothing you didn’t measure The rule of thumb: a peak figure moves the purchasing decision without moving the deployed efficiency. Only the achieved figure, grounded in your utilisation, tells you whether an intervention will actually shift your energy target. How do memory-boundedness and utilisation limit the FLOPS per watt you attain? The intuition that a more efficient chip yields a more efficient deployment breaks the moment your workload is memory-bound rather than compute-bound. It is a reasonable expectation that also happens to be structurally wrong for a large class of inference paths. Consider a decoder-heavy transformer inference path at small batch sizes. Each decode step reads the full KV cache and weights from memory but performs relatively little arithmetic per byte moved — the arithmetic intensity is low. The device spends most of each step waiting on HBM bandwidth. During that wait, the compute units are powered but idle. You are burning watts on silicon that is not producing FLOPS, so your achieved FLOPS per watt collapses regardless of how high the peak figure was. A device with a higher peak efficiency but the same memory bandwidth will not help here, because the bottleneck was never the compute ceiling. Utilisation compounds this. In configurations we have profiled, small-batch inference on a large accelerator can leave a substantial fraction of the compute array idle between kernel launches (observed pattern across porting engagements, not a benchmarked rate). Static power still flows. The efficiency ratio you attain reflects that idle silicon directly. The practical consequence is that FLOPS per watt is only interpretable once you know whether the path is compute-bound or memory-bound. On a compute-bound path, a more efficient device with more capable compute units can genuinely improve your achieved ratio. On a memory-bound path, the same device change moves almost nothing, and the correct intervention is a software change — quantisation, KV-cache reuse, kernel fusion, better batching — that raises arithmetic intensity or occupancy on the hardware you already have. How does FLOPS per watt relate to energy per inference and the power/thermal envelope? FLOPS per watt is the efficiency ratio; energy per inference is what you actually budget against. They connect through the work each inference requires. If a single inference needs a fixed number of FLOPS, then energy per inference is roughly that FLOP count divided by your achieved FLOPS per watt. Improve the achieved ratio and each inference costs fewer joules; the peak ratio never enters the arithmetic because it is not the rate at which you actually complete work. The deployment power and thermal envelope is the ceiling that closes the loop. An edge inference box has a sustained power draw it cannot exceed without throttling, and a thermal limit that caps sustained clocks. If your achieved FLOPS per watt is low, you draw more power per inference, which pushes you toward the envelope faster — and once you hit the thermal ceiling, the device down-clocks, which lowers throughput and, often, efficiency further. This is why envelope headroom is a release-readiness check, not an afterthought, and why the power/thermal budget belongs in the same conversation as the [AI-infrastructure readiness of an edge inference path](Inference Cost-Cut Pack). Three numbers, taken together, describe whether a deployment fits: Achieved FLOPS per watt on the profiled path — the efficiency you actually get. Energy per inference — achieved efficiency applied to the FLOP cost of one inference. Envelope headroom — how much power and thermal margin the deployment site leaves before throttling. A peak figure gives you none of these. It gives you a comparison between machines you are not running. When does a poor FLOPS-per-watt figure point to a port versus a different accelerator? This is the decision the metric exists to inform, and it turns entirely on why the achieved figure is low. A profiling pass that attributes the efficiency gap to a cause tells you which lever to pull. The same profiling discipline you apply before any inference port is what makes this attribution possible — you cannot decide between a port and a hardware change until you know where the joules are going. Decision rubric: attribute the efficiency gap first Dominant cause of low achieved FLOPS/W What it means Right intervention Runtime overhead (launch latency, unfused ops, eager dispatch) Software is leaving performance on the table Port / optimise the software path — compile, fuse, switch runtime (TensorRT, torch.compile) Low utilisation (small batch, serial stages) Workload shape underuses the device Software first — batching, scheduling; a bigger device makes this worse Memory-boundedness (low arithmetic intensity) Bottleneck is bandwidth, not compute Software (quantisation, KV reuse) or a device with higher bandwidth — not higher peak FLOPS Genuinely compute-bound at high utilisation Hardware compute ceiling is the real limit A different accelerator with better achieved efficiency on this path The pattern is clear: only the last row justifies buying different silicon on efficiency grounds. Everything above it says the gap lives in the software or the workload shape, where a port or an optimisation pass — not a purchase order — moves the number. Buying a higher-peak-efficiency accelerator to fix a memory-bound or overhead-bound path is the mistake the metric is supposed to prevent, and it is the mistake a peak-only reading walks you straight into. How do you measure achieved FLOPS per watt in a profiling baseline before a port? You measure it as part of the baseline profiling pass, alongside latency and utilisation, so the efficiency gap is attributed rather than guessed. The mechanics are straightforward with existing tooling: Count useful FLOPS for the inference path — from the model’s operation graph, or with a profiler that reports achieved FLOP throughput (Nsight Systems / Nsight Compute for NVIDIA devices report the counters you need). Measure power draw at the device during the same run — nvidia-smi power telemetry, board-level power monitoring, or the platform’s power management interface, sampled across a sustained window rather than a single burst. Sustain the load. A transient peak flatters the ratio; measure across a realistic sustained window so thermal throttling and steady-state draw are captured. Record utilisation and memory-bandwidth counters in the same pass, so the achieved figure comes with its explanation attached — you learn not just that efficiency is low but why. That last point is the whole value of doing this before a port. A single achieved FLOPS-per-watt number is a symptom; the profiling pass that produces it alongside utilisation and bandwidth counters is a diagnosis. The efficiency metric is one of the readings a profiling baseline records precisely so the port-decision step can weigh energy and footprint gains, not just latency — the same baseline that keeps the [inference-cost-cut profiling pass](Inference Cost-Cut Pack) honest, and the same reasoning that underpins the broader GPU performance and porting work we do. FAQ What should you know about flops per watt in practice? FLOPS per watt is throughput divided by power — how many floating-point operations a device completes per joule of energy. It is an efficiency ratio, not a speed: a device can be fast and inefficient or slow and efficient. In practice the achieved ratio, measured on your inference path, is what governs whether a deployment fits its energy budget, because it reflects the operations you actually complete per watt you actually draw. What is the difference between peak FLOPS per watt on a spec sheet and achieved FLOPS per watt on a real inference path? Peak is a property of the hardware in isolation, computed assuming perfect utilisation; achieved is a property of the executor — hardware plus runtime plus your specific workload. The peak figure moves the purchasing decision without moving the deployed efficiency, while the achieved figure, measured in a profiling pass, tells you whether an intervention will actually shift your energy target. The gap between them is a direct readout of how much of the device your workload leaves idle. How do memory-boundedness and utilisation limit the FLOPS per watt you actually attain? When a kernel spends most of its time moving data rather than computing on it, the compute units stall — you pay for bandwidth and static device power while the FLOP counter barely moves, so achieved efficiency collapses. Low utilisation compounds this: small batches and launch gaps leave compute units idle while static power still flows. On such paths a higher-peak-efficiency device changes almost nothing, because the bottleneck was never the compute ceiling. How does FLOPS per watt relate to energy per inference and the deployment power/thermal envelope? Energy per inference is roughly the FLOP cost of one inference divided by your achieved FLOPS per watt, so improving the achieved ratio directly lowers joules per inference. The deployment’s power and thermal envelope is the ceiling: low achieved efficiency draws more power per inference, pushing you toward throttling, which lowers throughput and efficiency further. Achieved efficiency, energy per inference, and envelope headroom together describe whether a deployment fits. When does a poor FLOPS-per-watt figure point to a port versus a different accelerator? It depends entirely on why the achieved figure is low. If the gap is runtime overhead, low utilisation, or memory-boundedness, the right intervention is a software port or optimisation on the hardware you have — a bigger or higher-peak device often makes utilisation problems worse. Only when the path is genuinely compute-bound at high utilisation does a poor figure justify a different accelerator. How do you measure achieved FLOPS per watt as part of a profiling baseline before committing to a port? Count the useful FLOPS for the inference path (from the operation graph or a profiler like Nsight Compute), measure device power draw over a sustained window (via nvidia-smi telemetry or board-level monitoring), and record utilisation and memory-bandwidth counters in the same pass. Sustaining the load rather than measuring a burst captures throttling and steady-state draw. Doing it in one baseline pass means the achieved figure arrives with its explanation attached — you learn not just that efficiency is low but why. The question a FLOPS-per-watt figure should always provoke is not “which device has the higher number” but “at what utilisation, on which path, through which runtime.” Answer those before the number leaves the spec sheet, and the metric becomes a diagnosis of where your joules are going — the reading that decides whether a port or a different accelerator is the intervention that actually moves your energy and footprint target.
