FLOPS per Watt: What It Measures and How to Use It in Port Decisions

FLOPS per watt on a spec sheet is a full-utilisation ceiling. How memory limits cut it, how it maps to energy per inference, and how to measure it.

FLOPS per Watt: What It Measures and How to Use It in Port Decisions
Written by TechnoLynx Published on 11 Jul 2026

FLOPS per watt is floating-point throughput divided by power draw, which works out to floating-point operations per joule of energy. A FLOPS-per-watt figure quoted at peak throughput assumes the accelerator is fully utilised. Treat the spec-sheet number as a theoretical ceiling. The figure that informs a port or accelerator decision is the achieved FLOPS per watt on your profiled inference path, which reflects your memory traffic, your batch sizes and your runtime.

How Does FLOPS per Watt Work in Practice?

Two terms get mixed up here. FLOPS (capital S) is a rate: floating-point operations per second, the standard measure of hardware performance. FLOPs (lowercase s) is an amount of work: the operations a task needs, such as one forward pass. Dividing hardware FLOPS by watts gives a measure of hardware energy efficiency. A watt is one joule per second, so the seconds cancel and FLOPS per watt is the same quantity as FLOPs per joule.

Counting conventions are the next trap. Hardware vendors count a fused multiply-add as two FLOPs, but computer vision papers often count it as one operation, so model FLOPs taken from a paper may need doubling before you compare them with a hardware rating. Precision matters as well. The FLOPS rating changes with data type, so a FLOPS-per-watt figure without its precision attached is incomplete: NVIDIA rates the L4 at 120 TFLOPS for FP32, 242 for FP16 and 485 for FP8.

Do not reach for the GPU utilisation percentage as a proxy either. NVIDIA’s Nsight Compute guidance warns that utilisation percentages can move in either direction when the total work changes, so absolute duration is the ground truth for performance progress. Energy per unit of work answers the efficiency question in a way a busy percentage cannot.

Peak Versus Achieved FLOPS per Watt on a Real Inference Path

Sze et al. (IEEE SSCS Magazine, 2020) point out that operations-per-watt figures are normally reported at peak throughput, which assumes maximum utilisation and the most favourable spreading of overhead, while real processors seldom run at peak and lose efficiency at lower utilisation. A 2021 study of inference energy trends computed GPU efficiency the same way, peak FLOPS divided by TDP, and said openly that this is efficiency at full utilisation and that real efficiency varies by workload.

Three things pull the achieved figure below the printed one.

Clocks fall under power management. AMD’s ROCm engineering blog explains that tensor-core arrays draw so much power that the best operating point runs below the boost clock, and modern GPUs lower their clocks during dense compute phases. Theoretical FLOPS figures assume the boost clock. Your kernels run at whatever clock the power limiter settles on.

Tensor-core FLOPS do not reach the whole model. Tensor cores cannot run every operation, so the theoretical tensor-core FLOPS increase does not show up as an equal gain in deep learning inference. The 2021 study therefore adapted its efficiency figures separately for CV and NLP tasks. These differ even for the same GPU: an A100 in mixed precision comes out at 130.88 GFLOPS/W for CNNs and 183.23 GFLOPS/W for NLP models.

TDP is not the power you draw. The MLPerf Power authors call reliance on TDP a myth, because TDP is a thermal design limit and often overstates real consumption by a wide margin. ML.ENERGY’s measurements show average GPU power varying with workload and load, and rarely reaching TDP, even in heavy large-model training. So a peak/TDP ratio rests on two assumptions real workloads break: full-utilisation throughput and TDP-level power draw.

Load changes how far TDP-based estimates miss. The 2026 WattGPU study found that TDP-based estimates of GPU power had a median error of 4.4% in offline, fully loaded scenarios, but 190.1% in server scenarios (26.0% when scaled by load). For server scenarios, TDP-based heuristics are a limited way to rank GPUs. The same caution applies to sizing a cluster from datasheet peak FLOPS, which is not a capacity plan either.

How Do Memory-Boundedness and Utilisation Cap the FLOPS per Watt You Attain?

Memory-boundedness caps achieved FLOPS, and low utilisation means paying static power while waiting. The Roofline model by Williams, Waterman and Patterson describes the first cap: attainable performance is the lower of peak floating-point performance and peak memory bandwidth multiplied by operational intensity (FLOPs per byte of DRAM traffic). The ridge point, where the two limits meet, is the minimum intensity a kernel needs to reach peak FLOPS. A kernel to the left of it is memory-bound however strong the compute is.

Over twenty years peak compute has outgrown memory bandwidth, pushing that ridge point right. The 2024 “AI and Memory Wall” analysis reports that peak server FLOPS grew about 60,000× over two decades while DRAM bandwidth grew about 100×. The practical result shows up in autoregressive decoding: GPT-2 runs much slower than BERT-Base despite similar end-to-end FLOPs, because its matrix-vector operations have low arithmetic intensity. WattGPU’s latency model reached the same conclusion from the data, with memory bandwidth the most important predictor of inter-token latency and FP16 TFLOPS ranking fourth, with much smaller importance.

Compute-bound models are not exempt from memory-bound operations. Benchmarking across TPU, GPU and CPU (Wang et al., 2019) found that even ResNet-50 spends about 13% of its time in memory-bound operations, and that FLOPS utilisation rises with batch size because larger batches increase computation per byte moved. Small models with little parallelism cannot make full use of a large accelerator. Irregular workloads can be extreme: recurrent networks reached less than 9% FLOPS utilisation on the GPU in that study.

The utilisation loss feeds directly into energy because of static power. ML.ENERGY explains that a GPU burns static power for the whole run, so if only 60% of its compute units are busy, 40% of the time it draws power while making little progress. Choi, Vuduc et al. (IPDPS 2013) made the same point formally in their energy Roofline: time costs can overlap, energy costs cannot, and constant power means time efficiency tends to imply energy efficiency, which makes “race to halt” a sensible first-order strategy. Data movement then adds energy that a FLOPs count does not reflect. Sze et al. note that data movement dominates energy over arithmetic, and MLPerf Power found that DLRM-v2 uses 2.3× the energy per sample of ResNet with only 1.06× the multiply-accumulates, a gap the authors attribute to data movement.

A memory-bound, under-filled inference path gives you low achieved FLOPS per watt on any accelerator, so buying a higher spec-sheet ratio does not fix it.

How Does FLOPS per Watt Relate to Energy per Inference and the Deployment Power Envelope?

Energy per inference is the model’s FLOPs per forward pass divided by the hardware’s FLOPs per joule. That relationship is why FLOPS per watt matters at all: it is the bridge from a hardware rating to joules per request. Take the 2021 study’s adapted figure for a T4 on CNN workloads, 312.15 GFLOPS/W. Each joule buys 312.15 billion FLOPs, so each FLOP costs about 1 ÷ 312.15×10⁹ J, roughly 3.2 pJ, and a model’s energy per inference is its FLOP count times that cost. That adapted figure still assumes full utilisation at TDP, so the energy it produces is an approximation. Your measured energy per inference will be higher wherever utilisation or memory traffic pulls achieved efficiency down.

ML.ENERGY states the inverse relationship directly: throughput per watt equals work per energy, so improving energy per token or per image is the same thing as improving throughput per watt. Batch size is the primary knob on the time-energy trade-off. Energy per token falls as batch grows and then levels off as the GPU saturates, and ML.ENERGY reports reductions of 3 to 5×. A 2025 measurement study made the point concretely: a single Llama 405B prompt consumed 21.7 Wh, while a batch of 100 consumed 60.4 Wh in total, about 0.6 Wh per prompt. Serving systems that prioritise latency often run below their best batch size, which caps the gain.

The power envelope at the deployment site is a separate need from energy. Average and peak power decide power delivery and cooling, and energy over time decides running cost and battery life. In server mode, WattGPU notes that the simple formula of mean power times inter-token latency stops being a meaningful efficiency metric, because mean power averages over idle and busy periods. TechnoLynx’s Inference Cost-Cut Pack profiling baseline records achieved FLOPS per watt, energy per inference and power headroom at the site. That last check belongs in a release-readiness decision for an AI feature as much as in the port assessment.

Also scope the measurement deliberately. MLPerf Power measures full-system power, because accelerator-only figures miss the host, memory and interconnect that run alongside it. A GPU-only reading is fine for comparing kernels on one device, but it understates what the site actually supplies. Our explainer on how MLPerf Power measures energy efficiency in practice covers the methodology in more depth.

Worked Example: The Higher TFLOPS per Watt Was the Wrong Pick

The WattGPU authors (2026) illustrate the failure with one GPU-selection case. They modelled a low-load Llama 3.1 8B server with a 25 ms inter-token latency target. The L4’s 72 W TDP made it look like the lowest-power option, but their predictions showed it would miss the latency target. Among GPUs that met the target, the H100 had the highest spec-sheet efficiency at 2.09 TFLOPS/W against the A30’s 1.00 TFLOPS/W, yet choosing the H100 would draw 43% more power than the A30. Because the server runs around the clock whichever GPU is chosen, the authors note that the power difference carries straight through to total GPU energy.

The assumptions matter: this is a predictive model’s output for one model, one latency target and a low request rate, not a measurement of your workload. At low load a large accelerator spends much of its time partly idle and paying static power, so a lower ratio on a part that fits the load can win. The 2025 measurement study saw the same thing: small models on a four-A100 setup used the most energy because they could not use the resources, and one 13B model consumed 0.18 Wh on two L4s but 0.08 Wh on a single H100. Over-provisioning and under-provisioning both cost energy. Which one you face depends on the model and its memory requirements.

For choosing efficient AI inference infrastructure, LynxBenchAI measures throughput, latency, energy and cost per unit of work across accelerators and precisions, not spec-sheet ratios. It measures the hardware under defined workloads, not your inference path, so it narrows the candidates and your own profiling settles the choice.

How Do You Measure Achieved FLOPS per Watt as Part of a Profiling Baseline?

Measure achieved FLOPs per joule on the real inference path, under power-limited settings, using the energy counter. Profile before you optimise. That principle applies to efficiency as much as to latency: the context that tells you which intervention will pay comes from the profile, not from the spec sheet.

  1. Pick the path and the load. Use Nsight Systems first to find which kernels dominate, then Nsight Compute for those kernels. Run at the batch sizes and request rates you expect in deployment, because efficiency at full offline load says little about a latency-bound server.
  2. Count the work. Derive FLOPs from operation or instruction counts with the proper weighting, then divide by kernel duration. Keep the multiply-add convention consistent with the hardware’s (two FLOPs per FMA).
  3. Time it properly. Collect kernel durations with Nsight Systems or CUDA events, and monitor the GPC clock with nvidia-smi or NVML to find the settled frequency.
  4. Run power-limited, not clock-locked, for the deployment number. NVIDIA’s CUTLASS guidance separates fixed-frequency tests (architectural efficiency) from fixed-power tests, which mimic real use with DVFS throttling but vary more from run to run and device to device. Nsight Compute’s default clock locking suits isolated kernel analysis, while system-representative behaviour needs clock and cache control switched off.
  5. Warm up until power settles. In fixed-power tests, the CUTLASS team observed clocks oscillating for about the first 3 seconds before settling.
  6. Control cache residency and input data. Synthetic loops that keep inputs resident in L2 inflate results, so CUTLASS rotates buffers totalling at least twice the L2 size. Fill values change switching activity and power, so use representative inputs.
  7. Read energy from the counter, synchronised. The NVML energy API returns cumulative energy, so two reads around the region give the energy used, and the CPU-side call must be synchronised with asynchronous GPU execution.
  8. Run long enough and repeat. MLPerf Power requires at least 60 seconds of power data, looping shorter workloads. Studies of nvidia-smi’s sampled power readings recommend at least 32 iterations or 5 seconds and four separate trials.
  9. Compute and cross-check. Achieved FLOPs per joule is counted FLOPs divided by measured joules, and energy per inference is joules divided by inferences completed. If the profiler’s roofline disagrees with your expected FLOPs, check the roofline section for your data path, the clock basis and padding in the matrix shapes.

A study of nvidia-smi found that on A100 and H100 only 25% of runtime is actually sampled for power. The error is about ±5% rather than the ±5 W NVIDIA documents, and naive energy measurements were off by up to 70%, falling to around 5% with careful practice. The same study found readings on one GPU almost perfectly linear, so before/after comparisons on the same device stay reliable, while comparisons across devices carry random error.

When Does a Poor FLOPS-per-Watt Figure Point to a Port Rather than a Different Accelerator?

A poor achieved figure points to software work, such as kernel or runtime changes, when the profile shows resources well below their limits. When the limiter is near its practical peak, the remaining moves are algorithmic change or different hardware. Nsight Compute’s triage treats a resource above about 80% of peak as near its limit and below about 60% as underused. For memory-dominated kernels already close to practical bandwidth, further gains need algorithmic change, not tuning.

What the profile shows What it usually means Intervention to weigh first
Compute and memory throughput both low, GPU often idle Latency-bound or starved; static power wasted Batching, overlap, runtime changes; a port if the current stack cannot batch or schedule well
GPU waiting on CPU preprocessing or I/O The rest of the system is the bottleneck Fix the host pipeline; a new accelerator will idle the same way
Many short kernels Per-kernel startup overhead of a few microseconds CUDA graphs, which cut energy for smaller models in one study
DRAM-bound, well below practical peak Poor reuse or too few bytes in flight Kernel work: reuse, vectorised loads, async copy
DRAM-bound, near practical peak Bandwidth wall for this algorithm Quantisation or a less memory-hungry algorithm, or hardware with more bandwidth per FLOP
Compute pipe above about 80% of peak Near the hardware limit Reduce algorithmic work or restructure the kernel
Large accelerator, low request rate Over-provisioned for the load A smaller accelerator that still meets the latency target

Software alone is a real lever. MLPerf Power found software-only changes on the same hardware gave up to 28% better energy efficiency for under 1% performance loss, and careful quantisation up to 70%. Precision changes are not free wins, though. ML.ENERGY measured FP8 using up to 56% more energy than BF16 at batch sizes 8 to 16, because quantisation overhead and underused FP8 tensor cores cancelled the gain until the batch grew. One more caution applies when the alternative is a newer GPU: Nsight Compute advises re-evaluating the dominant limiter on every generation, since a kernel that saturated the bus on one part may underperform on the next without modification. A different accelerator can still need kernel work, so re-profile on the new part.

These efficiency numbers feed the broader performance and porting assessment before a migration, alongside latency. Whether a rewrite out of the Python stack earns its cost is a separate question, covered in when porting Python inference to C++ or WASM pays off. The judgement this page argues is narrower: a spec-sheet ratio can change what you buy without changing what you deploy, while a measured joules-per-inference figure tied to a named limiter tells you which of those two moves actually reaches the energy target.