MLPerf Power: How the Energy-Efficiency Benchmark Works in Practice

How MLPerf Power measures full-system inference energy, how to turn samples per joule into GenAI run-cost, and when a result stops being a feasibility…

MLPerf Power: How the Energy-Efficiency Benchmark Works in Practice
Written by TechnoLynx Published on 11 Jul 2026

MLPerf Power is the MLCommons methodology that adds a measured energy reading to an MLPerf benchmark run. A result then reports efficiency (samples per joule for datacenter and offline inference) as well as throughput. In practice it measures full-system power with a SPEC-certified analyser, during the execution phase only, for at least 60 seconds in most scenarios. Cooling devices and storage nodes are not yet included. That makes it a fair way to compare hardware on a standard task, between results at the same scenario and accuracy target. It does not measure the energy cost of your own configuration; matching its scenario, accuracy target and measurement boundary to how you will serve the model is left to you.

How does MLPerf Power work, and what does a result mean in practice?

An MLPerf Power result answers one question: how much useful inference did this whole system deliver per unit of energy while it ran a fixed MLCommons inference workload? For throughput benchmarks, MLPerf Power records throughput in samples per second and power in watts. It then divides one by the other, and since a watt is a joule per second, the result comes out in samples per joule. Latency benchmarks such as tiny inference report the inverse of energy instead. The paper treats the two as comparable because each benchmark fixes different execution factors.

The framework is wide. The MLCommons Power Working Group describes it as running “from microwatts to megawatts”, across datacenter, edge, mobile and IoT deployments and across both MLPerf Training and inference. The MLPerf Power paper drew on 1,841 reproducible measurements from 60 systems. For a GenAI buyer, the part that matters is datacenter inference, where the generative models are among the most energy-hungry. The paper finds the three generative AI benchmarks in MLPerf Inference significantly more energy-hungry than every other benchmark except 3D U-Net.

The practical meaning is narrow, and it is useful. A high samples-per-joule figure tells you a given hardware and software stack converts energy into finished inferences efficiently on that task, at that accuracy, under that scenario. It does not tell you your service’s cost per request to run.

What exactly does MLPerf Power measure that a normal MLPerf submission does not?

A standard MLPerf Inference submission reports performance under a defined scenario. According to TokenPowerBench, such a submission does not systematically capture end-to-end power or normalise energy to LLM service units such as joules per generated token. MLPerf Power adds a physical power measurement, logged from an analyser while the workload runs. That measurement is scoped to the whole system, not the accelerator.

The full-system scope is deliberate. The MLPerf Power authors call it “a common misconception” that measuring accelerators or GPUs alone is enough. Different components are active at different stages, so isolating component power misrepresents energy use. In datacenter systems the measurement covers the compute nodes, the interconnect fabric and any drives inside the compute blades.

Two exclusions shape every number you read:

  • Phase. Only the execution phase counts. Initialisation, data setup and offline or online pre-processing are not counted.
  • Boundary. The paper states that there is not yet a method for including cooling devices or storage nodes, and it rejects Power Usage Effectiveness (PUE) as a measure of ML system efficiency, putting building efficiency outside MLPerf’s scope.

The sources differ on cooling. The MLCommons 2025 announcement says that a full understanding of AI system efficiency requires measuring all components, “including… the cooling system”. The primary paper says cooling is not yet in the measured scope. We read the announcement as stating the goal and the paper as stating the current method. When you interpret a published result, trust the paper.

What does working with MLPerf Power involve on the bench?

Running MLPerf Power requires a certified power analyser, not just a script on any box. MLPerf uses the SPEC PTDaemon tool for power measurement, and MLCommons publishes the measurement code under Apache 2.0 in its power-dev repository. In the inference setup, a Director connects to a SPEC-approved power analyser through PTDaemon API calls and tells the system under test to start the loadgen. It then logs time-stamped current, voltage, phase and power readings, usually converted to CSV afterwards.

The inference methodology requires a SPEC-certified meter such as the Yokogawa WT310. It also requires at least 60 seconds of power data in most scenarios, so that brief fluctuations do not dominate the reported figure. Small devices bring a known caveat. The authors note that meters show larger error bars when measuring AC wall power through adapters under 75 W. Phones are not yet supported, partly because battery-powered devices cannot be connected to external power without altering normal operation. Tiny systems use a separate timestamp-pin method, covered in our piece on MLPerf Tiny for constrained edge hardware.

Reproducing a submission therefore means a certified analyser, a Director controlling the run and the MLCommons power code.

How do I turn a performance-per-watt figure into inference cost for my GenAI workload?

Invert it. Throughput per watt is the inverse of energy per output, so samples per joule becomes joules per sample, and joules convert directly to kilowatt-hours and then to money at your tariff. The work is in the steps after that: deciding what one “sample” means for your service, and correcting for every way your serving setup differs from the benchmark run.

A sample in the benchmark is not your request. Energy per response is energy per token multiplied by the number of output tokens, and output length varies enormously by task. ML.ENERGY’s analysis found problem-solving prompts produced about 10 times more output tokens than text conversation, and used about 25 times more energy per response (mean 4,625 J against 184 J). Longer outputs also prevented larger batches. Input length matters as well. TokenPowerBench (2025) measured energy per token for Llama 3 70B rising roughly threefold as prompts grew from 2K to 10K tokens, because prefill must process every input token. Model size raises energy: Llama2-70B uses 4.2 times GPT-J’s energy per inference. The question of when a 32B model fits belongs to model choice rather than to this conversion.

Worked example: from joules per sample to energy per million requests

Assumptions are stated as assumptions.

  1. Start from the published figure. MLPerf Power reports Llama2-70B at 111.4 J per sample on a large datacenter system.
  2. Convert to kilowatt-hours. One kWh is 3,600,000 J, so 111.4 J is about 0.0000309 kWh, which is about 30.9 kWh per million samples.
  3. Price it. The energy cost per million samples is 30.9 multiplied by your electricity price per kWh. Use your contracted tariff, not a national average.
  4. Re-express per token, if you need to. If a benchmark sample produces T output tokens, energy per token is 111.4 / T joules. If your requests average more output tokens than the benchmark’s, scale up in proportion before pricing.
  5. Add what the benchmark excludes. Cooling, storage, idle reserve capacity and pre-processing all sit outside the figure. Google’s 2025 production study put idle machines at 10% and data-centre overhead at 8% of its own per-prompt total. Use those shares as a sense of scale, not as factors to copy, because they describe Google’s fleet.
  6. Check utilisation. The benchmark ran at whatever load the submitter tuned. If your traffic keeps the GPUs half busy, your energy per output will be worse, because static power is drawn whether or not useful work happens.

Inference energy recurs for as long as the service runs. That is why getting step 4 and step 6 wrong costs more than getting the tariff wrong. TokenPowerBench builds the same kWh-to-dollars conversion into its summary, so budget and sustainability discussions start from the same sheet as the technical metrics.

How do I match a benchmarked scenario to my actual use case so the number is meaningful?

Pick the scenario whose load shape matches your traffic. A number from a different scenario does not describe your service. The MLPerf Inference paper defines them as follows.

Scenario What the benchmark does Reported metric Use cases it represents
Single-stream One query at a time; the next is sent when the previous completes 90th-percentile latency Responsive client applications, one user at a time
Server Queries arrive at random (Poisson), each with a latency bound of 15 to 250 ms depending on the benchmark Queries per second sustained while meeting the QoS bound Online services: chat, translation, consumer-facing endpoints
Offline A throughput benchmark: samples processed per second Samples per second, and samples per joule with power Batch jobs: overnight summarisation, document processing, backfills

The trap is reading an offline efficiency figure as if it described a chat service. Larger batches lower energy per token. ML.ENERGY reports that increasing batch size can unlock reductions of 3 to 5 times, and TokenPowerBench measured a two- to threefold spread between its smallest and largest batches. Larger batches also raise per-request latency, so a latency deadline caps how far up that curve an interactive service can go. A latency-bounded server run cannot batch as far as an offline run. For anything a user waits on, the server-scenario result, or your own measurement under your latency target, is the one to price.

Compare like with like on accuracy and precision

Every MLPerf benchmark has a minimum accuracy target, and several workloads run at both 99% and 99.9% of reference accuracy. That choice moves efficiency a lot. The MLPerf Power paper found datacenter systems at 99.9% accuracy were on average half as efficient as those at 99%. It also found that recent submissions using more advanced quantisation keep strong efficiency at the higher target. MLCommons describes the gap as having narrowed significantly in later versions as reduced-precision techniques spread. Two results at different accuracy targets, or from different benchmark versions, are therefore not a hardware comparison. Read the accuracy target and the precision before the samples per joule.

When should I not trust an MLPerf Power result as a feasibility signal?

Treat an MLPerf Power figure as evidence about hardware efficiency on a standard task, never as your run-cost. LynxBenchAI, the TechnoLynx benchmark product, measures throughput, latency and energy across accelerators on one scale. It measures the hardware, not your production service. Be most careful in these cases:

  • Your model is much larger than anything benchmarked. The largest LLM in the MLPerf Power paper is Llama2-70B, and the authors say they cannot benchmark multi-trillion-parameter models. TokenPowerBench notes that Llama 3 405B-class deployments remain largely uncharacterised.
  • You will serve across nodes. According to TokenPowerBench, current MLPerf Power workflows emphasise single-node setups. Multi-node serving brings parallelism strategies with inter-GPU communication and network power draw that a single-node figure does not show.
  • You are comparing against a different boundary. Google’s study measured its median Gemini text prompt at 0.24 Wh on a comprehensive boundary, 2.4 times the figure from a narrower, high-utilisation approach. Llama 3.1 70B ranges from about 580 to about 3,600 prompts per kWh depending on the measurement approach. Google’s authors warn that without a standardised boundary, reported figures can vary by orders of magnitude for similar tasks.
  • Your utilisation will be lower than the submitter’s. Production systems hold idle reserve capacity for spikes and failover, and that energy is real cost. Where a non-GPU component (CPU-side preprocessing, for example) limits batch size, energy per token rises whatever the accelerator’s rating. Host-side limits like that are where a CPU spec for GenAI workloads starts to matter.
  • Your pipeline is more than the model. Pre-processing and data setup sit outside the measured execution phase.

The error can also run the other way. Google argues that production serving can be more efficient than benchmark experiments, because prompts are batched at scale and benchmarks often use open models that are not on the efficiency frontier. The direction of the gap therefore depends on your deployment. The TokenPowerBench authors frame the split well: MLPerf answers which hardware is more efficient on a standard task, while their tool answers the real energy cost of your configuration on your hardware. For the wider question of how to read MLPerf numbers without overreading them, see MLPerf results and where the benchmark stops short.

How does inference energy cost decide whether a use case is automatable, speculative or research?

Energy cost reclassifies a use case when the per-task run-cost, priced at realistic production volume and boundary, is comparable to the value the automated task returns. Treat the sorting below as an engineering judgement to test against your own numbers, not as a published threshold.

TechnoLynx’s generative AI R&D work turns benchmark power and performance figures into a cost-per-request estimate rather than relying on vendor peak-throughput claims.

Feasibility comes first. The first question in any generative AI engagement is whether the outcome is plausible at current capability. Repeatable tasks that people do today are far more likely to show measurable ROI than problems that need super-human capability. Energy enters once a use case passes that test, inside the ROI estimate for the part that can be automated.

The three outcomes sort roughly like this:

  • Automatable. The model already does the task to the required quality, and the run-cost per task, priced from a measured energy per response that includes real output lengths, batching limits and overheads, sits well below the value of the task. Short-output, high-volume work tends to land here, because energy per response scales with output tokens.
  • Speculative. The task works, but the margin depends on assumptions the benchmark cannot confirm: an offline efficiency figure for a latency-bound service, an accuracy target looser than you need, or full utilisation you have not shown. Long contexts and long reasoning outputs push work here, because, as the ML.ENERGY and TokenPowerBench measurements show, both raise energy per response.
  • Research. The required model sits beyond anything measured, or the energy per answer reaches a scale the benchmark does not cover. The MLPerf Power authors note that as models grow to multi-trillion parameters, energy reaches multiple kilojoules for a single question-answer inference.

The broader go/no-go decision, which weighs this energy estimate against data, quality and integration risk, is covered under GenAI technical feasibility.

Back See Blogs
arrow icon