Ask most teams how they plan to make their AI greener and the answer arrives as a procurement line item: buy carbon offsets, move the workload to a cloud region running on renewables, tick the box. That reduces the accounting footprint of the compute you already provisioned. It does nothing about the compute you shouldn’t have provisioned in the first place. The largest, most controllable efficiency lever in a deployed AI system is the same one that governs latency — right-sizing the model to the actual device capability baseline before you choose an architecture. This matters most at client-side scale. A server-side model that over-provisions compute burns extra energy in one place you can meter and optimise later. A client-side model that over-provisions burns it on every inference across millions of user devices, in every session, forever, on hardware you don’t control and can’t retune after shipping. The waste doesn’t happen once during training. It compounds across the fleet. Why offsets solve the wrong problem Carbon offsets and renewable cloud regions are real tools, but they operate downstream of the decision that actually determines energy consumption. They change where the electrons come from or who accounts for them. They don’t change how many joules each inference costs. If your image-classification model runs a 200M-parameter backbone where a distilled 40M-parameter one would hit the same accuracy target, no cloud region on earth makes that a good decision — you’re paying the 5x compute tax on every frame regardless of the grid mix behind it. The reason this framing keeps winning is organisational. Offsets live in the procurement and sustainability functions, where “greener AI” is treated as a purchasing question. Right-sizing lives in the engineering function, where it’s treated as a latency question. The two rarely meet, so the biggest lever stays outside the conversation labelled “environmental.” Our position is blunt: environmental efficiency and latency efficiency are the same discipline wearing two badges. Both are governed by measurement first, then architecture. The measurement divergence The divergence point between teams that ship efficient AI and teams that don’t is where they profile. Teams that establish a device capability and inference-cost baseline before selecting an architecture can distil, quantise, and route work to the cheapest sufficient path. Teams that skip profiling pick an architecture on paper — usually the largest one that fits their accuracy target on a beefy dev machine — and discover the energy and latency cost only after it’s in front of users, when the fix is a rewrite. Profiling here means the same instrumentation you’d use to prevent a latency miss: measure inference wall-clock, memory footprint, and — critically — energy per inference across a representative slice of the device fleet, not just the flagship phone on the engineer’s desk. On mobile SoCs you can read this from platform power-monitoring APIs; on a laptop or edge box you can bracket it with package-level power counters (Intel RAPL, nvidia-smi power draw) around a fixed inference loop. The same baseline that surfaces a latency miss surfaces the energy waste, because they share a root cause: compute the median device doesn’t need. This is why the profiling-first methodology travels beyond client devices. On server GPUs the currency is a power budget rather than a battery, but the discipline is identical — establish the executor’s real inference cost before you commit an architecture. If you’re reasoning about that server-side trade-off, our breakdown of what the DeepSeek-on-H100 pairing means for retail CV cost walks the same baseline logic through a datacentre GPU. How distillation and quantisation cut energy without breaking latency Two levers do most of the work, and they compose. Knowledge distillation trains a smaller student model to reproduce a larger teacher’s outputs, collapsing parameter count while retaining most of the accuracy. Quantisation reduces the numeric precision of weights and activations — FP32 down to INT8, or lower — so each operation moves fewer bits and, on hardware with integer or low-precision units, executes on cheaper silicon paths. Distillation shrinks how much you compute; quantisation makes each computation cheaper. The reason these hold the latency target while cutting energy is that both reduce the same underlying quantity: arithmetic operations and memory traffic per inference. Energy on a mobile or edge device is dominated by data movement, so an INT8 model that reads a quarter of the bytes of its FP32 parent draws proportionally less power even before you count the faster arithmetic. Frameworks make this reachable without hand-rolling kernels — PyTorch’s post-training quantisation and quantisation-aware training, ONNX Runtime’s INT8 execution providers, and TensorRT’s calibration flow all target this. If you want the precision-format detail, our explainer on FP4 and when to use it for edge inference covers where sub-INT8 formats hold up and where they collapse. In configurations we’ve tested, right-sizing a client-side vision model through combined distillation and quantisation cuts inference compute — and energy per inference — by roughly 2–5x while holding the same latency budget (observed pattern across TechnoLynx client-side deployments; not a published benchmark). At fleet scale that multiplier is the whole story: the same reduction applied across every device and every session compounds into aggregate energy and carbon savings that a single baseline-driven architecture decision locks in, with no post-deployment rewrite. Choosing the inference path: on-device, edge, or cloud Right-sizing the model is half the decision. The other half is where the inference runs, because the path changes both the energy profile and who pays for it. Sending every frame to the cloud spends network-transport energy and datacentre compute; running on-device spends battery but avoids the round trip. There’s no universal winner — the answer depends on model size, request frequency, network conditions, and how many devices you’re multiplying across. Decision table: matching inference path to workload Factor On-device Edge (near-user server) Cloud Energy per inference Lowest for small models; battery-bound Moderate; shared silicon amortises Highest per-request once transport is counted Transport energy None Short hop Full round trip, every request Best when Model fits device baseline; high request frequency Model too large for device but latency-sensitive Model can’t be right-sized below device or edge capacity Fleet-scale carbon Multiplies smallest per-inference cost Amortises across users in a locale Concentrates load in a meterable, greenable location Retune after ship Hard — can’t touch client hardware Possible — you own the box Full control The table encodes one principle: prefer the cheapest sufficient path, and let the right-sized model’s footprint decide what “sufficient” means. A model distilled to fit the median device baseline unlocks on-device inference, which removes transport energy entirely across the fleet. A model that can’t be shrunk below device capacity forces the round trip, and now the cloud region’s grid mix — the thing offsets address — finally becomes the relevant lever, but only after right-sizing has failed to remove the trip. The routing story for client-constrained inference runs deeper in our look at applying deployment discipline to client-side ML inference, and for the hardware-fit side, where DGX Spark fits in the edge latency/cost trade-off covers the near-user server tier. This right-sizing discipline is the same one we bring to constrained-inference work in media and telecom deployments, where fitting a model to a diverse device fleet directly reduces per-inference compute at the scale that matters. A worked measurement plan Concrete beats abstract. Here’s the sequence we’d run before committing an architecture for a client-side vision feature, with explicit assumptions. Define the fleet slice. Enumerate the device tiers your users actually run — say a flagship, a three-year-old mid-range, and a low-end SoC — weighted by real install-base share, not by what’s convenient to test. Fix the workload. Pin a representative input (resolution, frame rate, batch size) and an accuracy target the feature must meet. Without a fixed accuracy floor, energy comparisons are meaningless. Baseline the candidate architecture. Run the largest plausible model on each tier and record latency, memory, and energy per inference (from platform power APIs or bracketed package counters). Right-size and re-measure. Distil, quantise, and re-run the same loop. The delta in energy per inference, multiplied by your fleet weighting and session frequency, is your aggregate saving. Route on the result. If the right-sized model clears the latency target on the median tier, ship on-device. If it clears only on the top tier, consider an edge tier or a device-tiered rollout. Cloud is the fallback when nothing right-sizes below capacity. The output isn’t a single number — it’s a defensible architecture decision with the energy consequence attached before anyone writes shipping code. That’s the whole point of profiling first. FAQ How to make AI more environmentally friendly? The largest controllable lever is right-sizing the model to the device capability baseline before selecting an architecture, not buying offsets after the fact. An over-provisioned client-side model wastes energy on every inference across every user device, so shrinking it through distillation and quantisation compounds savings at scale. Offsets and renewable regions help only after you’ve eliminated compute you never needed. Why is right-sizing the model to the device capability baseline the biggest controllable energy lever for client-side AI? Because the waste multiplies. A model that over-provisions compute burns the excess on every inference across millions of devices and every session, not once during training. Right-sizing to the median device’s real capability removes that per-inference tax fleet-wide with a single architecture decision, which no downstream procurement choice can match. How do distillation and quantisation reduce energy per inference without breaking the latency target? Both reduce the same underlying quantity — arithmetic operations and memory traffic per inference. Distillation collapses parameter count by training a small student to mimic a larger teacher; quantisation lowers numeric precision so each operation moves fewer bits and runs on cheaper silicon paths. Since energy on edge devices is dominated by data movement, both cut power while staying inside the same latency budget. How does choosing the right inference path (on-device, edge, cloud) change AI’s energy and carbon footprint? On-device inference removes network-transport energy entirely but is bounded by the device’s capability; cloud spends transport energy on every request but concentrates load where it can be metered and greened; edge sits between. A right-sized model unlocks the cheapest sufficient path, so the model decision and the routing decision compound. How do I measure energy-per-inference across a diverse user device fleet before committing to an architecture? Define a fleet slice weighted by real install-base share, fix a representative workload and accuracy floor, then baseline the candidate architecture on each tier using platform power APIs or bracketed package-level power counters. Right-size, re-measure the same loop, and multiply the per-inference energy delta by fleet weighting and session frequency to get the aggregate saving. Why do offsets and renewable cloud regions matter less than avoiding over-provisioned models at client-side scale? Offsets and renewable regions change where the electrons come from or who accounts for them; they don’t change how many joules each inference costs. An oversized client-side model pays a compute tax on every frame regardless of the grid mix, and that tax multiplies across the fleet — a cost no downstream accounting choice removes. How do latency efficiency and environmental efficiency converge on the same profiling-first discipline? Both are governed by the same root quantity: compute the median device doesn’t need. The device capability and inference-cost baseline that surfaces a latency miss also surfaces the energy waste, because they share a cause. Establish the baseline first, then select the smallest architecture that meets the target — and you’ve optimised both at once. Right-sizing is a methodology, not a one-time cleanup: the moment your device fleet shifts or your accuracy target moves, the baseline is stale and the smallest-sufficient architecture changes with it. The question worth carrying into your next planning cycle isn’t “how do we offset this model” — it’s “what’s the smallest architecture that still meets the target on the median device we actually ship to?” Answer that with a profiling-first inference-cost baseline, and the energy question answers itself.
