Ask most teams how to make AI more environmentally friendly and you get a procurement answer: pick a low-carbon cloud region, buy offsets, publish the footprint of the training run once. None of that is wrong. It is also almost entirely outside the control of the people building the system. The operational answer is narrower and far more useful — the energy consumed per inference, multiplied by every session on every device, decided at architecture selection rather than at reporting time.
Everything actionable depends on where you draw the measurement boundary.
What does “environmentally friendly AI” cover in practice?
Scope the footprint to training and hosting, and you get a fixed number. You can offset it, you can move it to a greener region, and you can report it — but you cannot reduce it, because the decisions that produced it are already behind you.
Scope it to inference across the deployed fleet, and you get a variable. Energy per inference is a function of model size, precision, batching behaviour, and the device the model actually runs on. Every one of those is a design choice a delivery team makes, and every one of them can be profiled.
That is the divergence. Reporting-side levers change the carbon coefficient applied to your consumption; engineering-side levers change the consumption. The two are not interchangeable, and treating them as such is the most common reason a sustainability commitment produces no measurable change in the running system.
Where the levers actually sit
| Lever | Changes what | Controlled by | Effect on energy per inference |
|---|---|---|---|
| Carbon offsets | Reported carbon | Procurement / finance | None |
| Low-carbon region selection | Reported carbon coefficient | Infrastructure | None |
| Training-run footprint disclosure | Reporting completeness | ML team | None (retrospective) |
| Model right-sizing (distillation, quantisation) | Actual compute per request | Delivery team | Direct — the primary lever |
| Batching and scheduling | Utilisation per unit of work | Platform team | Direct, workload-dependent |
| On-device vs server placement | Where the energy is spent | Architecture | Direct, shifts the profile |
Why energy per inference is the operative metric
A training run happens once. Inference happens on every session, on every device, for the life of the product. For anything shipped at consumer scale — a vision feature in a mobile app, an on-camera detector, a per-request generative call — the cumulative inference cost dominates long before the training footprint is amortised.
Expressed operationally, the metric is compute cost per request against a fixed latency target, measured on the median device in the fleet rather than on a developer workstation. The workstation number is the one that flatters you; the median-device number is the one that governs what users actually consume. Once that figure exists, right-sizing through distillation and quantisation typically moves it by 2–5x while holding the same latency budget (observed pattern across TechnoLynx constrained-inference engagements; not a published benchmark).
Note what that implies: the environmental question collapses into the latency question. Both are answered by the same profiling pass — where the compute goes, at what precision, on what hardware. Teams already doing serious computer vision engineering under device constraints have most of the instrumentation in place; they have simply been reading it as a performance artifact rather than an energy one. The same applies to edge workloads in telecom and media infrastructure, where per-stream compute budgets and per-stream energy budgets are the same constraint wearing different units.
What has to exist before you can claim right-sizing
- A profiled baseline: energy or compute per inference at the current architecture, on representative hardware.
- A declared latency budget the reduced model must still meet.
- A precision map — which layers tolerate INT8 or FP16 and which do not, established by measured accuracy rather than assumption.
- A fleet distribution, so “median device” means something specific.
- A non-AI comparison point, if the feature replaces a previous approach.
Without the baseline, distillation and quantisation are guesses dressed as sustainability work. Tooling here is unremarkable and available: ONNX Runtime and TensorRT for the quantised deployment path, PyTorch profiling hooks and vendor power counters for the measurement side.
Water use and data-centre cooling belong in the reporting boundary, not this one — they are real, but they scale with consumption you have already fixed by the time you are reporting. Reduce the consumption and the cooling load follows.
The open question we keep returning to is comparative: how do you defensibly measure an AI feature’s energy cost against the non-AI baseline it replaced, when the baseline was never instrumented in the first place? For the deeper treatment of constrained-inference architecture decisions, see the parent discussion on designing vision systems for edge and device deployment.
