No single metric decides a Jetson AGX Orin gateway rollout. The lead metric should come from whichever constraint actually binds the product (the power budget, the unit cost or a hard latency deadline). Weight the other metrics around it, in the open, then check the whole weighting on a pilot of real units before a multi-year, 50,000-unit SKU is locked in. A September 2026 arXiv survey of AI deployment optimisation puts the risk plainly: the most common deployment failure is optimising the wrong metric. Orin shows how that happens. The same model on the same board can have a different winning configuration depending on whether you rank by latency or by energy.

Why 275 TOPS says little about your gateway

NVIDIA’s datasheet rates the 64GB module’s GPU at up to 170 INT8 sparse TOPS and its Deep Learning Accelerators at up to 105 INT8 sparse TOPS. Add them together and you get 275. The 32GB module is rated at 108 plus 92, which comes to 200.

Achievable peak compute sits well below the rating. A 2025 roofline study of the AGX Orin found achievable peak compute far below the theoretical specification. It also found that all five DNN inference workloads it tested were memory-bound in the default MAXN mode, so peak compute was not what limited them.

On fixed hardware, the software stack moves the result too. The September 2026 survey reports that ResNet-152 on a Jetson AGX Orin showed a 125× spread in latency across five inference frameworks: 2.3 ms under TensorRT and 285 ms under ONNX Runtime. Closing the gap between a model and the runtime it is deployed on is GPU performance engineering work, not a hand-off.

Power mode adds one more variable. The 64GB module offers 15 W, 30 W and 50 W modes, with up to 60 W available, and nvpmodel sets how many GPU, CPU, DLA and PVA cores are online, and at what clocks, to stay inside each budget. A gateway capped at 15 W runs fewer cores or lower clocks than the spec-sheet configuration. An August 2026 Jetson benchmark found the AGX Orin underused at 15 W, with an idle baseline of about 10.9 W against about 5.3 W for the Orin Nano Super.

Which metric should lead the Jetson AGX Orin gateway choice?

Lead with the metric that matches your binding constraint:

• Energy per inference when the gateway runs on a power or battery budget.
• Cost per device, spread over the inferences it will run in service, when unit cost dominates at fleet scale.
• Tail latency when a downstream deadline is hard.

Combine the remaining metrics into a weighted score only after the lead metric is fixed.

LynxBenchAI publishes measured results showing how devices run the public PyTorch stack that custom models are built on. That makes its results broadly indicative for custom PyTorch models, but it does not run or measure your own gateway workload. Start from the published LynxBenchAI results and methodology, then confirm the choice on your own units.

The September 2026 survey frames deployment as a constrained optimisation over accuracy, latency, throughput, memory footprint and energy. Its selection algorithm takes the binding constraints and the metric weights as explicit inputs, rather than burying them in a convention.

Is watts-per-inference more decisive than throughput on a power budget?

For a gateway on a fixed power or battery envelope, usually yes. A faster configuration that draws a lot more power can cost more energy for every result it produces. Energy per inference is power multiplied by latency, so a configuration with 2× lower latency but 3× higher power draw uses 1.5× more energy per inference.

Orin has a concrete case. For Swin-T, TVM used 0.085 J per inference against TensorRT’s 0.108 J, even though TVM’s latency was higher (5.28 ms against 3.95 ms), because it drew 16.0 W against 27.4 W. Ranked by latency, TensorRT wins. Ranked by energy, TVM wins. Which ranking applies depends on which constraint binds. The default power mode is no safe choice either: the 2025 roofline study found MAXN was not the most energy-efficient of the 96 Orin power modes it sampled.

A power cap and an energy budget can lead to different choices. A 2025 study of LLM inference on the Orin AGX found that one low-GPU-clock mode cut power by 51% against MAXN but raised total energy. The authors judged it suited to power-capped setups rather than to saving energy. If your limit is instantaneous watts (a PoE port, a thermal ceiling), cap the power. If your limit is battery capacity, minimise joules.

Throughput stays as a secondary check: the board still has to keep up with the incoming frame rate. Achieved utilisation also feeds back into energy. When throughput is low, the fixed idle overhead is spread across fewer inferences, which raises joules per inference. SPEC’s power methodology recommends measuring both the maximum-performance point and an active-idle period, because they bound the system’s work done per unit of power.

Trading tail latency against average throughput when the workload is not real-time

For anomaly detection that is not strictly real-time, give more weight to average throughput, but keep a hard ceiling on a tail percentile. The tail still matters when a downstream SLA depends on it, even when nothing in the gateway itself is time-critical.

Independent benchmarks make this split explicit. MLPerf Inference’s offline scenario reports throughput in samples per second with no latency constraint. Its server scenario reports the queries per second a system sustains under random Poisson arrivals while staying inside a latency bound. Reddi et al. (2020) found that the server scenario’s constraint cost some systems as little as 3% of their offline throughput and others about 50%. So an offline number has little bearing on what a latency-bounded gateway will deliver.

Headroom also matters more than it looks. Under the M/M/1 queueing approximation, response time is twice the model time at 50% utilisation and ten times the model time at 90%. The September 2026 survey adds that near saturation, a 5% cut in admitted request rate can reduce p99 latency by 30% or more. Power-saving modes have their own cost: they add latency when a device wakes from an inactive state. A gateway that sleeps between bursts will show that in its tail, not in its average.

Be specific about which latency you mean. Kernel time, model latency, request latency and service latency are four different quantities, and for the same model on the same hardware they can differ by an order of magnitude.

Which metric do independent benchmarks put first?

MLPerf attaches a metric to each scenario instead of crowning one number. MLPerf Inference deliberately provides no summary score, because weighting some tasks above others is highly subjective. Its four scenarios (single-stream, multistream, server, offline) came from surveying its customer and vendor members about real usage models. Single-stream, for example, is scored on 90th-percentile latency.

MLPerf Power applies the same logic to energy. Throughput benchmarks are scored in samples per joule, and latency benchmarks in the inverse of energy per inference. The authors also insist on measuring full-system power rather than accelerator power alone. Our article on why there is no single benchmark number and how to build defensible evidence for each comparison takes the measurement side of this further.

The vendor’s own published figures show what a MaxQ configuration trades. NVIDIA’s MLPerf v3.1 results for the NVIDIA Jetson AGX Orin Developer Kit list ResNet at 6,423.63 offline samples per second. The power-limited MaxQ configuration lists 3,526.29 samples per second at 23.57 W system power. That is about 55% of the throughput (3,526.29 ÷ 6,423.63), with a measured system power figure attached.

Deriving the weights from the binding constraint

Derive the weights in a fixed order:

1. Apply the cheap hard filters first.
2. Fix the lead metric from the binding constraint.
3. Assign weights to the rest.

The September 2026 survey’s procedure checks accuracy and memory first, because they cost little to evaluate, and leaves latency and energy, which need measurement on the target hardware, until after the clearly infeasible candidates are gone.

The weights are not a formality. In the survey’s worked Orin example, constraints of at least 73% accuracy, at most 10 ms latency and at most 0.25 J eliminated 5 of 11 candidates on accuracy alone. Among the survivors, equal weights picked Swin-T on TVM, and latency-heavy weights picked ResNet-152 on TensorRT.

Binding constraint	Lead metric	Secondary checks	Measure it under
Power or battery budget	Energy per inference at the deployed power mode	Throughput meets the required input rate; model fits in memory	Full-system power, sustained run
Hard latency deadline	Lowest latency after memory and accuracy filters, with energy as the tie-breaker when latencies are within 5%	Memory fits; accuracy floor met	Deployed batch size and arrival pattern

The weighting decides between boards as well as between runtimes. The August 2026 AgriJetsonBench study found that the Orin Nano Super beat the AGX Orin at a matched 15 W budget in INT8, while the AGX Orin won on throughput and latency in its native high-load FP16 mode. Its guidance: choose AGX Orin when latency or frame rate matters more than maximum runtime, and evaluate architecture and precision directly, because a speed advantage does not imply an energy advantage.

The board ranking and the weighting behind it shift with precision and power mode. Treat both as inputs to revisit, not constants.

Power numbers depend on where you put the meter

Watts-per-inference is only as good as the power reading behind it. A 2023 calibration study found that the internal power sensors on Jetson devices, AGX Orin included, underestimate actual power by up to 50% in most cases, and that calibrating them against an external setup brings the error within ±3%. On the AGX Orin there is no whole-board sensor, so total power is estimated by summing individual rails.

The AgriJetsonBench authors compared 76,691 paired samples and found internal telemetry read about 1.99 W lower than an external board-input logger on average. They treat the external logger as the primary energy reference. tegrastats and the Jetson Power GUI remain useful for trends, temperature and logging. Any reported figure should state:

• the power mode
• the precision
• the timing boundary
• whether power is accelerator-only, board-level or platform-level

The September 2026 survey notes that the ratio between accelerator-only and platform power is typically 1.5–2.5×.

How do we pilot-test the metric weighting on real gateways before committing the SKU?

Put the weighting on several real Orin gateways, built to the production configuration and running in representative conditions. Measure them at staged checkpoints, and set thresholds from the spread you actually observe across units, not from one development kit or the datasheet.

LynxBenchAI publishes measured results showing how devices run the public PyTorch stack, which gives the pilot a reference point for each candidate board before your own gateways are measured; the first step is to read its published results and methodology.

TechnoLynx can design and run that pilot for you. That covers defining which metrics and thresholds it must validate against the SKU commitment, sizing the sample and test conditions, instrumenting the Orin units, and deriving margins from the measured variance. The first step is agreeing the error budget the multi-year commitment can tolerate.

Several units are needed because nominally identical Jetson boards are not identical. An EdgeSys 2021 study of 13 identical Jetson AGX Xavier boards, run under the same inputs and frequency settings, found that board pairs differed at the 1% significance level in about 77% of sampled configurations for runtime and about 99% for power. Most runtime differences fell within ±5%, but some ranged from −10% to +20%. Power mostly fell within ±10%, with some cases from −30% to +20%. The authors concluded that a performance or power model built on one board predicts poorly across the rest, so data must come from several boards.

Orin’s own unit-to-unit spread is one more thing to measure in your own pilot.

Data-centre GPUs point the same way. A 2022 study found 8% average performance variation (maximum 22%) across GPUs with the identical SKU. It also found that variability rose under lower power limits: 18% at a 150 W cap against 9% at 300 W. That study measured data-centre GPUs, not a 15 W Orin. Treat it as a warning, not a number. NVIDIA’s own thermal design guide tells integrators to build in margin for module-to-module variation.

Representative conditions matter too. Pilot practice calls for samples that reflect the diversity of the full deployment, and for blocking by site or line when those factors could sway the outcome. On Jetson, TensorRT engines are tuned to the device they are built on: validate each engine on the deployment device, and run INT8 calibration on the target. Our guide to de-risking edge deployment through quantisation, thermal validation and toolchain porting for Jetson targets covers those engine-level checks.

A staged pilot, in order

1. Fix the metrics, thresholds and pass criteria before any unit is measured. Then record a baseline. Predetermined criteria are what separate a pilot from “try it and see”.
2. Source enough units to estimate spread. A small pilot has a high potential to underestimate variability.
3. Lock and record the configuration. That means JetPack, TensorRT and CUDA versions, power mode, precision and batch size. Driver and toolkit versions, thermal throttling and run order all shift deployment measurements.
4. Measure every unit the same way. Use sustained platform-level power over at least 60 seconds, and latency at mean, p50, p95 and p99 over at least 1,000 iterations after warm-up. Take power from an external meter or a calibrated sensor, and include an active-idle measurement.
5. Deploy to field sites in stages, with interim checkpoints. Watch trends over time, not single readings. If software thermal management cannot hold the SoC temperature, hardware clock throttling engages, so check the field enclosure, not only the bench.
6. Analyse the spread across units, derive margins and stress-test them against the worst units.
7. Re-validate after the first production tranche, before the rest of the order is committed.

What margin should thresholds carry for production variance across 50,000 units?

There is no universal percentage. Take the margin from the spread measured across pilot units, then add a safety factor for effects the pilot cannot see at its scale. Do not assume datasheet tolerances.

Pilot size and margin trade against each other. A 2021 sample-size paper on clinical pilots warns that a small pilot has a high potential to underestimate variability. That context is not a hardware fleet, but the lesson carries over: a smaller pilot needs a wider margin or a planned re-check.

If the fleet is safety- or SLA-critical, use more units, more batches and wider margins. If time and budget are tight, run a smaller pilot, write down the residual risk you are accepting, and re-validate on early production.

The estimate is fragile in known ways:

• Firmware updates, process changes or component substitutions between pilot and production can invalidate it.
• Sites that differ from the pilot sites can bias it.
• Thresholds set too tight produce false rejects, and thresholds set too loose let genuine problem units through.

Rare long-tail failures may only show up at full fleet scale. On the thermal side, NVIDIA specifies a ±3 °C temperature-sensor uncertainty that the design’s headroom has to absorb. In the data-centre study, 8 of the 10 worst-performing GPUs on SGEMM were also ResNet-50 outliers.

Who can design the pilot and check the hardware choice before we commit to 50,000 units?

TechnoLynx designs and runs Jetson AGX Orin pilot validation for this kind of commitment. That means turning your binding constraint into an explicit metric weighting, instrumenting real gateways with calibrated power measurement, and setting threshold margins from measured unit-to-unit variance. A specialist earns its place where the SKU decision is irreversible and the pilot has to stand up in front of procurement. For the earlier step of comparing candidate devices, LynxBenchAI is the measuring tool: its published results indicate how hardware runs the public PyTorch stack, not your workload.

This engagement is not a fit if you are looking for hours of benchmarking with no decision attached, or if the hardware is already fixed and only deployment tuning remains. In that case, engine-level optimisation is the better starting point.

FAQ

What is Jetson AGX Orin?

Jetson AGX Orin is an NVIDIA embedded module that pairs an Arm Cortex-A78AE CPU complex with an NVIDIA Ampere GPU. Its maximum module power is up to 60 W for the 64GB version and up to 40 W for the 32GB version.

Is higher throughput better or worse?

Higher throughput is better only if the latency bound still holds. In MLPerf testing, adding a latency constraint cut throughput by anywhere from 3% to 50% depending on the system.

How much power per AI query does an Orin use?

It depends on model, precision and power mode. In NVIDIA’s MLPerf v3.1 results, the power-limited AGX Orin configuration ran ResNet at 3,526.29 samples per second at 23.57 W system power, which works out to about 6.7 mJ per image (23.57 ÷ 3,526.29).