Judge each AI hardware accelerator as an executor, the silicon plus the software stack that drives it, because performance belongs to that combination and never to silicon alone. After that, we would apply three filters. The first is operator coverage, including fallback paths that silently shift work to the CPU. The second is stack maturity (CUDA, ROCm, oneAPI, or an NPU toolchain such as TensorRT, OpenVINO, RKNN, QNN or the Edge TPU compiler). The third is fit to the deployment envelope of power, lifecycle and board support.

LynxBenchAI measures GPUs and CPUs with stock PyTorch across backends and precisions; run it on your device and compare the result with that device’s 26Q3 page. It does not run your own model. The 26Q3 release covers no edge NPUs. The wider shortlisting of hardware and stacks for AI compute covers how this step fits into procurement.

Which AI hardware accelerators exist, and what each one is built for

What matters for a shortlist is the kind of silicon and how much of your model it can actually run. The rows below cover the build styles you will meet, from datacenter cards to edge boards.

Type	Built for	What to verify before it goes on the list
GPU	Programmable parallel compute for training and inference across model families	The vendor stack (CUDA, ROCm, oneAPI) for your kernels and precisions
NPU (discrete ASIC or integrated in an SoC)	AI inference at lower power than GPU-only designs	Operator coverage: in OpenVINO, the NPU plugin has the most limited operator set
TPU	Google’s tensor ASICs, including the Edge TPU for quantised models at the edge	A model runs entirely on the Edge TPU only if it is a TensorFlow Lite model, fully 8-bit quantised, compiled for it and limited to supported operations
DLA	NVIDIA’s fixed-function engine for convolutional networks on Jetson	Layer support, plus whether GPU fallback exists in your TensorRT and JetPack version
FPGA	Reconfigurable logic for fixed, low-latency dataflows	Toolchain and engineering effort for your model
Fixed-function ASIC	One model class run very efficiently	Whether your architecture compiles at all, especially transformers

One evaluation of ten edge accelerators (the NPU Hardware Evaluation white paper, 2026) found that all three ASIC NPUs it reports failed on transformer-based models. The authors call this a basic limit of fixed-function silicon. On a mixed or changing model portfolio, that one finding should weigh more than any throughput figure.

Is AI possible without a GPU, and when can an NPU carry inference alone?

Yes, AI runs without a GPU. The real question is which part of the work. In the 26Q3 results, CPUs produce real scores. The AMD Ryzen 5 9600X scores 5.2 GT and 2.7 in inference, while the NVIDIA GeForce RTX 4060 scores 106.3 GT and 155.0 in inference. GT is an ordinal aggregate, so that gap tells you the order of the two devices, not a ratio between them.

A dedicated accelerator card or NPU can carry inference on its own when the model fits its operator set and precision. In an independent benchmark of microcontroller-class NPUs (Benchmarking Ultra-Low-Power μNPUs), dedicated neural hardware gave up to two orders of magnitude better end-to-end latency than MCUs without it. The same ten-accelerator evaluation found the Hailo-8’s multi-stream throughput competitive with, or above, an RTX A5000 for several lightweight convolutional models, inside its operator envelope. NVIDIA describes the DLA as significantly more power-efficient than the GPU for CNN work, and as an independent pipeline where redundancy matters.

Precision support matters here as well as silicon. The Apple M1 produces no INT8 inference result in 26Q3 for Flan-T5 Small, DistilGPT2 or DistilBERT (a 0.0 means that precision gave no result on that executor). Yet it does score 9.7 at INT8 on MobileNetV3. Training is a different matter: we would keep it on GPU-class executors unless a measured result says otherwise.

Why headline TOPS and spec sheets mislead

The μNPU study concluded that GOPS is not a reliable predictor of end-to-end latency and that memory bandwidth has a very large effect. A 2026 cross-platform INT8 study states the design rule bluntly: for edge accelerators behind a bus, provision by data movement, not by TOPS. TOPS is proportional to the number of multiply-accumulate units times the clock frequency, which describes a synthetic peak, not the throughput or latency a product designer cares about. Hailo, an AI accelerator vendor, makes that argument and reports only a weak correlation between TOPS and ResNet-50 results.

Datacenter parts show the same gap. SemiAnalysis’s late-2024 training benchmark measured H100 and H200 at roughly 720 TFLOP/s of BF16 against a marketed 989.5, and MI300X at about 620 against a marketed 1,307. That left the part with the higher paper figure about 14% slower on BF16 matrix multiplication. Google Cloud’s accelerator benchmarking guide makes the general point: effective FLOPS are what matter, since peak numbers are almost never reached in real workloads.

Inside one release, the 26Q3 results show why one number can’t stand for a device. The AMD Instinct MI300X scores 1023.3 in compute against 487.1 for the NVIDIA H100 80GB HBM3. In inference it scores 1304.0 against the H100’s 1678.1. In 26Q3, the compute and inference categories order the same two executors in opposite directions.

Is CUDA necessary for AI, and how far does ROCm move the numbers?

CUDA is not necessary: AMD, Intel and CPU executors run real AI workloads and appear in the 26Q3 results. Even so, SemiAnalysis’s MI300X and MI325X tests found AMD’s software stack lagging CUDA, and that gap showed up in measured performance and accuracy. LynxBenchAI measures each device through the backend it actually runs on (cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU) and binds every score to the driver, framework and runtime on that machine. The MI300X entry in 26Q3 is therefore a ROCm result, not a verdict on AMD silicon. Where a shortlist needs a workload moved off CUDA, TechnoLynx’s AI Porting & Deployment Pack starts by listing your kernels and precision targets. It then builds and runs them on each candidate stack.

The stack can move results a long way. In a study of BERT and GPT-2 on one RTX 3090, GPU execution gave 5.4x over CPU, ONNX Runtime added 1.2x and TensorRT graph optimisation another 1.5x, all on the same silicon. The platform layer matters too. The RTX 5090 scores 597.5 GT natively on Linux and 545.7 GT under WSL in 26Q3.

What the public evidence says about ROCm maturity

The public evidence cuts both ways. SemiAnalysis’s 2025 inference benchmark found that 25% of the models it tested failed accuracy tests on AMD, and that ROCm’s SGLang CI coverage was below 10% of NVIDIA’s. The same report found that sometimes a single-node H200 and sometimes a single-node MI325X had the better performance per dollar, depending on workload and latency target. A vLLM-on-ROCm setup guide (GIGAGPU) reports that one MI300X beats a single H100 80GB on Llama 70B because the 192 GB of HBM avoids tensor-parallel overhead. Our reading is that ROCm can win on a well-trodden path and lose badly one step off it. That is why the audit has to use your kernels rather than someone else’s headline.

Before signing for MI300X with vLLM on ROCm, we would pilot against these known issues:

• ROCm version: the GIGAGPU guide recommends ROCm 6.3 or newer, because earlier versions lack FP8 on MI300 and have broken paged-attention kernels on RDNA3.
• Quantisation format: AWQ works, while GPTQ is hit-and-miss on ROCm 6.3.
• Attention path: vLLM uses Composable Kernel attention on CDNA3 and Triton-based attention on RDNA, and the Triton path is slower.
• Graph capture: HIP graphs are still flaky on some RDNA SKUs, with --enforce-eager as the workaround.
• Accuracy: run your own evals on both stacks, given the accuracy failures SemiAnalysis reported.

Porting effort and other non-CUDA stacks

Porting effort depends on which kernels you need, not on a generic ROCm-versus-CUDA verdict. Two examples from AMD’s HIP porting guide: code must not assume a warp size of 32, since GCN and CDNA use 64, and CUDA’s __launch_bounds__ minimum-blocks parameter has to be converted to warps per execution unit. Hidden differences like these don’t show up on a feature checklist, so the audit compares behavioural and numerical outputs across stacks.

On Gaudi 3, Intel’s support matrix lists Gaudi Software 1.24.2 with PyTorch 2.13 and a vLLM hardware plugin at 0.30.0, and marks Megatron-LM and Slurm as not supported. Intel validates only the latest version and the two before it. That validation window is the kind of commitment we would want written into a multi-year contract, alongside the driver and framework versions it covers. On Qualcomm’s DSP family, including QCS8550, the ten-accelerator evaluation found near-complete model coverage (MobileNetV3 on QCS8550 was the exception) but a fragmented SDK spread across SNPE, QNN and AI Engine Direct. An INT8 model moved to a new target, such as Qualcomm, needs re-validating on that target. The 2026 INT8 study saw a bring-your-own quantised graph on the Hexagon NPU compile and run without error while accuracy fell from 0.75 to 0.005.

The decision rule is straightforward. Shortlist the non-CUDA stack for production only if your required kernels are stable, behave the same and port at acceptable effort. Otherwise keep CUDA as the baseline and treat the alternative as experimental. Stacks change quickly, so any such audit is a snapshot to repeat near the decision date, and a clean audit still needs monitoring afterwards.

Which edge boards run TensorFlow Lite and ONNX Runtime without falling back to the CPU?

Full TensorFlow Lite and ONNX Runtime operator coverage is not guaranteed on an accelerator delegate. Coverage has to be checked per model, because NPU delegates can hand unsupported operators back to the CPU. TechnoLynx’s AI Porting & Deployment Pack does this per-model audit as part of porting to an edge target, starting from your model’s operator list.

The mechanism is graph partitioning. NXP’s i.MX guide states that delegates need not support the full TensorFlow Lite operator set, and that unsupported operations run on the CPU with reference kernels while the graph is split into segments. On the Edge TPU it is harsher: the compiler partitions only once, so the first unsupported operation and everything after it runs on the CPU. Google’s Coral documentation warns that even a small CPU-resident fraction can slow inference by an order of magnitude.

The fallback can be silent. In the ExecuTorch evaluation (2026), the LiteRT and ONNX Runtime NPU paths claimed only a limited number of graph nodes, so most inference ran on the CPU. An agent-driven deployment study passed verification on RKNN, and only afterwards found that four of YOLO-World’s operators had fallen back to the CPU. ONNX Runtime’s QNN execution provider has the same behaviour: unsupported operators fall back to CPU, with potentially significant performance loss.

In the ten-accelerator evaluation, an RTX A5000 with TensorRT compiled all twelve reference models. We would prefer Jetson-class hardware with general GPU compute when full, unmodified coverage across frameworks is a hard requirement. On Cortex-M or Ethos-U targets, expect a narrower operator subset. TensorFlow Lite for Microcontrollers has a more limited operator list, with optimised kernels only for Cortex-M, Xtensa and Ethos-U. Ethos-U itself is reached through TOSA and the Vela compiler with symmetric INT8 quantisation.

Profiling exposes the fallback. NXP’s benchmark output reports how many nodes each delegate executed, for example 29 of 31 nodes on the Neutron delegate. An industrial edge framework from Advantech (2026) lists positive verification of the acceleration path, including detection of CPU fallback, as a gate before deployment. Versions matter too. NXP recommends matching the TensorFlow Lite converter to the runtime version, since backward compatibility is not guaranteed.

For small UAVs, the field record matters as much as the operator list. We would treat a board with only bench results as unproven for flight, whatever its scores. One UAV integration guide (EMS Drone) recommends a two-hour thermal-chamber run at worst-case ambient with throughput loss held under 10%, and NVIDIA notes that Jetson Orin Super Mode may need carrier-board and thermal redesign.

How mature is the compiler and SDK for our model on each candidate board?

We would judge compiler and SDK maturity, and the quantisation effort needed to hit power and throughput targets, by putting your actual model through each vendor’s toolchain and measuring fps, latency, accuracy and power. LynxBenchAI’s published results show how each covered device runs the public PyTorch stack your own models are built on, so they are broadly indicative for custom PyTorch models. They do not run your model. The 26Q3 release covers GPUs and CPUs, not edge NPUs, so for an edge board the port itself is the measurement. That port is the work TechnoLynx’s AI Porting & Deployment Pack does.

Effort varies a lot by toolchain. In the agent-driven deployment study, Real-ESRGAN reached OpenVINO in about 19 minutes. The RKNN conversion ran for about 90 minutes and was not completed. The same model can fail to compile on one NPU, run unoptimised on a second and approach peak efficiency on a third, depending on how it was quantised, sparsified or pruned.

Quantisation adds hidden effort and accuracy risk. NXP warns that dynamic-range quantisation on GPU or NPU targets keeps activations in FP32 and runs slower than an FP32 model. The 2026 INT8 study found that INT8’s speedup on CPUs can be negative: up to 2.1x faster on cores with dot-product instructions, 1.7x slower on cores without them. It also measured DETR losing 42.9% mAP under INT8 on a discrete GPU. Selective mixed precision avoided accuracy loss in one FP16 study. On an RTX 3090, hybrid FP16 kept SST-2 accuracy exactly at the FP32 baseline, while blanket FP16 lost 0.34 points.

We would shortlist boards in this order:

1. Screen each SDK’s operator support and quantisation options against the model architecture.
2. Drop boards that would need prohibitive model rework.
3. Port and quantise the actual model on the remaining boards, then benchmark fps, latency, accuracy and power under matched batch size, precision and preprocessing.
4. Keep the boards that meet both the power and throughput targets at acceptable engineering effort, and record that effort with the SDK and model versions it applies to.

Who can port our model to each candidate board and tell us what it costs in accuracy?

TechnoLynx ports your model to each shortlisted accelerator and validates it there, so you can see the accuracy, latency and effort on every candidate before committing. The engagement lists your kernels and precision requirements, picks the stacks and hardware to audit, builds and runs your model on each, compares numerical outputs and performance across them, and delivers a maturity and porting-risk report with a recommended shortlist and migration plan. This is the AI Porting & Deployment Pack. The closest published evidence is adjacent work: in our WebSDK case, choosing a framework by real operator support led us to reverse a runtime choice. The system was validated against the 90–95% per-dimension accuracy target at the browser inference stage, and against the ≤200ms end-to-end latency target (a design requirement, not an operational benchmark) on the qualifying device tier. That was browser inference on low-end Android rather than an edge NPU. What carries over is de-risking a model choice on constrained hardware.

The pack does not suit buyers who want a procurement list ranked by TOPS, or the cheapest possible build. If your workload is a standard PyTorch model on a GPU the 26Q3 release covers, the device pages and a LynxBenchAI run may answer the question without an engagement.

Can the board run the whole pipeline offline, including the sensor?

Three separate questions sit inside “can it run offline”. Does the full preprocessing and detection pipeline run standalone with no network call? Does it meet worst-case latency even on a fallback quantised model? Does the board support the sensor’s interface natively? Each has to be checked on the actual hardware with the actual pipeline. LynxBenchAI cannot answer these for you, because it measures the covered stack rather than your pipeline. Porting the full pipeline to the board falls under TechnoLynx’s AI Porting & Deployment Pack.

Model-only numbers understate the problem. The μNPU study notes that most benchmarks time only the forward pass and leave out NPU initialisation, memory I/O and CPU pre- and post-processing, which can significantly affect end-to-end performance. It measured the MAX78000 spending over 90% of end-to-end time on memory operations for one model. In our WebSDK case, the naive frame-capture path on target devices was slower than inference itself, a bottleneck that no amount of model compression would have closed.

Worst-case latency depends on the thermal state and the fallback path. NVIDIA’s TensorRT benchmarking guide warns that two runs of the same engine can disagree by tens of percent if clocks float or thermal throttling starts partway through. A measured edge comparison found the Jetson Nano prone to latency spikes. The Advantech framework recommends a defined lower-throughput fallback mode over silent switching between accelerator and CPU. Sensor paths belong in the inference path too: that framework asks whether every sensor and I/O path has been validated with representative data and timing. We would treat a datasheet’s “native” interface claim as unverified until frames arrive at the expected cadence. A custom capture layer is real engineering effort that a spec-sheet comparison leaves out. Shortlist only boards that pass all three checks.

Unattended field deployments: remote management belongs on the shortlist

Monitoring and updating a board reachable only over 4G

A board on a crane or a remote mast reachable only over intermittent 4G needs a device management platform built for cellular links, plus a local fail-safe that does not depend on the network at all. We would choose a platform with store-and-forward telemetry and resumable or delta OTA updates if drops are frequent, and simpler polling if the link is mostly stable. The Advantech framework lists the OTA and rollback procedure in the evidence packet a field deployment should carry. A UAV integration guide makes the same point about safety: the hardware failsafe line must be independent of the serial link to the flight controller. For a crane, we would agree the safe state (halt, manual control, or a degraded known state) with the site’s safety owner.

Catching a failing unit without a site visit

Over a constrained uplink we would send health indicators and their trend, sampled and compressed on the device, not raw feeds. Alerting should fire on degradation, not on failure. We would build a tailored lightweight pipeline when accelerator health feeds a safety-relevant detection, and choose a managed fleet-monitoring service for large, uniform, low-power fleets. Staff time for triage does not disappear just because diagnostics are remote. Soak-test telemetry under realistic thermal, storage and concurrency conditions belongs in the evidence before go-live.

Comparing AI hardware accelerators fairly with results from a single release

Compare devices only within one release, on the same stack, the same models and the same precisions. Read each category separately, never just the aggregate. LynxBenchAI scores are comparable only within one release, here 26Q3. MLPerf applies the same discipline: the same system and framework must be used for a set of results reported in a single context, and the Closed division requires the reference model.

Worked example: three datacenter GPUs in 26Q3

Device	GT	Training	Inference	Compute
NVIDIA H200	1045.0	812.0	1914.6	480.8
AMD Instinct MI300X	986.7	727.2	1304.0	1023.3
NVIDIA H100 80GB HBM3	939.5	709.3	1678.1	487.1

In 26Q3, the H200 scores highest of the three in training and inference, and the MI300X scores highest in compute. The per-model scores split again. At FP16 inference on MobileNetV3, MI300X scores 733.8 against 2833.0 for the H100. On DistilBERT it scores 2816.1 against the H100’s 2611.7. A shortlist for a transformer-heavy service and one for a vision service could read this table differently, and both readings would be right.

Two cautions apply when reading it. Each LynxBenchAI test is one timed window after a discarded warm-up, not a median of repeats. NVIDIA’s guidance for trtexec, by contrast, compares throughput with median latency and runs a configurable warm-up. We would therefore treat small gaps between close neighbours with care and large gaps as decisive. Also, a 0.0 is a missing precision, not a slow one: the RTX 3080 and A100-SXM4-80GB produce no FP8 inference results in 26Q3. For pilot design and how to keep vendor-run numbers honest, see benchmarks, pilots and fair comparisons for AI compute candidates.

How to check an AI accelerator on your own machine before trusting its spec sheet

Run real software on the device. LynxBenchAI treats empirical execution of real workloads as the reference standard, and spec-based estimation as a fallback for when you cannot run anything. The same message appears in NVIDIA’s Triton documentation, which says the gains from dynamic batching and multiple instances are model-specific and must be found by experiment. It appears again in Google’s LiteRT guidance, which recommends benchmarking a delegate per model because each supports a fixed set of operations.

1. Run LynxBenchAI on the device with stock PyTorch and keep the record of the device, backend, driver, framework and runtime it binds the score to.
2. Compare your result with the 26Q3 device page for the same device and platform, since native Linux and WSL entries differ. The RTX 3080 scores 160.5 GT natively and 158.8 GT under WSL.
3. Run your own model with the vendor’s tool (trtexec, the LiteRT benchmark tool, or onnxruntime_perf_test) and check how many nodes the accelerator actually executed.
4. Record GPU, driver, CUDA version and clock state, since NVIDIA says numbers from an unrecorded environment should not be used as a deployment baseline.
5. Compare outputs against a reference run (cosine similarity and element-wise tolerance), and re-validate any INT8 model on each target.

LynxBenchAI runs stock PyTorch workloads, not your application. Steps 3 to 5 cover those.

A shortlist is a dated claim about executors, so the most useful thing to carry into the contract is a list of which kernels, versions and precisions you verified, and a commitment to re-run them before the hardware arrives.

FAQ

What is the AMD equivalent to NVIDIA CUDA?

ROCm is AMD’s equivalent of the CUDA toolkit. It ships HIP (the CUDA-like runtime) along with rocBLAS, MIOpen, Composable Kernel and hipBLASLt.

Which GPUs does ROCm support for vLLM?

According to GIGAGPU’s setup guide, vLLM runs on AMD GPUs from the MI250 through the MI300X, and on the consumer RX 7900 XTX and RX 9070 XT.

Does quantization affect accuracy?

It depends on how it is done and on the target. Google’s Coral documentation says 8-bit quantisation does not significantly affect accuracy. Results vary by model and target, though: one 2026 study measured DETR losing 42.9% mAP under INT8, so we would check accuracy on every target.

Sources

1. Edge AI Deployment Beyond Models: A BSP-Aware Systems Framework for Industrial Embedded Platforms, arxiv.org
2. AI accelerator performance and benchmarking  |  Generative AI  |  Google Cloud Documentation, docs.cloud.google.com
3. NPU Hardware Evaluation v1.0, arxiv.org
4. From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes, arxiv.org
5. Edge Devices Inference Performance Comparison (Bulletprove), arxiv.org
6. TensorFlow models on the Edge TPU — Coral docs, gweb-coral-full.uc.r.appspot.com
7. Working with DLA — NVIDIA TensorRT, docs.nvidia.com
8. Benchmarking Ultra-Low-Power 𝜇NPUs, arxiv.org
9. Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators, arxiv.org
10. Evaluating Edge AI Accelerator Performance: Why TOPS Are Not Enough — Hailo, hailo.ai
11. MI300X vs H100 vs H200 Benchmark Part 1: Training – CUDA Moat Still Alive (SemiAnalysis), newsletter.semianalysis.com
12. AMD vs NVIDIA Inference Benchmark: Who Wins? (SemiAnalysis), newsletter.semianalysis.com
13. GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference, arxiv.org
14. vLLM on ROCm: Setup Guide for AMD GPUs (MI300X, RX 7900 XTX) GIGAGPU, gigagpu.com
15. Porting CUDA code to HIP — ROCm Documentation, rocm.docs.amd.com
16. Support Matrix — Gaudi Documentation 1.24.0 documentation, docs.habana.ai
17. UG10166 i.MX Machine Learning User's Guide, nxp.com
18. 1Introduction, arxiv.org
19. UAV Edge AI and Onboard Computing: Selecting Companion Computers and AI Accelerators for Autonomous Drone Operations, ems-drone.com
20. Jetson FAQ, developer.nvidia.com
21. Case Study: WebSDK Client-Side ML Inference Optimisation, technolynx.com
22. Performance Benchmarking — NVIDIA TensorRT (trtexec), docs.nvidia.com
23. MLPerf Inference Rules — MLCommons, github.com
24. Benchmark MLPerf Inference: Datacenter — MLCommons, mlcommons.org
25. Optimization — NVIDIA TensorRT Inference Server 1.9.0 documentation, docs.nvidia.com
26. LiteRT Delegates  |  Google AI Edge  |  Google AI for Developers, ai.google.dev