Maximising Efficiency with AI Acceleration

AI acceleration choices between GPUs and NPU, ASIC or FPGA accelerators: why under-use, operator gaps and cost per inference decide the purchase.

Maximising Efficiency with AI Acceleration
Written by TechnoLynx Published on 21 Oct 2024

A GPU is already an AI accelerator, so the real question behind “AI accelerator vs GPU” is how specialised your hardware can safely be for the workload you actually run. Dedicated ASIC NPUs give up flexibility to gain efficiency. They deliver strong performance per watt on the operators they support, and in the white paper each failed to compile several of the twelve test models. Before either purchase, though, check whether the hardware already installed is under-used. That makes profiling your own workload the first step. Low utilisation is a fleet-wide cost problem in its own right, and our piece on the hidden cost of GPU underutilisation shows how that cost builds up across a fleet.

AI Acceleration Is a Trade Between Specialisation and Flexibility

The central distinction in AI hardware is specialisation versus flexibility. Hivenet’s guide to AI accelerators makes the point that the labels overlap: “AI accelerator” does not describe something separate from a GPU, a GPU is not a “training chip” and an NPU is not automatically a “small inference chip”. What separates them is how specialised the hardware is, what work it speeds up, what software it needs, and how much freedom you keep when the workload changes.

A review of FPGA-based accelerators sets out the three-way trade clearly. GPUs offer high throughput and mature programming ecosystems, at a cost in power. ASICs deliver strong energy efficiency through custom dataflows but are expensive to develop, cannot be changed after fabrication and are slow to adapt to new architectures. FPGAs sit between the two, with reconfigurable logic that can be updated as models evolve. The same review groups NPUs and TPUs under ASIC-style accelerators built for specific network requirements. Our own GPU vs TPU vs CPU comparison sets out how those three designs trade flexibility for throughput. The framing that matters here is the one it opens with: the useful question is which design matches this workload’s bottleneck in this software stack at this scale, not which chip is fastest.

What a Dedicated AI Accelerator Card Does That a GPU Does Not

A dedicated ASIC inference card implements a fixed dataflow. The white paper “NPUs for Inference”, which benchmarked ten edge accelerators against an NVIDIA RTX A5000 running TensorRT, describes ASIC NPUs as implementing fixed, highly optimised dataflows and memory hierarchies. That buys exceptional performance per watt, but the supported operations, quantisation formats and network topologies are limited by the fixed architecture. The Hailo-8 is a good example: up to 26 TOPS at a typical 2.5 W, with all of its memory on-chip and no external DRAM. Its pipelined design lets six concurrent jobs overlap, which gives close to six times the throughput at unchanged per-request latency for lightweight convolutional models.

The on-chip memory design also sets a limit. Models too large for the Hailo-8’s on-chip SRAM are split into several execution contexts that run one after another. In the white paper, ResNet-101 managed 112 FPS against 771 FPS for ResNet-50, a drop far larger than the 1.7× difference in parameter count would suggest. A GPU can run almost any parallel workload: training, fine-tuning, custom kernels, new architectures and several models on the same machine. That flexibility costs silicon and energy a narrow inference job may never use. This is why a dedicated card can be the better buy for one fixed model and the wrong one for a changing portfolio.

The Failure Mode: Models That Compile on the GPU and Not on the Accelerator

In the “NPUs for Inference” white paper, the A5000 with TensorRT compiled and ran all twelve reference models. Each of the three ASIC NPUs failed on at least one of the suite’s small vision-transformer models (Tiny-ViT-5M, ViT-Tiny), which the authors read as a fundamental limitation of fixed-function silicon that will matter more as architectures diversify.

Data-centre accelerators show a similar operator-support gap. The 2026 xPU-athalon study found that Groq’s speed advantage on computational primitives applies only to the operators it supports, and that it lacks PyTorch’s scaled dot product attention kernel. The same study reports compilation times up to 5000× longer on novel accelerators than on GPUs. It also reports that deploying a LLaMA model on nine Groq racks took about 1.5 years of coordination and repeated compiler issues, despite advertised LLaMA support. The white paper makes the consequence concrete: adapting the operator mix to the target hardware is the highest-leverage optimisation on dedicated NPUs, and it can decide between a failed compile and a working high-throughput deployment.

In practice, a model change that brings in an unsupported operator or numerical format can force a new compiler path, a model conversion or a fallback to the CPU. Hivenet’s guide puts it more bluntly: an accelerator that does not support your model is not an acceleration strategy.

NPU vs GPU for AI: Edge Inference Versus Data-Centre Serving

At the edge, the benchmarks favour NPUs on power and GPUs on large batches and matrices. On an Intel AI PC system-on-a-chip with integrated CPU, GPU and NPU, the GPU was 22.6% faster on large matrix multiplication, with twice the throughput, and 2.7× faster on LSTM networks. The NPU’s peak power draw was 35 W against the GPU’s 75 W for comparable or better results. For video classification the NPU did better at small batch sizes and the GPU at larger ones. A 2025 arXiv comparison of small language models on edge backends found the RaiderChip NPU more than 50% and 70% faster than an NVIDIA Jetson Orin Nano Super GPU at F16 and Q4K precision respectively, and identified memory bandwidth as the dominant factor in throughput.

Latency complicates that picture. In the “NPUs for Inference” white paper the A5000 reached 0.20 ms single-stream latency on ResNet-18, against 1.20 ms on Hailo-8 and 1.8 ms on Axelera Metis, because a request to a pipelined NPU has to pass through the full pipeline before it produces output. The same paper reports NPUs drawing 35–70% less power than GPUs at comparable throughput. So an edge NPU fits a stable, supported model, a tight power budget and enough concurrent streams to fill its pipeline. In those tests the GPU suited the lowest single-stream latency and models that change.

In the data centre the answer moves with the workload. A 2025 arXiv analysis of LLM inference accelerators found that no single architecture dominates, with performance differences of up to 3.7× depending on batch size and sequence length. At batch size 1, performance correlated strongly with internal memory bandwidth (r = 0.88), and compute mattered more at larger batches. The xPU-athalon study found that Cerebras, SambaNova and Gaudi have 10–60% higher idle power than NVIDIA and AMD GPUs, so those systems have to be kept busy to deliver their promised efficiency.

When a Dedicated Inference Accelerator Beats a GPU, and When It Does Not

The table below sums up the conditions the sources support. No single row settles a purchase; together they show where the burden of proof lies.

Condition Favours a dedicated accelerator (NPU, ASIC, FPGA) Favours a GPU
Model stability Model changes rarely and fits the supported operator set Models change often, or need custom kernels or new architectures
Architecture Models within the device’s supported operator set Small vision-transformer models, at least one of which each ASIC NPU tested failed to compile
Power budget Power-constrained edge deployment Power is secondary to coverage and toolchain maturity
Batch and concurrency Small batches or many concurrent streams on a pipelined NPU Large matrices and large batches
Latency Predictable latency on a supported, stable model Lowest single-stream latency
Volume Stable, high-volume inference where benchmarks show better economics Volume too low to repay migration and operational costs
Software A mature SDK for the device and the model Toolchain maturity and full model coverage matter most
FPGA specifically Workloads that need reconfigurable, application-specific execution Teams that rely on standard libraries, which FPGAs lack in a CUDA-like form

What Does “GPU utilisation” Measure, and Why Is the GPU-Busy Percentage Misleading?

The GPU utilisation metric measures time, not work. NVIDIA’s definition, quoted in a DevOpsBeast guide to DCGM observability, is the percentage of the sample period during which one or more kernels was executing. One tiny kernel running on a single streaming multiprocessor of an H100 for the whole window reads as 100%, while the other 131 SMs sit idle. The guide calls it the most-watched and least-informative GPU metric there is, because it stays near 100% under any steady load.

This is the normal state of LLM inference. Autoregressive decode launches a stream of small kernels, one per token, so GPU_UTIL sits at 100% while SM occupancy is around 15% and the tensor cores are almost idle. The same guide describes a deployment where the dashboard showed full utilisation and a capacity meeting drafted a GPU order, until someone measured throughput and found the H100 serving 40% of the tokens per second it should. The team did not have a capacity problem. They had a measurement problem.

A Triage Checklist for GPU-Busy Readings

The DevOpsBeast guide’s 30-second rule compares tensor-pipe activity with DRAM activity:

  • Tensor high, DRAM moderate: compute-bound, and the GPU is being used well.
  • DRAM high, tensor low: memory-bound. Reduce bytes moved (batching, quantisation, KV-cache changes) before buying hardware.
  • Both low while GPU_UTIL reads 100%: a stall from small kernels, launch overhead, a host-side bottleneck or a starved dataloader.
  • PIPE_TENSOR_ACTIVE missing: the profiling counters were never enabled. They are not in dcgm-exporter’s default set on every install and have to be added to the counters CSV.
  • Throughput drops with no application error: check throttle reasons, because thermal and power throttling can take 20–40% of throughput with no application-level signal.

For a tensor-active reading, the guide treats 0.6 on an H100 running matmul-heavy work as healthy and 0.05 as paying for hardware that sits in traffic. How these signals fit with logs and traces is covered in the three pillars of observability applied to GPU utilisation.

Where Does the Missing Throughput Go When the GPU Reads Busy?

Memory-bound decode. Prefill is compute-bound, while decode reads the full model weights and the growing KV cache for every token. More compute does nothing in decode, and only batching or moving fewer bytes helps. The xPU-athalon study draws the same prefill/decode split across accelerators. How this plays out on a large reasoning model is shown in our analysis of decode cost in practice.

Unbatched requests. The DevOpsBeast guide describes a 24-GPU H100 inference fleet where a misconfigured client sent one request at a time with no batching window. Tensor-active averaged 0.06, while DRAM-active averaged 0.74.

Small kernels and host-side stalls. In the guide’s triage, launch overhead, host-side bottlenecks and dataloader CPU starvation are stalls that show while GPU_UTIL reads 100%.

Default configuration. Our own sizing estimate is that default NVIDIA GPU configurations typically give up 20–40% of available AI throughput, which stays hidden until delivered FLOPs are benchmarked against rated capacity. That figure is an estimate, not a measurement.

A model shaped for different hardware. Google Cloud’s accelerator benchmarking documentation gives the example of gpt-oss-120B, whose attention head dimension of 64 wastes cycles on TPUs built around 256-wide matrix units. The low score reflects a software-architecture mismatch, not poor hardware.

At fleet scale, Cast AI’s 2026 State of Kubernetes Optimization Report found GPU utilisation averaging 5% across the clusters it analysed. That figure covers the clusters analysed, not your environment.

Should I Buy More GPU Capacity or Profile What I Have First?

Profile first. A memory-bound workload scales its bottleneck linearly when you add GPUs, and most of each new GPU is wasted. The right fix is on the memory side: bigger batches to spread the cost of weight reads, quantisation to shrink the bytes moved, or KV-cache changes. With the right four profiling metrics, the DevOpsBeast guide argues, you know whether to batch or to buy.

Best practices only make sense in context, and that context comes from profiling, not from reading blog posts. If a profile does show the hardware is the limit, a move to other silicon is a porting job, which porting AI inference across runtimes and hardware covers.

Pricing Idle Capacity Before Comparing Accelerators

The true cost is the money spent on capacity divided by the work that capacity actually delivers, not by the capacity rated on the spec sheet. GPU utilisation cannot supply that denominator, because it reads near 100% whether or not useful work is happening. Delivered throughput, together with tensor-active and DRAM-active, is the better starting point. The full effective-cost formula, with the fleet-level figures, is set out in our piece on the hidden cost of GPU underutilisation.

How Do I Compute TCO per Useful FLOP Rather than per Purchased FLOP?

Put sustained, useful output in the denominator. Google Cloud’s documentation recommends dividing the amortised capital and energy cost of one chip by its maximum sustained tokens per second per chip, which turns a benchmark into a cost per thousand or million tokens. For training it adds goodput, which counts only computation that advances the model and excludes time lost to faults, network stalls and checkpoint recovery. The same idea applies to inference: throughput that breaks the latency SLA, or that sits behind a stall, is not useful throughput. Comparing that metric across cloud providers is covered in AWS vs Azure on cost per useful FLOP.

Worked Example: Performance per Dollar Reverses a Raw Ranking

Google Cloud’s documentation works through three chips. Chip_A delivers on average 0.79 of Chip_B’s performance and 0.60 of Chip_C’s, so on raw speed Chip_C wins. With prices of $100, $180 and $200 respectively, the order changes. Normalising each chip to its price (taking B and C as 1.0 in their own comparison):

  • A against B: 0.79 / $100 = 0.0079 per dollar, against 1.0 / $180 ≈ 0.0056 per dollar, so A gives about 1.42× the performance per dollar.
  • A against C: 0.60 / $100 = 0.0060 per dollar, against 1.0 / $200 = 0.0050 per dollar, so A gives about 1.20×.

The slowest chip turns out to be the best value: on Google’s averages, 42% ahead of Chip_B and 20% ahead of Chip_C. The documentation also recommends counting performance per watt as part of the cost.

What Savings Are Realistic from Tuning Utilisation Versus Renting More Cloud GPUs?

The saving depends on how far the current deployment sits from what its hardware can deliver, and profiling is what shows that gap. In the DevOpsBeast example, fixing batching raised tensor-active from 0.06 to 0.55, nearly tripled throughput on the same hardware, and shelved a drafted $1.2M expansion. That is one fleet with one misconfiguration, so treat it as evidence of what can happen, not as a forecast. Adding cloud GPUs to a memory-bound workload does not remove its bottleneck. It copies the bottleneck onto more instances. Where capacity really is short, the rent-or-own question is the next one, and cloud GPU vs on-premise AI accelerators works through it.

Comparing an AI Accelerator with a GPU on Cost per Inference Before You Buy

Peak TOPS or FLOPS alone is the wrong unit for comparing AI inference infrastructure. Hivenet’s guide notes that two accelerators can quote similar peak arithmetic and deliver completely different application performance, because precision, memory bandwidth, batch size, operator support, compiler quality and utilisation all come between the spec and the result. The procedure below follows Google Cloud’s inference benchmarking method:

  1. Fix the model you will actually serve, because models co-designed for one platform may not use another platform fully.
  2. Prepare the model for each target: quantisation, sparsity and pruning interact differently with each platform, so model optimisation has to be evaluated together with the hardware.
  3. Set a strict latency SLA, for example a P99 of 100 ms with limits on time to first token and time per output token.
  4. Increase concurrency gradually until the hardware breaks the SLA, and record the sustained throughput per chip at that point.
  5. Divide amortised capital and energy cost by that throughput to get cost per thousand or million tokens.
  6. Add migration and operational cost for the specialised option, which can outweigh a theoretical efficiency gain if models change often.
  7. Repeat across the batch sizes and sequence lengths you expect, because the best platform moves with batch size, sequence length and model size.

Two caveats belong next to any result this procedure produces. The 2025 arXiv analysis saw up to 40% variation on the same hardware across software stack versions. The xPU-athalon study warns that metrics such as compute and memory-bandwidth utilisation may be defined differently on each platform. Published per-device results to check candidate hardware against are in the LynxBenchAI AI hardware directory.

Who Can Profile Our Inference Stack Before We Commit to New Accelerators?

TechnoLynx’s GPU Performance Engineering practice profiles and tunes the production AI serving path to find cost and latency gains before anyone recommends a new model or new serving hardware. The work starts with profiling your deployed workload, because that context decides which optimisations are worth making. It continues with optimising the levers the profile points to.

Some teams do not need this. A team that already runs DCGM profiling metrics and Nsight in production can often work through the triage above on its own.

The Purchase You Are Really Making

Choosing between a GPU and a dedicated accelerator also means choosing a compiler, a runtime, model support and a migration path. Hivenet’s guide makes the point that the software environment becomes part of the product. For that reason we would treat a cost-per-inference win on a vendor benchmark as provisional until your own model compiles, meets its latency target and stays within the supported operator set through the next model change. The comparison that matters puts your best-tuned current deployment against the specialised option, both measured under the same SLA.