Neither AMD nor NVIDIA is cheaper per token across the board for AI inference. In SemiAnalysis’s 2025 inference benchmark, teams that own and run their own GPUs found NVIDIA ahead on performance per dollar for some workloads and AMD ahead for others. Teams renting for less than six months found NVIDIA ahead every time. The result turns on batch size, sequence length, model size, the workload and the latency you promise users. A spec-sheet peak is not measured performance, so list price divided by peak FLOPS settles almost nothing when choosing inference infrastructure.
AMD vs NVIDIA for inference: which workloads each one wins
The fundamental trade-off is interactivity, meaning how many tokens per second each user gets, set against how many tokens per second each GPU produces across all users. A faster response for each user means smaller batches. Fewer tokens then share the fixed hourly cost of the hardware, so the same GPU costs more per token as you push it towards low latency. That is why a single “winner” figure is misleading: the published results cross over as the latency target moves.
| Workload and constraint | Leans towards | What the measurements show |
|---|---|---|
| Large dense model (Llama3 405B), chat-shaped 1k in / 1k out, owned hardware | AMD MI325X, until latency is relaxed past about 60 s | MI325X served at consistently lower cost than H200 and H100 on vLLM, MI300X matched H200 on vLLM, and H200 on TensorRT-LLM won again past the 60-second latency mark (SemiAnalysis, 2025) |
| Reasoning-shaped 1k in / 4k out, owned hardware | Split at about 100 s end-to-end latency | Above 100 s the MI325X showed up to 20% better performance per dollar than H200; below it the H200 won clearly |
| 70B dense model at low latency | NVIDIA H100 / H200 | Both NVIDIA parts on vLLM beat both AMD GPUs at low latency; the MI325X only pulled ahead at higher concurrency |
| GPT-OSS 120B FP4 summarisation, owned hardware | MI355X below 225 tok/s/user, B200 above | MI355X on vLLM had lower TCO per million tokens than B200 on vLLM, and beat B200 on TensorRT-LLM below 225 tok/s/user (SemiAnalysis InferenceMAX v1) |
| DeepSeek R1 FP8 1k in / 1k out, single-node serving | AMD MI355X on performance per TCO | MI355X on SGLang beat B200 on TensorRT-LLM and SGLang in most cases on performance per TCO (SemiAnalysis InferenceX v2, 2026) |
| FP4 with disaggregated prefill and wide expert parallelism combined | NVIDIA | AMD was competitive with a subset of these optimisations enabled, not with all three together |
| Short-term rental, under six months | NVIDIA | NVIDIA won performance per dollar for sub-six-month Neocloud rentals regardless of latency target |
The serving engine can change the winner. In InferenceMAX v1, for owners working on a four-year economic life, the MI325X on vLLM beat the H200 on vLLM on cost per million tokens at every interactivity level. Once NVIDIA’s TensorRT-LLM came in, the H200 won. In that AMD versus NVIDIA comparison the engine decided the winner, so check which engines a comparison uses. Generation matters as well: SemiAnalysis found the older MI300X not competitive with the H200 in most of its 2025 scenarios, on either absolute performance or performance per dollar.
Where NVIDIA still earns a higher hardware price
Tight tail-latency targets are one case. A 2026 ClusterBid ecosystem review argues that the CUDA Graph and TensorRT-LLM path behind consistently low tail latency has no ROCm equivalent, and recommends CUDA on H200 or B200 for SLAs below 50 ms p99 time to first token. Range is another: InferenceX v2 measured the H200 serving from 30 to 90 tok/s/user, against only 13 to 35 tok/s/user for the MI325X, which limits a provider who needs to offer several speed tiers. Distributed serving features count too. SemiAnalysis reported AMD lacking disaggregated prefill, smart routing and NVMe KV-cache tiering, and concluded that without disaggregated prefill NVIDIA wins on raw performance and performance per dollar. In the 2026 data the gap had narrowed for FP8 disaggregated serving but not for the full combination frontier labs actually run.
When does AMD’s larger HBM change the cost of serving a large model?
Memory capacity matters because it sets how many chips a model needs, and peak compute says nothing about it or about memory bandwidth, which bounds decode speed. NVIDIA specifies the H200 at 141GB of HBM3e and 4.8TB/s, AMD specifies the MI325X at 256GB and 6TB/s, and in the next generation the MI355X carries 288GB against the B200’s 192GB.
You see the effect in parallelism. In SemiAnalysis’s tests, on Llama3 405B the MI300X could run at tensor-parallel degree 4 or 8, while the H100 typically needed 8. Fewer GPUs per replica means more replicas per node and less communication per token. SemiAnalysis concluded that serving large dense models is AMD GPUs’ strong suit, with the MI325X ahead across all latency scenarios on that test. ClusterBid makes the same argument at the boundary: where a model barely fits in the H200’s 141GB, the MI300X’s 192GB removes CPU offloading altogether.
Two caveats keep this from becoming a rule. The first is that the inference software must exploit the capacity. In dstack’s DeepSeek R1 benchmark on 8x MI300X against 8x H200, the authors note that the MI300X’s capacity and bandwidth should allow higher throughput at large batch sizes, but the inference backends still needed optimisation to deliver that. The advantage became visible only at around 128 concurrent requests. The second is precision. With input fixed at 512 tokens, Ambati and Diep (2025) found the MI300X closing the gap as output length grew, from 49% to 66% of H100 throughput in FP8 and from about 56% to 80% in FP16. When KV cache spills to host memory, the picture reverses: SemiAnalysis expects NVIDIA’s Grace-based parts, with 900GB/s CPU-GPU bandwidth, to do better there, and calls AMD’s CPU-offload software poor.
Our reading is that the capacity argument is strongest for large dense or MoE models that would otherwise span more GPUs or offload, served at high concurrency. It is weakest for a model that already fits comfortably and is served at tight latency.
How far behind is ROCm for production serving, and what does catching up cost?
ROCm can serve production inference, but closing the gap to CUDA still costs engineering weeks and some throughput, and how much depends on how deeply your stack relies on NVIDIA-only tooling. ClusterBid’s 2026 estimate puts a vanilla PyTorch stack with no custom CUDA extensions at about 2 to 4 engineering-weeks to port and validate. A stack built on TensorRT-LLM with custom plugins, CUDA Graph capture and NCCL tuning needs 12 to 20 weeks to reach equivalent performance on ROCm, if it can be done at all. The same review expects a 10% to 30% throughput loss on most workloads after porting, which then takes more tuning to win back. These are one industry review’s estimates rather than controlled measurements, so we would treat them as a planning range rather than a quote.
TechnoLynx’s view is that vendor-specific APIs such as CUDA and AMD’s HIP are not faster by magic: the same code performs essentially the same on either side. What a vendor API offers is access to lower-level features that permit deeper optimisation, and using that access takes real commitment that is not always worth it. A team committed to TensorRT-LLM with custom plugins faces the longest port. In our reading, a team on vLLM or SGLang, both of which run on either vendor, loses the least. Our own CUDA vs ROCm comparison for modern AI puts it bluntly: choosing between the two is a hardware decision dressed up as an API decision.
Accuracy is a gap to test as well. In SemiAnalysis’s 2025 review, AMD had run no nightly accuracy tests until three weeks before publication, and 25% of the models tested failed accuracy checks on AMD. That may since have improved, but a cost-per-token comparison that never compares output quality on the same model is incomplete.
The gap between spec sheet and delivered performance belongs to the same story. Ambati and Diep found the MI300X sustaining on average 45% of its theoretical peak FLOPs, against up to 93% on H100 and B200. They attribute the gap to software stack efficiency and to frequency scaling under load. Despite 1.5x the H100’s theoretical compute, the MI300X reached only 37% to 66% of H100/H200 performance in their end-to-end LLM inference tests. Why headline numbers mislead in general is covered in our piece on what GPU benchmark scores mean for AI workloads.
The gap is also moving quickly. InferenceX v2 recorded AMD’s SGLang performance on DeepSeek R1 FP4 improving by up to 2x between December 2025 and January 2026. The same report found some fork-image performance work missing from the official vLLM ROCm image for the MI325X, and the MI355X still running a forked vLLM build. Whatever ROCm result you rely on, record the exact image it came from.
Comparing cost per million tokens on your own workload, not a vendor’s
Measure one number: TCO per million tokens at the interactivity level your application actually needs, on the serving engine you would actually run. SemiAnalysis calls TCO per million tokens the metric customers care about, with throughput only a step towards it. The steps below run in order, because each one depends on the one before.
- Write down the workload, not the chip. Record input and output lengths, demand by hour of day and the latency target, because utilisation follows from these.
- Fix a realistic interactivity level. A 4x throughput lead at 5 tok/s/user means little for a chatbot nobody would use at that speed. For GPT-OSS 120B, SemiAnalysis puts the practical range at roughly 150 to 200 tok/s/user.
- Sweep concurrency and parallelism on each candidate. Plot throughput against per-user latency while increasing concurrent users, and repeat for each tensor-parallel setting the GPU allows. The points where nothing improves on both axes at once form the Pareto frontier.
- Give each vendor its best engine. Benchmark vLLM or SGLang on both, and add TensorRT-LLM on NVIDIA, as SemiAnalysis did for the H200. Serving flags multiply into more combinations than anyone can sweep exhaustively, so even published results relied on configurations supplied by NVIDIA and AMD engineers.
- Pick the highest-throughput point inside your latency limit. NVIDIA’s sizing method rules out every point beyond the latency constraint and takes the best throughput among those left. The number of instances is then peak requests per second divided by requests per second per instance.
- Normalise by TCO per GPU-hour. If one GPU delivers 1.5x the throughput of another but costs 2x as much per hour, the slower GPU is the cheaper choice.
- Correct for the load you will actually see. Price each candidate at your expected request rate, which dominates real cost, as the next section shows.
- Check output quality on both. Run your accuracy evaluation on each stack before trusting the cheaper number.
Public benchmarks are a baseline for this exercise, not a substitute for it. InferenceX feeds random single-turn tokens and disables prefix caching, which its authors say gives a good baseline Pareto frontier but not a realistic production workload. NVIDIA’s GenAI-Perf works against any OpenAI-compatible endpoint, including vLLM and SGLang, so the same client can drive both vendors’ servers. For a desk-side version of the same discipline, see our guide to measuring local LLM throughput. It measures the hardware with stock PyTorch rather than your serving stack, so it informs which devices to sweep.
How do I model GPU total cost of ownership without guessing at utilisation?
Model the cost per useful hour, not the cost per hour. An owned accelerator costs the same whether it is busy or idle, so its cost per useful hour is F / u. Here F is the fixed cost per accelerator-hour (capital, maintenance, energy, facility and staff) and u is the fraction of all hours spent on useful work. Renting on demand costs the hourly price p. Owning wins when F / u is less than p, so the break-even utilisation is u* = F / p.
Worked example. Assume an owned H100 with an illustrative all-in capex of $30,000. Fully busy, it costs about $1.72 per GPU-hour. Against AWS’s $6.88 on-demand price, the break-even is 1.72 / 6.88, or about 25% average utilisation. Between 30% and 90% utilisation, the cost per useful hour moves by a factor of three. In the same model, capital recovery and maintenance make up 72% of the hourly cost, colocation about 18% and electricity about 6%. SemiAnalysis’s InferenceMAX analysis points the same way, putting colocation and electricity below 20% of TCO. A 20% gap in tokens per MW between GPUs therefore moves TCO by less than 4%. In the xPU-athalon study, the A100, H100 and MI300 all idled at about 20% of TDP, so idle draw did not separate them.
The trap is treating u as something you enter. A 2026 concurrency-aware cost study argues that utilisation is an output of request rate, model architecture, quantisation and latency SLO, and that every one of the 15+ public calculators it surveyed treats utilisation as an input. On identical H100 hardware, effective cost ranged from $0.21 to $15.25 per million output tokens across its configurations and load range. A per-token calculator can therefore be wrong by more than an order of magnitude, and always in the direction that makes self-hosting look cheaper.
Three corrections make the model honest. Measure u over the hardware’s whole life, including ramp-up and idle weeks. Treat busy time with caution, because a GPU serving single requests is “busy” while producing far fewer tokens than one serving large batches. Price the SLA: tightening the latency target forces a lower request rate and a higher cost per token. The study’s own tool, vllm-cost-meter, computes load-adjusted dollars per million output tokens against a live vLLM server.
Which workload patterns favour cloud GPU rental versus owning hardware?
Daily-cycling inference favours a mix of owned and rented capacity, busy queued training favours owning, and short-term rental favours NVIDIA. A training cluster with a queue of runs stays above 80% busy, while an inference fleet follows daily demand, so the cheapest design often owns the base load and rents the peak. In Safeguard’s illustrative model, with owned capacity costing half the on-demand rate, owning enough for average demand and renting the rest cost 34% less than renting everything and 12% less than owning for the peak. Against a multi-year reservation you pay for idle hours either way, so utilisation drops out and the question becomes whether you can run the hardware more cheaply than the provider. Newer generations also undercut what you bought: on Oracle’s price list, GB200 capacity rents for 37% less per peak dense BF16 PFLOP-hour than H100.
Market structure tilts rental towards NVIDIA. SemiAnalysis counted over 100 Neocloud providers renting NVIDIA GPUs on short terms, against a handful for AMD, and found the resulting scarcity pushing AMD rental prices up enough that NVIDIA won performance per dollar in the rental market at every latency target. By its 2025 figures, for chat-shaped workloads the MI300X would need to rent at $1.9/hr, and the MI325X below $2.5/hr on a one-month term, to compete with the H200. The AMD cost case is an owner’s case. The wider ownership economics, beyond the AMD and NVIDIA question, are covered in our analysis of renting vs owning GPUs for AI.
Tune the serving path before pricing a vendor switch
Production cost and latency problems rarely need a new model, and the same reasoning applies to new hardware: batching, caching, quantisation, kernel work, serving configuration and runtime choice sit below the model and can each be tuned on their own. NVIDIA reports H100 throughput on Llama 2 70B rising by up to 1.5x in a year from software alone, through GEMM and attention kernels, kernel fusion and chunked prefill. In InferenceX v2, enabling multi-token prediction cut DeepSeek-R1 FP4 on Dynamo TRT from $0.251 to $0.057 per million total tokens. FP8 cut saturation cost by 23% to 42% on H100 NVL in the concurrency study, but made Llama more expensive on an A100, which has no native FP8.
The same levers exist on AMD. AMD’s ROCm documentation lists FP8 and FP4 quantisation (cutting memory use by 2x to 4x), attention-backend choice, parallelism strategy and engine arguments such as batch size and graph modes. It cautions that TunableOp GEMM tuning is not guaranteed to beat the default hipBLASLt algorithm, and that the MI300X and MI350X use different FP8 variants, so quantised models must target the right one. A comparison that pits a tuned NVIDIA stack against a default AMD one, or the reverse, measures tuning effort rather than hardware.
TechnoLynx’s GPU performance engineering work starts at this point: profiling the deployed serving path to find the cost and latency gains available before anyone recommends replacing a model or the hardware under it. If the numbers still favour moving after that, the next step is porting AI inference across runtimes and hardware, starting with a feasibility assessment before any workload moves.
An AMD-or-NVIDIA result has a shelf life of months
Any cost-per-token verdict you reach reflects only the software as it stood when tested. SemiAnalysis notes that inference software improves continuously, with releases sometimes only days apart, on top of the yearly hardware jumps. dstack says plainly that its results reflect only the backends as they stood on the day of testing. What we would keep is the method, not the verdict: your interactivity target, your measured request rate, your engine on each vendor and your accuracy check. With those in hand, rerun the comparison after each major vLLM, SGLang or ROCm release. Without them, every new benchmark headline restarts the argument.
