A GPU-hour is cheaper on one cloud. That fact, on its own, tells you almost nothing about whether your inference workload will cost less per request after you migrate to it. Yet the most common way teams pick a cloud for AI inference is to open a pricing page for AWS, Azure, and Google Cloud, line up the on-demand hourly rates for comparable GPU instances, and migrate toward the cheapest row. That comparison is answering the wrong question. The provider is one variable among several, and it is rarely the one that sets your true cost-per-request. Instance type, GPU generation, batching behaviour, and quantisation overhead — all sitting inside your serving boundary — move cost-per-request far more than the headline hourly rate. A cross-cloud price table tells you a GPU-hour is cheaper on provider X. It does not tell you whether your workload will reach the utilisation that makes that hour cheaper per request. Why the cheapest GPU-hour is not the cheapest inference Cost-per-request is a ratio: what you pay for the instance, divided by how many requests that instance actually serves in the time you rent it. The hourly rate is the numerator. Utilisation is the denominator. Most price comparisons obsess over the numerator and ignore the denominator entirely. Here is the mechanism that catches teams out. Suppose your current deployment runs at 30% GPU utilisation because your serving stack batches poorly and your model spends most of its time memory-bandwidth-bound rather than compute-bound. Migrating that same workload to a cloud with a 15% cheaper GPU-hour lowers the numerator by 15% — and changes the denominator not at all. Your cost-per-request drops by 15% at best, and often by less once you account for egress, storage, and the operational cost of the move. The real lever — the 3x headroom sitting idle in that 30% utilisation figure — travels with the workload to the new cloud untouched. This is the divergence point the sticker-price table hides. The bottleneck was inside the serving path, not on the invoice. Choosing a cloud on hourly rate alone is how teams migrate a workload and find cost-per-request essentially unchanged, having paid a migration bill for a saving that the current provider could have delivered with a batching or quantisation fix. What actually differs between the three clouds for inference The clouds do differ — just not primarily on the axis the pricing pages emphasise. The differences that matter for inference sit in the GPU generations on offer, the managed serving stacks, and the network topology inside an instance. AWS exposes NVIDIA GPUs through the p and g instance families (P5 with H100, G6 with L4, and so on) and pushes Inferentia via its own inf2 accelerators and the Neuron SDK. Azure offers the NC and ND families with comparable NVIDIA silicon and tight integration into Azure Machine Learning’s managed endpoints. Google Cloud pairs NVIDIA GPUs on its A3/G2 machine types with its own TPU line, which requires JAX or a supported TensorFlow/PyTorch path rather than CUDA. That last point matters: a workload written against CUDA, cuDNN, and TensorRT ports cleanly across the NVIDIA offerings on all three clouds but does not run on a TPU without a compilation and numerics change. Portability is a real variable, and it is not on the price table. The managed serving layers diverge too. Whether you run Triton Inference Server yourself, lean on SageMaker’s managed inference, use Azure ML endpoints, or deploy to Vertex AI changes how much control you have over batching, concurrency, and instance packing — the exact levers that set the denominator in your cost-per-request ratio. Two clouds can rent you the same H100 at the same price and still produce different cost-per-request because their default serving stacks batch differently under your traffic shape. AWS vs Azure vs Google Cloud for AI inference: what to compare Compare on the variables that move cost-per-request, in this order. The hourly rate is deliberately last, because it is the variable that varies least between plausible candidates. Variable Why it moves cost-per-request How to compare across clouds GPU generation & memory Newer silicon + more HBM raises the batch size and utilisation ceiling Match on GPU model (H100 vs H100), not instance family name Serving-stack batching Sets the denominator: requests served per GPU-hour Profile the same model + traffic on each candidate’s serving path Quantisation support FP8/INT8 kernels shrink memory pressure and raise throughput Confirm the target GPU + runtime supports your precision Portability (CUDA vs TPU) Determines whether migration needs a code change at all CUDA workloads: NVIDIA-only; TPU needs a JAX/XLA port Egress & storage Recurring cost that a per-GPU-hour table omits Model your actual data movement, not the compute alone On-demand hourly rate The numerator only Compare last, after the above are held constant The point of holding the top rows constant is that only then does the hourly rate become a fair comparison. Compared in isolation, it is a number attached to instances that will serve wildly different request volumes. How do the serving stacks affect batching and utilisation? Batching is where most of the cost hides, and it behaves differently depending on how each cloud’s serving layer is configured. Dynamic batching — grouping incoming requests into a single GPU forward pass — is the single biggest lever on utilisation for most transformer inference workloads. A serving stack that batches aggressively can lift a memory-bandwidth-bound model from 30% to 70% utilisation without touching the hardware, and that is a larger cost-per-request improvement than any realistic cross-cloud price gap. The catch is that the batching behaviour you get depends on the runtime, the model, and the traffic shape, not on the cloud logo. Triton Inference Server behaves the same way on AWS, Azure, or GCP if you configure it identically; the managed endpoints (SageMaker, Azure ML, Vertex AI) each apply their own defaults that you may or may not be able to tune. This is why a like-for-like comparison has to profile the actual serving path — the batching, the p95 latency under realistic concurrency, the utilisation — rather than trust a spec sheet. We treat the serving path as the primary object of study; if you want the mental model, our note on mapping the serving path in a machine learning architecture diagram walks through where the cost actually accumulates between request and response. Quantisation compounds this. Moving a model to FP8 or INT8 reduces memory traffic and can raise the batch size the GPU sustains, but the overhead and accuracy trade-off vary by runtime and GPU generation. A cloud whose available GPUs have first-class FP8 kernel support can beat a nominally cheaper cloud whose silicon forces you back to FP16. Whether that helps your specific workload is a question profiling answers and a price table cannot. When is a migration the right lever versus a serving-path fix? This is the decision the whole comparison exists to inform. A cross-cloud migration is worth it when profiling shows the saving genuinely lives on the other cloud — a materially newer GPU generation you cannot get on your current provider, a managed serving stack that fits your traffic better, or a portability advantage (a clean CUDA target, or a TPU path that suits a JAX workload). A serving-path fix is the right lever when profiling shows your current cloud has idle headroom: low utilisation, poor batching, a precision you have not exploited. Diagnostic: migrate, or fix in place? Run this before committing to a move. It is a triage rubric, not a benchmark. Measure current GPU utilisation under production traffic. Below ~50% (observed pattern across inference audits, not a benchmarked threshold) means significant headroom exists on the current cloud — a serving-path fix likely wins. Check dynamic batching is enabled and tuned. If it is off or using defaults, that is a fix-in-place saving, not a migration saving. Confirm your precision. Still on FP16 when the GPU supports FP8/INT8? Quantisation is a current-cloud lever. Compare the GPU generation, not the price. Only a genuinely newer generation on another cloud justifies the move on hardware grounds. Model egress and re-platforming cost. A 15% hourly saving evaporates fast against data-transfer and engineering time. Then, and only then, compare hourly rates across the shortlisted candidates on matched hardware. If steps 1–3 surface headroom, the migration’s projected savings were probably going to travel with the workload anyway. The avoided migration — keeping the workload where it is and fixing the serving path — is often the larger, faster, and lower-risk win. The kernel- and batching-level detail behind these levers is genuinely its own discipline; GPU profiling resolves whether a cheaper instance actually lowers cost-per-request by showing whether your model is compute-bound or bandwidth-bound in the first place. And the unit-economics framing — treating cost-per-request as the KPI rather than hourly rate — is what makes any cross-cloud comparison meaningful; our broader engineering and R&D services page is where that framing sits in the larger picture. This article is a comparison of clouds; it is deliberately not a general treatment of inference reliability across providers, which our comparison of AWS vs Azure for running production AI reliably covers from the SLO and operational angle instead. A worked example: matching the numbers Assume, for illustration, a transformer inference workload serving 10 million requests a month. On the current cloud it runs on an H100 instance at $4.00/GPU-hour (illustrative figure), at 30% GPU utilisation. Migrate on price alone: move to a cloud at $3.40/GPU-hour, same 30% utilisation. Cost-per-request falls 15%. Migration and egress cost is real and recurring. Fix the serving path in place: enable dynamic batching and FP8, lift utilisation from 30% to ~65% (illustrative). The instance now serves roughly twice the requests per hour at the same $4.00 rate. Cost-per-request falls by nearly half — on the current cloud, with no migration bill. The arithmetic is illustrative, but the shape is the recurring pattern: the denominator moves more than the numerator. This is why a cross-cloud comparison that stops at hourly rate is answering a smaller question than the one the buyer actually has. FAQ What does working with aws vs azure vs google cloud involve in practice? All three clouds rent GPU instances by the hour and layer managed serving stacks (SageMaker, Azure ML, Vertex AI) on top. In practice, the meaningful differences for inference are the GPU generations on offer, the batching behaviour of each serving stack, and portability — CUDA workloads run across all three NVIDIA offerings but a Google Cloud TPU needs a JAX/XLA path. Comparing them well means comparing on cost-per-request under your own traffic, not on the published GPU-hour rate. What actually differs between AWS, Azure, and Google Cloud for AI inference beyond the headline GPU-hour price? The GPU generations available, the managed serving layers and how much batching and packing control they give you, precision support (FP8/INT8 kernels), and portability. Recurring egress and storage costs also differ and never appear on a per-GPU-hour table. These variables move cost-per-request far more than the hourly rate does between plausible candidates. Why can a cheaper cloud instance leave my cost-per-request unchanged after migration? Cost-per-request is the instance cost divided by requests served per hour. A cheaper GPU-hour lowers the numerator but does nothing to the denominator — utilisation and batching. If your real bottleneck is a serving path running at low utilisation, that inefficiency travels with the workload to the new cloud, so the saving is limited to the hourly-rate gap and often less after migration overhead. How do the GPU instance types and serving stacks on each cloud affect batching, utilisation, and cost-per-request? Newer GPU generations with more HBM raise the batch size and utilisation ceiling, and each cloud’s serving stack applies different batching defaults. Dynamic batching can lift a memory-bandwidth-bound model from around 30% to 70% utilisation without changing hardware — a larger cost-per-request gain than most cross-cloud price gaps. The stack, model, and traffic shape drive this, not the cloud logo, so a like-for-like comparison has to profile the actual serving path. When is a cross-cloud migration the right lever versus a serving-path fix on my current provider? Migrate when profiling shows the saving genuinely lives elsewhere: a materially newer GPU generation you cannot get today, a managed stack that fits your traffic better, or a portability advantage. Fix in place when profiling shows idle headroom — low utilisation, untuned batching, or an unexploited precision. If utilisation is low and batching is on defaults, the projected migration saving was probably going to travel with the workload anyway. How does an inference cost audit help me compare clouds on cost-per-request instead of hourly list price? An audit profiles inside the serving boundary — utilisation, p95 latency, batching, and precision — on each candidate cloud, so you compare cost-per-request rather than sticker price. It surfaces whether a projected cross-cloud saving is real or an artefact of an unfixed serving-path bottleneck. That is the difference between a migration that pays for itself and one whose savings evaporate on arrival. Where the real risk actually sits The cross-cloud comparison is worth doing — but the axis that matters is cost-per-request measured inside the serving boundary, not the hourly rate on the pricing page. Before you commit engineering time and egress bills to a migration, the question to answer is narrower and more useful than “which cloud is cheapest”: does the saving actually live on the other cloud, or is it idle headroom on the one I already pay for? The Inference Cost-Cut Pack exists to answer exactly that — profiling the serving path per candidate so the migration decision rests on measured cost-per-request rather than a sticker-price table. The failure class it guards against is the migration whose projected savings evaporate because the real bottleneck was never on the invoice.
