“AI in cloud” is not one thing. It is at least three: consuming a vendor’s managed AI API, training your own models on rented accelerators, and serving inference from cloud infrastructure you operate yourself. Those three share a billing portal and almost nothing else. Each has a different cost curve, a different data-residency question, and a different failure mode once it is carrying production traffic.
That matters because of where the conversation usually starts. Someone asks “which cloud should we use for AI?” — and the more useful question, the one that has to come first, is “which part of our AI workload belongs in a cloud at all?” Answer the second question and the first one often answers itself, or turns out to have three different answers for three different components.
What is AI in cloud computing, in practice?
Strip away the marketing layer and the term collapses into three patterns.
Pattern 1 — the managed AI service. You call a vendor’s endpoint. Someone else owns the model weights, the accelerators, the scaling behaviour, and the upgrade schedule. You own the prompt or the request payload and the interpretation of the response. This is what most people mean when they say “AI cloud services”, and it is the pattern with the fewest engineering decisions and the least control.
Pattern 2 — rented training capacity. You bring your own model, your own data, and your own training code, and you rent accelerators by the hour. The cloud is a hardware lease with a scheduler attached. PyTorch, CUDA, and a container image are yours to manage; the machine underneath is not.
Pattern 3 — self-operated inference on cloud infrastructure. Your model, your serving stack — often something like ONNX Runtime or TensorRT behind a container orchestrated by Kubernetes — running on instances you provision. You own latency, autoscaling policy, model versioning, and the incident pager. The cloud provides the metal and the network.
A single product frequently uses all three at once: a managed speech-to-text API at the front, a monthly fine-tune on rented GPUs, and a self-hosted ranking model in the hot path. Treating that as one purchase decision is how teams end up with a blended bill nobody can attribute.e.e.
The three patterns compared
| Managed AI service | Rented training capacity | Self-operated inference | |
|---|---|---|---|
| You own | The request and its interpretation | Model, data, training code | Model, serving stack, SLOs |
| Cost driver | Per token / per request / per API call | Accelerator-hours × utilisation | Instance-hours, largely fixed once provisioned |
| Cost behaviour at volume | Scales linearly with usage — no discount for predictability | Bursty; idle GPUs bill anyway | Improves with volume as fixed capacity fills |
| Data-residency question | Where does the vendor process and retain the payload? | Where does the training corpus live and get copied? | Which region holds the model and the request logs? |
| Typical failure mode | Silent model version change, rate limiting, dependency on vendor uptime | Job dies mid-run; checkpointing and cost overruns | Latency regressions under sustained load; scaling misconfiguration |
| Who carries the pager | The vendor, up to their SLA | Your team, for the job | Your team, fully |
The table is the useful artefact of this article. Take a workload, write it into one of the three columns, and most of the downstream procurement questions become answerable.
Where the naive reading breaks
The most common cost surprise is not “the cloud was expensive”. It is that a per-request price which looked cheap during a pilot stayed exactly the same per request after the product found its audience. Managed AI services are priced for elasticity, and elasticity is genuinely valuable while demand is unknown. Once inference volume becomes predictable and roughly flat, you are paying a premium for an option you no longer exercise. In our experience, that inflection point is where the migration conversation from managed API to self-operated inference usually starts — and where it should start, rather than eighteen months later during a budget review.
Training has the inverse shape. Training demand is genuinely lumpy: a burst of accelerator-hours, then weeks of nothing. Rented capacity fits that shape well, and buying hardware to match a peak you hit quarterly rarely pencils out. So the honest split for many teams is training in the cloud, steady-state inference somewhere cheaper and more predictable — which may be reserved cloud capacity, on-premise hardware, or the edge if latency or connectivity forces the issue.
The second break is data residency, and it fails later and more expensively. A managed API means your payload crosses an administrative boundary you do not control, and the relevant question is not “is this provider compliant?” but “for this specific data class, in this specific region, with this specific retention setting, is the processing lawful and auditable?” That question has a different answer for a customer support transcript than for a de-identified sensor reading. It also has to be answered before the contract, because the remedy after an audit finding is a migration, not a configuration change.
The third is ownership of reliability. Moving inference to a managed service moves the machinery but not the accountability — your users still call you when responses stop coming. Write down, before migration, who detects the incident, who communicates it, and what the fallback path is when the vendor is degraded but not down. Undocumented ownership is the failure mode that turns a two-hour vendor incident into a two-day one.
The split worth making explicit
The measurable outcome of this whole exercise is a workload split written down before procurement: which components run on managed AI services, which on rented accelerator capacity, and which stay on-premise or at the edge. With that split explicit, per-inference and per-training-hour costs can be compared against a fixed baseline instead of against a blended cloud bill that hides which component is actually growing.
A short pre-procurement checklist covers most of it:
- For each AI component, name which of the three patterns it belongs to. Components that plausibly fit two patterns are the ones to model in both.
- State the expected request volume and its variance. Flat and predictable argues against per-request pricing; lumpy argues for it.
- Name the data class crossing each boundary, its region, and its retention setting.
- Record who holds the pager for each component after migration, and what the degraded-mode behaviour is.
- Establish the fixed baseline cost you will compare the cloud figure against — including the on-premise or reserved-capacity option, even if you expect to reject it.
We keep coming back to this checklist because the ordering is what does the work. Providers get chosen last, and by then the choice is narrow enough to be boring.
This explainer stays deliberately at the definitional layer — it names the distinctions and does not recommend a vendor. The surrounding architecture and delivery questions, including how cloud and DevOps practice fit together around these workloads, are covered in our broader work on AI in cloud and DevOps, which goes further into the operational side than a single explainer usefully can.
Frequently Asked Questions
What does ‘AI in cloud’ mean, and what does it mean in practice?
AI Cloud refers to machine learning infrastructure and inference endpoints hosted by hyperscalers, consumed via API rather than self-hosted hardware. Here, term means running AI workloads on infrastructure someone else owns. In practice it resolves into three distinct patterns — calling a vendor’s managed AI API, renting accelerators to train your own models, and operating your own inference stack on cloud instances. The phrase is only useful once you say which of the three you mean.
What is the difference between using a managed AI service, training your own model in the cloud, and self-hosting inference on cloud infrastructure?
The difference is what you own. With a managed service you own only the request and its interpretation; with rented training capacity you own the model, data, and training code but not the machine; with self-operated inference you own the serving stack, the latency budget, and the pager. Cost drivers and failure modes follow directly from that ownership boundary.
Which parts of an AI workload actually benefit from running in the cloud, and which do not?
Lumpy, bursty work benefits most — training runs, periodic fine-tunes, evaluation sweeps — because you pay for peaks you only occasionally hit. Steady-state inference at predictable volume benefits least, since elasticity you never exercise is a premium you keep paying. Latency-critical or connectivity-constrained inference often belongs at the edge regardless of cost.
How do cloud AI costs behave — what drives the per-training-hour and per-inference figures?
Per-training-hour cost is accelerator-hours multiplied by how well you utilise them, so idle or poorly parallelised jobs bill at the same rate as efficient ones. Per-inference cost on a managed service scales linearly with request volume and does not reward predictability. Self-operated inference is closer to a fixed cost that improves per request as provisioned capacity fills.
What data-residency and access-control questions need answering before an AI workload moves to a cloud provider?
Name the data class crossing each boundary, the region where it is processed and retained, the retention setting in force, and who can access it on the vendor side. Answer those per data class rather than per provider — a support transcript and a de-identified sensor reading rarely have the same answer. Settle it before the contract, because the post-audit remedy is a migration.
When does it make sense to keep training or inference on-premise or at the edge instead?
Keep inference local when volume is high and flat, when latency budgets are tight, or when connectivity is unreliable. Keep training local when utilisation is high enough that owned hardware is genuinely busy rather than idling between runs. Data-residency constraints can also force the decision independently of cost.
What are concrete examples of AI workloads running in the cloud, and which of the three patterns does each one belong to?
A speech-to-text endpoint in a call-centre product is a managed AI service. A monthly fine-tune of a domain model on rented GPUs is rented training capacity. A ranking or defect-detection model served from your own containers behind Kubernetes is self-operated inference. Most real products combine at least two of the three.
What are the real benefits of running AI in the cloud, and which of those benefits disappear once your inference volume becomes predictable?
The durable benefits are elasticity for bursty work, no capital commitment, and fast access to accelerator generations you would not buy. Elasticity is the one that erodes: once inference volume is predictable and flat, per-request pricing charges you for optionality you no longer use. That is the point at which reserved capacity or on-premise serving becomes worth modelling.
Does moving AI workloads to the cloud change who is responsible for reliability and incident response, and how should that ownership be written down before migration?
It moves the machinery, not the accountability — your users still hold you responsible when responses stop. Before migration, record who detects an incident, who communicates it, what the degraded-mode fallback is, and where the vendor’s SLA ends. Ownership that is only assumed is the difference between a short vendor incident and a long one for your users.
Three decisions that expose whether AI Cloud fits
Most teams discover whether AI Cloud was the right choice within the first billing cycle—not from cost alone, but from whether the promised elasticity actually matched their inference pattern. If AI Cloud is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
