An NVIDIA data centre GPU is a card built and licensed for shared, around-the-clock operation in servers. That shows up in four places: stacked HBM memory, NVLink for multi-GPU work, hardware partitioning (MIG) so several tenants can share one card, and a GeForce driver licence that excludes data centre deployment. These are the cards you will often find behind the cloud computing GPU instances you can rent, so the useful questions come down to which tier your workload needs, what that tier costs, and whether renting or owning it makes more sense at your utilisation.
How a data centre GPU differs from a GeForce card
The cleanest dividing line is legal, not technical. NVIDIA’s GeForce software licence states that GeForce or Titan software is licensed only for GeForce or Titan hardware you own and “is not licensed for datacenter deployment”. NVIDIA’s stated reason, as reported by DCD, is that GeForce and Titan GPUs “were never designed for data center deployments with the complex hardware, software, and thermal requirements for 24x7 operation”. The same report notes that many observers read the clause as a way of steering customers towards the more expensive data centre line. Both readings can be true at once, and for a business the clause settles the question either way.
Data centre parts use stacked HBM memory: the A100 has 40 GB of HBM2 at 1,555 GB/s, and the H100 SXM5 has 80 GB of HBM3 at over 3 TB/s, according to NVIDIA’s architecture documents. They scale across cards through NVLink, which on H100 gives 18 links and 900 GB/s of bidirectional bandwidth. The line was also never built around a monitor: the former Tesla cards were originally unable to output images to a display at all. Multi-Instance GPU is another marker, since MIG is a feature of A100, A30, H100, H200 and Blackwell-class parts and is not supported on workstation or consumer RTX cards.
Reading NVIDIA data centre GPU names: generation, tier, form factor
An NVIDIA data centre GPU name gives you the architecture generation first, then the card’s position in the lineup. The first letter is the architecture, each named after a scientist: T for Turing (T4), A for Ampere (A10, A100), H for Hopper (H100, H200), L for Lovelace (L4) and B for Blackwell (B200). Within a generation, a higher number generally means a more powerful or feature-rich card, and some names carry the memory size, as in “A100 80GB”. The older “Tesla” label (as in Tesla V100) is the former name of the product line, not a separate tier.
Compute capability defines the hardware features and supported instructions of each architecture: Volta V100 is 7.0, Ampere A100 is 8.0, Hopper H100 is 9.0, and B200 is 10.0 in NVIDIA’s MIG documentation. MIG itself requires compute capability 8.0 or higher.
An H100 SXM5 and an H100 PCIe share a name but are different products: the PCIe card uses HBM2e at about 2,000 GB/s, has no NVLink and draws 350 W, while the SXM5 is roughly 30% faster on FP16 compute and 67% faster on memory bandwidth at 700 W, per a 2026 spec summary. The H100 NVL and H200 NVL are PCIe cards joined by an NVLink bridge, and the H200 NVL supports 2- or 4-way NVLink at 900 GB/s per GPU.
Quick reference: the current lineage
| Card | Architecture | Compute capability | Memory (vendor figure) |
|---|---|---|---|
| V100 | Volta | 7.0 | 16 GB HBM2, 900 GB/s |
| A100 | Ampere | 8.0 | 40 or 80 GB; 40 GB is HBM2 at 1,555 GB/s |
| H100 SXM5 | Hopper | 9.0 | 80 GB HBM3, over 3 TB/s |
| H100 PCIe | Hopper | 9.0 | 80 GB HBM2e, over 2 TB/s |
| H200 | Hopper | 9.0 | 141 GB HBM3e, 4.8 TB/s |
| B200 | Blackwell | 10.0 | 180 GB per NVIDIA’s MIG guide |
Sources disagree slightly on B200 memory: one 2026 spec summary gives 192 GB HBM3e.
Which GPUs data centres use for training and for inference
Large-model training runs on top-tier cards (A100, H100, H200, B200), because training needs lots of VRAM per GPU and fast links between GPUs. a16z points out that training a transformer needs about 8 bytes per parameter on top of the weights, so a 12 GB consumer card could barely train a 4-billion-parameter model. In practice, large training runs use clusters of top-end data centre GPUs with many cards per server and high-bandwidth connections. Fabrics such as NVLink are needed to train certain large models, because gradient synchronisation, tensor parallelism and pipeline parallelism all depend on fast GPU-to-GPU communication.
For inference, GPU memory often matters more than raw compute. An LLM has to fit in GPU memory before throughput matters, and token generation (decode) is limited by memory bandwidth, while prompt prefill is limited by compute. The KV cache grows linearly with sequence length, so long-context serving can make VRAM the main bottleneck. That is why the H200 matters: it uses the same Hopper architecture as the H100 but has 141 GB at 4.8 TB/s, nearly double the capacity and 1.4x the bandwidth, within the same power profile.
Below the top tier, a16z reports that large LLMs probably still need H100s or A100s, but smaller models such as Stable Diffusion run on A10, A40 or even RTX-class cards. Latency-tolerant batch work can cut compute cost by as much as 3-4x by using less powerful GPUs, for example A10s instead of A100s on AWS. At the bottom of the range, BentoML puts an entry-level T4 at a few hundred dollars a month, against more than $60,000 a month for a high-end H100 cluster. Where the cheaper card stops being cheap is its own question, covered in our piece on low cost GPUs for AI inference and when cheaper hardware costs more.
What an NVIDIA data centre GPU costs, and why one name has many prices
NVIDIA does not publish list prices, and what a buyer pays depends on volume and on whether the GPU comes as a complete system or through a vendor. Analyst estimates put the H100 at $25,000 to $40,000 per chip, and NVIDIA’s CEO told CNBC in 2024 that Blackwell would cost $30,000 to $40,000 per unit. A 2026 spec summary gives the same $25,000 to $40,000 range for new H100 SXM5 units as of Q1 2026.
Rental prices spread even further. One survey of 25 cloud providers found the same H100 renting for between $0.80 and $11.10 an hour, a 13.8x range. That gap is product difference sold under one GPU name:
- Interconnect class. A bare PCIe H100 and an NVLinked SXM H100 in an 8-GPU machine are different machines for multi-GPU work.
- Commitment terms. Reserved 1-to-12-month deals run 20-40% below on-demand, and spot runs 40-65% below, with the risk of interruption.
- Bundling and provider tier. A hyperscaler instance includes CPU, RAM, disk, an SLA and compliance support. A marketplace listing may come with no uptime promise.
The prices also move fast. AWS announced reductions of up to 44% on P5 (H100) instances and 33% on P4d (A100) instances. Eugene Cheah at Latent Space traces H100 rental from about $4.70 an hour at launch, to over $8 at the peak, to commodity rates of $2.85 down to about a dollar. Over the longer run, Epoch AI finds that hardware at a fixed performance level gets about 30% cheaper each year. Any quote is a snapshot, and that matters most to anyone thinking about buying.
What a cloud computing GPU instance actually gives you
When you rent a cloud computing GPU, you may be renting part of a card rather than the whole thing. MIG lets an A100 be partitioned into as many as seven separate GPU instances, which NVIDIA built with cloud providers’ multi-tenant use in mind. MIG gives each slice dedicated compute and a hardware-enforced memory boundary. Its limits are fixed profile sizes and no NVLink peer-to-peer between instances. Kubernetes time-slicing is weaker: pods share one address space, and an out-of-memory event or Xid error can take down workloads running alongside them. Before you size a job, check which kind of sharing an instance type uses.
Virtualisation and disaggregation also cost some performance. In a research system that pools GPUs across a datacentre network, the DxPU authors measured performance above 90% of a native GPU server in most cases. The overhead grew with more GPUs: ResNet fell from 92.7% to 82.4% of native as more GPUs were allocated. That is one research architecture, not every provider’s stack. In that system, single-GPU overhead was usually small but varied by workload, and it grew as more GPUs were allocated. For the same reason, a multi-GPU training job belongs on SXM instances with NVLink, not on several PCIe cards that happen to share a host.
When does renting data centre GPUs in the cloud make more sense than buying them?
Renting wins when sustained utilisation is low, and buying wins only when the card stays busy for long enough to outrun depreciation. The sources agree on that rule but not on the exact threshold. MillionMiner puts the crossover at roughly 40%. Digital Applied says renting wins below about 20% and owning wins above about 40-60% sustained, and calls the band in between one that “needs real workload analysis”. Read these as heuristics, not laws. The 20-40% band is where profiling your actual duty cycle decides it.
Worked break-even for one H100
Digital Applied’s reference case assumes a $25,000 H100 against a $3.00 per hour on-demand rental.
- Raw break-even: $25,000 ÷ $3.00 = 8,333 GPU-hours, or about 347 days of continuous use.
- At 40% utilisation, the same 8,333 hours take close to 30 months.
- Once power, cooling, networking and depreciation are added, Digital Applied reports that most analyses put practical break-even at 18 or more months of near-100% utilisation.
The result moves with the assumptions. MillionMiner’s version (a $30,000 card against $2.50 an hour) crosses near 14,000 GPU-hours, about 19 months at full use. Resale shortens break-even: used A100s still trade at $4,000 to $9,000, and a 40-50% residual after two years cuts effective break-even by about a third. Falling rental rates (the AWS cuts above) lengthen it.
| Situation | Leaning | Why |
|---|---|---|
| Below ~20% sustained utilisation, or bursty demand | Rent (on-demand or spot) | Idle owned hardware never pays back before it is obsolete |
| 20-40%, workload still changing | Rent, measure, revisit | The sources disagree here; actual duty cycle decides |
| Steady 24/7 inference | Own, or reserve long-term | MillionMiner: ownership pays back inside two years at steady 24/7 load |
| Own hardware, no facility | Colocate | A hosting fee (about $0.35/hr in MillionMiner’s example) replaces running your own facility |
| Infrastructure spend above ~$50M a year | Consider your own data centre | a16z’s threshold for when self-hosting may become cheaper |
Digital Applied names buying for the peak as the most common mistake it sees. It describes teams that size hardware for a launch quarter and then run it at 15% once demand settles. Its suggested sequence is to validate on a managed API, move steady load onto rented GPUs, and own only when utilisation is durably above about 40%. The deeper TCO arithmetic, including NVLink cluster design, is in our article on data centre GPUs for AI workloads: own vs rent, TCO and NVLink. The broader comparison with other accelerators is in our cloud GPU vs on-premise AI accelerator total cost analysis.
Choose the tier from a profile of the workload
Tier choice should follow measured workload need. Two vendor facts back this up. NVIDIA’s Hopper tuning guide says code that follows best practice for Ampere or Turing typically speeds up on H100 without changes. So moving up a generation is easy to try, and also easy to pay for without knowing what you gained. NVIDIA also reports, citing SemiAnalysis benchmarks, that B200 cost per million tokens on GPT-OSS-120B fell from $0.11 at launch to $0.02 within two months, from software alone. a16z also notes that 10x gains from software optimisation are not uncommon.
Before choosing between an L4 and an H200, we would profile three things: whether the model and its KV cache fit in memory at your real context lengths, whether decode is bandwidth-bound on the candidate card, and how much of the time the GPU sits idle. Idle time is the number the rent-versus-buy sources turn on. That last point is expanded in our article on the hidden cost of GPU underutilisation. Teams running inference as part of a production platform can see how we approach this engineering on our AI for AI infrastructure and production SaaS page.
FAQ
Does NVIDIA make CPUs as well as GPUs?
Yes, as part of combined products: CNBC reports that the GB200 pairs two Blackwell GPUs with an Arm-based CPU.
