A data center GPU is an accelerator validated for continuous operation in dense rack environments. It has ECC memory, 80 GB or more of high-bandwidth memory, NVLink for traffic between GPUs and Multi-Instance GPU (MIG) partitioning, and it is validated for server airflow and long-term enterprise drivers. Whether to buy one or rent it comes down to average utilisation, which is the one number you control. Against public on-demand prices the break-even for an 8-GPU H100 server is roughly 20 to 40 percent average utilisation, and one guide says owning wins above 40 to 60 percent sustained.
The own-versus-rent decision reduces to one axis you control: utilisation. The rest of this article covers how a data center GPU differs from the card in a workstation, which parts carry AI work today, what full TCO contains, and why NVLink changes the picture once a model spans more than one GPU.
How a data center GPU differs from a consumer card
Raw compute is not the difference. Consumer GeForce cards can match data center parts on compute and memory bandwidth per dollar. The differences are in features built for continuous, multi-tenant production use:
- ECC memory. It detects and corrects bit errors during long runs. Any single error is rare, but a GPU running continuously for months has far more chances to hit one than a card used in short sessions.
- Memory capacity. Consumer cards typically carry 16 to 32 GB, while data center cards carry 80 GB or more. That limits which models a consumer card can serve without aggressive quantisation.
- NVLink. NVLink support has been limited or removed on recent GeForce generations. Data center GPUs rely on it for multi-GPU training and large-model inference.
- MIG partitioning. One physical GPU can be split into isolated instances. On A100, H100, H200 and B200 the maximum is seven. Each instance has its own memory, cache and compute cores, isolated at the hardware level. MIG has been supported since the Ampere generation. Consumer GPUs do not support it, so they need either dedicated hardware per workload or software-only sharing with weaker isolation.
- Thermal design for racks. The H100 PCIe card, for example, uses a passive heat sink that depends on system airflow, and it runs at up to 350 W. Consumer cards are designed for open-air desktop cases.
- Licensing. Clause 2.8 of NVIDIA’s GeForce software licence says the software “is not licensed for datacenter deployment”.
None of this makes consumer cards obsolete. They remain a reasonable choice for prototyping, individual experiments and non-critical work, where giving up ECC, NVLink and enterprise support is a fair trade for the lower price. A cost analysis published on arXiv found that sub-30B-parameter models are feasible on a single RTX 5090, which it priced at about $2,000, and it lists an A100 80GB at about $15,000. If your team is asking whether a cheaper card is good enough for serving, our piece on when low-cost GPUs for AI inference end up costing more covers that question in detail.
Which data center GPUs run AI training and inference today
The older A100 is still offered for rent, and AMD’s MI300X appears in the same analyst purchase price list as the NVIDIA parts. The table collects what the vendors and the cited analysts state. Per-unit prices are one analyst tool’s list and should be read as indicative.
| GPU | Memory | Power (as stated) | Indicative unit price |
|---|---|---|---|
| A100 80GB | 80 GB, 2 TB/s bandwidth | not stated in sources used | ~$15,000 (arXiv cost analysis) |
| H100 SXM | 80 GB or 94 GB | up to 700 W TDP | $27,000 (Silicon Analysts) |
| H100 PCIe | 80 GB or 94 GB | 350 W TDP | not stated |
| H200 SXM | 141 GB | not stated | $38,000 (Silicon Analysts) |
| B200 | 180 GB | not stated | $40,000 (Silicon Analysts) |
| GB200 (NVL72) | 186 GB | 1,200 W per chip | $32,500 per unit (Silicon Analysts) |
| AMD MI300X | 192 GB HBM3 | 750 W typical board power | $15,000 (Silicon Analysts) |
NVIDIA’s current data center line goes beyond these. It includes the GB300 NVL72 (72 Blackwell Ultra GPUs with 36 Grace CPUs in one rack), the Vera Rubin NVL72, and RTX PRO Blackwell Server Edition cards aimed at enterprise inference and visual computing. Generations move quickly. The Silicon Analysts list above already includes the MI350X at $22,000.
Peak spec figures are not measured performance: real throughput depends on model, precision, software and interconnect. Measured results by device are listed in the LynxBenchAI AI hardware directory.
What a data center GPU costs to buy versus rent
Buying means paying for much more than the chip. CNBC reported in March 2024 that analysts put the H100 at $25,000 to $40,000 per chip, and NVIDIA’s CEO priced Blackwell at $30,000 to $40,000. At the system level, SemiAnalysis puts an H100 server at around $190,000, or about $250,000 all-in once storage and networking are added for a typical hyperscaler. It puts a GB200 NVL72 rack at about $3.1 million, or about $3.9 million all-in. A complete per-GPU line item includes the CPUs, host memory, power supplies, the baseboard with its NVLink switches, a share of the scale-out network and storage, installation, burn-in and spares.
Renting is quoted per GPU-hour, and listed cloud GPU rates differ between providers. Lambda lists on-demand H100 SXM at $3.99 and B200 at $6.69. The Safeguard analysis cites AWS on-demand H100 at $6.88 and Oracle’s list price of $10.00 for H100 and $16.00 for a GB200 GPU. Rental prices also change: in June 2025 AWS cut P5 (H100) instance prices by up to 45 percent.
When does owning data center GPUs beat renting?
Owning beats on-demand renting once average utilisation passes the ratio of your owned fixed cost per hour to the rental price. Sources plug in different capex, lifetimes and rental prices.
Worked example: the 25 percent break-even
The assumptions here are the Safeguard analysis’s illustrative grid point, not a quote: $30,000 all-in capex per H100, a four-year life and an 8 percent cost of capital.
- Operating cost excluding capex is about $0.48 per GPU-hour. That is $0.10 of electricity (9.64 cents per kWh, PUE 1.2), $0.31 of facility capacity at $175 per kW per month, and $0.07 of staff.
- Capital recovery and maintenance add about $0.41 per hour for every $10,000 of capex, so $30,000 adds about $1.23.
- Fixed cost per hour: $0.48 + $1.23 ≈ $1.72 when the GPU is fully busy.
- Break-even against AWS on-demand at $6.88: $1.72 ÷ $6.88 = 25 percent average utilisation.
Run the same $1.72 against Lambda’s $3.99 H100 rate and the break-even rises to about 43 percent ($1.72 ÷ $3.99). SemiAnalysis estimates an all-in owned cost of $1.525 per hour even for a poorly run operator on 13 percent debt, against favourable cloud deals of about $2 per hour.
Why the realistic threshold is 40 to 60 percent
The Digital Applied 2026 guide says renting wins below roughly 20 percent utilisation, owning wins above 40 to 60 percent sustained, and the band in between needs real workload analysis. Safeguard says its headline break-even hides average utilisation over the hardware’s life, including procurement, installation and burn-in, gaps between projects, and the nights and weekends of an inference service. Digital Applied’s higher band adds power, cooling, networking and depreciation. The same guide shows how payback stretches: an H100 at $25,000 against a $3.00 rental pays back in about 8,333 GPU-hours, roughly 12 months at 24/7. At 40 percent utilisation that becomes nearly 30 months.
Other studies give their own figures. A DCS Global analysis puts on-premises cost parity at about 18 months for 60 percent utilisation. Lenovo’s 2025 five-year comparison of an 8× H100 server ($871,912 on-premises against $4,306,416 on AWS on-demand) puts break-even at a usage ratio of 0.2025, or about five hours a day, assuming full GPU use during those hours.
Reservations change the question
A reservation removes utilisation from the comparison. You pay for every hour of the term whether or not you use it, so both options become fixed costs. Safeguard cites Azure’s three-year ND H100 v5 reservation at $5.40 per GPU-hour, 56 percent below on-demand. Against that rate the only question is whether you can run the hardware for less than the provider charges. If your $1.72 estimate holds, you can at any utilisation. Lenovo’s version of the same point is that discounted reserved pricing raises the number of daily usage hours needed to justify buying.
Training and inference give different answers
A training cluster with a queue of runs stays above 80 percent busy. Meta reported more than 90 percent effective training time for Llama 3. That makes training the workload that fits owned hardware best. Inference follows people: traffic rises during working hours and falls at night and at weekends. Safeguard works an illustrative daily curve swinging between 0.5 and 1.5 times its average, where a fleet sized for the peak averages 66.7 percent utilisation. The cheaper design is often to own the base load and rent the peak.
The bottlenecks differ as well. Peer-reviewed modelling of distributed LLM workloads finds inference primarily bound by off-chip memory rather than compute. Token generation is limited by memory bandwidth, while prefill is limited by compute.
This is why profiling comes before choosing AI hardware. The TCO decision should rest on a profiled workload, not on generic spend. Digital Applied suggests a practical order: start on a managed API to measure real volume, move the steady portion onto rented GPUs, and buy only once utilisation stays above about 40 percent.
Decision table: own, rent or mix
| Workload pattern | Lean | Why |
|---|---|---|
| Experimental or bursty, under ~20% average use | Rent on demand | Renting wins below roughly 20% utilisation |
| 20–40% average use | Measure before committing | The sources call this band workload-dependent |
| Steady, above 40–60% sustained, with rack space ready | Own | Clears the break-even once full TCO is included |
| Training queue running above 80% busy | Own | Training is the workload that best fits owned hardware |
| Inference with a daily demand cycle | Own the base load, rent the peak | Each owned unit pays off only if busy for enough of the hours |
| Three-year commitment is acceptable | Compare owned hourly cost to the reserved price | Utilisation cancels out against a reservation |
Data center GPU TCO beyond the purchase price
Full TCO covers capex (GPUs, servers, storage, networking, setup), operating costs (electricity, cooling, maintenance, staff, software licences) and the cost of scaling as the workload grows.
Capital dominates GPU TCO. SemiAnalysis attributes this to NVIDIA’s margins: an example GPU server costs $7,025 a month in capital against $1,871 in hosting. Safeguard reaches the same conclusion, saying that getting the capex, the life and the cost of capital right matters more than anything else. The Silicon Analysts model of a 1,024-GPU cluster over three years puts hardware at 60.0 percent of TCO, colocation at 19.6 percent, staff at 6.3 percent, refresh at 4.7 percent, power at 4.1 percent, networking at 3.3 percent and software at 2.0 percent.
Estimates of how much non-hardware costs add vary widely. The same Silicon Analysts tool says three-year non-hardware costs typically add 50 to 70 percent on top of hardware, with hardware at 35 to 50 percent of TCO. DCS Global says power, cooling, facilities, staff and licences add 3 to 5 times the hardware cost over five years. Silicon Analysts says the split depends on electricity rates, GPU generation and deployment model. Budget for the range rather than picking one number.
In Safeguard’s H100 base case, power price matters less than utilisation: moving electricity from 5 to 20 cents per kWh changes the hourly cost by about 10 percent. Moving utilisation from 40 to 80 percent halves the cost per useful hour, from $4.30 to $2.15. Facilities overhead depends on PUE: Google reports 1.09 across its fleet in 2025, while Uptime Institute’s 2025 survey average is 1.54. Size electrical capacity for rated power, but budget energy at operational draw. LBNL models AI servers at about 70 percent of rated, based on H100 nodes averaging 74 percent.
DCS Global advises planning a three-year refresh: hardware depreciated over five years will be technically obsolete in three. Safeguard gives three to five years as the common planning range and notes that Amazon shortened some server lives from six to five years from January 2025. A shorter life is a reasonable way to price in the falling cost of newer compute.
Rack power and hosting are the hidden gate
Rack density limits where you can host the hardware. A single H100 server draws 10 to 12 kW, and a rack of eight draws 80 to 96 kW, which DCS Global puts at 10 to 15 times a standard compute rack. GB200 NVL72 racks draw 120 to 130 kW and require liquid cooling. The air-cooled DGX B200 moves 1,550 CFM through a 10U chassis, which is why many existing halls can hold only a few per rack. Colocation is tight too. CBRE reported North American vacancy at a record low of 1.6 percent in H1 2025 and expects pricing to stay at $200 or more per kW per month. Silicon Analysts treats colocation as the usual route below about 10 MW and owned facilities as sensible only above about 50 MW, where construction at $15 to 20 million per MW can be amortised. Digital Applied’s conclusion that owned hardware beats rental comes with the condition “provided you already have somewhere to rack and cool it”.
Why NVLink matters when a model spans several GPUs
NVLink matters because multi-GPU inference and training wait on communication between GPUs. In tensor parallelism, each layer is split across two, four or eight GPUs, and after every layer each GPU must exchange its partial results with all the others before the next layer can start. During that exchange the Tensor Cores often sit idle. Minimising the time spent communicating between GPUs is therefore critical.
A pair of H100 PCIe cards can be joined by three NVLink bridges. Bridges only connect one adjacent card, though. In HGX H100 and H200 systems, NVSwitch lets every one of the eight GPUs talk to any other at 900 GB/s at the same time. The fabric is non-blocking, so the peak rate does not fall as more GPUs communicate. NVIDIA reports up to 1.5 times higher real-time throughput for Llama 3.1 70B inference on H200 with NVSwitch than with point-to-point links. Recent generations have widened the link. A100 connects at up to 600 GB/s. Blackwell’s fifth-generation NVLink runs at 1,800 GB/s per GPU and joins 72 GPUs in one domain in the GB200 NVL72. NVIDIA states that sixth-generation NVLink on Vera Rubin reaches 3 TB/s per GPU, 12 times PCIe Gen6.
The effect also appears in independent modelling. A 2024 performance study finds that the NVLink Switch System adds roughly another factor of 2 to H100 training throughput over InfiniBand alone. It also finds that for Llama2-13B on eight GPUs, communication time is about 1.6 times memory time.
NVLink also affects the own-versus-rent decision in two ways. First, the NVLink switches are part of the per-GPU capex: a DGX H100 node carries four of them. That is part of why a node’s capex is much more than its accelerators. Second, whether the model fits in one GPU’s memory decides the parallelism you need. If it needs tensor parallelism, Tensor Cores often idle during each exchange, and NVIDIA reports up to 1.5 times the real-time inference throughput on H200 at two-way tensor parallelism with NVSwitch. Which case you are in should come from a profile, not from the model card, before pricing either option. Profiling the workload and fixing what it shows is what GPU performance engineering delivers.
FAQ
Are data centers filled with GPUs?
GPU buying is concentrated: Uptime Institute’s 2024 survey found that hyperscale and IT services operators are the dominant buyers of GPUs but make up a small share of respondents, and that 33 percent of respondents support HPC on accelerator hardware. Uptime expects densities to rise as servers drawing 1 kW per rack unit or more ship more widely.
