Our advice: to find out whether a cloud GPU option is allowed for regulated data, get a written determination from counsel, the data protection officer, and the IRB or data-governance board. That determination should cover the exact services, regions, backups, logs and keys the deployment would use, before anyone prices an instance. Compliance removes options. Cost and benchmarks only rank the options that are left. Once that gate is passed, an honest cloud computing cost analysis measures throughput per dollar on your own workload, counts the engineering time to port the pipeline, and adds performance per watt wherever on-prem power is fixed. It does not rely on vendor-reported or host-submitted numbers.

Before comparing costs, how do I confirm a cloud GPU option is legally allowed for our regulated data?

Eligibility comes from a formal review, never from a benchmark. We recommend a review in three parts: governance or IRB sign-off for the placement, a legal or DPO reading of the rules in each jurisdiction involved, and contract checks such as a Business Associate Agreement (BAA) and in-region hosting. Tenant isolation and encryption are inputs to that review. Each control only counts once it is matched to a named requirement: an IRB approval, a HIPAA BAA, a GDPR transfer rule, FERPA or a national sovereignty law.

TechnoLynx’s AI Governance and Trust offering builds approval evidence packs for auditors, procurement committees and regulators. The legal decision itself stays with counsel, the DPO or the IRB.

What to confirm for each type of data

Data and proposed placement	Confirm before any costing
PHI on a cloud GPU service (US)	A fully executed BAA mapped to the specific services, regions and subprocessors used, and on AWS each service on its HIPAA-eligible list
Controlled-access human genomic data under the NIH GDS Policy	Adherence to the NIH Security Best Practices, written into new or renewed Data Use Certifications from 25 January 2025; repositories are expected to adopt the NIST SP 800-53 Moderate baseline
EU health, genetic or biometric data	An exception under Article 9 GDPR (otherwise processing is prohibited, and Member States may add conditions), plus a transfer assessment if cloud storage sits outside the EEA
US student education records	That the arrangement preserves the school’s “direct control” over the records and is set out in contract terms; FERPA does not prohibit cloud hosting
Data under a national sovereignty or critical-infrastructure rule	A legal or DPO determination for every option, on-prem included, against the specific law

The last row has no shortcut. When the data is subject to cross-border or sovereignty rules, only counsel or the DPO can say whether an in-country region run by a foreign provider is acceptable. The US Department of Education’s Privacy Technical Assistance Center says something similar for schools: FERPA may not be the only statute that applies, state laws can be stricter, and legal staff should review any cloud option.

Can patient-derived imaging data sit on a shared HPC GPU partition?

Our answer: only if the partition’s isolation is enforced, verified and documented, and the IRB approval and data-use agreement actually cover that environment. A partition with sound technical isolation still fails if the data-use agreement never mentions it.

Be sceptical of isolation claims. The LeftoverLocals study illustrates the threat potential of GPU vulnerabilities. Its authors note that conservative GPU virtualisation, such as NVIDIA vGPU or AMD MxGPU, physically partitions the GPU and so stops users from sharing local memory. They also note that cloud providers such as AWS and Azure are unlikely to offer shared GPU instances. A shared institutional cluster is a different case, so ask which partitioning mechanism is enforced and get the answer in writing.

Accountability does not pass to whoever runs the cluster. Under FERPA, the entity that discloses records stays legally accountable for protecting them wherever they are hosted. Under GDPR transfers, the exporter carries primary responsibility.

Do we need a BAA before PHI touches a cloud GPU instance, and are our instance types eligible?

Yes, you need the BAA first. Under AWS’s terms, a covered entity or business associate agrees not to use AWS’s HIPAA Eligible Services with PHI until it has entered into an AWS BAA. Microsoft explains why: a cloud provider handling PHI for a covered entity becomes a business associate, and HIPAA requires the two parties to sign a BAA. Federal HIPAA guidance on cloud computing helps cloud providers and their customers understand their responsibilities under the HIPAA Rules.

Eligibility is decided per service, and sometimes per feature, not per GPU. AWS lists the services eligible for ePHI and still requires customers to configure them in line with HIPAA. The list also carves out features: SageMaker AI is eligible except for Studio Lab, Ground Truth Plus and the public and vendor workforce options. Google’s BAA applies only to its named Covered Services. It does not cover PHI held in your own on-premises storage or in third-party applications. So check each service the deployment uses against the list, not just the instance.

What goes into the BAA matters too. Accountable’s HIPAA policy guide for digital pathology says never to upload whole-slide images until a fully executed BAA is mapped to the exact services, regions and subprocessors used. It lists the terms to require, including data location, flow-down of obligations to subcontractors, and no model training on your PHI without separate authorisation.

Signing the BAA does not make you compliant. Microsoft says plainly that offering a BAA does not make your Azure solution compliant, and that it does not inspect, approve or monitor what you deploy. Google says it supports HIPAA compliance within the scope of a BAA, but you remain responsible for evaluating your own compliance.

Buying capacity as spot or reserved changes the price, not the service or region the work runs in. The compliance question stays the same: is that service in scope? On Google Cloud, the BAA covers the whole infrastructure (all regions, zones and network paths) plus the listed products, so the BAA itself does not restrict which region you pick. Third-party hosted LLM and speech endpoints need the same check. Google’s BAA does not support third-party services, and PHI must not go to a third party unless that party has its own BAA in place.

Does an in-country cloud region satisfy data-residency rules, including backups, logs and key custody?

Not on its own. Residency is about where data sits. Sovereignty is about which laws it is subject to where it is stored, processed or transmitted. AWS’s Digital Sovereignty Lens defines data residency as keeping data in a certain jurisdiction, and data sovereignty as data being subject to the laws of the place where it is, whether stored, processed or transmitted. The same document describes a sovereign workload through five questions: where it is located, who can reach it, whether it keeps working when conditions change, whether it can move elsewhere, and whether you can prove the controls are in place.

Check where copies such as backups go, not just primary storage. AWS Outposts, which puts AWS hardware on your own premises, connects to an AWS region and lets customers send content such as EBS snapshots and RDS backups there. The European Data Protection Board recommends making access logs tamper-proof and making audits cover the importer’s processors and sub-processors. Support staff are part of the boundary too. Google’s Assured Support limits a regulated workload’s support cases to personnel who meet set location and personnel conditions, and Assured Workloads limits which regions you can select according to the compliance regime you choose.

For the EDPB, encryption counts only under strict conditions, one being key control. The EDPB’s 2021 recommendations count encryption as an effective supplementary measure only when the keys stay solely under the control of the data exporter, or of a trusted entity in an equivalent jurisdiction, among other conditions. The same document notes that US importers covered by FISA 702 can be obliged to hand over data in their possession, and possibly the keys needed to read it. AWS’s data residency paper, which is the vendor’s own view, describes how the CLOUD Act lets authorities send requests directly to providers. It argues that customers can reduce the remaining risk with encryption and other cloud controls. For EU data, the EDPB’s text is the regulator’s position and carries more weight than a provider’s whitepaper.

GPU workloads add a complication. A model cannot compute on ciphertext, so the data is unencrypted in GPU memory while it is processed. The EDPB says that when a cloud provider needs data unencrypted to do its job, and problematic third-country law applies, it cannot envisage any effective technical measure to stop that access. Customer-managed keys protect data at rest. They do not settle the question for the moment the model runs.

Holding your own keys is possible, at a cost. Google Cloud pairs its External Key Manager with Key Access Justifications to give customers cryptographic control over access. AWS KMS supports bring-your-own-key material that customers generate and keep outside AWS. AWS’s external key store goes further: encryption uses key material you hold outside AWS. AWS warns, though, that this is a major departure from shared responsibility, and that for most customers the extra availability and latency risk will outweigh the perceived security benefit. Where possible, it advises choosing a Region within a 35-millisecond round trip of the external key manager.

Does staying on-prem automatically satisfy sovereignty rules?

No. Physical control answers the question of where data is. Access control, continuity, portability and auditable evidence of controls still have to be addressed. The NIH’s genomic data expectations apply the NIST SP 800-53 Moderate baseline to repositories, in addition to local, state and federal law and institutional policy. The Education Department’s FERPA guidance asks whether a cloud solution offers equal or better security than your own data centre, a comparison that only makes sense if your own data centre is assessed too. AWS’s residency paper, written by an interested party, argues that keeping data on your own servers in your own country is not in itself an adequate basis for security. If you must stay on-prem, you still need documented access controls, tamper-evident logging, a secure way to update models and the evidence an auditor will ask for.

A hybrid such as on-prem inference with cloud backup does not avoid the review. The backup copy is storage like any other and goes through the same checks. Compliance decisions also do not stay settled. The EDPB tells exporters to re-evaluate protection at intervals, and AWS’s sovereignty guidance treats compliance as continuous, not a one-off exercise. Once eligibility and cost are both known, defending the result to leadership is a separate task, covered in justifying cloud vs on-prem placement under compliance and performance constraints.

How to compare cloud and on-prem GPU cost on throughput per dollar for your real workload

We take cost per unit of work as the measure: per million embeddings, per million tokens, per inspected frame. Measure it on your own workload on each eligible option, using the pricing model you would actually buy and including the engineering time it takes to get there. Spec-sheet throughput tells you what the silicon could do. It tells you nothing about the utilisation your pipeline actually reaches on it.

TechnoLynx runs this as a measured comparison through its GPU Performance Engineering work. It starts by fixing the cost-per-unit metric and the power budget, then ports a representative slice of your pipeline to each candidate and measures with the software stack held constant. LynxBenchAI published results serve as the device-level reference when drawing up the hardware shortlist.

Compare over three to five years, not year one

Metro Colo Advisory’s repatriation analysis treats total cost of ownership as a three-to-five-year comparison. Dedicated infrastructure loses year one to migration and capital outlay, wins the later years if the workload is steady, and usually crosses over somewhere in year two. The same analysis lists the costs a cloud bill hides that the dedicated side must carry:

• hardware amortised over three to five years, with a refresh cycle;
• facility rent and power;
• interconnect;
• operations;
• a platform layer to replace managed services such as Kubernetes and observability;
• migration, including months with both environments running in parallel;
• a risk buffer for hardware failures.

Operations is usually the line companies forget. In the analysis’s worked example, leaving out roughly $70,000 to $75,000 a month for operations and platform would have shown $5.3 million a year instead of $4.4 million.

The same source sets three conditions for moving off cloud:

• steady utilisation of roughly 60% or more around the clock;
• spend large enough, usually $50,000 a month or more, that the overhead is small against the saving;
• a win against optimised cloud pricing, not list price.

For steady workloads, it puts the saving off cloud at 30 to 50% once every cost is counted. When work is bursty and moves little data, the gap is narrow or negative and staying on cloud is the right answer. This is one advisory firm’s analysis, and its conditions are more useful to you than its percentage.

Egress goes on the cloud side of the ledger. egresscost.com’s 2026 comparison gives entry egress rates of $0.09/GB on AWS, $0.087/GB on Azure and $0.12/GB on GCP Premium Tier. NAT Gateway processing at $0.045/GB and cross-AZ transfer at $0.01/GB each way come on top. All three large providers now waive egress fees on data you move out when you leave, so steady outbound traffic is the cost to model, not the exit.

For LLM serving, a cost-benefit study of 54 deployment scenarios found that on-prem breaks even within a few months for small models, around two years for medium models and around five years for large ones. It found on-prem viable mainly at 50 million tokens a month or more, or under strict data residency mandates. For a per-option model at the accelerator level, see the total cost analysis of cloud GPUs versus on-premise AI accelerators. Sizing the cloud side before costing it is covered in right-sizing and costing a cloud GPU migration against on-prem infrastructure.

Worked example: turning a published measurement into cost per unit

These steps use AIMultiple’s September 2025 benchmark: vLLM serving Llama-3.1-8B-Instruct, priced at RunPod on-demand rates, with storage and network fees excluded. The numbers are someone else’s measurements on someone else’s workload, so repeat the steps on your own.

1. Measure sustained throughput at the batch size and precision you will serve. In this benchmark, one H100 delivered 23,243 tokens per second, counting input and output tokens.
2. Divide throughput by the hourly price. At $2.69 an hour, the source reports 8,642 tokens per second per dollar.
3. Convert to cost per unit: our arithmetic on the source figures gives about 83.7 million tokens an hour and about $0.032 per million tokens.
4. Repeat for each pricing model you could buy. For A100s, GetDeploying puts the median on-demand price at $1.86 per GPU-hour across 39 providers as of 31 October 2026. That is about $1,339 a month at 720 hours, against a cheapest verified in-stock on-demand price of $0.89. Reserved prices start at $0.66 (24-month term) and spot at $0.68, neither verified in stock.
5. Add the amortised porting cost (next section) before you rank the options.

The setup behind those numbers matters. The benchmark ran an independent vLLM instance on each GPU, which takes inter-GPU communication out of the result entirely. A tensor-parallel deployment of a larger model would behave differently. Render farms apply the same idea: per-frame wall-clock time multiplied by node cost per hour gives cost per frame.

Porting time belongs in the cost per unit

Calling a vendor-specific API instead of a standard one does not make the same code faster. What a vendor path offers is access to lower-level features, and making use of them takes real engineering commitment that does not always pay off. Whether it pays off for your pipeline is a question profiling answers, not a datasheet or a blog post. That is why porting effort cannot be left out of an accelerator comparison.

Put it in the formula directly. Cost per unit equals the hourly price divided by units per hour, plus the one-off porting and validation effort divided by the units you will process over the amortisation period. Because porting is a fixed cost, expected volume growth changes the ranking. An option with a cheap hourly rate and an expensive port can win at high volume and lose at low volume, so run the comparison at your current volume and at your projected volume. Porting estimates depend heavily on the workload, so give them a range, not a point.

Should perf-per-watt be part of the comparison when on-prem power is fixed?

Yes, whenever the power envelope is the binding limit. In that case the question is how much work fits under a fixed number of kilowatts, not how fast one card runs. Racks that fit equals the hall’s usable power divided by sustained power per rack, and the per-rack figure varies widely. The Uptime Institute puts NVIDIA’s NVL72 at 132 kW per rack. Its other figures for the same GPU generation:

• 64-GPU racks run at 80 to 90 kW, but will likely need about 50% more racks to match NVL72 training performance;
• four 8-GPU servers per rack stay under 50 kW and can be air-cooled, with fans taking up to 10% of full load.

Cooling, power delivery and floor loading (1.4 tonnes for an NVL72 rack) all add demands at this density.

Measure efficiency the same way on every candidate: throughput divided by average power draw. The DEEP-GAP study (2026) recorded power from device telemetry through nvidia-smi. It found NVIDIA’s L4 delivered substantially more performance per watt than the T4 within a similar power envelope, while the T4 stayed competitive for large-batch work where power limits or existing deployments dominate. For a cross-system reference, MLPerf Power gathered 1,841 reproducible measurements from 60 systems under rules written by a consortium of more than 20 organisations. Uptime also notes that public data is still not enough to settle the trade-offs between cluster performance, cost and deployment speed, so measured per-watt numbers on your own workload are worth more than extrapolations. In the cloud, power is already in the hourly price, so perf-per-watt shows up there only through cost per unit.

Why vendor-reported and host-submitted benchmark numbers shouldn’t drive a cost comparison

Vendor and host numbers come from a party that benefits from the result, so treat them as claims to verify. The “benchmarking crimes” paper lists the typical failures:

• missing platform specifications and software versions;
• relative numbers reported without absolute ones;
• no proper baseline;
• microbenchmarks presented as overall performance;
• no indication of variance.

Any comparison, chip generations included, needs software versions recorded to be reproducible.

We treat a benchmark as normalised and independent when the party running it has no stake in the result and the following conditions hold:

• Identical work on both sides: matched pairs only, such as same scene and user on both sides with at least three tasks per side (38 scenes of 1,419 tasks passed).
• Locked and recorded settings: operating system, engine build and settings held constant, and the GPU driver recorded and named when a newer card needs a newer driver.
• Available hardware and software: MLPerf’s rules require submitted systems and software stacks to be generally available, with the framework and accelerator libraries counted as part of the stack.
• Spread, not just a midpoint: the same render-farm study found the same hardware gained 1.6x on overhead-bound scenes and over 5x on compute-bound ones, and so reports the median, interquartile range and full range.
• Cost and energy alongside speed: perf per dollar and perf per watt for each generation, not just raw throughput.

The DEEP-GAP study swept batch sizes to find where each chip’s throughput saturates. The L4 reached up to 4.4x the T4’s throughput and hit peak efficiency at batch sizes of 16 to 32, while the T4 needed larger batches. How results are normalised and read across devices is a measurement question in its own right, and we leave it to LynxBenchAI’s published methodology and results rather than arguing it here.

Showing renters a verified, normalised score instead of the host’s peak claim

Show renters a verified, normalised price-performance score produced by an independent benchmarking harness, displayed apart from the host’s submitted peak figure. Normalise per GPU model and driver combination unless you have measured that driver variance is small.

LynxBenchAI gives an AI infrastructure marketplace an outside reference point: its published results show how each GPU runs the public PyTorch stack that a team’s own models run on, so they are broadly indicative for custom PyTorch models too. They do not measure any renter’s specific application. TechnoLynx can design and run the per-instance harness, test whether driver variance justifies per-driver scores for your GPU and provider mix, and set a refresh schedule so scores keep up with driver and firmware changes.

Make the distinction obvious on the page. Label the verified score with the harness version, test date, driver and benchmarked workload, and show the host’s claim separately and visibly marked as self-reported. Show absolute throughput and price-performance next to any ratio, because ratios alone hide whether a configuration is even sensible. Where you can, show a range rather than a single figure, since one midpoint hides the spread an operator needs for planning.

Per model and driver, or one number per GPU model?

Whether driver variance warrants per-driver scores has to be determined empirically for your GPU and provider mix. If driver-induced variance is small compared with the differences between GPU models, one number per model is enough and easier for renters to scan. If it is comparable or larger, publish per model-and-driver scores and say which basis each figure uses. The render-farm methodology above records and names the driver precisely because an exact match across generations is not always possible.

Each option has a cost. Per-driver granularity makes the interface more complex and can overwhelm non-technical renters. Driver and firmware updates can make a displayed score stale. And a verified score still reflects only the benchmarked workload, not every renter’s use.

Who can run a like-for-like cloud vs on-prem GPU comparison on our own pipeline?

TechnoLynx runs this comparison on your actual pipeline rather than a stand-in, through its GPU Performance Engineering work. The engagement covers four steps:

• frame the cost-per-unit metric and the power budget;
• port a representative slice of the pipeline to each option that passed the compliance gate;
• measure throughput, latency, memory, energy and cost per unit of work with the software stack held constant;
• hand over a reproducible cost and performance baseline, plus a guide to how many racks fit your power budget, for future procurement.

For the hardware shortlist, LynxBenchAI’s published results show how each device runs the public PyTorch stack; they do not measure your application. Whoever you bring in, ask whether they disclose stack versions, use matched pairs and report absolute figures with their spread, and check that they are not selling the hardware being compared.

There are other realistic routes:

• MLPerf results, whose rules require an available software stack for each submission;
• a colocation or TCO adviser for facility and contract terms.

FAQ

What are the three primary rules under HIPAA?

The Security Rule, the Privacy Rule and the Breach Notification Rule. Google Cloud’s HIPAA documentation describes complying with all three as a responsibility shared between the provider and the customer.

Is AWS SOC 2 certified?

AWS lists SOC 1, 2 and 3 among its certifications, validated by independent third-party auditors, with reports available through AWS Artifact. They provide inherited evidence to support your own audit activities.

What is data residency in AWS?

AWS states that customers choose the geographic locations where their data is stored, and that the data does not move unless the customer decides to move it.