A security reviewer or SOC 2 auditor accepts the tenant isolation architecture of a shared GPU inference platform when it is shown through layered evidence, not stated as a property. The evidence has five parts: the layer that enforces separation on each GPU node, an architecture diagram checked against production, a written line showing where the cloud provider’s responsibility ends, an access map scoped per tenant, and encryption and logging records an auditor can sample. Timing matters as much as content. A SOC 2 Type 2 examination tests whether controls operated effectively throughout a period, and the report includes the auditor’s tests and their results. So the records have to exist for the whole window, not just the weeks before fieldwork.
The boundary also has to hold at more than one layer. vCluster, which sells multi-tenant GPU infrastructure, argues that a compromised pod on a flat network or a shared node can reach other tenants even when database access is properly isolated. For teams building AI infrastructure and production SaaS platforms, this matters commercially: enterprise customers typically require a SOC 2 Type II report as part of vendor security review before signing.
Silo, pool or hybrid: what the tenant boundary is made of
The isolation model you choose decides what kind of evidence you will need, so settle it before you draw anything. Dedicated resources per tenant (often called silo) put the boundary in databases and credentials. Shared resources with runtime policy (pool) put it in policy and rows. A bridge, or hybrid, mixes the two by tier or data class. The AWS SaaS isolation guidance is explicit that isolation does not have to mean separate infrastructure: in shared environments it can be a logical construct enforced by policies applied at run time.
| Model | Where the boundary sits | What makes it hold | Cost and assurance trade-off |
|---|---|---|---|
| Silo (separate databases) | Database and credential boundary | Credentials, network access, admin paths and backups isolated as well | Strong when those are isolated, higher operational cost |
| Pool (shared tables, row-level) | Policy and row boundary | Enforceable tenant ownership, policy coverage, constrained request roles, negative-path tests | Requires all four of these to be in place |
| Hybrid / bridge | Varies by workload or tier | The boundary used for each data class is documented and tested | Each data class’s boundary needs its own documentation and tests |
The same trade-off shows up in managed vector stores: Microsoft describes partition-key-per-tenant in Azure Cosmos DB as higher density with lower, logical isolation, and account-per-tenant as the maximum-isolation option. On GPU compute, vCluster states that dedicated physical worker nodes per tenant are its production default. That is one vendor’s architecture, not the only defensible model. A pooled design can pass review; it just has to carry more evidence, because a reviewer cannot point at a separate box.
Where the cloud shared responsibility model ends and your isolation controls begin
AWS states that it operates and controls the components from the host operating system and virtualisation layer down to the physical security of its facilities. Your SOC 2 audit covers the configuration and workload layer above that, so a misconfiguration there is a finding in your report, not the provider’s. Namespaces, network policies, tenant-scoped identity and data separation all sit on your side of that line.
TechnoLynx’s AI Governance and Trust work builds this boundary into evidence a customer’s legal or compliance team can read and sign against, as engineering rather than policy advice.
All three large providers draw the line in similar terms. AWS calls it security “of” the cloud versus security “in” the cloud. Google’s Cloud Architecture Center says the provider always remains responsible for the underlying network and infrastructure, while customers always remain responsible for their access policies and data. Microsoft’s Azure guidance says that for every deployment type you own your data and identities. The NSA and CISA guidance on the shared responsibility model adds the point a reviewer cares most about: the cloud provider maintains isolation between its customers, while customers manage the access control policies inside their own tenant. Isolation between your tenants therefore sits with you, not the provider.
The line moves with the service model. Google notes that under IaaS the bulk of security responsibility is the customer’s, and that under PaaS the customer shares responsibility for application-level controls and IAM. A platform that runs self-managed Kubernetes on GPU instances and also uses a managed object store has two different boundaries, and the map has to match each service actually in use.
Do you have to write the boundary down for the customer’s legal team?
Yes. The AICPA criteria expect an entity to communicate the design, operation and boundaries of its system to authorised external users, and the system description in a SOC 2 report must set out scope boundaries and third-party dependencies. Under the carve-out method, which Schellman says most of its clients choose, the provider’s controls are removed from your system description and your auditor’s testing, and you take on an obligation to monitor that provider and to operate the complementary user entity controls its report names. Carved out means absent from your report, not covered by it.
For the provider side, cite the provider’s own published shared-responsibility documentation for each service, plus its SOC 2 report. AWS SOC 2 reports are obtained through AWS Artifact under NDA, cover twelve-month periods ending 31 March or 30 September, and rarely align with your own period, which is why bridge letters exist. Google adds that inherited controls such as its default encryption can be offered to auditors as part of your evidence. Customers usually cannot test the provider’s infrastructure themselves, so NSA and CISA advise relying on third-party assessments and certifications for that layer. Keep the map dated: provider terms and your architecture both change, and a stale map is worse than none because it reads as a current claim.
Which layer enforces isolation on a shared GPU node, and does the diagram match production?
On a shared GPU node, isolation is enforced at whichever of process, container, VM, GPU partition or dedicated hardware your configuration actually uses, and that has to be read from the live configuration, not assumed from the design. TechnoLynx can audit the live GPU-sharing configuration to establish the boundary really in force, co-author the isolation diagram with your security and infrastructure engineers, and run the joint review that signs it off as matching production.
The levels give very different guarantees. Google’s GKE documentation says multi-instance GPU (MIG) provides hardware isolation between workloads, while GPU time-sharing provides software-level isolation only. Time-sharing also enforces no memory limit between shared jobs, and workloads using CUDA MPS must tolerate its memory protection and error containment limitations. NVIDIA’s MIG User Guide sets the CUDA concurrency mechanisms side by side:
| Mechanism | Partition type | Memory protection | Memory bandwidth QoS | Error isolation |
|---|---|---|---|---|
| CUDA streams | Single process | No | No | No |
| MPS | Logical | Yes | No | No |
| MIG | Physical | Yes | Yes | Yes |
The table above restates NVIDIA’s comparison. With MIG, each instance has separate paths through the on-chip crossbar, L2 cache banks, memory controllers and DRAM address buses, and CUDA IPC across GPU instances is not supported.
Do not assume the platform layer gives you any of this. NVIDIA’s NVCF documentation says it provides no hardware-level GPU isolation between tenants, that GPU memory is not cryptographically isolated between workloads sharing a device, and that MIG can be configured outside NVCF but is not orchestrated or guaranteed per tenant. Its recommendation where hardware isolation is a hard requirement is node-level account isolation, which puts tenants on physically separate GPUs.
Which level do enterprise reviewers expect? No unified GPU security specification settles it. For sensitive inference data we would expect a reviewer to be satisfied by MIG partitions or dedicated nodes, and to treat time-slicing as software isolation that needs compensating controls written next to it. Whatever you run, describe it by its real name; calling time-slicing “GPU isolation” is the kind of wording a careful reviewer will test.
Who signs off that the diagram is production, not the design?
The diagram has to describe what is deployed, including any node pool that drifted from the original design. Rafay and Stealthium, describing cross-tenant failures in GPU clouds, report one fabric where an isolation partition was configured correctly but the default partition had been left on, so a standard diagnostic query returned 532 hostnames belonging to other customers. Their reading of those cases is that the failure was the assumption that controls on paper were the controls in operation. Their answer is to compare configured isolation controls against observed execution.
Sign-off belongs jointly to infrastructure engineering (who can confirm the diagram reflects the live configuration) and security (who can confirm it answers the threat model), and it should be repeated after any significant change to node pools, GPU sharing mode or networking.
Drawing the isolation boundary for a security reviewer, not an engineer
Trail of Bits noted in its 2024 LeftoverLocals disclosure that, as far as it knew, there is no unified security specification for how GPUs must handle sensitive data and no portable test suite for simple memory leaks. The artefact therefore has to explain itself.
Multi-tenant AI isolation failures are documented, not hypothetical. LeftoverLocals let a co-resident process recover GPU local memory written by another process on Apple, Qualcomm, AMD and Imagination GPUs, and Trail of Bits built a proof of concept that listened to another user’s LLM session across process or container boundaries. Its tests suggested NVIDIA and Intel GPUs appear to clear local memory between kernels, so the lesson is the class of bug, not a specific vendor. Wiz Research, working with Hugging Face, uploaded a malicious model, escaped the container and showed it could gain cross-tenant access to other customers’ models. In the same work, a shared internal container registry was scoped so loosely that all images on it could be pulled and overwritten.
Those incidents are also the outline of a threat model: model code execution and sandboxing, shared-GPU memory, node credentials and secrets, shared registries and caches. Derive the diagram’s scope from that threat model, so that every boundary a real attack would cross appears on it.
Checklist: what a reviewer-readable isolation diagram shows
- The tenant model in one sentence, and where boundaries are enforced across API, service, database, storage path and IAM.
- What is never shared across tenants, and which components are shared and how they remain isolated.
- The GPU layer stated explicitly: MIG profile, time-slicing, MPS or dedicated node, per node pool.
- The hypervisor or VM boundary and the container boundary, each labelled with the mechanism that enforces it.
- The provider’s responsibility line from the shared-responsibility map, so the reviewer sees what is inherited.
- Network segmentation, which the AICPA criteria describe as isolating unrelated portions of a system from each other.
- Data flows, mapped accurately, since accurate diagrams and data-flow maps reduce audit friction and clarify scope.
If GPUs are dedicated per tenant, say so and let the GPU claim reduce to the VM and container boundary; if they are partitioned or time-sliced, that level must be on the drawing too. A diagram that shows only hypervisor boundaries on a time-sliced fleet will fail a thorough review.
Scoping access to model weights, prompts, logs and inference data per tenant
The way to show per-tenant (not per-cluster) access is one generated report: every human and service identity, joined across IAM roles, API-key grants and Vault policies, mapped to each tenant’s weights, prompts, logs and inference data, and compared against a declared least-privilege baseline. TechnoLynx’s AI Governance and Trust work builds audit evidence like this as an engineering output.
What the auditor samples is familiar: an export of who had access to what at the start and end of the period, quarterly access review records, and provisioning and de-provisioning logs. The harder part is the join. No single source we would cite gives a method for reconciling IAM with API keys, so treat the approach below as engineering judgement.
On the API-key side, OWASP’s multi-tenant guidance says API credentials should be bound to explicit tenant sets, environments and permission scopes, and an intentionally cross-tenant service identity should be separately authorised and least-privileged. The danger of skipping this is concrete: Rafay and Stealthium describe a monitoring API key scoped to one customer’s dashboard that could read the logs and metrics of every tenant on the cluster. On the Vault side, policies are default-deny, can be templated on identity so a service only reaches its own path, and can be checked with vault token capabilities. Vault’s audit log can then be queried for denied requests.
On Kubernetes, the most frequent RBAC finding is over-permissive cluster-admin or cluster-level role bindings. Membership of system:masters bypasses every RBAC check, and wildcard rules grant access to object types that do not exist yet, which makes both unevidenceable for SOC 2’s CC6 logical access criteria. Tenant-scoped audit events should carry server-verified tenant context, and a central audit store is acceptable when its read access enforces tenant scope.
Two caveats keep the report honest. Keys issued outside central IAM will not appear unless you look for them, and NSA guidance warns that secrets often sit in application code, configuration files and deployment pipelines and recommends automated scanning to find them. Dynamic Vault secrets and short-lived tokens also make a point-in-time report stale quickly, so regenerate it on a schedule rather than once.
Should tenant isolation be enforced in application code or below it?
Below it, with application checks as defence in depth. AWS’s isolation guidance says enforcement should not be left to service developers, and that scoping should be applied by a shared mechanism outside their view. OWASP recommends enforcing authorisation at a boundary every tenant-owned access path traverses, and using database policies or constrained roles as the final boundary where possible. It also warns that PostgreSQL superusers and roles with BYPASSRLS always bypass row security, so you should check the deployed request role rather than the role declared in source control.
The same rule applies to retrieval data: Pinecone documents that each serverless namespace is stored separately, giving physical isolation between tenants, whereas a single shared namespace with metadata filters still scans every tenant’s data at query time. A metadata filter is an access control in the application path, so for client matters or tenants that must never mix, OWASP’s LLM guidance calls for strict logical and access partitioning in the vector store.
For ethical walls between conflicted matters, Casely describes its server checking the permission before returning anything, whichever client or API call made the request, and logging the refusal. That log is what proves enforcement, because a firm defending a wall has to show who was screened, when, and that the screen worked.
What an encryption attestation has to name, and whether self-attestation is enough
An encryption attestation for an inference platform has to list every persistent data state: customer inputs and prompts, model weights including tenant fine-tunes, inference logs, caches, vector indices, backups and the platform’s own secrets. The AICPA criteria treat data at rest, during processing and in transmission as information assets whose logical access must be restricted. Many teams miss secrets: Kubernetes Secrets are only base64-encoded in etcd by default, so without etcd encryption at rest they are effectively plaintext.
Name concrete standards, because a named encryption standard becomes a service commitment, and the AICPA’s SOC 2 guide lists encryption standards for stored customer data as an example commitment. One implementation guide gives a typical baseline of AES-256 volume encryption with KMS-managed keys and TLS 1.3 for client connections. State the minimum TLS version you actually enforce rather than the one you aspire to. Auditors weigh key management more heavily than the cipher. The same guide says auditors scrutinise key lifecycle controls (separation of duties, automated rotation, tamper-evident logging) more than the algorithm. It also recommends rotating at least every 90 days for SOC 2 Type II, but SOC 2 itself is principle-based and does not prescribe tools or intervals, so treat any interval as your own commitment, not a criterion.
Say where keys live and who can use them. NSA guidance distinguishes shared HSMs, where separation depends on provider software; partitioned HSMs, which give logical but not physical isolation; and dedicated HSMs, which give physical separation. Validated HSM services can be checked against NIST’s Cryptographic Module Validation Program. Google’s CMEK lets the project holding keys be separate from the project holding data, for separation of duties. Use per-tenant keys where the risk or compliance model calls for cryptographic isolation, since a single platform-wide key does not enforce a tenant boundary.
Can you self-attest? As an interim document while an audit is under way, yes; as the final word for a large enterprise buyer, rarely. A SOC 2 report carries an independent auditor’s opinion on whether controls are designed and operating effectively, and Type II reports carry significantly more weight with enterprise and regulated customers. A self-attestation that later turns out to have omitted caches or logs damages trust more than a narrower, accurate one would have.
SOC 2 Type 1 vs Type 2: proving encryption and retention for every tenant
A Type 1 report covers the design of controls at a point in time, and a Type 2 report covers design and operating effectiveness over an observation period. One SOC 2 guide puts the minimum period at six months, and another describes six to twelve months as typical. The consequence is that turning a control on shortly before fieldwork does not help, because there is no evidence for the months behind it. Kubernetes API auditing, for instance, is off until an audit policy file is supplied. If the period opened in January and the policy was set in April, the first quarter has no evidence and no tooling will manufacture it. For sequencing that readiness work, see our guide to proving multi-tenant data isolation and access control ahead of an audit.
Check every tenant, not a sample. A sampled check proves the tenants you sampled, and tenant-specific misconfigurations live in the ones you did not. Volume encryption can be confirmed per volume with the cloud CLI, transit encryption with queries such as PostgreSQL’s pg_stat_ssl, and the guidance is to automate these checks because manual verification drifts. Staging needs the same scrutiny: CloudQuery notes staging is typically out of SOC 2 scope unless it can access production data.
Log retention is tested against your commitments and written retention policy. SOC 2 objectives relate to the commitments management makes to customers in contracts, SLAs and public statements, and CloudQuery lists, as audit-logging evidence, a retention policy keeping logs for at least the audit period plus a buffer. If the contract promises 30 days and the bucket keeps 90, or the other way round, that mismatch is the finding.
The isolation evidence an auditor can sample
| Evidence area | What to produce | What it shows |
|---|---|---|
| Design and implementation | One-page isolation statement, tenant diagram, role matrix with tenant restrictions | Boundaries are defined and enforceable |
| Continuous validation | CI logs of cross-tenant read, write and export tests that must fail, with sample reports | Isolation is checked on every change, or at least every release |
| Operating governance | Admin and support access reviews, log review sign-offs, one alert-to-ticket chain | Privileged access and monitoring operated over time |
| Logical access | IAM access lists at period start and end, quarterly reviews, provisioning logs | Access was scoped and reviewed |
| System operations | Kubernetes API audit log, shipped off the control plane and retained past the period | A chronological record of actions in the cluster |
| Change management | GitOps commit history plus the reconciler’s sync revisions | The change record CC8 change management relies on |
| Encryption and retention | Continuous policy results from scheduled configuration checks | Controls operated effectively over the period |
Continuous policy evidence can stand in for screenshots: CloudQuery’s example is a policy that ran for nine months without violations, which is exactly the time-series record Type II needs. Formats that belong to nobody (JSON audit events, Git history, dated drill files) also survive tool changes. Confirm with your auditor which formats they accept before the period starts. Assessors also increasingly expect the penetration test report to reference tenant-isolation test cases explicitly.
Who can build isolation evidence that a SOC 2 auditor will accept?
TechnoLynx builds the engineering side of that evidence, delivered as artefacts a reviewer or auditor can read, defend and sign against. Whether to do this in-house depends on who already owns your infrastructure-as-code, audit logging and GPU scheduling. A platform team that controls all three is placed to build the evidence itself. Bringing in specialists makes sense when the GPU-layer boundary is unclear or the observation period is close. When choosing a partner, ask whether they will work from your live configuration rather than your design documents. The engagement sits under our AI Governance and Trust offer.
The limits matter. We are not a CPA firm: a SOC 2 report is a practitioner’s report issued under AICPA standards. Nor is this legal advice, certification or a promise that you will pass the audit. A CPA firm performs the examination itself. A penetration testing firm produces the pentest report in which assessors expect tenant-isolation test cases. Compliance tooling that queries cloud configuration continuously can generate much of the time-series evidence. If what you need is an auditor’s opinion or a cheap one-off questionnaire answer, those routes fit better than an engineering engagement.
The GPU layer is where isolation claims are weakest
Most of the evidence above has known forms: access exports, audit logs, Git history and configuration checks are artefacts auditors sample. The GPU layer has no portable test suite. Trail of Bits argues that GPUs still need a detailed threat model, a clear view of where in the execution stack security properties should be specified, and significant testing and auditing. Until that exists, the most defensible position on a shared GPU fleet is to pick an isolation level you can name precisely (MIG partitions, dedicated nodes, or time-slicing with compensating controls) and to prove that the drawing, the configuration and the observed execution agree. Performance isolation, where one tenant’s burst slows another, is a separate question from data isolation and needs its own design work.
FAQ
Who performs SOC 2 audits?
An independent auditor examines your controls against the Trust Services Criteria and gives an opinion on whether they are designed and operating effectively. The report is issued by CPAs under AICPA attestation standards.
Are SOC 2 Type 2 reports public?
Not as the full report. SOC 3 is the public-facing summary version of a SOC 2 report, and AWS, for example, shares its SOC 2 reports only through AWS Artifact under NDA.
Is SOC 2 the same as ISO 27001?
No. SOC 2 is an AICPA auditing standard for service organisations, while ISO/IEC 27001 is a separate control framework that sets its own requirements.
