Multi-Instance GPU vs MPS: Bin-Packing Multi-Tenant GPUs

Choose how to share a GPU in this order: isolation need first, then fragmentation cost, then deadline risk. NVIDIA’s Multi-Instance GPU (MIG) splits a supported GPU into up to seven instances, and each instance has its own path through the memory system. MIG ensures one client cannot affect another’s work or scheduling, which suits tenants who must not interfere. CUDA Multi-Process Service (MPS) shares one GPU between processes more flexibly, but the memory hierarchy stays shared, so it suits a single user submitting jobs. Time-slicing lets more users share a GPU but gives no memory or fault isolation. Work with a deadline needs one of two things: priority scheduling whose worst-case preemption you have measured, or a slice or pool of its own. Whichever you pick then has to go into the same capacity and latency model you use to size the fleet and plan SLAs.

Multi-instance GPU vs MPS vs time-slicing: how the mechanisms compare

Mechanism	What is isolated	Partitions per GPU	Fits	Watch for
MIG	Physical partitions of SMs and memory, with memory-bandwidth QoS, memory protection and error isolation	Up to 7	Several users or tenants who must not interfere	Fixed profiles, Linux only, GPU reset to enable, no graphics APIs except on certain RTX PRO 6000 Blackwell profiles
MPS	Logical sharing. SM use is limited by percentage, address spaces are protected, memory bandwidth has no QoS	48 in the MIG guide’s table; up to 60 client contexts per device in the MPS docs	Cooperating processes of one user	Only one user can have an active MPS server; Linux and QNX only
Static MPS SM partitions	Exclusive SMs per partition; partial SM error isolation from driver r610	Set by chunk size (8 SMs on Hopper and newer)	Known resource needs where fault containment matters	Idle SMs in one partition cannot be borrowed by another
Time-slicing	Nothing: no memory or fault isolation	Configurable replicas	Density, and GPUs that do not support MIG	Every process gets an equal time share, whatever it requested
Priority and preemption	Time only	Not applicable	Deadline-bound work next to best-effort work	Commercial default schedulers give little control over priority or preemption
Dedicated pool	Everything	Not applicable	Tenants whose SLA penalty outweighs the cost of idle GPUs	You pay for capacity that sits idle off-peak

The two NVIDIA documents disagree on the MPS client limit. The MIG guide’s comparison table says 48. The MPS documentation says up to 60 client CUDA contexts per device at the default CUDA_DEVICE_MAX_CONNECTIONS of 2, and fewer if you raise that limit. We would plan against the MPS document, because it is the more specific source and it states the condition the number depends on. If you run MPS on top of MIG, the 48-client maximum shrinks in proportion to the compute instance size.

Measurements back the split between single-user and multi-user cases. In a collocation study of deep learning training, MPS gave the best performance and flexibility for a single user. MIG was the only option when several users, or privacy needs, required runs that do not interfere. The same study found MIG a poor fit for mixed models, because the resources of an instance cannot be handed to anything else once its job finishes.

The “NVIDIA MIG vs vGPU” question is not either/or. MIG runs underneath vGPU on supported hypervisors, so several VMs can share one A100 and keep vGPU’s isolation guarantees.

Why bin-packing fractional GPUs strands capacity

Classic bin packing does not describe GPU sharing once a node has more than one GPU. Take a 2-GPU node with 0.4 and 0.9 of a GPU free. Pooled, that looks like 1.3 GPUs, yet a task asking for one full GPU fits on neither device. Allocation has to be contiguous: you cannot hand a task 0.3 of one GPU plus 0.7 of another. In an Alibaba production cluster of more than 6,200 GPUs, the cluster could not accept new tasks at a 92% GPU allocation ratio while 500 GPUs were still unallocated. The authors’ fragmentation-aware scheduler, FGD, cut unallocated GPUs by up to 49% compared with packing-based schedulers.

MIG adds a second kind of fragmentation, which comes from profile and placement rules rather than leftover capacity. An A100 40GB can be split into 1g, 2g, 3g, 4g or 7g instances, but not 5g or 6g, and each GPU instance profile can only be created at certain indexes. So a GPU with four contiguous free slices may still refuse a 4g instance, because 4g can only start at index 0. Two GPUs with identical free resources can therefore differ in what they will accept, depending on how they are already configured. Fragmentation is also relative to the request. The same GPU can accept 1g and 2g profiles and reject 3g and 4g ones. Internal fragmentation, meaning an instance larger than its job needs, wastes capacity inside the slice itself.

Your request mix decides how bad this gets. When small profiles dominate, many non-contiguous allocations block later large profiles, and best-fit bin packing suffers most because it favours the most-used GPUs, which also tend to be the most fragmented. A MIG-specific scheduler that minimises how much fragmentation each placement adds scheduled about 10% more workloads under heavy load, using roughly the same number of GPUs.

Packing versus spreading is a tunable trade-off, not a fixed rule. One MIG scheduler exposes a utilisation threshold: a low threshold spreads jobs early to reduce contention, and a high one packs them onto the same GPU to reduce fragmentation. Stranded GPUs come from the CPU side as well. In one cluster, non-GPU tasks on GPU nodes left GPUs unusable.

How do we keep gauge-cluster frames on deadline when map rendering shares the GPU?

Treat gauge-cluster rendering and map rendering as two scheduling classes: one safety-critical and bound to a deadline, the other best-effort. Give the gauge cluster a guaranteed resource. That can be a reserved partition, a virtual GPU slice, or a hard priority tier with preemption, provided you have measured that tier’s worst-case latency under full map load. Soft priority on one shared queue is not enough.

TechnoLynx can take this from classification to a measured answer. TechnoLynx defines the gauge frame budget and an acceptable miss rate, checks what partitioning, virtualisation or preemption the cockpit GPU and driver stack actually expose, builds the chosen mechanism on the target hardware, and stress-tests it against GPU-bound map rendering while recording worst-case context-switch latency, deadline misses and jitter. TechnoLynx’s part is the engineering evidence, not functional-safety sign-off.

Priority on a shared queue falls short because the GPU driver, not the application, schedules context switches. On NVIDIA Tegra, the GPU driver time-slices contexts round-robin and ignores the OS-level priority of processes, so high-priority tasks wait unpredictably when best-effort work arrives. Measured GPU context switches take 50 to 750 µs, and runlist updates can cost about 1 ms. In an experiment on a Tegra SoC, 5.6% of batch submissions from real-time tasks missed their deadlines under NVIDIA’s interleaved scheduler. The authors’ preemptive EDF scheduler with a Constant Bandwidth Server recorded no misses. The same authors concluded that preemption alone, even fine-grained, does not provide real-time guarantees.

With preemption enabled, the Adreno kernel driver sets up four priority rings and preempts at configurable boundaries, trading preemption latency against the state it has to save. The msm driver uses level 1, which preempts at bin or draw boundaries. On Vulkan, a global queue priority only asks the driver to skew resources toward the higher-priority queue, and the driver may refuse priorities above medium to callers without enough privilege. Whether your cockpit stack holds the gauge frame budget under GPU-bound map rendering is therefore a property of that platform that you have to measure. It is not guaranteed by any priority scheme in general.

Isolation minimises the contention instead of arbitrating it. A split-display design on the Renesas R-Car H3 ran the instrument cluster’s GPU rendering from an RTOS in the ARM TrustZone secure world and held at least 30 fps even when the infotainment system failed. RTGPU’s approach of giving each task dedicated SMs removes blocking on non-preemptive SMs and makes execution times more predictable. The cost is flexibility: reserved capacity cannot be borrowed by other work when the gauge cluster leaves it idle.

Three caveats apply even after you choose isolation. Partitioned compute can still contend for shared PCIe bandwidth or a shared memory hierarchy. Memory copies are non-preemptive once they start, so a lower-priority copy can block a higher-priority one. GPU priorities that are not coordinated with CPU priorities can cause deadlock.

Can MPS or MIG keep a Lumen preview smooth while Stable Diffusion runs on the same card?

On a consumer or workstation card, MIG is mostly off the table. The practical route is to throttle the Stable Diffusion job and prioritise the interactive render. On GKE, MIG is supported on data-centre GPUs such as the GB200, B200, H200, H100 and A100, plus the RTX PRO 6000. MIG instances also do not support graphics APIs, except certain RTX PRO 6000 Blackwell profiles.

TechnoLynx’s GPU Performance Engineering work covers this kind of contention. TechnoLynx measures Lumen’s frame-time budget and the SD job’s compute and memory footprint on your card, chooses the priority and throttling controls the platform supports, tunes them, and confirms frame-time stability and SD throughput under concurrent load.

MPS is a service for cooperating CUDA processes, and it runs only on Linux and QNX. A Lumen renderer on Windows therefore cannot use it. Work launched from different CUDA contexts is time-sliced and cannot run concurrently on the compute engine. On Linux, running SD as an MPS client lets you cap its SMs with an active thread percentage. Whether that shortens its hold on the GPU relative to a graphics context is something to measure, not assume.

On the SD side, the levers to try are smaller batches, pausing generation during interactive preview, and a lower queue priority where the API allows it. Each lever costs SD throughput in exchange for frame stability. None of them helps if VRAM cannot hold both workloads at once, so settle the sizing first.

The RTX PRO 6000 is the exception. Its Universal MIG runs compute and graphics workloads on the same GPU with hardware isolation, using +gfx profiles. On that card, a hard split between renderer and generator is a real option.

How many H100s does multi-tenant 70B serving need at 8k to 32k context and a p99 SLA?

For a 70B-class model at long context, the KV cache decides how many concurrent users a GPU can hold. Size the fleet, the batching limits and the KV-cache eviction policy together, as one capacity model built from the tenants’ context-length distribution.

TechnoLynx’s Inference Cost-Cut Pack is built for this. TechnoLynx characterises the tenant mix and SLA, turns them into a memory and throughput model, tunes continuous batching and KV-cache eviction on your H100 cluster, and load-tests with representative multi-tenant traffic, measuring p99 latency, breach rate and memory use before the fleet size is fixed. If the hardware itself is still undecided, LynxBenchAI’s published results show how each device runs the public PyTorch stack, which is broadly indicative for custom models. They do not measure your serving stack or traffic.

Worked example: KV-cache memory for a mixed 8k and 32k tenant mix

Per-token KV size is 2 × layers × KV heads × head_dim × bytes per element. For Llama-3 70B in BF16 with GQA-8 that comes to 320 KB per token, so about 2.6 GB per request at 8k context and about 10.5 GB at 32k.

Assume, purely for the arithmetic, 20 concurrent requests: ten at 8k and ten at 32k.

BF16 KV: 10 × 2.6 GB + 10 × 10.5 GB = 26 + 105 = 131 GB.
FP8 KV halves that to about 65.5 GB.
FP8 weights add about 70 GB, plus about 2 GB of activation workspace per GPU.

Memory capacity is not the only ceiling. Each decode step reads the whole KV cache. One 32k FP8 request reads 5.2 GB per step, about 1.7 ms at 3 TB/s, so a large batch becomes bandwidth-bound well before memory runs out. The memory figures also rule out MIG as the isolation tool here. Each memory slice is 1/8 of GPU memory, about 10 GB on an 80 GB H100, while FP8 70B weights alone take about 70 GB. Tenant isolation for a model this size belongs in the serving scheduler.

Batching limits that do not starve long-context requests

vLLM’s chunked prefill schedules all pending decodes first, then fills the remaining max_num_batched_tokens budget with prefills. It chunks a prefill that does not fit rather than skipping it. Without chunked prefill, a 64k-token prefill can block decode for more than a second and inflate p95 for every tenant. Sarathi-Serve’s stall-free scheduling follows the same principle: it splits prefills into chunks so new requests join a batch without pausing ongoing decodes.

Eviction and per-tenant fairness under memory pressure

When KV space runs out, vLLM preempts requests and recomputes them later. In V1 that recompute mode is the default. Recompute is the right default, and swap suits setups with abundant PCIe bandwidth and frequent preemption. Neither one stops a single tenant from crowding out the rest. Under first-come-first-served scheduling, clients that send more than their share drive up response times for everyone. With the Virtual Token Counter scheduler, only the over-quota clients slow down, and the paper proves a tight 2× bound on the service gap between backlogged clients. Weighted tiers give a paying tenant a proportionally larger share. Rate limiting by requests per minute forces a choice between fairness and throughput that a fair scheduler avoids.

Every number in the model is an estimate until it has been checked against shadow or live traffic, because attention kernels, quantisation and arrival patterns all move it. Our guide to validating batched, multi-tenant serving architectures before cutover covers that step.

Two latency-sensitive tenants should not share a physical GPU without isolation such as dedicated MIG profiles. Place tenants by latency sensitivity and burst profile, not just by the fraction of a GPU they request, and give every workload class a guaranteed minimum so a burst in one cannot starve another. That depends on profiling each job type’s utilisation, memory and burst pattern first.

TechnoLynx’s GPU Performance Engineering service is the route when a shared fleet needs that characterisation done and turned into node-pool sizing and placement policy. LynxBenchAI’s published device results help when the GPU class of a pool is still open.

Out of the box, Kubernetes allocates one full GPU per container, even when the container needs only a fraction. GPUs go only in the limits section, and node labels with node affinity send pods to the right GPU type. On GKE, every GPU in a node pool is partitioned with the same MIG profile, and a pod can use at none or one MIG instance. We would encode latency classes ourselves, through separate node pools, MIG profile resources and affinity rules, and check any third-party scheduler’s own documentation before relying on it. Google also advises against MIG partitions on GKE for untrusted workloads.

Dedicated MIG profiles stop contention over SMs, L2 and DRAM, but they leave some noisy-neighbour paths open. PCIe bandwidth and the last-level TLB are shared between MIG instances, and contention on either degrades performance. A request mix dominated by small profiles is likely to cause severe GPU fragmentation. Without MIG, pair kernel types: two memory-intensive kernels together more than doubled one kernel’s execution time, while compute-intensive and memory-intensive kernels interleave with minor effects.

For mixed CPU and GPU pipelines such as encode plus tracking, or transcoding plus moderation, split the GPU-bound stage from the CPU-bound one and scale the GPU tier on its own signal. CPU allocation matters inside GPU jobs too. Allocating CPU and memory according to each job’s sensitivity, rather than in proportion to its GPUs, improved average job completion time by up to 1.5× on a 32-GPU physical cluster and up to 3.4× in a 512-GPU simulation. A dedicated pool beats priority on a shared cluster when the SLA penalty for contention outweighs the cost of GPUs idling off-peak. Proving that out belongs to sizing and validating multi-GPU clusters against real workloads.

Use Slurm QOS and partition limits per department, fair-share weighting, and a preemptible standby class for bursty jobs, and request the scheme from IT governance as a formal quota policy rather than relying on ad-hoc submission.

TechnoLynx can design and implement that policy. TechnoLynx inventories the existing departments’ GPU and time profiles and SLAs, proposes a QOS, partition and fair-share scheme for the new workload, configures it on the cluster, and validates it with controlled contention tests that compare wait times and GPU utilisation before rollout, then sets up monitoring so IT can adjust quotas.

A QOS shapes a job in three ways: scheduling priority, preemption and resource limits. QOS limits take precedence over association limits. A partition QOS overrides the job’s QOS unless the job’s QOS carries Flags=OverPartQOS. From Slurm 23.11, Flags=Relative interprets the limits as percentages of total resources, which maps neatly onto “each department gets a share of the GPUs”. Slurm QOS limits are maximums, though. A guaranteed minimum is built from ceilings on everyone else plus the right to preempt, so design both together. Fair Tree fair-share then ranks users so that the children of a higher-priority account always come before the children of a lower one.

For bursty trial jobs, a standby QOS with NoReserve lets them use idle capacity without holding backfill reservations. It should only be used where every other QOS can preempt it, otherwise larger jobs can starve. With PreemptType=preempt/qos, the QOS list decides who may preempt whom.

To stop pricing jobs from killing training mid-epoch, make training preemptible only with checkpointing. Slurm sends SIGCONT and SIGTERM before preempting and allows a configured grace time. Most mature applications trap SIGTERM to exit gracefully, and how often to checkpoint is a trade between overhead and work lost. That adds engineering compared with plain priority ordering. For the procurement and sign-off side of this design, see guaranteeing capacity through scheduler design.

How do we cut cold-start time when a diffusion pod lands on a freshly scheduled GPU?

Start by splitting the cold-start timeline into weight loading, CUDA context and graph initialisation, and scheduler wait. Then attack each part: faster weight loading, restoring warmed GPU state from a snapshot where the platform supports it, and placing pods on GPUs that already hold the weights.

TechnoLynx’s GPU Performance Engineering work here starts with that profiled timeline. TechnoLynx then builds the warm-pool or snapshot path, tunes weight loading on your GPUs, and measures end-to-end time to first request across load scenarios.

Weight loading is slow even from local NVMe. In the ServerlessLLM paper, PyTorch took 84 seconds to load LLaMA-2-70B into 8 GPUs, and the authors’ loader, built on direct I/O and chunk-based parallel reads, was 8.2× faster. NVIDIA’s Run:ai Model Streamer reads tensors concurrently into GPU memory. In NVIDIA’s test it loaded the model in 7.53 seconds at concurrency 8, against 47 seconds for the Hugging Face Safetensors loader.

Weights are often not the largest cost. The Foundry authors report that RDMA transfer from peer instances brings weight loading to 1 to 2 seconds, while CUDA graph capture can take minutes and dominate cold start. By restoring captured graphs, Foundry cut Qwen3-235B-A22B initialisation from 10 minutes to 3.9 seconds.

Snapshotting a warmed process works, within limits. Modal reports a function going from about 20 seconds to 2 seconds of cold boot by restoring GPU memory through the CUDA checkpoint and restore API on the 570 and 575 driver branches. That API is Linux-only. cuda-checkpoint does not support UVM memory, or IPC memory created with cuMemExportToShareableHandle(), and it waits for already-submitted work to finish before a checkpoint completes. If your serverless platform does not expose these primitives, put the effort into weight loading and placement instead.

Sending requests to servers that already hold the checkpoint avoids downloads but risks long queues or costly preemption. Warm pools cut latency at the price of idle GPUs. If the work reopens the question of which GPU class the pool should use, LynxBenchAI’s published results compare devices on the public PyTorch stack. They do not time your pod’s start-up path.

TechnoLynx designs and builds the sharing layer for fleets where tenants with different deadlines share GPUs. That covers choosing between MIG, MPS, time-slicing, priority or dedicated pools for each workload class, configuring Kubernetes or Slurm to enforce it, and folding the result into fleet sizing. The work starts by characterising your workloads and SLAs, so the mechanism follows from measurements rather than defaults. For single-GPU contention, cold start and placement, the offer is GPU Performance Engineering. For LLM serving capacity, it is the Inference Cost-Cut Pack. Building in-house makes sense when your platform team already has profiling capacity and time on its hands. A specialist makes sense when an SLA or launch date will not wait for that learning curve. When choosing a partner, ask how they will measure worst-case behaviour on your hardware.

This is not a fit if you want hours billed against a ticket queue or the cheapest possible build. TechnoLynx supplies the engineering evidence, not certification or regulatory approval. For simple cases, a single workload class or a well-documented MIG layout on GKE, NVIDIA’s and your cloud provider’s documentation, or your scheduler vendor’s support, may be all you need.

FAQ

Does Kubernetes use GPU?

Yes. Kubernetes has stable support for AMD and NVIDIA GPUs through device plugins, and containers request them as a custom resource in the limits section.

What does GPU utilization mean?

It depends on the metric. SM activity (SMACT) is the fraction of time SMs are active, and SM occupancy (SMOCC) is how much parallelism each SM holds relative to its maximum. Together they show more than a single “GPU busy” percentage.

What is GPU scheduling?

GPU scheduling decides which process’s work runs on the GPU and when. At driver level, each process has its own GPU context, and contexts from different processes are time-sliced to share the hardware.