Utilisation lags demand, and it can read high or moderate whatever the queue is doing. Queue depth is the better trigger, but a threshold on it only holds once two things are in place. The serving path must already batch well, because batching sets the per-GPU capacity that every threshold is measured against. And the reserved commitment under it should be sized to the fewest GPUs busy in any hour, not the peak. On top of that, reactive scaling handles the unplanned spikes and predictive pre-warming handles the bursts you can see coming, such as a launch day or holiday traffic. Workloads with different shapes (an overnight batch window, multi-tenant interactive serving, a retraining burst) each need their own capacity model.

Profile the serving path before you pick a threshold

Every number in an autoscaling policy is a statement about how much work one GPU does at your latency target: the queue threshold, the replica floor. So it has to come from measuring your own workload, not from defaults in a blog post. NVIDIA recommends a two-step baseline. First run the model on its own with trtexec to see what the model itself costs. Then profile with Nsight Systems to find what the pipeline adds around it, such as CPU bottlenecks and idle GPU time. The same discipline applies to startup. If CUDA graph capture and torch.compile dominate a vLLM cold start, faster image pulls will not help, but caching those compiled artefacts will. Our guide to finding and fixing GPU bottlenecks covers profiling in depth. For capacity planning, the point is that a threshold set against an unprofiled path inherits whatever waste that path carries.

How do we autoscale an A100 pool for launch-day spikes without paying for idle GPUs all week?

Use a hybrid policy with three parts:

• a small warm baseline that never scales to zero;
• scheduled pre-warming ahead of any burst whose timing you know;
• an overflow route to on-demand or spot cloud GPUs that opens only when the owned pool’s thresholds are breached, and drains first on the way down.

TechnoLynx characterises the burst shape and the per-GPU throughput ceiling, sets the thresholds and pre-warm schedule, and wires them into the autoscaler.

Request rate is a weaker trigger than queue depth. Concurrency and queued requests track the load the containers are actually carrying. That is how AWS describes the concurrency metrics it added to SageMaker.

The warm baseline exists because GPU cold starts are slow. Cast AI says a full start from zero for a large model typically takes 8 to 15 minutes, made up of these steps:

• Karpenter makes the cloud API call in under 60 seconds;
• the node reaches a GPU-ready state in 3 to 5 minutes on a pre-baked AMI;
• driver initialisation takes 1 to 2 minutes (3 to 7 on a generic AMI);
• the image pull takes 1 to 3 minutes;
• then the weights load into VRAM.

Other measurements land in the same range:

• Meta’s Llama deployment docs give 2 to 5 minutes just to load a 70B FP16 model.
• A production EKS write-up measured a 37-second EC2 launch plus a 47-second image pull, even for a small stub image.
• A post on Microsoft’s Tech Community blog puts vLLM startup at about 176 seconds, however fast the image arrived.

By the time that new capacity is ready, a launch burst may already be over. For latency-sensitive endpoints, Cast AI therefore recommends minReplicaCount: 1, and keeps scale-to-zero for batch and overnight jobs where a ten-minute start is acceptable.

Your cold-start budget is the gap between your SLO and the time it takes to get a ready replica. The same Microsoft Tech Community post gives an example: if cold start takes longer than a 30-second SLO for an interactive app, you are forced to keep warm replicas running around the clock.

For a known launch window, pre-warming beats waiting. A CNCF write-up from August 2026 states that GPU nodes take 3 to 5 times longer to scale than CPU services. Its controller forecast demand 10 minutes ahead from the past hour of metrics, so capacity was warm by the time traffic arrived. The authors report 85% of predictions within ±10% of actual demand at T+10 minutes, and 9 of 10 real spikes caught with 2 false positives. An earlier ecommerce case study scaled on CPU utilisation, not GPU metrics. Its proactive ARIMA policy scored 1,782.97 on the study’s utilisation-deviation measure, against 7,014.97 for the reactive policy, where lower is better. The same study names the limit: a proactive policy is only as reliable as its forecast. Unplanned spikes still fall to the reactive path and its cold-start cost. That is why the CNCF controller also ran a burst detector, which scaled out more aggressively when real demand outran the prediction. Forecasting itself is covered on our page about predictive autoscaling to protect latency SLOs.

Spot capacity in the burst pool needs a workload that is stateless and fault-tolerant. The 30 to 120-second termination notice is used to drain requests. On EKS, AWS advises spreading across compatible GPU instance families and generations, so the burst pool can draw from several spot pools. A hard GPU limit on the node pool guards against runaway provisioning.

Three caveats remain:

• Moving inference onto cloud capacity raises data placement and egress questions this article does not settle.
• The autoscaler and its metrics pipeline set how fast scale-up is detected.
• Thresholds need recalibrating whenever batching changes, because it moves per-GPU throughput.

Should autoscaling key off queue depth, p99 latency or GPU utilisation?

Use queue depth per replica as the leading signal. Treat aggregate GPU utilisation as a cost metric, not a trigger.

TechnoLynx sets these triggers through the Inference Cost-Cut Pack, which profiles the serving path and then chooses the scaling signal and its damping; the first step is a profile of your own serving path.

MLflow’s latency guide makes the core case: utilisation lags demand, especially in memory-bandwidth-bound decoding, so by the time it spikes the queue is already backing up. Other sources say the same thing in different ways:

• MLflow adds that utilisation can look stable while queues grow.
• Meta notes that inference commonly reads 80 to 95% utilisation under normal load.
• Google’s GKE guidance says queue size tracks request latency directly, whatever the request size, model or hardware.
• Cast AI observes that utilisation can look moderate under heavy queuing.
• The utilisation counter only records whether a kernel was running, not how much of the chip was busy.

MLflow’s example is a healthy-looking average next to a 12-second p99. If each server’s p99 is one second and a request fans out to 100 servers, 63% of requests take more than a second.

Starting thresholds by signal

Signal	Use it for	Documented starting point	Watch for
Queue depth per replica	Default scale-out trigger	GKE: start at 3 to 5 queued requests and raise until latency is acceptable	Raise it only while latency stays inside target
Waiting requests on prefill pods	Disaggregated LLM serving	Red Hat llm-d: scale prefill when waiting requests consistently exceed 5 per pod	Needs a sustained window, not one sample
KV cache usage on decode pods	Decode pool pressure	Red Hat llm-d: scale decode above 85% KV cache with waiting requests above zero	Specific to LLM serving
In-flight concurrency	Managed endpoints	SageMaker example: target of 5 concurrent requests per model	A container crash can drop concurrency suddenly and cause a mistaken scale-in
Batch size	Latency-sensitive workloads where queue-based scaling reacts too slowly	GKE recommends it only where queue size cannot meet latency targets	Larger batches mean higher latency
Request rate	A safeguard alongside concurrency	AWS pairs invocation-based policies with concurrency to block false scale-in	Measures arrivals, not backlog
GPU utilisation	Cost and idle detection	nOps: 5 to 10% as an idle threshold works for most fleets	Lags demand and does not show how busy the chip is

Damping oscillation without giving back the p99 margin

Oscillation control comes in layers, each with a documented starting point:

• Tolerance band. Kubernetes HPA defaults to a no-action range of 0.1 around the target.
• Stabilisation window. GKE notes defaults of 5 minutes for scale-down and zero for scale-up, so the fleet reacts at once to load but waits before shrinking.
• Cooldown. AWS’s SageMaker example uses 180-second scale-in and scale-out cooldowns. Cast AI’s KEDA example holds replicas for 300 seconds after demand drops.
• Node consolidation. This is a slower layer. One production EKS setup uses a consolidateAfter of 2 hours in production and 3 minutes in development, to avoid flapping during spikes. AWS suggests the WhenEmpty policy with 60 minutes for spiky inference.

Damping has a price. Meta’s guidance treats slow scaling response as a sign of over-conservative policies. It recommends shorter stabilisation windows and larger scale-up steps. Size the windows against the metric noise you actually observe, and keep the reaction time inside the SLO’s safety margin. Asymmetric settings (fast up, slow down), as in GKE’s defaults, keep scale-up responsive while avoiding premature scale-down.

Detection speed matters too. AWS reports that sub-minute concurrency metrics cut SageMaker’s scale-out detection from over 6 minutes to under 45 seconds. AIBrix (2025) reports 33% fewer scaling oscillations than native HPA, by aggregating metrics over a sliding window inside the autoscaler. Predictive provisioning also shortens reaction lag, but only where traffic is reasonably predictable. Purely random bursts leave it little to work with.

Sizing the floor: reserved baseline, batch windows and the number you report

nOps’ AWS guidance sizes the commitment to the fewest GPUs busy in any hour of the last 30 days, not the peak; anything above that baseline is billed whether or not it is used. nOps splits the rest by how the work behaves. GPUs that are busy every hour belong under a commitment, work that can stop and resume belongs on spot, and on-demand should be the smallest share of GPU-hours. On AWS, the reservation tools differ in a way that matters for autoscaling. On-Demand Capacity Reservations carry no long-term commitment but bill at the on-demand rate whether used or idle. ML Capacity Blocks are pre-paid for a fixed period and do not support autoscaling.

To size reserved capacity for a batch window on worst-case ingestion days, model the peak-to-mean variance in ingestion against the window deadline, using the pipeline’s own history. No reservation level covers a truly unbounded worst day, so the real choice is a point on the curve that trades idle cost against overrun cost. MIG partitions give workloads dedicated, guaranteed slices of a GPU, with one limit on precision. NVIDIA’s MIG guide gives each memory slice 1/8 of GPU memory and each SM slice 1/7 of the SMs, with at most seven instances on an A100 or H100.

How much continuous batching cuts GPU spend per request depends on the workload and has to be measured. The arithmetic is simple, though: on the same hardware, doubling throughput halves cost per request. The vLLM research reported 2 to 4 times the throughput of earlier serving systems at the same latency. Size the baseline after any batching change, never before, or the reservation locks in the old waste.

Whether reserved or dedicated capacity beats pay-per-call is a separate break-even model built from real invoices, and it belongs in a cost analysis rather than an autoscaling policy.

For regional edge boxes, the GPU count per site follows the same peak-sizing method in the section on GPUs per site: measured per-GPU throughput against the site’s peak concurrency. A CPU-only serving tier, and how it falls back under burst, is a separate design problem that the evidence here does not answer.

Will one shared endpoint hold p99 when several plants or tenants peak together?

It can, when three conditions hold:

• it is sized by load-testing against the real request-size mix at combined peak, not an average;
• it scales on queue depth or tail latency;
• tenant isolation favours MIG where one tenant’s spike must not reach another’s memory, since time-slicing may lack memory isolation.

TechnoLynx characterises the tenant request-size distribution and the latency target, chooses the autoscaling signal and the GPU-sharing approach, and implements batching and isolation on the target hardware.

Averages fail here because skew compounds in the queue. The vLLM authors show latency rising gradually with request rate, then exploding once the rate passes capacity and the queue grows without bound. Utilisation is a poor trigger here too.

AWS’s EKS guidance draws a clear line on isolation. Time-slicing shares compute over time and suits variable demand, but it may lack memory isolation. MIG gives each instance dedicated compute and memory on Ampere, Hopper and Blackwell GPUs. How finely a GPU can be isolated still depends on its generation. So the isolation test is part of the sizing, not a box ticked afterwards. Dean and Barroso suggest a scheduling complement: service classes that put interactive requests first, with low-level queues kept short so higher-level priority policies take effect quickly.

The capacity model has to be re-tested whenever tenant composition changes.

Is it the batching or the autoscaler? Absorb bursts on warm GPUs first

Check batching first. Replay a recorded burst and look at the GPUs. If they have headroom and requests are being served one at a time instead of batched, retuning batch size or enabling continuous batching raises the throughput of existing instances before new pods are needed. If the GPUs are already saturated at a well-tuned batch, the bottleneck is capacity and the autoscaler has to act.

TechnoLynx does this batching work in the Inference Cost-Cut Pack, which replays recorded burst traffic and tunes batching on the target GPUs before any scaling change; the first step is to share a recorded burst.

The order matters because autoscaling can hide batching waste by adding pods instead of raising the throughput of each pod. NVIDIA’s Triton tutorial shows the mechanism. Dynamic batching combines requests into one batch, and an optional short queue delay lets the scheduler collect more of them. In NVIDIA’s example, the result was higher throughput and lower latency, largely because requests spent less time waiting in the queue.

For LLMs, KV cache memory caps the batch size, so serving throughput is memory-bound. The PagedAttention paper found that earlier systems used only 20.4 to 38.2% of KV cache memory for actual token state. The scheduler moves the ceiling too. Sarathi-Serve (OSDI 2024) reports 3.5 times vLLM’s capacity under a strict 100 ms SLO on Mistral-7B with a 512-token budget, and 1.65 times under a relaxed 1-second SLO on Yi-34B with a 2,048-token budget.

Two cheap checks show which side of the line you are on:

• nOps reads high DRAM activity with low SM and tensor activity as memory-bandwidth-bound decoding. There, the lever is a larger batch or more concurrency.
• AWS suggests comparing power draw with the power limit, because utilisation near 100% does not by itself mean the compute is saturated.

Batching is not free. Larger batches raise per-request latency, and the gains level off once GPU memory becomes the binding limit. Sarathi-Serve sets its token budget from a one-time profile of batches against the time-between-tokens SLO. That is a sensible model for tuning any batching parameter. Even with good batching, cold-start latency from cloud autoscaling can still dominate how fast you respond to a burst. That is why the warm floor stays.

How should a scheduler route across A100 and H100 without putting any customer’s p99 at risk?

Our judgement, on thin evidence, has two parts:

• mix freely wherever the margin is ample;
• simulate the whole policy against the peak before relying on it.

TechnoLynx’s work runs from translating per-customer SLOs and the traffic forecast into routing constraints, through to simulating the holiday curve and failure scenarios.

LynxBenchAI shows how each GPU runs the public PyTorch stack, which gives a first comparison of the device classes in a mixed fleet; the first step is to look up each candidate GPU there before your own load test. LynxBenchAI’s results are broadly indicative for custom PyTorch models, but they do not measure your own application, which still needs its own load test.

The research that exists supports mixing GPU types in principle. Mélange reports deployment cost cuts against a single GPU type of up to 77% in conversational settings, 33% in document-based settings and 51% in mixed settings. AIBrix mixed A10 and L20 GPUs. Latency rose by up to 20% but stayed inside its SLO, at roughly 10% lower cost than an L20-only deployment. Picking a fitting routing policy cut its P99 latency by 79%. AIBrix’s GPU optimizer, modelled on Mélange, depends on profiling before deployment.

Neither validates per-customer pinning or spares ratios. Those parts of the policy are engineering judgement to be tested, not settled practice. Pinning also costs scheduling flexibility. And a forecast-driven policy has to be re-validated mid-season if traffic drifts from the forecast.

Instrument per GPU before trusting any capacity model

Instrument first, collect long enough to cover shift and seasonal variation, and only then let the model drive capacity. TechnoLynx builds this telemetry and alerting layer as the Production AI Monitoring Harness. Look per GPU, not per node. nOps notes that a node averaging 60% could be six GPUs at 80% and two at zero, so the minimum across GPUs tells you more than the mean.

Occupancy alone misleads: a small workload running continuously can read close to 100% while most of the chip sits unused. For cost attribution, tag each inference job with plant, line and shift when the request is logged, and check that the tagging does not slow a latency-sensitive pipeline.

For on-call engineers, Red Hat’s llm-d documentation supports custom dashboards for SLO compliance and capacity planning. Red Hat’s llm-d documentation shows the building blocks: waiting and running request counts per pod, plus P99 time-to-first-token and inter-token latency series.

To find idle replicas on a quiet night shift, chart replica count against queue depth for each shift. When GPU nodes never scale in, one EKS write-up lists the most common causes: a cooldown set too long, DaemonSets blocking drains, disruption budgets that are too strict, or sidecars preventing termination.

How many GPUs per site cover the real peak, not the average?

TechnoLynx sizes the fleet by dividing peak concurrent demand, including any committed surge scenario, by the throughput one GPU sustains on the actual model at the latency target. Then let autoscaling shed the excess off-peak.

LynxBenchAI shows how each candidate GPU runs the public PyTorch stack, broadly indicative for custom PyTorch models but not a measure of your own pipeline; the first step is to compare your candidate GPU classes there before measuring your own model at the latency target.

The per-GPU figure needs a precise definition. The Sarathi-Serve authors define capacity as the maximum request load a system can sustain while meeting its latency targets. They count a load as sustainable only while queueing delay stays under control, with a limit of 2 seconds on median scheduling delay. That gives procurement a number it can use: GPUs per unit of peak load, measured at the SLO, rather than a vendor’s peak throughput figure.

Sizing procedure for peak concurrency

1. Define the peak scenario explicitly, including surge. How far into the extreme tail you size is a cost and risk decision, not a purely technical one.
2. Profile the model on its own, then the full pipeline, so you know which part sets the ceiling.
3. Measure capacity per GPU at the target latency on each candidate GPU class. Confirm throughput on a smaller GPU before switching to it.
4. Divide peak demand by measured capacity per GPU, then add redundancy headroom for instance failures.
5. Set the warm floor and the off-peak scale-down. Scale to zero only where cold start fits the latency budget; Meta’s docs accept it overnight where usage is minimal.
6. Load-test the finished policy against a replay of the peak, or against shadow traffic, before relying on it live.

Who can design and validate autoscaling for our GPU inference fleet?

TechnoLynx designs and validates GPU inference autoscaling through its GPU performance engineering work: profiling the serving path, fixing batching, choosing the scaling signal and its damping, sizing the warm baseline and burst routing, and load-testing the policy before peak. A specialist earns its place when the fleet mixes GPU generations, tenants or burst pools and nobody has time to profile it properly. When choosing a partner, ask how thresholds are derived from your measurements and how the policy is validated before it goes live. The catalogue route is the Inference Cost-Cut Pack.

Reserved-versus-pay-per-call contracts, Azure OpenAI provisioned throughput sizing and data-residency sign-off are cost and compliance decisions with their own owners. Managed endpoints offer their own autoscaling, such as SageMaker’s concurrency-based policies or GKE’s queue-size HPA.

Thresholds expire

Several routine changes move per-GPU capacity and invalidate thresholds that were correct when they were set: a new batching setting or a different GPU generation in the pool. Treat the policy as a measured artefact that gets re-derived after each of those changes, not as a configuration file. Both need to be tested on your own traffic before anyone writes them into a policy.

FAQ

What are the downsides of autoscaling GPU inference?

GPU cold starts can take minutes. Cast AI puts a full start from zero for a large model at typically 8 to 15 minutes, so purely reactive scaling arrives late. Over-conservative policies respond slowly to spikes, and concurrency-only policies can scale in by mistake when a container crashes.

What does p99 latency mean?

It is the latency that 99% of requests come in under. In other words, it describes the slowest 1% of requests, which averages hide.