A cost-safe autoscaling policy for GPU inference adds capacity when a leading signal moves (queue depth, batch size or p99 time-to-first-token, not GPU utilisation). It adds that capacity quickly and gives it back slowly, in bounded steps, only once demand has stayed low. It lets in-flight work finish within the termination grace period before a pod goes, and it sizes warm capacity against a measured cold-start baseline. The policy is a set of tuned parameters (stabilisation windows, scale-down rate caps, tolerance), scheduled and predictive pre-scaling, and a test under realistic load before the policy goes near production. Most of this article covers Kubernetes autoscaling with the Horizontal Pod Autoscaler (HPA) and the tools around it.

How do you scale down after a spike without flapping or stranding requests?

Scale down slowly and in steps, only after demand has stayed low for a sustained window, and remove a GPU pod only after its in-flight requests have finished. By default the HPA scales up immediately, with no stabilisation window. Scale-down should be gated by a cooldown and capped in how many replicas it can remove at once.

The HPA’s asymmetric defaults are the baseline, not the answer

The Kubernetes HPA v2 API already ships asymmetric behaviour: the stabilisation window defaults to 0 seconds for scale-up and 300 seconds for scale-down. When metrics call for more replicas, the target scales up immediately, adding at most 4 pods or doubling the current replicas per policy period. After the 300-second window, a single policy allows 100% of running replicas to be removed, so the target can drop straight to its minimum. The window delays the decision but does not limit its size. For a GPU fleet that means a whole burst’s worth of capacity can vanish in one step.

Scaling policies limit the rate at which replicas are removed. The Kubernetes documentation gives the pattern: one policy allowing at most 4 replicas to be removed per minute and another allowing at most 10% per minute, with the larger change selected by default. Add a tolerance so small fluctuations around the target trigger nothing. The cluster-wide default is 10%. The HPA control loop runs every 15 seconds by default, so these settings act on a short, regular cadence rather than continuously. The same idea appears in research as a Resource Removal Strategy, which removes only a fraction of the excess capacity each cycle. That design also imposes a minimum delay between consecutive corrective actions to stop the controller oscillating.

Why flapping happens and how to stop it

Flapping is a loop of opposing scale events, where one scale event triggers its opposite. Microsoft’s Azure Monitor documentation names the usual causes: small or no margins between thresholds, scaling by more than one instance at a time, and scaling in and out on different metrics. Its worked example has a scale-out threshold of 80% CPU and a scale-in threshold of 60%. Removing an instance at 60% load would push the remaining instances to 90% and force an immediate scale-out, so the engine skips the scale-in. Azure checks for possible flapping only before scale-in, never before scale-out, so that capacity is always available. That asymmetry is the right instinct for GPU serving, where a missed scale-out costs latency and an early scale-in costs a cold start.

How long to hold depends on the burst shape. If bursts are short and follow a predictable decay (a launch, a broadcast), a time-boxed scale-down delay tied to that decay is the simpler choice. If bursts are unpredictable, a metric-based cooldown on queue depth or latency, with a minimum hold time before any GPU is released, is safer. Too aggressive and the fleet thrashes, which wastes GPU hours and causes latency spikes. Too conservative and idle GPUs sit there after the spike, inflating cost.

Node-level release is handled separately, by the Cluster Autoscaler. By default, the Cluster Autoscaler waits until a node has been unneeded for 10 minutes before removing it. It also waits 10 minutes after a scale-up before it resumes scale-down evaluation. On accelerator nodes it treats a node as a scale-down candidate when GPU requests fall below 0.5 of allocatable. Tune pod and node scale-down together. A slow pod policy on top of a fast node policy, or the reverse, produces surprises.

Draining in-flight and streaming requests

Scale-down has to know about requests, not just metrics. When Kubernetes terminates a pod, it marks the pod Terminating and removes it from Service endpoints, so no new traffic arrives. It then runs the preStop hook, sends SIGTERM, and waits out a termination grace period that defaults to 30 seconds. If shutdown usually takes longer than 30 seconds, the grace period should be raised with terminationGracePeriodSeconds. Streaming LLM responses and long generations are the requests most at risk from that default. For those, the preStop hook should stop admission and wait for the server’s active-request count to reach zero, and the grace period should cover the longest generation the service allows.

The scale-up ceiling is set by GPU supply, not by demand

Set the HPA’s maximum replicas to the GPUs you can actually get: quota, reserved capacity and realistic on-demand or spot availability. Theoretical peak demand is the wrong number. On AWS, an Auto Scaling group that consumes Capacity Reservations without On-Demand fallback launches only into those reservations, and it might not reach its desired capacity if reserved capacity runs short. With fallback enabled, instances launch as On-Demand when no reserved capacity is left. Warm pools are bounded too: if the pool is depleted or an Availability Zone is out of capacity, a scale-out falls back to a cold start. A SaasDash.ai capacity-planning guide makes the GPU-specific point that GPU instances may have availability constraints and that setup time runs to tens of minutes.

Design for that limit openly. Azure’s guidance is to throttle demand while capacity is limited, and an MLflow guide recommends a maximum queue-depth threshold that returns HTTP 429 with a Retry-After header instead of letting the queue grow without bound. A known, controlled degradation at the ceiling is better than an autoscaler that keeps asking for GPUs that are not coming.

How do you combine scheduled and predictive scaling so pre-scaling matches expected demand?

Use scheduled actions for events you know about, predictive scaling for recurring daily and weekly patterns, and reactive scaling to correct forecast error. Let the largest recommendation win. Microsoft’s architecture guidance says the same in vendor terms: schedule instances to start before a busy period, and include metric-based rules in that scheduled profile to handle unexpected demand. AWS, Google and Kubernetes autoscalers resolve competing signals by taking the maximum. On EC2 Auto Scaling, each active policy computes desired capacity independently and the group uses the highest. Google Compute Engine picks whichever signal, schedule or metric, yields the most VMs. In Kubernetes, a KEDA cron trigger’s desiredReplicas therefore acts as a dynamic minimum, because the HPA evaluates all metrics and takes the one that needs the most instances.

Predictive scaling has limits that matter for event-driven traffic. AWS predictive scaling scales out when the forecast rises but does not scale in when it falls, and it suits workloads with recurring patterns by day of week or time of day. It needs at least 24 hours of data and works better with two full weeks. Google’s predictive autoscaler needs at least 3 days of history. For one-off events such as a marketing campaign that the forecast underestimates, AWS documents scheduled actions that raise minimum capacity above the forecast, with a second action restoring it afterwards. On Google Compute Engine a real-time signal that exceeds the prediction takes priority, and short, unpredictable peaks are covered reactively rather than in advance.

Lead time is the parameter that makes pre-scaling useful. Google’s initialisation-period setting controls how far ahead of predicted load the autoscaler starts new instances, and AWS offers a pre-launch buffer that starts instances ahead of the forecast hour. The forecast horizon has to cover provisioning, joining the cluster and warm-up together, and one practitioner guide’s example schedules the action 10 minutes ahead for instances taking 5 minutes to boot and pass health checks. For GPU inference the warm-up includes model load, which the next section measures. Another practitioner guide recommends stepping towards the forecast peak in increments, re-checking live signals between steps, so an overshooting forecast wastes less. It also recommends never letting the forecast be the only trigger: pair it with live metrics and guardrails such as a maximum step size and a minimum warm-up time. Choosing and validating the forecasting model belongs to predictive autoscaling: forecasting demand to protect latency SLOs, which covers forecasting in depth.

Sizing pre-scaling per event instead of a flat buffer

A flat buffer applied to every event over-provisions the quiet ones and may still under-provision the big ones. Sizing per event by expected audience fixes both. One way is to tier upcoming events from the traffic of comparable past events plus what is known in advance (competition stage, teams, time slot, marketing push). Only top-tier events get a large scheduled floor. Mid and low tiers should rely on predictive plus reactive scaling, with the tier thresholds revisited as each event adds data. This tiering is a judgement-driven practice rather than a published, measured method, so treat the first season’s thresholds as hypotheses to check against what actually happened.

Three caveats limit it. A newly scheduled or unusual event can be put in the wrong tier, and an under-tiered marquee event gets only reactive scaling, which has ramp-up latency of its own. Tiering also needs enough history per event type to separate tiers at all. With a handful of past events, a generous scheduled floor for anything plausibly large is the honest choice. Models trained only on normal weeks tend to fail during promotions and launches, so include the noisy periods in any evaluation.

How much cold-start latency can a warm pool of standby GPU workers remove?

A warm pool takes model load time off the critical path of a burst, and the price is the idle cost of the standby capacity. TechnoLynx sizes warm pools and predictive triggers against a team’s own large-model traffic traces and measured load times through the Inference Cost-Cut Pack. The work starts by framing the burst and cold-start problem against your SLA targets, then models pool size and trigger thresholds before anything is changed in production.

The reactive path is slow. When the HPA is combined with the Cluster Autoscaler, the Cluster Autoscaler FAQ puts the total from rising load to new pods running at usually about 5 minutes, including 3 to 4 minutes of node provisioning on GCE. That is before the model loads. A Cast AI guide puts a full cold start from zero at 8 to 15 minutes for large models (node boot, driver initialisation, image pull and model load) against 30 to 60 seconds for a pod-only restart. For LLaMA-2-70B, one survey reports a 130 GB checkpoint taking at least 26 seconds to download over a 5 GB/s network, and another 84 seconds to load onto 8 GPUs. AWS measured 80 to 460 seconds to move 60 to 200 GiB of weights from S3 to GPU memory on p5.48xlarge instances with default Run:ai Model Streamer settings. Measure the number for your own model, precision and storage path before sizing anything. MLflow’s guidance is to record how long a cold pod takes to serve its first token under realistic load.

Warm pools, standby buffers and what they cost

Warm capacity comes in grades, and cost falls as readiness falls. In AWS’s own example, a warm pool of pre-initialised EC2 instances cut launch time from 243 seconds to 36 seconds. Keeping pool instances stopped means paying only for volumes and attached Elastic IPs. If the pool is empty when a scale-out arrives, instances launch cold. GKE capacity buffers make the same trade explicit: standby nodes resume in typically 30 to 35 seconds against 60 to 120 seconds for creating a new node, and Google recommends a small active buffer to absorb the first spike plus a larger standby buffer for sustained growth. At the pod level, the Cast AI guide advises minReplicaCount: 1 for latency-sensitive endpoints, keeping scale-to-zero for batch and endpoints where a roughly 10-minute startup is acceptable.

These are VM- and node-level measurements, and a ready node is not a ready model. A 2024 survey of serverless LLM serving notes that keeping GPUs warm works for smaller models like BERT but falls short for LLMs because of the GPU load they require. Torpor, a system piloted in Alibaba Cloud, keeps idle models in host memory and swaps them to GPUs on request, which cut Llama2-13B startup to 4.4 seconds, over an order of magnitude below cold starts there. Under Alibaba’s billing scheme, users paid 10% of the GPU cost while a function was inactive and held in host memory. Holding weights in host memory is a middle grade worth considering when a full GPU warm pool is too expensive.

The decision depends on burst shape and budget. If bursts are frequent or predictable from leading signals, use predictive scaling to grow the pool ahead of demand. If they are sporadic and the budget tolerates idle capacity, keep a fixed pool sized to the worst burst.

Cutting model load time on a fresh instance for 34B and 70B tiers

What dominates load time depends on model size, so measure the phases before choosing a fix. In AWS’s EKS measurements, torch.compile took about 53 seconds (65% of startup) for a 64 GiB model, while weights loading took about 423 seconds (92% of startup) for a 203 GiB model. The crossover was near 100 GiB, because compile time tracks architecture and weights loading scales with file size. In 16-bit weights a 34B model is roughly 68 GB (34 billion parameters × 2 bytes) and a 70B model roughly 140 GB, so, by file size alone, the two tiers fall on opposite sides of that line.

For the compile-bound tier, caching torch.compile artefacts removes recompilation on later pod starts. Two changes needing no code modifications cut a 64 GiB model’s startup with Run:ai Model Streamer from 82 seconds to 65 seconds on first launch and to 16 seconds for later launches on the same node. For the weight-bound tier, faster weight loading gives the larger reduction. NVIDIA reports Run:ai Model Streamer loading a 15 GB Llama 3 8B model in 47.56 seconds at concurrency 1 and 14.34 seconds at concurrency 16 on GP3 SSD. ServerlessLLM’s multi-tier checkpoint loading is reported as 8.2 times faster than PyTorch and SafeTensors for LLaMA-2-70B. Pre-pulling images with a DaemonSet, using GPU images with drivers already installed and caching weights on a shared volume remove the steps before the load itself.

LynxBenchAI’s published results show how each device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models when you choose the device for a warm pool. They do not measure your serving workload or your load times.

Where KV-cache-aware routing helps, and where it does not

KV-cache-aware routing and prefix caching help only when requests reuse contexts. Multi-turn sessions, shared system prompts and long contexts queried repeatedly benefit. When reuse exists, the gains are real: the Cake system cut time-to-first-token by 2.6 times on average in long-context prefix-caching scenarios by loading cached KV state and computing it in parallel. Where the cache lives matters. AttentionStore found about 80% of cache hits at the disk level, where low I/O bandwidth constrains time-to-first-token. Treat it as a layer on top of warm capacity, never as a replacement for it.

How low can replicas go in known quiet periods without breaching the SLA?

Replicas can drop to a calendar-aware floor during known low-volume shifts, but that floor has to come from the measured worst-case ramp-up when volume returns, not from a round number. TechnoLynx designs these calendar-aware policies, blended with reactive queue-depth triggers, and validates them by replaying historical worst-case traces before rollout. The engagement starts by pinning down the SLA, the shift calendar and the worst-case load scenarios to test against.

Calendar floors and scale to zero

A KEDA cron trigger scales from the minimum to the desired replica count inside a time window. To scale to zero outside working hours, set minReplicaCount: 0 and raise replicas during working hours. With KEDA’s default 5-minute cooldownPeriod, the actual scale-down happens 5 minutes after the window ends. The Cast AI guide recommends KEDA over the plain HPA when you need minReplicaCount: 0 or queue-based signals. Combine the cron trigger with a queue-depth trigger so the floor sets the minimum and live demand can still push capacity higher.

The safe floor follows from one rule in Microsoft’s guidance: keep enough spare capacity to absorb demand while new instances start. If startup latency would breach response-time targets, keep instances warm rather than scaling from zero. In practice the floor has to serve the steepest volume increase observed at a shift boundary for as long as a new replica takes to become ready. Take the measured arrival growth over that ready time, divide by per-replica throughput at the SLA, and that is the minimum. Revisit it when shift patterns, customer mix or available GPU generations change, because a floor tuned on last quarter’s transitions can quietly stop holding.

How do you validate an autoscaling policy before production: replay peak traces or simulate load?

Replay historical worst-case traffic against the candidate policy in an environment whose contention resembles production, and accept the policy only if it meets the SLA throughout. Microsoft’s guidance is to validate an autoscaling strategy under realistic load, starting with a load test that checks performance targets as demand changes. Synthetic tests are fine for parts of this. MLflow recommends a cold-start surge test: scale the pool to zero or its minimum, inject a sudden spike and measure time-to-first-token for the first 100 requests. Its go/no-go bar includes an autoscaler that stays stable for 24 hours under variable load without thrashing.

A staging slice of the shared cluster is the stronger environment when co-tenant interference matters and the capacity exists. Synthetic contention is the fallback, and its results should be treated as lower-confidence evidence. One survey of predictive autoscaling puts the gap plainly: most autoscaling studies run on minikube testbeds or synthetic traces, while production clusters have noisy telemetry, evictions and varying node configurations.

A replay run that produces usable evidence follows a fixed order:

1. Select the traces: the worst historical peaks, the steepest ramps and the sharpest shift transitions, with real request sizes.
2. Freeze the candidate policy as versioned configuration (HPA or KEDA objects, probes, grace periods, cluster autoscaler flags).
3. Start from the state production would be in: the floor, a depleted warm pool, cold nodes.
4. Replay at recorded timing and record every scaling decision, its trigger, the requested change and its completion time. Microsoft recommends correlating exactly these events with application performance.
5. Score SLA compliance per interval at the percentile the SLA names, plus flapping events and GPU-hours used.

That record (configuration version, traces, per-interval results, scaling-event log) is the engineering content we would put in an evidence package for a quality or regulatory reviewer. Whether it is sufficient for sign-off is that reviewer’s call, not the engineer’s. It also proves only what was tested. A future peak more extreme than anything replayed is still untested.

Which signal should trigger GPU inference scale-out: queue depth, TTFT, GPU memory or request rate?

Trigger scale-out on queue depth or on time-to-first-token (predicted, or measured at p99), not on GPU utilisation, CPU or memory.

TechnoLynx tunes scale-out triggers for GPU inference fleets through GPU Performance Engineering.

GPU utilisation is a poor trigger for LLM serving. The llm-d documentation notes that utilisation is often pegged near 100% during active batching regardless of real load, and that resource metrics show saturation only after latency has already spiked. The Cast AI guide observes the opposite symptom, with utilisation looking moderate under heavy queuing. The two sources describe different behaviour, but both reject utilisation as the scaling signal in favour of queue-based signals. Google’s GKE guidance recommends queue size when optimising throughput and cost, noting that queue time adds to latency and queue size is independent of request size, model and hardware. It suggests starting with a threshold of 3 to 5 and raising it until latency is acceptable. For latency-sensitive services where queue-based scaling is too slow, it recommends batch size instead.

In one measured study, a model-driven latency controller missed fewer latency targets than a throughput proxy. In that arXiv study on an OpenShift cluster of H100 GPUs, a queueing-model controller tracking a fourfold load ramp missed a latency target in 7 of 127 control cycles. A decode-throughput analyser missed 27 of 128 cycles under the same load, while provisioning 4% and 28% fewer replicas. The same paper notes that a threshold on measured latency reacts only after degradation and does not say how much capacity to add. The HPA can consume these signals through custom metrics in the v2 API, and through external metrics such as the length of a queue outside the cluster.

Scale-out signals compared

Signal	What it tells you	Where it misleads
GPU utilisation	Saturation, only once it has occurred	Pegged near 100% under batching regardless of load
CPU or memory	Host-side pressure	Lagging; the decode pool has queued for seconds by the time CPU spikes
Request rate	Arrival volume	Ignores prompt and output length, so the same rate means different GPU work
Queue depth	Unmet demand; adds directly to latency	Stays low while batch space remains, then rises at the growth point
Batch size	Concurrent work and its latency cost	Needs the model server’s maximum batch size measured first
In-flight requests	Load per replica without extra plumbing	The default signal in llm-d’s KEDA path; less direct than an SLO signal
p99 TTFT	The user-facing SLO itself	Measured latency reacts after degradation unless predicted
KV-cache utilisation	Decode memory pressure	Same value maps to different TTFT as prompt mix changes

How do you configure HPA so new GPU pods load their checkpoint before they receive traffic?

Gate traffic with probes, not with the autoscaler. A pod with a readiness probe starts without receiving traffic and gets traffic only once the probe succeeds. Readiness probes exist for exactly this kind of slow initial work, such as loading files and warming caches. A startup probe holds off liveness and readiness checks until initialisation completes, giving a slow-loading model server time to finish. The vLLM documentation’s fix for startup-probe failures is to raise failureThreshold, sized by removing the probes and measuring how long the server takes to report ready. MLflow recommends loading weights into GPU memory before the pod is marked ready, which lengthens startup but removes the first-request penalty. Point the readiness endpoint at “model loaded and a warm-up request served”, not at “process is listening”.

How many checkpoints fit resident on one GPU, for example SDXL-style checkpoints on an A10G, depends on checkpoint size, precision and runtime overhead, and it has to be measured per variant. Over-eager pre-emptive scale-out wastes resources, so tune TTFT thresholds against both the SLA and the bill.

Who can help us tune GPU inference autoscaling without overprovisioning?

TechnoLynx designs and tunes autoscaling policies for GPU inference fleets through the Inference Cost-Cut Pack: leading-signal triggers, bounded scale-down, warm capacity sized to measured cold-start cost, and replay validation before rollout. Building in-house is sensible when your team already runs the serving stack’s metrics, probes and cluster autoscaler confidently. A specialist pays off when cold starts, flapping or idle GPUs persist after the obvious settings have been tried. When choosing a partner, ask whether they will measure your own model’s load phases and replay your own peak traces, rather than recommend a buffer size from experience. Autoscaling is a serving-path lever: cost and latency pressure in production AI infrastructure and SaaS often responds to batching, routing, caching and scaling configuration without replacing the model.

The pack works on the serving path and the workload you actually run, not on the cloud contract. It also cannot produce a regulatory sign-off: we supply the engineering evidence, and your quality function judges it. If your traffic is steady and your SLA is loose, the HPA defaults with a sensible floor may be all you need, and a short internal load test is the right next step.

FAQ

What is the difference between the cluster autoscaler and HPA?

The HPA changes the number of pods in a workload to match a metric. The Cluster Autoscaler adds only as many nodes as are needed to run pods that cannot be scheduled. Used together, the time from rising load to new pods running is the sum of HPA reaction, Cluster Autoscaler reaction and node provisioning, usually about 5 minutes in the Cluster Autoscaler FAQ’s account.

What triggers auto scaling?

An autoscaler compares live metrics such as response times, queue lengths, CPU and memory against predefined thresholds or schedules, and decides from that whether to scale.