Predictive autoscaling adds capacity ahead of a forecast load, so new replicas are already serving when the spike lands. Reactive scaling adds capacity when a metric exceeds a predefined threshold. The difference matters because a GPU replica is slow to arrive. A 2026 study of LLM serving on Kubernetes puts new-replica startup at two to ten minutes once multi-gigabyte weights have to load, which makes purely reactive scaling structurally late. A reactive policy is always behind by that startup delay, so the transient that triggered scaling has already caused SLO violations before the new capacity can help.
That does not make prediction the default. The sensible order is to build a plain queue-depth scaler first (KEDA on request-side metrics). Add forecasting only when the measured boot budget exceeds the queue your SLO can tolerate. If your spikes fit inside the reactive scaler’s reaction time plus a tolerable queue, you do not have a prediction problem.
Predictive autoscaling versus reactive scaling: where the minutes go
Reactive policies such as a typical Kubernetes Horizontal Pod Autoscaler (HPA) configuration can respond only after a demand increase becomes visible. Predictive autoscalers act earlier by provisioning from a workload forecast rather than from demand already visible. For GPU inference the gap between the two is minutes, not seconds.
Through the Inference Cost-Cut Pack, TechnoLynx cuts inference cost and p95 latency on the workload you actually run.
TechnoLynx benchmarks the inference workload you actually run with LynxBenchAI, so the scaling decisions below rest on measurements from your own fleet.
The minutes come from a chain whose links cannot overlap. Cast AI’s breakdown runs like this: Karpenter makes the cloud API call in under 60 seconds and has a GPU node ready in 3-5 minutes with a pre-baked AMI. The GPU Operator then needs 1-2 minutes for drivers on a GPU-optimised AMI, or 3-7 minutes where kernel modules must compile. The image pull adds 1-3 minutes, and the weights still have to load into VRAM, so a large model on a fresh node typically takes 8-15 minutes in total. ScaleOps makes the same point from the other side: each stage must finish before the next begins, so optimising any single stage gives only a marginal improvement. A CNCF post from August 2026 on predictive GPU autoscaling shows what this looks like in an incident. The spike arrived at 06:00, HPA crossed its threshold at 06:05, and the first GPU nodes finished provisioning at 06:45, after the spike was already over.
A measured result: delay-aware lookahead against reactive KEDA
The 2026 Kubernetes LLM-serving study cited above also tested the mechanism on Qwen2.5-7B, vLLM and an A100 in a real Kubernetes cluster with a 60-second startup delay injected. A delay-aware lookahead controller cut time-to-first-token (TTFT) SLO violations from 63.5% to 3.7%, compared with reactive KEDA scaling. The predictive controller saw the demand slope rising at about 45 seconds, before any queue formed, and its new replica was ready at about 105 seconds. KEDA saw the queue only at about 90 seconds, and its capacity arrived at about 150 seconds, 45 seconds into the overload. In the authors’ wider ablation, lookahead on its own (forecasting demand at t plus the startup delay rather than reacting to the present) cut violations 14-fold and was the largest single factor. You also do not need to know the startup delay exactly, because any conservative upper bound works.
An OpenStack Monasca study that emulated a 6-minute VM start-up shows the same lag. The new VM only served from about minute 83, after the cluster had run at full CPU for about eight minutes.
Which signals warn of a spike before the latency SLO breaks?
Request-side signals lead demand. GPU utilisation does not, because it pegs near 100% during active batching whatever the real load is. Elastic’s vLLM tuning guide puts it bluntly: DCGM_FI_DEV_GPU_UTIL means a kernel is resident on the device, so it reads 100% for a well-tuned server and for a badly tuned one alike.
| Signal | What it tells you | Caveat |
|---|---|---|
| GPU utilisation (DCGM_FI_DEV_GPU_UTIL) | That a kernel is resident, not that another replica is needed | Reads 100% on well-tuned and badly tuned servers alike |
| vLLM queue time (vllm:request_queue_time_seconds) | The earliest saturation signal in vLLM’s metric set | Alert on a small absolute threshold, since it climbs before waiting counts show |
| Waiting requests (vllm:num_requests_waiting) | A value that stays above zero means add a replica | A request can wait milliseconds without being counted as queued at scrape time |
| KV-cache usage (vllm:kv_cache_usage_perc) | Whether the engine can admit another request; Elastic suggests scaling near 60% | Measures the pre-allocated block pool, not physical VRAM |
| Demand slope (running plus waiting, smoothed) | Rising demand before any queue forms | Needs a bounded margin to absorb under-prediction |
| External event schedule or forecast | A known change before any metric moves | Not a recurring pattern a forecaster has seen |
In cache-gated LLM serving, token weight matters more than request count. With requests held at 8 per second, raising input length from 128 to 4,096 tokens took KV usage from 2% to 34% and raised TTFT 17-fold. One inference autoscaling guide sets out the queue-versus-batch trade-off. Queue size is a sensitive spike indicator under continuous batching. A batching policy tuned for peak throughput can therefore push individual requests past the SLO, so batch limits belong in the scaling design rather than being left at defaults. Whichever signals you choose, noisy or highly variable metrics can make event-driven scaling unstable, so they need smoothing before they drive actions.
Mixing predictive and reactive policies for recurring and one-off spikes
Cloud vendors’ autoscalers already run predictive and reactive policies together. In Amazon EC2 Auto Scaling, each active policy computes desired capacity independently and the group takes the maximum. Google’s Compute Engine autoscaler lets a real-time signal take priority whenever it exceeds the prediction. Predictive scaling on AWS only scales out, never in, which removes the risk of ending up short because of an inexact forecast. Research hybrids follow the same pattern. A 2025 edge-computing hybrid of an LSTM forecaster and a reactive autoscaler held average latency under 30 ms, while default HPA showed frequent spikes above 300 ms. Hybrids also fall back to the reactive component while the forecaster has no training data yet.
Forecasts need history. Google’s predictive autoscaler needs at least 3 days of history before predictions take effect. AWS needs at least 24 hours of data, and its forecasts work better with two full weeks. A one-off campaign or match is a different case. It is known in advance, but it is not a recurring pattern the model has seen, which is the gap scheduled scaling exists for. Pre-provisioned or scheduled capacity is the more reliable tool for the opening of a planned surge, and reactive autoscaling earns its keep after the first minutes. Start by characterising traffic to right-size inference capacity, because the pattern decides the mechanism:
| Traffic pattern | Primary mechanism | Supporting mechanism |
|---|---|---|
| Recurring daily or weekly peak (evening viewing, market open) | Forecast-driven predictive scaling, once enough history exists | Reactive queue-depth scaling for overshoot |
| Scheduled one-off event (final, launch, on-sale) | Scheduled pre-warm sized to the expected leading edge | Reactive scaling after the opening minutes |
| Unpredictable spike (breaking news) | Real-time reactive scaling, since instances are not created in advance | Active buffer sized to cover the initial burst |
| Fast boot (nodes ready in about 30 seconds) | Reactive HPA is fine | None needed |
| Truly random demand | No forecaster will help | Real-time reactive scaling |
How much warm headroom a multi-minute cold start needs
The forecast horizon or warm floor has to cover the measured boot time, stage by stage. If the measured total exceeds the horizon, lengthen the horizon or pre-warm on a schedule. A 10-minute forecast does not cover a 25-minute boot. The warm floor itself should absorb the spike that arrives while new replicas are still booting.
Worked example: active and standby buffers
GKE’s capacity-buffer documentation gives a sizing rule: active buffer replicas equal the expected scale-up rate multiplied by the time standby nodes take to resume. In its example, a workload scales up at 2 Pods per second, standby nodes resume in 30 seconds and cold nodes take 90 seconds. The active buffer needs 2 × 30 = 60 replicas to absorb the first 30 seconds. The standby buffer needs (2 × 90) − 60 = 120 replicas while the cluster autoscaler provisions cold nodes in the background. Standby-first is the cheaper option where a scale-up delay of about 30 seconds is tolerable. For GPU replicas the same arithmetic holds, but you have to put your own measured resume and cold-node times into it rather than these figures.
How much over-provisioning is right is not a fixed number. The BACC autoscaler (2026) treats a fixed-period SLO as a violation budget to spend: it over-provisions when the budget is nearly gone and saves replicas when plenty remains. On a bursty trace replayed on Kubernetes, HPA met the 5% budget in 0 of 3 runs at a 9.724% violation rate. BACC met it in 3 of 3 at 4.337%, using 49.7% more replicas. On a regular trace, both controllers met the budget, and BACC used 16.7% fewer replicas. The lookahead study above estimates the price of adapting at about 1.5 replicas (about 19%) over a hindsight-optimal static allocation, and static capacity is cheaper only when load is predictable. Mélange (2024) found that 84% of its SLO violations came from request-rate bursts or co-location with large requests, the load that over-provisioning exists to absorb.
GPU type matters to cost too. Mélange found that L4 and A10G were more cost-efficient than A100 and H100 at low request rates, and that the advantage shrank as rates rose. Your own model and traffic still need profiling on top.
Can GPU inference pools scale to zero overnight?
They can, but the first request after idle waits through a cold start. Scale-to-zero removes idle GPU cost entirely, at the price of cold-start latency that depends on what is already warm. Cast AI advises minReplicaCount: 1 for latency-sensitive workloads and reserves scale-to-zero for batch jobs and background consumers. Even the detection adds delay: after scale-to-zero, KEDA can wait up to one polling interval (30 seconds by default) before it notices the first event.
A measured keep-warm breakeven exists for energy. A 2026 study derived a request-rate threshold above which keeping a model loaded saves energy: about 13 requests per hour on H100, 7 on A100 and 18 on L40S with standard PyTorch loading. The cold-start penalty can be cut without holding a pool. Pre-pulling images with a DaemonSet, using GPU-optimised AMIs and caching weights on a ReadOnlyMany volume all shorten the chain. A public vLLM-on-KEDA reference kit warns that its default emptyDir cache downloads the weights again in every new pod, and baking weights in cuts cold start more than any autoscaler setting. On EC2, warm pools of stopped instances cost only volumes and Elastic IPs. A depleted pool falls back to a cold start, as does an Availability Zone that is out of capacity.
Pricing headroom against the risk of SLO breaches
The cost comparison has three columns: always-on peak capacity, a capped predictive reserve, and the business cost of breaches during the event. A reserve cap stated in GPU cards, held only inside the predicted window and released outside it, turns each false positive into a bounded, priced mistake instead of an open tab. Because predictive scaling can only add capacity, alerting when a policy over-forecasts is useful to prevent unnecessary cost. Spot pricing cuts active compute costs. Cast AI cites 50-80% below on-demand for consistently interruptible workloads such as queue workers that tolerate termination and restart. If inference is split between edge and regional nodes, moving work between them can be part of the scaling response, but the failover needs the same surge testing as the scaler, or the move itself becomes the visible QoE incident.
Turning these into a per-request figure is its own exercise, covered in cost-per-request modelling for bursty, shared and scaling inference. The guardrails that stop a policy from overspending are the subject of designing cost-safe autoscaling policies for GPU inference fleets. Before any of that costing, it is worth remembering that latency and cost pressure in production AI infrastructure often yields to profiling and tuning the serving path (batching, routing, scaling configuration, runtime) before anyone needs to replace the model.
Driving KEDA from a forecast or an external event signal
Kubernetes can scale pods on an external event or forecast rather than on CPU, using KEDA’s external, push and cron scalers or Kedify’s predictive trigger. This only protects the SLO if the measured lead time (node provisioning plus model load) fits inside the time between the signal and the demand.
TechnoLynx builds this trigger path end to end. It maps the SLO budget against detection latency, provisioning time and model-load time, and chooses between a KEDA external scaler and a custom metrics adapter to suit your existing GPU Operator and Kubernetes stack. It then measures lead time and false triggers, and rolls the policy out with reactive scaling kept as a fallback.
KEDA is an open-source, CNCF-graduated component that extends HPA with event-driven triggers and scale-to-zero. It ships with more than 70 built-in scalers, and custom sources plug in through its External Scaler gRPC interface. The operator creates and manages an HPA for each ScaledObject. Four mechanisms matter for event-driven spikes:
- Cron scaler. It holds a time window (start to end) at
desiredReplicas. Because HPA takes the maximum across triggers, that value acts as a dynamic minimum underneath your queue-based trigger. - External push scaler. The
external-pushscaler keeps a long-lived gRPC stream open and can send an activation event to KEDA at any time, independent of the polling interval, so a radar alert, match kick-off or broadcast cue can drive scaling directly. - Predictive trigger. Kedify’s predictor trains a Prophet model with a 10-minute horizon, retrained daily, on metric data flowing from KEDA. Scaling modifiers then blend the live and predicted values, for example
(mm + mmInTenMinutes)/2. - Queue-depth trigger. The vLLM reference kit scales on
vllm:num_requests_waitingat 4 waiting requests per replica, which its author calls a starting point to tune against the TTFT target. It caps scale-up at 2 pods per minute so one spike does not grab every GPU in the pool.
KEDA also has a hard limit. It is an external metric adapter on the HPA machinery, pod-centric and not coordinated with node provisioning, which stays with the reactive cluster autoscaler. Pods scheduled ahead of capacity wait in Pending until nodes are provisioned for them. In the MAS-H² prototype (2026), a burst of Pending pods was the intended sign that proactive scheduling had started ahead of node provisioning. The pod-level trigger has to fire early enough to cover node, driver, image and weight time, not just pod start. One more safeguard helps: a KEDA fallback block holds a set replica count if the event source becomes unreachable, rather than failing silently.
The decision rule that follows from all this runs in order:
- Measure the end-to-end lead time on the target cluster: node ready, driver and device plugin registered, image pulled, weights in VRAM, first inference served.
- Measure the gap between the external signal firing and demand arriving, minus the signal’s own detection and delivery latency.
- If lead time fits inside that gap, drive pre-warming from the event through an external push scaler or a custom adapter.
- If it does not fit, keep a standing warm pool sized to the worst-case burst, add partial reactive queue-depth scaling, or both.
- Expect false positives from the signal, and price each one before go-live.
Testing the policy against a replayed surge before trusting it live
Validate an autoscaling policy with an open-model surge scenario that reproduces the real event’s arrival shape. Run predictions in shadow or forecast-only mode first, and promote the policy only when scale-out latency, false-trigger rate and headroom meet thresholds agreed in advance.
It captures the event’s traffic profile (peak concurrency, ramp rate, session patterns) and agrees the thresholds with whoever owns the SLA. It then builds the load generator and shadow harness against your staging fleet, instruments scale events, tunes the policy over repeated surge cycles, and keeps the harness ready for rehearsal before each future event.
The generator must use an open model. Arrivals are set by rate, so the load does not back off as the system slows, and the leading edge of the arrival curve is reproduced, not only the plateau. Pass criteria should describe the first minutes on their own, because an average over twenty minutes can hide a three-minute failure entirely. Force a cold replica and fire the spike at it. Measuring only against a warm pool misses the distribution users actually feel. The alt.qa write-up turns this into a CI gate that asserts first-wave p99 stays inside the SLA during a simulated cold-pool spike. In a healthy run you see the queue spike, desired replicas jump, available replicas follow after about one cold start, and TTFT recover. If replicas rise and TTFT does not recover, the bottleneck is elsewhere (usually node provisioning or image pulls) and scaling faster will not help.
Forecast-only mode comes first. AWS predictive scaling starts in forecast-only mode, producing forecasts without acting on them. The CNCF team ran their controller in shadow for more than 500 hours. Their forecasts were 85% within ±10% of actual demand at T+10 minutes, and their burst detector caught 9 of 10 real spikes with 2 false positives, which they judged acceptable. That burst detector exists because forecasts fail at discontinuities. It watches the rolling deviation of predictions from actuals and scales harder when real demand breaks the band. Uncertainty gating protects against the opposite error. In SLO-Scaler (2026), removing uncertainty quantification raised the violation rate to 8.5% and scaling events to 61, because the scaler reacted to every point prediction above threshold.
An acceptable false-trigger rate is not a universal number. It is the cost of an unnecessary scale-out weighed against the cost of under-scaling during the real event, and a rate that is fine for a minor fixture can be wrong for a marquee final. Every significant change to the pipeline, content mix or audience size calls for another run.
Pre-event validation checklist
- Arrival model is open (rate-based) and reproduces the leading edge, not only the plateau.
- At least one run starts from a cold pool, with first-wave p99 asserted against the SLA.
- Scale-out latency is recorded from signal to Ready replica, stage by stage.
- Predictions ran in shadow or forecast-only mode, and their accuracy was checked before go-live.
- A burst detector or uncertainty gate is in place for forecast misses.
- A hard max-replica cap and a runbook for disabling predictions exist.
- The false-trigger threshold was set from the cost of extra capacity against the cost of under-scaling, and signed off by the SLA owner.
Who can build predictive autoscaling for our GPU inference fleet?
TechnoLynx builds and tunes predictive autoscaling for GPU inference fleets that have to hold latency SLOs through event-driven spikes. On build versus buy: the signal choice, lead-time measurement and surge harness are engineering work on your serving path, which TechnoLynx delivers as an engagement. On scope: the Inference Cost-Cut Pack profiles and tunes batching, routing and scaling configuration on your serving path before anyone proposes a new model. When choosing any partner, ask to see a replayed-surge result on your own stack, not a steady-state benchmark.
TechnoLynx runs the replayed-surge testing described above against your own staging fleet before go-live.
This is the wrong engagement if your nodes come up in well under a minute or your spikes fit inside a reactive scaler’s reaction time. A tuned queue-depth KEDA baseline is the answer there, and your team can run it. Teams whose bottleneck is supply rather than speed, such as GPU quota or regional stock, need their cloud provider’s capacity programmes first.
Start with the cold-start waterfall
Every decision above depends on one number: how long it takes, on your fleet, from “scale” to “first inference served”. ScaleOps’ advice is to put a timestamp on each stage (node provisioned, container started, model downloaded, CUDA initialised, first inference served) so you can see which one dominates before you pick a pattern. If that number is around 30 seconds, stop at a reactive baseline. If it is ten minutes, a reactive policy stays structurally late by that delay whatever its thresholds. Lookahead, a warm floor sized to the gap, and a surge replay that proves both are what it takes.
