Regression Testing for Model Serving Changes Before Cutover

Regression testing for model serving changes is a sequence of tests, not a single test. You fix the evaluation set and the acceptance rule before anything runs, replay or shadow real traffic against that rule, and only then canary the change behind rollback gates written down in advance.

Production AI systems regress in ways unit tests do not catch: quality drifts silently after a model update, edge cases fail only under load, and release gates are missing on workflows that need them. The harness that catches these failures is an engineering artefact in its own right, separate from model accuracy and separate from a monitoring dashboard.

A regression testing sequence for model serving changes

The stages are not substitutes for one another: contract tests in CI, a replay corpus at merge and a canary at ship is the usual combination.

Stage	What feeds it	Repeatable?	Side-effect risk	What a pass shows
Captured replay	A recorded corpus on disk, which ages as the data it references changes	Fully: the same corpus can run against two commits and be bisected	Controllable: the corpus can be filtered and scrubbed before it executes	The candidate answers that corpus the way the baseline did
Live shadow (mirroring)	This second’s production traffic	No: yesterday’s traffic is gone	Highest: mirrored writes happen for real unless blocked	The candidate handles today’s distribution, provided you record and join the responses yourself
Canary	A small share of real users	No: the next canary sees a different population	Real, but bounded by the traffic share	The candidate does not fall over under production conditions; a subtly wrong answer can still pass

The order matters. Start from a captured corpus, because a diff you cannot re-run is a diff you cannot investigate. Move to live mirroring only once your normalisation and containment are mature, and canary at ship.

Can you replay real production traffic against a shadow deployment before go-live?

Yes. For an autoscaling or routing policy, capture a real peak, replay it as a step change against the policy, and watch time-to-serve and error rate during the provisioning window. TechnoLynx builds this replay harness; when the goal is a cheaper or faster serving path, the work runs as the Inference Cost-Cut Pack. The first step is defining the target spike profile from your historical traffic. The replay is then instrumented to reproduce capacity and thermal alarm conditions before the policy is trusted with a live event.

The shape of the load decides what the test can prove. CloudOptimo’s analysis of AWS Auto Scaling points out that a load test ramping gradually from zero to peak gives the autoscaler time to add capacity ahead of demand, so it does not validate spike behaviour. A spike test applies baseline load, jumps straight to peak and holds it there. The figure worth recording is time-to-serve: the time from the scaling trigger to the first request completed by a new instance. The same analysis puts a lean application at about 90 seconds, and one that pulls dependencies, warms a cache or starts a JVM at 4 to 5 minutes from launch to production-ready. Warm-up configuration matters as well, because new instances count towards capacity only once warm-up completes.

Keeping a shadow run from touching users

In a shadow deployment, each request is copied to the candidate while users receive only the stable version’s responses. The candidate’s responses are logged for comparison, never returned to users. Amazon SageMaker offers the same pattern for changes to the model, the container or the instance inside one endpoint.

Shadowing is not safe by default. A mirrored request that charges a customer charges them twice, and no proxy setting prevents it. Mirroring only GET and HEAD is a sensible first step, but read handlers often write too: last-seen timestamps, counters, cache entries and audit rows. For a model endpoint, use permitted shadow data, discard candidate responses and disable duplicate tool actions. The safer end state is an isolated world: the candidate with its own database, cache and outbound network policy, created from a definition and destroyed afterwards.

Shared state can also spoil the measurement. The Google SRE Workbook notes that if the canary and production share a cache, the inflated hit rate invalidates the candidate’s performance numbers. Mirroring is not free either: it adds the candidate’s cost on top of the production traffic it copies.

Then comes the acceptance rule. Replay supplies the workload, but the test still needs a rule for what counts as a pass, because a response with HTTP 200 can carry the wrong result. Compare the fields the change could affect, and normalise the ones you expect to differ, such as timestamps and generated IDs.

An offline replay does not measure live latency or reproduce an interactive tool loop. A harness tuned on one event’s audience shape may also miss the next event’s. That is why pre-production load testing and production monitoring are both required, and neither is enough alone.

As camera count grows, the batch size follows the number of sources, until memory, decode or compute capability limits the server. These parameters change with the number of streams. With adaptive batching enabled in DeepStream’s nvstreammux, the batch size follows the number of sources, and Triton sets max_batch_size and per-GPU instance groups in each model’s configuration.

It can. A kernel hotfix such as moving NMS onto multiple CUDA streams should be followed by end-to-end re-profiling of every stage sharing the device, decode included, at each step of a staged rollout. Benchmarking the patched kernel on its own is not enough. TechnoLynx handles this under GPU Performance Engineering: it maps every stage on the device, re-profiles the whole pipeline with Nsight Systems and Nsight Compute under realistic concurrent load, and keeps the rollback path ready until no stage has regressed.

The mechanism is contention for shared GPU resources. In one study, researchers ran Llama3.1-8B decode alongside a lightweight sleep kernel on an H100. P90 time between tokens rose from 7.53 ms to 16.56 ms, because the decode kernels had to wait for SM resources. Splitting the SMs between workloads did not remove the effect: with separate CUDA green contexts, memory-bandwidth pressure from a copy kernel still slowed decode by up to 1.3×. A pipeline optimisation study found contention at pipeline level too. Batched inference cut a stage from 36 to 23 seconds, but adding multiprocessing pushed it back to 25 seconds, which the authors attribute to larger batches increasing GPU memory contention.

NMS is a natural candidate for this kind of fix, because greedy NMS is sequential and each iteration is linear in the size of the proposal set. When looking for the new bottleneck, NVIDIA’s DeepStream FAQ suggests nvidia-smi dmon to check GPU, decoder, encoder and memory utilisation, along with temperature and power. Cross-stage regressions can stay hidden in isolated tests and appear only under realistic concurrent load. A staged rollout also delays the fix, which is a real cost when the original problem was a latency spike users could already feel.

How do you prove optimised kernels or FP8 weights match the current baseline within tolerance?

Build a reconciliation harness. Run both pipelines on identical inputs and diff their outputs against tolerance rules agreed before the first comparison, as a gate that blocks the cutover. TechnoLynx agrees with your quant or ML team what “acceptably reconciled” means for each output, designs the shared-input staging and the diff layer, and wires the harness into nightly CI with alerting, so it guards the cutover and every later kernel change.

Why bitwise equality is the wrong bar

Expecting identical outputs sets the wrong target, because common kernels give different results when the batch size changes. Thinking Machines Lab shows that common matmul kernels are deterministic from run to run but not batch-invariant: change the batch size and each element in the batch can get a different result. Server load decides the batch size, so a request’s output depends on what else the server was doing at the time. Batch invariance requires each element’s reduction order to stay fixed whatever the batch size, and Thinking Machines Lab reports giving up about 20% of performance against cuBLAS to achieve it for matmuls. The authors of one batch speculative decoding method verified output equivalence and still reached only about 95% exact match. They attribute the remaining divergence to floating-point non-determinism and argmax tie-breaking, not to algorithmic error.

A stronger way to set tolerances than any ignore list is to run the known-good pipeline against itself twice and record where it disagrees with itself. That gives a measured noise floor for each field, and only disagreement above that floor gets escalated.

For generative changes such as a new batching scheme or speculative decoding, greedy decoding removes sampling variance. That lets you measure exact match and partial match, meaning the fraction of tokens that match before the first divergence. Early divergence usually points to position-ID or KV-cache misalignment, so partial match helps you locate a bug, not only score the run. Metrics such as ROUGE can hide corruption, because repetitive output can still score reasonably. When a scenario drifts past its tolerance, bisect by kernel or stage before trying again.

A pass is a statement about the scenarios you tested, not proof for every possible input.

What quality drop is acceptable before rejecting a quantised or ternary model?

Decide the threshold before the first run, and measure it on a held-out set sampled from real production prompts that was never used to tune the current system. The right number depends on the product. The comparison method does not: the “Adding Error Bars to Evals” paper recommends statistical inference on question-level paired differences between the two models, rather than comparing population-level summary statistics, and power analysis to check that the eval can detect the drop you care about.

Matching accuracy alone does not show matching behaviour. Dutta et al. (NeurIPS 2024) found that even when a compressed model and its baseline have similar accuracy, individual answers flip from correct to incorrect and back. They propose KL-divergence and flip rate as distance metrics. The FP8 formats paper (Micikevicius et al., 2022) gives a useful reference point: for every network except MobileNet v2, FP8 training accuracy fell within the run-to-run variation of 16-bit training. A threshold tighter than the baseline’s own seed-to-seed variation cannot be tested, so measure that variation first.

Proxy metrics can point the wrong way. The Nemotron-H report found FP8 loss curves typically less than 0.1% worse than BF16, yet downstream evaluations were as good or better, and loss did not reliably predict downstream results. The same report found that its FP8 recipe had to be verified on models of at least 8B parameters, because results from smaller models did not generalise. For serving, the lesson is to validate on the model size you actually deploy.

Ternary weights add a kernel question. The Bitnet.cpp authors checked their ternary kernels on the bitnet_b1_58-large model against Float16 on WikiText2 perplexity and on HellaSwag and WinoGrande accuracy: some kernels were lossless and others showed negligible loss. Earlier implementations had not achieved lossless inference, because they did not match the BitNet b1.58 training scheme. Labels also hide differences: checkpoints, quantisation and serving defaults change outputs even when the public model names match. Making the eval set represent your traffic is the subject of building credible, data-specific evaluation harnesses.

Showing that a p99 win came from the batching change, not the rest of the release

To attribute it to KV-cache-aware batching, rerun the old and new configurations on the same hardware and software stack, change one variable at a time (driver, tokenizer, scheduler), and record per-stage timing in both runs.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack your models are built on, which gives the rerun a fixed reference for the hardware and software stack, so the first step is to check its results for the GPU you serve on before comparing the old and new batching configurations.

Replay is what makes the reruns possible, because the same corpus can run against two commits and be bisected. Spinnaker’s canary guidance gives the matching discipline: compare against an equivalent baseline deployed at the same time, so that cache warm-up and heap size are not mistaken for effects of the change. If the driver and the scheduler change together, the result cannot be attributed to either. Breaking time down into queueing, batch wait, KV-cache handling and decode kernels is the job of stage-level latency attribution, a separate subject. Here the point is that the breakdown has to run inside a reproducible harness to count as evidence.

For a claim made to a customer, a harness they can run themselves is worth more than a report. They can check the attribution on their own stack, though that only helps when their workload mix and hardware are close enough to the original benchmark conditions.

Staging a retuned model on a non-production replica before the fleet push

Push to the whole fleet only after three gates pass. First, the model and its over-the-air update mechanism work end to end on a non-production replica. Second, acceptance metrics such as IoU or mAP meet a threshold fixed in advance on a frozen, representative batch. Third, a per-site comparison against each site’s current production model shows no unacceptable regression. After that comes a staged rollout, with monitoring that catches a failing appliance within minutes. TechnoLynx defines the acceptance criteria, builds a staging harness that mirrors production appliance behaviour including the update mechanism, runs the frozen-batch comparisons and sets the alerting and rollback procedure for the staged rollout.

The replica is only as useful as its fidelity. A staging environment should closely replicate production servers, caches, hardware and configuration, because that is what exposes configuration drift and performance regressions that only appear at release. The indicators you rely on must be measured the same way in pre-production tests and in production monitoring. During rollout, a ring strategy by region, tenant or traffic type guards against an early sample that turns out not to be representative. The Google SRE Workbook suggests that the first small stage should watch the clearest signals, such as crashes and request failures.

The limits are worth stating plainly. A frozen batch covers only the cases it contains, not the drift that arrives after rollout. Sites left out of the test set remain untested, and fast alerting shortens the exposure window without closing it. How much of this evidence a given model needs is a question of risk, covered in validating models before they go live, from risk tiering to evidence packages.

To show that a fix would survive a repeat of a specific incident, such as a Friday peak that crashed a transcoding fleet, replay the recorded peak-hour pattern against the new policy at equal or greater load than the incident, and use that run as the acceptance test. Passing it is evidence rather than proof, because replayed traffic misses new codecs or client mixes, and failure modes nobody has seen yet remain possible.

Load-testing a new blended GPU fleet against last year’s peak

A new fleet mix needs three things before it carries peak season: a replay of last year’s peak against a staging copy of the fleet, live visibility into the serving pipeline during the real spike, and measured confirmation that each fix restores the latency target under that load. TechnoLynx covers the measurement side: end-to-end profiling and per-stage latency breakdown of the inference pipeline under the replayed peak, plus a workload-based evaluation harness when the fleet mixes GPU models.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack your models are built on, so the first step for a blended fleet is to check its results for the GPU models you plan to mix before the replay tests your own workload.

Replaying the recorded peak against a staging copy

GoReplay’s guide sets out a practical workflow. Capture production traffic safely, filter or mask sensitive payloads, and route the replay into staging. Then compare responses, latency and error patterns, watch downstream queues, caches and resource use, and investigate every divergence before release. The same guide treats a candidate that is 15% slower on p95 than the previous release as a defect, even if every functional test passes. TiProxy’s replay tooling for TiDB shows the controls worth having in any replay tool: it runs at production rate by default, mapping each connection to its production counterpart, and a speed option replays faster to compress the run.

The hardware mix needs evidence of its own. LynxBenchAI publishes results showing how each device runs the public PyTorch stack your models are built on, which makes them broadly indicative for custom PyTorch models too. They do not run your workload, and that is what the replay is for.

Seeing the pipeline during the real spike

RadView’s SLA testing guide recommends three layers: load testing against SLA thresholds at every release, SLO burn-rate alerts at graduated thresholds, and periodic audits. It also recommends setting internal SLOs 20 to 40% tighter than the contractual SLA, so that operators are warned while there is still room to act. During a spike, the signals to watch are time from scaling trigger to the first new instance serving traffic, healthy host count, connection counts against their limits, and error rate during the provisioning window. Finding exactly where in the pipeline time is lost is a stage-attribution question, separate from the pass or fail verdict.

Confirming a fix restores the latency target under peak

Take a fix such as moving NMS onto multiple CUDA streams to meet a latency target. A fix that speeds up the kernel but shifts the queue somewhere else has only masked the stall. GoReplay’s guide describes a candidate whose endpoint met its latency SLO in isolation, while p99 blew up only under the whole system’s concurrency.

Who can run this load test for us if we have no spare engineering capacity?

TechnoLynx can run the replay and load test for you. TechnoLynx defines the spike profile from your historical traffic, implements and instruments the replay harness, and validates the fleet’s behaviour under it against real-spike telemetry. When the goal is a cheaper or faster serving path and not only a verdict, the work runs as the Inference Cost-Cut Pack.

This engagement is not a good fit for plain web load generation against an API with no GPU-resident stages. It also does not replace your own on-call monitoring during the live event, which stays with the operators who run the service.

Which metrics and thresholds should trigger an automatic rollback?

Choose them before the first request reaches the candidate. The usual signals are the error-rate gap and the p95 and p99 latency gap against the stable version, saturation, a business or quality proxy, and cost per request. Flag truly critical metrics, such as error rate, as critical, so that a single breach fails the whole canary. TechnoLynx instruments model-serving paths with the tracing and metrics these gates read, so the canary compares like with like.

A rollback gate checklist for a model-serving canary

The thresholds are written down before rollout. RadView’s example gates fail a release if p99 exceeds 500 ms for more than 60 seconds, or if the 5xx rate exceeds 0.5% in any five-minute window.
The canary is compared with an equivalent baseline deployed at the same time, at the same size and with the same traffic, not with production as a whole.
Each metric has enough data: Spinnaker’s guidance asks for at least 50 time-series points per metric and suggests a three-hour canary lifetime as a starting point.
The metric interval is no longer than the canary itself; an hourly metric judging a 30-minute canary gives a muddied signal.
The gates watch tail latency, because p99 can regress while p50 looks normal.
Alarms trigger rollback automatically during the bake period, as SageMaker’s canary traffic shifting does with CloudWatch alarms.
Check that rollback removes both the traffic and any side effects the candidate triggered asynchronously.
The canary configuration itself has been tested against a known-good and a known-bad version, ideally first in a non-blocking branch of the pipeline.

Gating a firmware or encoder update on feature parity

A canary measures error rate and latency, not whether responses are correct, so a subtly wrong answer gets through. For firmware that feeds a model, the gate that belongs before the canary is a parity test. On every firmware build, replay a fixed set of broadcast frames through the old and new pipelines, diff the resulting feature vectors against the training-time values within tolerance, and fail the build on a breach. One migration guide describes a parity mode that sends each request to both systems and logs every mismatched response. Their sequence is to roll out mirroring first to test capacity and switch to parity before cutover. AutoMQ’s guidance on replay testing adds the habit that turns this into a gate: produce artefacts (counts, output diffs, records that changed classification), because “no exceptions in logs” says nothing about correctness.

Should regression gating be per model or across the whole API?

No published guidance we could find settles this, so what follows is our judgement. Gate each model against its own known-good version and configuration, and add API-level gates on top for the risks that cross models. Each model has its own equivalence tolerance and its own failure modes, and a single blended quality score lets a regression on a small model hide behind a large, stable one. A routing-policy evaluation makes a similar point: a global average can hide a policy that helps research tasks but harms document workflows.

API-level gates still earn their place, because workloads that share a GPU interfere with each other. A co-located kernel can raise decode’s P90 time between tokens even when the model itself is untouched. In practice we would use per-model gates for output equivalence and quality, API-level gates for latency, error rate and saturation, and report every result broken down by model and tenant.

How much production traffic do you need to replay to trust a serving change?

Enough to cover the risks the change could introduce. A fixed duration does not prove coverage, and some systems can use offline replay followed by a small canary instead of a long shadow run. The Google SRE Workbook notes that the more uniform the requests, the less volume you need, and that a handful of queries tells you nothing about a system serving varied requests. On the statistical side, power analysis tells you whether a sample can detect the size of change you care about. Spinnaker’s minimum of 50 data points per metric sets a floor for the canary that follows.

Start with a narrow replay that proves the harness works before testing the business change, then widen the window towards the inputs that carry the most risk. We would build the corpus by strata rather than by the clock: the peak window, a slice for each model and tenant, and the rare paths added deliberately, since organic traffic under-samples them.

A clean replay still proves less than it seems. Production traffic is a biased sample by construction: it over-represents what customers do constantly and under-represents error paths and rare integrations. A replay also interleaves requests differently from the original, so it says nothing about races, and matching at replay rate does not show the system holding at peak. A 100% match retires one specific risk, unintended behaviour change on the traffic you see most, and the change should still ship behind a canary that watches error rate and latency.