Validating AI system changes means putting each change through a sequence of gates, before and after it ships. On this page that covers a new model version, a drop to lower precision, a CUDA library swap and a move to more GPUs. The gates are a comparison against a reference run, an automated regression check against the current production model, shadow or canary traffic with pass/fail criteria set before deployment, a rehearsed rollback of the whole deployment bundle, and independent review. Production AI systems regress in ways unit tests cannot catch, so change validation is a reliability discipline and not an accuracy exercise.
Teams running AI infrastructure and production SaaS meet this problem whenever the serving stack, kernels or libraries change, not only when the model does.
How do you validate a change to a production AI system before it goes live?
Start with a veto. Google’s ML Test Score rubric says that after a model is trained, and before it touches real traffic, an automated system must inspect it and either approve it or block it from production. The same rubric separates two failure shapes. Catching slow degradation across many versions needs loose thresholds against a validation set. Catching a sudden drop in one new version needs a comparison against the previous version’s predictions with tighter thresholds. In our view the veto should run both comparisons.
Automated evaluations and monitors miss a share of what goes wrong. A study of four years of Microsoft production incidents in generative AI cloud services found that 38.3% of GenAI incidents were reported by humans rather than automated monitors, compared with 13.7% for other cloud services. The same study recorded an 11.0% false alarm rate among incidents that monitors did detect. Incident symptoms split into performance degradation (49.8%), deployment failure (35.7%) and invalid inference (14.5%). Infrastructure issues caused 27.2% of incidents, and the authors note that faulty GPUs can process requests incorrectly and produce gibberish output.
The gate sequence, in order
- Reference-run comparison. Compare the candidate with the current production model on a held-out reference set, alongside schema, data sanity and performance threshold tests.
- Blocking regression gate. The model registry refuses the promotion call until every required approval and test artefact is present, instead of only recording approvals afterwards.
- Shadow, then canary. Run the candidate on live inputs without serving its outputs, then send a small slice of traffic to it, with gates on quality, latency and cost that were defined before deployment. One MLflow production checklist routes 5–10% of traffic to the canary and monitors it for a defined window, typically 30–60 minutes.
- Progressive ramp. Increase traffic step by step only while the gates stay green, and keep the previous version warm for a defined window after full rollout.
- Automated rollback. A metric regression after promotion triggers rollback without anyone having to intervene.
- Independent review. Someone with no stake in whether the model is judged valid reviews the validation work.
Bit-for-bit matching is the wrong pass bar on GPUs
One candidate pass criterion is that the changed system must reproduce exactly the same numbers as before. On GPUs, non-deterministic operations rule out exact recomputation as a verification method. In parallel matrix work the order of floating-point operations is not guaranteed to stay the same between executions, and the resulting differences accumulate through a model’s layers. Fused multiply-add rounds once where a separate multiply and add round twice, so the fused result is more accurate than the two-step one.
PyTorch’s documentation states that completely reproducible results are not guaranteed across PyTorch releases, individual commits or platforms, or between CPU and GPU executions, even with identical seeds. It adds that the SDPA attention backends accumulate in different orders, so bitwise-matching numerics across backends are not guaranteed even for the same inputs and dtype. NVIDIA’s cuBLAS documentation promises bit-wise identical results only for the same toolkit version, on GPUs of the same architecture with the same number of SMs. The promise does not hold across toolkit versions, with multiple active CUDA streams, or when fixed-point emulation is used.
Serving load introduces a further source of variation. Thinking Machines showed that when kernels are not batch-invariant, a request’s output depends on the batch size it runs in, and batch size depends on how busy the server is. With batch-invariant kernels enabled, all 1,000 completions in their test were identical. Determinism is achievable, then, but only by building batch invariance into the kernels.
Determinism still has its uses. The ML Test Score points out that deterministic training lets you diff-test a refactor: if the old and new feature code train identical models, the refactor was correct. We would use bitwise checks where the stack can guarantee them, such as within one pinned toolkit version on one architecture, and a stated tolerance everywhere else.
Which test set, metric and tolerance should a changed model be compared against?
The test set is a held-out reference set, and its identity travels with the result. The MLflow registry checklist requires the raw scores, the evaluation dataset ID and the test run ID to be attached to the registry entry before a promotion request is submitted. Aggregate scores can hide a regression, so the set should be sliced. Google Cloud’s MLOps guidance gives the example of a churn model that improves overall accuracy while its accuracy per customer region varies widely.
The metric depends on what the output is. For tensor outputs, NVIDIA’s Polygraphy tool compares frameworks elementwise by default and can instead compare the mean, median or maximum absolute and relative error across a tensor. Its documentation gives an example comparing TensorRT using TF32 against ONNX Runtime with absolute and relative tolerances of 0.001 on the median error. It also warns that the default tolerances suit FP32 and may not suit reduced precision. For generated text, a 2025 paper on validating GPU computation compares each response with a trusted reference response by cosine similarity of embeddings, and accepts the response if similarity meets a pre-trained threshold. With a threshold of 0.5 on a −1 to 1 scale, that check reached 76.5% accuracy on a held-out test set. At that threshold its precision was 66.9% and recall 81.8%, so the gate itself makes errors.
The tolerance is a trade-off between false alarms and missed regressions. Google’s SRE workbook warns that, in canary evaluation, an overly strict definition of acceptable behaviour produces false positives, while a loose one lets bad canaries go undetected. A numerical tolerance is a business decision expressed in units the system produces (basis points on a price, a loss gap, a per-class detection rate), so in our view the people who own those outputs should set it with the engineers, and it should be written down before the comparison is run.
| Output type | Comparison | Example metric | Watch for |
|---|---|---|---|
| Tensors from a ported or recompiled model | Elementwise or statistical error against the reference framework | atol/rtol on median, mean or max error, as Polygraphy supports | FP32 default tolerances applied to reduced precision |
| Generated text | Semantic similarity to a reference response | Cosine similarity against a pre-trained threshold | The gate’s own accuracy, precision and recall |
| Model quality after a retrain or update | Candidate vs current production on a held-out reference set | Threshold and regression tests, sliced by segment | Overall gains hiding per-segment losses |
| Training or optimisation loss | Final loss gap to the FP32 baseline | Relative gap under a fixed threshold | Faster variants that need a per-problem check |
How do you prove an FP16, FP8 or INT8 model still matches its FP32 baseline?
You prove it on your own model, with your own reference set, against a threshold fixed in advance. Published results show parity is achievable but not automatic. A 2023 study of FP8 training and inference matched FP16 validation accuracy for GPT models from 111M to 13B parameters. When it swept fixed scaling biases, it reported only the values that reached more than 99.5% of the FP16 accuracy, because not all of them did. A 2025 FP8 training recipe found differences from BF16 typically within 1–2 points across reasoning benchmarks. A 2024 study of more than 500,000 evaluations on the Llama-3.1 family found FP8 weight-and-activation quantisation effectively lossless and well-tuned INT8 at 1–3% average accuracy degradation. The same study found its INT8 scheme caused noticeable drops at 70B and needed SmoothQuant to recover. Results depend on model size, even within one family.
A 2026 paper on mixed-precision SGD gives a worked tolerance protocol. The authors compared mixed-precision communication-avoiding SGD against an FP32 baseline under a stated 0.5% validation threshold on final loss. A faster variant fell outside the error budget their theory verifies, and they recommend it only with a per-problem check. The protocol states the threshold first, then reports which configurations pass, then marks the configuration that needs its own evidence.
Two practical points follow. INT8 usually needs a calibration step before accuracy is acceptable, as NVIDIA’s Polygraphy documentation notes. And a parity result from a paper describes the authors’ workloads, not yours. The detailed protocol for proving kernel equivalence after porting or precision changes, including how to attribute divergence to a specific kernel, has its own page.
Three GPU pricing changes that need three different validation plans
Three decisions look alike because all of them touch the same validated pricing code: scaling CUDA pricing kernels across GPUs, replacing kernel internals with vendor libraries, and choosing a backtesting engine. Each one changes something different, so each needs its own reference and its own rollback unit.
| Change | What stays fixed | Reference run | Main risk | Rollback unit |
|---|---|---|---|---|
| Scale kernels across GPUs | Kernel code and pricing maths | Single-node results on the same scenarios | Hidden cross-path dependencies, aggregation order | |
| Swap kernel internals for cuBLAS or cuSOLVER | Input/output contract the pricing cache depends on | Legacy kernel outputs | Library numerics shifting across toolkit versions | Prior kernel behind the same interface |
| Build or buy a backtesting engine | Strategy logic, data and cost specification | A second, independent engine | Engine-level defects and cost-model differences | Previous engine kept runnable |
Can validated CUDA pricing kernels scale across GPUs, or be swapped for cuBLAS, without a rewrite?
Validated CUDA pricing kernels can be scaled by sharding scenarios across GPUs when paths are independent, and their internals swapped for vendor libraries behind an interface-preserving wrapper. These are two separate decisions. Scaling is a parallelisation decision, and swapping internals is an interface and migration decision.
TechnoLynx handles both through GPU Performance Engineering. The engagement confirms that paths are truly independent, defines the interface contract your pricing cache depends on, shards scenarios across GPUs or swaps internals behind a wrapper, and validates against single-node and legacy outputs before cutover. The first step is the independence and contract review (GPU Performance Engineering).
Scaling across GPUs without touching the pricing maths
In a multi-GPU option pricing implementation described at MIPRO 2012, every CUDA thread simulated one Monte Carlo path, and the kernel was invoked repeatedly on each GPU. The authors rejected static distribution of work because it would not suit systems with different types of GPU, and used dynamic load balancing: one main thread distributes blocks of paths and gathers results while the others compute.
NCCL is compatible with virtually any multi-GPU parallelisation model, including such a design. NVIDIA describes NCCL as a library of topology-aware inter-GPU communication primitives (AllReduce, Broadcast, AllGather and others, plus point-to-point send/receive) that works with single-threaded, multi-threaded or multi-process control. NVIDIA is explicit that NCCL is not a full parallel programming framework, so the sharding logic remains your code. The no-rewrite route therefore starts by confirming that paths are truly independent.
Does the multi-GPU version reproduce single-node results within tolerance?
The multi-GPU version should be signed off against single-node results under a tolerance and test protocol agreed in advance, not against an expectation of identical bits. A paper on a GPU linear-programming matching solver shows what that evidence looks like. The authors found relative error in the dual objective below 1% within the first 100 iterations and still falling, and saw stable error behaviour across single- and multi-GPU runs. They concluded that NCCL-based aggregation and column sharding preserved numerical consistency. The tolerance itself (basis points on price, per instrument or per book) belongs to the risk and quant owners.
Replacing kernel internals while keeping the contract
The migration route is an interface-preserving wrapper. The function signature, input layout and output semantics the pricing cache expects stay fixed, and a cuBLAS or cuSOLVER call sits behind them, with a switch that can select the legacy path. Several facts about the vendor libraries shape the regression test. cuBLAS has used Tensor Cores automatically wherever possible since version 11.0.0 unless pedantic compute modes are selected. The 16-bit pedantic mode uses standardized half-precision arithmetic in every phase and is intended for numerical robustness studies, testing and debugging. When the compute type is wider than the output type, split-K summation can produce intermediate overflows, which can be controlled through cublasSetMathMode with CUBLAS_MATH_DISALLOW_REDUCED_PRECISION_REDUCTION. On the solver side, NVIDIA’s cuSOLVER 13.4 documentation marks cuSolverMG as deprecated and directs multi-GPU users to cuSOLVERMp. A migration plan built on cuSolverMG would need revisiting.
Where the work is moving the pricing code to a new runtime or target, as well as changing its internals, the AI Porting & Deployment Pack covers the port and its validation. The rollback plan for a library swap that diverges unexpectedly keeps the prior kernel selectable behind the same interface, with production divergence monitoring. Because vendor numerics can shift between versions, the regression suite reruns on every toolkit upgrade.
Can change validation run automatically in the build pipeline?
Yes, and a gate that depends on someone checking results by eye scores lower on Google’s rubric. The ML Test Score awards half a point for a test run manually with documented results, and a full point only when a system runs it automatically and repeatedly. Martin Fowler’s CD4ML article describes threshold tests, or ratcheting, as a simple quality gate that stops new models degrading against a known baseline. Google Cloud’s MLOps guidance adds tests that call the prediction service API with expected inputs, to catch a model version that expects different input, and load tests that capture queries per second and latency.
Speed and resource regressions belong in the same pipeline. The ML Test Score recommends slicing serving latency, throughput and RAM metrics by data and model version as well as by code version, because regressions can come from the underlying compute library or infrastructure. It also notes that canarying catches problems like numeric instability long after the engineering decision that caused them, so catching them earlier in unit or integration tests is preferable. The canary is the last gate, not the main one.
MLflow’s guidance on registry automation puts it plainly: “Automated promotion without automated rollback is half a system.”
What should a rollback plan cover when a library change diverges in production?
A rollback plan covers the model, the serving code, the data and feature pipeline, and the configuration, reverted together to a known-good combination. Reverting only the weights while a changed prompt template or feature pipeline stays live often reproduces the bug or creates a new one. One practical rule from the same guidance is that anything which can change outputs gets a version and is recorded with the deployment. Because cuBLAS promises bitwise reproducibility only within one toolkit version, we would record the CUDA toolkit and library versions in that manifest alongside the container image.
Rollback is also a widely used incident response in practice. The Microsoft incident study found rollback widely used and efficient for incidents triggered by configuration or code changes. Under tight deadlines, on-call engineers prioritised quick approaches like rollback to reduce downtime. To be quick, a rollback needs preparation: the previous good version kept warm so that rollback takes minutes, not hours, and full provenance on every registry transition so the system knows exactly what it is reverting to. Blue/green deployment makes the switch itself a trivial reversal of the router change. Ongoing drift monitoring and revalidation after deployment are covered in keeping deployed models trustworthy through revalidation, monitoring and rollback.
A worked failure: every gate passed and quality still dropped
Anthropic’s postmortem shows why quality evaluations have to run on production systems too. Between August and early September, three infrastructure bugs intermittently degraded Claude’s response quality. The first affected about 0.8% of Sonnet 4 requests, and a load balancing change on August 29 increased the share of affected traffic. Anthropic’s validation already included benchmarks, safety evaluations, performance metrics, spot checks and small canary deployments. The evaluations did not capture the degradation, partly because the model often recovers from isolated mistakes, and the company says it relied too heavily on noisy evaluations. Its fixes were more sensitive evaluations that can tell working implementations from broken ones, and quality evaluations run continuously on true production systems. It also named user reports as a necessary signal.
The lesson we draw is narrow. A change to load balancing or serving infrastructure is a model change from the user’s point of view, and the evaluation set must be sensitive enough to tell a correct implementation from a subtly broken one.
When is independent expert review worth paying for?
Independent review earns its cost when the people who made the change are the only people checking it and the output feeds a material decision. The Federal Reserve’s SR 11-7 guidance (2011) says validation is generally done by staff who are not responsible for model development or use and have no stake in the outcome. Where developers do some validation themselves, an independent party should critically review that work. KPMG’s 2024 framework for AI/ML model risk adds a review of the validation process itself by independent internal or external parties. SR 11-7 also expects validation to continue after a model goes into use, with at least an annual review.
Other triggers are a divergence nobody can explain, a change to numerical libraries under a pricing or risk model, and a change where the team cannot state the tolerance it is testing against. Producing an audit-ready evidence pack in a regulator’s format is a separate piece of work from the engineering review.
Who can validate a GPU kernel or library change before it reaches our production system?
TechnoLynx validates changes to GPU-accelerated production systems before cutover: scaling kernels across GPUs, swapping internals for vendor libraries, and moving to lower precision. An engagement frames the problem and defines the interface contract and tolerance with your owners, chooses the approach, implements it, runs bit-for-bit or tolerance regression tests against single-node and legacy outputs, and deploys with a rollback path and production divergence monitoring. The catalogue offer is GPU Performance Engineering. On building in-house or bringing in a specialist, we would weigh whether your team already owns the numerical reference and the regression harness. If it does, outside help adds independence. If it does not, outside help also adds the method.
The engagement supplies engineering evidence, not the regulatory or model-risk sign-off itself. SR 11-7 expects such validation work to be critically reviewed by an independent party. Teams with a mature registry, a blocking promotion gate and an owned tolerance may need nothing beyond their own MLOps tooling.
Should we buy a GPU backtesting engine or build our own?
In our judgement, buying a GPU-accelerated intraday backtesting engine makes sense when a commercial platform covers your strategy types and your latency and throughput needs at an acceptable integration cost. Building makes sense when you need tight control over proprietary alpha logic, custom data pipelines, or performance beyond what the platforms deliver. TechnoLynx supports the decision through GPU Performance Engineering: it catalogues your strategy types, throughput needs and current backtesting bottlenecks, scores platform coverage against the feasibility of a custom engine, and, if you build, develops and benchmarks the vectorised engine on your hardware.
Vendors’ coverage and speed claims are claims to test against your own workloads, not facts to plan on. What the evidence does show is that the engine is itself a source of backtest unreliability, whichever route you take. A 2026 study ran 15 benchmark strategies through five independent backtesting engines, holding strategy logic, data and cost specification fixed across engines. At zero transaction cost all engines agreed exactly. With costs included, divergence reached 3.71% in total return for high-turnover rotation strategies under the heaviest cost regime. The authors found a Backtrader default that silently divides the user-specified commission rate by 100, and excluded three engines for calendar, arithmetic and API problems. They recommend at least two independent validators chosen for implementation diversity, such as one event-driven and one vectorised engine, plus an audit of each engine’s cost model.
Buying means relying on another party’s cost model, the component the study found drives divergence. For a very small team, a third option is to optimise the engine it already has.
The tolerance is the decision nobody wants to own
Every gate above depends on one number someone has to write down before the test runs: how far the changed system may drift from its reference and still ship. Only the owners of the output can say whether a gap of a few basis points on a bond book, or a point on a reasoning benchmark, is acceptable. In our view the most useful hour in any change-validation project is the one spent agreeing that number with the people whose decisions depend on it.
