Sensitivity analysis verification only holds up when the re-check is a fixed procedure and not a fresh analysis each time. Whenever the digital twin, the alloy library or the hardware cost inputs behind a what-if re-costing change, the same frozen historical production data should go through a versioned harness. The outputs should be compared against locked reference results, with everything else pinned and logged: the software stack, the GPU architecture, the seeds, the deterministic flags, the sensitivity design, the metrics and the thresholds. Holding all of that constant is what lets you attribute a difference in output to the change under test. Fixed execution parameters are what let a verifier replay a request under identical conditions.
The rest of this article covers each part of that protocol in turn. It explains why the harness is frozen, what to pin, how to keep GPU nondeterminism out of the numbers, what to record on each run, how much to re-verify after a change, and when somebody other than the author should run it.
One frozen replay harness, re-run after every twin or alloy library update
Yes, the same verification can be repeated on every revision, provided the harness itself never changes. It replays one fixed historical production campaign against the simulated model each time the twin or the alloy library is updated. Inputs, versions, comparison metrics and thresholds should be held constant, so that only the model or library version varies.
TechnoLynx can build this kind of replay harness as reliability engineering. In an engagement, the first step would be to frame the verification problem by defining what “apples-to-apples” means for a particular twin and alloy context: which inputs, metrics and thresholds are fixed. From there TechnoLynx would harden a version-locked harness, automate the replay on every twin or library change with tagged configurations, and confirm that repeated runs on an unchanged model give identical results before the protocol is trusted to detect change. It would then deploy the harness into the release pipeline, as part of its production AI reliability work.
Simulation-based verification compares model outputs with a referent, meaning data, information or theory against which results can be checked, and the referent can be the real-world system itself. NASA-STD-7009B also has configuration-management records include test cases spanning the model’s limits, defined as benchmark input/output sets that verify proper execution. For digital twins specifically, one TEVV framework lists simulation testing with historical data as a testing activity and backtesting against historical data as predictive validation. A 2025 Karlstad University thesis describes a snapshot-based framework that tests a heating-furnace twin automatically against real production data.
The minimum protocol, in order
- Lock the historical production campaign dataset, the comparison metrics and the pass/fail thresholds as the reference, and record their versions.
- Re-run the same harness, unmodified, whenever the twin model or alloy library changes.
- Compare new outputs only with the frozen reference and with earlier run baselines, never with a re-sampled or re-selected dataset.
- Version-tag each run’s harness configuration, so that any deviation can be traced to the model or library change and not to the test setup.
Each step has a condition attached. If the historical campaign data is updated or re-sampled between cycles, the comparison stops being apples-to-apples. Any change to the harness or the metrics invalidates direct comparison and forces a re-baseline. The harness also checks against past production behaviour only. Passing replay says nothing about future or out-of-distribution operating conditions, or about alloy and process combinations that never appeared in the historical campaign. NASA-STD-7009B makes the same distinction when it says a model can be used to extrapolate outside its domains of verification and validation but still within its limits. An extrapolated what-if result should be labelled as one.
What sensitivity analysis verification has to pin before any re-run
The guiding principle is to pin anything that can influence the numbers. The EigenAI verifiable-inference design (2026) puts it this way: “every parameter that can influence numerical outcomes is fixed and committed”. Each request in that design fixes the model, the container digest, the GPU architecture, the driver and toolkit version, the decoding policy and the PRNG seed. These are treated as immutable execution parameters, so that any verifier can later replay the request under identical conditions. Its containers are built on fixed CUDA/driver pairs referenced by digest, with persistence mode and ECC memory enabled. The authors traced one case of nondeterminism to faulty memory and cured it by turning ECC on.
At the framework level, the determinism switches are specific. NVIDIA’s framework-reproducibility guidance sets PYTHONHASHSEED before the Python process starts, then seeds random, NumPy and torch.manual_seed inside it. In PyTorch, torch.backends.cudnn.benchmark = False makes cuDNN select its convolution algorithm deterministically. That selected algorithm can still be nondeterministic unless torch.use_deterministic_algorithms(True) or torch.backends.cudnn.deterministic = True is also set. torch.use_deterministic_algorithms() also raises an error when an operation is known to be nondeterministic and has no deterministic alternative. That error is useful in a harness, because the run fails loudly instead of drifting quietly. TensorFlow’s enable_op_determinism gives identical outputs for the same inputs on the same hardware.
Pins also have limits. The PyTorch documentation says these steps limit nondeterminism “for a specific platform, device, and PyTorch release”, so a framework upgrade is a configuration change and not a no-op.
Two further items belong in the pinned configuration alongside the software stack:
- The sensitivity design itself. One-at-a-time designs leave much of the input space unexplored, so they systematically underestimate uncertainty and misestimate sensitivity. The chosen method, the parameter ranges and the number of samples all belong in the pinned configuration, because sensitivity indices depend on those choices. Convergence also depends on sample count, and plotting sensitivity against the number of evaluations is a common way to check it. A re-run that silently changes any of these is a different analysis.
- The statistical comparison method. This covers the metric, the tolerance and the test used to call two results “the same”. NASA-STD-7009B requires a maintained record of acceptance criteria that explicitly includes criteria for sensitivity.
How do you stop GPU nondeterminism from changing results between identical runs?
Turn on deterministic algorithms, fix every seed, hold batch size and parallelism constant, and re-run only on the same GPU architecture. Where full determinism costs too much, report a distribution across runs and not a single number. The cause is floating-point non-associativity: the order of additions changes the rounded result. Parallel GPU kernels do not guarantee that order unless they are told to.
The effect is large enough to matter for what-if numbers:
- In deep reinforcement learning, with all other sources of randomness controlled, GPU nondeterminism alone compounded into a 12.41% relative standard deviation in final performance (Nagarajan et al., 2018).
- In TensorFlow ResNet training on CIFAR-10, GPU nondeterminism, not weight initialisation or minibatch order, accounted for between 74% and 87% of the standard deviation in scores (Morin and Willetts, 2020).
- Under bfloat16 with greedy decoding, a reasoning model showed up to 9% variation in accuracy when only GPU count, GPU type and evaluation batch size changed (Yuan et al., 2025).
- Thinking Machines Lab (2025) sampled 1,000 completions of one prompt at temperature 0 and got 80 unique outputs. With batch-invariant kernels, all 1,000 were identical.
The research above comes from training and inference workloads. The mechanism, order-dependent floating-point addition in parallel GPU kernels, is not specific to them.
Determinism does hold when the conditions are pinned. EigenAI matched 100% of outputs on the same host, across different hosts with the same SKU, and under ±20% batch-size variance. Its A100 versus H100 comparison matched 0% bitwise. The cross-architecture deviations were around 10⁻⁷ in logits, which the authors attribute to architecture-dependent rounding. The EigenAI authors conclude that bit-exact re-verification needs a separate verifier set per GPU architecture. Separately, the Hawkeye study (2026) found that Hopper Tensor Cores sum all 16 products in a single stage with a 25-bit internal mantissa, while Ampere uses a two-stage process with 24 bits.
Precision is another lever. Yuan et al. (2025) found that FP32 computation gave near-perfect reproducibility across 12 runtime configurations where BF16 was substantially unstable. FP32 roughly doubles memory use and inference time compared with BF16.
None of this comes free, but the cost can be measured. Deterministic operations are often slower, and PyTorch’s own documentation names regression testing as the payoff. Batch-invariant matmuls cost about 20% of throughput compared with cuBLAS in the Thinking Machines work. EigenAI’s deterministic GEMM kernels reached 97–99% of cuBLAS throughput for quantised matmuls on Hopper.
Where determinism is impractical, the residual variance should be documented and the result reported as a spread across seeds. Morin and Willetts recommend comparing distributions of outcomes, not single values that “may seem favourable simply by luck”. A narrow confidence interval suggests a stable result, and a wide one signals sensitivity to experimental conditions.
What to log on each run to show it matches the validated configuration
For every reported number, you should be able to name the code commit, the data version and the environment specification that produced it. The table below lists the minimum record, one row per item.
| Logged item | What it proves |
|---|---|
| Code commit and harness version tag | The figure maps to an immutable point in history |
| Container digest, driver, CUDA and framework versions | The software stack matches the validated one |
| GPU model, architecture and accelerator count | The comparison sits within one architecture’s verifier pool |
| All seeds and determinism flags | Randomness was controlled the same way |
| Declarative configuration: sensitivity method, parameter ranges, sample count, metrics, thresholds | The analysis itself was not quietly altered |
| Input data version and pedigree for the replayed campaign | The replayed data’s version and transformations are on record |
| SHA-256 digests of each persisted artefact (model weights, alloy library) | Any byte-level modification of the artefact shows up |
| Hash of outputs, plus the metrics themselves | Re-runs can be checked by equality and not by judgement |
| Warning and error messages, with explanations | Nothing unexplained was ignored |
| Version identifier and justification for the change being tested | The difference is traceable to a named update |
Artefact integrity and environment drift are separate checks. An unchanged digest cannot show that a dependency changed, and a changed dependency does not prove that model behaviour changed. The Modelstamp authors (2026) treat detected drift as an early warning that may warrant compatibility testing. In their framing it is not proof that outputs changed, and the absence of detected drift is not proof that the environments are equivalent. In a re-verification pipeline, a drift report should trigger the frozen replay. Drift detection alone does not determine whether model behaviour changed.
Full re-verification or a statistical sample after a model or hardware change?
The default is a full replay of the frozen panel. The panel is the reference, and on a deterministic stack each replay is checked by byte equality. NASA-STD-7009B defines regression testing as selective checking that modifications have not caused unintended effects. Make the selection when the panel is built and then keep it fixed, not re-picked every cycle, because a re-picked subset breaks comparability.
| What changed | Re-verification scope |
|---|---|
| Twin model or alloy library, with the same stack and architecture | Full frozen replay, compared with the locked reference and the previous baseline |
| Driver, CUDA, framework or container | Replay the unchanged model first. If outputs are bit-identical to the old baseline, proceed. If not, re-baseline before attributing any difference to the model |
| GPU architecture (for example A100 to H100) | A new per-architecture baseline. Bitwise equality is unavailable, so compare against a pre-agreed tolerance and statistical test |
| Harness, metrics, thresholds or the historical dataset | Re-baseline, because direct comparison with earlier cycles is no longer valid |
| Only hardware cost inputs, for a re-costing | Replay with the new cost inputs recorded as the single changed variable |
A statistical sample is defensible only when a full replay is genuinely unaffordable. Even then, the sampling rule and its seed have to be frozen and logged like any other parameter. Thresholds work best when they are set in advance. SR 11-7 (Federal Reserve, 2011) treats outcomes that consistently fall outside predetermined thresholds of acceptability as the point where adjustment, recalibration or redevelopment is warranted. Setting thresholds across hardware revisions is covered in more depth in our piece on sign-off, isolation and threshold-setting across hardware and model revisions.
When should an independent party run the sensitivity verification?
An independent party should run it whenever the result feeds a decision where somebody has a stake in the answer: a board allocation, a procurement award, a regulatory filing, or any re-costing that a losing option will dispute. SR 11-7 says validation work done by developers must still be critically reviewed by an independent party, by staff who have no stake in whether the model is judged valid. Saltelli et al. note that sensitivity analysis can be done by the modellers or by a third party. They warn that when a third party exposes unexpected sensitivity, the original inference becomes hard to rescue. They also observe that reproduction rarely happens except where an issue is already contested. That is a reason to arrange it before a result is challenged, not after.
A deterministic harness reduces independent checking to a byte comparison. In EigenAI’s procedure, auditors re-execute the declared request, and matching hashes are enough to validate the result. The Hawkeye authors point out that an auditor only needs to reproduce a computation once, so a slow reproduction is tolerable. The independent run should use the same GPU architecture as the reference.
Packaging also decides whether an independent check succeeds. Artefact reviewers often run on hardware unlike the author’s, and artefacts usually fail for mundane reasons such as a missing dependency, a hard-coded path or an unobtainable dataset.
Why sensitivity analyses fail to reproduce, and the checks that catch it
Reproduction fails on two fronts: flawed sensitivity methods and undocumented environments. A review of highly cited sensitivity analyses by Saltelli et al. found that 34% used one-at-a-time methods, that only 8% had models that were clearly linear, and that even on the most generous reading over 20% relied on inadequate methods. On the recording side, the CRA guide names an undocumented environment as the most common reason a result fails to reproduce. It lists unpinned dependencies and reporting a single lucky seed among the recurring failures.
The checks that catch these failures before anyone relies on a result:
- Confirm the design explores the input space. A one-at-a-time design does not, so treat its result as suspect.
- Verify linearity or monotonicity before trusting correlation-based indices.
- Plot each sensitivity index against the number of evaluations to check convergence.
- Re-run the unchanged model on the pinned stack and require bit-identical output hashes.
- Run several seeds and report the spread, because a fixed seed is not proof of reproducibility.
- Before deserialising a persisted model, check its environment against the recorded manifest, as Modelstamp does.
All of this proves consistency, not correctness. The Modelstamp authors draw this line when they separate verification against a reference state from reproducibility in the stronger sense of rebuilding the result from scratch. A frozen harness can show that this year’s twin agrees with last year’s on the same campaign. It cannot show that either version is right for an alloy the mill has never rolled. Saltelli’s distinction applies here: a well-run sensitivity analysis makes an inference worth considering, but whether the inference can “stand in court” depends on auditing the whole modelling chain.
