A simulator or digital twin earns trust once a written statistical method answers four questions. How close is close enough? On which conditions? With how many real runs? And where are the gaps? In practice that means four commitments: agree the accuracy tolerance with whoever makes the decision, state the validity domain the evidence covers, size the real-world runs by power and confidence instead of by round numbers, and report results by stratum with the data gaps written down. The definition of validation from the Society for Computer Simulation already holds two of these. A model is validated within a domain of applicability, and its users have to define the range of accuracy they consider good enough for decision-making. Oberkampf, Trucano and Hirsch add the uncomfortable part: validation never finishes, because a computational model’s accuracy cannot be shown for every possible condition, so what counts as “sufficient” is set by intended use and budget.

So scarce real data is no excuse for skipping the methodology. It is the reason the methodology has to be written down.

How many scenarios are enough to back an ODD coverage claim?

A scenario set is enough when it covers an exposure-weighted distribution of the conditions inside the operational design domain (ODD) to an agreed degree of confidence. A count such as 1,200 tells you nothing on its own. A study proposing coverage metrics for scenario databases puts it directly: a usable coverage metric should be a percentage, because kilometres driven or scenarios simulated can grow without limit, and the number of failures found is no measure either, since the total number of failures is unknown. The same authors split coverage into two types. Do the collected scenarios cover every relevant aspect of the ODD? And do they cover everything that is actually in the driving data?

Their worked case on highD data illustrates this. Applied to highD data, the time-based metric showed that about 75% of time instants were covered by at least one scenario. In the uncovered 25%, no actor matched a listed scenario category, or none was present; adding the category “ego vehicle has no leading vehicle” would bring time coverage to 100%.

Brute-force counting also fails on the road. A RAND report (2016) estimated that test driving alone would need hundreds of millions, and sometimes hundreds of billions, of miles to give clear statistical evidence of autonomous vehicle safety.

Finding the rare classes a random sample misses

Rare but critical scenario classes are the weak point of sampling in proportion to exposure. Stellet et al. note that undirected sampling meets small critical subspaces only with small probability, so test duration grows as critical events get rarer. Importance sampling deals with this by raising exposure to risky events and reweighting the probability measure, and the estimate stays theoretically unbiased whichever sampling distribution is used. Only the number of tests saved changes. In one study, the TrimFlow method reached the required accuracy on a car-following collision-rate estimate after about 1.0 × 10^5 scenario tests, against 7.2 × 10^5 when sampling from the naturalistic driving distribution. That is a reduction of 86.1%. Either way, compare the sampled distribution with the exposure-weighted target, class by class, so under-sampled critical classes show up as gaps you can name.

The sequence we would put in front of an OEM runs in this order:

1. Define the ODD’s distribution of operating conditions from exposure data. The cited work states that statistical quantification of risk needs reliable naturalistic driving data, so the quality of the exposure data limits how credible everything that follows can be.
2. Tag every scenario against the ODD description. Where the ODD is written to ISO 34503, the tags listed in ISO 34504 can be used.
3. Compute coverage percentages for both coverage types.
4. Fill the critical gaps by adjusting the sampling distribution, not by naive sampling.
5. Stop when a confidence threshold agreed in advance is met, and record that threshold as a negotiated value, not a universal one.

On justifying scenario counts to regulators, there is no shared standard to copy. Stellet et al. conclude that no standard notion of coverage exists for automated driving, because acceptance criteria are informal and the ODD is unstructured. The coverage-metrics authors likewise leave appropriate threshold values to future work and context. The threshold is therefore something you agree with the specific OEM or regulator, and the methodology is what makes that conversation concrete. The transfer importance sampling authors note that ISO 21448 requires risks to be monitored continuously during an AV’s operation.

How do we define “within tolerance” for several output traces at once?

“Within tolerance” for several traces has to be a single joint criterion fixed in advance: a stated error metric per channel, plus one acceptance rule that applies to all channels together, reported on every replay run. TechnoLynx builds this kind of replay-and-compare validation for a manufacturing digital twin. The work covers agreeing with the process owner what “within tolerance” means across, for a rolling mill, the roll-force and temperature traces together, choosing the statistical error metric, building the replay of archived campaign inputs on the target compute environment, and documenting the procedure so a later campaign can rerun it unchanged. Where the replay has to run on a new compute environment, the build is delivered through the AI Porting & Deployment Pack, and the engagement starts by fixing the joint tolerance definition before anything is coded.

Why joint, rather than one tolerance per channel or an average? Marginal checks can be fooled. A study on subtrace-conditional validation showed opposing biases from two misspecified input models cancelling each other out, so the marginal output distribution looked consistent with observed data. Conditioning on partial historical traces correctly detected that both inputs were wrong. Per-channel bands have a related blind spot: they can pass a run where moderate errors in force and temperature happen at the same moment in a way that matters physically.

The metric has to be named, because each one behaves differently on the data:

• Root mean square error gives an average error per point in the original units.
• Normalised Euclidean distance is unitless and can use replicated simulations for the standard deviation, but it blows up where that standard deviation is close to zero.
• Dynamic time warping measures the distance between two sequences, and one online digital twin validation split its threshold into a warning level and a failure level.
• The area validation metric measures the area between the simulated cumulative distribution and the experimental empirical distribution in the units of the response quantity. A value of zero means no evidence of disagreement, not perfect agreement.

Thresholds should come from observing the system under normal conditions. In a gantry-crane study, each threshold was set as the maximum metric value seen across 50 simulated replications, and across 10 normal operating trajectories for the runtime checks. The authors note there is no ready-made answer, and that a more sensitive threshold can be had by accepting a stated rate of false alarms under normal conditions. The same study found that metrics which looked fit in a simulation study for one fault type did not all detect a different real fault. Its authors recommend checking every metric against every type of fault where time and resources allow. When several channels are each tested, a correction such as Bonferroni or Holm controls the family-wise error rate. If the requirement is phrased as “a stated share of runs fall inside the band”, a statistical tolerance interval makes it precise: it bounds a proportion P of the population at confidence 1 − α, and for distribution-free intervals, fixing any two of sample size, confidence and coverage determines the third.

That also answers what a vendor’s methodology should make you report. You need the named metric per channel, the threshold rule, the number of replications behind it and the joint pass/fail rule. Oberkampf and colleagues hold that validation metrics and their success and failure criteria should be specified together with the application requirements. A methodology that rests on visual overlay or average error alone does not meet that bar.

A worked example on another mill’s historical campaign is worth asking for, with the right expectations. It tests whether the procedure reproduces on a second site rather than being tuned to this campaign. Its thresholds will not transfer exactly if process conditions differ.

Can we trust the twin on conditions it was never validated against?

Only inside the domain of applicability it was validated for. Outside the domain of applicability, the SCS definition requires a new validation effort. Parts of the system the model leaves out cannot be validated at all, because they are never simulated. Where the validation domain and the application domain do not overlap, Oberkampf, Trucano and Hirsch argue that inference can only be made using both physics-based models and statistical methods. They note that choosing experiments just on the invalid side of the boundary sharpens it, with greater leverage when the model’s predictions are used to design those experiments. A validation framework for simulated counterfactuals models the move from validated to unvalidated conditions as an added extrapolation penalty, and describes that assumption as indispensable, only partly verifiable and able to fail silently. The same thinking runs through autonomy safety cases: a paper on out-of-distribution detection for safety assurance notes that once the system is outside its ODD, its safety guarantees cannot be assumed to hold.

For a rolling-mill twin validated on one campaign, a pass holds only within that campaign’s domain of applicability. Trusting it on an alloy grade never trialled needs a documented validity domain built from further historical campaigns across varied grades, or from limited physical trials on intermediate grades. A grade whose parameters fall inside the validated range carries far less extrapolation risk than one outside it.

The automotive case below is why we state the domain in terms of phenomena, not just parameter ranges. In a study linking several test setups, about 70% of collisions in the 3D software-in-the-loop setup happened when a low sun blinded the camera. That effect mainly appeared in the 3D setup and was hard to predict from the simpler 2D setups.

Who can build and document a replay validation for our digital twin?

TechnoLynx builds the validation around a simulator or digital twin. That covers the replay pipeline that runs archived inputs through the model, the joint tolerance and metric definitions it is scored against, the validity-domain analysis that bounds how far the result extends to untested conditions, and the documentation that lets the next campaign rerun the procedure. On building in-house or bringing in a specialist, we would weigh whether the replay must run on a new compute environment and whether the method has to survive an external reviewer. Whoever you choose, ask to see the tolerance definition and threshold rule in writing before any results, plus a replay on an earlier campaign. The implementation is delivered through the AI Porting & Deployment Pack, starting from the tolerance definition.

Are 20 real runs enough to detect a sim-to-real gap?

Twenty real runs can support a sim-to-real comparison only if each one is paired with the simulated runs of the same scenario and the minimum detectable gap is worked out before anyone claims equivalence. A protocol for evaluating small improvements runs baseline and variant under identical conditions, analyses the per-pair differences with a bias-corrected and accelerated (BCa) bootstrap interval and a sign-flip permutation test, and recommends exploiting pairing and stating decision rules explicitly. The BCa interval adjusts for bias and skew in the resampled differences, which is why it is useful when the number of pairs is small.

Bootstrapping does not create information; it gives an honest interval from the pairs you have. In its experiments, the same protocol never declared significance with three paired runs, even when the BCa interval was strictly positive, and its authors report it becomes less conservative from five runs upwards.

Stratifying the 200 simulated trials to match the obstacle scenarios of the 20 real ones is the right instinct. It has a cost, though: each stratum’s simulated sample shrinks, and pairing only means something if the simulator reproduces those scenarios faithfully. Two published methods combine the large simulated set with the small real one. Prediction-powered inference combines abundant biased simulator data with a small unbiased real sample to produce unbiased outcomes, and a context-aware variant estimates the bias separately per context, trusting the twin where it is reliable and correcting it heavily where it is not. Transfer importance sampling uses a cheap, scalable setup for exploration and then estimates risk with a small number of runs in the trusted, costlier setup, reducing both bias and variance.

Worked example: the smallest gap 20 paired runs can resolve

Assumptions: the per-pair difference between real and simulated outcomes is roughly normal, as the t-based method assumes, the test is two-sided at α = 0.05, and the target power is 0.8.

• For a t-test, the required sample size is approximately n ≈ c / ζ², where ζ is the effect size in standard deviations.
• At α = 0.05 and power 0.8, two-sided, c = 7.87 (the case known as Lehr’s rule).
• Solving for ζ with n = 20 gives ζ ≈ √(7.87 / 20) ≈ √0.39 ≈ 0.63. Twenty pairs detect, with 80% power, a gap of about 0.63 standard deviations of the per-pair difference.
• If the gap that matters operationally is 0.3 standard deviations, n ≈ 7.87 / 0.09 ≈ 87 pairs are needed.
• The approximation is stated for samples of about 30 or more, so at 20 the result is only a rough guide.

If the minimum detectable gap is bigger than the gap that matters, the 20 runs cannot support an equivalence claim. Failing to detect a difference with an underpowered test is not evidence of no difference. At that point the options are more real runs, a tighter paired design or a narrower claim.

What is the cheapest way to get more real validation data before the window closes?

One source of extra real validation data is logging from operations that are already running, not new planned trials. The continuous-validation study does exactly this. Instead of designing a separate validation experiment, it treats data from the system’s normal operation as the real-world side of the comparison. It delimits experiments by routine cycle or by time window, and warns that the number of cycles chosen smooths measures such as the mean. The trade-off is control. Replications from normal operation depend on the system repeating an identical operation, with no control over the time between repeats. Logs from the field also keep their value beyond validation: Stellet et al. stress that recorded field data should be reusable in closed-loop simulation so that scenarios found there can be retested reproducibly.

Take an agricultural obstacle-detection programme with a harvest window as an example. If other farms are operating this season, we would treat retrofitting logging there as the fastest lever, because nothing new has to be scheduled. The weakness is ground truth. Estimating missed detections needs reference data of the environment, and because a safety-critical miss can only occur in a dangerous situation, estimating miss rates is not symmetric with false alarms. Passively logged encounters therefore need a labelling pass to a stated standard before they count as validation evidence.

A hardware-in-the-loop rig is a sensible parallel track for rare obstacle cases that the logs still lack, provided nobody treats it as paired real data. The multi-setup study found that relying only on the cheaper test setups produced biased risk estimates. When the window closes, it closes. Whatever the logs and the rig have not covered by then goes into the gap log, not into the claim.

When is synthetic or replayed data acceptable evidence for a thin stratum?

Synthetic or replayed data can stand in for a thin stratum only when some real data from that stratum exists to check it against, and only through a method that corrects for its bias. An analysis of synthetic data says that validation data from the target distribution is essential, and notes that a few hundred real samples can be enough to validate even when they are too few to train on. Without that, the reasoning goes in a circle: the lack of target data that motivated augmentation also makes it impossible to check whether augmentation helped or hurt. Naive pooling breaks statistical validity. Treating synthetic samples as real and running standard statistics on the combined set gives confidence intervals with the wrong coverage and inflated Type I error. Bias-correcting procedures such as prediction-powered inference restore valid intervals, but only for specific estimators. One digital twin study reports good performance with only 1 to 10% of the training data being real.

Resampling your own data cannot widen what that data covers. Purely bootstrapped augmentation, with no extra source information, cannot make a dataset more representative of conditions the base data never covered.

How should sensitivity be reported by stratum so weak cohorts stay visible?

Report sensitivity separately for each device type and data-completeness tier rather than as one pooled figure. Set the completeness thresholds before looking at results, and report cohorts below threshold as their own labelled stratum, with counts and the reason they could not be fully evaluated. Pooling hides failures. Research on hidden stratification in medical imaging found that models with high overall performance can consistently miss rare subsets, with relative performance differences of over 20% on clinically important subsets. The same logic applies to simulators. In one framework for LLM-based simulators, conditional calibration across demographic strata detects differential fidelity.

A clinical-trial addendum on estimands and sensitivity analysis gives the reporting discipline. Pre-specify the main analysis and its sensitivity analyses in the protocol. Change one assumption at a time, so that any difference can be traced to its cause. Report whether each analysis was pre-specified or done post hoc. On missing data, the same text says data counts as missing only relative to a particular estimand, and recommends studying a plausible range of assumed differences between missing and observed values to see whether the conclusions hold. It also warns that integrating data sources without specifying what each one estimates can be a source of bias.

Two caveats keep this honest. Over-stratification shrinks each stratum’s sample, and estimate spread falls only with the square root of the run count. And a threshold chosen after seeing results can be set to hide a weak subgroup, which is exactly what registering it in advance prevents.

Replay-based verification of historical events, such as replaying a flood forecast over past storms, needs one cleaned, time-aligned forcing-and-observation dataset per event before any replay runs. Every gap is flagged: minor random gaps are interpolated by a documented method, and structural gaps (a gauge down for part of the storm) are masked from skill scoring and reported per event. Andam-Akorful et al. (2021) note that missing values handled badly can bias observed streamflow records and feed uncertainty into hydrological model outputs, particularly when the record is short.

Is our historical dataset representative enough? That depends on the statistic

Sufficiency depends on the statistic being estimated, how variable it is, and how far operating conditions have moved since the data was recorded. TechnoLynx takes on the perception version of this question. That means defining the operating envelope a recorded dataset has to cover (dust density, sun elevation and soil type for a field robot), auditing the footage against it, deciding which gaps need synthetic augmentation and which need more real capture, and then running and packaging a robustness test of the embedded perception pipeline against pass/fail thresholds. The work starts with the envelope definition. The three cases below each need different evidence and a different person to sign them off.

Twelve years of daily returns and a Sharpe ratio

For signal-to-noise ratios such as the Sharpe ratio, a power rule links the true effect size, the sample size and the error rates. With the effect measured per square-root year, the sample size comes out in years. At α = 0.05 and power 0.8, two-sided, c = 7.87. By the large-sample rule, 12 years separates from zero only an annualised signal-to-noise ratio of about √(7.87 / 12) ≈ 0.81 or more. The textbook distribution of the Sharpe ratio assumes normal returns, skew does more damage than kurtosis, and positive autocorrelation makes the standard error of the mean an underestimate. Picking the best of many backtested variants inflates the winner’s ratio further, so the interval has to account for how many variants were tried. Should the data-length limitation go to the committee? We would say yes, stated as the interval around the estimate, not as a footnote.

One afternoon of dusty footage

How long the footage runs is not a measure of coverage. As with scenario counts, we would measure the share of the operating envelope the recording covers. The low-sun collisions in the multi-setup study show how strongly a perception failure can concentrate in one corner of the envelope. Synthetic dust and backlight augmentation can extend a limited set, but only with real footage from the target conditions held back to validate it. The envelope itself has a limit: the same safety-assurance paper notes that open-world data distributions cannot be fully specified at design time.

A decade of concessional-lending data

The methodology note should run sensitivity analyses on its assumptions about the lending terms, changing one assumption at a time. Regime shifts that no sensitivity analysis captures should be named as a remaining risk. Sign-off belongs with the supervisor and the bank’s model-risk function, not with the engineering team.

Checklist: what a defensible validation plan states before the first run

• The decision the model supports, and the range of accuracy the decision-maker accepts for it.
• The validity domain, with boundary experiments just on the invalid side where they are feasible.
• A coverage metric expressed as a percentage of the ODD or operating envelope, with a threshold agreed with the reviewer.
• Named validation metrics with success and failure criteria, and a family-wise correction across channels.
• Thresholds derived from replications under normal conditions, with the accepted false-alarm rate stated.
• A real-run count from a power calculation, together with the minimum detectable gap it implies.
• The pairing scheme between real and simulated runs.
• The main analysis and its sensitivity analyses, pre-specified before results are seen.
• Missing data defined against the estimand it affects.
• Where synthetic or replayed data is used, the real target-stratum data used to check it.
• How requirements are monitored, and violations logged, once the system is in operation.

The plan also becomes the baseline for the next validation. Oberkampf and colleagues describe regression testing as running a fixed set of tests against an established baseline and treating any deviation as either a bug or a reason to set a new baseline. Stated tolerances and validity domains belong in the validation packages and release-readiness reviews that production AI reliability produces. What no plan settles on its own is the coverage threshold itself. For scenario coverage and twin tolerances alike, no shared standard exists yet, so the number has to be argued for and agreed with the person who will sign it off.

FAQ

What is a bootstrap method in statistics?

The bootstrap estimates the uncertainty of a statistic by resampling the observed data with replacement many times and recomputing the statistic on each resample. The BCa variant adjusts the resulting interval for bias and skew, which helps when samples are small.

How does a larger sample size affect the confidence interval?

A larger sample narrows the interval, but slowly. The relative spread of an estimate falls with the square root of the number of runs.