A model validation sign-off report for numerical parity should let an automotive OEM check four things without coming back to you. The first is a pass/fail verdict for each output, derived from layer-wise cosine similarity and maximum error against thresholds stated in advance. The second is the exact dataset version and the model or engine build that verdict applies to. The third is a record showing tuning and validation data were kept apart, and the fourth is a named person who authorised the result. Getting a harness to print cosine and max-abs-error numbers is the easier part. The harder part is turning those numbers into a document that a reviewer at the OEM can trace, reproduce and sign against.
What should a model validation sign-off report contain for the OEM to accept it without back-and-forth?
Start by finding out whether your customer already has its own report format. IATF OEM members publish their own customer-specific requirements for IATF 16949, with separate documents from groups including Stellantis, Volkswagen AG, Renault Group and Volvo Group. In the PPAP framework, customer-specific requirements are a named element that sits on top of the AIAG standard, and Ford, GM and Stellantis each define their own.
The decision logic is simple. If the OEM publishes an acceptance checklist for this kind of evidence, mirror it item by item, including the way it names dataset versions and artefacts. If it does not, use a superset template. Leaving out an artefact the reviewer expected costs a whole rejection cycle, while including one they did not need costs a few pages.
Whichever template you use, the report is one piece of evidence inside the safety case, not a document that stands alone. Under ISO 26262, the safety case records the hazards, the safety goals, how safety requirements were refined, and the verification and testing results that show the requirements are met, so that the implemented software can be traced back to the hazards it mitigates. The maximum error rate an ML component may have follows from its safety requirements and their ASIL. At every ASIL level, each failure on the test set has to be assessed, which is a heavy requirement for ML components because their test error is typically non-zero. AMLAS, the assurance guidance for ML components in autonomous systems, requires validation results to be documented. Automotive SPICE v4.0 asks for bidirectional traceability between verification results and verification measures. In a parity report, that means every pass/fail line points to the test that produced it, and every test points to the line it supports.
A report structure to use when the OEM has no checklist
The laboratory reporting rules in ISO/IEC 17025:2017 clause 7.8 make a sound skeleton. They are a model structure borrowed from test laboratories, not something the OEM requires.
| Report section | What it holds for a parity report | Basis in the reporting rules |
|---|---|---|
| Title and unique report ID | An identifier that is never reused, with every page recognisable as part of the complete report and a clear end marker | Unique, non-duplicable identification |
| Method | The harness, comparison metric, error statistic and tolerances used | Identification of the method used |
| Item under test | Reference model, converted model or engine, and their build hashes | Unambiguous identification of the item |
| Sampling | Which frames or sequences were compared, and how they were chosen | Reference to the sampling plan and method |
| Dates | Date the comparison ran and date the report was issued, both stated even when they are the same day | Performance and issue dates |
| Results | Per-output values and verdicts, with units where they apply | Results with units of measurement |
| Deviations | Anything run differently from the stated method, or excluded from it | Additions, deviations and exclusions |
| Scope statement | The results apply only to the builds and data tested | Results relate only to items tested |
| Authoriser | Name of the person who authorised release | Identification of the authorising person |
Do we need to trace every test image back to the original validation dataset version?
Trace every result to a versioned dataset snapshot, and make sure that snapshot can be resolved down to individual files. Once that holds, any single image can be traced without a separate per-image register. The AI Model Passport framework recommends that each dataset gets a unique, resolvable identifier that links back to the data unambiguously. DVC gives a concrete way to do this: its .dvc placeholder files, which are versioned in Git, record an MD5 hash for each tracked file or directory, plus its size and, for a directory, the number of files in it. MLflow’s dataset tracking keeps a link to the original data source and hashes datasets used in its evaluation workflow.
There is one caveat. Organisations version data in different ways, so the identifier written in the report has to be one the OEM’s reviewers recognise and can check against their own records.
Which supporting artefacts should go with the report?
No source we hold says what OEMs typically demand as attachments, so what follows is our judgement of a sensible superset, not a published requirement. We would attach:
- full harness logs for every comparison run, including the configuration each run used;
- confusion matrices for the safety-relevant classes, for both the reference model and the deployed model;
- per-frame diffs of detection outputs, sampled on a plan agreed with the OEM in advance;
- a layer-wise error summary showing where in the network any divergence enters.
Confusion matrices earn their place because AMLAS warns that a single performance measure is rarely enough for an ML component, that a classifier usually involves a trade-off between false positives and false negatives, and that relying on one measure can produce systems that pass acceptance yet behave in unintended ways. Existing tooling already produces layer-wise summaries. Texas Instruments’ TIDL Layer Trace Inspector, for example, compares reference and test layer outputs and produces a network-wide error summary with MAX, absolute MAE and relative MAE per layer. Even with every artefact attached, acceptance still depends on the OEM’s contractual and regulatory terms, which no generic template captures.
Raw numbers, plots or a pass/fail summary: what goes in the signed parity report?
We would include all three, in layers. Verdicts go on the first page, the numbers that produced them go in the body, and plots go in an appendix for the reviewer who wants to see the shape of the error. The Federal Reserve’s SR 11-7 model risk guidance (2011) is written for banks, but it describes effective validation reports as including a clear executive summary with the model’s purpose and an accessible synopsis of results, including major limitations and key assumptions. TechnoLynx supports this through its AI Governance and Trust offer, which builds approval evidence packs that survive an external auditor, a procurement committee or a regulator.
How layer-wise cosine similarity and max error become pass/fail criteria
Write the comparison into the report as a rule a reviewer can recheck: the metric, the tolerance, and the statistic the tolerance is applied to. NVIDIA’s Polygraphy shows what that rule looks like in practice. Its distance-metric comparison checks each output against an L2 tolerance (default 1e-5) and a minimum cosine similarity (default 0.997, which corresponds to a cosine distance of 0.003), and both values can be set separately for each output. Each result is stored as a pass/fail flag together with the measured L2 norm and cosine similarity. Polygraphy’s tolerance-based comparison results also record the minimum absolute and relative tolerances needed for the outputs to count as equivalent.
The error statistic changes what “pass” means, so the report has to state it. Polygraphy can apply tolerances elementwise, or to the maximum, mean, median or a quantile of the absolute and relative errors, and it describes the max check as the strictest of these. Its default elementwise check fails an element only when that element exceeds both the absolute and the relative tolerance. That makes it less stringent than the max check, which can fail when one element breaks the absolute tolerance and a different element breaks the relative one. The same atol and rtol can therefore pass under one statistic and fail under another, so a report that leaves out the statistic cannot be reproduced.
Bind the verdicts to the safety-relevant outputs, and attach layer-wise results as diagnostics. Polygraphy’s own documentation presents per-layer comparison as the way to find where error enters when the final outputs do not match. Per-layer thresholds belong in the pass/fail table only if the OEM has agreed to them. The tolerance band itself (which classes it covers, which distance ranges, and FP32 against FP16 or INT8) is a separate agreement that has to be settled with the OEM before the comparison runs, not after the numbers come in.
Recording dataset versions and model or engine build hashes
The dataset snapshot identifier from the traceability section covers the data side. The model side needs a lineage of its own: a hash for each artefact along the conversion chain (training checkpoint, exported ONNX graph, compiled engine), with the toolchain versions that produced each step. AMLAS requires a model development log that documents and justifies the key decisions in the learning process, including the choice of development toolchain such as TensorFlow or PyTorch. Extending that log to cover the conversion steps keeps the report and the log consistent.
Who inside the supplier needs to sign the validation report before it goes to the OEM?
TechnoLynx builds the report so that at least one named authoriser signs on the report itself. ISO/IEC 17025 requires results to be reviewed and authorised before release, by personnel specifically authorised to do so. It also requires the authoriser to be identified even when the same person ran the test. Interpretations, such as a statement that a divergence is acceptable, may only be released by personnel authorised to give opinions, and the basis for the opinion must be documented. A Tier-1 PPAP work instruction from TI Automotive shows the pattern for physical parts: its dimensional results carry an OK/NOK qualification for each item and an authorisation signature with name, job title and date.
Who else signs comes from the customer relationship. The Development Interface Agreement under ISO 26262-8 assigns each safety activity, verification and confirmation measures included, to a party marked responsible, approving, supporting, informed or consulted in a RASIC matrix, and the DIA work products include joint confirmation reports and a final sign-off package. The required independence of reviewers and assessors follows an ASIL-to-independence mapping (I1, I2, I3). The PPAP Part Submission Warrant carries sign-off from both supplier and customer. SR 11-7 makes the general point clearly: validation should be done by people with no stake in the outcome, and any validation done by developers should be critically reviewed by an independent party.
Showing the tuning set and the validation set never overlapped
Proving separation after the fact needs records kept at the time the data was split and used. An audit trail cannot be rebuilt from memory. Leakage between such sets is a documented risk. Kapoor and Narayanan (2022) surveyed fields adopting ML and found data leakage errors affecting 329 papers across 17 fields, in some cases producing wildly overoptimistic conclusions. In automotive perception specifically, Volvo Cars and Chalmers researchers note that improper splitting, duplication or the inclusion of future information can give misleadingly high validation results while hiding poor generalisation.
Published ML assurance guidance describes what the separation should look like. The ISO 26262 adaptation for ML proposes that a procedure for producing training, validation and test sets from the dataset is specified. AMLAS requires three separate datasets (development, internal test and verification) and reserves the verification set for model verification. It requires that information about the verification data be hidden from developers, that internal test data was not used to create the candidate model, and that a log record the measures taken to keep verification data away from the development team.
Records that make the separation auditable
| Record | What it shows |
|---|---|
| Split procedure | The specified procedure for producing the training, validation and test sets |
| Content hash of each set at the time of the split | The sets have not changed since they were assigned |
| Preserved evaluation set | Practitioners in implementation roles stressed keeping a preserved evaluation set and running automated similarity checks before new data enters existing datasets |
| Per-sample role and timestamps | FG-Trac shows the idea: its audit interface reports each sample’s assigned role (train, validation or test), the timestamps of traceable events and the training epoch in which it was last used |
| Tamper-evident log digests | FG-Trac anchors sample-level events in Merkle-root digests so the trail can be verified independently |
| Verification access log | The steps that kept verification data away from developers |
A cryptographic ledger is not where we would start. We would start with hashed, write-once evaluation sets and access logs, and add stronger guarantees only where the OEM or an assessor asks for them.
Backtesting a replacement model against the cases the old one missed
After missed detections in the field, the people who decide whether to trust the model again should get a retrospective record. It should show the replacement matches or beats the current model on historical cases, with the misses broken out as a named subset rather than averaged into the total. SR 11-7 describes testing documentation as covering the purpose, design and execution of test plans, summary results with commentary, and detailed analysis of informative samples. It also notes that validation can set thresholds for acceptable error by analysing the distribution of outcomes around predicted values. The missed cases are exactly those informative samples.
There is a catch that ties back to separation. AMLAS warns that any single sample can be encoded into training and a model found that avoids that specific failure, without becoming robust to the wider class of failures the sample belongs to. If the misses informed the new model, report them as development evidence and back the claim with fresh, sealed cases of the same kind. Otherwise the backtest measures memory, not generalisation.
Who can build the parity evidence pack for our OEM sign-off?
TechnoLynx builds the engineering evidence behind a parity sign-off through its AI Governance and Trust offer: harness outputs turned into pass/fail criteria, dataset and build traceability, separation records and backtests against historical misses, assembled for the OEM’s review. If you are deciding between doing this in-house and bringing in a specialist, the deciding question is whether your team already has a harness and a versioned data pipeline. If it does, the work is mainly evidence design. If the regression checks still run by hand, the Production AI Monitoring Harness builds the repeatable comparison and logging first. When choosing any partner, ask to see how they record the error statistic and build identity, since those two details decide whether a report can be reproduced.
The scope has firm limits. This is engineering evidence for release readiness, not safety certification or regulatory sign-off, and it does not replace your in-house QA or functional safety organisation. Signing is still done by your authorised personnel and your DIA counterparts, and an independent functional safety assessor remains the right route for the confirmation measures your ASIL requires.
Every new engine build reopens the report
A sign-off covers one build against one dataset snapshot, so we would issue a new report for a new compiler version, a re-quantisation or a different target GPU. ISO/IEC 17025 requires any change to an issued report to be clearly identified with its reason, and a full replacement to carry a new unique identifier and a reference to the report it replaces. On the customer side, the DIA includes assessing supplier change requests for revision triggers, ASIL impact and re-confirmation before a change is approved. Planning the report as a template that the harness fills in on each build is what keeps that cost from growing with every release.
FAQ
What is cosine similarity?
Cosine similarity is the cosine of the angle between two vectors. It ranges from -1 to 1: 1 means the vectors are identical or parallel, 0 means they are orthogonal, and -1 means they point in opposite directions. In Polygraphy it is one of the distance metrics used to compare the outputs of two runners.
What is the purpose of using a validation dataset?
In the ISO 26262 adaptation for ML, the validation set is used to assess how well the trained model generalises and to select hyperparameters. AMLAS likewise keeps development data apart from the verification data reserved for model verification.
