Two engineers stand in front of the same model and disagree about whether it is done. One of them ran the held-out evaluation, watched accuracy clear the agreed threshold, and called it validated. The other points out that nobody ever wrote down what the model was supposed to do under fog, partial occlusion, or a sensor that drifts a few degrees off calibration — so the number, whatever it is, answers a question nobody asked. Both of them are using the word validated. Neither of them means the same thing. This is the single most common confusion in production AI reliability work, and it has a precise resolution. Verification and validation are not two names for one check. Verification asks whether you built the system to the agreed specification; validation asks whether the built system meets the real-world need it was commissioned for. Collapse the two into one accuracy figure on one held-out set and you lose the ability to tell which question you actually answered — which is exactly the gap that surfaces at handoff, when an engineering reviewer is asked to sign against a claim and finds there is nothing reproducible behind it. What Verification and Validation Mean in Practice Both terms come from systems and software engineering, and the cleanest informal phrasing has survived for decades: verification is “did we build the system right?” and validation is “did we build the right system?” The first measures against a written specification. The second measures against the need the specification was meant to capture — which is not the same thing, because specifications are written by humans who do not know everything about the operating environment in advance. For a production AI model this distinction becomes sharp rather than philosophical. Verification of a vision model checks that the inference graph matches the agreed architecture, that the preprocessing pipeline normalises inputs as specified, that latency stays inside the contracted budget on the target hardware, and that the model produces the documented outputs for documented inputs. Every one of those is checkable against a written spec with a reproducible pass/fail. Validation is the harder question: when this model runs against the actual distribution of cameras, lighting, products, and edge cases it will see in the field, does it deliver the outcome the buyer commissioned it for? You can pass verification cleanly and fail validation badly, and that combination is where most disappointing deployments live. The reason the two terms drift together in casual use is that a single held-out test set looks like it answers both at once. It does not. A held-out set drawn from the same curated data the model trained on verifies that training converged and generalises within that distribution. It says almost nothing about whether the distribution itself matches operating conditions nobody specified. We see this pattern regularly: a model that “validated” at 97% on the internal set lands on the line and the number that matters — caught defects per shift, false-stop rate — moves in a direction the held-out set never predicted. Why the Difference Matters at Handoff The divergence point is not a lab artifact. It shows up at handoff, and it shows up in two failure shapes that look like opposites. The first is the verified-but-not-validated model. Internal tests pass. The architecture matches the spec, latency is in budget, the eval harness is green. Then it meets an operating condition nobody wrote into the specification — a packaging change, a new camera firmware, a seasonal lighting shift — and it underperforms against the real need while still satisfying every documented requirement. Nobody lied; the specification was simply incomplete, and verification can only ever be as good as the spec it checks against. The second is the unverified-but-demoed model. It looks right in a demo, the stakeholders nod, and it ships on the strength of a convincing screen. But there is no reproducible evidence behind the demo — no documented input set, no recorded pass/fail, no signer. When it misbehaves three weeks later, there is nothing to roll back to and no record of what “working” was supposed to mean. A demo is a performance; verification is a record. Getting V&V right is what turns reliability work into something an engineering reviewer can sign against rather than take on trust. When verification and validation are separated explicitly, a buyer can trace every reliability claim to a specific test and a specific signer. That traceability is the actual ROI: it cuts the handoff disputes and re-work cycles that come from each release re-litigating what “validated” meant, and it surfaces specification gaps during the work instead of after deployment. The recurring cost of skipping the separation is paying for that discovery in production, where it is most expensive. The Four Methods of Verification, Mapped to AI Evidence Classical systems engineering recognises four methods of verification — inspection, analysis, demonstration, and test. They map cleanly onto the evidence a production AI model should produce, and the mapping is worth making explicit because it tells QA and engineering what kind of artifact each requirement needs. Method What it checks AI model evidence it produces Inspection Conformance you can confirm by examining the artifact Model card, architecture diagram, data-lineage record, config and dependency manifest Analysis Properties derived by reasoning, not direct observation Worst-case latency analysis, numerical-precision impact study, failure-mode reasoning, fairness analysis across documented subgroups Demonstration Function shown under nominal operation Recorded inference run on a documented input set with expected outputs, end-to-end pipeline smoke test Test Measured behaviour against quantified pass/fail criteria Eval-harness scores, regression-suite results, latency and throughput measurements under defined load Two cautions on this table. Demonstration is the weakest method for an AI system precisely because it is the most persuasive — a clean demo is what the unverified-but-demoed failure mode rides in on, so demonstration evidence should never stand alone where a test could measure the same thing. And the test row is where most of the reproducible weight should sit: the eval harness and regression suite are the artifacts an engineering reviewer reads first. Our work on regression testing for production AI and catching model drift before release covers how those test artifacts are built so they stay meaningful across retrains. A Concrete V&V Example for a Production Model Take a defect-detection model on an electronics assembly line. The verification side asks a set of spec-anchored questions: does the model accept the camera’s native resolution and the documented preprocessing, does it return the agreed defect classes with confidence scores, does it run inside the per-frame latency budget on the deployed GPU, and does the inference graph match what was reviewed? Each answer is a recorded test result against a written requirement — benchmark-class evidence, because the test is named and reproducible. The validation side asks a different set: against a labelled set drawn from this line’s production stream — not the curated training data — does the model catch the defect types that actually cost money, at a false-stop rate the operations team can live with, across the shift-to-shift variation in lighting and component mix the line really sees? That requires measuring under real operating conditions rather than spec-sheet conditions, which is the reasoning behind treating empirical, workload-bound measurement as the reference standard rather than a curated-set proxy. Validation evidence is only as trustworthy as the realism of the conditions it was gathered under. The handoff dispute this example prevents is concrete. Without the split, “the model is validated” means whatever the last person to say it intended. With the split, the buyer can ask: show me the verification record — which requirement, which test, which signer — and separately, show me the validation evidence against real line data. Two questions, two evidence trails, two signers. That is the difference between a claim and a demo. Who Signs, and What a V&V Plan Specifies Up Front Responsibility for the two activities is usually different, and naming it early prevents the most common organisational failure: everyone assuming someone else owns validation. Verification is typically owned by the engineering team that built the system, checking its own work against the spec, often with QA executing the test suite. Validation is co-owned with whoever holds the real-world need — the customer’s operations team, a domain expert, and in regulated settings a reviewer or regulator who decides whether the evidence is sufficient. The signer of a validation claim should be the party who carries the consequence if it is wrong. A software verification and validation plan written before work starts should make those answers explicit rather than discovering them at handoff. At minimum it specifies the requirements that verification will check and the method (inspection, analysis, demonstration, or test) for each; the operating conditions validation will measure against and where that real-condition data comes from; the quantified pass/fail thresholds for both; who signs each result; and how the whole package is re-run when the model is updated. That last point matters more for AI than for conventional software, because the system under test changes every time you retrain. How Does V&V Change When the Model Is Retrained? Conventional software is verified once unless the code changes. An AI model changes its behaviour every time it is retrained on new data, even when no line of code moves — which means verification and validation are not one-time gates but a loop that re-runs on every model update. The regression suite re-checks that previously-correct behaviour did not silently break; the validation set is re-measured against current operating conditions; and drift telemetry decides when a retrain and re-validation are even needed. The relationship between the trigger and the re-run is exactly what model drift detection in production AI — signals, thresholds, and telemetry is built to make observable. Treating V&V as a one-off “validated and shipped” event is the assumption that quietly expires the first time the data moves. This is also where the discipline connects to the artifact it populates. V&V is not the validation pack — it is the practice that fills the pack’s eval harnesses, regression suites, and release-readiness evidence. The discipline of the engineering practice that catches AI failures before customers do is the broader frame; the production AI reliability technology page describes the artifact those V&V activities produce. And the emerging TEVV — Test, Evaluation, Verification and Validation — expectations now appearing in AI assurance guidance are essentially this same loop made mandatory: the demand that an AI system carry reproducible, signable evidence for both questions before it is trusted with consequence. FAQ How does verification and validation work, and what does it mean in practice? Verification and validation are two distinct checks, not one. Verification asks whether you built the system to the agreed specification — matching architecture, latency budget, documented outputs — using reproducible tests against written requirements. Validation asks whether the built system meets the real-world need it was commissioned for, measured against actual operating conditions rather than curated data. In practice they produce separate evidence trails with separate signers. What is the difference between verification and validation for an AI system, and why does it matter at handoff? Verification checks the system against its specification (“did we build it right?”); validation checks it against the real-world need (“did we build the right system?”). It matters at handoff because the two failure shapes are opposites: a verified-but-not-validated model passes every internal test yet fails operating conditions nobody specified, while an unverified-but-demoed model looks right in a demo but has no reproducible evidence behind it. Separating the two lets a reviewer sign against a claim instead of taking it on trust. What does a concrete software verification and validation example look like for a production model? For a defect-detection model on an assembly line, verification confirms the model accepts the documented input, returns the agreed classes, and runs inside the latency budget — each a recorded test against a requirement. Validation measures the model against labelled data from that line’s real production stream: does it catch the defects that cost money at an acceptable false-stop rate across real shift variation? Two questions, two evidence trails, two signers. Who is responsible for verification versus validation — engineering, QA, customer, or regulator? Verification is typically owned by the engineering team checking its work against the spec, often with QA executing the test suite. Validation is co-owned with whoever holds the real-world need — the customer’s operations team, a domain expert, and in regulated settings a reviewer or regulator who decides whether the evidence is sufficient. The signer of a validation claim should be the party who carries the consequence if it is wrong. What evidence does each V&V step need to produce so it is reproducible and signable? Each step needs a named requirement, a verification method, a quantified pass/fail result, and a signer. Verification maps onto the four classical methods — inspection (model cards, lineage), analysis (worst-case latency, precision impact), demonstration (recorded runs), and test (eval-harness and regression results). Validation needs evidence gathered under real operating conditions with documented thresholds. A demo without a documented input set, recorded result, and signer is not evidence. How does the V&V approach change when the model is updated or retrained? An AI model changes behaviour on every retrain even when no code changes, so V&V is a loop that re-runs rather than a one-time gate. The regression suite re-checks that previously-correct behaviour did not break, the validation set is re-measured against current operating conditions, and drift telemetry decides when a re-validation is needed at all. Treating “validated and shipped” as permanent is the assumption that expires the first time the data moves. When V&V is treated as two separable questions with their own evidence and their own signers, the argument at the start of this piece never happens — because “is it done?” splits into “does it match the spec we agreed?” and “does it meet the need we measured?”, and both have a recorded answer. The harder question is the one the specification could not anticipate: which operating condition will the field reveal that nobody thought to write down, and is the validation loop watching closely enough to catch it before a customer does?
