Independent model validation means the people judging a model had no part in building it, no role in using it and no stake in whether it gets approved. US supervisory guidance has said since SR 11-7 that reporting-line separation can support independence, but independence should be judged by actions and outcomes. A credible validation therefore shows four things: a validator who sits outside the development chain, results the validator reproduced rather than read about, tests the developer did not pick, and a sign-off claim no broader than the evidence under it.

TechnoLynx’s part is the engineering underneath that: reproducible test harnesses, role-separated approval records and evidence packs that a model-risk committee or auditor can read and sign against, built as engineering work within our AI governance and trust practice and never as regulatory sign-off or certification. We would start by scoping which models and which claims the evidence has to support, because that decides how much testing is enough.

What makes independent model validation genuinely independent

Under SR 11-7, validation should be done by people who are not responsible for developing or using the model and who have no stake in whether it is found valid. The same guidance treats independence as a means rather than an end: it exists to line up the validator’s incentives with the goal of finding problems. A model owner cannot also be a member of the validation or audit teams that provide effective challenge.

Independence also needs influence. SR 11-7 expects validators to have explicit authority to challenge developers and escalate findings, and it says that influence can be shown by a pattern of actual cases where models or their use changed because of validation. FDIC examination procedures go further and ask whether validators are incentivised to withhold findings or are being leaned on by the business line or developers.

What “independent” means under SR 26-2, PRA SS1/23 and the ECB guide

In the US, SR 26-2 (issued jointly by the Federal Reserve, OCC and FDIC on 17 April 2026) describes effective challenge as work by people with the expertise to challenge, enough independence to stay objective, and the organisational standing to force change. It names misaligned incentives between reporting lines, such as development and validation groups, as a conflict of interest to manage. An OCC bulletin rescinds OCC Bulletin 2011-12 and states that the guidance does not set enforceable standards or prescriptive requirements. The OCC also points out that its model risk guidance does not require community banks to perform annual model validation.

In the UK, PRA SS1/23 is relevant to UK-incorporated banks, building societies and PRA-designated investment firms with internal model approval for regulatory capital. Its validators should not take part in development or have a stake in approval, and the PRA accepts separate reporting lines and incentives, or an independent party reviewing the test results, as ways to secure that impartiality. Firms with internal model approval are expected to go further and demonstrate independence through separate reporting lines for validators on one side and developers and owners on the other.

The ECB’s three organisational arrangements

The ECB guide to internal models sets out three possible organisational arrangements and expects the choice to reflect the institution’s nature, size, scale and risk complexity.

Arrangement	What it looks like	When it fits
(a) Separate units, different senior managers	Validation and development report to different members of senior management	Expected for large and complex institutions, which the ECB says should implement the most robust option
(b) Separate units, same senior manager	Units act separately below a shared senior manager	Allowed with additional requirements under Delegated Regulation (EU) 2022/439, regularly checked by internal audit
(c) Separate staff, same unit	No separate unit, but validators are different people from developers	Choice depends on nature, size and complexity; the EBA describes the same three setups for IRB validation

Extensive use of machine learning in internal models may push an institution towards a more robust arrangement than its size alone would suggest.

Where reporting lines cannot be separated, Lloyd’s guidance for insurers is a useful pattern: it accepts an independent intermediate step such as a review committee independent of the shared executive, periodic external validation, or the involvement of another director.

Can a small bank or fintech outsource only the validation step?

Yes, but responsibility stays with the institution. The EBA allows operational validation tasks to be outsourced internally or externally while stating that responsibility stays with the validation function. The PRA says boards and senior management remain ultimately responsible for model risk under any outsourcing arrangement. Even a vendor of validation tooling describes its role under SS1/23 as supporting evidence, with the validation judgement and sign-off remaining the firm’s.

The sources set these practical conditions:

• A written scope and an internal owner. SR 11-7 expects a clearly written, agreed scope of work and a designated internal party who can evaluate the results, verify the scope was completed, track issues to closure and fold the work into the firm’s framework.
• The business does not set the scope. FDIC procedures call it a fundamental control that the business line and developers cannot develop or modify the validation scope.
• The outside validator is genuinely outside. The ECB applies its independence rules to third parties and expects people who have done development or risk-control work not to validate within the same institution until a prudent cool-off period has passed. A third party that developed a rating system should not also validate it.
• Pay is part of the independence test. Lloyd’s expects the independence assessment of external validators to cover their remuneration structure and other mandates with the firm.
• In-house knowledge is kept. The ECB expects institutions to retain enough competence to take direct control of an outsourced activity in extremis.

The alternative is a second internal team, separated from development under one of the ECB’s three arrangements. Our view: keep validation in-house when you can staff a function that has its own line and real authority to block a model; outsource when you cannot, or when your team is small enough that every validator once worked on the models being reviewed. How to set up that internal function from scratch is covered in our piece on building an evidence-backed model validation function.

Diagnostic checklist: is the separation real or only on paper?

A policy document is necessary and nowhere near sufficient. Before an auditor asks, check that you can produce each of these:

1. A documented separation policy naming roles, expertise, authority, reporting lines and continuity, plus controls on external validators.
2. A reporting chart showing the validation and modelling reporting lines, or the governance, committee or external step where they report to the same person.
3. Instances where validation findings changed or stopped a model.
4. Remediation confirmed by the validators or an independent third party, not by the developer who fixed it.
5. A model registry gate that records a distinct approver: in one AWS reference workflow, a registered model sits at PendingManualApproval until an authenticated member of an approver group managed in Active Directory approves it. Another AWS reference implementation assigns model approval for staging and production to a separate role with its own permissions.
6. A plan for independence over time, since a reviewer who re-reviews changes they recommended loses some independence cycle by cycle.

Why does the validator have to reproduce the results instead of reviewing the documentation?

Because reading documentation tests the writing, and reproducing results tests the model. The EBA records as good practice that the validation function reproduces the documented estimation steps or reviews the code directly, and treats documentation as usable only if a third party can replicate the estimation and arrive at the same results. Where the validator calculates its own metrics, it is expected to compare them with the developer’s. The PRA counts models whose results cannot easily be repeated among those carrying higher model risk, and expects development documentation detailed enough for an independent expert to replicate parameter estimation and model results.

For machine learning the bar is concrete. The ECB expects documentation of ML components to allow replication, including parameters and hyperparameters, stored random seeds where training relies on simulation, and in some cases the exact ordering of training observations. In engineering terms that means pinned library versions, a fixed container image (Docker or equivalent) and recorded seeds. A validator who cannot rebuild the model has learned what the developer believes, not what the model does.

What calibration, out-of-time and out-of-sample evidence should a credit-risk validator expect?

Out-of-time and out-of-sample performance tests are part of the rigorous statistical process the EBA handbook expects for validating IRB models. The validator should expect the sample used to measure generalisation to be independent of the training sample, and back-testing to compare forecasts with actual outcomes over a period not used in development, at a frequency matching the forecast horizon.

Calibration validation compares forecast default probabilities with realised default rates. The named tests are:

• the binomial test, applied to one rating grade at a time;
• the chi-square (Hosmer-Lemeshow) test, which checks several grades together;
• the normal test, a multi-period test for a single grade;
• the traffic-lights approach, a multi-period back-testing tool for a single grade.

For machine-learning predictors, calibration plots and Brier scores are commonly used to assess whether predicted probabilities are reliable. The PRA adds comparison against challenger models where practicable. Where data is too thin for the tests to be robust, the EBA expects the validator to state its level of confidence in its own results. We would have the validator choose the out-of-time window itself rather than inherit the developer’s holdout, since a split chosen by the developer is part of what is being validated.

Model verification versus model validation

In PRA SS1/23, process verification is one part of model validation. PRA SS1/23 defines validation as independent review of conceptual soundness, process verification of the implementation, and ongoing performance monitoring. Process verification checks that inputs are accurate and complete and that the code is correct, change-controlled and auditable. Validation asks the wider question: is the model performing as expected for its design objectives and business use? Even a correct model can be used inappropriately outside its design scope.

What an independent validation report has to contain

SR 11-7 expects validation reports to state which aspects were reviewed, highlight potential deficiencies across a range of conditions, say whether adjustments or compensating controls are needed, and open with an executive summary covering model purpose, results, major limitations and key assumptions. Combining the regulator and market texts, an audit-ready report carries:

Section	Why it is there
Scope, including whether it is limited by the model’s risk grade	Listed in FDIC exam procedures
Opinion on data relevance, assumptions and expert-judgement overlays	Listed in FDIC exam procedures
Deficiencies ranked by materiality, their combined effect, and confidence in the conclusions	EBA handbook expectation
Tests performed, sample sizes and data preparation	EBA handbook expectation
Known limitations with impact and materiality, and the tools used	Lloyd’s validation pack contents
Actions taken to secure independence and objective challenge	Lloyd’s validation pack contents
Examples of escalation from failed tests and their outcomes	Lloyd’s validation pack contents
A clear outcome, including return for redevelopment where material flaws are found	A rejected model is not used until issues are resolved

The report should be understandable by a knowledgeable third party.

Sampling for rare, high-consequence failures outside credit

In vision inspection or sim-to-real robotics the failures that matter are rare, so a sample drawn at the natural rate contains few of them. We would write a separate sampling plan for each of the two cases below.

Rare defect classes in a vision-inspection sample

Estimating sensitivity on a rare defect class needs enough examples of that class, so the sample must be sized and stratified or enriched for it. The arithmetic is unforgiving: by the rule of three, a model that misses none of 30 defect images still has a 95% upper bound of about 3/30, or 10%, on its miss rate. One published anomaly-detection study with 38 anomalies among 1,687 samples (2.25%) noted that standard asymptotic assumptions may fail at that imbalance, used exact Clopper-Pearson intervals for sensitivity, and stratified every cross-validation fold to keep anomalies represented. The same study reported post-hoc power of 0.85 to detect sensitivity differences of 0.15 or more at α = 0.05 with those 38 cases.

Enrichment does not bias specificity by itself. Sensitivity is estimated among true defects and specificity among true negatives, so oversampling defects leaves specificity unbiased as long as the negatives are a random sample of the negative population. Pooled accuracy and positive predictive value do depend on prevalence, so any figure mixing the two classes must be reweighted to the production defect rate, or the enrichment silently inflates it.

Rare divergences in sim-to-real validation

We would power a sim-to-real correlation package for the collision-relevant divergences, not only for mean behaviour. We would set the coverage plan per scenario class and per model type rather than reuse a template built for fewer model types. We would flag scenario classes with too few physical runs to estimate a correlation as such, with a stated minimum-coverage threshold below which no claim is made, rather than average them into a headline number.

How narrow should a “certified for third-party use” claim be?

A “certified for third-party use” claim should name exactly what the evidence covers: a specific robot and firmware combination, a class of robots, or a class of scenarios, never an unqualified “certified simulator”. TechnoLynx can do the engineering behind a scoped claim: cataloguing which robot and firmware combinations and scenario classes the correlation package actually covers, defining the test matrix each scope needs, running the sim-to-real measurements on the target simulator stack, checking the statistical confidence per scope segment and drafting the scope-limit language. The first step is that catalogue of what was really tested; the certifying decision stays with whoever issues the certificate.

We would choose the scope from the evidence. If the package tested one combination, the claim covers that combination. A class-level claim is defensible only when correlation evidence spans several configurations or scenarios and shows they behave consistently. Every published correlation metric should carry the conditions it was measured under (robot models, firmware versions, scenario classes, environmental ranges) and the explicit exclusions, so a third party cannot read it more broadly than it was earned. Those statements go stale as firmware and scenario catalogues change, so they need an owner and a review trigger.

Banking guidance states a similar principle. SR 11-7 expects that where validation cannot be done because of data paucity, the gap is documented and reported to users and senior management. The PRA counts using a model outside its intended purpose as a model exception, for which firms need escalation processes. A scoped certificate is that same discipline applied to a robot simulator: the opinion is bounded by what was tested, and it remains the issuer’s own.