Independent model validation is credible only when you can show two things. First, the people who challenged the model had no part in building it and no stake in whether it passed. Second, their challenge produced evidence that an approver, auditor or examiner can check without taking anyone’s word for it. US supervisory guidance says independence should be judged by actions and outcomes, and that separate reporting lines are one way to support it. One sign of it is a validation function with the authority to challenge, and a record of models that actually changed because a validator objected.
That means the work has two halves: a separation that holds up when someone examines it, and an evidence package built for the people who have to sign off. Each half is weak without the other.
What makes a validator genuinely independent of the model’s builders?
The Federal Reserve’s SR 11-7 guidance (April 2011) says validation should generally be done by people who are not responsible for developing or using the model and who do not have a stake in whether it is found valid. It also says independence is not an end in itself. Its job is to line up incentives with the goals of validation, and there may be ways to keep reviewers objective other than separate reporting lines. The guidance hangs all of this on “effective challenge”, which depends on incentives, competence and influence. Incentives are stronger when the challenge sits further from development and is backed by sensible pay practices and culture.
SR 26-2 now supersedes SR 11-7 (and SR 21-8 on BSA/AML models), drawing on fifteen years of supervisory experience. According to a 2026 comparison of the two letters, the revised guidance was issued jointly by the Federal Reserve, OCC and FDIC on 17 April 2026. It keeps the framework of development, validation and governance and adds explicit proportionality. SR 26-2 describes effective challenge as work by people with the right expertise, enough independence to stay objective, and the organisational standing to force change. It also names conflicts of interest between reporting lines, such as development and validation groups, as something governance must address.
One sentence in SR 26-2 is easy to misread. It says the quality of validation depends on how rigorous and effective the review is, rather than on how the risk function is organised. We read this as a restatement of the old outcomes test, not a licence to drop separation. A rigorous review by someone whose bonus depends on the model owner still fails, because rigour on paper means little when the reviewer’s incentives point the other way.
Independence tests a validator has to pass
| Test | What meeting it looks like | What fails it |
|---|---|---|
| No development or use role | The reviewer neither built nor operates the model and gains or loses nothing from the verdict. | A reviewer who helped tune the model, or whose team consumes its output. |
| No stake through pay or budget | The reviewer’s pay, promotion and budget do not depend on the business that owns the model; second-line validation typically reports through the Chief Risk Officer. | Validators reporting to the model owner, with findings left open for years. |
| Competence | Enough technical skill and familiarity with the line of business to give an objective, critical opinion. | A generalist reviewer who cannot follow the method being challenged. |
| Authority and influence | Explicit authority to challenge developers and escalate, reporting to someone with enough stature to get issues fixed. | Recommendations that can be ignored without consequence. |
| Control of scope | Neither the business line nor the developers can set or change the validation scope. | A scope written by the team being validated. |
| Freedom from pressure | Validators are not rewarded for holding back findings, and the culture does not push models through. | Deadlines that make a finding feel like a career risk. |
For a failure-prediction model, the independent reviewer who challenges its failure-mode assumptions must meet those same tests. That means no development role, no stake in the outcome, and enough domain knowledge of the failure modes to recognise a wrong assumption. Validation may be done by an independent internal team as well as by an outside firm. We see the main risk as a shared management chain that makes the separation nominal.
Can another internal team do the validation, or do we need an outside firm?
Either can satisfy the requirement. The FDIC’s examination module says validation may be done by internal staff or by third parties, as long as they are independent of model development or subject to an independent review that deals with the conflict of interest. The question is not simply “internal or external”. It is whether the chosen party passes the tests above, and whether you can show that it does.
An internal team is acceptable when its reporting line, budget owner and incentives are separate from the developers’ all the way up, not just at the level of team names. External firms meet the same independence bar, not a lower one. The ECB guide to internal models applies its internal validation independence requirements to third parties as well. It expects anyone who has done development or risk-control work for an institution to stay out of validation there until a prudent cool-off period has passed. Where a third party developed a rating system, the ECB expects the institution to make sure that party does not also validate it.
Hiring an outside validator does not hand over the responsibility. The PRA’s SS1/23 says boards and senior management remain ultimately responsible for model risk under outsourcing or third-party arrangements. SR 11-7 expects a clearly written, agreed scope of work whenever external resources are used for validation. It also expects a designated internal party who can evaluate the results, verify the scope was completed and track the issues found.
Does it have to be a big-name audit firm?
SR 26-2 puts the weight on rigour and expertise. We read that as meaning a specialist boutique can qualify when its independence is documented and its scope and credentials match what the reviewing party requires.
In a banking-as-a-service programme, we would count the sponsor bank among those reviewing parties. A 2025 analysis by the law firm Shumaker notes that the chartered bank, not its fintech partner, bears the regulatory consequences of fair-lending and BSA/AML failures, whatever the partnership agreement says. The same analysis sets out a contract covenant giving the bank a right to require independent third-party testing of the fintech’s BSA/AML programme at set intervals. A sponsor bank therefore sets its own bar, which may be stricter than SR 26-2. If its policy names a required tier of audit firm, that requirement overrides the general argument about competence.
What the validator should check on a gradient-boosted credit model
SR 11-7 gives three core elements: conceptual soundness, ongoing monitoring and outcomes analysis. Outcomes analysis can cover the accuracy of estimates and the model’s ability to rank-order. In credit terms, that means calibration and discrimination. Ongoing validation checks that changes in markets, products or clients have not created new model limitations. Back-testing should use a period that was not used to develop the model. The bank, not its fintech partner, bears the regulatory consequences of fair-lending failures. We would treat a validation that leaves out any of the three core elements as incomplete, however independent it was.
Which SR 11-7 independence criteria will examiners test, and how should reporting lines be documented?
Examiners test whether validators are independent in practice and whether policy requires it. The FDIC module tells examiners to check that validators are required to be independent of development and that reporting lines are reasonable. It also tells them to check whether validators are incentivised to withhold findings or are improperly influenced by business-line staff or developers. In the US, SR 26-2 is now the reference text, with the same independence and effective-challenge principles.
TechnoLynx can build the evidence side of that record through AI Governance and Trust: validation runs that can be reproduced, findings tied to the model version, and boundaries between development and validation environments that can be shown to hold. The work starts with a map of where the two teams currently share code, data or people. Judging whether an arrangement satisfies a particular examiner stays with your model risk function.
On documentation, SR 11-7 expects policies that set out roles, staff expertise, authority, reporting lines and controls on external resources. It also expects reporting lines and incentives to be clear, with potential conflicts identified and dealt with. The PRA goes further and expects each lifecycle role to be documented with the organisational independence and seniority it requires. Incentives belong in the record too. SR 11-7 suggests performance evaluation standards that reward the quality of validation and the degree of critical review.
Should the separation go in the engagement letter? SR 11-7 already expects a written, agreed scope for external work. We would add the reporting-line separation, conflict declarations and any prior work for the vendor or model owner to that same document. Doing so costs little, and a claim of independence that the engagement letter never mentions is hard to defend later.
If examiners have flagged independence before, we would not assume a new structure clears the finding. We would answer the specific gap named in the exam letter, show how the separation works in practice, and confirm with the examiners before go-live where any doubt remains.
Three ways to separate validation, from the ECB guide
Outside the US, the ECB lists three organisational arrangements for keeping internal validation independent of model development:
| Arrangement | Structure | ECB view on fit |
|---|---|---|
| (a) | Development and validation in different units reporting to different members of senior management. | Large and complex institutions should use the most robust option. |
| (b) | Different units reporting to the same member of senior management, acting separately below that level. | Comes with additional regulatory requirements that internal audit should check regularly. |
| (c) | Separate staff within the same unit. | Possibly suitable for small entities that are not systemically important. |
The ECB adds that extensive use of machine learning in internal models may call for a more robust arrangement. The PRA expects firms with internal-model approval to show separate reporting lines for validators on one side and developers and owners on the other. For ECB internal models that rely heavily on machine learning, that points towards option (a), not away from it.
Firewalling shared code, data and people between developers and validators
This covers two different governance problems. One is separation of duties between development and validation inside an institution. The other is who may sign off a certification stamp when some reviewers have ties to vendors.
Through AI Governance and Trust, TechnoLynx builds the evidence that the separation between development and validation holds, from reproducible validation runs to environment boundaries a reviewer can check, and the first step is a map of where the two teams currently share code, data or people.
The place to start is an inventory of what development and validation share (repositories, feature pipelines, data snapshots, credentials and people). That inventory turns into access controls and a documented exception list that a reviewer can check.
Separation of duties between development and validation
The Federal Reserve’s own supervisory stress-test models give a clear example. Its Committee on Supervisory Model Oversight (COSMO) helps guide how the models are developed, implemented and used, and the Fed’s Inspector General stated in 2022 that, for this reason, COSMO is not an independent validation group. Independent validation sits with the System Model Validation group, set up in 2016. The group is staffed by dedicated full-time employees who do no supervisory modelling, and it reviews conceptual soundness, code and model performance. It judges whether a model meets its stated purpose but does not decide whether the model is used. The lesson carries over: a body that helped build a model cannot validate it, however senior that body is.
An earlier arrangement shows the pressure points. In 2015 the development oversight group and the validation unit both reported to the same division director. Validation relied on about 85 reviewers working part-time alongside their day jobs. The Inspector General flagged risks such as dependence on key people and insufficient scrutiny of models. The function then moved to dedicated reviewers under a permanent committee at the Federal Reserve Bank of Minneapolis.
For shared code and data, a useful pattern is the Fed’s own: its supervisory models use detailed portfolio data from firms but generally not the firms’ models or estimates. Validators should take the raw data and leave the developers’ derived features, notebooks and scoring code behind, or at least record each place where they cannot. A firewall only works if access controls enforce it; a policy that says “do not use the dev repo” is not a firewall. SR 11-7 allows developers to do some validation work, provided an independent party reviews it critically and carries out additional work of its own.
The formal record has several layers. The model inventory should name the people responsible for development and for validation. Internal audit evaluates whether the validators are objective, competent and properly placed in the organisation. It does this without duplicating validation itself, and internal audit staff who carry out validation activities should not also assess the overall framework.
Who can sign off a certification stamp when reviewers may have vendor ties
MLPerf is a useful comparison, mostly for how it differs from model validation. Its results are reviewed by a committee that can depart from the rules or exclude a submission by a two-thirds vote of the submitters. The rules say the review exists to ensure fairness, not to argue over details in order to disqualify competitors. We read that as independence through adversarial peer review, not through separation, relying on rivals checking each other.
A single-vendor verification stamp has no rivals in the room, so the MLPerf model does not transfer directly. For that case, written conflict-of-interest disclosure and recusal should be required for any reviewer with ties to the vendor whose work is being stamped. Where the internal technical reviewers cannot all be cleared, the stamp needs an external review panel.
What compute does a validation team need to re-score a model without the developers’ pipeline?
The validation team needs an environment it controls, sized to re-run the full population the model scores, not just the developers’ sample. What these reproduction requirements imply for infrastructure is our own reading. The ECB expects machine-learning documentation to allow replication down to hyperparameters, random seeds and, in some cases, the order of training observations. If you run that replication on the developers’ cluster with their credentials, you bring back the dependency the separation was meant to remove.
LynxBenchAI publishes measured results for AI accelerators, which a second-line team can use to size a dedicated re-scoring or challenger-training environment before committing to hardware or cloud reservations. It measures the hardware, not your model, so the first step is to compare published results for candidate devices against your re-scoring workload profile on the LynxBenchAI results. How to read benchmark results belongs to LynxBenchAI’s own methodology, not to this page.
A short isolation checklist for the validation environment:
- A separate cloud account or on-premises allocation, with credentials the first line does not hold.
- A read-only snapshot of source data taken at a recorded point, not a live link to the developers’ feature store.
- Validation code written or independently reviewed by validators, with every reused developer component listed as an exception.
- Capacity for full-portfolio re-scoring within the validation turnaround.
- For vendor models without code access, capacity for the sensitivity analysis and benchmarking that SR 11-7 says banks may have to rely on instead.
Isolation has a cost: a duplicate environment and replicated data. It also guarantees nothing if the data pipelines or code still come from the first line, which is why the checklist reaches beyond hardware.
What has to be in the approval file before a model counts as independently validated?
The approval file needs the validation findings attached to the specific model version, before approval. Those findings cover what was reviewed, under which conditions, what deficiencies appeared and whether compensating controls are warranted. SR 11-7 says validation reports should set out the aspects reviewed and any deficiencies, with a clear executive summary that states the model’s purpose and its major limitations and key assumptions.
TechnoLynx does the engineering behind that file through AI Governance and Trust. The work covers deciding which metrics and conditions the gate has to evidence, then building the calibration, parity or stress tests. It then wires the metric computation into the registration or test pipeline and attaches the resulting package to the model registry entry. The first step is to agree with the approvers which conditions the gate must cover. In our view, production AI reliability is an engineering discipline separate from accuracy, and independent validation evidence works as a release gate, not a document produced after the decision.
Contents of an independent validation evidence package
| Item | What it shows the approver |
|---|---|
| Written scope, validator identity and reporting line | Who validated, against what agreed scope, and why they count as independent. |
| Results for conceptual soundness, ongoing monitoring and outcomes analysis | That all three core elements were covered. |
| Findings graded by severity, with developer responses | That critical issues were answered before implementation; in the Fed’s own programme, developers must respond to every level 1 finding first. |
| Validator confirmation of remediation | That fixes were checked by validators or an independent third party, not just claimed by developers. |
| Documented limitations and validation not performed | That gaps were reported to users and senior management, as SR 11-7 requires. |
| Inventory entry | Purpose, inputs, last update, expected validity period, policy exceptions and named responsible individuals. |
| Replication record for ML components | Parameters and hyperparameters, plus random seeds where training relies on simulation, sufficient to replicate training. |
| Findings tracker | Date, status, remediation timeframe and owner for each finding. |
Calibration evidence attached to the model version
Expected Calibration Error (ECE) measures the gap between predicted probabilities and observed frequencies across probability bins. The Brier score is the mean squared error of probabilistic predictions. We would compute both inside the registration pipeline on the validators’ own holdout set, store them as metadata on the registry entry, and gate approval on an agreed threshold. Report them by segment as well as in aggregate, because a model can pass in aggregate while miscalibrated in a segment that matters.
Parity evidence for a quantised model
When an FP32 model is quantised to INT8 for deployment, run both precisions on the same lab-validated challenge set and compare per-class sensitivity against the floor validated in the lab. A numerical parity test and a field challenge-set test answer different questions: one asks whether the arithmetic held, the other whether detection held on real inputs. SR 11-7 expects material changes to models to be validated. We would treat parity evidence as valid only for the hardware, firmware and driver combination it was tested on. Re-validation when the model or hardware changes is where TechnoLynx’s Production AI Reliability work runs that harness again.
Linking thermal and power behaviour to detection reliability
For edge devices that throttle under heat, the evidence has to connect each operating condition (temperature, power state, clock speed) to detection results under that condition, with failures per class and per condition, not an average across conditions. We would not assume an evidence package built for one regime, such as model risk management, satisfies a notified body or the FDA. It has to be mapped to each reviewer’s requirements explicitly.
Review findings in the approval record before go-live
SR 11-7 says that if validation finds significant deficiencies, the model should not be used, or used only under very tight constraints, until they are resolved. SR 26-2 accepts that an urgent business need may justify use before validation finishes. In that case, sound practice means paying closer attention to the model’s limitations. In both cases the findings, constraints and decision belong in the approval record before go-live, not in a meeting note.
When a vendor’s FDA 510(k) summary describes the model one way and pilot results suggest another, an independent technical audit should reconcile the two before the committee decides. Any duty to notify the FDA, other regulators or the vendor then depends on how material the discrepancy is, and that is a question for regulatory counsel, not engineers.
Who can build the validation evidence our approvers and examiners will accept?
TechnoLynx builds the engineering side of independent validation evidence: reproducible validation pipelines, calibration and parity harnesses, isolated validation environments, and evidence attached to each model version in the registry. An engagement starts by agreeing which conditions the approval gate has to evidence and mapping where development and validation currently share code, data or people. The work then moves to implementation and to re-validation when the model or hardware changes. The catalogue offer is AI Governance and Trust.
It is not the right engagement in every case. TechnoLynx does not act as your second-line validator of record for an SR 26-2 credit model. It does not give regulatory sign-off or a fair-lending legal opinion. And if we helped build a model, we should not validate it, on the same reasoning as the ECB’s cool-off principle. If you have a separated second line with its own statistical tooling and pipeline engineers, building in-house is sound. If the sponsor bank names a tier of audit firm, or the question is legal materiality, a specialist model-risk firm, an audit firm or counsel is the right route.
Reproducing the developers’ results instead of reviewing their documents
A credible independent validation reproduces the key results on data and code the developers did not control. We would include calibration curves, out-of-time and out-of-sample performance, and a comparison against a challenger model. SR 11-7 asks validators to evaluate the developmental evidence and to carry out additional analysis and testing where needed. It also says ongoing monitoring should include benchmarking. Reading the developers’ notebook and agreeing with it is not validation.
Vendor models follow the same rule. Banks are expected to validate their own use of vendor products, relying more on sensitivity analysis and benchmarking where code is not accessible. Under SR 26-2, vendor documentation is an input to validation, not evidence of it. A vendor that refuses to disclose enough for validation has produced a finding in its own right. That is a good reason to negotiate a right to commission independent testing before signing, much as sponsor banks write such rights into fintech contracts.
Hybrid and novel architectures, such as an XGBoost model feeding a neural component, need the method adapted: challenger models suited to the architecture, plus explicit tests on edge cases and under-represented segments that a generic checklist misses. The ECB says validation of ML-based models may need to happen more often so that material deficiencies are caught in reasonable time. A team of three data scientists cannot validate its own model. For a team that size, an outside validator is the realistic route, and the institution still has to oversee that validator’s work. The same logic of separation and scoped sign-off applies well beyond banking, and our piece on what counts as independent model validation across industries follows it there. In radiology AI, for example, independent external validation is described as a way to provide reassurance on performance, calibration, value and risk.
We would confirm that a validator has no reporting-line or financial tie to the vendor or the pipeline team with two checks. For internal candidates, review the org chart and the budget owner, since shared incentives do not always show on a chart. For external candidates, ask for disclosure of any prior work for the same vendor. The ECB’s expectation that anyone who has done development work sits out a prudent cool-off period before validating gives a clear standard for the second check.
Which explainability evidence satisfies examiners for complex models, and what is still unsettled?
The clearest expectation concerns replicability. The ECB expects all internal models to be documented so that a qualified third party can understand the methodology, assumptions, limitations and use, and replicate development and implementation. Where code cannot be inspected, SR 11-7 points to sensitivity analysis and benchmarking. Validation reports need an accessible synopsis of the model and the results. The ECB also asks internal audit to consider whether its audit techniques are relevant and meaningful for ML-based models.
We see the method as still unsettled. We read the supervisory texts cited here as setting outcomes, not techniques. Until that settles, we would treat explanations as claims to validate in their own right. That means checking their stability across retrains and seeds, and checking them against the segment and challenger results already in the package, so that the explanation sits on top of reproducible evidence instead of standing in for it. An examiner may disagree with your choice of explanation method. They have little ground to object if the method is documented, its limits are stated, and its output can be re-run by someone who never worked on the model.
