Model validation documentation holds up on the hard cases when a qualified outsider can open the approval file, see what the model does, where it underperforms, how it has changed and where its data is incomplete, and then replicate or challenge the work. The ECB’s guide to internal models sets this test: documentation should let a qualified third party independently understand a model’s methodology, assumptions, limitations and use, and replicate its development and implementation. Judged against that test, an elevated false-positive rate, a lost data segment or years of informal recalibration are not things to keep quiet about. Each one is a record that needs writing.
A weakness on its own need not sink an approval. Canada’s OSFI, in Guideline E-23, says a model may be approved despite identified weaknesses or limitations, provided compensating mitigants are in place or the stakeholder group justifies using a model with known limitations. What damages a file is a weakness the reviewer finds before the file mentions it, because the question then changes from “is this bounded?” to “what else is missing?”
Model validation documentation for the hard cases
| Hard case | What the approval file records | What it must not do | Supervisory anchor |
|---|---|---|---|
| Known limitation (for example, more false positives in one segment) | A metric for each segment with confidence intervals, plus a mitigating control or use restriction | Drop the segment from scope without saying so | OSFI E-23; PRA SS1/23 |
| Recalibration history | Rationale, data and method change, independent review or validation report, approval, version record | Leave a change without its rationale or version record | NAIC predictive-model white paper; Regulation (EU) 529/2014 Art. 8 |
| Key-person dependence | Documentation detailed enough to replicate the model, kept as corporate memory | Leave the method in one analyst’s head | OCC 2000-16 (rescinded); ECB guide |
| Regime change in historical data | Limitation statement, adjustment or added conservatism, sensitivity analysis | Exclude the awkward period without analysis | EBA GL/2017/16; SR 11-7 |
How do we document an elevated false-positive rate on corporate-action days?
Measure it separately and report the gap with its uncertainty. Split the evaluation data into corporate-action days and normal days, compute the false-positive rate in each group, and put a confidence interval on each figure. The difference between the two groups, with its uncertainty, is the limitation statement. The false-positive rate is false positives divided by all actual negatives, so each group needs enough genuine negatives to give a stable denominator.
TechnoLynx runs this under its Production AI Reliability offer as a stratified subgroup analysis on the production model. The work starts from a curated test set of corporate-action days and ends with the measured evidence and the limitation statement packaged for the approval file.
Supervisory text supports reporting a range rather than a single number. SR 11-7 (the US Federal Reserve’s 2011 guidance, since superseded) notes that model uncertainty can be quantified through the confidence interval around a statistical point estimate, and that a range of outputs helps avoid spurious precision. The same guidance expects testing to find the situations where a model performs poorly or becomes unreliable. A false-positive spike around corporate actions is exactly that kind of situation.
When corporate-action days are too rare to give a tight interval
Corporate-action days are rare, so the subgroup can be small and its interval wide. A study of binomial confidence intervals for rare events argues that for small proportions the margin of error has to be judged relative to the proportion itself. It recommends keeping the relative margin of error between 0.1 and 0.5, because higher values lead to imprecision and poor coverage. In the authors’ case studies the Wald interval had notably poor coverage, while the Clopper-Pearson, Wilson and Agresti-Coull intervals did better in one scheme but still needed larger samples for a tight margin. If the corporate-action sample is too small to settle the question, the honest file says the gap is suspected but not yet bounded, and names how many more corporate-action days of data would settle it.
To show a failure pattern is bounded and not just noticed, combine four things. The test set is fixed before results are seen. Every corporate-action day in the window is included. The corporate-action calendar defines the grouping mechanically. And the gap is shown to persist across more than one period.
Exclude corporate-action days from scope, or document a degraded-accuracy zone?
Documenting a bounded degraded-accuracy zone with a mitigating control is usually the more defensible route, because supervisory texts describe such controls. The OCC’s current supervisory guidance on model risk management (Bulletin 2026-13) describes informing stakeholders of limitations and setting controls such as limits on model use or closer performance monitoring. The PRA’s SS1/23 goes further. It expects firms to consider restrictions when significant deficiencies are found, including allowing use only under strict controls or mitigants, or barring the model from specific purposes. A manual-review flag on alerts raised on corporate-action days is one such control. SS1/23 also makes model users accountable for taking known limitations into account when they use the output.
Exclusion is still an option, but it is a use restriction too, and it needs its own evidence: corporate-action days are identified reliably and nothing in that window is scored silently. How a conditional approval with a control attached gets through review is covered in our piece on what it takes to get a model past independent validation.
Who can build the evidence behind a documented model limitation?
TechnoLynx builds the measured evidence behind a documented limitation under its Production AI Reliability offer: the stratified evaluation, the false-positive rate for each group with its confidence interval, and a limitation statement and control recommendation that a model-risk committee can sign against. TechnoLynx frames the subgroup analysis on the production model, builds the stratification pipeline, checks whether the measured gap is statistically supportable, and packages the result for the documentation pack. We treat that pack as an engineering output in its own right, part of our AI governance and trust work, and not as a policy document.
The engagement produces the measured evidence and the limitation statement for the documentation pack. It is not regulatory sign-off, legal advice or certification, and the committee and validators still make the decision. Outside help makes most sense when the stratification pipeline does not exist yet, or when the committee wants the analysis built outside the development line. When choosing any partner, ask how they report uncertainty on a small subgroup.
What does a regulator expect to see in a pricing model’s recalibration history?
Supervisory texts expect a change-by-change trail: for each material change, the rationale, the data and method changes, the independent review or validation and the approval, tied to version records. A memo plus the final parameter set shows what the model is now, but not why it changed or whether the change was tested. The NAIC white paper Regulatory Review of Predictive Models sets out questions a state regulator may ask as “information elements” in its Appendix B. One of them is how often each risk characteristic is updated and whether the model is refreshed periodically.
TechnoLynx’s AI Readiness Scorecard scores a team’s programme, including a memo-plus-parameter-set trail like this one, with an evidence trail and a remediation backlog.
The recalibration record, field by field
| Field | What it answers | Where the expectation comes from |
|---|---|---|
| Trigger | Why the model was recalibrated: a performance threshold breached or a persistent deviation from expectations | OCC Bulletin 2026-13; PRA SS1/23 |
| Data and method change | What changed in calculations, assumptions, parameters and data compared with the previous version, and why | NAIC white paper |
| Validation result | Whether the adjusted model beat the original against realised outcomes | SR 11-7; PRA SS1/23 |
| Independent review | Whether the adjustment or recalibration underwent independent review | SR 11-7 |
| Approval | Which body approved the change, and when | Regulation (EU) 529/2014 Art. 8 |
| Version trail | Current and previous model version numbers, software and version used | Regulation (EU) 529/2014 Art. 8; NAIC white paper |
| Obsolescence plan | How decay is measured, and the plan and timeline for updating or replacing the model | NAIC white paper |
The validation row deserves the closest attention. SR 11-7 describes parallel outcomes analysis, where both the original and the adjusted model are tested against realised outcomes. If the adjusted model does not outperform the original, more changes or a redesign are likely needed before it replaces the original. SR 11-7 also says recalibration should follow the principle of conservatism and undergo independent review. The PRA’s SS1/23 likewise expects testing of material changes to compare model output before and after the change against actual outcomes. The NAIC white paper adds a time dimension: reviewers should check stability over time, because a winter storm in year 3 of 5 can distort both the testing and validation datasets.
TechnoLynx treats this record as release-gate documentation. A recalibration without its validation and approval entries should not count as released. How change-triggered revalidation and rollback run in production is covered in our guide to keeping deployed models trustworthy through revalidation, monitoring and rollback.
Gaps an auditor flags in a memo-plus-parameter-set process
Look for a missing trigger rationale, no before-and-after comparison, no independent review of the change and no change control over the parameters, each a supervisory expectation. The OCC’s rescinded Bulletin 2000-16 already described documenting every modelling change for independent review as best practice, with code under change control so only approved parties could alter it. In a 2016 report, Florida’s Auditor General cited actuarial standards requiring documentation of any data adjustments and their rationale, clear enough that another qualified actuary could objectively appraise the work.
The same Florida report also audited the regulator’s own reviews. It found that the state insurance regulator’s policies did not specify the nature and extent of independent reviews of rate filings, nor the documentation to keep as evidence of them. For 3 of 43 applicable health insurer rate filings, the regulator could not produce a completed Rate Filing Summary form showing that the filing had been reviewed. That audit concerned the regulator’s own process, but the lesson for filers is the same: a review that left no record cannot be shown to have happened.
Prior acceptance is not a guarantee. The NAIC white paper states that a previous approval does not necessarily guarantee continued approval, for example when statutes change or later experience undermines a model’s indications. The paper is guidance for state regulators and does not change the legal standards for approving rating plans. Its third level of information is typically requested only when a reviewer’s concerns remain unresolved after the first two levels. A thin recalibration trail is a reliable way to end up there. And a complete trail depends on systems that actually captured each parameter change at the time, which a memo written afterwards cannot recreate.
How do we disclose lost records and a partial reconstruction without it reading as a quality failure?
Write it as a controlled nonconformance-and-remediation record. The record states what was lost, how the affected scope was bounded, what evidence the reconstruction rests on, what remains uncertain, and what corrective action prevents it happening again. That structure follows the logic IATF 16949 commentary attaches to traceability: traceability is key to corrective action, and assessors will look for an audit trail to judge compliance with the standard. The same commentary notes that traceability only helps while the chain stays unbroken. A disclosure therefore has to show where the chain broke and what now stands in for it.
The commentary we cite here covers the surrounding duties, not the reconstruction itself. Commentary on clause 8.5.2.1 calls for detailed records of nonconforming and suspect products, including the reasons and any investigations. Commentary on clause 7.5.3.2.1 requires a documented retention policy that meets statutory, regulatory, organisational and customer requirements, with records stored securely enough to prevent damage, loss or unauthorised access. The retention periods that commentary lists, such as five years for corrective action records, are explicitly examples to be adjusted to each organisation’s requirements. Check the standard’s own text and your customer-specific requirements before treating any of this as the clause wording.
For an automated optical inspection (AOI) model, machine-generated logs make strong reconstruction evidence: if each line and shift logged the hash of the deployed model version, those logs can show which baseline was in effect even when the tracker that recorded it is gone. Where the logs themselves have gaps, the record should state the reduced confidence for that period plainly, label reconstructed entries as reconstructed with their date and method, and never pass them off as originals. The aim is defensible handling, not avoiding a finding, and no wording replaces a corrective action that actually stops the loss recurring.
Financial-model guidance also expects data problems to be documented and reported. OCC Bulletin 2000-16 called input-data auditing an indispensable and separate part of validation. Where a bank chooses to use a model despite data problems, audit, risk management and modelling staff should each be independently responsible for informing senior management. SR 11-7 adds that when data paucity prevents validation before use, that fact should be documented and reported, and the uncertainty offset by compensating controls. The NAIC white paper suggests reviewers obtain a statement on whether data is missing for a systemic reason, and an explanation of material data limitations and how the analysis was adjusted because of them. The ECB guide expects documents to be kept for an appropriate period with legal and regulatory retention periods in mind. Our article on how evidence packs differ between a general security review and a healthcare review covers how this evidence is packaged for different reviewers.
Could another analyst reproduce last month’s numbers without the person who built the model?
If the answer is no, the documentation is not finished. The practical test is to have a second analyst reproduce a prior period’s output unaided, from the documents alone, with ownership held by a function and not by a person. OCC Bulletin 2000-16, although rescinded, put it well: model documentation creates a corporate memory when key modelling staff leave, and the most rigorous policies require enough detail to replicate the model precisely. For machine-learning components, the ECB guide expects documentation to allow replication down to parameters and hyperparameters, with random seeds stored where training relies on simulation.
For a process like monthly chargeback allocation, that means documenting the data sources, the transformation logic, the assumptions and the validation checks, each kept current. SR 11-7 assigns developers responsibility for documentation that stays up to date as the model and its environment change. Its standard for detail is that parties unfamiliar with a model can understand how it operates, its limitations and its key assumptions. Tacit judgement, such as the exception handling the analyst does “by feel”, needs capturing too. SR 11-7’s expectation that overrides are documented and analysed gives that judgement a home in the record.
Ownership closes the loop. The PRA’s SS1/23 expects a documented model owner for every model, accountable for performance monitoring, inventory accuracy and supplying what validators need. The NAIC white paper expects a filing to include a contact who can put the regulator in touch with the experts who built the model. The ECB guide asks institutions to keep enough in-house knowledge to take direct control of outsourced activity if they have to. A reproducibility test only covers the scenario it runs, so a single pass is not permanent proof. Turning the chargeback model itself into an approved, versioned artefact with a governance sign-off is a related step. The templates for that are covered in standardising model documentation and approval with schemas, gates and go/no-go decisions.
When should regime changes in historical data be called out as a modelling limitation?
Call them out whenever a policy shift, moratorium or subsidy change means older data no longer represents the portfolio the model now serves. Document a sensitivity analysis alongside the limitation. The PRA’s SS1/23 notes that a model’s continued suitability can be affected by changes in its assumptions, including assumptions about the continued relationship between variables in historical data. Where development data does not represent the portfolio, the PRA expects the impact to be assessed, reflected in the model’s tier and made known to users and owners. Any adjustments or proxies used to compensate should be documented and independently validated.
The EBA’s guidelines on PD and LGD estimation (EBA/GL/2017/16) are explicit that in calibration, non-comparable historical data should not lead to data being excluded. It should instead trigger an appropriate adjustment and a higher margin of conservatism. The same guidelines classify deficiencies arising from changes in relevant processes or the external environment as a separate category of uncertainty. Those guidelines govern internal-ratings capital models, so applying the principle to an accounting provisioning model is our judgement and not the text’s scope, but the logic carries over: keeping the regime in the data and adding visible conservatism is easier to defend than a quiet cut-off.
On the sensitivity analysis, OCC Bulletin 2000-16 described best-practice model reports as including “what if” scenarios built on reasonable alternatives for the major assumptions, which show decision makers how robust or fragile the outputs are. For a provisioning model trained on concessional lending, that means estimates with those loans included, excluded and reweighted, shown side by side.
Whether a supervisor has raised development-bank-specific concerns depends on its published findings and the portfolio’s own history, and that answer belongs with a credit-risk validation specialist. A methodology note arguing why a decade of concessional-lending data is a valid training basis is a separate document again, and it should cite the regime-change limitation, not replace it.
Keeping the hard-case records alive between approvals
Each of these records loses value once it stops being maintained. A limitation bounded last year may have widened, the recalibration trail stops wherever the last undocumented tweak went in, and the reproducibility test passed for a model that has since changed. The OCC’s current guidance points the same way: an ongoing monitoring plan should keep assessing model limitations over time, with procedures for responding to issues both before and after approval. The practical question each team still has to settle is who owns these entries between review cycles and which event forces an update. Assembling these records into the full audit-ready model documentation and approval package is covered separately.
