What Is Drift Monitoring in Model Risk Management?

Drift monitoring is the ongoing check for changes in a deployed model’s input distribution and in the relationship between its inputs and outputs. It tracks performance against defined thresholds and initiates corrective action when necessary. Supervisors count it as part of validation. The 2011 US guidance, replaced in 2026, named ongoing monitoring a core element of validation, and the PRA’s SS1/23 includes model performance monitoring in its definition of validation. A defensible monitoring programme rests on four things a dashboard cannot supply on its own: statistical tests a validator recognises, a hold-and-review procedure for flagged output, proof that the production model is the approved one, and an audit trail showing that each alert was handled.

Drift monitoring in model risk management: the continuous leg of validation

Ongoing monitoring targets two forms of drift. Data drift is a change in the distribution of input data. Concept drift is a change in the relationship between inputs and target. Data drift detection uses a two-sample hypothesis test to check whether current requests are drawn from the same distribution as the training data. Options include feature-wise Kolmogorov-Smirnov (KS) tests with Bonferroni or false-discovery-rate correction, or maximum mean discrepancy with a permutation test for multivariate data. One banking validation approach to concept drift matches new data to similar development samples by nearest neighbour, then compares the residual distributions using KS, Cramér–von Mises or Anderson-Darling tests.

Scarce labels are the more common scenario. In that case teams monitor label-independent signals, such as input and prediction distributions, as a proxy for performance. That proxy is weaker than it looks, because tying drift-detector outputs directly to model performance is still described as an open research problem. Where outcomes do arrive, we would use an input-drift metric only to cover the gap until they do. It should never replace outcomes analysis.

The tests themselves should be chosen during development. SS1/23 expects model developers to outline the monitoring pack that will track the model in use. That is the point where monitoring connects to the wider work of building model risk management into the ML lifecycle.

Production reliability is a separate discipline from accuracy

A model can keep its validated accuracy and still fail in production. The ML Test Score rubric by Breck et al. at Google lists seven monitoring tests:

dependency changes trigger a notification
data invariants hold
training and serving are not skewed
models are not too stale
models are numerically stable
computing performance has not regressed
prediction quality has not regressed

Only the last test is about accuracy. Breck et al. note that regressions in latency, throughput and memory can arrive suddenly or as slow leaks. Slicing those metrics by data and model version, and not only by code version, is useful. Staleness matters for non-stationary data such as financial applications: if the pipeline fails to train and deploy sufficiently up-to-date models, the model is stale, and its effect on prediction quality sets how often to update. The CSA AICM auditing guidelines also look for defined rollback triggers, such as performance falling below thresholds or capability regressions, and for a rollback that restores a known good state.

Production AI reliability is an engineering discipline of its own. Its deliverables are eval harnesses, regression suites, drift detection, alert-quality work and release-readiness reviews; a monitoring dashboard is not one of them. The regression side matters when a new model version replaces the previous one. That is the job of evaluation harnesses that catch drift and regressions before a new version reaches traffic.

What does SR 26-2 change about the monitoring evidence a bank keeps?

The updated interagency guidance still expects ongoing monitoring and outcomes analysis. What changes is the framing: the guidance is now risk-based, tailored to each bank’s model risk profile, size and complexity, and it leaves more of the response to deterioration to the bank’s own policy.

Topic	SR 11-7 (2011)	SR 26-2 (updated guidance)
Status	Replaced in 2026	Current; risk-based, tailored to model risk profile, size and complexity
What ongoing monitoring confirms	That the model is appropriately implemented, used and performing as intended	How far the model performs as expected given changes in products, exposures, activities, clients, data relevance or market conditions
Programme content	Ongoing testing and evaluation with response procedures, including process verification and benchmarking	Regular assessment of limitations and procedures for responding to issues, before and after approval
Frequency	Periodic review at least annually, more often if warranted	Frequency and scope depend on the model’s nature, new data or approaches, and materiality
Outcomes analysis	Back-testing against expected ranges or confidence intervals; investigate discrepancies that are significant in magnitude or frequency	Compare outputs with real-world outcomes; persistent deviations outside established thresholds may warrant adjustment, recalibration or redevelopment
Response to deterioration	Adjustment, recalibration or redevelopment, governed by conservatism and independent review	Overlays, adjustment or redevelopment, depending on the bank’s model risk management policy
Internal audit	Verifies policies and records of model use and validation	Evaluates whether practices are rigorous and effective; generally does not duplicate validation

The OCC bulletin adds that the guidance may also apply to banks with $30 billion or less in assets where model risk is significant. It also says the agencies plan a request for information on AI, including generative and agentic AI.

We do not read this as a lighter evidence burden. Under the updated guidance, the response to deterioration depends on the bank’s own policy, and monitoring frequency and scope depend on model materiality. Under the UK PRA’s SS1/23, monitoring frequency follows the model tier, reports are independently reviewed, thresholds come from development testing criteria, and escalation must promptly reach owners, users, validators and senior management.

Thresholds a validator will accept: significance is not meaning

Every drift threshold has to answer two questions. Is the change statistically detectable? Is it operationally meaningful? At large production sample sizes, even small KS distances are highly significant. In one study of a deployed monitoring framework, samples of about 332,000 observations gave a critical KS distance of 0.00333 at α = 0.05. Every KS test the authors reported was highly significant, yet none was operationally meaningful.

The threshold therefore has to come from validation, not from the test’s default. SR 11-7 describes validation setting acceptable error levels by analysing how outcomes are distributed around predictions. When outcomes fall consistently outside that range, the model should be redeveloped. SS1/23 expects monitoring thresholds to rest on the testing criteria used during development. Setting those thresholds takes domain knowledge, and it is hard to do without generating false alarms; change-point methods may prove more robust.

For a model monitored by the population stability index (PSI), the familiar cut-offs are: below 0.10 means little change, 0.10 to 0.25 moderate change, and above 0.25 significant change. They are an industry rule of thumb used without reference to type I or type II error rates, so on their own they are a convention, not a defensible threshold. A statistical analysis of PSI shows that these benchmarks shift with the bin count and with both sample sizes. In its null-case analysis, PSI > 0.10 raises too many false alarms unless both samples exceed 400, and the power of both fixed tests falls as sample size grows. The defensible cut-off is the one calculated for the committee’s own bins, sample sizes and chosen significance level.

SS1/23 expects the documentation to detail the monitoring tests and the rationale for choosing them. For PSI, we would write down the following before the first monthly run:

the baseline or vintage window
the bin definitions
the minimum sample size per period
the recalculation cadence
how the significance-based cut-off was derived

We would avoid a baseline window set so far back that it understates recent drift. We would equally avoid one so recent that it hides a slow secular shift. Either way, the choice belongs in the document, so a reviewer can challenge it. We would add a characteristic-level stability index per input to show which feature drives a PSI move, and KS as a binning-free cross-check. Neither replaces a documented primary metric.

Worked example: 3-sigma versus CUSUM on the same drift

In a clinical-imaging paper’s simulation of 100 CT slices a day, the OOD rate was 0–1% in the first month and 3–5% in the second. The 3-sigma chart raised a flag on day 37, but the following days fell back inside the limits, so that flag may have been an anomaly. The CUSUM chart flagged two days after the induced shift, and its subsequent signal supported that flag. The authors conclude that 3-sigma works for detecting individual items, while CUSUM is better at catching sustained drift in batches, because daily averaging hides deviations from a 3-sigma rule. Tuning CUSUM’s k parameter trades fewer false positives against timely detection of a real shift. A single flag is not a confident detection, which is why additional run rules are worth defining.

Monitoring question	Test to reach for	Known weakness
Has the score distribution shifted from baseline?	PSI against a calculated, significance-based benchmark	Sensitive to bin count and sample sizes
Has an individual feature’s distribution changed?	Feature-wise KS with multiple-testing correction	Significant at large samples even when the change is negligible
Is this single item out of distribution?	3-sigma control chart	Struggles with temporal drift in batched daily averages
Is a small, persistent shift building up?	CUSUM control chart	The k setting trades false positives against detection delay

Which backtests will a model validator accept?

Kupiec and Christoffersen: the recognised pair for exceedance models

For VaR models, the regulatory benchmark is the Kupiec proportion-of-failures test paired with the Christoffersen independence test. Kupiec checks whether the realised exceedance rate matches the model’s tail probability. It is the foundational backtest in Basel capital frameworks. Christoffersen asks whether the violations cluster in time. A model with the right average rate but clustered exceedances is still misspecified. A review of VaR backtesting stresses that unconditional coverage and independence are separate properties and that an accurate model must satisfy both. When Kupiec passes and Christoffersen rejects, the clustering signals a static or stale volatility model. GARCH-based VaR or filtered historical simulation usually passes the independence test where simple historical VaR fails.

Basel sets the backtesting window. Basel MAR32 compares one-day VaR against actual and hypothetical P&L over the prior 12 months. The Basel traffic-light scheme counts violations over the previous 250 trading days: four or fewer is green, five to nine calls for further tests, and ten or more marks the model as problematic. Results in the amber zone bring a capital add-on, and the bank must document every exception with an explanation.

Unconditional coverage tests have low power at the one-year sample sizes the regulatory framework uses. Kupiec in particular is weak at one year and can pass a model whose violations cluster. The joint conditional-coverage test is less able to catch a model that fails only one of the two properties. We would therefore report the two tests separately rather than only the joint result. Basel specifies 250 daily observations, but longer windows are recommended alongside it during validation reviews.

We would carry the same framework over to books that are not standard trading VaR. Hit-function backtests make no assumption about the stochastic behaviour of profits and losses. They only require that hits are Bernoulli with the nominal probability. On that basis, Kupiec and Christoffersen can be adapted to, say, a weather-derivative exposure model. The catch is that lower-frequency data produces fewer observations per year, and an already low-powered test gets weaker. We would lengthen the window and state the reduced power in the validation report, not invent new statistics. Whether a given committee accepts the framework also depends on the institution’s own model validation policy, not on how standard the test is in the literature.

Building the board evidence pack around a control chart

We would present a finding to the board through a recognised control framework with thresholds set before the fact, not a narrative written after it. SS1/23 expects the board to receive regular reports on significant changes in model performance. It also expects the board to understand the operating boundaries of the most material models, the impact of poor performance and the mitigants in place. The OCC’s derivatives handbook asks for an annual summary to senior management that includes validation and back-testing results and explicitly identifies the models’ limitations. SPC separates common-cause variation from special-cause variation. An evidence pack built on SPC should contain:

the control chart, with limits drawn from the baseline
the test statistic and its p-value
the threshold and where it came from
the exceptions log
the action taken

SR 11-7 describes effective validation reports as opening with a clear executive summary and an accessible synopsis of results, including major limitations.

How do you tell an audit committee about drift without losing the statistics?

Lead with one sentence that says three things: whether the shift is likely real or within normal variation, the number behind that judgement, and the decision the committee is being asked to take. If the test is significant at the committee’s agreed level, we would say the shift is likely real and give its size. If the test is not significant or lacks power, we would say plainly that the evidence cannot yet distinguish a shift from normal variation, and recommend continued monitoring. Do not offer reassurance that the model is stable.

Give the number, not just the words. A 2020 briefing by Full Fact, Africa Check and Chequeado reports that the IPCC uses “very likely” for a probability of 90% or higher, while study participants’ typical median reading of the phrase was 65–75%. The same briefing found that both numeric and verbal statements of uncertainty were understood. It notes that the UK Government Statistical Service recommends quantifying uncertainty early in a publication, and where that is not possible, giving a reasoned judgement of its likely size and direction. A p-value can be explained in one clause: it says how surprising this result would be if nothing had changed. For a backtest, also inspect the 95% Clopper-Pearson interval for true coverage. If the interval contains the nominal rate, the model is statistically consistent with it.

For an ambiguous result, we would choose the wording with particular care. A defensible version reads: “The data so far can neither confirm nor rule out a shift; monitoring frequency has been increased and the committee will see the next reading on [date].” That sentence shows the bank is in control, because control lies in the process the committee can see working, not in certainty the data does not support.

A flagged output: hold it, send it to manual review, or widen the intervals?

We would hold the output and route it to review, with a named authority signing off on what happens next. Widening the confidence intervals is not, on its own, a response. SS1/23 expects firms to consider restricting model use when performance tests show that a significant breach has occurred or is likely. That can mean permitting use only under strict controls, or barring the model from specific purposes. Escalation must reach owners, users, validators and senior management promptly. Auditors look for a documented process for reviewing detected anomalies, and for evidence that the actions taken were recorded. Practitioners interviewed about monitoring said a monitoring system should not trigger actions such as automated retraining. It should give humans actionable insight, without causing alert fatigue.

A checkpoint holds the output until a trained person reviews it, and routing rules decide which cases go there: low confidence, high stakes, or an appeal. Recording the score, the threshold and the route taken makes the decision rule inspectable. Two limits shape the design. Review capacity sets a ceiling, so the escalation rules must match the number of trained reviewers available. And when only low-confidence cases are escalated, confidently wrong outputs pass unseen, so a random sample of routine cases needs review as well.

Widening intervals falls short because, in effect, it is an unreviewed post-model adjustment. SS1/23 expects post-model adjustments to follow a documented process, to be independently reviewed and to be reported with and without the adjustment. An interval widened quietly meets none of those conditions. It changes what downstream users see without anyone having decided the model is still fit for use.

When a unit is flagged as outside the model’s known envelope, we would give sign-off to a named model risk authority defined in the monitoring policy before any flag fires. That authority decides whether to release, re-baseline or escalate the held unit, and the event, decision and rationale go into the case file whatever the outcome.

On the floor, we would move a flagged unit to the manual inspection schedule and log the event. We would not trust the model’s countdown for that unit class again until model risk management has reviewed the pattern of flags and signed off. Sign-off can be quick when flags are isolated and explained; when they are not, it goes through recalibration or re-baselining.

What is the minimum interim safeguard while re-validation is pending?

The minimum safeguard is a temporary human-in-the-loop confirmation step on the affected outputs, with written exit criteria. Halting the workflow is not necessary. SR 11-7 set the precedent. When validation cannot be completed before use, that fact is documented and communicated, and the uncertainty is mitigated by compensating controls, especially when an existing model is applied in a new setting. We would treat a classifier whose inputs have shifted, for example after a new sequencing platform feeds a tumour-board report, the same way: manual confirmation until the re-validation evidence is accepted. SS1/23 asks for triggers that activate validation and remediation when post-model adjustments are used for prolonged periods. The checkpoint’s effect should be tested, not assumed.

How can you prove the model in production is the one the committee approved?

Proof comes from signing and hashing the approved artefacts. Model artefacts are signed, version-controlled and validated before deployment, and checkpoint hashes are compared regularly to detect unauthorised modifications. TechnoLynx designs and builds this control as part of its AI governance and trust work. That covers the signing step tied to registry approval, the verification gate in the deployment pipeline, and the continuous production check with its alert path, validated against the detection time the committee requires.

Under the OpenSSF Model Signing specification, a private key produces a signature that anyone holding the public key can verify. The signature acts as a tamper-evident seal covering the model’s integrity, authenticity and provenance. In the MLflow Model Registry, each version has a number and links back to the run that produced it. Aliases such as @champion and tags such as pre_deploy_checks: “PASSED” carry the deployment state. The audit trail should then bind the approved version to a specific deployment instance, recording when and where it was deployed. The Cloud Security Alliance’s AICM auditing guidelines ask for cryptographic checksums of model checkpoints to be computed and compared regularly: at least annually depending on risk, or after any change of hands. SR 11-7 already required that model code could not be altered except by approved parties, and that every change was logged and auditable.

Two limits matter. First, what gets approved should be the whole bundle, because the same guidelines define the baseline as covering weights, tokenisers, quantised variants and runtime configuration, not just a single weights file. Second, a passing hash proves identity, not behaviour. It shows the bytes in production are the approved bytes; whether those bytes still behave as approved on today’s data is the drift monitor’s job. Signing keys also become an operational dependency that needs its own security and audit.

Can the check run same-day, not only at release?

Yes. We would run the same verification on a schedule, or on events, against the live artefact, with detection time set by how often that job runs. TechnoLynx implements the release-gate check in the CI/CD pipeline against the committee-approved registry entry, builds the periodic production re-check and its alert hook, and validates the whole path in the model-risk sandbox before it is enabled against live traffic. Auditors test this by deliberately creating artefact inconsistencies and version mismatches, then confirming that alerts fire within the expected timeframes with the correct severity. A failure of the verification job itself must be reported as well. Alerts should fire when monitoring telemetry drops, and escalation should follow automatically when an alert goes unacknowledged. Testing the change in a sandbox first matters because the integrity check is itself a change to the deployment pipeline, and it should not introduce the very risk it exists to catch.

How do you stop any one team from silently pushing an unapproved model?

Every deployment has to be attributable to an individual credential, signed-artefact deployment has to be enforced by the platform rather than by policy text, and the strictness should scale with the model’s risk tier. TechnoLynx designs the signed-artefact deployment control that ties each release to an individual credential. That work also integrates per-person or per-team keys with Kubernetes admission control, connects deployment events to the audit trail the committee attests from, and defines the risk-tiering rules that decide which models take the stricter path.

SR 11-7 required that only approved parties could change model code, and that every change was logged and auditable. The CSA guidelines check that approvals for model promotion are role-based and documented. SS1/23 scales the rigour of controls to the model tier. For the credential layer, secrets managers such as HashiCorp Vault or cloud key management services provide per-person or per-team keys. A registry backend with role-based access controls adds governance on the model side. We would treat linking a credential to a specific committee approval as integration work. Kubernetes admission controllers can reject a workload that fails a policy check at deploy time. That is the layer where a silent swap is actually stopped, because a role-separation policy that lives only in a document stops nothing.

What should an alert’s audit trail record so a later investigation holds up?

For each alert and escalation, the trail records the model version, what triggered the alert, who acted, what evidence they consulted, what they decided and what follow-up was created. The records sit in an immutable, append-only trail with tamper evidence. A paper on LLM audit trails defines the trail as a chronological, tamper-evident, context-rich ledger. It links technical provenance (models, data, deployments, monitoring) with governance records such as approvals and waivers, so an organisation can reconstruct what changed, when, and who authorised it.

A checklist for each monitoring event:

Event type, such as a drift alert, validation success, deployment, rollback or configuration update.
Timestamp, plus the model name and version affected.
Triggering mechanism, for example a monitoring alert identifier, a named user or a scheduled run.
Associated artefacts: dataset version, code commit hash, configuration version and validation results.
Actor: the user account or service principal that acted.
For an incident: when it was detected, who triaged it, what evidence was consulted, the decision (rollback, patch, temporary suspension) and the follow-up obligations.
For a backtest exception: an explanation of that exception, as Basel MAR32 requires for every exception.

Tamper evidence can be concrete. The same paper’s proof of concept stores each event with a SHA-256 hash computed over its payload plus the previous entry’s hash, starting from a fixed genesis value, so any modification or deletion becomes detectable. An auditor can replay the hash chain to confirm that the trail has not been altered or truncated since it was written. Access to the logs should itself be logged, with timestamps, actor IDs and the model data accessed.

There is one limit to keep in view. An audit trail proves the order of events, not why they happened, so causation and responsibility still depend on interpretation and organisational judgement. Drift metrics carry a similar limit: tying them to model performance is still an open research question. What makes a threshold defensible is the record: it was written down before the alert fired, the response matched the one the policy named, and the log shows both happened.