AI model risk management is not a validation gate that a model passes once. It is a monitoring discipline that runs for as long as the model is in use. On 17 April 2026, the Federal Reserve, OCC and FDIC issued SR 26-2. SR 26-2 supersedes SR 11-7 and keeps ongoing monitoring as a core expectation: checking whether a model still performs as expected as products, clients, data relevance or market conditions change. We would translate that into four engineering commitments: drift thresholds set before launch, drift tests that run on a schedule rather than whenever an analyst has time, a named owner for every breach, and an audit trail written while the model runs.

Why AI model risk management cannot stop at the validation gate

SR 11-7 already said validation activities should continue after a model goes into use, to track known limitations and find new ones. SR 26-2 ties the frequency and scope of monitoring reports to the nature of the model, the availability of new data or methods, and the model’s materiality. Glacis’s 2026 reading of the guidance is that it sets no universal calendar, so cadence follows the institution’s risk-based policy and how the model is actually used.

When outcomes analysis finds material errors or persistent deviations outside the institution’s established performance thresholds, adjustment, recalibration or redevelopment may be warranted. The PRA’s SS1/23 asks for thresholds for acceptable performance, plus criteria for deciding when a breach calls for recalibration or redevelopment. OSFI’s Guideline E-23 lists drifts and threshold violations among the events that should prompt a model review.

Machine learning makes all of this harder. AI models are highly sensitive to shifts in their input data, and their high dimensionality makes drift harder to detect and diagnose. Some AI systems also update their parameters as new data arrives. Such a model can become a substantively different model without ever triggering formal re-validation. Drift itself takes several forms: concept drift (changes in the target variable), data drift (changes in the inputs), and changes in upstream pipelines. Our position is that catching these failures is its own engineering discipline, separate from model accuracy.

Which monitoring metrics and retraining thresholds should a bank commit to before launch?

A bank’s inventory entry for the model should name its monitored metrics, a numeric alert threshold for each, and a named model owner. Thresholds should be marked provisional until production data exists. Openlayer’s 2026 guidance puts the bar bluntly: a system listed as “monitored” without named metrics and numeric thresholds is not monitored in any auditable sense.

TechnoLynx builds this layer as the Production AI Monitoring Harness: the eval, regression, drift and slice-metric harness behind the monitoring plan.

The PRA expects performance monitoring to measure against thresholds built from the testing criteria used during development. It also expects developers to hand over an outline of the monitoring pack that will track the model in use. Before launch there is no production data, so the first thresholds rest on the testing criteria used in development. The plan should commit to revisiting them once a defined volume of production traffic has built up. SR 11-7 addressed validation limited by data paucity. Where validation is limited by scarce data, that limitation should be documented, reported, and offset with compensating controls. It also called for “early warning” metrics that begin measuring very shortly after a model is introduced.

A provisional monitoring plan for a contract-review LLM

Signal	Example starting threshold	Owner who acts
Task quality on recent data (e.g. weighted F1 on extraction)	5% relative drop, evaluated weekly on the trailing four weeks; retrain after four consecutive breaches (setting used in a 2023 study of one language-model classification task)	First-line model owner
Groundedness of answers (retrieval pipelines)	Flag responses below 85% (Openlayer 2026 example)	First-line model owner
Toxicity of outputs	Block outputs above 0.15 probability (Openlayer 2026 example)	First-line model owner
Group outcome gap	Review when selection-rate gaps exceed 5 percentage points (Openlayer 2026 example)	First-line model owner
Input distribution drift (PSI)	Per-method threshold, e.g. PSI 0.3 on categorical columns (Evidently docs); tracked monthly at minimum, weekly or daily for high-risk models (go.ai 2026)	First-line owner, reviewed by validation
Overrides of model output	Track the rate and the reasons; overrides indicate limitations in the model	Model users and owner

The figures in the table are examples from their sources, not regulatory floors. Openlayer itself says to calibrate its example thresholds to the institution’s risk appetite and the applicable regulatory floor.

First-line model owners define acceptable output behaviour and maintain the thresholds. Validators judge whether the monitoring is adequate, and validation should be done by people with no stake in the outcome. Internal audit, as the third line, provides independent assurance that the framework works. Role definitions are not uniform, though. Glacis reads the current guidance as treating a three-lines model as optional rather than prescribed, and NIST notes that a traditional three-lines approach may be impractical for smaller organisations. The owner column therefore has to be confirmed with the bank’s own model risk team before the plan is filed. The evidence the plan starts from is covered in more depth in our piece on getting a model past independent validation for go-live.

What counts as a champion challenger model when there is only one fine-tuned LLM?

With a single fine-tuned LLM, we would take the best alternative available as challenger: the prior model version, a rules-based baseline, or a different prompting or fine-tuning configuration. We would run it in shadow mode on the champion’s traffic. The PRA lists comparing model estimates with comparable but alternative estimates as a monitoring test. Sphere’s 2026 guide for banks describes LLM validation as evaluation harnesses run against golden question sets by a team independent of the build team.

Two details matter. First, a prompt change or a refresh of the retrieval corpus counts as a model change and needs its own change-control trail. Second, a 2023 study found that incrementally fine-tuning the champion worked better than re-fine-tuning the pre-trained model on all the data at every retraining flag. We would therefore treat the next retrained champion as a challenger. The caveat is real: a variant of the same model shares that model’s blind spots, so for a material use case we would not rely on it as the only challenger.

How do you validate a contextual bandit or RL policy against seasonal shift before it goes live?

We would validate a contextual bandit or reinforcement-learning policy the way we would validate a quant model. Test it on regimes it was not trained on, wrap it in guardrails, and run it in shadow mode with seasonality-aware checks before it takes over from the existing A/B test. Context features cannot catch a shift nobody anticipated, and guardrails cannot fix a poorly specified context space.

The reason lies in how bandits fail. A contextual bandit picks actions whose rewards depend on the current context, such as a user’s profile and history. One study of non-parametric bandits under covariate shift gives a procedure that adapts without knowing when or how much the context distribution shifts. The authors also state that the procedure has no way to detect more severe shifts in rewards, where an arm it had discarded becomes the best one.

Two controls address this. First, encode the seasonal signal in the context: calendar features, the mix of traffic sources, and session characteristics, so that a seasonal visitor is not scored as a typical one. Second, run shadow evaluation with guardrails that watch for distribution shift linked to seasonality. Shadow mode has a cost: a later cutover. Guardrails tuned to known seasons may not generalise either, so whether features and guardrails are sufficient has to be reassessed each time a new kind of shift appears.

For RL policies that allocate capital, published practice points the same way. A 2025 deep-hedging framework recommends clipping actions to desk risk limits, halting trading when liquidity deteriorates, and logging both observations and predicted actions. It also calls for retraining policies periodically as volatility regimes change and for clear criteria for throttling or disabling the strategy. A 2026 study of option hedging found that hedging-oriented RL models were more variable across assets and regimes, with rankings compressing and becoming asset-specific in calmer markets. A single aggregate backtest can therefore hide differences between regimes. For each stressed regime, the report should include:

• RMSE of terminal hedging P&L, average trading cost and shortfall probability
• Rolling Sharpe, turnover, realised cost per basis point, and policy drift relative to benchmark hedges
• Stability across random seeds and initialisations, since large variation signals instability
• Behaviour near the edges of the input ranges the policy was trained on, since models that only work within certain ranges should be monitored for when those limits are approached

Automating monthly PSI and KS drift tests with Evidently so a committee can audit them

Yes, PSI and KS drift tests can run as a scheduled job instead of an analyst’s spreadsheet. Evidently’s own documentation describes batch monitoring jobs that run at regular intervals or when new data or labels arrive.

TechnoLynx builds this scheduled drift job as part of the Production AI Monitoring Harness: PSI and KS tests run on schedule with pinned settings, and every run is logged for validators.

One Airflow DAG runs daily on a cron schedule and applies Evidently’s KS test to numerical features and chi-square to categorical ones. Evidently’s batch blueprint stores each run in PostgreSQL with the timestamp, test name, threshold, drift score and drift flag. The D3Bench study notes Evidently’s integration with MLflow, Airflow, Prefect and PostgreSQL.

A scheduled drift job, step by step

1. Fix a reference window per site. The reference can be validation data, earlier production data or a curated golden set. We would keep one per site so that one site’s normal behaviour does not mask drift at another.
2. Extract the current window on schedule. We would version schemas so that a renamed field cannot silently break a test.
3. Pin methods and thresholds. Evidently picks a drift method automatically from column type and data volume, and either can be overridden. According to D3Bench, its statistical tests default to a 0.05 p-value threshold and its distance metrics to 0.1. Write these settings into configuration explicitly, because the same test name can produce different results across tools.
4. Correct for many features. Running per-feature KS tests across many features inflates false positives. In a 2026 test with injected drift, KS with Benjamini–Hochberg correction found all five drifted features from drift magnitude 0.2 upwards, with zero false positives at magnitude zero.
5. Write every run, not just failures. Record each run with the model version hash, test configuration, threshold and outcome, so that validators get an artefact rather than a summary written afterwards.
6. Branch on the result. Route detected drift to an alert task and log a no-drift outcome as a record in its own right. Pair the drift tests with data-quality checks such as null shares and schema compliance.

For alerting and governance built on top of this job, see our guide to building drift detection, alerting and governance pipelines.

We would detect when engineering adds a new model or hosting vendor the same way: instrument change events in the deployment pipeline, such as new container images, Terraform providers or API endpoints, so that each one opens a risk re-review with a named owner rather than surfacing during a customer audit. OSFI’s E-23 lists model modifications, including changes to supporting operational components, among the events that should prompt a review.

Who can build the drift pipeline and monitoring harness our model risk committee will sign against?

TechnoLynx builds the monitoring and drift-detection pipeline that a bank’s model risk function signs against: the scheduled tests, thresholds, alert routing and the logged record behind each result. The engagement is the Production AI Monitoring Harness, part of TechnoLynx’s AI engineering for finance work. Whether to build in-house or bring in a specialist comes down to whether your team can treat the evidence trail as an engineering deliverable alongside its model work. We hold that approval-grade evidence of this kind is an engineering output in its own right, not a policy document.

The engagement does not suit everyone. It does not provide certification or policy advice. It cannot serve as your independent validation either, because validation should be done by people with no stake in the model’s outcome. A strong in-house MLOps team can assemble a similar stack from open-source parts. D3Bench rates Evidently highly for integration with MLflow and Grafana. Commercial monitoring platforms are another route, and independent validation is still required either way.

What documentation do model-risk reviewers expect when the architecture, output encoding or loss function changes?

Reviewers expect a documented justification of the new design choice against the alternatives considered, written clearly enough that someone unfamiliar with the model can follow it. The format follows the institution’s own policy, because SR 26-2 explicitly lets practice vary with each organisation’s risk profile. The underlying expectation is long-standing: documentation should let parties unfamiliar with a model understand how it works, its limitations and its key assumptions. NIST’s AI RMF Playbook lists “evaluated alternative approaches” among the items documentation policies should cover.

Material changes in model structure or technique should be validated before implementation. For a loss-function choice such as L1 against L2, we would compare the candidates on outlier robustness, fit to the business loss and backtested performance. As a reference point, L1 loss is less sensitive to outliers than L2. Record the business rationale alongside the numbers.

Moving from a scorecard to a neural network with a different output encoding needs two further checks. Downstream decision thresholds built on the old score scale need re-validating. For credit decisions, creditors using complex algorithms must still give specific reasons for adverse actions. For selection or mining runs, log the configuration, data snapshot, code version and candidate results. The expectation is that any model state can be rebuilt as of any point in time.

Who owns the response when a drift alert fires?

We would name the role that owns the response to drift separately from the role that monitors for it. A RACI then gives one role final sign-off when functions disagree, and every cycle from detection to resolution is logged. Under the PRA’s SS1/23, model owners are accountable for monitoring performance against the board-approved risk appetite, and firms need escalation processes that promptly notify owners, users, validators and senior management of a model exception.

TechnoLynx operates the detection side of this through the Production AI Monitoring Harness. The RACI itself remains the institution’s decision.

We would write down in advance the tie-breaker for when data science flags decay and the business function disagrees. SR 11-7 gave validation staff explicit authority to challenge and elevate findings, reporting to someone with enough standing to see issues addressed. The PRA expects the approving individual or body to make sure validation recommendations are acted on. Managing such issues should be documented, recording their nature and tracking remediation in the model inventory.

Two practical points apply. First, escalation windows such as “N days below threshold” should be calibrated per model rather than copied across fleets. Second, Openlayer draws a useful line between observation and control: a drift alert that notifies without halting inference is a monitoring signal, not a governance control. How this escalation produces evidence a committee can use is the subject of our article on ongoing monitoring and drift escalation in model risk management.

Decision rights for go-live, retraining and retraction

We would decide before the triggering event who can pause, roll back, retrain or retract a model, against which numeric gate, and record it with a signature. Sign-off given after the fact does not hold up under audit. A validation report should state what was tested, what thresholds were set, and which residual risks were accepted by named owners.

TechnoLynx supports this on the engineering side through its AI Governance and Trust work. That means agreeing the numeric pass/fail metrics that act as the gate, building the shadow or canary and monitoring pipelines that produce the evidence, and operating them so triggers fire automatically on a breach. Ownership of the decision rights stays with the institution.

Before go-live: who can delay the launch?

A launch gate needs a named role with authority to delay or roll back, and that technical pass/fail authority should be kept separate from whoever owns the contractual deadline. We would record who approved the launch, against what evidence and on what date. The inventory should also carry the date its thresholds were last reviewed and approved. Shadow or canary traffic supplies the evidence for the gate, but it does not assign anyone the right to decide.

After go-live: who decides between re-validation, retraining or retraction?

After deployment, a standing policy does the work. NIST’s AI RMF Playbook calls for policies naming the functions and people responsible for AI monitoring and incident response. SR 11-7 listed the options available to senior management: adjust model output, restrict use, rely on other models, or apply other compensating controls. The PRA adds that firms should consider restricting a model’s use when performance tests show a significant breach has occurred or is likely.

Drift beyond a defined threshold reopens the validation question. The policy document matters less than what happens at the first genuine breach: a decision by a named person, recorded within the window the policy promised, choosing between recalibration, retraining or restriction.

FAQ

What is PSI in machine learning?

The Population Stability Index (PSI) is a single number that measures how far a feature’s binned distribution has shifted from its reference. One 2026 architecture uses it, with quantile-based bins, as a complement to per-feature KS tests. In the D3Bench study, PSI was one of five methods that identified drift consistently in both use cases.