A model’s risk tier sets how deep validation has to go before it goes live. The tier comes from two ratings: how material the decisions it drives are, and how complex and opaque the model is. The tier then sets the rigour, intensity and frequency of validation, and how independent the review has to be. We would let that tier decide which gates a retrained or swapped model has to clear (an offline regression harness, then shadow and canary traffic) and what goes into the evidence pack the approver signs. In that sequence, the shadow-mode canary comes last. It does not replace the steps before it.

Accuracy on a held-out set is not the same as release readiness. Production models regress in ways unit tests miss, from silent quality drift after a model update to edge-case failures under load. The work that catches these failures produces specific artefacts: task-specific evaluation harnesses, regression suites, release-readiness reviews and validation packages.

Which risk tier does our lending model fall into, and what validation depth does that require?

A lending model’s tier combines a materiality rating and a complexity rating. The Prudential Regulation Authority’s SS1/23 (2023) expects the rigour, intensity and frequency of validation, independent review and monitoring to follow that tier. A model that drives production credit decisions already rates high on purpose, so the open question is usually complexity.

TechnoLynx’s AI Governance and Trust practice builds the engineering side of a tiering decision. It starts by identifying the model and assigning its risk tier, then builds an evidence pack sized to that tier. These are artefacts an auditor can read, not regulatory sign-off.

The UK and US texts agree on this. SS1/23 asks firms for a model definition, an inventory and a risk-based tiering approach. It judges materiality on size-based measures (exposure, book value, number of customers) and on the model’s purpose and importance to business decisions. In the United States, SR 26-2 replaced SR 11-7 (2011). The Federal Reserve, OCC and FDIC issued it on 17 April 2026. The revised guidance makes four points:

• Inherent risk rises with complexity and with the number of assumptions a model needs.
• Purpose and exposure together determine materiality, and models used to meet regulatory requirements or manage financial risk generally count as higher risk.
• Higher-materiality models warrant more comprehensive oversight.
• Validation rigour follows the model’s approach, use and materiality.

SR 11-7 also tied validation rigour to model complexity and materiality, but it is no longer the current US guidance.

Tiering inputs at a glance

Tiering input	Raises the tier when
Exposure	The model applies to a large portfolio, book value or customer base
Purpose	Outputs inform regulatory, capital or financial-risk decisions
Complexity	The model is hard to explain in non-technical terms, or its output is hard to anticipate from its input
Data	Development data does not represent the portfolio, or the model uses alternative or unstructured data
Challenger coverage	Little comparison against challenger models or benchmarks has been done
Interconnection	The model feeds or depends on many related models and data sources

A higher tier means more rigorous and more frequent validation, independent review and re-validation. SS1/23 asks each firm for one consistent, firm-wide tiering approach, and SR 26-2 tailors expectations to each organisation’s model risk profile. Each model’s tier assignment is also reassessed independently at every validation and re-validation.

Does adding a neural component change the tier compared with a pure XGBoost model?

Only if it raises the complexity rating, which SS1/23 describes by properties. SS1/23 describes complexity by properties: a model carries more risk when it is hard to explain in non-technical terms or when its output is hard to anticipate from its input. Where adding a neural component makes either worse, the complexity rating rises, and with it the tier of a lending model. The model can earn some of that back with challenger comparisons. Validators expect a complex model to be benchmarked against simpler alternatives, to show that the extra complexity buys real predictive gain. Credit decisions carry one more constraint. The CFPB’s Circular 2022-03 requires creditors using complex algorithms to give specific reasons for adverse actions, and an opaque model is not an excuse.

How rigorous does validation need to be for a model that only predicts execution cost?

We would scope it below a credit-decision model, but it still has to be validated. An execution-cost model has a narrower purpose, which lowers its qualitative materiality. SS1/23 still expects every model to get an independent review in proportion to its tier. Exposure can push the tier back up: SR 26-2 treats exposure as something you can measure in numbers, such as portfolio size. An execution-cost model that sits under a large order flow has high exposure, which raises its tier.

Validating an engine that sets capital add-ons for many banks

A capital-setting engine serves a regulatory purpose and has wide exposure, and both raise its materiality. Its validation should be independent of the development group and include out-of-sample backtests. We would keep a standing evidence pack ready before any challenge arrives. Its purpose alone puts it there: SR 26-2 treats models used for regulatory requirements as higher risk, and SR 11-7 already expected a more extensive framework where models inform capital decisions. Aggregate risk adds to this, because model risk grows with the number of related models and shared data sources.

When should a team outside model development re-run the backtest before release?

For any high-tier model, and for any release where the validation so far was done by the people who built the model. SS1/23 says validators should take no part in development and have no stake in whether the model is approved. SR 11-7 accepted that developers may do some validation work, but required an independent party to review it critically and add its own testing. SR 26-2 asks for clear accountability that deals with conflicts of interest between development and validation reporting lines. For algorithmic trading, the PRA’s SS5/18 expects testing by a competent team that was not involved in writing the code.

The re-run should be a backtest in SR 11-7’s sense. That means comparing forecasts with actual outcomes over a period not used in development, at a frequency that matches the forecast horizon, and investigating results that fall outside the confidence intervals. How far the independent work goes depends on the tier, and for a model change, on how material the change is. Independence is judged by actions and outcomes, not by reporting lines alone. The go-live negotiation itself, including when a model gets conditional approval and what stratified evidence an independent validator asks for, is covered on its own page.

What should a validation evidence package contain if a regulator or affected party challenges an output?

It should let someone unfamiliar with the model understand how it works, its limitations and its key assumptions. SR 11-7 set that bar for documentation, and SR 26-2 keeps the focus on identifying limitations and appropriate use. A pack ready for a challenge holds:

• the model’s tier and the information used to assign it, which validation reassesses;
• the model’s assumptions, stated and validated against the data;
• backtest results against the expected ranges or confidence intervals around the forecasts;
• the validation report: what was reviewed, deficiencies across a range of conditions, and any compensating controls;
• an executive summary with the model’s purpose, results, major limitations and key assumptions;
• for credit decisions, the specific reasons the model can give for an adverse action;
• the log of identified issues and their fixes, tracked to an auditable standard.

We treat the record of who developed, validated and approved the model, and whether that trail is tamper-evident, as a separate engineering problem. Our page on building evidence packs that separate audit workflows from approval gates covers it. A complete pack is no guarantee that a supervisor will accept the conclusion. It guarantees the argument can be examined.

What happens when validation finds a significant deficiency?

The release stops or is narrowed. SR 11-7 said a model with significant deficiencies should not be used, or used only under very tight constraints until the problems are fixed. If the deficiencies could not be fixed within the model’s framework, the model should be rejected. SS1/23 lists the options: strict controls, limits on use, or a ban on specific purposes. Sometimes an urgent business need forces use before validation is finished. In SR 26-2, sound practice then involves closer attention to the model’s limitations, informing stakeholders of them, and controls such as limits on use or closer monitoring.

Do our analysts need a summary, retrieved precedent or a full agent for triage?

Measure how analysts actually spend their time and what “done” means for a case. Then pick the smallest tier that covers the bottleneck you measured, rather than whatever the vendor shortlist is selling.

TechnoLynx helps make this decision through its Generative & Agentic AI R&D offer. The output is a recommendation on which tier the evidence supports.

Do analysts mostly read case history, or query several systems by hand? We would measure it rather than ask: self-reported estimates are a weak basis, and logs of what analysts open and query are the evidence worth having. Would retrieval-augmented generation (RAG) over policy documents and past case notes close the gap without an agent? Only if most of the time goes on finding and summarising information.

What “done” means	Smallest tier that fits	What validation has to show
A drafted note	Summary generator	The note is faithful to the case record and invents no facts
A recommended disposition grounded in past cases	RAG over policy documents and case notes	Retrieved precedent is relevant, and the disposition matches what an experienced analyst would choose
A fully investigated case	Agentic investigator	Each action across systems is correct and logged, and escalation fires when it should

Choosing the tier is really a decision about validation scope. Each definition of “done” needs a different evaluation, and the agentic tier needs the deepest one. Where “done” differs by case type or severity, a mixed design is the right answer: summaries for routine cases and deeper tiers for high-severity ones.

How do I build a citable evaluation harness when I can only hand-label a small test set?

A citable, low-cost harness has three parts:

1. A fixed failure-mode taxonomy, turned into pass/fail criteria.
2. A small hand-labelled gold set, stratified across those categories.
3. Optionally, an LLM judge. Use one only if the labelling budget runs out, and only after its agreement with a human rater has been measured.

TechnoLynx builds this structure as the Production AI Monitoring Harness. The work starts by turning the failure-mode taxonomy into criteria that can be scored. It then sets the sampling for each category, implements the scorer, and reports judge-human agreement next to the coverage figures.

For a design you can cite, we would start from CheckList (Ribeiro et al., ACL 2020). It defines three test types:

• Minimum Functionality tests: small, focused labelled sets for one behaviour.
• Invariance tests: a change to the input that should not change the label must leave the prediction unchanged.
• Directional Expectation tests: the output should move in a known direction.

It treats the model as a black box, so the same suite can run against different or third-party models. Its authors found severe bugs in commercial models that standard benchmarks rated as accurate as humans. It also has a five-point severity scale, from “not a bug” to “no model with this bug should be in production”, which works well as the severity dimension of a pass/fail taxonomy. To adapt a support-ticket or chatbot harness to another agent domain, we would rewrite the taxonomy and the judge prompts, not just swap the data.

How do we check an LLM judge’s agreement with human raters before relying on it?

Measure Cohen’s kappa against a human-labelled subset, for each failure-mode category. Kappa corrects agreement for chance. Then compare the result with how well your human raters agree with each other. Correlation alone is not enough. The Judge’s Verdict study (2025) shows that a judge can correlate perfectly with humans while being consistently too harsh or too lenient, which kappa detects and correlation does not. Kappa is (Po − Pe) / (1 − Pe): observed agreement corrected for the agreement you would expect by chance. That study tested 54 LLM judges in two steps. First it filtered on correlation (r ≥ 0.80), which 36 passed. Then it applied a kappa-based human-likeness test, and 27 reached its top tier. Its human-to-human baseline was κ = 0.801.

Other studies add to the picture:

• Psychosis-response safety (2026). A study of judges scoring responses to users showing psychosis found kappa against human consensus of .75, .68 and .56 for three judges, and .74 for a majority-vote jury. Agreement on individual criteria ranged from .34 to .97. Those figures come from 16 test prompts, and the authors warn that per-criterion kappa may shift as the sample grows.
• No Free Labels (2025). This study found that a judge grades poorly on questions it cannot answer itself. On those questions, GPT-4o’s pairwise kappa fell from 0.78 to 0.14 when it used its own answers as references. Human-written reference answers improved agreement with human annotators. A small judge given human references beat a larger judge.
• Free-form QA (2025). Judges show positional, verbosity and self-enhancement biases. Combining several judges by majority vote improved alignment with human judgement.

A calibration procedure that follows from these studies:

1. Hand-label a held-out subset with at least two human raters, and compute their kappa as the baseline.
2. Give the judge a human-written reference answer for every item, and run it at temperature 0.
3. Compute judge-versus-human kappa for each category, not only overall.
4. Where the judge falls clearly below the human baseline, score that category with humans only, and say so in the results.
5. Repeat the check whenever the judge model, prompt or rubric changes.

What is a defensible sample size per failure-mode category for a hand-built test set?

There is no single correct number. We would treat a sample size as defensible when it is derived from the precision you need and reported alongside the result. Sample within each category, and oversample categories that are rare but severe, so each category’s pass rate stands on its own.

Worked example: how wide is the interval on a small category?

Assumptions: a pass rate near 50%, which gives the widest interval, and a 95% normal-approximation interval with half-width ≈ 1.96 × √(p(1−p)/n).

Items in category (n)	Calculation	Half-width	What a 60% pass rate really means
25	1.96 × √(0.25/25) = 1.96 × 0.10	±0.20	Roughly 40% to 80%
50	1.96 × √(0.25/50) ≈ 1.96 × 0.071	±0.14	Roughly 46% to 74%
100	1.96 × √(0.25/100) = 1.96 × 0.05	±0.10	Roughly 50% to 70%

Halving the interval width costs four times the labelling. Near 0% or 100% the normal approximation breaks down, so use an exact binomial or Wilson interval there. If one question yields several scored responses, bootstrap at the question level, because repeated responses to one question are not independent samples. The No Free Labels authors handled this the same way. Reporting a range instead of a single point estimate is also what SR 11-7 recommended to avoid false precision.

What does an investment committee need to see before approving more capital on stress-tested results?

An investment committee needs a structured go/no-go document, not raw stress-test output. It should contain four things:

• the thresholds agreed before testing began;
• the measured result against each threshold, including exchange-conformance status;
• the scenarios covered, with their assumptions and limits;
• the remaining risk, stated plainly enough for the committee to weigh it.

We would fix thresholds before the test window opens. Thresholds chosen after seeing the results measure the team’s optimism rather than the model.

Conformance testing needs its own line in the pack. The FIA (2024) recommends that exchanges require it whenever a participant deploys a new exchange-facing interface, and again after material changes. A conformance test confirms that the participant’s system interacts with the exchange properly. That is a baseline, not evidence of how the strategy behaves in a stressed market. The PRA’s SS5/18 expects trading algorithms and risk controls to be tested before deployment, and firms to assess latency and system capacity under normal and severe but plausible conditions. The FCA’s multi-firm review of algorithmic trading controls found that firms deployed algorithms slowly and in phases, with small pilot trades closely scrutinised. A staged rollout plan belongs in the committee pack for the same reason.

Can these results be packaged like exchange-conformance threshold evidence used elsewhere? The structure carries over: threshold, result, pass or fail, date. The meaning does not, because conformance thresholds reflect one venue and one test window. When the pack covers two systems, such as an execution-cost model and an RL execution policy, each should get its own go/no-go on a shared template, so the committee can compare the quality of evidence between them.

How do we document stress-scenario assumptions and limits so the committee does not overtrust them?

For each scenario, write down what was assumed, what range was covered, and where the model’s behaviour was not tested. Every model rests on assumptions about the data and the relationships in it, and those assumptions should be stated and checked against the data. SS1/23 asks for sensitivity analysis to find the operating limits within which performance stays acceptable. SR 11-7 expected testing to include scenarios outside ordinary expectations and to identify where the model becomes unreliable.

A scenario section the committee can rely on states:

• each scenario’s source (historical replay, synthetic or hybrid) and the period or generator used;
• the assumptions behind each scenario and the evidence that they hold;
• the operating limits found by sensitivity analysis;
• material differences between the test environment and production;
• the market regimes and tail events the scenarios do not cover.

What caveats and residual risks must the package disclose rather than omit?

Major limitations and key assumptions belong in the executive summary. Sometimes part of the validation cannot be done before use, for example because of data paucity. SR 11-7 required that fact to be documented and communicated, and the resulting uncertainty to be covered by compensating controls. SR 26-2 asks for the same openness when urgency forces use before validation is complete. The pack should also state three things plainly:

• a passing stress test covers only the scenarios it included;
• conformance results apply to one venue and one test window;
• open issues remain on the tracker.

Packaging is a communication step. It cannot turn weak underlying tests into strong evidence.

How do we build a regression harness that compares the old and new model before every promotion?

A regression harness runs the baseline and the candidate on the same held-out cases, fixtures and scoring path. The set might be claims scenarios, prompts and seeds, historical deals or simulated order flow. The harness scores both models against pre-set thresholds, checks each slice as well as the total, and gates promotion automatically.

For model swaps, TechnoLynx builds this harness as part of the LLM Selection Pack, and the Production AI Monitoring Harness keeps it running as a promotion gate. The first step is to define the baseline and the candidate, the replay dataset, and the metrics and thresholds, together with your risk or compliance owner.

The gate policy is written down before it is wired into CI. OneUptime’s 2026 guide to LLM regression gates lists what the contract should fix for each metric and critical segment:

• the dataset and rubric versions;
• the candidate and baseline versions;
• decoding settings and number of repetitions;
• whether the metric is deterministic, human-labelled or judge-based;
• the maximum acceptable regression and the absolute floor;
• the confidence level and decision rule;
• what happens when a score is missing.

The same guide gives the rules for running the gate:

• Compare candidate and baseline case by case, not across unrelated aggregate runs.
• Clear the gate only when the confidence interval on the mean difference stays above an allowed regression (−0.02 in its example).
• Add hard constraints that no average can make up for, such as zero unsafe tool calls.
• Report timeouts and malformed judge output as evaluator errors. Never drop them silently or count them as passes.
• Run cheap checks on every pull request, and the large, repeated, judge-based runs nightly or before release.

Breaking performance down by segment (loan type, region, credit band) shows where a model struggles. CheckList-style behavioural tests expose capability failures in models that benchmarks rate highly. Yan et al. (CVPR 2021) name the specific failure a swap introduces: “negative flips”, cases the old model got right and the new one gets wrong. We would count negative flips in each segment, not only the net change in accuracy.

Which statistical test decides “same enough” versus “regression”?

For two classifiers scored on the same cases, McNemar’s test is a non-parametric paired comparison. Dietterich preferred it to a difference-of-proportions test. It uses only the two disagreement counts, B and C: the cases where one model is right and the other wrong. The statistic is χ² = (B − C)² / (B + C).

• The chi-squared approximation needs B and C to be reasonably large, around 25 or more.
• For small counts, an exact binomial test is the better choice.
• Above about 50, the continuity-corrected McNemar test gives almost the same p-values as the binomial test.

One of those two disagreement counts is the negative-flip count. If the models can be refitted repeatedly, the 5x2cv paired t-test is slightly more powerful, at about ten times the computation. Comparing several candidates calls for follow-up pairwise tests with a Bonferroni correction.

A significance test answers “is there a difference?”, which is not the question a release gate asks. For “same enough”, a non-inferiority test fits better: set the largest regression you will accept in each segment, then require the paired confidence interval to stay above it. For adjusted models, SR 11-7 described the same logic as parallel outcomes analysis. If the adjusted model does not beat the original on real outcomes, further changes are likely needed before it replaces the original.

How do we keep the harness from going stale?

A harness only catches failure modes that its held-out set contains. When fraud patterns, the claims mix or the market regime shift, the replay set needs refreshing on a schedule. Once the model is live, thresholds on how far input distributions have drifted can trigger a review. Jensen-Shannon divergence and Wasserstein distance are two ways to measure that drift. Every result should name the version of the held-out set it ran on. Even a passing harness may not satisfy a risk committee that expects independent review.

How do I run a shadow-mode canary before promoting a retrained model?

“Shadow-mode canary” joins two gates that answer different questions. Shadow mode copies live requests to the candidate with no effect on users. A canary serves the candidate’s output live to a slice of users, with rollback ready. The guide separates three stages:

Gate	Who sees the candidate’s output	What it proves	Conditions that matter
Shadow	Nobody. Production answers the user, and the candidate’s response is scored offline	The candidate does not behave very differently on real traffic	Stop on a rubric drop or on failures that appear only in the candidate
Mirror	Nobody. Only a fraction of traffic is copied, 10 to 25% by default in that guide	The same as shadow, at a cost you can afford	Use it when the candidate is expensive to run
Canary	A percentage of real users	The candidate is at least as good with users in the loop, judged on signals such as retry and escalation rates	Rollback triggers written in advance, users grouped by risk, a 24 to 72 hour hold

In that guide, rollback triggers are written into the configuration before any traffic moves. Any one of the guide’s triggers reverts the canary on the next request, among them:

• the guardrail trip rate exceeds 1.5 times its trailing seven-day baseline over 15 minutes;
• p99 latency stays above 1.3 times baseline for 10 minutes;
• a failure cluster appears that only the candidate produces.

The guide’s fix is to start with the lowest-cost user group and widen from there. It also computes the required sample size before launch. It treats offline evaluations as necessary but not sufficient, because a candidate that wins on shadow can still lose on canary signals such as retry rate. For a regulated model, we would let the tier decide how far a canary goes before independent sign-off. At a high tier, shadow results are evidence for the validator, not a substitute for validation.

Who can build the release gates for our retrained models?

TechnoLynx builds the regression gates a retrained or swapped model has to clear before promotion. That covers the replay regression harness with its pre-agreed thresholds, the judge calibration, and the CI integration that gates promotion.

Whether you build in-house or bring in a specialist, your risk owner should own the thresholds. A specialist helps most when the harness has to convince an independent validator or committee by a deadline. Judge a partner on two things: whether they hand over a harness you can rerun without them, and whether their evidence would stand up to your validators. The work is scoped through the Production AI Monitoring Harness, with the LLM Selection Pack when the release is a model swap.

This work has limits. TechnoLynx produces engineering evidence. It does not give legal advice, regulatory sign-off or certification. CheckList’s open-source release already offers reusable test suites and perturbations. A buyer looking for the cheapest hours, rather than a harness they will own and rerun, would be better served elsewhere.

Deciding whether a retrain counts as a material change

Each firm needs its own criteria for when a model change is material. SS1/23 expects firms to set materiality criteria for model changes, and to define the validation, independent review and approval steps those changes trigger. SR 11-7 required material changes in structure or technique to be validated before implementation. That definition decides whether a routine retrain on fresh data passes through the regression harness alone, or reopens full validation and the tier assignment with it. If the definition is vague, every retrain becomes an argument. If it is precise, the harness, the canary and the evidence pack become routine work.