Model-agnostic attribution methods such as Shapley values, permutation importance and LIME run on a neural-network forecaster, alongside TimeSHAP for recurrent models and Integrated Gradients for deep networks. We would treat an attribution as evidence only once it is mapped to economic drivers, read regime by regime, disclosed with its instability and reproduced by someone independent on data the model’s builders never touched.

TechnoLynx builds that evidence as engineering work within its AI engineering for finance practice, from the attribution pipeline to stable, reproducible backtests, and the first step is framing with the model-risk team which trustee and regulator questions the evidence has to answer.

Which model interpretability techniques work on a neural-network forecaster?

Every technique below runs on a trained network without changing it. Where they differ is in what they explain and how they fail on sequential financial data.

Technique	What it gives you	Where it fits a forecaster	Known weakness
SHAP (Kernel SHAP)	Additive attribution of each prediction to its inputs, with a unique solution in its class	Treats the network as a black box through weighted regression over perturbed inputs, so it fits any architecture	Rankings of correlated inputs can invert between training seeds
TimeSHAP	Feature-, timestep- and cell-level attributions for recurrent models	LSTM and RNN forecasters with a lookback window	Compute cost and attribution variance, both reduced by its pruning step
Integrated Gradients	Gradient-path attribution satisfying the Sensitivity and Implementation Invariance axioms	Differentiable networks; needs only a few gradient calls	In the ROAR image benchmark its unsquared estimates ranked features worse than a random assignment
Permutation importance	Increase in prediction error when one input is shuffled	Global importance over backtest periods with observed outcomes	Measures influence on accuracy, so it cannot be computed for forecasts whose outcome is not yet known
LIME	Local linear surrogate around one prediction	Explaining a single forecast to a reviewer	The perturbation scheme and proximity kernel can significantly change the explanation
Gradient saliency	Sensitivity of the output to each input	Quick sensitivity check	On time series, saliency methods failed to identify importance over time reliably in a NeurIPS 2020 benchmark; two-step temporal rescaling improved them

SHAP and permutation importance answer different questions, and they can disagree. In a Bank of England working paper on economic forecasting (Buckmann and Joseph, 2022), a random forest ranked the three-month Treasury bill higher by permutation importance than by Shapley values, and the authors argue Shapley values suit forecasting better because they explain the predicted value rather than its accuracy. The same paper adds a further step: Shapley regressions that test whether each variable’s contribution is statistically significant, so results can be communicated like a regression table.

Intrinsic or post-hoc, model-agnostic or model-specific: which fits a neural forecaster?

Two distinctions shape the choice, and the first is about vocabulary. One useful definition treats interpretability as the structural transparency of a model and explainability as the ability to map it onto domain knowledge. For a forecaster, a SHAP chart is interpretability; tying it to term premium or inflation expectations is explainability, and the second is what a trustee is really asking for.

Intrinsic methods build interpretability into the model through trees, additive models or constrained layers, while post-hoc methods apply a second technique to a model that is already fitted. Intrinsic interpretability costs expressivity, the property that makes neural networks attractive for complex, unknown functions. There is a middle route: the Hybrid-Lift longevity model anchors an LSTM’s predictions to actuarial baselines with a mean-bias correction, and its authors propose embedding actuarial constraints directly in the training loss.

Model-agnostic methods (Shapley values, permutations, occlusion, LIME) run on any supervised model, whereas model-specific ones such as coefficients, tree importances and backpropagation attributions do not transfer. The practical advantage is comparison: the same attribution run across several model families shows whether an interpretation survives a change of algorithm.

We would default to model-agnostic, post-hoc attribution for a neural forecaster and run it on the linear benchmark as well, because a reviewer can then see where the network agrees with a model they already trust and where it departs. The Bank of England’s 2019 default-risk paper compared feature influences from a gradient tree boosting model with those of a logit regression as that sense-check.

SHAP rankings can flip between training seeds

A paper titled The Attribution Impossibility proves that when features are collinear no ranking can be faithful, stable and complete at once, and in its survey 68% of 77 public datasets showed rankings that inverted when the same model was retrained with a different seed. For a pair of collinear features, the ranking is a coin flip. Forecasting inputs are often correlated by construction (yields at adjacent maturities, several inflation measures), so a single-seed SHAP ranking is one draw from a distribution, not a finding.

The Hybrid-Lift authors say their SHAP influence hierarchy is sensitive to the trained weights and the KernelExplainer approximation, and should be read as indicative rather than as a stable ordering. On U.S. Treasury yields, the factor directions recovered from a linear and a random-forest forecaster sat between 39 and 73 degrees apart and shared at most four indicators.

The impossibility paper also gives a workable screen: for flagged pairs, train five models and compute a Z statistic, and if it falls below 1.96, report an ensemble-averaged ranking (its Dash method) over at least 25 models. Its recommended disclosure names the correlated group, gives the estimated flip rate, says the members should be read as interchangeable contributors and confirms that the between-group ranking is stable. Group-level reporting is the honest answer for a trustee too: inflation expectations as a group can be a stable driver while the order of the individual series inside it is noise.

Which explainability outputs can a risk committee actually sign against?

A risk committee can sign against aggregated, per-horizon driver summaries and regime-segmented backtest charts delivered inside the model risk management lifecycle, not against raw SHAP plots. Supervisors already expect explanations to differ by audience: EIOPA’s 2021 AI governance principles say consumers, auditors and supervisory authorities need different explanations. SR 11-7, the Federal Reserve and OCC model risk guidance, asks for validation reports with clear executive summaries and an accessible synopsis of results, major limitations and key assumptions. The Bank of England’s 2019 paper lists six stakeholder types, from developers through first- and second-line checkers to conduct and prudential regulators, each asking a model different questions.

Both layers can come from one computation: a board summary that ranks driver groups per horizon in plain language with the instability disclosure attached, and the full attribution detail for the validation team. The summary’s weakness is what it leaves out (edge cases, within-group instability), so the board version should say where the detail lives and who has read it.

SR 11-7 expects validation to continue after a model goes into use, a review at least annually, and validation of material changes. An attribution summary from the previous training run describes a model that no longer exists. Where this sign-off fits in an institution’s model and vendor approval process, and how deep validation must go, are governance decisions in their own right; SR 11-7 ties rigour to material impact on decisions such as risk management and capital planning.

Do the attributions make economic sense at each forecast horizon?

Checking economic sense means stating, before looking, which drivers should dominate at each horizon and then comparing. The ECB describes the yield curve as readable in terms of market expectations of monetary policy, economic activity and inflation over short, medium and long horizons. For a yield-curve forecaster that suggests policy-rate inputs should weigh most at short horizons and inflation-expectation and term-premium proxies further out; if the attribution shows the reverse, someone should be able to explain why. Buckmann and Joseph note that explaining predictions at different horizons can reveal differential signals from the same predictors.

The Bank of England’s 2019 default-risk work shows the check in use: loan-to-value and current interest rate came out among the top influences, in line with the economics literature, while a year variable was flagged because its rationale might not hold in live deployment.

Two limits keep the check honest. Attribution describes correlates inside the trained model, not causal drivers, a point the IIF’s 2018 explainability paper makes about the word “drivers” itself. A SHAP result that matches expectations shows the model is consistent with prior beliefs, not that it is correct. The Treasury yield study frames its own driver naming the same way, as a regularised association under a permutation null rather than a structural model.

What does a regime-conditioned backtest add to feature attribution?

Attribution says which inputs matter on average; a regime-conditioned backtest shows when the model’s reasoning holds and when it degrades. RegimeFolio, a portfolio-optimisation study, assigns each trading day to a low, medium or high volatility regime using the 33rd and 67th percentiles of the trailing 252-day VIX distribution. Its regime-wise SHAP showed momentum indicators dominating in low volatility and volatility measures dominating in high volatility. Its forecast errors also followed regime-dependent patterns that pooled, regime-agnostic training obscured.

Hybrid-Lift gained most where unit-root behaviour broke the Li-Lee mean-reversion assumption, showed no benefit for near-linear Switzerland and Japan, and showed a conservative bias during the 2020 pandemic shock. In the Bank of England unemployment forecasting comparison, differences between models were small before the global financial crisis and larger after it. SR 11-7 asks for testing across a variety of market conditions, including scenarios outside ordinary expectations, and for monitoring when a model that only works in some ranges approaches their edges.

A per-regime report a reviewer can read contains, for each regime:

• the rule that defines it, and the alternatives that were tried;
• out-of-sample error against the benchmark model;
• the top driver groups by attribution, with their stability across seeds;
• the periods where the model was overridden or performed worst.

RegimeFolio’s authors note that VIX-only classification is a limitation and that macroeconomic indicators or credit spreads could segment the market differently; alternative VIX-based definitions kept the result but with reduced magnitude. Regime backtests can be overfitted like any other backtest: Bailey and co-authors argue that standard hold-out tends to be unreliable for investment backtests and propose estimating the probability of backtest overfitting directly. History also caps coverage, and behaviour outside every historical regime is a separate extrapolation-testing question.

Building an explainability package a trustee can challenge in a meeting

The IIF found no single panacea for the explainability challenge. What works is a package of three parts that answer different questions: attributions mapped to the fund’s economic drivers (why did it do this), regime-conditioned backtests (how has it behaved), and documentation that describes the hybrid neural-network and rules architecture in the categories the governance template already uses (how is it governed).

TechnoLynx builds that package as an engineering deliverable: it selects attribution methods suited to the forecaster’s architecture and compute budget, aggregates raw attributions into the fund’s driver taxonomy, builds the regime-segmented backtests, checks that outputs are stable and reproducible across retraining runs, and packages them as reports for trustee and regulator meetings.

The Pensions Regulator’s guidance on AI in pension schemes expects trustees to answer, for each significant use of AI, what decisions it may influence, who is accountable, how it is tested, how a decision can be challenged and how humans can override it. It also warns that AI outputs may appear authoritative even when they are incomplete or inaccurate. An attribution report serves the challenge question only if a trustee can ask why the forecast moved and get an answer in named drivers.

Mapping raw inputs onto named drivers needs a defensible aggregation rule, or it reads as storytelling after the fact. Group-level reporting from the attribution-impossibility paper supplies one: report the total attribution of a correlated group and treat its members as interchangeable. Shapley regressions then test each driver’s statistical significance and present it like a familiar regression table.

SR 11-7 requires documentation detailed enough that parties unfamiliar with a model can understand how it operates, its limitations and its key assumptions. Where a template written for linear or scorecard models has no room for a two-component system, extend it rather than force the model into it: the rules engine described as a constraint layer, the network as a bounded forecasting module with its own limits and overrides. Reconciling the rules engine and the network into one examiner-readable narrative is a piece of work of its own.

Documented practice comes from banks: the IIF reported in 2018 that several banks began using machine-learning challenger models as part of validating their capital and provisioning models. For the questions auditors and model-risk reviewers bring, see machine learning explainability.

Why aren’t the vendor’s own explainability outputs enough for trustees?

SR 11-7 says validation should generally be done by people who are not responsible for development or use and have no stake in whether the model is judged valid. It adds that a developer is an important source of information but cannot be relied on as an objective or sole source on model quality.

LynxBenchAI benchmarks throughput, latency, memory, energy and cost per unit of work on your own hardware, so the hardware side of a vendor’s performance claims rests on your own measurements rather than the vendor’s figures, and the first step is running it on the hardware the forecaster will use in production.

Effective challenge, the guidance’s organising principle, depends on incentives, competence and influence, and incentives are stronger the further the challenge sits from development. For vendor products, banks are expected to validate their own use, rely more on sensitivity analysis and benchmarking where code is not accessible, and monitor outcomes on their own data. The PRA’s SS1/23 applies its principles to vendor models and to every modelling technology. Vendor attestations can support due diligence but do not replace internal evidence that the model behaves within approved bounds.

Attribution instability sharpens the point: a vendor’s SHAP report is one training run, and the deployer cannot tell from it whether the ranking would survive a retrain. Organising the validation function itself, with the separation and evidence trail that make it credible, is the subject of independent model validation: separation, evidence and scope.

Regime evidence for a yield-curve forecaster

Evidence that a yield-curve forecaster behaves across regimes combines out-of-sample error against a simple benchmark, attributions per regime and a significance test on any driver the model is credited with. A study of monthly U.S. Treasury zero-coupon yields from 2006 to 2025 compared a linear factor-augmented Nelson-Siegel model and a random forest against a random walk, and found the forest more accurate at maturities of two years and beyond. Its block-permutation test certified the business-cycle factor recovered from the linear model’s residuals, but not the one from the forest’s, which returned p = 0.135. The more accurate model left a driver signal too faint to certify at that sample length.

SR 11-7 defines back-testing as comparing outcomes with forecasts over a period not used in development, at a frequency matching the forecast horizon, and asks long-horizon models to add early-warning metrics. Stress scenarios belong in the same pack: the 2019 Bank of England paper ran its models through the Bank’s stress-test scenario and used its Shapley-based QII attribution to explain why the two models’ predictions differed.

How should a validator reproduce SHAP and backtest results on a holdout nobody touched?

We would have the validator assemble the holdout, control access to it and rerun both the attributions and the calibration from scratch. The order matters:

1. Build the holdout chronologically after the training window and fit every scaling parameter on training data only, the anti-leakage rule the Hybrid-Lift authors apply.
2. Ration access to it. Reusing a holdout adaptively, for hyperparameter tuning or feature selection, overfits to the holdout itself; the reusable-holdout method Thresholdout is one principled way round this, and otherwise each dependent validation needs fresh data.
3. Retrain across seeds before computing attributions, run the Z-test on correlated pairs and average models where rankings are unstable; the Bank of England forecasting study averaged ten models trained with different seeds.
4. Re-derive calibration twice: once with the vendor’s method to reproduce the claim, and once with an independent test such as a conditional calibration test, which a banking backtesting study found better at detecting misspecification among traditional backtests of risk forecasts.
5. Validate the interpretation itself with repeated data-splitting plus stability analysis, which assesses the bias and variance of the discovery together.
6. Investigate every discrepancy against the vendor’s figures, as SR 11-7 expects for differences from benchmarks.

The Hybrid-Lift paper is candid about the trap in step one: its mean-bias correction was calibrated on the same 2012 to 2020 period used to evaluate out-of-sample performance, so the reported improvements partly reflect that window, and a separate test set was impractical with nine years of data. A validator who finds that kind of overlap should report the affected result as in-sample.

How can attributions and calibration be benchmarked so auditors can defend them?

Attributions are benchmarked with removal tests, and calibration with comparative backtests against a reference model. Hooker and co-authors’ ROAR benchmark removes the features an attribution method ranks highest, retrains, and measures how far accuracy falls; on large image datasets many popular methods did no better than random, and only ensemble variants such as VarGrad and SmoothGrad-Squared beat a random ranking. Retraining matters because removing inputs without it creates a distribution shift that confounds the result. A cheaper deletion test neutralises features in importance order and tracks the predicted score. These benchmarks come from image classification; saliency methods have also been benchmarked on synthetic time-series data across recurrent, convolutional and Transformer networks.

For calibration, traditional backtests only test whether a model is correct and cannot rank alternatives, and in one simulation misspecified models hit almost exactly the expected share of VaR violations. Comparative backtesting with strictly consistent scoring functions ranks the internal model against a standard one and sorts the outcome into red, yellow and green zones at a fixed confidence level. Calibration and attribution are linked: in image classifiers, calibration correlated with more faithful maps from one attribution method and improved its stability. SR 11-7 sets the bar for the reference itself: benchmark models should be rigorous and benchmark data accurate and complete.

Mortality forecasts against the accepted actuarial model

A neural mortality-improvement forecast is easiest to defend as a challenger to the accepted actuarial projection, not a replacement for it. The Hybrid-Lift study benchmarked its LSTM against the Li-Lee model on 2012 to 2020 out-of-sample data, improving RMSE by 17.40% in Sweden and 12.57% in West Germany while staying comparable in Switzerland and Japan. It also checked that projected death rates rise monotonically with adult age, as the Gompertz law requires, and all six populations passed. That test is the actuarial counterpart of the economic-sense check: a hard constraint the model must not violate, fixed before the results are seen.

Cost and throughput figures under real production conditions

For the hardware part of such a case, LynxBenchAI benchmarks throughput, latency, memory, energy and cost per unit of work on your own hardware; it measures the hardware, not the reader’s own pipeline.

Who can independently validate a neural-network forecaster’s explanations?

TechnoLynx builds independent validation evidence for neural-network forecasters as engineering, evidence an auditor or model-risk committee can read, defend and sign against. Building this in-house works when the second line has machine-learning depth and real separation from the builders; a specialist helps when it does not, or when the vendor’s model is opaque. When choosing any partner, check that they had no hand in building the model, that they will rerun results rather than review slides, and that their report states limits. Reproducing interpretability and backtest results works as a release gate for a production model. TechnoLynx runs the engagement within its AI engineering for finance work, starting with the model-risk team’s list of questions.

The engagement produces the artefacts an approval needs; it is not regulatory sign-off, certification or legal advice, and the accountable owner still signs. Teams whose validators already run multi-seed attribution and comparative backtests may need only a method review, and an audit firm’s model-risk practice or an academic reviewer are realistic routes where the open question is governance rather than engineering.

Write the rejection criteria before reading the attributions

One control on an attribution report is an intervention plan written before the analysis, stating what result would raise trust, what would lower it and what would send the model back. Without one, any SHAP chart can be rationalised after the fact in either direction. Revalidation triggers belong in the same plan: material changes to use, input data or model version, and any monitoring signal that crosses an agreed threshold. For a neural forecaster that retrains on a schedule, the plan is what stops last quarter’s explanation being presented as this quarter’s model.