A team ships a lending model, runs SHAP over the test set, attaches a feature-importance bar chart to the model card, and marks the explainability box as done. Then the model-risk committee meets, and the first question is not “which features mattered.” It is: “For this applicant, who was declined, why should we trust this decision, and can we defend it if the regulator asks?” The chart does not answer that. It answers a different question, one the data scientist found interesting but the reviewer never asked. That gap is the whole subject of this article. Machine learning explainability is only useful when it is engineered as evidence for the audience that has to sign against it — not generated as a plot because a library made it easy. The naive approach treats explainability as a single library call. The expert approach starts from the reviewer’s question and works backwards to the artefact that answers it. What does machine learning explainability actually mean in practice? The word “explainability” collapses two very different things. One is an engineer’s diagnostic aid: an attribution that helps you understand whether the model learned something sensible or latched onto a spurious correlation. The other is a governance artefact: a document that lets a non-engineer read an explanation, understand the decision it justifies, and put their name against it. Both are legitimate. They are not the same deliverable, and confusing them is where most teams lose weeks at approval time. In practice, an explanation is fit for purpose only when three things line up: the decision being explained, the specific case it applies to, and the rubric the reviewer will judge it against. A global attribution over a validation set tells you how the model behaves on average. It says almost nothing about the individual applicant whose loan was declined — and the individual case is exactly what a lending or diagnostic reviewer must defend. When explainability is engineered as evidence rather than produced as a chart, the format is chosen to satisfy that review, not to look complete on a slide. This is a pattern we see regularly in regulated engagements: the explainability work was technically done and still failed review, because it answered “which features the model uses” when the committee needed “why this particular decision is trustworthy under our approval rubric.” The rework is not more modelling. It is re-framing the same signals into an artefact a reviewer can actually act on. Global versus local explanations: when does a reviewer need each? Global and local explanations answer structurally different questions, and a review that needs one is not satisfied by the other. A global explanation describes model behaviour across a population — which features drive predictions in aggregate, how the model responds to a range of inputs, where its dependence on a given variable is strong or weak. This is what a governance reviewer wants when the question is about the model as a whole: is it relying on a prohibited attribute, is its behaviour stable, is the overall logic sensible and documentable. Global permutation importance and partial-dependence-style summaries live here. A local explanation describes a single decision — for this input, why did the model produce this output. This is what a reviewer needs when the decision itself must be defensible per case: a declined loan, a flagged diagnosis, an escalated transaction. SHAP local attributions and LIME both operate at this level, approximating the model’s behaviour in the neighbourhood of one instance. The mistake is treating one as a substitute for the other. A model-risk committee approving a lending model typically needs both: a global view to confirm the model is not systematically discriminatory, and a local view to confirm that individual adverse decisions can be explained to the affected person and to a regulator. If your evidence pack has only the global chart, the per-case defence is missing; if it has only local attributions, the systemic-behaviour question is open. Which explainability methods produce evidence an auditor can sign against? The honest answer is that no method produces sign-off-grade evidence by itself. Methods produce signals; evidence is what you get after you match a signal to a decision, a case, and a rubric. That said, the methods differ in how well their output survives a reviewer’s scrutiny, and the differences are worth naming. Explainability method fitness for review Method What it produces Reviewer question it answers well Where it falls short for sign-off Global feature importance Ranked contribution of features across a dataset “Does the model rely on a prohibited or nonsensical variable overall?” Says nothing about any individual decision; averages hide case-level behaviour SHAP (local) Per-decision attribution with additive, consistent contributions “For this case, which inputs pushed the decision this way?” Attribution is not causation; reviewers over-read magnitude as importance-in-reality LIME (local) Local linear approximation around one instance “What simple rule roughly explains this one prediction?” Instability — re-running can shift the explanation; hard to defend as reproducible Counterfactuals Minimal input change that flips the decision “What would this applicant have needed to be approved?” Requires actionable, non-prohibited features; can surface many valid counterfactuals Intrinsic model structure (e.g. a decision tree or scorecard) The model’s own decision logic “Can we read the decision rule directly?” Only available when the model class is inherently interpretable Counterfactual explanations deserve a specific mention because they map cleanly onto the question a regulated reviewer actually asks. “What would have had to be different for this outcome to change” is directly defensible to an affected individual, and it sidesteps the trap of presenting a correlational attribution as if it were the model’s reasoning. When the underlying model is an ensemble — bagged trees, for example — the interpretability trade-offs shift again; we cover that structural angle in our note on how bootstrap aggregation produces defensible, stable predictions. None of these methods removes the reviewer’s judgement. What good engineering does is remove the ambiguity the reviewer would otherwise have to resolve in a clarification round — by stating the method, its scope, its known limits, and how the output maps to the decision under review. How do you match an explanation to the audience? The single highest-leverage move in explainability is choosing the audience before choosing the method. An explanation that a data scientist finds obvious is not the artefact a model-risk committee needs, and neither is what a buyer’s procurement lead is looking for. Same model, same underlying attributions, three different fit-for-purpose artefacts. Audience-to-artefact rubric Engineer / data scientist — wants to know the model learned something real. Local SHAP plots, dependence plots, and error-slice diagnostics are the right register. Precision matters more than readability; the audience can interpret the raw output. Buyer / procurement lead — wants to know the model is fit and the risk is bounded. Summarised behaviour, known failure modes, and a plain-language statement of what the model does and does not do. Raw attribution plots read as noise here. Auditor — wants traceability: what method, over what data, producing what artefact, stored where, reproducible how. The explanation’s provenance matters as much as its content. Model-risk owner — wants to defend the decision under a rubric. Per-case explanations tied to the applicable control, plus the systemic view that rules out prohibited behaviour. This is the audience the sign-off package is built for. When the audience is fixed first, the method selection and the artefact format follow. When the method is fixed first — because SHAP was easy to run — the artefact often lands on the wrong audience and gets sent back. This is the core of claim C17: explainability is only useful when engineered as evidence for the reviewing audience, matched to the decision, the case, and the rubric. Where does explainability stop being sufficient evidence of trust? Post-hoc explanation methods have a hard limit that is easy to talk past. SHAP and LIME do not reveal the model’s reasoning; they build an approximation of the model’s local behaviour and present it in a readable form. The approximation can be faithful or unfaithful, and the reviewer usually cannot tell which from the artefact alone. Treating a post-hoc attribution as if it were the model’s actual causal logic is the most common overreach we see, and a sharp reviewer will catch it. This is why explainability is a component of trust evidence, not the whole of it. An explanation says “here is a plausible account of this decision.” It does not, by itself, establish that the model is accurate, stable under distribution shift, or well-calibrated. Those properties come from validation, and the validation package produced under a reliability discipline is what the explanation artefacts interpret and reference. The two are complementary: validation establishes that the behaviour is sound; explainability makes a specific decision legible. Neither substitutes for the other. The practical boundary is this: explainability stops being sufficient evidence of trust the moment it is asked to carry a claim about correctness or robustness. Those belong to the validation and monitoring surfaces. An evidence pack that leans on explainability to prove the model is right — rather than to make its decisions legible — has a structural gap that review will find. How do explainability artefacts fit into an approval-grade evidence pack? Explainability outputs are not a standalone deliverable; they are line items in an evidence pack, scored against a readiness rubric. Under frameworks like the NIST AI Risk Management Framework and the ML Test Score readiness rubric, explainability sits alongside validation results, monitoring plans, data documentation, and change control. The explanation format has to be chosen so it maps to a rubric line, carries into the sign-off package directly, and survives an auditor’s request to reproduce it. Concretely, an approval-grade explainability artefact declares its method and scope, ties each explanation to the decision and case it covers, states its known limits, and stores its provenance so it can be regenerated. When it does those things, the explainability portion of a model-risk review compresses from repeated clarification rounds to a single pass — the measurable outcome the discipline is aiming at. When it does not, each review cycle re-litigates what the attribution means, and the recurring cost is the number of review iterations, not the modelling effort. This is the connection between explainability and the broader AI governance and trust discipline: the explanation is engineered to satisfy the review, not to satisfy the engineer who ran the library. We treat these outputs as scored line items precisely because a rubric forces the audience question to be answered up front. If an explanation cannot be mapped to a rubric control, it is not evidence yet — it is a plot waiting for a home. Our companion piece on what explainability means inside a regulated evidence pack drills into how those line items are assembled and scored. How does explainability differ for classical ML versus LLMs? For a classical model — a gradient-boosted classifier, a logistic scorecard, an ensemble of trees — explainability has decades of grounding. The feature space is fixed and named, attributions like SHAP have clear semantics, and counterfactuals are well-defined. A reviewer can reason about “the income feature contributed this much to the decline” because income is a stable, meaningful input. For large language models and generative systems, the ground shifts. There is no fixed, human-named feature space; the “features” are tokens and internal representations that do not map onto the concepts a reviewer cares about. Post-hoc attribution over tokens rarely produces something a model-risk owner can sign against, because token-level saliency does not explain why the model produced this content in terms the rubric understands. Explainability for LLMs tends to move toward behavioural evidence — documented evaluation over defined tasks, confidence and calibration analysis, and traceable prompt-to-output records — rather than feature attribution. For a reasoning model, this looks less like a SHAP plot and more like the evidence structure we describe for producing approval-grade evidence for a reasoning model. The reviewer’s question is the same; the artefact that answers it is different. The lesson carries across both worlds: the model class changes the method, but not the discipline. You still start from the reviewer’s question and engineer the artefact that answers it. FAQ What’s worth understanding about machine learning explainability first? Explainability methods produce an account of how a model behaves — either globally across a population or locally for a single decision. In practice it is only useful when engineered as evidence for a specific audience: matched to the decision being made, the individual case, and the rubric the reviewer will apply, rather than generated as a chart because a library made it convenient. What is the difference between global and local explanations, and when does a reviewer need each? A global explanation describes model behaviour across a population — useful for confirming the model is not systematically relying on a prohibited variable. A local explanation describes a single decision — needed when an individual adverse outcome must be defensible per case. Regulated reviews of lending or diagnostic models typically need both; a pack with only one leaves a real question open. Which explainability methods produce evidence an auditor or model-risk committee can actually sign against? No method produces sign-off-grade evidence on its own — methods produce signals, and evidence is what you get after matching a signal to a decision, case, and rubric. Counterfactuals map especially well to the regulated question (“what would have changed the outcome”), while SHAP and LIME require care because their attributions approximate behaviour and are not the model’s causal reasoning. How do you match an explanation to the audience so it survives review? Choose the audience before the method. An engineer wants raw diagnostic attributions; a buyer wants bounded-risk summaries; an auditor wants traceable provenance; a model-risk owner wants per-case explanations tied to a control plus a systemic view. When the audience is fixed first, the method and format follow; when the method is fixed first, the artefact often lands on the wrong audience and is sent back. What are the limits of post-hoc explanations, and where does explainability stop being sufficient evidence of trust? Post-hoc methods like SHAP and LIME build an approximation of local behaviour, not the model’s actual reasoning, and the reviewer usually cannot tell how faithful it is from the artefact alone. Explainability makes a decision legible; it does not establish accuracy, stability, or calibration. It stops being sufficient the moment it is asked to prove the model is right rather than to make its decisions readable — those claims belong to validation. How do explainability artefacts fit into an approval-grade evidence pack and map to rubrics like NIST AI RMF or ML Test Score? They are scored line items, not a standalone deliverable. An approval-grade artefact declares its method and scope, ties each explanation to the decision and case, states its limits, and stores reproducible provenance so it maps to a rubric control under the NIST AI RMF or the ML Test Score readiness rubric. Done well, this compresses the explainability portion of review from repeated clarification rounds to a single pass. How does explainability differ for classical ML models versus LLMs and generative systems? Classical models have fixed, named features, so SHAP attributions and counterfactuals carry clear meaning a reviewer can reason about. LLMs have no human-named feature space — token-level saliency rarely explains output in terms a rubric understands — so explainability shifts toward behavioural evidence: documented task evaluation, calibration analysis, and traceable prompt-to-output records. The reviewer’s question is the same; the artefact changes. The next time an explainability deliverable is declared “done,” the useful test is not whether a chart was produced. It is whether a non-engineer can read it, defend the decision it justifies, and sign against it under the rubric that governs the review — and if they cannot, the failure to fix is not the model, it is the fit between the explanation and its audience.
