A feature-importance chart is not explainability evidence. It becomes evidence the moment a reviewer can trace one model output back to its contributing factors during an audit walk — and not a second before. That distinction is where most “explainable” models quietly fail their first regulated audit. The common move is to attach SHAP or LIME output to a model, screenshot the chart, and declare the model explainable. It looks convincing in a demo. It answers the wrong question. An auditor examining a HIPAA or GxP workflow does not ask “which inputs mattered on average.” They ask something narrower and harder: can you show why this model produced this output for this patient record, and can you trace that reasoning against the workflow’s approval criteria without a multi-day engineering pull. That gap — between attribution as a capability and explainability as auditable evidence — is the subject of this article. What machine learning model explainability actually means in a regulated workflow Strip away the tooling and explainability is a claim about traceability: given a model output, you can reconstruct the factors that produced it and defend that reconstruction to someone who did not build the model. In an ordinary ML project that claim can live in a notebook. In a regulated workflow it has to live in the evidence pack, structured so a reviewer can find the answer to a single-decision query without re-running anything. The naive interpretation treats explainability as a single dashboard bolted to the model. The expert interpretation treats it as evidence that maps to a specific regulated question, produced and documented per regulated step. The divergence point is auditability. Post-hoc attribution methods answer “which inputs mattered.” They do not answer “was this decision defensible under the workflow’s approval criteria.” Those are different questions, and only the second one survives an audit. This is why an explainability claim is really a documentation claim. The method, its limitations, and the evidence it produces at each regulated step have to be written down inside the pack. Bolt the chart on as a demo artefact and you have a capability the team believes it has — right up until an auditor asks for the record behind one decision and the reconstruction turns into an engineering project. Intrinsic models versus post-hoc explanation methods The first real decision is whether explainability is a property of the model or a layer added afterwards. The two paths carry different evidence obligations. Intrinsic (interpretable-by-design) models — logistic regression, single decision trees, shallow rule sets — produce a reasoning trace as a native side effect. The coefficient, the split, the rule that fired is the explanation. There is nothing to approximate. The trade-off is capacity: you accept a simpler hypothesis space in exchange for a reasoning trace you never have to defend as an approximation. Post-hoc methods — SHAP, LIME, integrated gradients, attention rollups — sit on top of a model that is not itself interpretable and estimate what the model was probably attending to. They are approximations of the model’s behaviour, not readouts of it. That is not a criticism; it is a property, and it is the property you must document. SHAP’s game-theoretic attributions and LIME’s local surrogate fits are both answers to “what would a simpler explanation of this region of the model’s behaviour look like” — useful, but explicitly not “this is the causal reason the model decided.” When to reach for each Question Intrinsic model Post-hoc method (SHAP / LIME) Do you need the reasoning trace to be exact, not approximate? Yes — the trace is native No — the trace is estimated Is model capacity the binding constraint? You accept less You keep the complex model Can you document approximation limits honestly per step? Not required Required, or the evidence is fragile Third-party / hosted model you don’t control? Rarely available Often the only option Auditor asks “why this specific output”? Answered directly Answered with a documented caveat The heuristic we apply in regulated engagements: use the simplest model the accuracy budget allows, because an intrinsic reasoning trace is the cheapest explainability evidence you will ever produce. Reach for post-hoc methods when accuracy genuinely requires a model you cannot interpret directly — and then treat the limitations of the method as a first-class part of the evidence, not a footnote. What are the known limitations of post-hoc explainability, and how do you document them honestly? Every post-hoc method has failure modes that an honest evidence pack names before an auditor does. LIME’s local surrogate can be unstable — re-sampling the neighbourhood around the same input can shift the explanation, an observed pattern in practice rather than a fixed defect. SHAP’s attributions assume a feature-perturbation baseline that may not correspond to any real patient state, so “feature X contributed +0.3” is a statement about the model under that baseline, not about clinical reality. Attention weights in transformer models correlate with importance loosely at best; treating an attention map as an explanation is a well-documented overreach. Honest documentation does not hide these. It states them as scoped claims. The pattern that survives audit looks like this: Method named — “Per-decision attributions produced with SHAP (TreeExplainer), version pinned.” Baseline declared — what the perturbation baseline is and why it was chosen. Limitation stated — “Attributions are approximations of model behaviour under the declared baseline; they do not establish clinical causality.” Evidence scope fixed — which regulated step this attribution supports, and what it deliberately does not claim. An auditor is not unsettled by a stated limitation. They are unsettled by a confident chart with no stated limitation, because it signals the team has not thought about where the method breaks. Documenting the boundary is the credibility move, not the risk. Per-decision explainability versus global model explainability This is the distinction that separates a demo from an audit answer, and it is the one most teams collapse. Global explainability describes the model’s behaviour in aggregate: on average, across the validation set, these features carry the most weight. It is genuinely useful for model development and for a high-level governance narrative. It is not what an auditor examining a single adverse decision needs. Per-decision explainability answers the query an auditor actually raises: for patient record #4471, on the date this output drove a workflow action, what were the contributing factors, and does that reasoning hold against the approval criteria that governed the step. A regulated workflow needs per-decision explainability because audits are conducted one record at a time. A global feature-importance chart cannot answer a single-record query — and the moment a reviewer asks, the absence becomes visible. The broader question of what auditors and model-risk reviewers actually need sits alongside this: the reviewer’s job is to trace, not to admire aggregate statistics. Design the evidence around the single-record trace and the global view falls out for free; design it around the global chart and you will rebuild everything the first time a specific decision is questioned. Where explainability evidence sits in the pack — and where validation ends Explainability and validation are complementary sections of the same evidence pack, and confusing them is a recurring failure. Validation evidence shows the model performs: accuracy, calibration, drift monitoring, the record that the model meets its intended-use criteria across the validated population. Explainability evidence shows why a given output occurred. Reliability answers “does it work”; explainability answers “why did it decide this.” An auditor expects both, in different sections, cross-referenced. The line is clean once you see it. If the question is “is this model fit for its intended use,” that is validation — the territory covered when you work out what makes an AI or video workflow HIPAA- or GxP-ready. If the question is “walk me through the reasoning behind this specific output,” that is explainability. The two attach to different regulated steps, and the pack should make that attachment explicit rather than leaving a reviewer to guess which section answers which query. This mapping is the core of how we structure AI governance and trust evidence: each regulated step names the evidence it produces, and explainability is the section that carries the “why this output” answer for that step. How do you produce explainability evidence for third-party or hosted models? This is where teams get stuck, because you cannot run SHAP against internals you do not control. The honest answer is that you shift the evidence from internal attribution to observed input–output behaviour and documented boundaries. For a hosted model, the defensible pack captures: the exact inputs sent per decision, the output returned, the version identifier of the served model, and any provider-supplied attribution or confidence signal — treated with the same limitation discipline as your own methods. Where the provider exposes nothing, you document that boundary explicitly and lean harder on input-perturbation testing you can run from outside: change one input, record how the output moves, and build a local behavioural record. That is weaker than an intrinsic trace and you say so. Pairing this with a calibrated uncertainty estimate produced with Bayesian inference gives a reviewer a second, independent signal about how much weight a given output deserves. The evidence pack does not require you to have opened the model. It requires you to have documented, honestly, exactly how much you can and cannot say about why it decided — and to have that documentation attached to the regulated step, not sitting in someone’s notebook. The ROI: from a multi-day pull to a structured lookup The reason to do this work upfront is not compliance theatre. Explainability evidence mapped into the pack lets a reviewer trace a specific model output to its contributing factors in minutes rather than reconstructing it from raw logs. The per-decision audit query drops from a multi-day engineering pull to a structured lookup — an observed pattern across regulated engagements, not a benchmarked figure, but a consistent one. It also travels. An explainability answer built into the pack is a repeatable audit answer that moves across sites unchanged, which means you stop re-litigating “how does the model decide” at every audit cycle. That is the difference between explainability as a demo capability and explainability as a durable asset: one impresses in a meeting, the other holds up when an auditor points at a single record. FAQ How should you think about machine learning model explainability in practice? Explainability is a traceability claim: given a model output, you can reconstruct the factors that produced it and defend that reconstruction to someone who did not build the model. In practice, in a regulated workflow, that means the method, its limitations, and per-step evidence live inside the evidence pack — structured so a reviewer can answer a single-decision query without re-running anything. What is the difference between intrinsic (interpretable-by-design) models and post-hoc explanation methods like SHAP and LIME, and when should each be used? Intrinsic models — logistic regression, single trees, rule sets — produce a native reasoning trace that is exact, at the cost of model capacity. Post-hoc methods sit on an uninterpretable model and estimate what it attended to; they are approximations, not readouts. Use the simplest interpretable model your accuracy budget allows, and reach for post-hoc methods only when accuracy genuinely requires a model you cannot interpret directly — then document the approximation as first-class evidence. What are the known limitations of post-hoc explainability methods, and how do you document them honestly in an evidence pack? LIME’s local surrogate can be unstable across re-sampling; SHAP’s attributions depend on a perturbation baseline that may not map to any real state; attention weights correlate with importance only loosely. Document each honestly by naming the method and version, declaring the baseline, stating the limitation explicitly, and fixing which regulated step the evidence supports and what it deliberately does not claim. A stated limitation reassures an auditor; a confident chart with no stated boundary does the opposite. What explainability evidence does a HIPAA / GxP auditor actually expect behind a single model output, and where does it sit in the pack? An auditor expects a per-decision trace: for this record, on this date, the contributing factors and whether the reasoning holds against the step’s approval criteria. It sits in the explainability section of the evidence pack, cross-referenced to the specific regulated step, distinct from the validation section that shows the model performs. How does per-decision explainability differ from global model explainability, and which does a regulated workflow need? Global explainability describes the model’s average behaviour across the validation set; per-decision explainability answers why one specific output occurred. Audits are conducted one record at a time, so a regulated workflow needs per-decision explainability — a global feature-importance chart cannot answer a single-record query. How do you produce explainability evidence for third-party or hosted models where you don’t control the internals? Shift from internal attribution to documented input–output behaviour: capture exact inputs, outputs, the served model version, and any provider attribution or confidence signal, all under the same limitation discipline. Where the provider exposes nothing, run input-perturbation testing from outside and document the boundary explicitly. The pack does not require you to have opened the model — it requires an honest record of how much you can and cannot say about why it decided. How does explainability evidence relate to validation evidence — where does one end and the other begin? Validation evidence shows the model performs — accuracy, calibration, drift, fitness for intended use. Explainability evidence shows why a given output occurred. Reliability answers “does it work,” explainability answers “why did it decide this”; both belong in the pack, in different, cross-referenced sections attached to their respective regulated steps. The question that decides the design Before you attach a single chart, ask which query the evidence has to answer. If the answer is “which inputs mattered on average,” you are building a development aid. If the answer is “why did the model produce this output for this record, and does the reasoning survive the approval criteria,” you are building audit evidence — and the two look nothing alike once a reviewer starts asking. The failure class to avoid is explainability-as-demo-artefact: a capability the team is sure it has until the record behind one decision is requested. Map the method, its limits, and its per-step evidence into the pack from the start, and that request becomes a lookup instead of a fire drill.
