A procurement committee asks a fair question: “why did the model produce this output?” The naive answer is to point at an aggregate accuracy number or an attention heatmap and call the model explainable. That answer collapses the moment a regulator or approver asks a case-level “why” — because a saliency artefact explains a mechanism, not the decision the buyer has to defend. Machine learning explainability, in the context of a procurement-grade LLM evaluation, is not a research artefact you attach to a model card. It is a claim the buyer’s workflow must actually consume: the right method for the model class, an honest statement of what the output can and cannot support, and a chain back to the specific decision under review. An explainability signal scoped to the eval’s actual questions survives committee scrutiny. A decorative one — however visually convincing — does not. What machine learning explainability actually means in practice Explainability is often treated as a single property a model either has or lacks. In practice it is a family of methods, each producing a different kind of evidence, each valid only for certain model classes and certain questions. A gradient-based saliency map over a vision transformer tells you which input regions moved the logits. A SHAP attribution over a tabular gradient-boosted model tells you which features contributed to a score and by how much, additively. A token-attribution readout over an LLM prompt tells you which prior tokens the attention heads weighted — which is a statement about the forward pass, not about why the answer is correct or defensible. These are not interchangeable. Confusing “I can show which tokens the model attended to” with “I can justify this decision to an approver” is the most common failure we see when explainability shows up in a procurement pack. The distinction that matters for procurement is between a mechanistic explanation and a decision-level justification. A mechanistic explanation describes how the output was produced. A decision-level justification connects the output to the criteria the buyer’s review will apply. The first is cheap and often automatic; the second requires you to know, in advance, what questions the committee will ask. That framing carries directly into what machine learning model explainability means in practice, which treats the method-selection problem in more depth for individual model classes. Which explainability methods fit which model classes? The method has to match the model, and the output has to match the question. Choosing a method because it is popular rather than because it fits the model class produces an artefact that looks like evidence and functions as noise. The table below maps common methods to what they actually support — and, more importantly, what they do not. Method Fits model class Output supports Does not support SHAP / feature attribution Tabular, tree ensembles, some NN Additive per-feature contribution to a score Causal claims; correctness of the decision Gradient saliency / Grad-CAM CNNs, vision transformers Which input regions moved the logits Why the classification is right Attention / token attribution Transformer LLMs Which prior tokens were weighted in the pass Whether the answer is factually grounded Counterfactual / perturbation probes Model-agnostic How the output changes when inputs change A single canonical “reason” for one output Retrieval-source tracing (RAG) Retrieval-augmented LLMs Which documents the answer was conditioned on Whether the model faithfully used them Two things about this table are worth stating plainly. First, for generative LLMs the most decision-relevant explainability signal is often not an interpretability method at all — it is retrieval-source tracing: showing the committee which documents an answer was conditioned on. That maps a contested output to auditable provenance in a way an attention heatmap never can. Second, no row in the table supports the claim a committee most wants: “this specific output is correct.” Explainability narrows the space of unexplained behaviour; it does not certify a decision. What can an explanation legitimately claim? The line an explanation must not cross is the line between description and warrant. An explanation can legitimately claim that certain inputs, features, or retrieved sources influenced an output, and it can quantify that influence within the assumptions of the method. It stops short — always — of claiming the output is correct, safe, or compliant. Those are separate claims that need separate evidence. This matters because procurement committees, under time pressure, tend to promote an explanation into a justification it cannot bear. A SHAP plot showing that “prior claims history” drove a denial is a description of the model’s behaviour. Whether that denial was justified is a policy question the explanation cannot answer. We treat this as a claim-class boundary: an explainability output is an observed-pattern artefact about model behaviour, not a benchmark of decision quality. Blurring the two is how an approval gets re-litigated six months later when a disputed output surfaces and the pack cannot answer for it. Getting this boundary right is why explainability sits alongside, not inside, the model’s quality metrics. Which model metrics actually decide a serving configuration is a separate question from which explanation defends a contested output — and confusing the two produces a pack that measures the wrong thing. How explainability evidence fits into the procurement evidence pack A procurement-grade LLM evaluation is not a single score; it is an evidence pack that has to survive committee review. Explainability is one component of that pack. It answers the case-level “why” — not a general model-quality claim, which the eval’s accuracy, cost, and reliability components already carry. The structural point is that the explainability signal is chosen from the committee’s question list backward, not from the method catalogue forward. If the committee will ask “on what basis did the model flag this transaction?”, the pack needs feature or retrieval-source attribution for flagged cases, not an aggregate interpretability report. If the committee will ask “can we reproduce the reasoning?”, the pack needs deterministic decoding conditions logged, which is closer to experiment tracking that feeds release readiness than to any explainability method. We build the explainability component of the pack the same way — starting from the specific questions the review will pose, then selecting the method that answers each one on demand. This is the discipline our [production AI monitoring harness](Production AI Monitoring Harness) is built around, and it is the reason explainability is scoped per-eval rather than bolted on generically. For teams standing up this kind of review capability inside an AI-infrastructure SaaS platform, the evidence pack is the deliverable — the model choice is just the thing it defends. A scoped explainability signal: worked example Assume a lending-adjacent SaaS product running a RAG-based LLM that drafts eligibility rationales. The committee’s contested-output question is: “when a rationale cites a disqualifying factor, can we show it came from the applicant’s actual record?” Wrong artefact: a token-attention heatmap over the generated rationale. It shows which prompt tokens the model weighted — a mechanism, useless to the approver. Scoped artefact: retrieval-source tracing that links each disqualifying claim in the rationale back to the specific source document and passage. This answers the committee’s exact question on demand. Boundary stated: the trace proves the claim was conditioned on that source; it does not prove the model interpreted the source correctly. That residual gap is handed to human review, explicitly, in the pack. The scoped artefact costs more to build. It also survives the review, which the heatmap does not. How does the explainability requirement change when the deployment is regulated? Regulation raises the bar from “can you explain it” to “can you explain it to a specific party, on a specific timeline, in a form they accept.” Under regimes that grant individuals a right to an explanation of automated decisions, the relevant explanation is decision-level and case-specific — the applicant’s case, not the model’s aggregate behaviour. A global feature-importance chart does not satisfy that; a per-case attribution or retrieval trace, retained and retrievable, does. The practical shift is that explainability stops being optional documentation and becomes a retention and retrieval obligation. The pack has to demonstrate not just that a per-case explanation can be produced, but that it will be produced within the required window and stored against the decision record. This is where explainability starts to overlap with governance-facing evidence: the same boundary discipline that keeps an explanation from overclaiming also keeps the governance pack honest about what the automated system does and does not decide. We keep the two components separate but aligned so a governance reviewer and a technical reviewer read consistent claims. What does explainability miss that monitoring catches after deployment? Explainability is an evaluation-time artefact. It reasons about the model as it was assessed, on the data it was assessed against. It says nothing about drift, distribution shift, or the slow degradation that surfaces only in production. An explanation that was faithful at eval time can describe behaviour the deployed model no longer exhibits. This is the seam between the procurement pack and operations. A committee can approve a model whose explainability component is airtight, and still be blindsided when the input distribution shifts and the model’s flagged-case rate doubles for reasons no eval-time explanation anticipated. That is a monitoring problem, not an explainability gap — which is why the two are complementary components of the same evidence discipline rather than substitutes. What machine learning model monitoring means in practice covers the post-deployment half of this picture: the questions explainability cannot answer because they only exist once real traffic arrives. FAQ What does working with machine learning explainability involve in practice? Explainability is a family of methods — feature attribution, gradient saliency, attention tracing, counterfactual probes, retrieval-source tracing — that each produce a different kind of evidence about how a model produced an output. In practice it means selecting the method that fits the model class and answers a specific question, not attaching a generic artefact to a model card. Which explainability methods fit which model classes, and what does each output actually support? SHAP and feature attribution fit tabular and tree models; gradient saliency fits CNNs and vision transformers; attention and token attribution fit transformer LLMs; retrieval-source tracing fits RAG systems. Each output supports a description of how the output was produced within its own assumptions — none of them certifies that a given output is correct. What can an explanation legitimately claim, and where does it stop short of a decision-level justification? An explanation can legitimately claim that certain inputs, features, or sources influenced an output, and quantify that influence within the method’s assumptions. It stops short of claiming the output is correct, safe, or compliant — those are separate claims requiring separate evidence, and promoting a description into a warrant is how approvals get re-litigated later. How does explainability evidence fit into the procurement evidence pack that defends a model choice? Explainability is one component of the pack, answering the committee’s case-level “why” rather than a general model-quality claim. The signal is chosen from the committee’s question list backward — starting from the questions the review will pose, then selecting the method that answers each one on demand. How does the explainability requirement change when the deployment is regulated? Regulation shifts the requirement from “can you explain it” to “can you explain it to a specific party, on a required timeline, in a form they accept.” A per-case decision-level explanation must be producible within the mandated window and retained against the decision record, turning explainability from optional documentation into a retention-and-retrieval obligation. What does explainability miss that operational monitoring catches after deployment? Explainability is an evaluation-time artefact that reasons about the model as assessed on the data it was assessed against; it says nothing about drift or distribution shift. Monitoring catches the post-deployment degradation an eval-time explanation cannot anticipate, which is why the two are complementary components of the same evidence discipline. Which procurement-review questions should an explainability signal be able to answer on demand? The signal should answer the committee’s contested-output questions: on what basis was this case flagged, which sources did this answer condition on, and can the reasoning conditions be reproduced. If the signal cannot answer the specific questions the review will pose, it is decorative — and it will fail at the exact moment a case-level “why” is asked. An explainability signal earns its place in the pack only if it can name, before the review starts, which question it answers and where it stops. When a case-level “why” arrives and the answer is a heatmap, the failure is not the method — it is that the evidence was never scoped to the decision the buyer has to defend.
