A SHAP plot is not explainability evidence. It is a picture of a prediction that a data scientist can read. An approval committee cannot approve a picture. What a committee needs is an answer to a different question: can we explain the failure modes we are about to ship, on our task, at our stated risk tolerance — and can we defend that explanation eighteen months from now when a specific decision is challenged. That gap is where most LLM procurement evidence packs quietly fall apart. The team attaches an attention heatmap and an off-the-shelf interpretability notebook, marks “explainability” as satisfied on the intake form, and is surprised when the committee sends it back for “more detail.” The detail was never missing. The audience was wrong. What does ml model explainability actually mean in a procurement context? Explainability is not one thing. It is a family of techniques that answer different questions for different people. The classical toolkit — SHAP, LIME, partial dependence, feature attribution — was built to explain a single prediction to a technical practitioner who wants to debug or improve a model. That is a legitimate use. It is also not what a procurement committee is asking for. The committee is not debugging your model. It is deciding whether to accept accountability for the model’s behaviour across a population of decisions it will make in production. Its question is behavioural, not per-prediction: where is this model reliable, where is it not, and can we live with the “not” at the volume we expect to run. So there are really two explainability audiences, and conflating them is the root failure: The data-scientist audience wants to know why this output happened so they can iterate. Feature attribution and local surrogate models serve this well. The committee audience wants to know what class of outputs we should expect and which of those we cannot yet account for. That is a documented behavioural profile, not a saliency map. In our experience reviewing procurement packs, the packs that stall are the ones that ship the first audience’s artefacts to the second audience. The techniques are fine. The framing is a category error. Which explainability techniques hold up for an LLM, and where do they break? Classical explainability assumes a stable input-to-output mapping you can perturb and probe. For a gradient-boosted classifier or a bagged decision-tree ensemble, that assumption mostly holds — you can trace a decision back to feature contributions, and the same input reliably yields the same output. That is why the interpretability story for classical models is comparatively clean; techniques like feature attribution have a defensible mechanistic reading. If you want the classical-model version of this argument, what auditors and model-risk reviewers actually need from machine learning explainability covers it directly. For an LLM, three of those assumptions weaken at once, and the techniques inherit the weakness. First, attention is not explanation. Attention weights show where the model is looking, not why an output is what it is. A large body of work has shown that attention maps can be manipulated without changing predictions, which means they are — at best — a suggestive visualisation, not attributable evidence. Presenting an attention heatmap as “the reason for the output” is an observed-pattern overclaim that a competent reviewer will reject. Second, the input space is unbounded. Feature attribution over a fixed feature vector is well-defined. Attribution over free-form natural-language prompts is not — you cannot enumerate the input distribution, so any local explanation is conditional on a prompt you happened to test. The explanation holds on the benchmark set and may not hold on your prompt distribution, which is the distribution that actually matters. Third, outputs are non-deterministic. With temperature above zero, the same prompt produces different completions. A local explanation of “this output” is an explanation of one sample, not of the model’s behaviour. This is the point most transferred-from-classical evidence packs never confront. The honest response is not to abandon explainability. It is to shift what you explain from the prediction to the behaviour envelope — and to document reproducibility explicitly rather than pretending it away. A decision table: which explainability evidence for which model class Evidence type Classical ML (trees, GBMs) LLM (generative) What the committee reads it as Feature attribution (SHAP/LIME) Defensible per-prediction Conditional on tested prompt only Debugging aid, not behavioural proof Attention/saliency maps N/A Suggestive, not attributable Visualisation — do not cite as cause Failure-mode catalogue Useful Primary evidence Behavioural profile at a risk tolerance Reproducibility statement Rarely needed Mandatory (temperature, seeds, version) Where outputs can and can’t be replayed Held-out task-set behaviour Useful Primary evidence Reliability on our task, not a benchmark The pattern is clear: as you move from classical to generative, weight shifts away from per-prediction attribution and toward documented behaviour on a task the buyer actually cares about. Reading benchmark scores as behavioural evidence is a separate trap — what benchmark suites prove and where they fall short for LLM procurement unpacks why the leaderboard number is not your number. What does an approval committee need that a data scientist does not? A data scientist can accept “the model attends to these tokens.” A committee needs something it can put its name on. The difference is defensibility over time. Procurement-grade explainability evidence has to survive three moments a debugging notebook never faces. It has to survive the approval meeting, where a non-technical member asks what happens when the model is wrong. It has to survive the version review, where the vendor pushes an update and someone has to say whether behaviour changed. And it has to survive the challenge, where a specific downstream decision is contested and the buyer must point to what was known at approval time. None of those are served by a saliency map. They are served by a stable, versioned document that states the model’s failure modes, the conditions that trigger them, and the risk tolerance under which they were accepted. That document is the explainability section of the AI governance and trust evidence pack, and it is written for the committee’s job, not the data scientist’s. Concretely, committee-grade evidence answers: What are the named failure modes we observed on our task, and at what rate did we see them (observed-pattern, on the buyer’s held-out set, not a public benchmark)? Which of those failure modes are reproducible, and which are stochastic? What is the risk tolerance we are accepting each one at? How will we know if a new model version changes any of the above? How does explainability tie to the failure-mode catalogue? This is the join that makes the whole section coherent. Explainability, for an LLM, is largely the documentation of failure. You cannot explain every correct output — there are too many and the mechanism is opaque. But you can enumerate the ways the model fails on your task, characterise when each failure appears, and state your tolerance for it. The failure-mode catalogue is therefore not a separate deliverable that sits next to explainability. It is the explainability evidence, reframed for the committee. Each entry pairs a behaviour (“fabricates a citation when asked for a source it doesn’t have”) with a condition (“triggered by out-of-domain retrieval queries”), an observed frequency scoped to the buyer’s task set, and an accepted tolerance. That structure is legible to a committee in a way a feature-attribution plot never is, and it gives the later challenge a documented answer: this failure was known, characterised, and accepted at this tolerance. Confidence signals feed the same structure — how AI confidence scores in LLM evaluation should be read into a catalogue matters, because a raw model score is not a calibrated probability and treating it as one is its own failure mode. How do you represent explainability for behaviour that is non-reproducible? You state the non-reproducibility as a first-class property, and you scope every claim to it. This is the discipline most packs skip. For each documented behaviour, the pack should record the generation configuration under which it was observed — temperature, top-p, seed handling, decoding strategy — and whether the behaviour is deterministic under that configuration. A useful pattern is to distinguish three tiers: Reproducible — deterministic decoding (temperature 0, fixed seed where the runtime supports it); the same input replays the same output. This is the strongest evidence and the smallest set. Statistically characterised — stochastic decoding, but the failure rate is measured across enough samples to state a bounded estimate on the buyer’s task set (observed-pattern, not a benchmark). Unbounded — behaviour observed but not characterised at volume; explicitly flagged as a residual risk the committee accepts knowingly or requires more evidence on. Writing “unbounded” into the pack is not a weakness. It is exactly the honesty a committee needs to approve in one round rather than deferring for “more detail” — because the detail is present, and the boundary of what is known is drawn explicitly rather than papered over. What changes when the buyer is regulated versus unregulated? The techniques do not change. The evidentiary bar and the retention obligation do. A regulated buyer — under a model-risk framework or a sector supervisor — needs the explainability section to be auditable, versioned, and traceable to a named owner, because it may be examined by a third party who was not in the approval meeting. An unregulated buyer needs the same content to make a good internal decision, but can carry it more lightly. Practically, the regulated pack adds three things: an explicit statement of which explanation techniques were used and why, a reproducibility appendix tied to specific model versions and runtime configurations, and a change-control trigger that forces re-examination on any vendor version bump. The vertical procurement-evaluation methodology — how this gets operationalised against a buyer’s actual infrastructure and prompt distribution — is where the regulated bar becomes concrete rather than aspirational. How does explainability evidence get re-checked when the vendor ships a new version? Treat the explainability section as a versioned artefact with a diff, not a one-time attachment. When the vendor pushes a new model version, the committee’s question is narrow and specific: did any documented failure mode change, appear, or disappear. That is only answerable if the original evidence was structured for comparison — same task set, same generation configuration, same failure-mode taxonomy. Re-run the held-out set, re-characterise each catalogued behaviour, and produce a behavioural diff against the approved baseline. A model that scores higher on a public benchmark can regress on your task; the diff is what catches it. This is the ongoing-review value the ROI depends on: a stable reference for what changed in behaviour between versions, rather than re-litigating trust from scratch each time. FAQ How should you think about ml model explainability in practice? Explainability is a family of techniques that answer different questions for different audiences. Per-prediction techniques like SHAP and LIME explain why a single output happened, which serves a data scientist iterating on a model. Procurement-grade explainability instead documents the model’s behaviour envelope — where it is reliable, where it fails, and at what risk tolerance — because a committee is accepting accountability for a population of decisions, not debugging one prediction. Which explainability techniques actually hold up for an LLM versus a classical model, and where do they break down? For classical models like tree ensembles, feature attribution is defensible because the input-to-output mapping is stable and perturbable. For LLMs, three assumptions weaken: attention weights show where the model looks but are not attributable causes, the free-form prompt space cannot be enumerated so local explanations are conditional on tested prompts, and non-deterministic decoding means one output is a sample, not a behaviour. The honest response is to shift from explaining predictions to documenting the behaviour envelope. What explainability evidence does an approval committee need, versus what a data scientist needs? A data scientist wants per-prediction attribution to debug and improve the model. A committee needs a versioned, defensible document — a named failure-mode catalogue with observed rates, reproducibility status, and accepted risk tolerances — that survives the approval meeting, later version reviews, and a challenge to a specific decision. Shipping the data scientist’s saliency maps to the committee is the category error that triggers “come back with more detail.” How does explainability tie to the failure-mode catalogue in a procurement evidence pack? For an LLM, explainability is largely the documentation of failure, so the failure-mode catalogue is the explainability evidence reframed for a committee. Each entry pairs a named behaviour with its triggering condition, an observed frequency scoped to the buyer’s task set, and an accepted tolerance. That structure is legible to a committee and gives a later challenge a documented answer: the failure was known, characterised, and accepted at a stated tolerance. How do you represent explainability for behaviour that is non-deterministic or non-reproducible? State non-reproducibility as a first-class property and scope every claim to the generation configuration used — temperature, top-p, seed handling, decoding strategy. Sort behaviours into reproducible (deterministic decoding replays the same output), statistically characterised (stochastic but with a measured failure rate on the buyer’s set), and unbounded (observed but not characterised at volume). Writing “unbounded” into the pack is honesty, not weakness — it draws the boundary of what is known explicitly. What changes about explainability evidence when the buyer is regulated versus unregulated? The techniques stay the same; the evidentiary bar and retention obligation rise. A regulated buyer needs the section auditable, versioned, and traceable to a named owner because a third party may examine it, and the pack adds an explicit techniques statement, a reproducibility appendix tied to model versions, and a change-control trigger on any version bump. An unregulated buyer needs the same content to decide well but can carry it more lightly. How does explainability evidence get re-checked when the model vendor pushes a new version? Treat the section as a versioned artefact with a diff, not a one-time attachment. On a new version, re-run the same held-out task set under the same generation configuration and re-characterise each catalogued behaviour, then produce a behavioural diff against the approved baseline. A model that scores higher on a public benchmark can regress on your task, and the diff is what catches it. The uncomfortable part is that a good explainability section documents more absence than presence — more “we cannot yet account for this” than “here is why it works.” Committees that expect the opposite are the ones that keep deferring. The failure class to watch for is the pack that explains predictions to a data scientist and calls it evidence for a committee; the fix is a failure-mode catalogue written at a stated risk tolerance, versioned so the next vendor release can be diffed against it rather than re-argued from zero.
