A vision-language model captions a chest X-ray as “no acute findings” and helpfully highlights the lung fields with a clean attention heatmap. The caption is wrong. The heatmap looks right. That gap — between what a model attended to and what it actually used — is the whole subject of explainability in multimodal systems, and it is where most teams get the concept backwards. The common assumption is that explainability is a rendering problem: attach a saliency map to the vision encoder, pull an attention matrix from the fusion layer, overlay it on the input, and you have an interpretable model. For a single-modality classifier this shortcut is already shaky. For a system that fuses vision and language — captioning, visual question answering (VQA), grounded scene reasoning — it is actively misleading. The heatmap tells you where the model looked. It does not tell you whether looking there changed the answer. What does ML model explainability actually mean in a multimodal system? Explainability, in the sense that matters operationally, is the ability to answer a specific question: given this output, which parts of which inputs were causally responsible for it? In a CV+NLP system there are at least three candidate loci — the vision encoder, the cross-modal fusion, and the language decoder — and the whole reason multimodal explainability is hard is that a wrong answer can originate in any of them while every diagnostic surface still looks healthy. The distinction that carries the weight is between faithful attribution and post-hoc plausibility. Faithful attribution measures what the model actually relied on: if you perturb or ablate the input region an explanation credits, the output should move. Post-hoc plausibility measures what looks convincing to a human reviewer: a heatmap that lands on the object a person would also have looked at. These frequently disagree. A VQA model asked “what colour is the bus?” can answer “yellow” because school buses are yellow in its training text — a language prior — while its attention still politely rests on the bus in the image. The explanation is plausible and unfaithful at the same time. We see this pattern regularly when a model’s accuracy looks fine on a benchmark but collapses on a distribution where the language prior no longer holds. The explainability tooling never flagged it, because plausible-looking attribution is exactly what the tooling was optimised to produce. Why aren’t attention maps and saliency heatmaps faithful explanations? Attention weights are learned routing coefficients, not causal accounts. A transformer’s attention head can assign high weight to a token or patch that has almost no influence on the final logit — the residual stream carries information around the attention bottleneck, and later layers can override early routing. This has been shown repeatedly in the interpretability literature (an observed-pattern across the field rather than a single benchmark): attention is correlated with importance under some conditions and decoupled from it under others, and you cannot tell which regime you are in by looking at the attention alone. Saliency and gradient methods have their own failure. Raw gradients are noisy; smoothed variants (Integrated Gradients, SmoothGrad) are more stable but still answer “what is the model locally sensitive to?” rather than “what did the model use to produce this answer?” For a fused CV+NLP model the question is worse, because a gradient with respect to the image says nothing about how much the language decoder leaned on the text context versus the visual evidence. The practical test for faithfulness is an intervention, not an inspection. If an explanation claims the model used image region R, mask R (or replace it with a counterfactual patch) and check whether the output changes. If it doesn’t, the explanation was decorative. Perturbation-based attribution — occlusion tests, LIME-style local surrogates, and counterfactual editing of one modality at a time — is slower and uglier than an attention overlay, but it is the class of method that can actually be wrong in a detectable way, which is what makes it useful. How do you tell whether a model answered from the image or from language priors? The cleanest diagnostic is a modality ablation: run the same query with the visual input degraded, blanked, or replaced with an unrelated image, and see whether the answer holds. A model answering from the image should degrade sharply; a model answering from language priors will confidently return the same answer to a blank or mismatched picture. This is the multimodal analogue of a placebo control, and it is embarrassingly cheap to run relative to how much it reveals. The tell to watch for is silently confident wrong answers — high-probability outputs that survive the visual input being destroyed. Those are the outputs that ship undetected and later surface as a support ticket or, in a regulated setting, an incident. Instrumenting a modality-ablation pass over a validation set and tracking the fraction of answers that are invariant to visual destruction gives you a running measure of prior-reliance that no attention heatmap can provide. Quick diagnostic: is this a faithful explanation or a plausible one? Run each candidate explanation through these five checks. Any “no” downgrades the explanation to decorative. Check Faithful explanation Decorative explanation Masking the credited region changes the output Yes No Answer degrades when the image is blanked/swapped Yes (image-driven) No (prior-driven) Explanation localises to a specific component Yes (encoder / fusion / decoder) No (whole-model overlay only) Counterfactual edit of one modality moves the result Yes No Method can produce a detectably wrong claim Yes No (always looks reasonable) The bottom row is the one to internalise. A method that always produces a reasonable-looking result cannot be validated, and a diagnostic that cannot be wrong cannot be trusted. Which methods localise a failure to the encoder, fusion, or decoder? Localisation is the payoff, and it changes what you do next. When a caption is wrong, the expensive mistake is retraining the whole stack because you couldn’t tell which part failed. Teams that instrument component-level attribution locate the failure to a specific stage and fix that stage — cutting debugging cycles from weeks to days (an observed-pattern from our multimodal engagements, not a benchmarked figure). A workable decomposition, from cheapest to most involved: Vision-encoder probing. Freeze the encoder and evaluate its features on a proxy task (does a linear probe recover the object the caption got wrong?). If the encoder never represented the relevant object, the failure is upstream and no decoder fix will help. Fusion-layer intervention. Swap or corrupt the cross-attention between modalities and observe output change. If corrupting fusion barely moves an answer that should depend on the image, the model is bypassing fusion — a prior-reliance signature localised to the fusion stage. Decoder-side counterfactuals. Hold the visual features fixed and vary the text prompt/context. If small prompt changes swing the answer while visual evidence stays constant, the decoder is doing the deciding. None of this requires exotic tooling. Feature extraction, ablation hooks, and probing classifiers are standard in PyTorch, and instrumentation frameworks like Captum give you Integrated Gradients and occlusion out of the box; the discipline is in running the interventions rather than trusting the overlay. If you are reading raw performance numbers alongside these diagnostics, our note on reading TensorFlow benchmarks for multimodal CV+NLP models covers the throughput side of the same systems. How do explainability requirements differ across the four NLP–CV patterns? The intersection pattern decides the attribution approach — which is exactly why explainability should be scoped alongside the OCR/captioning/VQA/grounding choice, not bolted on afterward. Each pattern has a different failure geometry. Pattern Dominant failure mode Explainability question that matters OCR / document extraction Character confusion, layout misreads Which pixels produced this token? (region-to-text traceability) Captioning Hallucinated objects from language priors Did the caption’s nouns survive image ablation? VQA Answering from priors, not evidence Does the answer change when the image is swapped? Grounded scene reasoning Correct answer, wrong grounding Do the cited regions causally support the relation claimed? Grounded reasoning is the subtle one: a model can be right for the wrong reasons — correct answer, incoherent grounding — which is invisible to accuracy metrics and only surfaces under faithful attribution. This is the same grounding concern that shows up when a language head narrates a visual scene in generative pipelines; teams building on our generative AI work hit it from the other direction. For an application-side view of grounding retrieval to visual evidence, Vision RAG walks through one concrete pattern. Because faithful attribution surfaces what a model doesn’t know, it sits close to uncertainty estimation. Whether a wrong answer is a fixable data gap or irreducible ambiguity is a distinction we treat separately in aleatoric vs epistemic uncertainty, and the two instruments are complementary in a deployment monitor. What explainability evidence should you require before a regulated deployment? For a multimodal system entering a regulated setting, “we ship attention heatmaps” is not evidence. The evidence that holds up is intervention-based and component-localised. At minimum, require: a modality-ablation report showing the answer distribution under blanked and swapped visual inputs; a faithfulness check demonstrating that credited regions, when masked, move the output; and a component-attribution breakdown for the known failure cases. Regulated document workflows raise this bar further — our take on document intelligence for regulated teams covers the traceability side. Scoping this early is cheaper than retrofitting it. The multimodal CV scoping decision — OCR versus captioning versus VQA versus grounding — determines which attribution method is even applicable, so the explainability requirement and the architecture choice belong in the same conversation. FAQ How should you think about ml model explainability in practice? Explainability answers which parts of which inputs were causally responsible for a given output. In practice that means intervention — masking, ablating, or counterfactually editing an input region and checking whether the output moves — rather than inspecting an attention or saliency overlay. In a multimodal system the operative question is whether an answer came from the image, the text context, or the fusion between them. Why are attention maps and saliency heatmaps not the same as faithful explanations in vision-language models? Attention weights are learned routing coefficients, and the residual stream lets later layers override early routing, so high attention does not imply high causal influence. Saliency and gradient methods report local sensitivity, not what the model used to produce a specific answer. A faithful explanation must survive an intervention test: mask the region it credits and the output should change. How do you tell whether a VQA or captioning model answered from the image or from language priors? Run a modality ablation — blank, degrade, or swap the visual input and re-ask the question. An image-driven answer degrades sharply; a prior-driven answer stays confidently the same against a blank or mismatched picture. Tracking the fraction of answers invariant to visual destruction gives a running measure of prior reliance. Which explainability methods actually localise a failure to the vision encoder, fusion layer, or language decoder? Vision-encoder probing (does a linear probe recover the object the model got wrong?), fusion-layer intervention (corrupt cross-attention and watch the output), and decoder-side counterfactuals (fix visual features, vary the prompt) each isolate one stage. Localising the failure lets teams fix a component instead of retraining the whole stack. How do explainability requirements differ across the four NLP-CV patterns (OCR, captioning, VQA, grounded scene reasoning)? OCR needs pixel-to-token traceability; captioning needs to test whether nouns survive image ablation; VQA needs image-swap invariance checks against language priors; grounded reasoning needs verification that cited regions causally support the claimed relation. Because each pattern has a different failure geometry, the attribution method must be chosen alongside the intersection pattern. What explainability evidence should you require before deploying a multimodal CV system in a regulated setting? Require intervention-based, component-localised evidence: a modality-ablation report under blanked and swapped inputs, a faithfulness check showing credited regions move the output when masked, and a component-attribution breakdown for known failure cases. Attention heatmaps alone are decorative and do not constitute deployment evidence. The question worth carrying out of this is not “can we visualise the model?” but “can our explanation be wrong in a way we would detect?” For a multimodal CV+NLP system, that reduces to a scoping decision made at the start: which intersection pattern — OCR, captioning, VQA, or grounded reasoning — you are building, and therefore which attribution method has to be wired in before the first wrong caption ships silently.
