Explainability asks a deceptively simple question: can you account for why a model produced a given output? In production, the honest answer has a price tag — every explained request carries extra compute, and that cost lands on your cost-per-request whether you budgeted for it or not. That framing is unusual. Most teams meet explainability as a compliance checkbox or a debugging convenience, then reach for a post-hoc library — SHAP, LIME, integrated gradients — and bolt it onto a model that already shipped. The attributions look free because the library is free to install. They are not free to run. The moment explanations become mandatory rather than optional — a regulated credit decision, a customer dispute, an audit request — the team that treated explainability as an afterthought discovers each explained request can cost several times a plain inference. Nobody sized that. This article grounds explainability so you can decide where it earns its place on the serving path instead of applying it everywhere by default. What does machine learning model explainability actually mean in production? Strip away the marketing and explainability is the ability to attribute a model’s output to its inputs or its internal reasoning in a way a human can act on. That covers two structurally different things, and conflating them is the first mistake. The first is intrinsic interpretability: the model itself is legible. A logistic regression, a shallow decision tree, a linear model with a handful of features — you read the coefficients or trace the path and you already know why. No second computation is required; the explanation is the model. The second is post-hoc explanation: the model is a black box (a deep network, a gradient-boosted ensemble, a transformer) and you run a separate procedure after inference to approximate why it decided what it decided. SHAP samples feature perturbations, LIME fits a local surrogate, integrated gradients accumulates gradients along a path from a baseline input. Each of these is its own workload with its own compute footprint. The distinction matters because the naive reading treats “explainability” as a single property a model either has or lacks. In practice it is a spectrum of methods with wildly different costs, and the cost is what the serving path feels. Intrinsically interpretable models vs post-hoc explanation methods — when does each fit? The trade-off is not academic. It decides whether explanation is a fixed property of the model or a recurring per-request expense. Dimension Intrinsically interpretable Post-hoc explanation Where the explanation lives In the model’s structure In a separate computation run after inference Marginal cost per explained request ~Zero (read the coefficients) Meaningful — often a multiple of the base inference Fidelity Exact for the model’s own logic Approximate; a surrogate of the real decision Typical model class Linear models, shallow trees, GAMs Deep nets, GBMs, transformers Constraint it imposes Caps model capacity Adds latency and compute at serve time The honest guidance: if the decision the model supports is high-stakes and the accuracy gap between an interpretable model and a black box is small, choose the interpretable model and pay nothing at serve time. The industry-wide push toward ever-larger models has made this option feel quaint, but for tabular, structured problems it is frequently the right call — a well-regularised gradient-boosted model with SHAP is not automatically better than a monotonic GAM you can read directly (observed pattern across the tabular workloads we have reviewed; not a benchmarked accuracy claim). When capacity genuinely requires a black box, you are in post-hoc territory, and now the cost question is unavoidable. How much does adding explanations cost per inference request? This is the number teams skip, and it is the number that decides the feature’s economics. A post-hoc method is not a free annotation on top of a prediction. It is extra forward passes, extra sampling, or extra gradient work. Consider what each method actually does: Integrated gradients typically accumulates gradients over a series of interpolation steps — on the order of tens of forward/backward passes per explained input. If your base inference is one forward pass, an explanation with 50 steps is roughly 50× the base compute for that request (illustrative arithmetic from the method’s definition, not a measured latency figure). KernelSHAP samples many perturbed coalitions of features and runs the model on each. The sample count is a tunable, but “enough to be stable” often means hundreds of model evaluations per explained instance. LIME fits a local surrogate from perturbed samples — again, many model calls to build one explanation. None of these numbers is a benchmark; they follow directly from what the algorithms compute. But they tell you the shape of the problem: explanation cost is a multiple, not an increment. A request that costs x to serve plainly can cost several x — sometimes one or two orders of magnitude more — once you attach a faithful post-hoc explanation. That multiple is exactly the figure a cost-per-request analysis has to carry. If you want it measured rather than estimated, profiling the serving path turns the “several x” into a real number on your hardware — the incremental milliseconds and the incremental compute the explanation method adds to the request. The [inference-cost-cut engagement](Inference Cost-Cut Pack) sizes exactly this kind of per-request cost driver against a deployed serving path. Which request classes genuinely need an explanation? Here is the decision that most teams never make explicitly: they either explain nothing or explain everything. Both are wrong for the same reason — they refuse to segment traffic by whether an explanation will ever be consumed. Most production inference is never questioned. A recommendation surfaces, a routing decision fires, a low-value classification resolves, and no human ever asks why. Attaching an explanation to those requests pays compute for an artifact nobody reads. The fraction of traffic that genuinely needs an explanation — the disputed decision, the flagged transaction, the audited case, the borderline score a reviewer will actually open — is usually a small slice of total volume. A quick diagnostic to size your own split: Explanation-necessity checklist Is this request class subject to a regulatory right-to-explanation (adverse credit action, certain hiring or insurance decisions)? → Needs explanation. Can a customer or user formally dispute this output? → Needs explanation, at least on demand. Will an internal reviewer open this case and require a rationale? → Needs explanation. Is the output low-stakes, high-volume, and never individually reviewed? → Does not need a per-request explanation. Are you generating explanations “just in case” with no consumer identified? → You are paying for explanations nobody uses. If you tag every request class against that list, you almost always find that a minority of traffic drives the entire explanation cost, and a large share of your current explanation spend (if you explain by default) is dead weight. That is the wasted-margin finding, and it is the on-ramp to the cost decision. How do you match explanation fidelity to the decision it supports? Not every explanation needs to be faithful to the model’s exact reasoning. The right fidelity is a function of what the explanation is for. If an explanation exists to satisfy a regulator or defend a decision in a dispute, it must be faithful and defensible — a cheap, unstable surrogate that gives different attributions on re-run is worse than useless, because it manufactures a liability. If an explanation exists to help an internal engineer sanity-check a model during triage, a fast approximate attribution is often plenty; the engineer applies judgment on top of it. Over-engineering shows up as running a heavyweight, high-sample SHAP explanation for a low-stakes internal debugging view — paying for regulatory-grade fidelity on a request where an approximate feature ranking would have done. Under-engineering is the reverse: leaning on a cheap surrogate for a decision that will be legally challenged. Matching the method to the decision is where the compute budget gets spent well. How does explainability relate to a production AI feature’s contribution margin? Cost-per-request is not an abstraction; it is the denominator of the feature’s unit economics. If a feature earns a fixed amount per request and explanation multiplies the cost of the requests it touches, then explainability is directly eroding contribution margin on that slice of traffic. The relationship sharpens as usage grows. At low volume, explanation cost is a rounding error you can absorb. At scale, if you explain a class of traffic that does not need it, you are scaling a cost multiplier linearly with usage — the feature can get less profitable per request as it grows, which is the opposite of the leverage a software feature is supposed to have. Understanding how compute cost maps to a feature’s unit economics is what turns “explainability is expensive” into a number the P&L can act on. The AI infrastructure practice treats this as a core serving-path economics question, not a compliance side-quest. The reframe: explainability is a per-request cost driver, and like every cost driver it should be applied where it earns its keep and withheld where it does not. Inline explanations vs on-demand generation — how do you decide? This is the architectural decision that follows from everything above, and it is where the biggest savings usually hide. Serving explanations inline means every request in the covered class produces its explanation as it is served, whether or not anyone will ever look at it. Simple, but you pay the multiplier on the full class. Generating explanations on demand means the base inference serves cheaply, and the expensive explanation is computed only when a decision is actually disputed, flagged, or audited. This works when you can reconstruct the explanation later — which requires that you log enough context (the input, the model version, the seed or config) to reproduce the exact decision when someone asks. Approach Pay the explanation cost on Best when Inline Every request in the covered class Explanations are consumed on nearly every request; reconstruction is impossible or the model is non-deterministic and unlogged On demand Only disputed / flagged / audited requests The consumed fraction is small; you can log enough to reproduce the decision exactly later For most regulated or dispute-driven use cases, the consumed fraction is small and the decision is reproducible — which points to on-demand. The catch is disciplined logging: if you cannot faithfully reconstruct the input and model state, the on-demand explanation will not match the original decision, and a mismatched explanation in an audit is a problem, not a solution. Keeping model versions and run conditions tracked is what makes on-demand explanation defensible. FAQ What does working with machine learning model explainability involve in practice? Explainability is the ability to attribute a model’s output to its inputs or internal reasoning in a way a human can act on. In practice it comes in two forms: intrinsic interpretability, where the model’s structure is itself legible, and post-hoc explanation, where a separate procedure runs after inference to approximate why a black-box model decided what it did. The practical consequence is that post-hoc explanation is its own workload with its own compute cost. What is the difference between intrinsically interpretable models and post-hoc explanation methods, and when does each fit? An intrinsically interpretable model (a linear model, shallow tree, or GAM) carries its explanation in its structure at near-zero marginal cost, but caps model capacity. Post-hoc methods (SHAP, LIME, integrated gradients) explain a black box after the fact, giving you full model capacity but adding meaningful per-request compute. Choose the interpretable model when the accuracy gap to a black box is small and stakes are high; accept post-hoc cost only when capacity genuinely requires a black box. How much does adding explanations cost per inference request, and how does that flow into cost-per-request? Post-hoc explanation cost is a multiple, not an increment — integrated gradients runs tens of passes per input, KernelSHAP and LIME run hundreds of model evaluations. A request costing x to serve plainly can cost several x, sometimes far more, once a faithful explanation is attached. That multiple lands directly on the cost-per-request of the explained traffic, which is why it must be measured on the actual serving path rather than assumed to be free. Which request classes genuinely need an explanation, and which are paying for one they never use? Requests subject to a right-to-explanation, that can be formally disputed, or that a reviewer will actually open genuinely need explanations. Low-stakes, high-volume, never-individually-reviewed traffic does not. Teams that explain everything by default typically find a minority of traffic drives the entire explanation cost, and a large share of their explanation spend produces artifacts nobody reads. How do you match explanation fidelity to the decision the explanation supports without over-engineering? Match fidelity to consumption: regulatory or dispute-facing explanations must be faithful and reproducible, while internal debugging views can use fast approximate attributions. Over-engineering means running heavyweight, high-sample explanations for low-stakes internal use; under-engineering means leaning on a cheap surrogate for a legally challengeable decision. The method should track the stakes of the decision it supports. How does explainability relate to the contribution margin of a production AI feature as usage grows? Explanation cost lands in the denominator of a feature’s unit economics. If you explain traffic that does not need it, you scale a cost multiplier linearly with usage, so the feature can become less profitable per request as it grows — the opposite of software leverage. Applying explanation only where it earns its keep protects contribution margin at scale. How do you decide between serving explanations inline versus generating them on demand only when a decision is disputed? Inline serving pays the explanation multiplier on every covered request; on-demand generation serves the base inference cheaply and computes the explanation only when a decision is disputed, flagged, or audited. On-demand wins when the consumed fraction is small and you can reproduce the decision exactly, which requires logging the input, model version, and run configuration. Without disciplined logging, an on-demand explanation may not match the original decision — a serious problem in an audit. Where explainability earns its place Explainability is not a property you switch on for a model; it is a cost you place on specific request classes because a specific decision needs defending. Get that placement wrong and you either scale a cost multiplier across traffic nobody questions, or you find yourself unable to explain the one decision that ends up in front of an auditor. The engineering question is not “is our model explainable” but “for which requests, at what fidelity, served how — and what does that do to cost-per-request as we grow.” That is a serving-path economics question, and it belongs in the same conversation as every other per-request cost driver on the path.
