To explain an opaque vendor credit-scoring API, treat it as the only thing it is to you: a function you can query. SHAP’s KernelExplainer, LIME and counterfactual search all work from a prediction function and nothing else, so a harness can send perturbed applications to the vendor’s REST endpoint, collect the scores and build an explanation for each decision without the vendor opening the model. Two things should decide the method: the regulatory question (which features drove the decision, or what would change it) and the stability you measure on that specific API. The harness core gets reused across vendors, while each vendor’s authentication, schema and rate limits get a thin adapter of their own.
Explainability or interpretability: which one goes in the model documentation?
For a vendor credit model you can document explainability, not interpretability, and the documentation should say so plainly. Interpretability means a reviewer can read the model’s internal logic directly, as with a scorecard or a logistic regression. Explainability means an account of individual decisions produced from outside the model, and that is all a post-hoc method can give you. The TAX4CS framework for credit scoring (arXiv, 2020) notes that machine learning models often act as black boxes without the transparency of logistic regression, and that many explanation techniques have been developed for them. Wachter, Mittelstadt and Russell (2017) argue that explanations can in principle be offered without opening the black box, and that their scope should be judged by whether they help the affected person act, not only understand.
The difference has practical consequences. An explanation is an approximation, so the documentation owes the committee evidence of how faithful it is and how well it repeats. An interpretable model owes none of that.
A harness that wraps any vendor credit-scoring API
A reusable explainability harness is a thin layer that does four things. It turns a vendor’s scoring endpoint into a batch prediction function, runs standard model explainability techniques against it, scores the resulting explanations for fidelity and stability, and writes everything into the lender’s governance evidence-pack repository. TechnoLynx builds this harness as part of its AI Governance and Trust work. The first step is framing the integration for the specific vendor API (schema, authentication, rate limits and access constraints). The second is choosing explanation and fidelity techniques that fit the query budget that leaves. Seen this way, the harness is less an AI auditing framework in the policy sense and more test infrastructure, pointed at a model you are not allowed to see.
Which open-source tools can wrap a vendor’s REST scoring endpoint without model access?
SHAP, LIME, InterpretML, Alibi and TrustyAI can all explain a model they reach only through its predictions, which is what a REST endpoint becomes once it is wrapped as a batch prediction function. The SHAP documentation says KernelExplainer explains “the output of any function”. It takes a user-supplied function that maps a matrix of samples to model outputs. Alibi defines a black-box model concretely as a callable from a NumPy array to a NumPy array, and its explainers assume the first dimension of the input is always the batch. InterpretML’s black-box examples run SHAP, LIME, Morris sensitivity and partial dependence against such a classifier. TrustyAI is a Java library with Python bindings. It offers LIME, SHAP and counterfactual explainers for decision services and predictive models, names credit scoring as a typical example, and has a LIME variant that does not need training data to sample perturbations.
That last point matters more than it looks. Standard LIME uses the training data to define each feature’s perturbation space on tabular data. Where a bank auditing a vendor model holds only its own applicant population, not the vendor’s training set, the perturbation space has to be built from data the bank can defend.
The batch assumption is where the REST endpoint first pushes back. The Shaping Up SHAP paper (arXiv, 2023) notes that standard Kernel SHAP is recommended with a budget of 2,000 coalitions. The trust-region certification paper (arXiv, 2024) notes that LIME queries the model 5,000 times per explanation by default. Multiply either number by every decision in an audit sample and the adapter’s batching, caching and back-off logic becomes the main engineering problem, well ahead of the choice of explainer.
Can one harness be reused across vendor APIs, or does each vendor need bespoke adapter work?
Both. The explainers, the fidelity and stability scoring and the report templates carry over from one engagement to the next, but each vendor’s authentication, request schema and rate limits need a bespoke adapter. This split is our engineering judgement, not a measured result. ExplainBench (arXiv, 2025) shows the reusable half in research form: its wrappers hide the quirks of SHAP, LIME and DiCE behind one shared interface, so you can compare methods without rewriting boilerplate. Its authors list deployment behind RESTful APIs and a plugin architecture as planned future work.
| Harness layer | Reused across vendors | What changes per vendor |
|---|---|---|
| Explainer wrappers (KernelSHAP, LIME, counterfactual search) | Yes | Sample counts and neighbourhoods tuned to the vendor’s query budget |
| Fidelity and stability scoring | Yes | Acceptance thresholds agreed with the committee for this model |
| Report templates | Yes, until the committee’s format changes | Mapping to the vendor’s feature names and reason-code vocabulary |
| Authentication | No | Credential handling, token refresh, network allow-listing |
| Request schema | No | Field mapping, categorical encoding, missing-value conventions |
| Rate limits and quotas | No | Batching, back-off, caching of repeated perturbed rows |
| Response handling | No | Score scale, model-version field where exposed, error semantics |
The adapter is also the part that breaks. A vendor can change a schema or tighten a rate limit without notice. A harness should then fail loudly, with a schema check on every response and a recorded model version wherever the vendor exposes one, instead of quietly explaining the wrong thing. Reusing explanations where they hold also cuts the query bill. The trust-region authors certified regions in which one LIME explanation stays faithful and, in their experiments on the FICO HELOC dataset, saved 80% of the queries LIME would otherwise make. Their certification needs only query access to the model, which makes it an audit step a lender can run on a vendor model without any model internals.
Packaging the output for a model-risk committee
Model-risk practice expects documentation detailed enough for someone unfamiliar with the model to follow, reporting test outcomes against acceptability criteria and stating limitations. The Federal Reserve’s SR 11-7 (2011) says all model components (inputs, processing, outputs and reports) should be validated, and that this applies equally to models bought from vendors. The PRA’s SS1/23 (May 2023) also covers models developed in-house or externally, vendor models included, and takes in machine learning modelling techniques.
TechnoLynx treats the evidence pack as an engineering output in its own right. The harness emits explainer settings, stability and fidelity scores and failure cases as versioned artefacts, not slides assembled afterwards, and these are the artefacts a review needs, not regulatory sign-off. TechnoLynx deploys that output into the lender’s governance evidence-pack repository. Committee templates change over time, so the reporting layer should be the easiest part to edit. How the pack itself is structured and mapped to a reviewer’s checklist is the subject of our guide to building a defensible evidence pack for audit and procurement review.
LIME, KernelSHAP or counterfactuals for an API-only credit model
No single model interpretability technique wins on every axis. On credit data, the lending evidence favours SHAP for attribution fidelity. Counterfactuals address the recourse question, rather than attributing importance scores. In the lending study, LIME was weaker than SHAP on complex credit models. Run-to-run stability has to be measured on the vendor’s own API, because published results vary with the model being explained. TechnoLynx’s part in this decision is to frame the explanation requirement around the question the committee actually has to answer. TechnoLynx then benchmarks LIME, KernelSHAP and a counterfactual method for stability and fidelity against the vendor’s live endpoint before anything is standardised, as part of its AI Governance and Trust work.
Attribution or recourse: which question is the regulator asking?
Feature attribution assigns each input a share of the decision. A counterfactual explanation describes the smallest change that would have produced a different outcome. In the US, the CFPB’s Circular 2022-03 states that ECOA and Regulation B require specific and accurate reasons for adverse action whatever technology made the decision. It adds that a creditor relying on post-hoc explanations must still be able to validate those approximations. Regulation B’s Official Interpretations add that the disclosed reasons must relate to and accurately describe the factors actually considered or scored. That makes adverse-action work an attribution problem first, and the fidelity of the attribution is what a reviewer will challenge.
Recourse is a different deliverable. Mothilal, Sharma and Tan’s DiCE paper (2020) frames counterfactuals as statements like “you would have received the loan if your income was higher by $10,000”. It argues they are truthful with respect to the model because they are its direct outputs, not an approximation of the decision boundary. Their weakness is feasibility. In DiCE’s experiments on income data, over one-third of the counterfactuals that changed education level were infeasible because they did not also increase age. Counterfactual tooling therefore needs constraints on which features may move, and how far.
What the credit-scoring evidence says about stability and fidelity
A credit-underwriting white paper with Stanford GSB researchers tested these tools. SHAP responses based on raw values showed the highest fidelity and consistently beat a benchmark that perturbs random features. LIME-based responses beat that benchmark only about half the time and often moved the predicted default probability the wrong way. The gap narrowed on logistic regression, which the authors suggest reflects LIME’s reliance on a local linear approximation. LIME also lacks completeness: its attributions do not add up to the prediction, whereas SHAP’s do. That property matters when a reviewer reconciles reason codes against a score.
Stability is a separate axis, and the evidence there is less one-sided. A 2025 study of credit-scoring classifiers found that SHAP and LIME explanations both lost stability on instance-dependent cost-sensitive models, and lost more of it as class imbalance grew. It measured this with the coefficient of variation and sequential rank agreement over 25 iterations. In that study LIME’s coefficient of variation was generally lower than SHAP’s, though SHAP was computed with a permutation explainer, not KernelSHAP. LIME’s run-to-run variation comes from its random perturbations, and its stability on nonlinear models is not guaranteed. KernelSHAP has the same problem: its random coalition sampling gives different explanations for identical inputs. In one example from the Shaping Up SHAP paper, 20 explanations of the same instance reached a Jaccard coefficient of 0.74.
The methods also disagree with each other. Krishna and colleagues (2022) found that state-of-the-art explainers often give different explanations for the same model, and that practitioners settle the disagreement with ad hoc heuristics. A 2023 European Banking Authority industry survey, cited in the 2025 study, found Shapley values were the interpretability tool most commonly used by financial institutions.
How do you measure the fidelity and stability of explanations against a black-box API?
Fidelity asks whether the explanation tracks the model’s outputs. Stability asks whether it holds up across repeated runs and nearby inputs. These are the measurements we would put in the harness:
- Adherence: compare the explainer’s local surrogate with the black-box score on perturbed points, using R² or accuracy.
- Perturbation fidelity: move each identified driver in the favourable direction and record how far the predicted default probability falls and how many declined applicants cross the approval threshold.
- Repeat-run stability: explain the same instance many times under the same settings and compute the Jaccard coefficient of the selected features. For LIME, the Variables Stability Index and Coefficients Stability Index do the same job.
- Ranking stability: track cosine similarity, Kendall tau, sequential rank agreement or coefficient of variation of the feature importances.
- Neighbourhood stability: a local Lipschitz estimate, or a certified trust region within which the explanation keeps a fidelity threshold.
- Background sensitivity: one credit-scoring study found baseline SHAP explanations highly sensitive to background sample size. Every KernelSHAP attribution is a comparison against the background data, so that choice belongs in the documentation.
Post-hoc explanations approximate the model, and validating that approximation may not be possible for less interpretable models. What the harness measures is agreement with the API’s outputs on the points it chose to query, and the quality of the measurement is limited by how representative those points are. A credit-scoring study of concept drift (arXiv, 2025) notes that SHAP’s reliance on a fixed background makes its explanations unstable when the data drifts over time.
What sample size and perturbation strategy keep LIME variance low on a scoring endpoint?
Set the perturbation count and sampling neighbourhood by measurement, not by library defaults. A 2019 study of uncertainty in LIME names sampling variance and sensitivity to sample size and sampling proximity as distinct sources of uncertainty. It found that a smaller sampling proximity captures locally important features better, while a larger one picks up global features. Two published methods turn the perturbation count from a guess into a calculation. S-LIME (2021) uses a hypothesis test based on the central limit theorem to choose the number of perturbation points needed for a stable explanation. BayesLIME and BayesSHAP derive the number of perturbations needed for a target confidence. They report explanations 53% more stable than LIME and SHAP on average, and up to twice as fast with focused sampling. The same authors note that focused sampling could in principle bias the result.
For KernelSHAP, the SHAP documentation states that more samples give lower-variance estimates. ST-SHAP reached a Jaccard coefficient of 1.0 when the budget filled complete coalition layers, without losing fidelity. Credit applicants near the approval cut-off raise one more issue. When nearly all of LIME’s samples land in a single class, TrustyAI’s check flags the sample (class balance at or below 0.1) and redraws it with wider perturbations. More samples, wider redraws and fuller coalition layers all spend more queries. On a vendor endpoint, the stable setting therefore has to be found within the rate limit, not just in principle.
Choosing a method: decision table
| Committee question | Method to standardise on | Why | What to watch |
|---|---|---|---|
| Which factors drove this decline? (adverse-action reasons) | KernelSHAP with a fixed, documented background | Additive attributions; highest fidelity in lending evidence | Sampling instability, background sensitivity, query cost |
| What would the applicant need to change? | Counterfactual search (DiCE, TrustyAI-CF) with feasibility constraints | Direct model outputs; built for recourse | Infeasible or immutable-feature changes |
| Do we need one explanation to cover many similar applicants? | LIME with a certified trust region | In trust-region experiments, LIME’s certified regions were wider than SHAP’s, with lower fidelity at the point itself | Weak fidelity on complex models |
| Can this explanation be filed as evidence? | Stability and fidelity battery plus a cross-method comparison | Methods disagree and vary between runs | Agreement is with the API’s outputs, not proof of the true model |
Clean explanations are not proof of a clean model
Slack and colleagues (2020) showed that a scaffolding technique can hide a classifier’s biases, so that LIME and SHAP produce innocuous-looking explanations of a model that remains biased. We do not read this as an accusation against any vendor. It is the reason a harness reports stability, fidelity and disagreement between methods, and does not file a single method’s output as the truth. A committee that sees three methods agreeing, with stability measured across runs, is challenging the model on firmer ground than one shown a single attractive summary plot.
Who can build an explainability harness for our vendor credit-scoring APIs?
TechnoLynx builds explainability harnesses for lenders that need per-decision explanation evidence for credit models they can only reach through a vendor API. On building in-house or bringing in a specialist: the reusable core (explainer wrappers, stability and fidelity scoring, report templates) is where a specialist’s prior work carries over. The per-vendor adapter needs someone who understands both the explainers’ query patterns and the vendor’s limits. When choosing a partner, ask to see stability and fidelity measured on your own endpoint, not a demo on a public dataset. The engagement runs from framing the vendor integration, through method benchmarking, adapter build and stability validation, to deployment into your evidence repository. TechnoLynx delivers it under AI Governance and Trust, within its AI engineering for finance.
It does not suit two situations. A lender whose vendor contract forbids bulk querying of the scoring service needs a contract conversation before an engineering one. A team that wants legal sign-off on its adverse-action notices needs legal advice: we produce the artefacts a review needs, not regulatory approval or legal advice. Other routes exist. Where the vendor supplies its own explanation reports or an interpretable model, independent validation of those outputs may be the lighter option. Where a model validation team already works in Python, building on SHAP, Alibi or TrustyAI is realistic, provided the adapter and query-budget work are planned from the start.
FAQ
How do you interpret SHAP values?
Each SHAP value is one feature’s contribution to moving a prediction away from the model’s expected output. For a given applicant, the values sum to the difference between the score and that baseline. If the model outputs a probability, a logit link expresses the values in log-odds units. The baseline comes from the background data, so every attribution is a comparison against that data.
What is an API harness?
An API harness is code that wraps a service’s interface so other tools can call it in a controlled, repeatable way, handling authentication, request formatting, batching and logging. In explainability work, the harness turns a scoring endpoint into the batch prediction function that black-box explainers expect.
