Off-Policy Evaluation: Judging a New Bandit Policy from Logged Data

Yes, logged data from contextual bandits can tell you whether a new targeting or tracking policy would beat the one in production, but only under two conditions: the logs record the probability with which each action was chosen (the propensity), and the old and new policies choose overlapping actions often enough. Off-policy evaluation (OPE) is the estimation of a target policy’s value using data collected under a different policy. Inverse propensity scoring (IPS) and doubly robust (DR) estimators do this with well-understood statistical properties, and both are minimax optimal up to constants in the worst case, when no consistent reward model is available. When either condition fails, standard estimators can fail catastrophically and their confidence intervals become unreliable.

Why a bandit log cannot simply be replayed against a new policy

Contextual bandits have a partial information problem: you only ever see the reward for the action the production policy actually took, never for the actions it skipped. Logged entries are also biased, because actions the system favours are over-represented in the logs. The new policy may want to take exactly the actions the old one rarely took, so you cannot compute its value directly from the log; you need a statistical estimator.

The simplest shortcut is the direct method: train a regression model of reward given context and action, then score the new policy with it. That estimate is unbiased only if the reward model is, and in practice the model has seen far more rewards for good actions than bad ones. Vowpal Wabbit’s tooling rejects the direct method for evaluation, returning the error that it “can not be used for evaluation — it is biased”.

The randomness in the logged policy is what makes counterfactual reasoning possible at all. A deterministic production policy that always picks the same action for a given context leaves nothing to re-weight.

What does our bandit need to log so a new policy can be evaluated later?

Each decision should be logged as a tuple of context, action, reward and propensity, where the propensity is the probability the exploration policy assigned to the chosen action given that context. Vowpal Wabbit’s input format encodes exactly this as an action:cost:probability triplet attached to the context features, and the same triplet is used when evaluating a policy rather than training one. Exploration has to cover every action you may later want to evaluate: if the exploration rate is zero, arbitrary policies cannot be evaluated correctly.

Recording the tuple goes wrong in practice in many ways. Agarwal et al.’s paper on the Microsoft Decision Service names the failure modes directly: context, action and probability must be recorded at the moment of decision and the reward matched to that record afterwards. Features stored as references to database rows that a separate process updates will hold different values at learning time than at decision time. When downstream business logic such as editorial locking overrides the chosen action and the override is what gets logged, the recorded probability no longer belongs to the recorded action. A third error is feeding the identity or probability of the chosen action back in as a feature, which only exists after the decision is made. In the paper’s simulations, these logging faults made training-time performance estimates deviate from test-time performance by a factor of 1.2x to 8.7x.

Reward timing matters too. If rewards for some actions arrive faster than others, the join between decision and outcome becomes biased; the Decision Service avoids this by waiting a fixed duration before joining and assigning a default reward when nothing arrives. Shifting the clicks of the two most popular articles earlier in their MSN data produced training performance 1.3x higher than test performance.

Logging checklist for offline evaluation

Context features as values at decision time, not as references to mutable records.
The action the policy chose, before any downstream override.
The propensity of that chosen action under the logging policy.
The reward, joined after a fixed waiting window with a default for no response.
The identifier of the policy version that made the decision, plus the randomisation seed, so the decision can be reproduced.
Any sub-sampling applied to the log (for example, dropped non-click rows), so the evaluator can correct for it.
A non-zero exploration probability over every action a future policy might choose.

IPS or doubly robust: which estimator should we trust?

IPS re-weights each logged reward by the ratio of the target policy’s probability for the logged action to the logging policy’s probability. IPS is unbiased and assumes nothing about how rewards depend on context and action. Its weakness is variance: the classic IPS estimator has unbounded variance, because a propensity close to zero can push the estimate arbitrarily far from the truth. That variance grows with how different the target policy is from the logging policy.

Doubly robust estimation combines a reward model with the importance weights. Dudík, Langford and Li showed that DR yields accurate value estimates when either the reward model or the model of the past policy is good, even if neither is consistent. Wang et al. go further: with the same worst-case rates as IPS, DR will do better on finite samples whenever the reward model is even moderately good, meaning better than predicting zero. Uncapped, DR remains unbiased, and its variance is lower than IPS when the predicted reward correlates well with the actual reward.

DR does not remove the problem of large importance weights. Under large reward noise and large importance weights, DR can still have high variance even with a perfect reward model. Remedies include truncating large weights, which reduces variance at the cost of a little bias, self-normalised estimation, or the SWITCH estimator, which applies the reward model to actions with large weights and IPS or DR elsewhere. Gilotte et al. (Criteo) found that in practice no capping parameter for capped importance sampling gave a confidence interval narrow enough to decide between policies.

Estimator decision table

Situation	Estimator to start with	What to watch
No credible reward model, high-dimensional contexts	IPS	Variance as target diverges from logging policy
A reward model better than a constant is available	Doubly robust	Large weights under heavy reward noise
A few extreme importance weights dominate	SWITCH or capped weights	Bias introduced by capping
Reward model only, no propensities	Direct method is not an evaluation; treat as a hypothesis	Unbounded bias from the reward model

The Criteo test-bed recommends reporting both the IPS and the self-normalised estimate for a new policy, alongside an approximate confidence interval, rather than one point estimate. Disagreement between the two estimates is a warning not to rely on either point estimate.

Can a bandit log reconstruct a counterfactual A/B comparison after the fact?

Partly, and for more candidate policies than online A/B testing could cover. Counterfactual estimators let the same logged data evaluate many new policies without deploying any of them, which Swaminathan and colleagues describe as greatly improving scalability and timeliness compared with A/B tests. Vlassis et al. make the same point for recommendation systems: off-policy evaluation tests far more candidate policies than online A/B testing could. The Decision Service team reported running parallel offline experiments on exploration data that found a better exploration algorithm, learning rate and feature set without any additional live experiment.

The reconstruction estimates the expected outcome had the candidate been deployed, not the live result itself. In Amazon’s Alexa skill-routing work, OPE estimates differed from A/B measurements in absolute value but correlated with them at an r-value of 0.89. That makes OPE good at ranking a candidate against production and weaker at predicting the exact lift. Gilotte et al. also point out the asymmetry in errors: a false positive costs the length of an A/B test, while a false negative is a real improvement that is never tested. On their own recommendation logs, their capped importance sampling had a false negative rate of 0.64, which dropped to 0.33 with normalised capping.

What reporting you lose by giving up fixed random assignment

A fixed A/B split gives every unit the same known assignment probability for the length of the test, so arms can be compared with a simple difference in means and a standard confidence interval. An adaptive bandit logs a propensity for each chosen action, and comparisons are re-weighted by those logged propensities. In the Criteo test-bed, IPS variance scales roughly with the range of inverse propensities, and intervals widen as candidate policies differ from the logging policy.

OPE recovers the counterfactual comparison under support assumptions, and the same logged data serves to evaluate many candidates. What it does not recover is the live effect of the policy once participants react to it. In the reserve-price ad marketplace studied by a 2026 arXiv framework, the replay value, the off-policy value and the live launch effect are three different quantities, and treating the first two as the third is the core methodological risk. Off-policy estimates are also predictive only while the environment stays close to stationary. Questions such as how stable a policy stays through a regime shift, or which validation methodology a model risk function will accept, need their own evidence beyond a single OPE estimate.

When does off-policy evaluation break down, so a live test is still needed?

IPS formally requires full support: the logging policy must have given non-zero probability to every action in every context the target policy might choose. Sachdeva, Su and Joachims (KDD 2020) show that many real systems produce support-deficient data, particularly with large action spaces, and that standard methods can fail catastrophically on it. Their remedies are to restrict the action space, extrapolate rewards, or restrict the policy space. If the target policy puts weight on an action the logging policy never took, an unbiased estimate requires augmentation data covering that action.

Policy divergence is the second failure. When several historic logging policies are mixed naively, IPS degrades so badly as logging and target policies diverge that throwing away data improves the estimator’s variance. In the Criteo test-bed, confidence intervals widened as candidate policies moved away from the logging policy, and in the highest-variance settings the computed intervals did not contain the true value at all. Importance sampling variance also grows with the number of slots: the same test-bed notes that slot-filling policies would need tens of millions of impressions for a precise click-through estimate.

Effective sample size (ESS) is an overlap diagnostic: the square of the summed importance weights divided by the sum of squared weights. An ESS of 500 from 10,000 logged rows means 95% of the log tells you nothing about the target policy. For offline reinforcement learning, a survey of PAC guarantees suggests starting defaults of an ESS of at least 200 and a 99th-percentile weight of at most 30, explicitly as adjustable starting points that should tighten as the stakes rise. The same survey recommends interval estimates over point estimates and abstaining from deployment when coverage is poor.

Three further conditions call for a live test whatever the diagnostics say:

Propensities were never logged, or were logged after an override, so importance weighting has nothing valid to work with.
Unobserved factors drive both the action taken and the outcome, which biases standard OPE estimators.
The policy changes the behaviour of the system it acts on, such as bidders adapting to new reserve prices, which logged data cannot observe.

How off-policy evaluation fits a release pipeline as a gate

The release-gate question is not whether a candidate improves an offline metric but whether the evidence justifies launch, online validation, holding or redesign. A 2026 arXiv framework for marketplace policies puts the distinction plainly: strong offline evidence justifies validation, not deployment. In its case study, offline evidence was strong enough to earn validation capacity but could not remove the blockers around real propensities, participant response and interference. Its authors treat OPE run on simulated propensities as validation evidence, not as a substitute for an online experiment with logged assignment probabilities.

Within those limits, OPE is a strong pre-release check. We treat it as one of the reliability artefacts that catch a policy regression before customers do, alongside regression suites and drift monitoring, and distinct from the accuracy of the model behind the policy. In Amazon’s Alexa skill routing, each new policy is evaluated before and after deployment, with guardrail metrics deciding whether it ships. One model refresh showed a regression that OPE had predicted, and it shipped because the regression stayed within the pre-deployment guardrail tolerance. That team still tracks post-deployment performance, because weight clipping biases OPE and coverage gaps inflate its variance.

Release gate procedure for a new bandit policy

Check the log: propensities present, recorded at decision time, no overrides, rewards joined on a fixed window.
Run support diagnostics: action coverage, ESS, and how sensitive the estimate is to the weight cap.
Report both the IPS and the self-normalised estimate, with an approximate confidence interval.
Rank on a conservative lower bound of the lift rather than the point estimate, and require every guardrail metric to stay within its tolerated deterioration.
Deploy only if the interval is narrow and sits above the production policy’s value; if it is wide or overlaps, collect more data or keep the current policy.
Where participants can react to the policy, as bidders react to reserve prices, follow the marketplace framework’s sequence: shadow logging, a controlled online test, then a ramped launch.
After release, keep counterfactual monitoring running so the policy can be rolled back if it falls behind how the previous best policy would have performed.

If we move to a bandit, can we still tell finance why a given customer got a given offer?

Yes, if the bandit logs enough for each decision to be reproducible. A bandit’s choice for one customer is a random draw from the policy’s distribution over offers, so reproducing it needs both the policy version and the randomisation seed.

TechnoLynx’s AI Governance and Trust work builds approval evidence packs for auditors and procurement committees.

The Microsoft Decision Service shows the mechanism: randomisation is confined to one layer that samples from the policy’s distribution, each interaction seeds its own pseudorandom generator with a key and an application ID, and the client records the ID of the policy used for each decision. Together these make every decision reproducible offline, and its authors credit that reproducibility with diagnosing numerous failures, often by finding the point where the offline run deviated from the online one. With the logged tuple of context, action, reward and propensity alongside the policy ID, a reviewer can reproduce what the customer was offered, what the alternatives were and with what probability, and which model version decided.

That makes the decision reproducible at the level of the system. It does not by itself explain which customer features drove the policy’s scores; that needs explanation methods on the model and documented decision factors. Public buyers ask for this: Georgia’s procurement guidelines for AI tools (GS-25-002) tell buyers to define auditability requirements, including access to decision logs and justifications. The documentation an auditor expects for a bandit in a regulated workflow is covered in depth in our guide to contextual bandit audit evidence.

What evidence package defends an AI model shortlist under a procurement audit?

A defensible shortlist rests on a documented process with a trail of evidence for each decision, because public procurement auditors are required to obtain sufficient and appropriate evidence, mostly through document review and interviews. INTOSAI’s guidance on auditing public procurement also notes that large technical procurements may need technical experts on the audit team, so the technical rationale has to be legible to them.

Two public bodies give concrete templates. The District of Columbia’s AI Procurement Handbook requires vendors to complete an AI Factsheet capturing basic facts about the system. The handbook also requires documenting how the tool’s performance will be measured, so the vendor can be held to it. Its model contract language gives the District the right to audit relevant work processes and documents, with the contractor providing access at no additional cost. Georgia’s GS-25-002 guidelines ask vendors to disclose model functionality, decision processes and limitations, including training data, learning methods and how decision thresholds were set. The same guidelines call for measurable benchmarks and KPIs, audit rights and remediation procedures, plus an algorithmic impact assessment run in a context as close as possible to the intended use.

For the internal side, Raji et al.’s framework for internal algorithmic auditing produces a set of documents at each stage that together form an audit report. The framework’s minimum is a “transparency trail” of documentation, and its documentation stage collects records of data and model dynamics alongside design documents and reviews.

Evidence item	What it answers	Source of the requirement
AI Factsheet per shortlisted model	What the system is and what risks it carries	DC AI Procurement Handbook
Performance measurement plan and benchmark results	How each candidate was compared and on what tasks	DC handbook; Georgia GS-25-002
Vendor disclosure of data, methods and decision thresholds	Why the model behaves as it does	Georgia GS-25-002
Algorithmic impact assessment in the intended context	Bias, fairness, transparency and accountability risks	Georgia GS-25-002
Decision logs and audit rights in the contract	Whether the decision can be checked later	DC handbook; Georgia GS-25-002
Stage-by-stage audit documents	That the selection process itself was followed	Raji et al. internal audit framework

How to assemble and maintain this pack over time is the subject of our piece on building a defensible evidence pack for audit and procurement review.

Who can build the evaluation gate and audit trail around our bandit or model shortlist?

TechnoLynx produces a documented comparison for a model shortlist through its LLM Selection Pack. Whether to bring in a specialist mostly turns on whether your team already owns propensity logging and versioned MLOps pipelines; if it does, the remaining work is estimator choice and gating, which a strong in-house team can take on.

The engagement does not suit everyone. A team still at proof-of-concept stage, with no production policy generating logs, gets more from a well-designed randomised test than from offline evaluation machinery. Where the open question is how bidders or customers will react to a policy, no offline analysis from us or anyone else settles it: that takes a staged online validation, and the most useful thing OPE can do is decide which candidate earns that test.