A contextual bandit is a learning loop that repeatedly observes a context, chooses one of a fixed set of actions, and sees the reward for that chosen action only. It then updates its policy and goes again, trying to maximise cumulative reward without wasting too much traffic on poor choices. Because it sees only the chosen action’s reward, the algorithm has to spend some decisions on seemingly worse actions to learn about them. Epsilon-greedy, upper confidence bound methods such as LinUCB, and Thompson sampling are the main techniques for that exploration. A contextual bandit suits frequent decisions where context predicts the best action, rewards arrive quickly enough to learn from, and reward during learning matters most.
How a contextual bandit makes one decision
Take a news portal choosing which article to show each visitor. The context is what the portal knows about the user, the actions are the candidate articles, and the reward is whether the user clicked. The aim is a policy, a rule mapping context to action, that maximises long-term cumulative reward.
Using that context is what separates this from the plain multi-armed bandit. A multi-armed bandit estimates one expected reward per arm and looks for the arm that is best on average. A contextual bandit asks which action is best for this context, right now, which is why it suits recommendations, ranking and messaging where different users want different things. In the Burtini et al. survey of 2015, contextual bandits are also judged more widely applicable than the non-contextual variants, because it is rare for no context at all to be available.
The other boundary is full reinforcement learning. A contextual bandit assumes the consequence of an action is captured by its immediate reward and does not model how the action changes future state. When today’s action shapes tomorrow’s options (a notification that changes a user’s future engagement, a price that changes inventory), a one-step reward can optimise the wrong behaviour, and the problem needs the sequential credit assignment of reinforcement learning, where agents learn through trial and error across states. Some authors call the contextual bandit “the half-way point” between supervised learning and reinforcement learning. Methods such as advantage actor-critic (A2C) reinforcement learning belong on the far side of that line, and they bring the cost of modelling state with them.
Exploration costs reward now to save reward later
The fundamental challenge in bandit problems is balancing exploration and exploitation. Exploitation picks the action that currently looks best, but that action may only look best because the estimates are still imprecise. Exploration picks seemingly worse actions to gather information, which raises short-term regret but can lower long-term regret. Regret here means the reward lost against an oracle policy that always picks the best action for each context. Neither pure exploration nor pure exploitation works best in general. Explore too little and the policy risks locking onto a plausible but suboptimal choice. Explore too much and it over-explores, wasting opportunities to earn reward.
The LinUCB paper shows this in its tuning curves. Click-through rate traced an inverted U against the exploration parameter: set too small, the algorithms failed to find good articles, and set too large, they over-explored and gave away clicks. In practice the exploration setting is a business parameter as much as a statistical one.
Epsilon-greedy, LinUCB or Thompson sampling: how does each one decide when to explore?
The literature names three main techniques for the exploration-exploitation trade-off in contextual bandits: epsilon-greedy, upper confidence bound (UCB) methods such as LinUCB, and Thompson sampling. They differ in how they choose which actions to explore.
Epsilon-greedy: explore at random, a fixed share of the time
Epsilon-greedy exploits the current best estimate most of the time and, with probability epsilon, picks an action uniformly at random. In the Vowpal Wabbit documentation’s example, the learned policy’s prediction is taken 80% of the time and a random action the other 20%. A common refinement starts with a larger epsilon and lowers it over time, so exploration happens early, when estimates are weakest. For bandits with observable side information, Epoch-Greedy’s regret scales as O(T^{2/3} S^{1/3}) or better, where S is a sample-complexity term from supervised learning, and it needs no known time horizon.
Epsilon-greedy’s exploration is unguided: it picks exploratory actions uniformly at random, without using any measure of uncertainty.
LinUCB: explore where the model is uncertain
UCB methods estimate each action’s mean reward along with a confidence interval, then pick the action with the highest upper bound. An action wins either because its estimate is strong or because the model is still uncertain about it, so exploration goes where information is missing. LinUCB assumes the expected reward is linear in the features and shows that the confidence interval can then be computed efficiently in closed form. Its selection rule reads as an additive trade-off between the reward estimate and the reduction in model uncertainty. Computation is linear in the number of arms and at most cubic in the number of features, which matters when the feature vector is large. The authors present LinUCB as a generic contextual bandit algorithm, not one tied to news recommendation.
In their experiments, UCB methods consistently beat epsilon-greedy ones in the deployment bucket, and the gap was larger when data was scarce. In the parameter-tuning curves, though, epsilon-greedy with well-chosen parameters reached click-through similar to UCB in the deployment bucket, so both learned comparable policies.
Thompson sampling: explore in proportion to the chance of being best
Thompson sampling keeps a posterior distribution over each action’s reward, draws one sample from each, and plays the action with the highest sample. The effect is that each action is played with roughly the probability that it is the best one, given the data so far. In the linear contextual form (LinTS), a least-squares model’s coefficient and covariance estimates are used in a Bayesian way to produce those samples. The first theoretical guarantees for contextual Thompson sampling with linear payoffs gave a high-probability regret bound of Õ(d^{3/2}√T), within a factor of √d of the lower bound. On the empirical side, Chapelle and Li (NeurIPS 2011) found Thompson sampling highly competitive on simulated and real data and argued that, being very easy to implement, it should be a standard baseline.
Sometimes the context does the exploring
The assumption that every bandit needs deliberate exploration deserves scrutiny. In the large empirical bake-off by Bietti, Agarwal and Langford, an optimism-based method came out best overall, but a simple greedy baseline that explores only through the natural diversity of contexts was a surprisingly close second. The Greedy-First work argues that exploration-free greedy algorithms are attractive where exploration is costly or unethical, and in its simulations Greedy-First reduced exploration and beat Thompson sampling and UCB. The LinUCB paper found guided exploration beat random exploration by more when data was scarce. Our reading is that both hold: when contexts vary a lot, the traffic itself supplies much of the exploration, and when data is thin or contexts are narrow, guided exploration earns its keep.
Tooling: what a library actually gives you
Vowpal Wabbit exposes five exploration algorithms as flags: explore-first (--first), epsilon-greedy (--epsilon), bagging (--bag), online cover (--cover) and softmax (--softmax). Its documentation describes online cover as a theoretically optimal exploration algorithm that trains a set of policies optimised to disagree, so that it keeps trying every action not yet learned to be bad in a given context. Regression-oracle approaches reduce the problem to weighted least-squares regression. In that framework, RegCB with a linear oracle coincides with LinUCB, and across datasets RegCB always beat epsilon-greedy.
When should you use a contextual bandit instead of an A/B test?
Use a contextual bandit when the decision repeats often, context meaningfully predicts which action works, rewards arrive soon enough to learn from, and earning reward during learning matters more than estimating one stable average effect. If you need a clean answer about one feature, run an A/B test.
The difference is one of purpose. A bandit tries to act well while it collects data; an experiment collects data so that a careful analysis can follow. Sven Schmit of Eppo frames it as perishable decisions against evergreen ones: bandits suit the first, experiments the second. The Burtini et al. survey even shows that the classic A/B test is a bandit policy, epsilon-first, which explores uniformly for a fixed share of the horizon and then exploits the winner. That policy beats epsilon-greedy when the horizon is fixed and rewards are stationary, but all its learning happens up front, which leaves it exposed when the reward distribution shifts.
| Situation | Better fit | Why |
|---|---|---|
| Decision repeats often and context predicts the best action | Contextual bandit | It learns a per-context policy and earns reward while learning |
| Low traffic or many variants to test | Bandit | Traffic shifts to the better arm sooner, lowering opportunity cost (Stitch Fix, 2020) |
| Short-lived decision, such as a headline or a seasonal promotion | Bandit | The value of the decision perishes quickly |
| You need an unbiased effect estimate or controlled error rates | A/B test | Fixed allocation keeps assignment independent of outcomes; adaptive allocation biases simple estimates |
| Several metrics or guardrails matter | A/B test | A bandit needs all of them collapsed into one reward |
| Users must get a consistent experience across sessions | A/B test | Adaptive allocation makes that hard to guarantee |
| Actions change future state | Reinforcement learning, or a deliberately simplified policy | A one-step reward ignores downstream effects |
Schmit’s own conclusion is that real decisions usually carry nuance, which leads Eppo to favour experiments more often than not, while in certain scenarios bandits are terrific. We would add that the two are not rivals inside one product: a bandit can run the personalised policy while a stable baseline arm keeps measuring whether the policy beats the simple alternative on outcomes beyond the immediate reward, which is the comparison GrowthBook recommends.
How are contextual bandits used in recommender systems and personalisation?
They serve as a basic algorithmic framework for optimising recommendation decisions online, and they were adopted for online recommenders such as Digg, Yahoo! Buzz and news recommendation in general. The LinUCB paper modelled personalised news recommendation as a contextual bandit: select articles from contextual information about users and articles, and adapt the selection strategy from click feedback to maximise total clicks. Its LinUCB results showed a 12.5% click lift over a standard context-free bandit, with a larger advantage when data was scarcer. Simply adding user features gave both epsilon-greedy and UCB methods around a 10% click-through lift over the baseline epsilon-greedy.
That work also introduced a hybrid model. Some features are shared across all articles, so click information on one article transfers to others, and the hybrid LinUCB showed significant benefits when data was small.
Two cautions apply to personalisation. First, recommendation under user conformity has its own Thompson sampling variant, TS-Conf, which uses posterior sampling to balance exploration and exploitation. And a contextual policy’s gain is not automatically per-user personalisation. In a 2026 study of marketing-message bandits, Lee and colleagues show that when the best single arm cannot be identified from training data, the contextual policy’s edge may come from robustness to that uncertainty instead. On their ground-truth generator, the contextual oracle beat the best fixed arm by +72 to +143 percentage points when real conditional structure existed, and by roughly nothing when it did not.
Delayed rewards, drift and misleading offline numbers
This section covers three failure modes: delayed rewards, drift and misleading offline evaluation. Feedback delay is under-studied in the bandit literature, even though in many real applications the reward arrives long after the action. In Lee and colleagues’ simulations sweeping reward delays of 0, 7 and 14 days, the blind period equalled the delay, convergence shifted right by the delay, and at 14 days the bandit had not converged within 60 days. Teams therefore train on a fast proxy reward, which brings its own trap: their deployed same-day proxy correlated with conversion per user but pointed the wrong way per arm (ρ ≈ −0.8), steering the bandit toward the arms that converted worst.
Drift is the second failure. The best-known bandit algorithms assume no time-varying effects such as seasonality or novelty bias, and can easily be tripped up by them. Algorithms built for non-stationary contexts exist, for example one that uses replay phases to detect change and achieves dynamic regret of O(min{√(ST), Δ^{1/3}T^{2/3}}) over T rounds with S switches and Δ total variation in data distributions.
Evaluating a policy before it touches users is the third. Offline evaluation is hard because bandit logs carry only partial labels. The replay method of Li, Chu, Langford and Wang (WSDM 2011) is data-driven and gives provably unbiased evaluations. Lee and colleagues add that even a doubly robust static estimate cannot see learning efficiency or cold-start dynamics, which need online replay and then a live test. Their pre-deployment checklist runs in this order:
- Check that the proxy reward moves the real outcome per arm, not just per user.
- Compare fixed-arm, context-free and contextual policies to confirm a contextual bandit is warranted.
- Treat marginal arm tests as a screen and read learning efficiency from online replay, not a batch number.
- Use percentile intervals and a variance-reduced estimator, and respect the 1/K variance ceiling.
- Budget the cold-start and maturation lags explicitly.
Trusting logged data to judge a new policy is a subject of its own, covered in our piece on off-policy evaluation before you ship a policy. Microsoft’s Decision Service (Agarwal et al.) structures contextual bandit learning as a loop of explore, log, learn and deploy, with the explore and log steps designed to produce correct, unbiased data. Its authors describe applying contextual bandits in practice as fraught with technical debt. Recording the probability with which each action was chosen, versioning the policy that chose it and keeping the loop reproducible is MLOps engineering more than modelling.
Epsilon-greedy, LinUCB and Thompson sampling all learn workable policies when tuned. The question that matters before launch is whether the quickly observable reward tracks the outcome you care about, and whether the decision has the conditional structure a contextual bandit can exploit. If you cannot answer both before launch, a fixed-arm baseline and a properly analysed A/B test are the safer first step.