A team asks for “an AI model” to pick which offer to show a user. Before anyone reaches for a generative or predictive architecture, the first question is whether this is a generation problem at all. Often it is not — it is sequential decision-making under uncertainty, and that is bandit territory.

Contextual bandit algorithms are a reinforcement-style technique for choosing an action, observing a reward, and learning which action to take next given the situation you are in. They do not generate content. They allocate decisions and optimise cumulative reward from observed feedback. Getting that distinction right at the start is what keeps architecture selection honest, because a generative model and a contextual bandit imply entirely different data pipelines and evaluation loops.

What does a contextual bandit algorithm do, in practice?

Picture a recommendation surface. On each round, a user arrives with a context — a feature vector describing who they are, what device they’re on, what they’ve clicked before. The system picks one action from a fixed set (which article, which offer, which layout). It then observes a reward — a click, a conversion, dwell time — but only for the action it actually took. It never sees what would have happened had it chosen differently.

That last point is the whole game. This is called bandit feedback, and it is what separates the problem from ordinary supervised learning. A classifier trained on a labelled dataset sees the correct answer for every example. A bandit only sees the outcome of the choice it made. It has to learn a good policy while simultaneously paying the cost of every bad choice it makes along the way.

So the algorithm is doing two jobs at once. It estimates the expected reward of each action given the context — a fairly standard regression or classification model sits inside most bandit implementations. And it decides how much to trust those estimates versus how much to keep probing actions it is uncertain about. The second job is where bandits earn their reputation, and it is the part teams most often skip when they try to bolt a static predictive model onto a decision problem.

Across the recommendation and content-selection projects we have worked on, the failure mode is almost always the same: a model trained offline on logged clicks, deployed once, and never allowed to explore. It converges to whatever the logging policy already favoured and quietly stops learning. A contextual bandit is the structural fix for that, because exploration is built into how it selects actions rather than bolted on afterwards.

Why a contextual bandit is not a generative model

This is the reframe that matters most, because the word “AI” flattens genuinely different problem classes into one bucket. A generative model — a GAN, a variational autoencoder (VAE), or a diffusion model — learns a probability distribution over data and produces new samples from it. Its objective is fidelity to that distribution. You evaluate it with likelihood, reconstruction error, or perceptual quality metrics, and you train it on a corpus that you assemble in advance.

A contextual bandit learns a policy — a mapping from context to action — and its objective is cumulative reward under online feedback. There is no distribution over content to reproduce. There is a decision to make repeatedly, and a running tally of how good those decisions turned out to be. You cannot swap one for the other, because they optimise different things and consume different data.

The practical consequence: if you build a diffusion model to solve what is really an allocation problem, you get an architecture that produces plausible-looking outputs but cannot optimise online for click-through or conversion. If you build a bandit to solve what is really a generation problem, you have nothing that produces the content in the first place. The two sit in different regions of the AI taxonomy, and the sub-fields that touch generative architecture — the ones we cover across the generative AI practice — genuinely do not overlap with the decision-allocation region where bandits live.

Here is the distinction laid out directly.

Dimension	Generative model (GAN / VAE / diffusion)	Contextual bandit
Learns	A distribution over data	A policy: context → action
Objective	Fidelity to the data distribution	Cumulative reward from feedback
Feedback signal	Full data samples for training	Reward for the chosen action only
Evaluation	Likelihood, FID, reconstruction, perceptual quality	Regret, cumulative reward, online lift
Data pipeline	Curated corpus assembled ahead of time	Online logs of context, action, reward
Optimises online?	No — trained then served	Yes — learns from live feedback

If your answer to “what is the objective?” is generate content, you are in the left column. If it is allocate decisions and maximise reward over time, you are in the right column. That single question resolves most of the confusion before any code is written.

How do exploration and exploitation actually get balanced?

Every bandit strategy is a rule for resolving one tension. Exploitation means picking the action your current estimate says is best. Exploration means occasionally picking something else to reduce your uncertainty about it. Pure exploitation locks in early mistakes; pure exploration never cashes in what it has learned. Three families dominate in production, and they differ in how principled their exploration is.

Epsilon-greedy is the blunt instrument. With probability epsilon you pick a random action; otherwise you exploit the current best. It is trivial to implement and surprisingly hard to beat as a baseline, but its exploration is undirected — it wastes probes on actions it already knows are bad. In configurations we’ve tuned, a decaying epsilon (high early, shrinking as data accumulates) usually beats a fixed one.

Upper Confidence Bound (UCB) makes exploration principled. Instead of exploring at random, it adds an optimism bonus to each action’s estimated reward, sized by how uncertain that estimate is. Actions the model has barely tried get a large bonus and are tried more; well-understood actions rely on their estimate. This is directed exploration, and it concentrates effort where the uncertainty actually is.

Thompson sampling takes a Bayesian route. It maintains a posterior distribution over each action’s reward, draws a sample from each on every round, and plays the action with the highest sampled value. Actions with wide posteriors sometimes draw high, so they get explored; as evidence accumulates, posteriors tighten and exploration naturally decays. It tends to be robust in practice and pairs cleanly with linear or neural reward models.

The reinforcement-learning machinery behind these estimators — value estimation, policy updates, the reward-driven training loop — is shared with fuller RL methods; our walkthrough of the advantage actor-critic (A2C) algorithm in practice covers the version of that loop used when actions have downstream consequences the bandit deliberately ignores.

When is a contextual bandit the right fit?

The decision between a bandit, full reinforcement learning, and a plain supervised model comes down to two questions: does your action change the state the next decision is made in, and do you have the correct answer at training time?

• Supervised model — Use it when you have labelled ground truth for every example and no exploration cost. Classifying whether an image is defective is supervised; there is a right answer and you already know it. The moment the “right answer” depends on a choice you make and observe the reward of, supervised framing breaks.
• Contextual bandit — Use it when each decision is myopic: your action produces an immediate reward and does not meaningfully change the context of the next decision. Which article to recommend to this user, right now, is a bandit problem. The reward is immediate and the next user is independent.
• Full reinforcement learning — Use it when your action changes the state and rewards are delayed across a sequence. A robot navigating a room, or a dialog policy where this turn shapes the next, needs the credit-assignment machinery of full RL. That is more data-hungry and harder to stabilise.

The reason to reach for a bandit rather than full RL whenever the problem allows it is efficiency. Contextual bandits reach near-optimal action selection with far less data than full reinforcement learning (observed pattern across recommendation-style engagements; not a benchmarked rate), precisely because they do not have to solve the long-horizon credit-assignment problem. If your decisions genuinely are independent round to round, paying the RL data and stability tax buys you nothing.

What feedback loop does a contextual bandit require?

A bandit is only as good as the loop that feeds it. It needs three things logged on every round: the context it saw, the action it chose, and the reward it observed. Miss any one and the policy cannot learn — this is the operational reality that catches most first attempts.

The subtle requirement is logging the probability with which each action was chosen. Because a bandit only observes the reward for the action it took, offline evaluation and off-policy correction depend on knowing how likely that action was under the deployment policy. Teams that log clicks but not propensities cannot later re-evaluate a candidate policy honestly, and they cannot debug why the live system converged where it did. This is the same data-discipline concern that governs any production ML system; the broader argument for treating the data pipeline as the primary object sits in our data-centric approach to AI feasibility.

There is also a latency constraint that generative pipelines rarely face. The reward has to close the loop while it is still attributable to the action. A conversion three weeks later is hard to credit; a click within seconds is clean. Designing the reward signal so it arrives promptly and unambiguously is often more decisive for success than the choice between epsilon-greedy and Thompson sampling.

Python has matured tooling for this — Vowpal Wabbit’s contextual bandit modes, the contextualbandits library, and RL frameworks that expose bandit reductions — and our survey of Python reinforcement learning libraries for adaptive systems covers where each fits.

What is regret, and how do you know a bandit is converging?

Generative models give you a vague accuracy target. Bandits give you a concrete convergence metric, and it is the single most useful thing to understand about them for anyone signing off on an architecture.

Regret is the difference between the cumulative reward you actually earned and the reward you would have earned if you had known the best action all along. It measures the total cost of learning — of every round spent exploring or exploiting a mistaken estimate. A good bandit algorithm has sublinear regret: as rounds accumulate, the average regret per round trends toward zero, which means the policy is converging on near-optimal decisions. Under the right assumptions, that regret can be bounded, giving you a theoretical convergence guarantee rather than a hope.

Operationally, you watch cumulative reward against a baseline — the fixed logging policy, or a random policy — and you expect the gap to widen in the bandit’s favour and the exploration rate to decay. If cumulative reward is flat against baseline, either the context carries no signal, the reward is mis-specified, or exploration collapsed too early. Regret is the lens that tells you which.

FAQ

What’s worth understanding about contextual bandit algorithms first?

On each round a contextual bandit sees a context (features describing the situation), picks one action from a fixed set, and observes a reward only for the action it chose. It estimates the expected reward of each action given the context and balances exploiting the current best estimate against exploring uncertain actions. In practice it learns a good decision policy online while paying the cost of its own mistakes along the way.

How does a contextual bandit differ from a generative model like a GAN, VAE, or diffusion model — and why is it a separate category?

A generative model learns a distribution over data and produces new samples, optimising for fidelity to that distribution. A contextual bandit learns a policy mapping context to action and optimises cumulative reward from online feedback. They consume different data and are evaluated differently, so one cannot substitute for the other — that is why they occupy separate regions of the AI taxonomy.

What is the exploration vs exploitation trade-off, and how do strategies like epsilon-greedy, UCB, and Thompson sampling handle it?

Exploitation picks the action currently believed best; exploration probes uncertain actions to improve future estimates. Epsilon-greedy explores at random with fixed probability; UCB adds an optimism bonus scaled by uncertainty so exploration is directed; Thompson sampling draws from a posterior over each action’s reward, exploring more when posteriors are wide and less as they tighten.

When is a contextual bandit the right fit versus full reinforcement learning or a supervised model?

Use a supervised model when you have ground-truth labels and no exploration cost. Use a contextual bandit when each decision is myopic — the action yields an immediate reward and does not change the next decision’s state. Use full reinforcement learning when actions change the state and rewards are delayed across a sequence, which requires heavier credit-assignment machinery and more data.

What data and feedback loop does a contextual bandit require to learn effectively?

It needs the context, the chosen action, and the observed reward logged on every round — and, critically, the probability with which each action was selected, so off-policy evaluation stays honest. The reward should close the loop promptly enough to remain attributable to the action; delayed or ambiguous rewards undermine learning more than the choice of exploration strategy does.

What are realistic production use cases for contextual bandits, such as recommendation or content selection?

Recommendation, content and layout selection, offer allocation, and ad selection are the canonical fits — each is a repeated, myopic decision with an immediate, attributable reward. They suit any surface where you choose one option per impression, observe a click or conversion, and want the system to keep improving from live feedback rather than a one-time offline training run.

What is regret, and how do you measure whether a bandit is converging toward good decisions?

Regret is the gap between the reward you earned and the reward an oracle that always chose the best action would have earned — the total cost of learning. A converging bandit has sublinear regret, meaning average regret per round trends toward zero. Operationally, you track cumulative reward against a baseline policy and watch the exploration rate decay; a flat gap signals no signal, a mis-specified reward, or exploration that collapsed too early.

Before recommending any architecture, the honest first move is to classify the problem, not the technology. If the objective is to allocate decisions and optimise cumulative reward from observed feedback, no amount of generative modelling will serve it — and mislabelling a decision task as a generation task is exactly the taxonomy error an A3 feasibility assessment exists to catch early.