Two teams benchmark the same model on the same dataset and report exact-match scores several points apart. Nobody changed the weights. What changed sits between the model and the text on your screen: the decoding strategy. Beam search, greedy decoding, and sampling are not interchangeable knobs — they are different search procedures over the same probability distribution, and they produce different text. If an eval does not pin down which one ran, the number it reports is not reproducible, and a procurement committee cannot defend it.

That is the core problem this article addresses. A language model does not emit text directly. It emits, at each step, a probability distribution over the next token. Decoding is the algorithm that turns that sequence of distributions into an actual string. Beam search is one such algorithm, and understanding what it optimises is the difference between reading an eval number as evidence and reading it as an artifact of a configuration nobody wrote down.

What does working with beam search decoding involve in practice?

Greedy decoding is the simplest thing you can do: at every step, take the single most probable token and move on. It is fast and deterministic, but it is myopic. The locally best token can lead down a path where every subsequent choice is worse, and greedy has no way to back out.

Beam search widens that lens. Instead of tracking one candidate sequence, it keeps the k most probable partial sequences at every step — k is the beam width. At each new token, it expands every surviving beam by every possible next token, scores the resulting sequences by their cumulative log-probability, and prunes back down to the top k. When all beams hit an end-of-sequence token or a length limit, it returns the sequence with the highest total probability.

The practical consequence is that beam search finds sequences with higher aggregate likelihood than greedy does. It is a bounded search for the most probable whole output, not the most probable next token. For tasks where there is a single correct answer expressed in a fairly constrained way — translation, structured extraction, short-form question answering — that bias toward high-probability sequences tends to line up with what the reference answer looks like.

But “higher probability” is not the same as “better,” and this is where the naive reading breaks. Maximising sequence probability drives beam search toward safe, generic, high-frequency phrasing. In open-ended generation it produces the well-documented failure of repetitive, bland text — the model keeps choosing the likeliest continuation, which is often a loop. The strategy that flatters a translation metric can hollow out a creative or conversational task.

How beam search differs from greedy and sampling methods

The cleanest way to hold these apart is to ask what each strategy is trying to do with the distribution the model hands it.

Strategy	What it does	Determinism	Diversity	Where it fits
Greedy	Takes the single most probable token each step	Deterministic	Lowest	Fast baselines; short constrained outputs
Beam search (width k)	Keeps top k partial sequences by cumulative probability	Deterministic	Low	Translation, structured extraction, exact-match tasks
Top-k sampling	Samples from the k most probable tokens	Stochastic	Moderate	Conversational, creative generation
Top-p (nucleus)	Samples from the smallest token set whose probability mass exceeds p	Stochastic	Moderate–high	Open-ended generation with adaptive cutoff

Two distinctions carry most of the weight here. The first is deterministic versus stochastic. Greedy and beam search are deterministic — run them twice with the same input and you get the same output. Top-k and top-p sample, so they need a fixed random seed and a fixed temperature before an eval is repeatable at all. The second is what each optimises: beam search optimises for probability, sampling optimises for a controlled amount of surprise. That difference is exactly why an eval can move without the model moving.

If you are reasoning about how these token-level choices interact with the vocabulary itself, the mechanics of how text becomes tokens matter too — our explanation of how WordPiece subword tokenization shapes eval behaviour covers the layer directly beneath decoding.

Why beam search can change an exact-match or BLEU score without changing the model

This is the claim a procurement audience needs to internalise. Exact-match and BLEU both reward output that matches a reference string. Beam search, by construction, favours high-probability sequences — and reference answers in benchmark datasets are usually written in conventional, high-frequency language. So beam search tends to land closer to the reference than sampling does, which means it can post a higher exact-match or BLEU number on the same model with the same weights.

In practice, the gap between decoding configurations on match-based metrics is often several points — enough to reorder a shortlist of candidate models (an observed pattern across LLM evaluation work, not a single benchmarked constant). That is not a rounding difference. It is the difference between a model clearing an acceptance threshold and missing it, decided entirely by a config line that may never have been logged.

Three consequences follow, and each one is a reproducibility hazard:

• A vendor reporting beam-search numbers and a buyer re-running with sampling will disagree, and both will be technically correct about their own run.
• A model that looks strong under beam search may degrade at the sampling temperature actually used in production, because the metric was measured on a distribution the deployment never sees.
• Two internal eval runs weeks apart can diverge purely because someone changed the default decoding config in the harness.

The fix is not to declare one decoding strategy correct. It is to treat decoding as a first-class part of the eval specification — a variable that gets recorded alongside the number, the same way you would record the dataset version or the scoring function. This is one strand of the larger discipline we describe in how an evaluation spec links task, dataset, scoring, and run conditions, where decoding sits in the run-conditions column.

When beam search helps, and when it hurts

Beam search earns its place on tasks with a narrow target and a probability-aligned reference. It hurts on tasks where diversity is the point, and — more subtly for anyone auditing a model — it can hide failure modes that sampling would expose.

Here is the trade-off as a decision rubric:

• Use beam search when the task has a mostly-correct answer expressed in conventional phrasing: machine translation, grammar correction, structured field extraction, short factual QA where the reference is tight.
• Prefer sampling when you need varied, natural, or creative output, or when you want the eval to reflect the stochastic behaviour production will actually run.
• Be wary of beam search when auditing safety or robustness. Because it collapses onto high-probability paths, beam search can under-report failure modes — jailbreaks, hallucinations, degenerate loops — that sampling surfaces because sampling explores lower-probability continuations the model can still produce in deployment.

That last point is the one buyers miss. A clean beam-search eval can look reassuring precisely because it never visits the risky regions of the distribution. If your production traffic runs top-p sampling, an eval that only ran beam search has measured a model you are not deploying. When the eval touches safety behaviour specifically, the decoding strategy interacts with what a safety benchmark can and cannot prove in production — beam search can quietly suppress exactly the behaviours a safety eval is meant to catch.

What beam width controls, and its cost

Beam width k is the knob that trades output quality for compute. A wider beam searches more of the sequence space, which can improve match-based quality up to a point — but the returns diminish, and past a moderate width the extra beams rarely change the winning sequence while they always cost more.

The cost is concrete. Each decoding step now scores roughly k times as many candidate continuations, so both latency and memory scale with the beam width. A width-8 beam does substantially more work per token than greedy decoding, which raises time-to-first-token and cost-per-request. For a production feature, that cost lands on every single request, which is why beam width belongs in the same conversation as the rest of your serving economics — the kind of accounting we walk through in which machine-learning model metrics actually decide a serving config.

There is also a subtle length interaction: because beam search sums log-probabilities, longer sequences accumulate more negative score, so naive beam search is biased toward shorter outputs unless you apply length normalisation. That is another parameter that has to be recorded, because it too moves the number.

For a picture of how the decoding layer is evolving toward cheaper search, our explanation of how lookahead decoding cuts inference cost-per-request shows where the token-generation loop is heading beyond classic beam search.

How to record decoding so an eval is reproducible

If you take one operational thing from this article, it is that the decoding configuration must travel with every metric. An exact-match score without a decoding config is not evidence — it is a number that cannot be re-derived. We treat decoding as part of the reproducibility surface that a [production AI monitoring and validation harness](Production AI Monitoring Harness) has to capture, so that every reported metric is attributable to a known run. For teams building on top of an AI-infrastructure SaaS platform, this is what keeps vendor benchmark claims auditable rather than aspirational.

At minimum, record: the decoding strategy (greedy / beam / top-k / top-p), the beam width or the k/p value, the temperature, any length normalisation or repetition penalty, and — for stochastic strategies — the random seed. That short block is the difference between an eval a procurement committee can defend and one that quietly resets on the next run. It is also the piece that connects decoding to the broader question of what a governance pack must capture as reproducibility evidence, where decoding settings sit alongside the dataset and scoring provenance.

FAQ

What should you know about beam search decoding in practice?

Beam search keeps the k most probable partial sequences at each generation step, where k is the beam width, expanding and re-pruning them until they terminate, then returns the sequence with the highest cumulative probability. In practice it is a bounded search for the most probable whole output rather than the most probable next token, which biases it toward safe, high-frequency phrasing.

How does beam search differ from greedy decoding and sampling-based methods like top-k and top-p?

Greedy takes only the single most probable token each step; beam search tracks several high-probability sequences at once; top-k and top-p sample from a restricted set of likely tokens. Greedy and beam search are deterministic and optimise for probability, while sampling is stochastic and introduces controlled variation — which is why sampling needs a fixed seed and temperature to be reproducible at all.

Why can the choice of beam search change an LLM’s exact-match or BLEU score without changing the model?

Exact-match and BLEU reward matching a reference string, and reference answers are usually written in conventional, high-frequency language that beam search’s probability bias tends to land close to. So beam search can post higher match-based scores than sampling on identical weights, and the gap is often several points — enough to reorder a candidate shortlist.

When does beam search help output quality, and when does it hurt diversity or surface fewer failure modes?

Beam search helps on narrow-target tasks with conventional references — translation, structured extraction, short factual QA. It hurts on open-ended or creative generation where it produces bland, repetitive text, and it can under-report failure modes in safety or robustness audits because collapsing onto high-probability paths never visits the risky regions of the distribution that sampling explores.

How should the decoding configuration be recorded so an eval is reproducible and defensible in procurement review?

Record the decoding strategy, the beam width or k/p value, the temperature, any length normalisation or repetition penalty, and the random seed for stochastic methods, and attach that block to every metric. A score without its decoding config cannot be re-derived, so it is not defensible evidence under procurement scrutiny.

What does beam width control, and how does it trade off against latency and cost-per-request?

Beam width controls how much of the sequence space the search explores; wider beams can improve match-based quality but with diminishing returns. Each step scores roughly k times as many candidates, so latency, memory, and cost-per-request scale with width, and that cost lands on every request in production.

Decoding is the layer everyone forgets to write down, and it is the one that most quietly breaks reproducibility. The next time an eval number moves and the weights did not, the first question to ask is not “did the model change?” — it is “did the decoding config change, and can anyone prove which one ran?”