A team picks tree-of-thought because it topped a reasoning benchmark, ships it, and watches the inference bill climb. The strategy that looked obviously better on accuracy is now the most expensive line in the serving budget — and nobody priced the extra tokens before the decision was made.

That is the trap. Choosing a reasoning strategy sounds like a quality question right up until you count the tokens. Tree-of-thought fans out multiple candidate branches, evaluates them, and prunes — which means it can multiply the tokens spent on a single request several times over. Chain-of-thought walks a single reasoning path. Both improve on a raw one-shot answer, but they do not cost the same, and the accuracy gap between them is often much narrower than the cost gap.

The claim this article makes is simple: a reasoning-strategy decision only survives contact with production if it is measured in cost-per-request and cost-per-token at the p95 latency your product actually allows — not on a leaderboard’s accuracy column. Rank the two strategies on accuracy alone and tree-of-thought wins. Reprice the branch-and-evaluate overhead against the same latency budget, and chain-of-thought often wins on margin for the majority of requests.

How chain-of-thought vs tree-of-thought works in practice

Chain-of-thought (CoT) asks the model to reason step by step along a single path before committing to an answer. The token cost is the prompt plus one reasoning trace plus the final answer. It is roughly linear in the length of the reasoning the task demands.

Tree-of-thought (ToT) generates several candidate reasoning steps at each stage, scores them, keeps the promising ones, and expands from there. It is structurally closer to a search: a branching factor, a depth, and an evaluation pass at each node. If you keep three branches over four steps and evaluate each, you are not running one reasoning trace — you are running many, plus the evaluation calls that decide which to keep. The same mechanics show up in decoding strategy: keeping multiple candidate sequences is exactly what beam search decoding does at the token level, and it carries the same cost shape.

The practical consequence: ToT’s output-token count per request is a multiple of CoT’s, not a small delta. Across the agentic and reasoning workloads we have looked at, that fan-out typically lands somewhere in the 3–10x range on output tokens per request, depending on branching factor and how aggressively the search prunes (observed pattern across engagements, not a published benchmark). The exact multiplier is workload-specific — but that it is a multiplier, not an increment, is the part teams routinely underestimate.

Why tree-of-thought’s fan-out multiplies cost-per-request

Cost-per-request at the reasoning layer is dominated by output tokens, because generation is the expensive phase — each output token requires a full decode pass, and decode is where the GPU spends most of its time. When ToT expands branches, every branch generates tokens, and every evaluation step generates or scores tokens. The prompt is amortized once; the reasoning is paid for many times.

This is why an accuracy-first comparison misleads. A reasoning benchmark reports the accuracy ToT achieved. It does not report that ToT spent, say, six times the output tokens to get there. When you price those tokens at your provider’s rate — or at your own serving cost if you self-host — the marginal accuracy points can turn out to be some of the most expensive tokens in your product. An LLM token calculator makes the per-request arithmetic concrete before you commit a strategy to the serving path.

There is a latency dimension too. ToT branches can run in parallel if you have the capacity, but parallel branches consume more concurrent GPU memory and can push you into a higher serving tier or degrade the p95 latency of everything else sharing the pool. If the branches run sequentially to save capacity, latency climbs directly. Either way, the same p95 budget that CoT clears comfortably can be the constraint that ToT keeps violating. Pricing the extra branch-evaluation passes requires the per-config latency and utilisation numbers that GPU profiling supplies — the strategy comparison is only as trustworthy as the serving measurements underneath it.

How do you compare the two strategies fairly?

A fair comparison fixes what the business already fixes and varies only the reasoning strategy. Two constraints do the work: a p95 latency budget (the tail latency your product commits to) and an accuracy bar (the quality threshold the feature must clear). Hold both constant, run each strategy through your actual serving path, and read off cost-per-request and cost-per-token.

The discipline here is the same one that governs any config comparison: the numbers are only comparable if the run conditions are identical. If ToT runs on a bigger instance or a different quantization, you are comparing serving configs, not reasoning strategies. Which numbers you then let decide the choice is a question of picking the metrics that actually decide a serving config rather than the ones a benchmark happened to report.

Decision rubric: which reasoning strategy for this request class

If your request class shows…	Lean toward	Why
Reasoning is mostly single-path; one good trace suffices	Chain-of-thought	Fan-out buys little accuracy; you pay tokens for nothing
High-value, low-volume requests where a wrong answer is costly	Tree-of-thought	Accuracy gain justifies the token multiplier on few requests
Mixed: most requests easy, a minority genuinely hard	Routed hybrid	Run ToT only on the hard minority; CoT clears the rest cheaply
Tight p95 budget, high request volume	Chain-of-thought	ToT’s branch overhead risks tail-latency and capacity blowout
Accuracy bar unmet by either at acceptable cost	Neither yet	Revisit model choice or retrieval before spending on search

Evidence class for the rows above: observed-pattern — these are planning heuristics from reasoning-feature engagements, not a benchmarked ranking. Your own cost-per-request measurement overrides any row.

What does a worked cost-per-request comparison look like?

Here is an illustrative worked example. The numbers are assumptions to show the shape of the arithmetic, not a benchmark result.

Assumptions. A support-reasoning feature, 1,000-token prompt, provider priced at a flat per-output-token rate. CoT produces ~800 output tokens per request. ToT keeps three branches over three steps with an evaluation pass, producing ~4,800 output tokens per request — a 6x output multiplier. Accuracy bar: 85%.

Strategy	Output tokens/req	Relative cost/req	Accuracy (illustrative)	Clears 85% bar?
Chain-of-thought	~800	1.0x	84%	Just under
Tree-of-thought	~4,800	~6x	89%	Yes
Routed hybrid (20% to ToT)	~1,600	~2x	88%	Yes

Read this the way a margin owner would. ToT clears the bar but at six times the cost. The routed hybrid — send the 20% of requests a cheap classifier flags as hard to ToT, let CoT handle the rest — clears the bar at roughly a third of ToT’s cost. The strategy that topped the accuracy column is the wrong default; the routing decision is where the money is. This is the same fan-out arithmetic that governs agentic workloads where one request explodes into many calls — the reasoning layer is just an early, controllable instance of it.

When does tree-of-thought’s accuracy gain actually justify the cost?

The honest answer is: less often than the benchmark suggests, but not never. ToT earns its multiplier when three conditions hold together — the request is genuinely hard enough that a single reasoning path frequently fails, the cost of a wrong answer is high relative to the extra tokens, and the volume of such requests is low enough that the aggregate spend stays bounded. A high-stakes, low-volume decision — a complex diagnostic step, a legal-reasoning check — can be worth six times the tokens. A high-volume chat feature almost never is.

The failure mode we see most is applying ToT uniformly because it won a benchmark, then discovering the aggregate token bill dwarfs the accuracy value on the 80% of requests that CoT would have answered correctly anyway. The fix is not to abandon ToT; it is to route it.

How do you route so tree-of-thought only runs where it’s needed?

Routing turns a binary strategy choice into a per-request one. A lightweight classifier — often a small model or a heuristic on request features — predicts whether a request is hard enough to need branching. Easy requests take the CoT path; hard ones take ToT. The economics of the hybrid are governed by the split: if only 20% of requests route to a 6x path, your blended cost is roughly 2x CoT rather than 6x, while capturing most of ToT’s accuracy on the requests that actually needed it.

Two things make or break the router. First, the classifier’s own cost and latency have to be small relative to the reasoning it gates — a router that costs as much as CoT defeats the purpose. Second, the router’s error profile matters more than its raw accuracy: sending an easy request to ToT wastes tokens, but sending a hard request to CoT can miss the accuracy bar. Tune the threshold against your cost-per-request SLO, not against the router’s F1 in isolation.

How this feeds a defensible config-selection decision

All of this rolls up into a single decision your finance and engineering owners can both defend: which reasoning strategy, at which routing threshold, on which serving config, meets the accuracy bar at the lowest cost-per-request within the p95 budget. That is a config-selection decision under a cost-per-request SLO, and it is the same decision framework that governs model choice, quantization, and serving runtime — the reasoning strategy is one more variable priced the same way everything else is.

This is the method our [inference cost-cut engagement](Inference Cost-Cut Pack) applies against a buyer’s own deployed serving path: benchmark chain-of-thought and tree-of-thought — and the routed hybrid — to produce the cost-per-request delta before and after. For teams building on AI-infrastructure and SaaS serving platforms, the reasoning layer is often the fastest-moving line in the inference budget, and the one most likely to have been chosen on accuracy alone.

FAQ

What matters most about chain-of-thought vs tree-of-thought in practice?

Chain-of-thought walks a single step-by-step reasoning path before answering, so its token cost is roughly linear in reasoning length. Tree-of-thought generates multiple candidate branches at each step, scores them, and prunes — structurally a search. In practice that means tree-of-thought spends a multiple of the output tokens chain-of-thought does per request, so the choice is as much a cost decision as a quality one.

Why does tree-of-thought’s branch fan-out multiply cost-per-request compared to chain-of-thought?

Cost-per-request at the reasoning layer is dominated by output tokens, and every branch tree-of-thought expands generates tokens, plus the evaluation passes that decide which branches to keep. The prompt is amortized once but the reasoning is paid for many times. Across the reasoning workloads we have looked at, the output-token multiplier typically lands in the 3–10x range depending on branching factor and pruning (observed pattern, not a published benchmark).

How do you compare the two strategies fairly at a fixed p95 latency and accuracy bar?

Fix the two things the business already fixes — a p95 latency budget and an accuracy threshold — and vary only the reasoning strategy, running each through your actual serving path. If one strategy runs on a different instance or quantization, you are comparing serving configs, not strategies. Read off cost-per-request and cost-per-token under identical run conditions so the number, not a benchmark’s accuracy column, decides the choice.

What does a worked cost-per-request comparison between chain-of-thought and tree-of-thought look like?

Take a fixed prompt size and accuracy bar, measure output tokens per request for each strategy, and price them at your serving or provider rate. In an illustrative example with a 6x output multiplier, tree-of-thought clears an 85% bar at roughly six times chain-of-thought’s cost, while a routed hybrid sending 20% of requests to tree-of-thought clears the same bar at about a third of tree-of-thought’s cost. The routing decision, not the raw strategy, is where the margin is.

When does the accuracy gain from tree-of-thought actually justify its token cost?

When three conditions hold together: the request is genuinely hard enough that a single reasoning path often fails, a wrong answer is costly relative to the extra tokens, and the volume of such requests is low enough that aggregate spend stays bounded. A high-stakes, low-volume decision can be worth the multiplier; a high-volume chat feature almost never is. Applying tree-of-thought uniformly because it won a benchmark is the most common way the token bill outruns the accuracy value.

How do you route requests so tree-of-thought only runs on the queries that need it?

Use a lightweight classifier or heuristic to predict whether a request is hard enough to need branching, sending easy requests to chain-of-thought and hard ones to tree-of-thought. The blended cost depends on the split — routing 20% of traffic to a 6x path yields roughly 2x chain-of-thought cost while capturing most of the accuracy gain. Keep the router’s own cost small relative to the reasoning it gates, and tune its threshold against the cost-per-request SLO rather than its F1 in isolation.

How do these reasoning-strategy choices feed back into a defensible config-selection decision under a cost-per-request SLO?

They collapse into one decision: which strategy, at which routing threshold, on which serving config, meets the accuracy bar at the lowest cost-per-request within the p95 budget. That is the same framework that governs model choice, quantization, and serving runtime, with the reasoning strategy priced as one more variable. Measured that way, the choice is defensible to both engineering and finance because it is stated in the currency the business already uses.

The reasoning strategy that wins a benchmark and the one that wins your margin are rarely the same — the question worth answering before you ship is which one clears the accuracy bar at the lowest cost-per-request within the p95 latency your product already promised.