To stop paying tree-of-thought prices on every clause, run a cheap chain-of-thought pass first. Escalate to tree-of-thought, or to a frontier model, only when that pass reports low confidence or the clause is flagged as high severity. The Tree of Thoughts authors (Yao et al., 2023) say tree search can generate 5 to 100 times as many tokens as chain-of-thought. In the RouteNLP authors’ (2026) legal clause-extraction benchmark, a tiered router kept 99.2% of quality at a 78% cost saving. The catch is that a routing policy is only as good as the evaluation behind it. It needs a calibrated threshold, a rubric that scores interpretation coverage rather than a match to one reference answer, and human review to catch what slips through.
In a cascade, expected token cost is a weighted average of the cheap and expensive paths, set by how often the expensive path runs. The rest of this article is about controlling that frequency without losing the clauses that matter. For the wider engineering picture, see AI engineering for legal teams.
How do I stop paying for tree-of-thought on every legal clause?
Build a two-stage pipeline. A chain-of-thought pass handles every clause and reports a confidence score. A triage rule then sends a clause on to tree-of-thought only if its confidence falls below a calibrated threshold or its clause type is on a high-severity list; every other clause keeps the cheap result.
TechnoLynx builds this kind of two-stage routing pipeline for legal workloads through the Inference Cost-Cut Pack. The work starts by framing the problem as a two-stage inference pipeline with routing logic before any code is written. It then chooses the confidence and severity thresholding approach, implements the triage on the target hardware and batching setup, and validates escalation rates, cost per request and recall against human-reviewed ground truth before deployment. After deployment, monitoring tracks escalation volume and recall drift so the threshold can be adjusted as the clause mix changes.
Cheap-first is a sound default for clause work. It is not just a budget compromise. In the PROSLEX study of expert-annotated Indian statute prediction (2026), chain-of-thought gave the best results among the in-context methods tested, and GPT-4 reached a macro F1 of 0.75. The authors found that tree-of-thought branches could read the same statutory language differently, producing inconsistent predictions, while a single reasoning chain kept the model focused on one line of reasoning. In that study, tree search did not beat a single reasoning chain for statute prediction. Its extra token cost is documented; its benefit has to be shown clause type by clause type.
Cheap-first cascades have also been tested on legal data. FrugalGPT (Chen et al., 2023) cut costs by 73% against GPT-4 on the OVERRULING legal dataset while gaining 1% in accuracy. RouteLLM (2024) reported cost reductions of more than 2 times on general benchmarks (MMLU, MT Bench and GSM8K) without sacrificing response quality.
Setting the confidence threshold that triggers escalation
There is no universal threshold number. The optimal threshold is model-specific and depends on the ratio of error cost to escalation cost, so it has to be measured. Three methods from the literature fit together well:
- Answer directly, escalate on low confidence. In confidence-gated chain-of-thought (2025), each query is first answered directly, and only queries with low confidence are re-run with reasoning.
- Sweep the budget, then pick from the Pareto front. Sweep percentiles of the confidence distribution, so that each threshold sends a fixed share of queries to the expensive path, and plot accuracy against token cost. A threshold is Pareto-optimal if no other threshold gives equal or higher accuracy at a lower token cost.
- Fix the error-rate guarantee with conformal calibration. RouteNLP sets each tier’s threshold by conformal risk control on 500 labelled calibration examples. At that size, coverage violations came out at 4.2%, with the upper end of the confidence interval just above the 5% target.
The cost of a wrong answer belongs in the threshold as well. The escalation study “Act or Escalate?” (2026) derives the cost-minimising threshold as one minus the ratio of the escalation cost to the cost of acting wrongly.
Worked example (assumption stated): suppose an undetected error on a clause costs four times as much as escalating it. The threshold is then 1 − 1/4 = 0.75: any clause whose calibrated probability of being correct falls below 75% goes to the expensive path. If you raise the cost of a missed indemnity clause, the threshold rises with it, which is exactly where a severity trigger comes from. To bound total spend, feed the escalation rate into cost-per-request modelling, and check the result against your budget before you commit.
How much clause volume ends up on the expensive path
Across RouteNLP’s deployment domains (finance, customer service and legal together), 25–35% of queries required frontier models. The share depends heavily on the task.
| Study (year) | Legal task | Share sent to expensive path | Quality cost |
|---|---|---|---|
| RouteNLP (2026), benchmark | Clause extraction | 9% third tier + 5% frontier tier = 14% (top two of four tiers) | F1 90.9 vs 91.6 for the frontier model, 99.2% retained, 78% cost saving |
| RouteNLP (2026), benchmark | Risk assessment | 22% third tier + 15% frontier tier = 37% (top two of four tiers) | Accuracy 86.1 vs 88.3, 97.5% retained, 40% cost saving |
| Human-on-the-Loop legal discovery (2026) | Privilege review | 23.7% escalated to attorneys at τ = 0.5 | Privilege-waiver rate cut from 8.3% to 3.2% against fully autonomous review |
In RouteNLP, risk assessment sent more of its queries to the top two tiers than clause extraction did.
Where confidence gates mislead
A confidence gate fails in two directions, missing recoverable problems or escalating at high cost. In a cost-aware routing study (2026), a self-confidence gate under-escalated on 13.7% of test problems and over-escalated on 42.1%. Higher-solve frozen LLM routers cut under-escalation to 6–11% but over-escalated on 63–71% of problems. The same study found that post-answer confidence predicted failure well, with an AUROC of 0.88. However, it could not reliably pick which expensive protocol would pay off. Confidence answers “stay or escalate”, not “escalate to what”.
LLMs are miscalibrated in their self-assessment. Some models overestimate their own accuracy while others underestimate it, and the direction varies by model and domain. In one study of a small model on reasoning-trap tasks, verbalised confidence rose faster than accuracy under light reasoning budgets, and how reliable verbalised confidence is depends strongly on how the prompt asks for it. There is also a selection-bias trap. In the confidence-gated CoT study, low-confidence queries were the most token-intensive, so each escalated call cost more than average; for GPT-5.1, fewer reasoning calls did not offset those longer calls.
Thresholds also need recalibration. A calibration set has to reflect the difficulty and diversity of the real tasks, or the threshold transfers badly. RouteNLP found weekly recalibration sufficient in its live deployment. Re-run calibration whenever the contract mix changes, not only on a fixed schedule.
If the cheap tier is going to be self-hosted, LynxBenchAI’s published results show how candidate devices run the public PyTorch stack that custom models also run on. That makes them a reasonable guide when choosing hardware for that tier, although they do not measure your own routing pipeline.
Does tree-of-thought improve multi-statute interpretation enough to pay for it?
In PROSLEX’s expert-annotated statute prediction study, chain-of-thought outperformed it. Treat any coverage gain as something to measure on your own queries, against expert-annotated ground truth, before paying for it at scale. Where a team wants that comparison designed and run on its own statutes, TechnoLynx takes it on as Generative & Agentic AI R&D work, starting with the set of expert-annotated queries the comparison will be scored against.
Tree-of-thought does help on some puzzle tasks. On the Game of 24, it reached 74% success against 4% for chain-of-thought. The authors recommend it for tasks that need deliberate reasoning where chain-of-thought struggles. Whether competing readings of overlapping statutes are such a task is an empirical question. The PROSLEX result is a warning not to assume the answer.
The decision follows from the measurement. If tree-of-thought shows a substantial, statistically supported coverage gain and the token cost is acceptable at full volume, adopt it outright. If the cost is prohibitive, send only low-confidence or high-severity queries to tree-of-thought and check whether that subset captures most of the gain.
Can routing only a subset to tree-of-thought keep most of the quality gain?
The available routing studies, none on tree-of-thought, show subset routing recovering part of the gain, short of perfect routing. In the cost-aware routing study (2026), stronger protocols had recoverable value in all 10 settings tested. In six settings, a learned router improved on the cheap baseline by 7.2–37.5 points yet stayed 18.5–28.9 points below a fixed-order oracle. In confidence-gated chain-of-thought, gating matched always-on reasoning on several tasks while cutting tokens by 30–50%.
What is the token cost difference in production traffic?
The Tree of Thoughts authors report token costs on benchmark tasks, the Game of 24 and Creative Writing. One finding points to higher costs on live traffic than in simulation. In RouteNLP’s pilot, live traffic used the frontier tier 9.7% of the time against a simulated 7.3%, because live queries were more complex. Separately, in the confidence-gated CoT study, the queries that triggered reasoning were the most token-intensive. RouteLLM’s cost estimate, for instance, assumed short single-turn prompts with an average of 95 input tokens. The practical method is to run both paths in shadow mode on a sample of real traffic, log tokens per clause, and price the difference from those logs.
What rubric scores interpretation coverage separately from answer correctness?
An interpretation-coverage rubric checks whether an answer surfaces each legally relevant competing interpretation and reasons validly to it. TechnoLynx’s LLM Selection Pack builds evaluation suites scored on a team’s own tasks.
Answer match alone is not enough, because correct conclusions can rest on wrong reasoning. PROSLEX shows GPT-4 predicting the right statute while citing facts that do not establish its elements. The PLawBench authors (2026) describe models that reach correct answers through flawed or invented logic. Their scoring weights reasoning above conclusions, facts and cited provisions.
Three published designs are worth borrowing from:
- Interpretation coverage as disambiguation: ASQA (2022) scores a long answer by whether a reader could use it to answer each valid interpretation of an ambiguous question. Swap “interpretation of the question” for “live reading of the statute” and you have a coverage score.
- Separate dimensions: GreekBarBench (2025) scores facts, cited articles and analysis as three separate dimensions. It found that span-based rubrics improved agreement between LLM judges and human experts.
- Locked rubrics: Rulers (2026) compiles rubrics into versioned, immutable bundles so the scoring criteria cannot drift at run time. Its authors also note that locking will faithfully preserve a badly specified rubric.
Who signs off the ground truth behind the rubric?
Qualified legal reviewers, not engineers, should sign off every version of the ground-truth interpretation set. Published benchmarks offer a template. PLawBench ran a draft-review-verify workflow, with disputes escalated to senior experts. PROSLEX had a senior legal expert adjudicate cases where annotator overlap fell below a ROUGE-L of 0.75, about 37% of cases. An LLM judge can extend coverage once it is calibrated against that set. The oab-bench authors (2025), however, found that their judge scored legal essays more generously than human examiners and needed calibration in borderline cases.
Keeping the harness current as case law shifts
The interpretation set needs an update trigger: when monitored case law changes which readings are live for a statute, the affected items are revised, re-signed and the rubric is given a new version. Label Studio recommends routing contested items to experts, recording the final decision with a short rationale, and using it to refine the rubric. A stale set penalises answers that reflect newer authority, and a slow sign-off process becomes the bottleneck. Budget reviewer time for both.
Is a cheaper model’s drafting good enough if a lawyer reviews every letter?
It can be, but only a test on your own document types will tell you. A generic legal benchmark will not. The LLM Selection Pack builds an eval suite scored on a team’s own tasks, such as visa-letter petition types, to compare a fine-tuned open-weight model with a frontier model.
VidhikDastaavej (2025) had three legal professionals rate drafts on factual accuracy and completeness. Plain supervised fine-tuning often made the open-source models worse. A structured, retrieval-based wrapper around a Gemma model scored 8.82 for factual accuracy and 7.82 for completeness, ahead of GPT-4o, at the cost of 1.4–1.6 times the inference time. On statute explanations in PROSLEX, Llama-3.1-70B received the lowest expert rating (3.16), against 3.91 for GPT-4. The gap is therefore real in some settings and closable in others, depending on the scaffolding.
The main error classes appear across models, not just weak ones. GreekBarBench found that even top models failed to identify the correct statutory articles, with rare critical fact hallucinations. A citation-checking benchmark (2026) found pincites, verbatim misquotes and content misrepresentation the hardest errors to catch. Error patterns found on one petition type may not hold for another, so test each type you plan to automate.
A review process that catches errors whichever model drafts
Human review stays in the loop whichever model you pick. Stanford RegLab (2024) found that commercial legal research tools from LexisNexis and Thomson Reuters hallucinated 17–33% of the time. Newer models are not automatically safer either: GPT-5.1 produced hallucinated legal citations at 6.57%, against 1.23% for a mid-2024 GPT-4o release. LLM scoring is not a substitute for expert judgement in high-stakes evaluation, and the GreekBarBench authors warn against deploying systems on benchmark performance alone.
Review can be made cheaper without being skipped. ABA Formal Opinion 512 says the verification needed depends on the tool and the task. It adds that a lawyer who has tested a tool’s accuracy on a manually reviewed subset need not re-check everything by hand. The lawyer stays fully responsible for the work. A practical design follows: sample-test each petition type before trusting it, then concentrate reviewer attention where the router flags low confidence or high severity. Weigh the reviewer hours against the savings from the smaller model. If review time eats the saving, the frontier model is the cheaper option for that petition type.
Who can build a confidence-gated routing pipeline for our legal AI workload?
TechnoLynx builds two-stage routing pipelines for legal workloads: a cheap first pass, calibrated escalation, and validation of recall against human-reviewed ground truth. Should you build in-house or bring in a specialist? If your team already runs calibrated evaluation and inference tuning, the gap to fill is smaller. If not, the Inference Cost-Cut Pack covers framing the pipeline, setting thresholds, implementing the routing on your hardware and batching setup, validating against human-reviewed ground truth, and monitoring for drift. When choosing a partner, ask how they will measure recall on your clauses, not on a public benchmark.
Your lawyers still own the ground-truth sign-off, the severity list and the duty to verify. FrugalGPT notes that training a cascade is an upfront cost that is beneficial only when query volume exceeds the training data, so at low clause volume a single frontier model can be cheaper.
FAQ
What is tree-of-thought prompting designed to improve compared with chain-of-thought?
Tree-of-thought lets a model explore several reasoning paths, evaluate its own intermediate steps, and look ahead or backtrack before it commits to an answer. Chain-of-thought follows a single line of reasoning.
How can you reduce LLM inference cost?
FrugalGPT (2023) groups the options into three strategies: prompt adaptation, approximating an expensive model with a cheaper one, and cascades that choose which model handles each query. Cascades need labelled examples from a distribution similar to real traffic before they work well.
