Send every query to your largest model and you pay premium GPU-hours for prompts a 7B model would have answered correctly. RouteLLM’s premise is that most traffic does not need the flagship model — it needs a classifier that decides, per query, whether the strong model is worth the spend. Get that decision right and you shift the majority of requests to a cheaper model without users noticing. Get it wrong and you either degrade answer quality or save nothing at all.

That single decision — the routing threshold — is where most RouteLLM deployments live or die. It is not a hyperparameter you can guess from a blog post. It is a value you calibrate against your own traffic, because the saving you realise depends entirely on how your query difficulty is distributed, not on the router’s cleverness.

What does RouteLLM actually do?

RouteLLM sits in front of two (or more) models: a strong, expensive one and a weak, cheap one. For each incoming query, a small router model predicts whether the cheap model can answer it about as well as the expensive model would. If yes, the query goes to the cheap path. If no, it escalates to the strong model.

The router itself is lightweight — often a fine-tuned classifier or a matrix-factorisation model trained on preference data such as pairwise human judgments over which model gave the better answer. Its job is not to answer the query; it is to estimate the win probability of the weak model against the strong one for this specific prompt. The original RouteLLM work from LMSYS trained several such routers on human-preference data and released them as open reference implementations, so the mechanism is inspectable rather than proprietary.

The routing decision reduces to a comparison. The router outputs a score in [0, 1] — roughly, the probability that routing to the strong model is warranted. You compare that score to a threshold. Below the threshold, the query is “easy enough” and goes to the cheap model; above it, the query escalates. Everything interesting about cost and quality is encoded in where you put that line.

This is a narrower concept than agentic tool-routing or mixture-of-experts gating, both of which people sometimes conflate with RouteLLM. Mixture-of-experts routes within a single model’s forward pass; RouteLLM routes between separately served models. The distinction matters because the cost lever is different: with RouteLLM you are choosing which GPU deployment answers, and cheap-model traffic can run on entirely different — smaller, or fewer — accelerators.

Why the naive “always use the best model” approach wastes money

The intuition behind routing every query to the flagship model is defensible: it guarantees the best possible answer every time. The problem is that most production traffic is not hard. A large fraction of real queries — greetings, format conversions, simple factual lookups, short rewrites — are answered identically by a 7B model and a frontier model. Paying frontier GPU-hours for those is pure waste.

The economics compound because inference cost scales with model size and with token count. A frontier model burns more GPU memory bandwidth per token, needs more accelerators to serve at the same latency, and often produces longer, more elaborate answers. If you are trying to model this from first principles before deploying, the interaction between routing and per-request cost runs straight through tokenisation — how tokenisation ratios affect inference cost is the same lever seen from the token side rather than the model-selection side.

RouteLLM-style routing typically shifts 50–85% of queries to the cheaper model while preserving the bulk of strong-model answer quality (observed pattern across published RouteLLM evaluations and comparable routing deployments; the exact split is a function of your traffic, not a benchmarked constant). That shift translates to a comparable reduction in per-query inference cost — but only if your traffic actually contains that many easy queries. A workload of uniformly hard reasoning prompts will not see 85% savings no matter how good the router is, because there is nothing cheap to route to.

What determines the routing threshold, and how do you set it?

The threshold is a business decision dressed up as a number. Lower it and more queries escalate to the strong model — quality rises, cost rises. Raise it and more queries stay cheap — cost falls, quality risk rises. There is no universally correct value; there is only the value that matches your tolerance for quality regression on your traffic.

To set it honestly you sweep the threshold across a held-out sample of your own queries and measure two curves: the fraction of traffic routed cheap (the cost proxy) and the answer-quality gap between routed and always-strong (the quality cost). The threshold you want is the point where the quality gap is still inside your acceptance band and the cheap-routing fraction is as high as it will go.

Threshold-setting rubric

Step	What you measure	Why it matters
1. Sample real traffic	500–5,000 representative queries from request logs	The distribution, not a benchmark, drives your saving
2. Get reference answers	Strong-model answer for every sampled query	This is your quality ceiling
3. Score both paths	Weak-model answer + a quality judge (human or LLM-as-judge)	Tells you which queries the weak model already handles
4. Sweep the threshold	Cheap-routing fraction and quality gap at each threshold	Produces the cost-vs-quality trade-off curve for your data
5. Pick the knee	Highest cheap fraction inside your quality band	This is the threshold, not a value from a paper
6. Re-measure on live traffic	Actual routing split and escalation rate post-deploy	Traffic drifts; the threshold is not set-and-forget

The step people skip is the first one. A threshold copied from a published evaluation is calibrated to that evaluation’s query mix (often a chat benchmark like MT-Bench or a preference dataset), which almost certainly does not match a domain-specific support queue or a code-assistant workload. This is the same discipline as any inference-cost decision: you cannot optimise spend without profiling the actual workload first, which is exactly why our GPU performance and cost work starts from your request logs rather than a spec sheet.

What profiling data do you need before deploying a router?

Three things, all extractable from request logs and light instrumentation:

First, the query-difficulty distribution — what fraction of your traffic is genuinely easy versus genuinely hard. Without this you cannot estimate the ceiling on your savings. A sampled quality comparison between weak and strong models on real prompts gives you this directly.

Second, the cost per request on each path — GPU-seconds, or tokens times a cost-per-token figure, for both the cheap and expensive deployments. Modelling this ahead of deployment is straightforward with a token calculator for estimating inference cost before you deploy, which turns a routing split into a monthly spend delta.

Third, the sustained utilisation profile of each deployment — not peak, but the load your GPUs actually sustain. This is what determines whether routing lets you shrink or consolidate the expensive deployment, which is where the real infrastructure saving lands. Turning that raw telemetry into a decision is a job for ML observability tooling that converts utilisation data into cloud decisions rather than a one-off dashboard glance.

How does routing change the cloud-vs-on-premise break-even?

Model routing lowers the sustained-utilisation cost that drives the cloud-versus-on-premise break-even point. The expensive model, after routing, serves a smaller and burstier slice of traffic. That changes the calculus in a specific way: the strong model becomes a candidate for pay-per-use cloud endpoints (it runs less often, so reserved capacity is wasteful), while the cheap model — now carrying the steady bulk of requests at high, predictable utilisation — becomes the strong candidate for owned or reserved hardware.

In other words, routing does not just cut a bill; it re-shapes the workload into two profiles with different optimal homes. A blended workload that looked marginal for on-premise can split into a steady cheap-model tier that clearly justifies owned accelerators and a spiky strong-model tier that clearly belongs on elastic cloud. The break-even you should compute is per-tier, not for the blended whole. The underlying model choices that make the cheap tier cheap — quantisation, attention-cache reuse, decoding strategy — are worth understanding too; how DeepSeek inference works and the algorithmic choices that drive GPU cost walks through several of them.

When is routing not worth deploying?

Routing has real overhead: a router model to serve, a second deployment to operate, and a calibration process to maintain as traffic drifts. It pays for itself only when the saved GPU-hours exceed that overhead. Several workload shapes fail that test.

Uniformly hard traffic is the clearest case — if almost every query genuinely needs the strong model, there is nothing to route away and the router only adds latency and operational surface. Very low-volume workloads are another: the absolute GPU-hours saved may be smaller than the cost of running and maintaining the router path. And latency-critical single-token-stream applications can find the router’s own inference time eats into a tight budget, though a well-built router adds only milliseconds.

There is also a subtler failure: when the “cheap” and “strong” models are close enough in capability that the weak model’s answers are frequently almost good enough. In that regime the router is forced to make many low-confidence decisions, quality regressions creep in unpredictably, and the saving is not worth the risk. Routing works best with a clear capability gap between tiers, not a marginal one.

FAQ

How does RouteLLM work?

RouteLLM places a lightweight router model in front of a strong (expensive) and a weak (cheap) model. For each query it predicts whether the cheap model can answer about as well as the strong one, and routes accordingly. In practice it means most easy queries never touch your flagship GPU deployment, while hard queries still escalate — cutting cost without a visible quality drop, provided the threshold is calibrated to your traffic.

What determines the routing threshold between a strong and a weak model, and how do I set it from my own traffic?

The threshold is the score above which a query escalates to the strong model, and it encodes your tolerance for quality regression. Set it by sampling real queries, scoring both the weak and strong answers, sweeping the threshold, and picking the point where the cheap-routing fraction is highest while the quality gap stays inside your acceptance band. A threshold copied from a published evaluation is calibrated to that evaluation’s query mix, not yours.

How much inference cost can query routing realistically save without degrading answer quality?

Routing typically shifts 50–85% of queries to the cheaper model while preserving the bulk of strong-model quality, translating to comparable cost reductions — but this is an observed pattern, not a guaranteed rate. The realised saving depends on your query-difficulty distribution, which is measurable from request logs. A workload of uniformly hard queries will save little no matter how good the router is.

What profiling data do I need before deploying a RouteLLM-style router?

You need three things from your request logs and light instrumentation: the query-difficulty distribution (what fraction of traffic is genuinely easy), the cost per request on each path (GPU-seconds or tokens times cost-per-token), and the sustained utilisation profile of each deployment. Without the difficulty distribution you cannot estimate your savings ceiling; without utilisation data you cannot tell whether routing lets you shrink the expensive deployment.

How does model routing change the cloud-vs-on-premise GPU cost break-even point?

Routing splits a blended workload into two profiles: a steady, high-utilisation cheap-model tier and a burstier strong-model tier. The cheap tier’s predictable load favours owned or reserved hardware, while the strong tier’s infrequent use favours elastic pay-per-use cloud. The break-even should be computed per tier rather than for the blended whole, which often makes on-premise viable for the cheap tier where it was marginal before.

When is routing not worth it — which workloads see little or no benefit?

Uniformly hard traffic sees little benefit because there is nothing to route away; very low-volume workloads may save fewer GPU-hours than the router path costs to operate; and latency-critical single-stream applications can be hurt by the router’s own inference time. Routing also struggles when the cheap and strong models are close in capability, since the router is forced into many low-confidence decisions that leak quality regressions.

The question worth carrying out of this is not “should I use RouteLLM” but “what does my query-difficulty distribution actually look like” — because that distribution, measured from your own logs, is what decides whether routing is a large saving, a marginal one, or a net loss. That measurement is exactly the workload profiling a GPU performance audit exists to produce, and it is the input a routing threshold cannot be set without.