The cost of one LLM request is simple to state: average input tokens times the input price, plus average output tokens times the output price. Real LLM costs drift away from that line through the modifiers: cached prompt prefixes, batch discounts, reasoning tokens billed as output, tokenizers that count the same text differently, and, for self-hosted models, how busy the GPU actually is. The figure worth defending is the one measured on your own traffic, because a model that looks cheaper on paper can cost several times more once you see how it counts tokens.
This arithmetic applies to teams running models inside a product, the focus of AI for AI infrastructure and production SaaS.
How to calculate LLM costs per request and per month
Commercial APIs price per million tokens and charge substantially more for output than input. A 2026 arXiv study of concurrency-aware cost estimation lists GPT-5.5 at $5.00 input and $30.00 output per million, Claude Sonnet 4.6 at $3.00 and $15.00, and Gemini 3.1 Pro at $2.00 and $12.00, so output runs 5–6x the input rate. That asymmetry means the shape of your traffic changes what you pay. The same study gives an illustrative code-generation pipeline with 100-token inputs and 500-token outputs, which at GPT-5.5 list pricing comes to roughly $25.80 per million tokens in aggregate, far above the $5.00 input rate that tends to get quoted.
Worked example: one extraction workload on two models
Nadir’s 2026 cost model for invoice extraction assumes 1,000,000 calls a month at 1,200 input and 150 output tokens per call.
| Model (in / out per 1M) | Input cost per call | Output cost per call | Cost per call | Monthly at 1M calls |
|---|---|---|---|---|
| Haiku 4.5 ($1 / $5) | 1,200 × $1 ÷ 1M = $0.0012 | 150 × $5 ÷ 1M = $0.00075 | $0.00195 | $1,950 |
| Opus 5 ($5 / $25) | 1,200 × $5 ÷ 1M = $0.006 | 150 × $25 ÷ 1M = $0.00375 | $0.00975 | $9,750 |
The per-call figures and the monthly totals match the ones Nadir reports. The monthly figure is just per-call cost times volume, so the bill is only as good as the volume forecast. SoftwareSeni advises a 3x buffer when projecting monthly token volume, because test usage rarely predicts production. If the number has to stand up to finance alongside placement options, the guide to building a defensible cost-per-request model for sovereign cloud versus self-hosting covers the placement side this page leaves out.
Input, output, cached and reasoning tokens: which ones drive your bill
The table lists the modifiers that move a real bill away from the base formula, with the figure each source gives.
| Modifier | Effect on cost | Figure and source |
|---|---|---|
| Output tokens | Priced above input at major APIs | 5–6x input at GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro (2026 arXiv study) |
| Cache reads | Repeated prefixes billed at a fraction of input | 0.025–0.1x base input at Anthropic; 0.05–0.1x at OpenAI (vendor docs) |
| Cache writes | First storage costs more than plain input | 1.25x for a 5-minute write and 2x for 1 hour (Anthropic); 1.25x for 30 minutes and 2.5x for 12 hours on GPT-5.6 and later (OpenAI) |
| Batch processing | Asynchronous requests discounted | 50% off input and output at Anthropic; 50% at OpenAI with a 24-hour turnaround |
| Reasoning tokens | Billed as ordinary output, so volume rises | Can reach 10–30x the output tokens of a non-reasoning answer (AIMultiple) |
| Tokenizer | Same text, different token counts | 2.65x or more depending on the model (TensorZero) |
| Data residency | Regional processing uplift | 10% on OpenAI regional endpoints for models released on or after 5 March 2026; 1.1x for US-only inference on Claude 4.6 and later |
| Long prompts | Higher tier above a length threshold | Gemini 2.5 Pro input $1.25 up to 200k tokens, $2.50 above |
Caching pays only when later requests share the same prompt prefix. OpenAI’s documentation works through the maths: at a 0.1x read rate with 30-minute writes, one write and nine full reads across ten requests cost 2.15x the ordinary input cost of the prefix, against 10x without caching. There are conditions, though. On GPT-5.6 and later the minimum cacheable prompt is 1,024 tokens. Cached state lives on individual machines, and traffic above 15 requests per minute can be routed to machines that do not hold the entry.
Tokenizers also make list prices misleading. TensorZero found that on tool-heavy workloads claude-opus-4-7 cost 5.3x more than gpt-5.4 even though their list prices are only 2x apart, and that Gemini, the cheapest on text and structured data, became 46% more expensive than OpenAI on tool definitions. Anthropic says the tokenizer in Claude 4.7 and later produces about 30% more tokens for the same text, depending on the workload. Artificial Analysis blends at a 7:2:1 ratio of cache-hit, input and output tokens, which is a sensible default and probably not your ratio. Its cost-per-task measure is closer to what you will pay, because models that write longer answers or more reasoning tokens cost more per task even at identical per-token prices.
Which LLM API is the best value once accuracy on your task counts?
The best-value API is the one with the lowest cost per accepted output on your own task, and for some tasks the answer is not an LLM at all. In a 2026 arXiv study using pricing snapshots from 22 January 2026, fine-tuned encoders cost $12.44 (DistilBERT) and $32.98 (RoBERTa) per million classification requests, against $842.78 for GPT-4o and $1,174.95 for Claude 4.5 used zero-shot. On one of the study’s four benchmarks, few-shot prompting roughly doubled Claude’s input tokens for about 2.6 points of macro-F1, raising estimated cost from $326.67 to $599.67 per million requests. For fixed-label work, comparing LLM price sheets misses the cheaper architecture entirely.
Spending more does not reliably buy quality either. EvalEval’s 2026 review on Hugging Face cites a 9x cost difference for a two-point accuracy gap on Online Mind2Web, and the CLEAR study finding that accuracy-optimal agent configurations cost 4.4–10.8x more than Pareto-efficient alternatives with comparable real-world performance. So rank candidates on a cost and quality frontier measured on your own task, never on accuracy alone or price alone.
When does self-hosting an open-weight model become cheaper than per-token APIs?
Self-hosting cost depends on whether your offered load keeps the GPU busy at the latency you have committed to. At low offered load, effective cost per token climbs steeply. The 2026 concurrency-aware cost study measured effective cost on identical H100 hardware ranging from $0.21 to $15.25 per million output tokens across 42 vLLM benchmark runs. It found an underutilisation penalty of 2.5–24x at enterprise loads of 1–10 requests per second, and up to 36.3x near idle. Any calculator that assumes full utilisation understates true cost by a factor of 1/U, and the study says the error is worst at low traffic, which is where it most overstates the case for self-hosting.
The study treats a latency SLA as a price. Fixing a tail-latency target caps the load a deployment can absorb, which pins it to a higher cost per token than the saturation floor. In the study’s live workload, a Mixtral-8x7B FP16 deployment on two H100s reached at best $8.23 per million tokens against a theoretical floor of $0.87. Total model size is also a poor guide to serving cost. At peak load, Qwen3-30B-A3B FP8 came out cheaper per token than Llama 3.1 8B FP8, which the authors attribute to active parameter count and memory-access pattern. Self-hosting does have one structural edge: GPU time bills input and output tokens at the same rate, so generation-heavy workloads escape the API output premium.
The study’s workflow has three steps. Deploy the candidate configuration. Load-test it with your real prompt and output length distributions under your SLO, and record the highest arrival rate that still meets the SLO. Then compute effective cost from that goodput and compare it with API pricing. If the result shows a deployment deep in the cost cliff, the study’s remedies are provisioning changes: consolidate tenants, scale replicas to load, coalesce requests, or fall back to a serverless endpoint below the crossover. A 2025 arXiv cost-benefit framework takes a different route to the same question, estimating the on-premise break-even point against commercial services from usage levels and performance needs. Teams weighing a move to their own hardware can read about validating cost and quality before switching to a local fine-tuned LLM, which covers the quality side of that switch.
Hidden costs that belong in an LLM cost comparison
AIMultiple estimates that costs beyond API usage, such as embeddings and vector databases, reranking models, local caching infrastructure, logging, monitoring and audits, often make up 20–40% of total LLM operational expenses. Tools add their own line items. OpenAI charges $10.00 per 1,000 web search calls on top of token costs. Anthropic’s computer-use toolset adds about 4,500 input tokens to every request that declares it.
Retries and malformed outputs cost twice: once per request and once per task type. In Nadir’s extraction model, a cascade that sends every failed Haiku output to Opus 5 costs $3,900 a month, and half of that is escalation. Separating format failures from content failures, with local repair and a draft-conditioned retry, cuts the bill to $2,941.35. The quieter cost is in evaluation. If the scorer counts unparseable answers as wrong, it measures format compliance times accuracy, and in Nadir’s example that sends a whole task type to a model five times the price. Over time, SoftwareSeni suggests budgeting 15–25% of inference compute cost each year for responding to model drift, along with retraining and migration costs that year-one projections leave out.
Are LLM costs going down, and should falling prices change a vendor commitment?
Yes, prices are falling fast, but unevenly, and that does not make lock-in free. Epoch AI found that the lowest price for reaching a fixed benchmark performance fell by between 9x and 900x per year depending on the task, with a median of 50x. Using only post-January 2024 data, the median rose to 200x per year. That measure tracks the cheapest model at each capability level over time, not the price of the model you have already integrated. AvePoint’s 2026 analysis puts the risk on the other side plainly: an organisation with no alternative has no negotiating position, and if its vendor raises the flagship price, it pays or rebuilds.
Our reading is that falling prices argue for shorter commitments and an integration you can repoint, because the cheaper option next year may come from a different provider. Switching cost, according to SoftwareSeni, compounds with every month of deployment as proprietary formats, agent memory and API-specific integrations build up.
How much does it cost to evaluate candidate models before switching?
The cost of an LLM evaluation run is per-call price times the number of calls, and calls scale with test cases × vendors × configurations × reruns. To cap that spend, estimate it before the run, sample instead of running every case against every vendor, and have the harness enforce a hard ceiling.
LynxBenchAI publishes results showing how each device runs the public PyTorch stack that self-hosted candidates use, so where the comparison includes them, start by checking those results to narrow the hardware side before evaluation spend; it does not run your eval suite.
The amounts are not trivial. One HELM evaluation can cost $9,337 in API credits, according to Li et al. (2024), cited in a 2026 arXiv paper on efficient evaluation. Reruns multiply it: EvalEval estimates that eight reruns per cell takes the Holistic Agent Leaderboard’s roughly $40,000 sweep to about $320,000. On the other side, sampling with statistical guarantees cuts the number of test points needed. Cer-Eval saved 20–40% of test points at a 95% confidence guarantee, and reached the same accuracy on MMLU, AlpacaEval and MATH for GPT-4o with 60–80% of the test data. Factorized Active Querying matched the confidence-interval width of uniform sampling with up to 5x fewer queries. OpenAI lists running evaluations as a Batch API use case, at a 50% discount for runs that can wait up to 24 hours.
Capping evaluation spend, step by step
- Estimate per-call cost from each vendor’s price sheet and the measured token lengths of a small pilot batch, not guessed lengths.
- Set the confidence level and error margin the decision actually needs, for example estimating a score within a set error at 95% confidence.
- Draw a stratified or adaptive sample sized to that target, and keep the full suite for high-stakes or regulated decisions.
- Run through the batch endpoint where latency does not matter.
- Have the harness track cumulative spend and stop or down-sample at a cap that includes a buffer for retries and price changes.
The caveats are real. A sample estimates the score within a confidence interval, not with certainty. A cap that halts mid-run can leave one vendor under-sampled, so interleave vendors call by call rather than in sequence. Check that the scorer separates format failures from wrong answers before trusting the ranking.
Should we commit to one frontier LLM provider or route across several?
Single-vendor standardisation is the right call when volume is moderate, workloads are similar, and the cost of a future migration is acceptable. Multi-model routing earns its place when volume is high, tasks differ in what they need, or pricing and outage risk cannot be tolerated, and only if the routing overhead costs less than it saves.
Gateway providers and RouteLLM benchmarks report API cost reductions of 40–85% while keeping 95–98% of frontier-model quality. The overhead is real too. Requesty’s April 2026 benchmark measured gateway overhead from about 16 ms to about 124 ms per request, depending on architecture. Lost cache reuse adds a further cost, because a KV cache built by one model cannot be reused by another. In one measured 20-turn session, per-request gateway routing captured only 13 of the 40 percentage points of savings that the price difference promised, while phase-boundary routing captured 29. A cache-blind policy on a frontier model cost 4.88x the cache-aware baseline. The same author cites a smaller 2.1–2.4x penalty reported in production.
In Nadir’s model, the simpler cascade costs only about $10 more a month at 10,000 calls. Sticking with one vendor leaves no negotiating position if that vendor raises prices. It carries technical risk too: provider-side model updates can change behaviour, and pricing can move independently of your system design. AvePoint’s mitigation is an abstraction layer between application code and the model API, so that a vendor swap becomes a configuration change rather than a rebuild. SoftwareSeni’s switching-cost audit measures how deep the lock-in already goes: map every call to the vendor’s API, its proprietary formats and embeddings, the agent behaviour held in vendor memory, and the migration effort for each component. A middle path is to stay on one vendor with a portable integration and route only at workflow phase boundaries, which keeps most of the cache value. Whichever you choose, a multi-model setup needs ongoing evaluation as models and prices shift, or yesterday’s routing decisions go stale.
Does a cheaper open-weight model still come out ahead once extra manual QC is counted?
The answer comes from a pilot on a single matter, not from the per-token price. Track the all-in cost per GB, meaning inference cost plus the manual re-review that any accuracy drop triggers, against the client’s agreed cap before switching firm-wide.
TechnoLynx’s Inference Cost-Cut Pack works on the inference side of that sum: it profiles and tunes the serving path of the candidate model before anyone concludes a different model is needed, so the pilot’s per-GB inference figure reflects an optimised deployment rather than a default one.
The arithmetic is simple: all-in cost per GB equals inference cost plus re-review hours times reviewer cost, divided by GB processed. If that figure stays under the cap and the quality regression is acceptable, roll out wider. If the QC buffer eats the saving, stay on the current model or narrow the scope. Two caveats limit what one pilot can tell you. A single matter is one sample, not the full range of matter types and volumes. And the QC buffer is only as good as the measured precision and recall gap against the current tool.
What exposure do we carry if a smaller model misses a privileged document?
Professional-responsibility exposure, outside ethics counsel sign-off, and any duty to tell the client which model reviewed their documents are legal questions. They depend on jurisdiction and engagement terms, and the firm’s counsel has to answer them. Engineering can supply the evidence those people will ask for: recall and precision on privilege calls, measured on document sets representative of the matter, compared with the firm’s current review standard and documented well enough to reproduce. If the smaller model is materially worse, or has not been validated, it should not be used for privilege review on the strength of its cost saving. Sign-off on validated results also does not guarantee real-world performance will match, which is one more reason to keep monitoring after deployment.
RAG vs fine-tuning for headline and summary generation: which gives more quality per pound?
Run both at small scale on the same held-out set of headline and summary tasks from your archive, and compare cost per unit of quality gained, counting data cleaning, evaluation labour and iteration as well as GPU time. With around 18 months of articles, RAG over a base model or a single-vertical fine-tuning pilot is the lower-risk start. Full fine-tuning is justified once the pilot shows a quality ceiling that RAG cannot reach.
TechnoLynx’s LLM Selection Pack starts by framing that comparison as a controlled experiment on your own archive. From there it sets up data cleaning and the headline and summary metrics, builds the RAG pipeline and a fine-tuning pilot on the target hardware, measures cost and quality for each, and recommends the path with the best trade-off.
Whether 18 months is enough to fine-tune depends less on volume than on noise, duplicates and style drift across that period. Those can quietly invalidate any estimate of fine-tuning benefit. Budget the hidden costs of each route (curation, evaluation labour, reruns) explicitly rather than inside a contingency. For the GPU side, LynxBenchAI’s published results show how devices run the public PyTorch stack, which makes them broadly indicative for a custom fine-tuning job, but they do not measure your pipeline. These conclusions apply to editorial headline and summary generation, and should not be stretched to other tasks.
Who can help us compare LLM cost and quality before we switch?
TechnoLynx compares LLM options on cost and quality as controlled experiments on a team’s own data. Building this in-house makes sense when you already have an evaluation harness and someone who can load-test a serving stack. Bringing in a specialist is worth it when the decision is large, time-boxed, or has to stand up to finance or legal review. When choosing a partner, ask whether they measure on your workload or quote list prices, whether they price the cost of switching, and whether they will tell you to keep your current model when the numbers say so. The engagement is the LLM Selection Pack, and it starts by framing the comparison as a controlled experiment on your own data.
The engagement is a poor fit for a team that only needs a price sheet compared, or whose volume is low enough that the cheapest simple option is already the right answer. A spreadsheet with the formulas above does that job. Legal sign-off, procurement negotiation and cloud-bill FinOps are outside it, so firm counsel, a procurement lead or the cloud provider’s account team are the right routes for those. The numbers that matter in any route are the cost per accepted output and the switching cost, measured on your own traffic. If those two are missing, the comparison is not finished.
