A token estimator converts the real text of your prompts and expected completions into the token counts a specific model’s tokenizer produces. That count, not the number of requests, is what a hosted API bills and what GPU compute and memory are spent on. Providers meter input and output tokens separately and bill per million tokens, with cached input charged at a lower rate where they support it. Hardware follows the same unit: prompt tokens go through a compute-bound prefill phase, and every generated token goes through a decode phase that is bound by memory bandwidth because it reads the KV cache.
So a token estimate is the input that cost forecasts, capacity plans and the hosted-versus-self-hosted decision all depend on. If you need the basics of how a model counts text first, our page on GPT token count basics covers them. This page is about what you do with the count once you have it.
How does a tokenizer split prompt and completion text into tokens, and why do estimates vary across models?
A tokenizer breaks text into subword units learned from its training corpus, so common character sequences become single tokens and rare ones split into several. Two things move the count: which tokenizer you use, and what kind of text you feed it. Tokenizers are trained with different methods on different corpora. Text made of frequent character sequences compresses better, so a general essay produces fewer tokens than a technical paper of the same length.
The differences are non-trivial. An empirical study of ten tokenizers across eight text domains found compression ratios varying by nearly a factor of ten between the lowest domain (emojis) and the highest (code and essays). Among natural-language domains, essays compressed about 25% better than technical text. The same study found Claude splitting “antidisestablishmentarianism” into 9 tokens where Gemini and Grok used 5. It also tested the familiar rule of 0.75 words per token. For Claude and Gemini the rule overestimates tokens on the most frequent words, while for DeepSeek and Mistral it underestimates them, at closer to 0.6 words per token. On randomly sampled English words, the tokenizers examined landed between 0.35 and 0.45 words per token.
Language is a second axis. Ahia et al. (EMNLP 2023) note that vendors bill by token and that the tokens needed to convey the same information vary widely across languages. Our page on average tokens per word and tokenisation ratios goes further into those ratios.
Tokenizers also change between versions of the same model family. Anthropic’s documentation says Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text, and that the exact increase depends on content. The practical consequence is that a word-count heuristic, or a count taken with a different model’s tokenizer, can be off by a non-trivial margin. Run an LLM token counter with the tokenizer of the model you will actually call. Anthropic exposes a free token-counting endpoint for this, which counts messages, system prompt and tools together.
How does AI token cost pricing work, and why are input and output tokens priced differently?
The bill for any call is input tokens times the input rate plus output tokens times the output rate, and cached input is billed at a cheaper cache-read rate where the provider offers one. On OpenAI’s current price page, for example, gpt-6.1-sol lists $2.00 per million input tokens, $10.00 per million output tokens and $0.10 per million cached input tokens. Across the market, one 2026 pricing survey by Finout found output typically priced three to eight times above input, with five times the most common ratio.
The asymmetry follows how the hardware works. Prefill handles hundreds to thousands of prompt tokens in parallel and is compute-bound. Decode generates one token at a time, and each step reads the model weights and the whole KV cache, so it is memory-bound. A generated token takes a full forward pass to itself, while prompt tokens share one parallel pass. That makes output the expensive direction to serve. The direct consequence for budgeting is that output length is the token stream to watch most closely.
Several modifiers sit on top of the base rates:
- Caching. Artificial Analysis reports cache-hit discounts of 50-90% off standard input pricing. Caching needs exact prompt matches, and some providers (OpenAI, DeepSeek) cache automatically while others (Google, Anthropic, Amazon) need manual setup. Anthropic bills cache reads at 0.1x base input on most models and 5-minute cache writes at 1.25x, so a single read pays back the write.
- Batch. Anthropic’s Batch API discounts both input and output tokens by 50% for asynchronous work, and Google lists a 50% batch reduction for Gemini.
- Context tiers. Some models price by prompt length: on one Gemini Pro tier, input is $2.00 per million tokens for prompts up to 200k tokens and $4.00 above that.
- Hidden tokens. Reasoning models bill thinking tokens that never appear in the response.
Provider-side caching discounts price the reuse of previously processed input tokens. The serving mechanism, and what it costs in GPU memory, is covered in our explanation of how RadixAttention reuses KV cache.
What are the common failure modes of estimating LLM cost from request counts instead of tokens?
A request count is a poor unit of capacity planning because request costs vary by orders of magnitude. A 10-token response is perhaps 100x cheaper to produce than a 1,000-token response on the same model, so 1,000 short requests per minute and 1,000 long requests per minute can need ten times different capacity. On the prompt side, a 4,000-token prompt needs far more prefill computation than a 100-token one, even though both count as one connection. A workload that looks cheap by request count can be dominated by a handful of long-context prompts, such as 8k-token RAG contexts arriving where the model assumed short chat turns.
The failure modes are distinct, and each shows up in a different place:
| Failure mode | What the request-count model misses | Where it shows up |
|---|---|---|
| Flat per-call price | Prompt and output lengths vary per request, so cost per request varies with them | Budget overrun once real prompts arrive |
| Blended token rate | Output-heavy traffic is understated and cache-heavy traffic overstated | Forecast too low for output-heavy traffic and too high for cache-heavy traffic |
| Ignoring retries | A retry rate rising from 4% to 9% more than doubles the extra calls with no change in user traffic | Spend rises while traffic is flat |
| Dividing by requests | True cost per answer includes retries, tool calls, context bloat and reasoning overhead | Counting retries can change which model is cheapest |
| Requests treated as utilisation | Bursty traffic leaves self-hosted GPUs idle but still billed | Self-hosting looks cheaper than it is |
We would budget against cost per workload, and ideally per successful answer, rather than against a generic cloud-spend figure. Dividing total spend by validated outputs, not initial requests, is what exposes which model is really cheaper once retries are counted.
Working with a token estimator, step by step
In practice, using a token estimator means building three token streams per month and pricing each one separately. The steps run in this order:
- Sum the input components per request. That means the system prompt, retrieved context, conversation history and user message, plus the expected answer length for output. System prompts count as input on every request.
- Count with the target model’s tokenizer. Counts for the same text differ across model generations. Anthropic notes that its counting endpoint estimates without caching logic, so cache effects are modelled in the next step.
- Split input into cached and uncached. Identify the stable, byte-identical prefix that will hit the cache and the variable remainder; that gives the cache-hit ratio.
- Multiply by monthly volume. The result is three totals: cached input, uncached input and output tokens.
- Apply each candidate’s rates and add the multipliers. Inflate for retries, tool calls and agent loops.
- Re-run against real logs after a week of traffic. The gap between forecast and actuals is the optimisation backlog.
For a per-request version of this arithmetic, our LLM cost calculator walkthrough works through single calls.
How does context length translate into KV-cache memory per concurrent request?
KV-cache memory grows linearly with the number of tokens a request holds. A survey of KV-cache management gives the per-request footprint as 2 × layers × KV heads × head dimension × sequence length × bytes per element, before sharding, fragmentation and paging overheads.
Worked example: a 70B GQA model
The survey’s example is a Llama-70B-style grouped-query-attention model with 80 layers, 8 KV heads and head dimension 128. In BF16 (2 bytes) that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 320 KiB per token, which matches the survey’s figure. At a 128K-token context that comes to roughly 40 GiB per request. At 32K it is about 10 GiB, so one hundred concurrent 32K requests approach 1 TiB of KV state. That is far beyond what a single GPU can hold alongside weights, activations, CUDA graphs and allocator slack.
Capacity is only half of it. During decode, each step re-reads the growing prefix, so cumulative KV read traffic scales as O(TS + T²) for a prompt of S tokens and T generated tokens. In the survey’s example, a 128K prompt generating 32K tokens reads on the order of 1.4-1.6 PB of KV state over its lifetime, which is why KV bandwidth can matter as much as KV capacity.
How can token estimates and a tokens-per-second target feed serving capacity and autoscaling plans?
Token estimates turn a traffic forecast into input and output tokens-per-second requirements, which token-based capacity planning uses instead of requests. One worked capacity example in a serving-architecture guide by Brenndoerfer sizes a 7B chatbot for 100 requests per second at peak, with 500-token prompts and 200-token outputs. With continuous batching, a single H100 serves about 50 concurrent decode sequences at 4.2 ms per token, giving roughly 12,000 tokens per second or about 60 completed requests per second. The plan comes to 100 / 60 ≈ 1.7 instances, rounded up to 2 active plus 1 warm spare. The request count alone would never show that.
The per-instance number has to be delivered throughput under your latency targets, not a peak figure. The gap between the two is the subject of our page on delivered vs peak GPU FLOPS.
For autoscaling, token-based signals catch load that request counts miss. Tokens per second rises when users submit longer prompts even if the request count stays flat. vLLM exports KV-cache utilisation and queue depth in tokens through its Prometheus endpoint, which gives finer control than hardware-level GPU utilisation. That matters because GPU compute utilisation can read low on a server that is saturated on memory bandwidth during decode.
Cold start sets the planning horizon. Combined VM start-up and model loading for a 70B model can exceed three minutes, which is why warm pools are kept. A predictive scaler’s forecast horizon must be at least as long as instance start-up time, with 5-10 minutes suggested as a safety margin. Predictive scaling fails under distribution shift, such as a launch or news spike, so it needs reactive policies behind it. Services with strict latency targets often set an autoscaling utilisation target around 0.65-0.70. That headroom belongs in the capacity plan from the start, not as a surprise later.
How can token-level estimates inform cluster sizing and fabric bandwidth decisions before procurement?
Once the KV footprint at your projected context length and concurrency exceeds one GPU, the model and its cache have to be split across devices, and that turns a token estimate into an interconnect requirement. Tensor parallelism splits each layer across GPUs, which exchange partial results at every layer, so per-token traffic crosses the fabric on every decode step. The physical side of that traffic is covered in our comparison of GPU cluster interconnect choices, where cable and optics choices meet the bandwidth the sharding plan demands. Prefill-to-decode handoff moves KV state that is large and latency-critical, and migrating KV state during scale-in is technically complex.
Adding GPUs is not free on cost per token either. A concurrency-aware cost study found Mixtral 8x7B at TP=4 on A100 about 1.5x more expensive per token than at TP=2. Throughput rose 1.33x while GPU cost doubled, and the authors recommend the smallest tensor-parallel degree that fits the model and the latency target, scaling out with replicas. A token-level estimate tells you which of those regimes you are in before you buy anything.
TechnoLynx works on exactly this stage for teams in AI infrastructure and production SaaS: grounding cluster topology and fabric design in token-level demand before a GPU procurement or deployment is committed.
How do token estimates change the comparison between a hosted LLM API and self-hosted serving on cloud or on-prem GPUs?
A hosted API scales with tokens processed and has no idle cost, while a rented or owned GPU costs roughly the same per hour busy or idle. The break-even, as Packet.ai sets it out, is simple: self-hosted cost per token equals the GPU hourly rate divided by tokens actually generated per hour at your real utilisation, and self-hosting wins when that falls below the API rate for a comparable model.
Worked example: utilisation decides the break-even
Packet.ai’s example assumes a GPU renting at $2 per hour that sustains 100,000 tokens per hour when fully busy, which works out to $0.02 per 1,000 tokens. At 20% utilisation the same hour produces 20,000 tokens, so the effective cost becomes $0.10 per 1,000 tokens, five times higher. Against an API at $0.05 per 1,000 tokens, the GPU only wins above roughly 40% utilisation. That threshold follows directly: $2 / $0.05 per 1,000 = 40,000 tokens per hour, which is 40% of 100,000.
Measured data makes the same point more sharply. The 2026 concurrency-aware study measured effective cost on identical H100 hardware spanning $0.21 to $15.25 per million output tokens, depending on offered request rate. It shows that any utilisation-naive estimate understates true cost by a factor of 1/U. A latency SLA effectively sets a price, because a fixed latency target caps the load a deployment can sustain.
Token mix also changes the comparison. Self-hosted serving bills input and output tokens at the same GPU-time rate, so the API’s output premium makes generation-heavy workloads such as code generation and long-form writing relatively more attractive to self-host. An academic on-premise cost-benefit analysis assumed a 2:1 output-to-input ratio. It found break-even within a few months for small models, around 2 years for medium models and 5 years for large ones, viable mainly at 50M tokens per month or more, or under strict data-residency mandates.
There is no universal break-even volume, because input/output mix, model quality, rates, engineering cost and the capacity needed to meet the service target decide it. Packet.ai suggests starting on an API to observe real request shapes, and the concurrency study recommends load-testing at your own distributions and recording goodput at your SLO. A guide to reading an LLM inference benchmark helps here, since a published tokens-per-second figure is rarely the goodput your SLO allows.
What happens to cost estimates when context length or the prompt and completion mix shifts after launch?
Cost estimates drift upward quietly, because prompts grow and retries rise while user traffic stays flat. LayerLens gives an example: average input length growing from 900 to 1,450 tokens over six months, at 7.5 million requests per month and $0.012 per 1,000 tokens, adds roughly $49,500 per month. The arithmetic checks: 550 extra tokens × 7.5 million requests is 4.125 billion tokens, or about $49,500 at that rate.
The Opus 4.7 upgrade shifted both sides of the mix at once. OpenRouter measured Opus 4.7’s new tokenizer producing 32-34% more tokens than 4.6 on prompts above 10K tokens and 42-45% more on smaller prompts. Net costs still rose only 12-27% for prompts above 2K tokens, because caching absorbed much of the inflation. Below 2K, much shorter completions offset the tokenizer overhead entirely. At the longest prompts (128K and above), 93% of the extra tokens landed in cache. A tokenizer change can therefore be mostly harmless or expensive depending on your cache-hit ratio and prompt-size distribution. Counting a sample of real prompts with the new model’s tokenizer shows the actual increase, which Anthropic says depends on content.
Context growth also crosses pricing thresholds and memory limits. A prompt that drifts past 200k tokens moves to the higher Gemini Pro tier. On self-hosted GPUs, longer resident sequences grow KV memory per request and across concurrent requests. Token-rate monitoring catches longer prompts that request-based monitoring misses. Tracking token distributions and retry rates surfaces prompt creep and instability early.
The estimate you make before launch needs re-running against real logs once traffic arrives.
FAQ
How much text is a million tokens?
At the commonly quoted 0.75 English words per token, a million tokens is about 750,000 words. Measured ratios vary by tokenizer and text type, so treat that as a rough figure. Anthropic’s examples put an average 10 kB web page at about 2,500 tokens.
Which model consumes more tokens for the same text?
It depends on the tokenizer and the domain. One 2026 study found Claude using nearly twice as many tokens as Gemini or Grok for a single long word, and Anthropic says its Claude 4.7 and later tokenizer produces about 30% more tokens than earlier models.
