A workload that looks cheap when you count requests can be dominated by a handful of long-context prompts. That is the trap in almost every first-pass LLM cost model: it treats a “prompt” as a fixed unit and multiplies request volume by a flat per-call price. The cluster gets sized around that number, procurement signs off, and then production traffic arrives carrying 8k-token RAG contexts instead of the 300-token chat turns the spreadsheet assumed. Suddenly per-GPU memory is tight, the interconnect is saturated, and the “correctly sized” cluster is anything but. A token estimator is the tool that closes that gap before it costs you a procurement cycle. It converts real prompt and completion text into token counts, because tokens — not requests — are what actually drive compute, KV-cache memory, and the inter-GPU traffic a fabric has to carry. The request is the billing unit users think in; the token is the physical unit the hardware works in. Getting the translation right is the difference between a cost model that survives contact with production and one that quietly under-provisions you. What should you know about a token estimator for LLMs in practice? At its core, a token estimator runs the same tokenizer the model uses at inference time over representative text, then reports counts. A tokenizer is a deterministic function that splits a string into sub-word units — the atomic pieces a transformer actually processes. OpenAI’s tiktoken implements the byte-pair-encoding schemes used by the GPT family; Hugging Face’s tokenizers library backs most open models; SentencePiece underlies Llama-style vocabularies. Feed each of them the sentence “GPU cost planning is underrated” and you will get slightly different counts, because each was trained on a different corpus with a different vocabulary size. In practice, the estimator is doing three jobs at once. It counts prompt tokens (everything the model reads before generating), it projects completion tokens (what the model will write, which you have to estimate from expected output length since it does not exist yet), and it sums the two into the context length that determines memory footprint. The prompt side is exact once you have the text. The completion side is a distribution, not a number — and treating it as a fixed value is one of the more common sizing mistakes we see. The reason this matters more than it sounds: a well-built estimator does not report a single average. It reports a distribution — median, tail, and the shape between them — because GPU capacity has to be sized against the tail, not the mean. This is the same discipline behind reading how tokens drive inference cost in the GPT token count model: the headline average hides the requests that actually stress the hardware. Why do token counts, not request counts, drive GPU compute, memory, and interconnect demand? Consider two workloads that a request-count model treats as identical: both handle 10,000 requests per hour. Workload A is a customer-support chatbot with ~200-token prompts and ~150-token replies. Workload B is a document-QA system pulling ~6,000-token retrieved contexts with ~400-token answers. By request count they are the same line item. By token count, Workload B moves roughly fifteen times the text through the model — and the hardware demand scales with tokens, not calls. The mechanism is worth being precise about. Transformer inference has two phases. Prefill processes the entire prompt in one forward pass; its compute cost scales with prompt length (and, for the attention step, with length squared). Decode generates completion tokens one at a time, and each new token attends back over every prior token in the sequence. The state that makes decode efficient — the keys and values for every token processed so far — is the KV cache, and it lives in GPU memory for the full duration of the request. Longer context means a bigger KV cache means less memory left for concurrency. This is the point in the CCU’s framing that deserves the sharp analogy: just as an interconnect that was fine last generation is not safe at a higher data rate, a cost model built on request counts is not safe once token distributions shift. When you shard a large model across GPUs, every decode step generates activation traffic that crosses the fabric — over NVLink inside a node, over the network between nodes. That traffic scales with tokens in flight, so a workload whose token profile grows will pressure the interconnect long before the request count changes. If you want to see where that fabric pressure becomes the binding constraint, the reasoning in how 800G ConnectX-8 interconnect matters picks up exactly where token-level sizing hands off. How does a tokenizer split text into tokens, and why do estimates vary across models? Tokenizers do not split on words or characters — they split on learned sub-word units. The word “tokenization” might be one token in one vocabulary and three (token, iza, tion) in another. Byte-pair encoding builds its vocabulary by repeatedly merging the most frequent adjacent pairs in the training corpus, so a model trained heavily on English code will tokenize def __init__ compactly and tokenize Hungarian prose into many small pieces. The practical consequence is that there is no universal token count for a given string. A prompt that is 1,000 tokens under cl100k_base might be 1,150 under a Llama tokenizer and 1,300 under an older GPT-2 vocabulary. As a planning heuristic, English prose runs roughly 1.3 tokens per word across common tokenizers (an observed range, not a benchmarked constant — code, JSON, and non-Latin scripts diverge sharply from it). If you want to reason about that ratio properly rather than lean on a rule of thumb, the mechanics are laid out in how tokenisation ratios affect inference cost, and the concrete counting behaviour in how tiktoken counts tokens. The rule that follows is simple and non-negotiable: estimate with the tokenizer of the model you will actually deploy. Estimating a Llama deployment with tiktoken will systematically mis-size it. This is a benchmark-class distinction when you name the tokenizer and version — the counts are exactly reproducible — and it collapses into guesswork the moment you don’t. How does context length translate into KV-cache memory per concurrent request? This is where token counts become dollars. The KV cache holds, for every token in a sequence, one key vector and one value vector per attention layer. Its size grows linearly with context length, linearly with the number of layers, and linearly with the number of concurrent requests you want to hold in flight. A useful mental model: KV-cache bytes per request ≈ 2 × (layers) × (context length) × (hidden dim per head × heads) × (bytes per element) The factor of 2 is keys plus values; bytes per element depends on your cache precision (2 bytes at FP16, 1 byte at FP8). The number that matters operationally is not the model weights — those are fixed and loaded once — but the KV cache, because it is what scales with your traffic and eats the memory headroom that determines how many requests a GPU can serve at once. Worked example: sizing memory headroom before procurement Assume a 70B-parameter model on a GPU with 80 GB of HBM. The weights at FP16 consume roughly 140 GB — already requiring at least two GPUs — so assume a two-GPU shard leaving on the order of 20 GB of headroom per GPU for KV cache after weights and activations. The numbers below are illustrative planning figures, not a benchmark of any specific device. Context length Approx. KV cache per request (FP16) Concurrent requests per ~20 GB headroom What the request-count model saw 512 tokens ~0.3 GB ~60 1 request 2,048 tokens ~1.3 GB ~15 1 request 8,192 tokens ~5 GB ~4 1 request 32,768 tokens ~20 GB ~1 1 request Read the last column against the others. The request-count model treats every row as one identical unit of work. The token-level view shows the 32k-context request consuming as much memory as sixty short requests — and dropping your effective concurrency by more than an order of magnitude. Size a cluster on the request count and the long-context traffic will silently exhaust memory; size it on the token distribution and you preserve the headroom you actually need. How can token-level estimates inform cluster sizing and fabric bandwidth decisions before procurement? Once you have a token distribution instead of a request count, three quantities become projectable before you buy anything. Throughput in tokens per second — the operationally relevant measure, since a GPU’s decode rate is expressed in tokens, not requests — sets how many nodes you need for a target latency. KV-cache memory per concurrent request at your tail context length sets your per-GPU memory floor and therefore your concurrency ceiling. And the activation traffic per decode step across a sharded model sets the fabric bandwidth you need, which is where NVLink-versus-Ethernet and copper-versus-optical decisions get made. Here is a diagnostic checklist for pressure-testing a cost model before it goes to procurement: Are you sizing on tokens or requests? If the model multiplies request volume by a flat price, it is blind to context length. Stop here and rebuild it. Which tokenizer did you estimate with? If it is not the deployment model’s tokenizer at the correct version, the counts are systematically off. Is completion length a point estimate or a distribution? A single average completion length hides the long-generation tail that dominates decode cost. Did you size memory against the tail context, not the median? The median tells you nothing about the request that exhausts HBM. Have you projected activation traffic per decode step against fabric bandwidth? If not, the interconnect is an unmodeled risk. Have you computed variance between estimated and actual tokens per request from a real traffic sample? That variance is your sizing error bar. Running this against real traffic gives you the ROI anchor the whole exercise exists for: the reduction in over- and under-provisioned nodes when sizing is driven by tokens rather than request counts. It is also the concrete input that turns an abstract on-premise-versus-cloud debate into line items, connecting workload shape directly to interconnect and memory costs. We treat token-level demand modelling as the ground layer beneath any GPU cluster topology and procurement decision — get the token distribution wrong and every downstream fabric and memory choice inherits the error. What are the common failure modes of estimating LLM cost from request counts instead of tokens? The failures cluster into a few recognisable patterns, and each one has an early warning sign you can catch before it becomes a re-procurement. The first is the flat-per-call assumption. A cost model that prices every request identically is silently betting that your token distribution never shifts. It does — the day a RAG feature ships, or a customer starts pasting whole documents into the prompt. The warning sign is a growing gap between billed request volume and measured GPU utilisation; if utilisation climbs while request counts stay flat, tokens per request are rising and your model can’t see it. The second is estimating with the wrong tokenizer, which produces a consistent bias rather than random noise — always under- or always over-counting, depending on the mismatch. The third is treating completion length as fixed, which under-counts decode cost precisely for the reasoning and long-form workloads where decode dominates. And the fourth, the most expensive, is sizing memory against the average context length, which works until the first burst of long-context requests arrives simultaneously and exhausts the KV cache — at which point you are dropping requests or paging, not serving. None of these are exotic. They are what happens when a request is mistaken for a fixed unit of work. The correction is the same in every case: measure the token distribution of real traffic, estimate with the deployment tokenizer, and size against the tail. FAQ What does working with a token estimator for LLMs involve in practice? It runs the model’s tokenizer over representative prompt text to get exact prompt-token counts, projects completion tokens from expected output length, and reports the sum as a context-length distribution. In practice the useful output is not a single average but the median and tail, because GPU memory and throughput must be sized against the tail of the distribution, not its mean. Why do token counts, not request counts, drive GPU compute, memory, and interconnect demand? Transformer inference cost scales with tokens: prefill compute grows with prompt length, decode generates one token at a time attending over the whole sequence, and the KV cache that holds that state grows with context length. Two workloads with identical request counts can differ by an order of magnitude in tokens, and the hardware follows the tokens — including the activation traffic that crosses the fabric on every decode step. How does a tokenizer split prompt and completion text into tokens, and why do estimates vary across models? A tokenizer splits text into learned sub-word units — byte-pair-encoding merges in tiktoken, SentencePiece pieces in Llama-style models — so the same string yields different counts under different vocabularies. English prose runs roughly 1.3 tokens per word as a planning heuristic (an observed range, not a constant), but code and non-Latin scripts diverge sharply, so you must estimate with the tokenizer of the model you will actually deploy. How does context length translate into KV-cache memory per concurrent request? The KV cache stores a key and value vector for every token, at every attention layer, so its size grows linearly with context length, layer count, and concurrent requests. Because model weights are loaded once but the KV cache scales with traffic, it is the cache — not the weights — that sets how many concurrent requests a GPU can serve, which is why long-context requests collapse effective concurrency. How can token-level estimates inform cluster sizing and fabric bandwidth decisions before procurement? A token distribution makes three quantities projectable before purchase: throughput in tokens per second (setting node count), KV-cache memory at the tail context (setting the per-GPU concurrency ceiling), and activation traffic per decode step (setting fabric bandwidth). Together they turn an abstract sizing debate into concrete interconnect and memory line items and reduce the number of over- or under-provisioned nodes. What are the common failure modes of estimating LLM cost from request counts instead of tokens? The recurring failures are pricing every request at a flat rate, estimating with a tokenizer that doesn’t match the deployment model, treating completion length as fixed, and sizing memory against the average rather than the tail context length. Each is a case of mistaking the request for a fixed unit of work, and each is caught by measuring the token distribution of real traffic and sizing against its tail. A cost model is only as safe as the token distribution it assumes. If you are still sizing an LLM deployment from request counts — the failure class this whole piece is about — the next question to answer before any hardware order is committed is which requests sit in your tail, and what their context length does to KV-cache memory and fabric traffic when they all arrive at once.
