A team sizes an LLM deployment by counting words in a typical prompt, multiplying by a per-token price, and calling the number a budget. It survives the demo and breaks the moment real traffic arrives, because a token is not a word. The count a model actually processes — and pays GPU compute and memory for — is set by a tokenizer that splits text into subword units, and those units rarely map one-to-one onto the words you counted.

That gap is not a rounding detail. It is the difference between a capacity plan that holds and one that quietly runs over on context length, KV-cache footprint, and per-request cost the first week of production. If you want to know how many GPUs a given request volume needs, you have to reason in tokens, not words.

How does GPT token count work in practice?

A GPT-style model never sees your text as characters or words. Before anything reaches the transformer, a tokenizer maps the input string to a sequence of integer IDs drawn from a fixed vocabulary — typically on the order of tens of thousands to a few hundred thousand entries. Each ID is a token. The model’s cost, latency, and memory are all functions of how many of these tokens flow through it, not how many words a human would count.

Modern GPT tokenizers use byte-pair encoding (BPE) or a close variant. BPE builds its vocabulary by repeatedly merging the most frequent adjacent byte pairs in a training corpus, so common words often become a single token while rare words, code identifiers, and non-English strings fragment into several. The word “tokenization” might be one token in a large vocabulary or three in a smaller one. A snake_case variable name, a UUID, or a stretch of Cyrillic text can explode into many tokens each. Whitespace and punctuation carry tokens too, which is why formatting-heavy prompts cost more than their word count suggests.

The practical consequence is that token count is a property of the (text, tokenizer) pair, not of the text alone. OpenAI’s tiktoken, the SentencePiece tokenizers used by many open models, and the Hugging Face tokenizers library will each return a different count for the same input. If you are modelling cost or memory, you count tokens with the exact tokenizer your served model uses — measuring with the wrong one reintroduces the error you were trying to eliminate. We walk through the mechanics of one such counter in how tiktoken counts tokens for inference cost modelling.

Why a token is not a word — and how much the count inflates

The intuition that “one word is roughly one token” is close enough to feel safe and wrong enough to break budgets. For ordinary English prose, the tokens-per-word ratio typically lands around 1.3 to 1.5 — that is, 1,000 words of English text usually becomes something like 1,300 to 1,500 tokens (observed pattern across common GPT tokenizers on English text; not a benchmarked rate for your corpus). The multiplier is not universal:

Content type	Typical tokens-per-word range	Why
English prose	~1.3–1.5×	Common words are single tokens; suffixes and punctuation add fractions
Source code	~1.5–2.5×+	Identifiers, indentation, and symbols fragment heavily
Non-English (Latin script)	~1.5–2×+	Under-represented in vocabulary, more subword splits
Non-English (non-Latin script)	often 2–4×+	Byte-level fallback splits characters into multiple tokens

(Ranges are observed patterns across widely used GPT-family BPE tokenizers; treat them as planning heuristics and measure your own traffic before committing capacity.)

The pattern that trips teams is not the average — it is the variance. A cost model built on the English mean silently understates any traffic that is code-heavy, multilingual, or full of structured data. A support product that markets in one language and gets used in five will see its real token bill diverge from the plan by a factor that no amount of word-counting predicts. If your traffic mix is uncertain, the average tokens-per-word figure is the wrong anchor; the tokenisation ratios that actually affect inference cost explain why the tail matters more than the mean.

How prompt and completion tokens differ on the GPU

Every request has two token populations, and they load the serving GPU differently. Understanding the split is where token accounting stops being a billing exercise and becomes a hardware-sizing exercise.

Prompt (input) tokens are processed in the prefill phase. The model runs a forward pass over the entire prompt in parallel, which is compute-bound: it saturates the GPU’s matrix-multiply units and, per token, is relatively cheap in wall-clock time because the whole sequence goes through together. Prefill is where a long system prompt or a large retrieved context hits you — not in latency per token, but in the compute spike and the memory it leaves behind.

Completion (output) tokens are generated in the decode phase, one token at a time. Each new token requires a forward pass that attends to every previous token, so decode is memory-bandwidth-bound and inherently sequential. This is the phase that dominates user-perceived latency for long responses, and it is why generating 500 output tokens costs far more wall-clock time than reading a 500-token prompt.

Both populations share one expensive resource: the KV cache. Attention caches the key and value tensors for every token already in the sequence so they do not have to be recomputed on each decode step. The cache grows linearly with total context length — prompt plus completion so far — and it lives in GPU memory (HBM) for the entire life of the request. This is the mechanism behind runtimes such as vLLM’s PagedAttention and the prefix-reuse schemes we cover in RadixAttention and KV-cache reuse on GPU. The cache, not the model weights, is usually what caps how many requests a GPU can serve concurrently.

How do you turn a token count into cost and a memory footprint?

Two calculations matter, and both need actual token counts rather than word estimates.

Cost per request. Serving cost tracks tokens directly. If you rent an API, prompt and completion tokens are priced separately (completion is usually the more expensive of the two). If you self-host, the cost is your GPU-hour rate divided by the tokens-per-second the deployment sustains. Either way, the input is a token count — so a word-based estimate propagates its 1.3–1.5×-plus error straight into the dollar figure.

KV-cache memory per request. This is the sizing number people skip. A useful worked estimate:

KV-cache bytes per token ≈ 2 (key and value) × number of layers × hidden dimension × bytes per element.

For example, take a model with 32 layers, a hidden dimension of 4,096, storing the cache in FP16 (2 bytes). That is 2 × 32 × 4,096 × 2 ≈ 524 KB per token. A single request holding an 8,000-token context therefore pins roughly 4 GB of GPU memory in the KV cache alone — before you count model weights or activation buffers.

(Illustrative arithmetic on assumed model dimensions; substitute your model’s real layer count, hidden size, head grouping, and cache precision. Grouped-query attention and cache quantization reduce this substantially.)

Multiply that per-request footprint by your target concurrency and you get the memory budget the KV cache demands. Add model weights and you have the real occupancy of the card. This is the arithmetic that decides how many concurrent requests fit on an 80 GB H100 versus a smaller card — and it runs on token counts, full stop. If you want the estimate before writing serving code, a token estimator for LLM GPU cost planning turns the same inputs into a capacity figure.

Why word-based estimates understate cost, cache, and context length

The failure mode is consistent. A word-based plan is built on the friendliest possible assumption — English prose at roughly one token per word — and every deviation from that assumption pushes the real numbers up, never down. Three things break at once when real traffic hits:

Context length overruns first. A prompt template that a designer counted as “about 600 words” may tokenize to 900-plus, and once you add retrieved documents or conversation history, sequences bump into the context window limit sooner than planned. Requests that were supposed to fit start getting truncated or rejected.

Cache pressure follows. Because the KV cache scales with total tokens, the same inflation that stretched context length also inflates per-request memory. A GPU sized to hold, say, 40 concurrent requests at the word-based estimate holds meaningfully fewer at the real token count, and the serving layer starts queuing or evicting under load that the plan said was comfortable.

Cost lands last and loudest. The per-request dollar figure is the product of the two errors above, so it drifts furthest. A team that budgeted a per-request cost from word counts can find the production bill running well over plan — not because the model got more expensive, but because the accounting unit was wrong from the start.

The correct framing is the one we apply before any capacity conversation: measure the token distribution of representative real traffic — including the code-heavy and non-English tail — with the served model’s own tokenizer, and size from that distribution’s upper percentiles, not its mean. This is the same discipline that turns a vague “we need some GPUs” into a defensible cluster spec, which is where our GPU engineering practice starts every inference-cost engagement.

How does token count per request affect a GPU’s throughput ceiling?

Throughput is measured in tokens per second, and it is not a single number — it depends on the shape of your requests. A GPU serving many short completions spends proportionally more time in compute-bound prefill and can keep its matrix units busy; a GPU serving long completions spends most of its time in memory-bandwidth-bound decode, where the sustained ceiling is lower. Two deployments with identical request counts can have very different tokens-per-second ceilings purely because their prompt-to-completion ratios differ.

Batching complicates this further. Continuous batching (as implemented in vLLM and similar serving stacks) lets the GPU decode tokens for many requests in the same forward pass, which raises aggregate tokens/second — but only until the KV cache fills, at which point the scheduler must evict or queue and effective throughput falls. So the throughput ceiling is jointly set by compute for prefill, memory bandwidth for decode, and KV-cache capacity for concurrency. All three are driven by token counts. Measuring the tokens-per-second and latency of a local model on your own request mix is the only reliable way to find where that ceiling actually sits.

FAQ

What matters most about GPT token count in practice?

A tokenizer maps input text to integer IDs from a fixed vocabulary before the model runs; each ID is a token, and the model’s compute, latency, and memory scale with token count rather than word count. Because the split is done with byte-pair encoding, the count is a property of the text and the specific tokenizer, so you must count with the exact tokenizer your served model uses to get a number that predicts cost.

Why is a token not the same as a word, and how much does the count typically inflate for English, code, and non-English text?

GPT tokenizers split text into subword units, so common words may be one token while rare words, identifiers, and non-English strings fragment into several. English prose typically runs around 1.3–1.5 tokens per word, source code often 1.5–2.5× or more, and non-Latin-script text frequently 2–4× or more (observed patterns across common BPE tokenizers, not benchmarked rates). The variance across your traffic mix, not the average, is what breaks budgets.

How do prompt tokens and completion tokens differ in how they consume GPU compute and KV-cache memory?

Prompt tokens are processed in a parallel, compute-bound prefill pass; completion tokens are generated one at a time in a sequential, memory-bandwidth-bound decode phase that dominates response latency. Both populations feed the KV cache, which stores key and value tensors for every token in the sequence and grows linearly with total context length, occupying GPU memory for the request’s full lifetime.

How do you translate a token count into an inference cost and a GPU-memory footprint per concurrent request?

Serving cost tracks tokens directly — separate prices for prompt and completion on an API, or GPU-hour rate divided by sustained tokens/second when self-hosting. KV-cache footprint is roughly 2 × layers × hidden dimension × bytes-per-element per token; multiplying that by context length and target concurrency gives the memory the cache demands, which, added to model weights, sets how many requests fit on a card.

Why do word-based estimates understate context length, cost, and cache pressure once real traffic runs?

A word-based plan assumes English-prose token ratios, and every real deviation — code, multiple languages, structured data — pushes token counts up, never down. That inflation overruns context windows first, then increases per-request KV-cache memory (cutting achievable concurrency), and finally compounds into a per-request cost that drifts furthest of all because it multiplies both earlier errors.

How does token count per request affect the throughput ceiling (tokens/second) of a serving GPU?

Throughput depends on request shape: short completions stay compute-bound in prefill and keep matrix units busy, while long completions spend most time in bandwidth-bound decode with a lower sustained ceiling. Continuous batching raises aggregate tokens/second until the KV cache fills and the scheduler must evict or queue, so the ceiling is set jointly by prefill compute, decode bandwidth, and cache capacity — all functions of token counts.

Where this leaves your capacity plan

The uncomfortable part is that none of this can be resolved on paper. You can reason correctly about prefill, decode, and KV-cache scaling and still misestimate the deployment, because the one input that governs all three — the token distribution of your real traffic — is exactly the thing a pre-launch plan doesn’t have. The honest move is to treat the first weeks of production traffic as the measurement, size conservatively from the upper percentiles of a representative sample, and revise. Word counts feel like a shortcut; in inference capacity planning they are the failure class that shows up as an over-budget bill and a saturated card, and the artifact that prevents it is a token-distribution measurement taken with the served model’s own tokenizer.