A token is not a word. If your inference cost model treats the two as roughly interchangeable, every number downstream of that assumption inherits the error, and the error is not small or random. tiktoken — the byte-pair encoding tokeniser OpenAI ships for its models — will split the same English sentence, the same JSON blob, and the same paragraph of German into wildly different token counts, and the count is what actually drives sequence length, memory footprint, and per-request compute on the GPU. The common shortcut is to estimate tokens from word count — “roughly 0.75 words per token” is the folk ratio people repeat — then feed that guess into a throughput and cost model. That works well enough for a napkin estimate and badly enough to move an infrastructure decision to the wrong side of its break-even. The fix is not a better ratio. It is measuring the actual token distribution across representative traffic before you size anything. How does tiktoken count tokens, and what does it mean in practice? tiktoken implements byte-pair encoding (BPE). The text is first broken into bytes, then the encoder repeatedly merges the most frequent adjacent pairs according to a learned vocabulary until no more merges apply. The result is a sequence of integer token IDs, and the count of those IDs is what the model — and the billing meter, and the KV-cache — actually sees. The important consequence is that the mapping from characters to tokens is learned, not rule-based. Common English words often collapse to a single token. Rare words, inflected forms, and anything the vocabulary never saw during training get split into sub-word fragments. A word like tokenisation might be two or three tokens; a made-up product SKU might be six or seven. This is why the folk ratio breaks: it is an average over a corpus that looks nothing like your traffic. Here is the mechanism stated as something you can quote and reason about: tiktoken counts tokens by applying byte-pair encoding under a specific model encoding, so the same text produces a different token count depending on the encoding, and that count — not word count — determines sequence length, KV-cache memory, and per-token compute at inference time. That is the whole reason this matters for GPU planning. Token count per request is the input to the workload profile. Get it wrong and the profile is wrong. Why does the token count differ from the number of words in my text? Four things move the count away from your word intuition, and they compound. Punctuation and whitespace are tokens too. Leading spaces are frequently bound to the following word as a single token, but trailing punctuation, repeated newlines, and markdown syntax (###, |, backticks) each consume tokens the word count ignores entirely. A heavily formatted document costs more than its readable word count suggests. Code tokenises poorly relative to prose. Identifiers like getUserAccountBalance, operators, indentation, and bracket runs fragment badly. In our experience profiling code-assistant and RAG workloads, code-heavy requests run noticeably denser in tokens per character than plain English — an observed pattern across the traffic we have profiled, not a fixed multiplier, and one you should measure on your own corpus rather than assume. Non-English text inflates the count. BPE vocabularies trained largely on English represent English cheaply. Languages with different scripts, agglutinative morphology, or heavy diacritics get split into more fragments per word. A German or Japanese request can carry substantially more tokens than an English translation of the same meaning. Structured data is the worst offender. JSON, tables, and long numeric strings fragment into many small tokens. A single UUID can be several tokens on its own. None of this is exotic. It is the ordinary content real applications send. The point is that word count systematically under-estimates for exactly the traffic — code, structured payloads, non-English — that production systems tend to carry. If you have already read our companion piece on how average tokens-per-word ratios distort inference cost, this is the mechanism underneath that distortion. Do cl100k_base and o200k_base produce different counts for the same input? Yes, and this is the trap that catches teams who standardise on one number. tiktoken selects an encoding based on the model. The cl100k_base encoding backs the GPT-3.5 and GPT-4 generation; o200k_base backs the newer GPT-4o family. They have different vocabularies and different vocabulary sizes (roughly 100k versus 200k merges, per the encoding names), so the same string can tokenise to a different number of tokens under each. A larger vocabulary generally packs common text into fewer tokens, because more multi-character sequences earned their own token during training. The practical implication: if you profile your traffic under one encoding and then serve a model that uses another, your token distribution — and therefore your cost model — is calibrated to the wrong encoding. import tiktoken text = "def get_user_balance(account_id): return db.query(account_id)" for enc_name in ("cl100k_base", "o200k_base"): enc = tiktoken.get_encoding(enc_name) print(enc_name, len(enc.encode(text))) The rule is simple: always count under the encoding of the model you will actually serve. Do not carry a token distribution measured against one encoding into a cost model for a model that uses another. How do I measure tokens per request across real workload traffic? Do not estimate a scalar. Measure a distribution. A single “average tokens per request” figure hides the tail, and the tail is where memory pressure and cost overruns live — a request at the 99th percentile of sequence length is the one that determines your worst-case KV-cache footprint. The procedure we use when profiling an inference workload: Sample representative traffic. Pull a few thousand real prompts (and, if you serve them, real completions) from logs or a shadow capture. Synthetic prompts will not reproduce your code/prose/non-English mix. Tokenise under the serving encoding. Run each sample through enc.encode() for the correct encoding and record the length. Record prompt and completion separately — they scale cost differently, because completion tokens are generated autoregressively and dominate decode-phase compute. Build the full distribution. Compute the mean, median, p90, p95, and p99 of tokens per request. The spread between median and p99 tells you how bursty your sequence-length profile is. Segment by request type if your traffic is mixed (chat vs. code vs. document analysis). Each segment often has a distinct token profile, and blending them into one number destroys the signal. This distribution is exactly the kind of workload characteristic that feeds a proper GPU performance audit: sustained-versus-burst behaviour, per-request cost, and the sequence-length spread that determines batching efficiency. How does per-request token count translate into GPU memory and compute? Two distinct costs, and they scale differently — conflating them is a common modelling error. Compute per token is roughly linear in the number of tokens processed for the feed-forward and projection work, but attention cost grows with sequence length because each new token attends over the growing context. Longer sequences are more than linearly expensive on the attention path, which is why FlashAttention and similar kernels exist to keep that cost tractable. KV-cache memory is the sharper constraint for serving. Every token in the context occupies key/value cache entries across every layer and attention head, in the model’s serving precision. That footprint scales with sequence length times batch size, and it competes directly with model weights for HBM. When you under-count tokens, you under-provision KV-cache, and the system silently caps your effective batch size — which shows up as lower throughput and higher cost per request, not as an obvious error. Here is the chain stated plainly, because it is the whole argument: Token count per request sets sequence length; sequence length sets both per-request compute and KV-cache memory; those two set achievable batch size and GPU utilisation; and utilisation is precisely what the cloud-versus-on-premise break-even turns on. If you want the runtime mechanics of how prefix reuse reclaims some of that KV-cache cost, RadixAttention and KV-cache reuse covers the serving-side lever; this article stays on the counting that feeds the model. Worked example: how a token under-count moves the cost model Assume a support-assistant workload. Word-count estimate says 300 tokens per request average. You size a cloud GPU fleet against that. Assumption Word-count estimate Measured (tiktoken, o200k_base) Avg tokens / request 300 390 (+30%) p99 tokens / request ~450 (guessed) 1,120 (code + JSON tail) Effective batch size (KV-cache bound) planned 32 actual 20 Cost per 1k requests baseline ~1.5× baseline The point of the table is not the specific numbers — they are illustrative, framed for a hypothetical workload — but the direction and mechanism: a 30% under-count on the mean plus a badly under-estimated tail collapses the batch size the fleet was sized around. Under-counting tokens by even 20–30% (an observed pattern in workloads we have profiled, not a benchmarked constant) can push an infrastructure decision to the wrong side of its break-even, because utilisation targets set against a fantasy distribution never materialise. How does an accurate token distribution feed the cloud-vs-on-premise model? The infrastructure decision — rent cloud GPUs or buy on-premise accelerators — is a utilisation question over a 12–36 month horizon. On-premise wins when utilisation is high and sustained; cloud wins when load is spiky or uncertain. The token distribution is a primary input to that utilisation profile. A measured distribution sharpens three things the break-even model depends on: the sequence-length spread (which sets achievable batch size and therefore throughput per GPU), the sustained-versus-burst shape (which decides whether you are paying for idle capacity), and the per-request cost that anchors the total-cost comparison. Feed it a guessed token count and you are comparing two infrastructure options against a workload that does not exist. This is also why the audit treats token distribution as a first-class workload characteristic rather than a modelling afterthought. If you are building the cost case for on-premise hardware, our readiness checklist for on-premise accelerators walks the utilisation and profiling steps that sit alongside token counting. FAQ What should you know about tiktoken count tokens in practice? tiktoken applies byte-pair encoding: it breaks text into bytes and merges frequent adjacent pairs according to a learned vocabulary until it produces a sequence of integer token IDs. The count of those IDs — not the word count — is what the model, the billing meter, and the KV-cache all see, so it is the correct input to any inference cost or capacity model. Why does the token count from tiktoken differ from the number of words in my text? Because tokenisation is learned, not rule-based. Punctuation, whitespace, and markdown syntax each consume tokens; code and structured data fragment into many small tokens; and non-English text splits into more fragments per word than English. These effects push the token count above the word count, especially for the code, JSON, and multilingual traffic real applications tend to carry. How do I use tiktoken to estimate tokens per request across my real workload traffic? Sample a few thousand representative prompts (and completions) from logs, tokenise each under the encoding of the model you will actually serve, and build a full distribution — mean, median, p90, p95, p99 — rather than a single average. Record prompt and completion tokens separately and segment by request type, because chat, code, and document traffic each have distinct profiles that a blended average destroys. How does per-request token count translate into GPU memory and compute cost for inference? Token count sets sequence length, which drives two separate costs: per-token compute (with attention growing worse than linearly as context lengthens) and KV-cache memory (which scales with sequence length times batch size and competes with model weights for HBM). Under-counting tokens under-provisions the KV-cache, silently caps effective batch size, and shows up as lower throughput and higher cost per request. Do different models and encodings (cl100k_base, o200k_base) produce different token counts for the same input? Yes. cl100k_base (GPT-3.5/GPT-4) and o200k_base (GPT-4o family) have different vocabularies and sizes, so the same string can tokenise to a different number of tokens under each. Always count under the encoding of the model you will serve; carrying a distribution measured against the wrong encoding calibrates your entire cost model incorrectly. How does an accurate token distribution feed the cloud-vs-on-premise infrastructure cost model? The cloud-versus-on-premise decision is a utilisation question, and the token distribution sets the sequence-length spread, the sustained-versus-burst shape, and the per-request cost that the break-even depends on. A measured distribution turns those inputs into figures traceable to real traffic; a guessed one compares two infrastructure options against a workload that does not exist. Measure the distribution before you size the fleet. The failure mode here is not a wrong average — it is an unmeasured tail feeding a utilisation target that the hardware can never hit, and it is exactly the kind of workload characteristic a GPU performance audit exists to pin down before the money is committed.
