“A token is about four characters.” You have heard it, you have used it to size a budget, and it is close enough to be dangerous. The ratio is real, but it is not a constant — it drifts with language, code, whitespace, and the tokenizer’s own vocabulary. And because LLM providers bill per token, not per character, a cost model built on an assumed ratio starts diverging from the invoice the moment your traffic mix changes. That gap is the whole point of this article. The characters-per-token ratio is a fine rule of thumb for a sketch on a whiteboard. It is not a line item you can put in a cost model and defend under audit. The difference between those two uses is measured, not assumed. How many characters are in a token? For typical English prose tokenized with a modern byte-pair-encoding vocabulary — the kind OpenAI’s tiktoken cl100k_base and o200k_base encodings use — a token lands at roughly four characters on average. That number is an observed pattern on English text, not a specification, and it hides an enormous amount of variance underneath the average. Tokenizers do not split text on character boundaries or word boundaries. They split it on learned subword units. Common words like the or inference collapse into a single token; rare words fragment into several; a stray emoji or an unusual Unicode character can cost multiple tokens on its own. The “four characters” figure is the mean of a distribution that spans from one character per token (a single punctuation mark) to well under one character per token (multi-byte characters that a byte-level tokenizer splits apart). So the honest answer to “how many characters in a token” is: on English prose, around four, but the number you actually care about is the ratio on your traffic, and that can sit anywhere from under two to well over four depending on what your prompts and completions look like. Why does the characters-per-token ratio vary so much? The ratio moves because the tokenizer’s vocabulary was trained on a particular distribution of text, and your traffic may not match it. A few forces dominate. Language. Byte-pair-encoding vocabularies are heavily weighted toward English and other high-resource languages. English tends toward roughly four characters per token. Languages written in non-Latin scripts — and especially those where the tokenizer falls back to byte-level splitting — routinely produce far fewer characters per token, which means far more tokens for the same visible text. If your product serves a multilingual audience, an English-calibrated ratio will understate your token count on every non-English request. Code. Source code tokenizes differently from prose. Indentation, camelCase and snake_case identifiers, operators, and brackets fragment in ways natural language does not. A block of Python or JSON can carry a meaningfully lower characters-per-token ratio than the surrounding documentation, so a coding-assistant workload and a chat workload built on the same model will not share a ratio. Whitespace and formatting. Leading spaces are often bound into tokens, but runs of whitespace, tabs, and newlines — common in code, tables, and pretty-printed JSON — inflate token counts relative to their character length. Structured output formats you may treat as “free” formatting are not free at the token level. Vocabulary version. The ratio is a property of the specific tokenizer, not the model family in the abstract. Newer encodings like o200k_base carry a larger vocabulary than cl100k_base and tokenize the same text into fewer tokens. Switch models — or switch a provider’s default encoding — and your ratio shifts even if nothing about your prompts changed. This is a market-direction reality of a fast-moving model landscape, not a one-time adjustment. Why you can’t budget from a fixed characters-per-token constant Here is the failure mode we see most often. A team estimates traffic in characters — or in something they can measure easily, like request bodies or document pages — multiplies by an assumed four characters per token, multiplies again by the per-token price, and calls it a budget. It works right up until reality arrives. The problem is compounding. The assumed ratio is wrong by some percentage on your actual mix; that error propagates into token count; token count is what you are billed on; and the error rides through unchanged into the invoice. A budget that is 25% low on the ratio is 25% low on the bill, and it stays that way at every scale. It does not average out — it accumulates. It also drifts silently. Add a multilingual feature, ship a code-generation surface, change your system prompt to include more structured context, or upgrade to a model with a different tokenizer, and your effective ratio moves without anyone touching the budget spreadsheet. This is why a cost model built on an assumed ratio ages badly: the assumption is invisible once it is baked into a constant, so nobody re-checks it when the traffic that invalidated it lands. The correct framing is to treat the ratio as a measured input, not a constant. Cost is billed per token. A defensible model therefore starts from token counts on real prompts and completions and derives the ratio from them — the reverse of the naive direction. Getting this right is what lets a cost-per-token price translate into an accurate cost-per-request figure, and cost-per-request is the number a business actually plans against. If you want the full unit-economics framing that sits on top of this, our breakdown of what LLM context windows cost at inference works through the same per-request math from the context-length angle. How token count becomes cost-per-request Cost-per-request is the atomic unit of an LLM cost model, and it decomposes cleanly once you have measured token counts. Most providers price input (prompt) tokens and output (completion) tokens separately, often at very different rates, so you cannot collapse them into a single figure. Worked example (illustrative, assumptions stated) Assume a chat request where the prompt — system message plus context plus user turn — measures 1,200 input tokens on your tokenizer, and the model returns a 400-token completion. Assume illustrative prices of $0.50 per million input tokens and $1.50 per million output tokens. Input cost: 1,200 × $0.50 / 1,000,000 = $0.0006 Output cost: 400 × $1.50 / 1,000,000 = $0.0006 Cost per request ≈ $0.0012 Now suppose you had budgeted using an assumed 4.0 characters per token, but your prompts — heavy with JSON context and code snippets — actually run at 3.0 characters per token. Your real token count is about a third higher than the estimate, and so is your bill. On a million requests a day, that gap is the difference between a plan that holds and a plan that gets escalated. These are illustrative figures to show the mechanics; the prices, the token counts, and above all the ratio must come from your own measurement. The structure is what transfers, not the numbers. How do I measure the actual token count on my traffic? You do not estimate this — you count it, because the count is cheap and the estimate is where the error lives. Tokenize a representative sample of real traffic. Pull actual prompts and completions from logs, not synthetic examples. Run them through the exact tokenizer your production model uses — tiktoken for OpenAI-family models, the model’s AutoTokenizer from the Hugging Face transformers library for open-weight models served through vLLM, SGLang, or TensorRT-LLM. The tokenizer must match the model, or the count is fiction. Record input and output tokens separately, per request. They price differently and they scale differently — output length is driven by the model’s behaviour, input length by your prompt design. Derive the effective characters-per-token ratio from the counts, segmented by the axes that matter for your product: language, prose vs. code, and traffic surface. One global ratio hides exactly the variance that breaks budgets. Re-measure when anything changes — a new model, a new encoding, a new feature, a prompt rewrite. Treat the ratio as a monitored quantity, not a settled one. Most serving stacks already surface token counts in their usage APIs or response metadata, so step one is often a matter of instrumenting what you already have rather than building something new. If routing across models is part of your cost strategy, note that each model may carry its own tokenizer and its own ratio — our explainer on how model routing cuts LLM inference cost covers why per-model token economics matters when a router sends the same request to different backends. Comparison: assumed ratio vs. measured ratio Dimension Assumed constant (~4 chars/token) Measured on your traffic Source Rule of thumb, English prose tiktoken / AutoTokenizer on real logs Handles multilingual traffic No — understates non-English tokens Yes — segmented by language Handles code / structured output No — prose-biased Yes — segmented by surface Survives a model / tokenizer swap No — silently wrong Yes — re-measured on change Translates to cost-per-request Approximately, with hidden error Yes, audit-defensible Best used for Whiteboard sketch, order-of-magnitude Cost model, replacement decisions The table is not arguing that estimation is useless. It is arguing that estimation and measurement do different jobs, and the moment a number stops being a sketch and starts driving a spending or replacement decision, it has to be measured. FAQ How many characters are in a token? For typical English prose tokenized with a modern byte-pair-encoding vocabulary such as OpenAI’s cl100k_base or o200k_base, a token averages roughly four characters — an observed pattern, not a specification. That average hides wide variance: punctuation, rare words, and multi-byte characters can each shift the count sharply, and the number that matters is the ratio measured on your own traffic. Why does the characters-per-token ratio vary across languages, code, and whitespace? The tokenizer’s vocabulary was trained on a particular distribution of text, mostly high-resource languages, so anything unlike that distribution tokenizes less efficiently. Non-Latin scripts often fall back to byte-level splitting and produce more tokens per visible character; source code fragments on identifiers, operators, and indentation; and runs of whitespace inflate token counts relative to character length. How does token count relate to cost-per-request in an LLM inference deployment? Providers bill per token, and usually price input and output tokens at different rates, so cost-per-request is the sum of measured input tokens times the input price and measured output tokens times the output price. Because billing is per token, the accuracy of your cost-per-request figure depends entirely on how accurately you counted tokens, not characters. Why can’t I budget LLM cost from a fixed characters-per-token constant? An assumed ratio is wrong by some percentage on your real mix, and that percentage rides straight through into token count and then into the invoice without averaging out at scale. Worse, it drifts silently: a new multilingual feature, a code surface, or a tokenizer upgrade moves the real ratio while the baked-in constant stays put, so nobody re-checks the number that the change invalidated. How do I measure the actual token count on my own prompts and completions? Tokenize a representative sample of real logged traffic using the exact tokenizer your production model uses — tiktoken for OpenAI-family models, the matching AutoTokenizer for open-weight models — and record input and output tokens separately per request. Derive the effective ratio from those counts, segmented by language and traffic surface, and re-measure whenever the model, encoding, or feature set changes. How does cost-per-token connect to the calibrated ROI model in an inference cost audit? Measured token counts are what turn a cost-per-token price into an accurate cost-per-request figure, and cost-per-request is the number a replacement or optimisation decision actually rests on. An [inference cost audit](Inference Cost-Cut Pack) uses the measured token economics on your real traffic to build a calibrated ROI model, so the decision rests on real numbers rather than a back-of-envelope constant. Where the number stops being a rule of thumb The characters-per-token ratio is one of those figures that is correct as a heuristic and wrong as a foundation. It is a useful sketch for sizing a proof of concept and a liability the moment it becomes a constant in a spending decision. The dividing line is measurement: count tokens on your real prompts and completions, segment the ratio by the axes that move it, and re-measure when your traffic or your model changes. That measured token count is the input to the calibrated ROI model an [inference cost audit](Inference Cost-Cut Pack) produces — the model that turns cost-per-token into cost-per-request and lets a replacement or optimisation decision rest on real token economics. If you want to see how we scope that kind of work, our engagement approach starts from the same principle we have argued here: measure the thing you are billed on before you plan around it. The failure class is a cost model that drifts from the invoice because the ratio underneath it was assumed rather than measured.
