Ask most teams how big a prompt is and they count words. That is the first place a generative AI budget goes wrong. A large language model does not read words — it reads tokens, and the two rarely line up one-to-one. The word “tokenization” itself, the thing this article is about, is not a single token in most modern vocabularies. It splits. That split is the whole point. If you budget your API spend and your context window by counting words, you are estimating in a unit the model never uses. The gap looks harmless in a demo and becomes expensive in production, usually at the worst possible moment — mid-build, when prompts start overflowing the window or the monthly bill lands higher than the proposal said it would. What does an example of tokenization actually look like? Take a plain sentence and watch how a tokenizer breaks it apart. Consider the string: Tokenization isn't word-counting. A byte-pair-encoding tokenizer — the family behind OpenAI’s GPT models via the tiktoken library, and closely related to the WordPiece and SentencePiece schemes used by BERT and Llama — does not see four words. It sees a sequence of learned subword units. A rough segmentation looks like this: Token · ization · ` isn · ‘t · word · - · counting · .` That is eight tokens for four words. Notice several things at once. The leading space is attached to a token ( isn, ` word) rather than treated as a separator — most BPE tokenizers encode whitespace as part of the following token. The contraction isn’t splits at the apostrophe. The hyphen in word-counting becomes its own token. And Tokenization` — the term everyone assumes is one unit — decomposes into a common prefix and a suffix the vocabulary has seen often enough to store. This is what “subword segmentation” means in practice: the tokenizer learned a fixed vocabulary of frequent character sequences during training, and at inference it greedily matches your text against that vocabulary. Common words map to single tokens. Rare words, technical terms, and anything with unusual punctuation get chopped into pieces the vocabulary already knows. Why is a token not the same as a word? The tokenizer’s job is to represent any possible input using a fixed, finite vocabulary — typically on the order of 30,000 to 200,000 entries, depending on the model (per the published vocabulary sizes of models like Llama and GPT-4). A word-level vocabulary can’t do this: natural language has a long tail of names, misspellings, code identifiers, and morphological variants that no fixed word list covers. Subword tokenization solves the coverage problem by falling back to smaller and smaller pieces until, worst case, it reaches individual bytes. The consequence is that the token count for a given text is a property of that tokenizer, not of the text. The same paragraph produces different token counts under GPT-4’s cl100k_base encoding than under Llama’s SentencePiece model. This is one reason our own NLP tokenization walkthrough treats the encoding as a first-class part of the model choice, not an implementation detail you can defer. A useful rule of thumb, framed as an approximation you should verify against your own text rather than trust blindly: for ordinary English prose, one token is roughly three-quarters of a word — about four characters (observed pattern across common English corpora; not a guarantee for your data). So 1,000 words of clean English is on the order of 1,300 tokens. That heuristic collapses the moment your text stops being clean English. Where does tokenization sit in the generative AI family? Tokenization is specific to the generative — and more broadly, the language-model — family of AI systems. A convolutional vision model does not tokenize text; it consumes pixel grids. A tabular gradient-boosted model consumes numeric features. The token is the unit of work for transformer-based language models specifically: it is what the attention mechanism attends over, what the model predicts one step at a time during generation, and what every published context-window and pricing number is denominated in. That is why understanding tokenization is the concrete bridge from “this is a generative AI system” to “here is what it will cost and whether the prompt fits.” It is the same reasoning we apply in a GenAI feasibility audit: classify the system as generative, then turn that classification into token-denominated cost and context-window arithmetic before anyone commits to a model. If you want to see how the same token unit propagates into vector-database sizing downstream, our note on how embeddings power agent retrieval picks up where this one leaves off. How do token counts drive cost and the context window? Two numbers in every LLM deployment are denominated in tokens, and both are easy to underestimate if you think in words. The first is API cost. Commercial LLM APIs bill per token — separately for input (prompt) and output (completion) tokens, usually at different rates (per the published pricing pages of major providers). Your monthly spend is, to first order, requests × (input_tokens + output_tokens) × price_per_token. Every element there is a token count. Estimate the token counts wrong and the whole projection is wrong by the same factor. The second is the context window — the fixed maximum number of tokens a model can attend to in a single request, covering prompt plus generated output. A model advertised with a 128,000-token window (per published model specifications) does not hold “128,000 words.” Under the three-quarters heuristic that is closer to 96,000 words of clean English, and considerably fewer if your prompt is full of code, tables, JSON, or non-English text. When a retrieval-augmented system silently stuffs more context into a prompt than the window allows, the request fails or the oldest content gets truncated — a failure mode that shows up only under real document loads, not in the demo. Worked cost-and-fit estimate Here is the arithmetic a team should do before choosing a model. Assumptions are stated so you can substitute your own. Input Assumption Value Prompt text 1,500 words of clean English ~2,000 tokens Retrieved context 3 documents, ~800 words each, some tables ~3,600 tokens System + instructions fixed boilerplate ~500 tokens Expected output ~400 words ~550 tokens Total per request prompt + output ~6,650 tokens Context window needed must exceed total fits in a 8k window, tight Monthly volume 50,000 requests — Cost illustrative $X per 1k tokens 50,000 × 6.65 × $X The point of the table is not the numbers — those depend on your provider and your text. The point is that every row is a token estimate you can compute in advance, and that a word-based guess for the retrieved-context row (which contains tables, so it tokenizes worse than prose) is exactly where the estimate silently breaks. How do punctuation, whitespace, and rare terms inflate the count? The three-quarters rule of thumb assumes text that looks like the corpus the tokenizer was trained on. Real production inputs rarely do. A few patterns reliably push the token-to-word ratio up: Code and structured data. JSON, XML, and source code are dense with braces, quotes, indentation, and identifiers that don’t appear as whole words in the vocabulary. Each becomes one or more tokens. A block of JSON can run well past one token per two characters — far worse than prose. Non-English and mixed-script text. Languages the tokenizer saw less often during training fragment into more pieces, sometimes down to individual bytes for scripts like Chinese, Japanese, or Arabic. The same meaning costs more tokens. Rare or technical terms and long compounds. Domain vocabulary — chemical names, gene identifiers, part numbers — has no single vocabulary entry, so it decomposes. “Tokenization” splitting into two tokens is the mild version; a chemical IUPAC name can become a dozen. Repeated whitespace and formatting. Tabs, repeated spaces, and newline-heavy layouts each consume tokens that add nothing a human counting words would notice. Any of these can push a document that “looks like” 5,000 words to 9,000+ tokens. If your retrieval corpus is contracts, lab reports, or logs rather than clean articles, assume the ratio is worse than the heuristic and measure it directly. The same caution about production data diverging from the clean case runs through our piece on why GenAI fails on production data — tokenization inflation is one concrete instance of that broader gap. How should a team estimate tokens before choosing a model? Do not estimate in words and convert. Tokenize representative samples of your actual input with the actual tokenizer for each candidate model, and measure. tiktoken for OpenAI models, and the model’s own tokenizer via Hugging Face transformers for open-weight models like Llama, both run offline in a few lines and give you exact counts on real text. Then: Sample ten to twenty documents that look like your worst-case production input — the ones with tables, code, or non-English text, not the tidiest examples. Tokenize them with each candidate model’s tokenizer and record the token-to-word ratio you actually observe. Build the per-request total from the worked-example table above, using your measured ratio, not the three-quarters default. Check the total against each model’s published context window with headroom for output, and against the per-token price for the volume you expect. That turns two of the riskiest numbers in a GenAI proposal from guesses into arithmetic. Our token size calculator walkthrough covers the mechanics of doing this at scale across a corpus. FAQ How does example of tokenization work? A tokenizer breaks input text into subword units drawn from a fixed vocabulary learned during training. In the example Tokenization isn't word-counting., four words become roughly eight tokens because the term splits into a prefix and suffix, the contraction and hyphen break at their punctuation, and leading spaces attach to tokens. In practice it means the model’s unit of work is the token, and every cost and context estimate must be denominated in tokens rather than words. Why is a token not the same as a word, and how does subword segmentation split terms like ‘tokenization’? A word-level vocabulary cannot cover the long tail of names, code, misspellings, and morphological variants in real language, so tokenizers use a fixed subword vocabulary and fall back to smaller pieces — down to bytes — for anything unfamiliar. Common words map to one token; rarer terms decompose. “Tokenization” splits into a frequent prefix (Token) and a suffix (ization) because the whole word was not stored as a single vocabulary entry. How do token counts drive LLM API cost and consumption of the context window? Commercial APIs bill per input and output token, so monthly spend is requests times token counts times price — every factor is a token count. The context window is a fixed maximum number of tokens covering prompt plus output; a 128k-token window is not 128k words and holds far less code or non-English text. Underestimating token counts throws off both the bill and whether prompts fit. Where does tokenization sit relative to the generative AI family in the working taxonomy, and why is it specific to that family? The token is the unit of work for transformer-based language models: it is what attention operates over, what the model predicts step by step, and what context-window and pricing numbers are denominated in. Vision or tabular models consume pixels or numeric features, not tokens. Tokenization is therefore the concrete bridge that turns a “this is generative AI” classification into cost and context-window arithmetic. How do whitespace, punctuation, and rare or non-English terms change the number of tokens a text produces? They all push the token-to-word ratio above the clean-English heuristic of roughly three-quarters of a word per token. Code and JSON fragment on braces, quotes, and identifiers; non-English and mixed scripts split into more pieces, sometimes single bytes; rare technical terms and long compounds decompose into many tokens; and repeated whitespace and formatting consume tokens invisibly. A “5,000-word” document of contracts or logs can easily exceed 9,000 tokens. How should a team estimate tokens for a prompt or document before choosing a model? Do not estimate in words and convert — tokenize representative worst-case samples with each candidate model’s actual tokenizer (tiktoken for OpenAI, the model’s Hugging Face tokenizer for open-weight models) and measure the real ratio. Build the per-request total from prompt, retrieved context, instructions, and expected output, then check it against each model’s published context window and per-token price at your expected volume. The question worth carrying into model selection The naive question is “how many words is my prompt?” The useful question is “under this specific tokenizer, on my worst-case production text, how many tokens does one request cost — and does that fit the window with room to generate?” Answer the second question with measured counts, not the first with a guess, and the two riskiest numbers in a generative AI plan become arithmetic. Where that arithmetic first gets pinned down is the GenAI feasibility audit: the point where “generative” stops being a label and turns into a defensible cost and context-window estimate.
