NLP Tokenization Explained: How Text Becomes Tokens for Generative Models

Tokenization in natural language processing splits text into subword tokens and IDs.

NLP Tokenization Explained: How Text Becomes Tokens for Generative Models
Written by TechnoLynx Published on 11 Jul 2026

Tokenization in natural language processing is the step that splits raw text into tokens from a fixed vocabulary and maps each token to an integer ID the model can compute with. In modern language models those tokens are mostly subwords, units that sit between single characters and whole words. A large language model never reads your text directly. It is conditioned on the exact token sequence the tokenizer produced. That is why a component that looks like plumbing ends up shaping cost, context use and how well a model handles languages other than English.

What is the main goal of tokenization, and why does it matter so much?

The first goal is translation: models can only process numbers, so a tokenizer converts text inputs into numerical data. The harder goal is doing that with a fixed-size vocabulary, when language keeps producing new words. Jurafsky and Martin’s Speech and Language Processing (3rd edition draft) notes that vocabulary keeps growing as a corpus grows (Heaps’ Law), so no fixed vocabulary ever captures every word a model will meet, and unknown words become a serious problem for machine learning models. Some writing systems, including Chinese, Japanese and Thai, do not mark word boundaries with spaces at all.

Subword tokenization is the answer to both problems. If the subword vocabulary contains every base character, it is open-vocabulary: any string built from those characters can be represented, including one never seen in training. Common words stay whole as single tokens, while rare or unknown words break into smaller pieces. Sennrich and colleagues brought this idea to neural machine translation, and their subword models beat a back-off dictionary baseline on the WMT 15 English-German and English-Russian tasks by 1.1 and 1.3 BLEU.

Word, character, subword and sentence tokenizers compared

Tokenizers differ in the unit they cut text into. The table compares the four families you will meet.

Tokenizer type Splits text into What it does well Where it struggles Typical use
Word-based Words, on whitespace and punctuation Easy to set up with a few rules, often decent results English has over 500,000 words, and “dog” and “dogs” get unrelated IDs Rule-based pipelines, parsers that need grammatical words
Character-based Individual characters Much smaller vocabulary, far fewer unknown tokens Token sequences become extremely long Smallest units in data-driven tokenizers, used occasionally
Subword Frequent words kept whole, rare words decomposed Compact vocabulary that still captures meaningful pieces Vocabularies trained mostly on English split other languages into shorter pieces Byte-level BPE in GPT-2, WordPiece in BERT, Unigram in several multilingual models
Sentence Sentences Needed for tasks that detect structure, such as parsing Depends on language and genre NLTK’s Punkt, which must be trained on text in the target language

A short example shows the subword principle. A rare word such as “annoyingly” can be split into “annoying” and “ly”, two pieces that appear often on their own and still carry the meaning of the whole word. A rule-based word tokenizer handles the same kind of text differently: the Penn Treebank standard turns doesn’t into does plus n’t, keeps San Francisco-based together and separates out all punctuation. For more split-by-split cases there is our example of tokenization in LLMs, and the character tokenization explainer covers where character-level splitting earns its long sequences.

Sentence tokenizers do a different job altogether. Punkt learns a model of abbreviations, collocations and sentence-starting words without supervision, which is how it knows that the full stops in “Mr. Smith” do not end a sentence.

How BPE, WordPiece and Unigram split text

The three subword algorithms used with Transformer models are Byte Pair Encoding (BPE), WordPiece and Unigram. They differ in how they build the vocabulary during training and in how they cut new text at encoding time. The Hugging Face LLM Course summarises the differences like this:

  BPE WordPiece Unigram
Starting vocabulary Small, grows by merging Small, grows by merging Large, shrinks by removing tokens
Training step Merge the most common pair Merge the pair with the best score, favouring pairs whose parts are individually rare Remove the tokens whose loss over the corpus is smallest
What it stores Merge rules and a vocabulary Only the vocabulary A vocabulary with a score per token
How it encodes new text Split into characters, apply learned merges Longest matching subword from the start, then repeat on the rest Most likely split given the learned scores
Example models Llama, GPT-2 BERT, DistilBERT, Electra T5

BPE: merge what is frequent

BPE starts from individual characters and repeatedly merges the most frequent adjacent pair; in the Hugging Face example, “u” and “g” occur together most often across “hug”, “pug” and “hugs”, so they become the new token “ug”. Training stops after a set number of merges, so that number controls the vocabulary size. At encoding time the tokenizer simply replays the learned merges in the order they were learned, and the frequencies in the new text play no part. In practice BPE usually runs only inside words, after the corpus has been split on whitespace and punctuation.

In real settings, vocabularies are large. Jurafsky and Martin describe BPE run with tens of thousands of merges to reach 50,000, 100,000 or even 200,000 tokens, so most words are single tokens and only rarer words need several. Most modern language modelling efforts use byte-level BPE, which Radford et al. (2019) proposed with a base vocabulary of just 256 byte tokens. OpenAI’s tiktoken documentation lists the resulting properties: BPE is reversible and lossless, it works on any text including text absent from its training data, and on average each token corresponds to about 4 bytes. It also tends to expose common pieces, splitting “encoding” into “encod” and “ing” so the model sees “ing” in many contexts.

WordPiece: score pairs, then match greedily

WordPiece, developed by Google to pretrain BERT, marks pieces inside a word with a prefix, so “word” starts out as w ##o ##r ##d. Instead of merging the most frequent pair, it divides the pair’s frequency by the product of its parts’ frequencies, which favours merging pairs whose parts are rare on their own. At encoding time WordPiece keeps no merge rules: it finds the longest subword in the vocabulary from the start of the word, splits there and repeats. When no subword fits, the whole word becomes [UNK]; in the course’s toy vocabulary “bum” becomes ["[UNK]"], not ["b", "##u", "[UNK]"], whereas BPE would mark only the missing characters as unknown. The ## markers and their effect on evaluation are covered in more depth in our piece on BERT tokenization and WordPiece subwords.

Unigram and SentencePiece: probabilistic splits on raw text

Kudo’s Unigram work starts from the observation that subword segmentation is ambiguous: one vocabulary allows several segmentations of the same sentence. Unigram scores candidate splits with a unigram language model, and subword regularization trains the downstream model on several segmentations sampled probabilistically.

SentencePiece is a library, not a fourth algorithm: it applies BPE or Unigram directly to raw text instead of assuming spaces separate words. It treats input as a raw sequence of Unicode characters and replaces spaces with the meta-symbol ▁ (U+2581), so detokenization is a lossless string join in any language. Because it trains straight from raw sentences without language-specific pre-tokenizers such as MeCab or KyTea, it suits languages without explicit word boundaries. Its .model file holds the normalization rules, vocabulary and segmentation model together, so the same file gives the same tokens in C++, Python or Go.

Tokenization in natural language processing, step by step: a worked example

Encoding text for a Transformer happens in two stages, tokenization and then conversion to input IDs. Inside the tokenizer, the steps run in this order:

  1. Normalize. General clean-up such as removing needless whitespace, lowercasing or stripping accents, sometimes with Unicode normalization like NFC or NFKC.
  2. Pre-tokenize. Split into rough word-like units; a BERT tokenizer splits on whitespace and punctuation, and other tokenizers use different rules. The GPT-2 tokenizer, for instance, splits “We’re” into “We” and “‘re” and keeps a leading space attached to some tokens.
  3. Apply the subword model. Words the vocabulary lacks are broken into known pieces, for example “dreaming” into dream, ing and “carefully” into care, ful, ly.
  4. Add the extra inputs. The tokenizer adds any additional inputs the model expects, such as the special tokens in its vocabulary.
  5. Map to IDs. Each token is replaced by its integer index in the vocabulary.

A worked example from Hong and colleagues’ paper on Transformers puts this together: “Hello World!” becomes [Hello, world, !], then the IDs [7592, 2088, 999], and token embedding turns those IDs into a tensor of shape (3, N), where N is the embedding dimension.

Where normalization, stemming, lemmatization and stopword removal fit

In a classical NLP pipeline, tokenization runs before any other language processing, so it needs to be very fast. Stemming and lemmatization reduce word forms to a common base. Manning, Raghavan and Schütze’s Introduction to Information Retrieval describes stemming as a crude heuristic that chops off word endings and lemmatization as a vocabulary-based analysis that returns the dictionary form: faced with saw, a stemmer might return s while a lemmatizer returns see or saw depending on use. The Porter stemmer reduces operate, operating, operation and operational all to oper, and stemming generally raises recall while harming precision. Stopword removal drops frequent, low-value words using curated lists or statistical methods such as TF-IDF, and the lists need tailoring to the language and context.

In subword LLM tokenizers, normalization is a step inside the tokenizer before subword splitting. All of the tokenizer’s preprocessing has to be done exactly as it was when the model was pretrained. For how these steps sit in the wider family of methods, see our overview of natural language processing algorithms.

Tokenization versus embedding in an LLM pipeline

Tokenization and embeddings are two distinct but complementary processes. The tokenizer turns text into tokens and integer IDs. The embedding layer is a matrix of size V × N, vocabulary size by embedding dimension, that maps each ID to a vector in a high-dimensional space. The tokenizer’s vocabulary is fixed before the model is trained, while the embedding weights are learned during training.

The boundary matters because the model is conditioned on the exact tokenization it receives. The same surface text tokenized two ways maps to different internal representations, so the next-token distribution after “r a ce car” can differ sharply from the one after “race car”. Pairing a model with a tokenizer other than its own therefore breaks the rule that preprocessing must match pretraining exactly.

Why do different LLMs use different tokenizers?

No tokenizer is best for every application, and Mielke and colleagues’ survey concludes there likely never will be a single solution. Model families differ here too: Llama uses BPE, BERT-family models use WordPiece and T5 uses Unigram.

The choice has measurable effects on the model. Ali et al. found that tokenizer choice can significantly affect downstream performance and training cost, and that applying an English-centric tokenizer to multilingual LLM training led to severe performance degradation and up to 68% additional training cost. The same study found that the usual tokenizer metrics, fertility and parity, do not always predict downstream performance. Rust et al. found that a dedicated monolingual tokenizer matters as much to downstream performance as pretraining data size.

Because LLM training data is dominated by English, multilingual BPE tokenizers spend most of their vocabulary on English and split other languages into shorter tokens, which leads to poorer representations of meaning, longer contexts and higher training costs. Ahia et al. (EMNLP 2023) measured languages with their own scripts, such as Telugu, needing up to 5 times as many tokens as Latin-script languages to convey the same information. Petrov et al. found tokenization lengths for the same translated text differing by up to 15 times. Training data is not the whole explanation: Ahia et al. could not reach uniform fragmentation across scripts even with equal data, so properties of the script matter too. Heavier fragmentation also reduced in-context learning performance in the affected languages.

Those token counts feed into what API calls cost and how much fits in a context window. The practical arithmetic is covered in our guides to average tokens per word and GPT token count and inference cost.

For teams building generative AI systems, the tokenizer is part of the model’s contract, not a preprocessing detail to standardise across models. A retrieval pipeline that chunks documents by token count, for example, inherits the tokenizer’s behaviour on the corpus, so chunk sizes tuned for English text can hold far less content in another language; our notes on retrieval-augmented generation cover the retrieval side. Since the standard tokenizer metrics are weak proxies, the check worth running before committing to a model is to tokenize a representative sample of your own text, in every language and format you serve, and compare token counts and splits across the candidate models.

FAQ

What is the difference between a type and a token in linguistics?

A type is a distinct word in the vocabulary, and a token is each occurrence of a word in running text, so “the cat saw the dog” has five tokens but four types. Jurafsky and Martin use this distinction when describing Heaps’ Law, which links the number of types to the number of instances in a corpus.

How many tokens are in one word?

It depends on the tokenizer: with large BPE vocabularies most words are single tokens, while rarer words take several. tiktoken’s documentation, for OpenAI’s models, gives about 4 bytes per token on average. Our guide on how many characters per token turns this into a planning figure.

What is blockchain tokenization?

Blockchain tokenization is an unrelated use of the word: it means representing an asset or right as a digital token on a ledger, and it has nothing to do with splitting text for language models.

Back See Blogs
arrow icon