A subword tokenizer splits PN-4471-A/2 into fragments no human would choose — PN, -, 44, 71, -A, /2 — and once that split happens, the model reasons over pieces that no longer line up cleanly with the characters on the page. For most prose, nobody notices or cares. For a part number, a chemical identifier, or a lot code that has to trace back to an exact source region in a supplier document, that invisible split is where extraction provenance starts to fray. Character tokenization is the alternative most teams never seriously consider, because tokenization is treated as an implementation detail the model handles for them. In a regulated document-intelligence pipeline, it is not a detail. It is one of the concrete choices that determines whether an extracted value can survive an auditor’s derivation question. What does working with character tokenization involve in practice? Tokenization is the step that turns raw text into the discrete units a model consumes. A character-level tokenizer treats each character as its own unit: 4, 4, 7, 1, -, A are six tokens, mapped one-to-one to six positions in the input. There is no learned merging, no vocabulary of frequent substrings, no guessing about where a “word” begins or ends. That one-to-one property is the whole point. When a model produces an output span and you need to know which characters on the original page it came from, a character-level scheme gives you a direct index: token position n is character n. Nothing is fused or fragmented in a way that has to be reversed before you can point back at the source. Contrast that with what most production systems actually run. Modern transformer models — the architecture behind nearly every extraction model you would reach for — almost always use subword tokenizers such as Byte-Pair Encoding (BPE) or WordPiece, the schemes shipped with the tokenizers in Hugging Face and used by GPT-family and BERT-family models. Those tokenizers learn a vocabulary of common fragments from a training corpus, which is efficient for natural language and disastrous for strings that never appeared in that corpus in that exact form. How does character tokenization differ from word and subword tokenization? The three schemes sit on a spectrum from coarse to fine, and each trades a different thing. Scheme Unit Vocabulary size Sequence length Behaviour on PN-4471-A/2 Word Whole words Very large (100k+) Shortest Likely one out-of-vocabulary blob or dropped entirely Subword (BPE / WordPiece) Learned fragments Moderate (~30k–50k) Medium Unpredictable split into 4–6 pieces, boundaries vary by context Character Single characters Tiny (hundreds) Longest One token per character, boundaries fixed and stable Word tokenization was the older default and fails hardest on structured identifiers, because a serial code is almost never a known word — it becomes an unknown token or gets discarded. Subword tokenization, the current mainstream, handles unknown strings gracefully in the sense that it never fully breaks, but how it breaks depends on the surrounding context and the learned merge rules. That is the trap: it looks like it is working because it produces output, while the internal representation of the identifier is inconsistent from one document to the next. Character tokenization removes that inconsistency. The cost, which we come back to below, is that everything gets longer. When does character-level tokenization help extract structured identifiers accurately? The value of character granularity shows up precisely on the fields where subword tokenizers are weakest: alphanumeric codes, part numbers, chemical identifiers such as CAS numbers, lot and batch codes, and any string where a single transposed or dropped character changes the meaning. These are exactly the fields that dominate automotive supplier-compliance documents and regulated material declarations. There are three properties that make character tokenization the right granularity for these fields: Stable segmentation. The same identifier tokenizes identically every time, regardless of what surrounds it. That stability is what lets a downstream check compare extracted values field-to-field without normalising away tokenizer artefacts. Character-level alignment. Because each token is one character, the mapping from model output back to input offsets is exact. In our experience across regulated document-automation engagements, this is the property that most directly reduces mis-split codes reaching a human reviewer — an observed pattern in the extraction stage, not a benchmarked rate. Graceful handling of never-seen strings. A model that reasons over characters is not surprised by a part number format it has never encountered, because it was never relying on a learned fragment vocabulary for that field in the first place. This is the concrete facet behind claim C17: for the fields that carry legal and safety weight, choosing tokenization granularity deliberately affects the percentage of extracted values that keep an intact source-region link. It is not a universal recommendation to tokenize everything by character — it is a targeted choice for the fields where structure is the meaning. What does character tokenization cost, and when is the trade-off worth it? Nothing about character tokenization is free, and the cost is not subtle. Representing text one character at a time produces sequences that are several times longer than the subword equivalent — on the order of four to five times more tokens for typical English text, per the published behaviour of standard BPE vocabularies. Transformer self-attention scales quadratically with sequence length, so longer inputs mean more memory and slower inference, whether you are running on PyTorch with FlashAttention or serving through an optimised runtime. That trade-off is worth paying only where the accuracy and provenance gain matters. A quick rubric: Field type Recommended granularity Rationale Free-text descriptions, comments Subword Length cost dominates; provenance rarely disputed Part numbers, serial codes Character (or character-aware) Segmentation stability and exact alignment matter Chemical / CAS identifiers Character (or character-aware) Single-character errors change meaning Dates, quantities, units Character-aware or validated subword Structure matters but formats are constrained Long narrative paragraphs Subword Character length becomes prohibitive The practical answer for most pipelines is not “switch the whole model to character tokenization.” It is to apply character-level handling only to the structured fields that need it, either through a character-aware model variant, a field-specific extraction pass, or a validation layer that operates on character offsets. Reserve the expensive granularity for the values that would otherwise reach a reviewer mis-split. Why does the tokenization choice affect provenance at all? Provenance in a compliance extraction pipeline means one thing concretely: given an extracted value, you can point back to the exact region of the source document it came from and defend that derivation. That link is what an auditor tests. It is also what separates a document-automation pipeline you can put in front of a regulator from one that merely looks fast. When a subword tokenizer fragments an identifier unpredictably, the model’s internal positions no longer map cleanly onto source characters. You can often reconstruct an approximate span, but “approximate” is the wrong word to say to an auditor asking where a serial code came from. A character-level scheme keeps the value-to-source-region mapping exact and stable, which is why it belongs in the discussion whenever the extraction stage has to defend an audit rather than just populate a database. The same provenance discipline runs through our work on machine-learning monitoring for provenance-preserving compliance automation, where the extracted value’s traceability is the thing being watched over time. If you are still deciding whether a rules-based reader or a learned model should do the extraction at all, that upstream question is worth settling first; we walk through it in OCR vs AI for supplier-compliance documents. Tokenization granularity is a decision you make after you have chosen a learned extraction approach. How does character tokenization fit a staged pipeline rather than replacing it? The mistake worth avoiding is treating tokenization granularity as a single global switch. A production document-intelligence pipeline is staged — ingestion, layout analysis, region detection, field extraction, validation, and evidence assembly — and tokenization is a lever inside the extraction stage, not a property of the whole system. You can run subword tokenization for the narrative portions of a document and route the structured-identifier fields through character-level handling, without rebuilding anything upstream. This is why the granularity choice sits comfortably inside the broader picture of what document intelligence is and how it works in automotive supplier compliance. The pipeline stays the same shape; you are tuning one stage so the fields that carry audit weight keep their alignment. For teams working out where learned components like this belong in a traceable extraction flow, our services team scopes exactly this kind of stage-level decision against the compliance requirements that actually constrain it. FAQ What should you know about character tokenization in practice? A character-level tokenizer treats each character as its own token, mapping input positions one-to-one to characters. In practice this means the model reasons over the raw character sequence rather than over learned word or subword fragments, so an output span maps directly back to the exact characters it came from on the source page. How does character tokenization differ from word and subword tokenization? Word tokenization uses whole words and fails on unknown strings like serial codes; subword schemes such as BPE and WordPiece split text into learned fragments whose boundaries shift with context; character tokenization uses single characters with fixed, stable boundaries. The trade is granularity for sequence length — character tokenization produces the longest sequences but the most stable segmentation. When does character-level tokenization help extract structured identifiers like part numbers and codes accurately? It helps most on alphanumeric codes, part numbers, and chemical identifiers where a single transposed character changes meaning and subword tokenizers split unpredictably. Character granularity gives stable segmentation and exact input alignment, which in our regulated document-automation engagements reduces mis-split identifiers reaching reviewers — an observed pattern in the extraction stage, not a benchmarked rate. What is the cost of character tokenization — longer sequences, slower inference — and when is that trade-off worth it? Character tokenization produces roughly four to five times more tokens than subword schemes for typical text, and because self-attention scales quadratically with length, that means more memory and slower inference. The trade-off is worth it only for the structured fields that carry legal or safety weight; free-text and long narrative content should stay on subword tokenization. Why does the tokenization choice affect whether an extracted value keeps its provenance link back to the source region? When a subword tokenizer fragments an identifier unpredictably, the model’s internal positions no longer map cleanly onto source characters, so the value-to-source-region link becomes approximate. A character-level scheme keeps that mapping exact and stable, which is what lets an extracted value survive an auditor’s question about where it came from. How does character tokenization fit into a staged document-intelligence pipeline rather than replacing it? Tokenization granularity is a lever inside the extraction stage, not a property of the whole system. A staged pipeline can run subword tokenization for narrative text and route structured-identifier fields through character-level handling, so you tune one stage for the audit-weight fields without rebuilding ingestion, layout analysis, or evidence assembly. If the deciding question on your pipeline is where does an extracted value lose its link to the page, tokenization granularity is one of the first places to look before you blame the model — and the fields that break the provenance link are almost always the structured identifiers a subword tokenizer was never designed to keep intact.
