A token counter looks like a billing utility bolted onto an API. Count the tokens, multiply by the per-token price, and you have your monthly spend. That reading is not wrong, but it stops one layer too early — the same count that estimates your bill is also the clearest cheap signal you have for where an LLM inference request actually spends its time on the GPU.

Here is the reframe worth holding onto: token count is the primary driver of both the compute work an inference engine does and the latency a user feels, not just the number on the invoice. Input tokens set the cost of the prefill phase. Output tokens set the per-step cost of the decode phase. Read a token counter against measured latency and it becomes a probe into the engine’s compute profile — a way to tell, before you touch a single line of serving code, whether a slow request is slow because the prompt is long or because the generation is long. Those two problems have entirely different fixes.

What does a token counter for an LLM do in practice?

A token counter runs the model’s tokenizer over a string and returns how many tokens that string becomes. That is the whole mechanism. What matters is that the tokenizer is the model’s tokenizer — the same byte-pair or unigram vocabulary the model was trained on — so the count reflects the exact sequence the engine will process, not an approximation of “words.”

Most teams first meet a counter as a pre-flight check: will this prompt fit inside the context window, and roughly what will the call cost? Both are legitimate uses. The count that answers “will it fit” is the same count that answers “how much prefill work will the engine do,” and once you see that equivalence the counter stops being a billing gadget and becomes a compute estimate. A prompt of 4,000 tokens is not just four times more expensive than a prompt of 1,000 tokens on the invoice — it is roughly four times more attention and matrix work in the prefill stage, and that shows up as latency before the first output token ever appears.

We treat the counter as the cheapest measurement in the whole inference stack. It requires no GPU, no warm-up, no load, and it tells you the single most predictive variable for a request’s cost profile.

Input tokens versus output tokens: why the split matters

The most common mistake is treating “tokens” as one number. An inference request has two token counts that behave differently, cost differently, and are fixed by different things.

Input (prompt) tokens are everything you send in: system prompt, retrieved context, chat history, the user’s message. You know this count exactly before the request runs, because you control the input. Input tokens are consumed in a single parallel pass.

Output (generated) tokens are what the model produces. You do not know this count in advance — it depends on what the model decides to say, bounded by your max_tokens cap. Output tokens are produced one at a time, sequentially.

That asymmetry is the entire story. Public API pricing usually charges more per output token than per input token, and that price gap is not arbitrary — it reflects that generated tokens are produced in a fundamentally more expensive way. If you have modelled cost purely as total_tokens × price, you have blurred the two terms that actually explain your latency. The relationship between raw characters and token counts also varies by language and content type, which is why understanding how tokenisation ratios affect inference cost matters before you trust any per-word estimate.

How token counts map to prefill versus decode

Inside the engine, an LLM request runs in two distinct phases, and the two token counts map cleanly onto them.

Prefill processes all input tokens at once. The engine runs the full prompt through the model in a single forward pass, building the key-value (KV) cache for every input position. This phase is compute-bound — it saturates the GPU’s matrix units — and its cost scales with input token count (and, for the attention component, super-linearly as the prompt grows). Prefill is where a 30,000-token document-analysis prompt spends its time, and it is why time-to-first-token climbs with prompt length.

Decode generates output tokens one at a time. Each step runs a forward pass for a single new token, reading the entire KV cache built so far. This phase is memory-bandwidth-bound, not compute-bound — each step does relatively little math but must stream the whole cache from HBM. Decode cost scales with output token count, one GPU step per generated token, and it is why a request that generates 2,000 tokens feels slow even with a short prompt.

This is not a Python-glue distinction. It is the reason the same “slow endpoint” complaint has two unrelated root causes. Engines like vLLM and SGLang are built around exactly this split — techniques such as continuous batching and KV-cache reuse target the two phases separately, and some architectures go as far as separating prefill and decode onto different GPU pools because the phases have such different hardware appetites. A token counter tells you which phase dominates your traffic before you reach for any of that machinery.

Why token count drives latency, not just billing

The billing framing (“more tokens cost more money”) is true but flattening. The latency framing is sharper: input tokens set a mostly-fixed prefill cost you pay once per request, and output tokens set a recurring per-step decode cost you pay for every token generated.

Consider two requests that both bill for 2,100 tokens (as an illustrative decomposition, not a benchmarked figure):

• Request A — 2,000 input tokens, 100 output tokens. A long retrieved context, a short answer. Dominated by prefill. Time-to-first-token is high; once generation starts it finishes quickly.
• Request B — 100 input tokens, 2,000 output tokens. A short question, a long essay. Dominated by decode. Time-to-first-token is low; then the user waits through 2,000 sequential decode steps.

Same invoice line. Completely different user experience and completely different optimisation lever. Trimming Request A’s prompt helps; trimming Request B’s prompt does almost nothing. Capping Request B’s generation helps; capping Request A’s does almost nothing. This is why cost attribution split by tokens-in versus tokens-out — not a single blended token count — is the number that actually guides tuning.

Using a token counter to attribute latency before you optimise

The practical move is to log both counts alongside measured latency and look at the shape of the data. This costs nothing beyond a logging line and it replaces guessing with attribution.

Diagnostic checklist: is this request prefill-bound or decode-bound?

For a sample of real requests, record input_tokens, output_tokens, time_to_first_token, and total_latency, then read them against this rubric:

Signal	Prefill-bound (input dominates)	Decode-bound (output dominates)
Time-to-first-token	High and grows with prompt length	Low and roughly flat
Latency after first token	Small remainder	Large — most of the total
Correlates with input_tokens	Strongly	Weakly
Correlates with output_tokens	Weakly	Strongly
Dominant hardware limit	Compute (matrix units)	Memory bandwidth (HBM)
Lever that moves the number	Trim prompt, cache prefixes, shorten context	Cap max_tokens, speculative decoding, smaller model

If time-to-first-token dominates total latency and tracks input token count, you are prefill-bound: shorten prompts, prune retrieved context, or reuse cached prefixes. If total latency is dominated by the tail after the first token and tracks output count, you are decode-bound: cap generation length, or reach for decode-acceleration techniques. The counter does not fix anything — it tells you which category you are in so you stop optimising the wrong phase. This attribution is exactly the profiling-baseline step that should precede any port-or-tune decision, which is why we treat it as prerequisite work in an inference cost cut sprint rather than an afterthought. For the broader picture of where GPU inference cost hides across the stack, our GPU engineering practice treats token attribution as the entry point, not the whole map.

FAQ

How should you think about a token counter for an LLM in practice?

A token counter runs the model’s own tokenizer over a string and returns the number of tokens that string becomes. In practice it does two jobs at once: it tells you whether the input fits the context window, and it estimates the compute work the engine will do, because token count is the primary driver of both prefill and decode cost.

What is the difference between counting input (prompt) tokens and output (generated) tokens, and why does it matter for latency?

Input tokens are everything you send in and are known exactly before the request runs; they are consumed in one parallel pass. Output tokens are generated one at a time and are not known in advance. The two behave differently — input sets a mostly-fixed prefill cost, output sets a recurring per-step decode cost — so blending them into one number hides what actually drives latency.

How do token counts map to prefill versus decode work inside the inference engine?

Input token count drives prefill: the engine processes the whole prompt in a single compute-bound pass and builds the KV cache, so time-to-first-token grows with prompt length. Output token count drives decode: each generated token is a separate, memory-bandwidth-bound step that reads the whole cache, so total latency after the first token grows with generation length.

Why does token count drive both inference cost and per-request latency rather than just billing?

Because the same tokens that appear on the invoice are the tokens the GPU processes. Input tokens set the prefill work paid once per request; output tokens set the decode work paid per generated token. Reading counts only as billing misses that two requests with identical total tokens can have completely different latency profiles depending on the input/output split.

How do you use a token counter to attribute latency to prompt length versus generation length before optimising?

Log input_tokens, output_tokens, time_to_first_token, and total_latency for real requests, then check which count each latency component tracks. If time-to-first-token dominates and follows input count, you are prefill-bound; if the tail after the first token dominates and follows output count, you are decode-bound. That tells you which lever — trimming prompts versus capping generation — will actually move the number.

How do tokenizers differ across models, and how does that change the counts you get?

Different models use different tokenizer vocabularies and merge rules, so the same text can produce meaningfully different token counts across models. A counter is only accurate when it uses the target model’s own tokenizer; a count from one model’s tokenizer is at best an approximation for another, which is why the ratio of characters or words to tokens is not portable across model families.

When is a long-prompt (prefill-bound) request the constraint versus a long-generation (decode-bound) one, and what lever moves each?

A request is prefill-bound when a long input dominates latency — typical of retrieval-augmented or document-analysis calls — and the levers are trimming the prompt, pruning context, or reusing cached prefixes. It is decode-bound when a long generation dominates — typical of open-ended writing — and the levers are capping max_tokens, speculative decoding, or a smaller model. Matching the lever to the phase is the point of the whole exercise.

If you take one habit away, make it this: never log a single blended token count. Log input and output separately, put them next to time-to-first-token, and let the split tell you which phase owns your latency before you spend an engineering week tuning the one that doesn’t.