A retrieval-augmented chatbot never costs what its generation model costs. One user turn fans out into an embedding call, a vector search, an optional rerank, one or more retrieval-conditioned generation calls, and often a guardrail or summarisation pass — each with its own token profile, its own latency, and its own line on somebody’s bill. Price the feature off the answer model’s per-token rate and you have quoted a vendor price list, not measured a workload.
The tell is simple. Increase top-k from 4 to 12 and watch your cost figure. If it does not move, the figure is not a cost-per-request number.
Which calls make up a single RAG chatbot turn?
Before anything can be costed, the turn has to be enumerated as a call stack rather than as “a chatbot response”. A typical production path looks like this, and every hop consumes something billable:
| Hop | What it consumes | Scales with |
|---|---|---|
| Query embedding | Embedding tokens (query only — small, but per turn) | Query length; multi-query expansion multiplies it |
| Vector search | Vector DB read units or GPU/CPU-seconds on a self-hosted index | Index size, top-k, filter complexity |
| Rerank (optional) | Cross-encoder inference over k candidate pairs | Rerank depth (k), candidate chunk length |
| Generation | Prompt tokens (system + history + retrieved context) plus output tokens | Retrieved-context size, conversation history depth |
| Guardrail / moderation | A second model pass over input, output, or both | Whether it runs on input, output, or both |
| Summarisation of history | Periodic extra generation call amortised across turns | Turn count before compaction triggers |
Retrieved context is usually the dominant term in a RAG chatbot’s per-turn cost, because it lands in the generation call’s prompt-token count and is re-paid on every turn it stays in the window. That is the structural fact the naive model misses: retrieval quality work — bigger chunks, more of them, more overlap — is paid for at generation time, not at retrieval time.
We see this pattern regularly when a team’s answer-quality sprint lands and the inference bill moves a fortnight later, with nobody able to say which change caused it. The retrieval config changed; the cost model never had a term for it.
Why embedding and vector search feel free and are not
Compared to a generation call over several thousand prompt tokens, a query-embedding call is genuinely small. Practitioners are right that the generation hop usually dominates a text chatbot’s per-turn spend. Where the reasoning goes wrong is in the conclusion drawn from that — “so we can ignore the retrieval side” — because the retrieval side is what sets the generation hop’s size.
Two costs also hide outside the token ledger entirely. The first is the standing cost of the vector index: a self-hosted index on GPU or a managed tier with provisioned capacity is charged whether or not anyone chats, so at low traffic the per-request share of that fixed cost can exceed the generation cost outright. The second is reranking, which is inference in its own right — a cross-encoder scoring 50 candidates runs 50 forward passes for one user turn, and rerank depth is a cost knob that no per-token price list will surface for you.
Profiling makes each of these individually costable rather than lumped, which is why the decomposition work usually starts with a trace, not a spreadsheet.
Building the per-turn figure
The method is mechanical once the stack is written down. Attribute a unit cost to each hop, multiply by the per-turn quantity, and sum:
- Tag the turn at ingress with a conversation ID, a retrieval config version, and a cache-hit flag, so the hops can be reassembled into one request later.
- Emit token counters per hop — embedding tokens, prompt tokens split into system / history / retrieved-context, and output tokens. Splitting the prompt count is the step teams skip, and it is the step that makes the retrieval lever visible.
- Convert self-hosted hops to GPU-seconds, then to currency at your effective accelerator-hour rate including idle reservation. A rerank model sitting on a dedicated GPU at 12% utilisation costs what the whole GPU costs.
- Amortise the periodic calls — history summarisation, index refresh — across the turns they serve, rather than dropping them.
- Divide fixed index and endpoint cost by realistic traffic, and re-derive it at your actual traffic shape rather than a hoped-for one.
- Report the result decomposed by hop, not as a single number. A single number tells you the feature is expensive; the decomposition tells you which hop to attack.
The output is a per-turn cost figure with five or six named components, and it should survive both a retriever change and a model swap — if swapping the generation model invalidates the whole model, the model was really a price quote for that vendor.
Which levers actually move the number
The decomposition earns its keep by ranking the levers. In practice, on the retrieval-augmented chatbots we have worked on, the ordering tends to run: context trimming first, cache next, rerank depth after that (observed across TechnoLynx engagements; not a published benchmark, and the order flips when traffic is heavily repetitive).
- Retrieved-context trimming — cutting top-k, tightening chunk size, or reducing overlap directly reduces generation prompt tokens on every turn. It is also the lever most likely to cost answer quality, so it is measured as a cost-per-request delta and an answer-quality delta, never one alone.
- Cache hit rate — repeated or near-repeated queries are common in support chatbots. Caching at the embedding, retrieval-result, or full-answer level each has a different hit rate and a different staleness risk. Prefix caching in the serving runtime is a separate lever again, since it attacks the shared system-prompt portion of the prompt tokens.
- Rerank depth — halving candidate depth halves a hop that is pure inference cost, and often costs less quality than trimming the context that reaches the generator.
- History compaction policy — deciding when to summarise rather than carry raw turns changes the growth curve of prompt tokens across a conversation.
Each of these carries a p95 latency delta alongside the cost delta. Caching cuts both; reranking less aggressively cuts both; trimming context cuts cost and can raise cost later through follow-up turns when the answer was incomplete. That second-order effect is why a per-turn figure eventually needs a per-conversation companion, which is a different measurement problem than this one.
Setting the SLO
Once the stack is decomposed, a cost-per-request SLO per conversation turn becomes enforceable: a target figure, a decomposition budget per hop, and an alert when any hop’s share drifts. Tracked against active chatbot users, it converts directly into gross-margin-per-user, which is the form the number needs to be in before a VP of Engineering and a CFO can have the same conversation about it.
The wider argument for treating per-request cost as the control variable for production AI — rather than monthly spend — is developed in our production AI cost and infrastructure work. Where a buyer wants this decomposition run against their own deployed serving path rather than a worked example, that is the job of the Inference Cost-Cut Pack.
What remains genuinely unsettled is the quality side of the trade. Cost per hop is measurable to the token; the quality cost of removing a chunk is not, and every team we work with ends up building its own evaluation set to make that half of the equation legible. Until that exists, cost optimisation on a RAG stack is a one-sided ledger.
Frequently Asked Questions
What does cost-per-request applied to a RAG-augmented chatbot workload mean in practice?
Why do RAG chatbots cost 3-5× more per request than vanilla LLM calls? Cost Per Request RAG turns on one distinction. Cost Per Request RAG is simpler than it looks. Cost Per Request RAG turns on one distinction. Stripped down, Cost Per Request RAG is the following. Cost Per Request RAG is simpler than it looks. Cost Per Request RAG turns on one distinction. Cost Per Request RAG is simpler than it looks. Cost Per Request RAG turns on one distinction. Stripped down, Cost Per Request RAG is the following. Cost Per Request RAG is simpler than it looks. Cost Per Request RAG turns on one distinction. Cost Per Request RAG is simpler than it looks. Cost Per Request RAG turns on one distinction. Cost Per Request RAG rarely needs a long answer. Stripped down, Cost Per Request RAG is the following. In the context of CostPerRequest for a RAG Chatbot, it means pricing one user turn as the sum of every call that turn triggers — embedding, vector search, optional rerank, generation, guardrail — rather than pricing the generation call alone. The resulting figure is decomposed by hop, so optimisation effort can be aimed at the hop that actually carries the spend., a typical turn is query embedding, vector search, optional rerank, one or more retrieval-conditioned generation calls, and often a guardrail or summarisation pass. Each hop is attributed by emitting its own token counters or GPU-seconds under a shared request ID, converting self-hosted hops at your effective accelerator-hour rate, and amortising periodic calls across the turns they serve.
How does retrieved-context size (top-k, chunk size, overlap) change cost-per-request?
Retrieved context lands in the generation call’s prompt-token count, so increasing top-k, chunk size, or overlap raises per-turn cost on every turn that context is present. This is the divergence test: if your cost figure stays flat when top-k rises, the figure is not modelling the workload.
How do embedding and vector-search costs compare to generation cost in a realistic chatbot?
Query embedding is small, and generation usually dominates a text chatbot’s per-turn token spend. The important qualification is that retrieval sets the size of the generation hop, and the vector index carries a standing cost that is charged regardless of traffic — at low volume its per-request share can exceed generation.
How should multi-turn conversation history be accounted for in a per-request cost figure?
History is a prompt-token term that grows with conversation depth, so it belongs in the per-turn split alongside system prompt and retrieved context. Periodic summarisation or compaction calls are separate generation calls and should be amortised across the turns they serve rather than dropped.
Which levers — caching, rerank depth, context trimming — give the largest cost-per-request reduction without degrading answer quality?
Caching usually gives the cleanest reduction because it removes work rather than shrinking evidence, and rerank depth is often the next-safest cut. Context trimming moves the number most but carries the highest answer-quality risk, so it should always be measured as a paired cost delta and quality delta.
How do you set and monitor a cost-per-request SLO for a chatbot turn once the stack is decomposed?
Set a target per-turn figure plus a budget for each hop’s share, then alert when any hop drifts outside its share rather than only when the total breaches. Tracking the SLO against active chatbot users converts it into gross-margin-per-user, which is the form the number needs for a commercial conversation.
Decomposing RAG costs by pipeline stage
Embedding, retrieval, reranking, and generation each carry distinct cost and latency profiles—measure them separately to know where optimization pays off. Revisit it when your workload shifts.
