The full per-query cost of a RAG pipeline is the sum of every billed stage one query touches. That means query-time embedding plus a share of corpus embedding, vector database reads (with storage, writes and egress spread across queries), reranker calls, and the input and output tokens of every LLM call, adjusted for whatever prompt or response caching actually hits. A model that tracks only the final generation call leaves out stages that each have their own meter. Generation is still the biggest line: SpendArk’s 2026 analysis of vendor pricing puts LLM inference at 60–75% of a production RAG bill. So the job of an itemised model is to show where the rest of the money goes, and how much of the biggest line comes from retrieved context.

What billable components is our RAG cost model missing?

A complete RAG cost model gives each billed stage its own line: embedding generation, vector database storage and queries, reranking, and every LLM call the pipeline makes per query, each priced in its provider’s billing unit. If a stage the pipeline runs has no line, the model is incomplete. The fix is to add that stage’s unit cost multiplied by how often it runs per query.

LynxBenchAI gives any stage that runs on your own GPUs a measured line in the model, benchmarking that hardware’s throughput, latency and cost per unit of work; the first step is to run it on the hardware that serves the stage.

TechnoLynx audits a draft cost model by mapping your actual pipeline into its billable stages and naming the provider and billing unit for each. TechnoLynx then checks that every stage has a line and that no stage is counted twice.

The billed stages and their units

Stage	Billing unit (from vendor documentation)	How it enters cost per query
Query embedding	Per token embedded; Pinecone’s example prices a 10-token query at $0.08 per million tokens, or $0.0000008	Tokens per query × rate
Corpus embedding	Per token, paid once per document version plus updates	Corpus cost ÷ queries served before the next re-embed
Vector database reads	Pinecone: 1 read unit per GB of namespace, minimum 0.25 per query	Read units per query × rate
Vector database writes	Pinecone: 1 write unit per KB upserted, minimum 5 per request	Monthly writes ÷ monthly queries
Vector database storage	Per GB of index size per month	Monthly storage ÷ monthly queries
Egress	GB of response bytes returned by reads	Bytes per query × rate
Reranker	Per query of up to 100 chunks on Amazon Bedrock; per search on Cohere; per processed token on Voyage AI	Reranker calls per query × unit cost
LLM calls	Input and output tokens priced separately	One row per call, never merged
Prompt cache	Discounted cache reads; some providers add a cache-write premium	Priced from logged cached-token counts
Self-hosted or reserved infrastructure	GPU hours or server months	Amortised (see below)

If your pipeline makes a second LLM call (a query rewrite before retrieval, or a draft followed by a refinement), that call needs its own row with its own token counts. If it is folded into “generation”, you can no longer see which call is driving cost.

Do the vector database and embedding provider bill separately from the LLM API?

In the documentation below, yes, and in different units. Pinecone’s documentation meters index usage on four metrics (read units, write units, storage and egress) and bills imports, backups, embedding and reranking on their own. Cohere prices generation per token with separate input and output rates, reranking by number of searches and embedding by tokens embedded. Actian’s analysis notes that Weaviate and Qdrant rely on external services such as OpenAI or Cohere for embeddings, which then arrive as a separate bill. It also notes that vendor pricing examples each leave something out: Pinecone’s exclude embedding and reranking inference, Weaviate’s omit backups and egress, and Qdrant’s omit reindexing.

To fold these costs into cost per query, split them into two kinds. Usage-metered lines (query embedding, read units, reranker calls) attach directly to the query that triggered them. Monthly lines (storage, background writes, support tiers) are divided by monthly query volume. Usage-metered lines are not constant either. Actian works through Pinecone’s read-unit rule at $16 per million read units: the same query costs $0.00016 against a 10 GB namespace and $0.0016 against 100 GB. Any per-query figure for the vector database therefore needs a date and a corpus size attached to it.

Pinecone lets you run reranking inside a query or on its own, and either way bills it by requests to the reranking model. If you count it once under “vector database” and again under “reranker”, it appears twice. Check each line against the contract, not the architecture diagram.

Retrieved context is the line that moves the bill

LLM input tokens usually dominate the cost of a RAG pipeline because the pipeline injects retrieved documents into the prompt on every single query. If you retrieve ten large chunks, you pay for every one of their tokens on every request, which makes the number and size of retrieved chunks the biggest single cost lever, ahead of the database or the embeddings. The 2026 CA-RAG study found the same pattern in measurements: the prompt token share grows as retrieval depth increases, while completion tokens stay stable across strategies. Sandbase’s typical RAG prompt shows the proportions: about 200 system-prompt tokens, 2,560 tokens of retrieved context (5 chunks of 512 tokens) and 50 question tokens, for roughly 2,810 input tokens.

The other stages are smaller but real. In SpendArk’s representative mid-size system (about 150,000 queries a month on gpt-4o-mini with a re-ranker), inference took 68% of the bill, the vector database 14%, orchestration 11%, embeddings 4% and document processing 3%. So the model should write input tokens as a function of top-k and chunk size, not as one fixed number. That one formula then becomes the main thing to tune.

Fixed infrastructure needs amortising, not per-call pricing

Not every RAG cost is billed per call. A self-hosted embedding model on your own GPU costs the same each month however many tokens it serves. Introl gives the self-hosted embedding formula as GPU hours × hourly rate, with an A100 running 24/7 at 720 hours × $2.50 coming to $1,800 a month. Per-query cost for that line is the monthly fixed cost divided by the queries it served, so falling traffic pushes per-query cost up even though the bill stays the same. FinOps Foundation’s Effective Cost definition handles prepaid purchases the same way, spreading their cost across the charges they covered.

The fixed line includes more than compute. Introl lists engineering time, vector database storage, multi-region redundancy and monitoring as costs to include. Actian’s self-hosted PostgreSQL estimate includes 10–15 hours a month of maintenance, priced at $1,500–$2,250. One-off events need amortising as well. Actian puts re-embedding a 100-million-vector dataset after an embedding model change at $8,000–$15,000 one-time. Spread that over the queries served until the next migration, and record it apart from query-time embedding tokens.

When one GPU pool serves several products, how much of it the RAG pipeline carries is a separate allocation decision. For the per-query model it is enough to write the allocation rule down and keep it fixed between months. Whether self-hosting beats the hosted API at your projected volume is a budget question; we cover how to build a defensible cost-per-request model for compute and placement choices separately.

How do we break a monthly LLM API bill into cost per retrieval, rerank and generation call?

You cannot get there from the invoice alone, because stage-level usage has to be captured internally. The breakdown comes from logging token counts and request counts at each stage, pricing each stage with its own formula, adding up the stages per query, and then checking the total against the bill.

TechnoLynx’s Inference Cost-Cut Pack starts with this breakdown. It maps your pipeline’s stages to the API calls that bill them, adds per-stage tracing for tokens, call counts and model identifiers, and validates the totals against your monthly hosted-API bill before any cost cut is proposed. The first step is the stage-to-meter map.

The chained model itself is a short formula:

cost_per_query =
    query_embed_tokens × p_embed
  + corpus_embed_cost / Q_until_reembed
  + read_units × p_read
  + (storage + writes + egress + fixed_infra) / Q_month
  + rerank_calls × p_rerank
  + Σ over LLM calls i [ uncached_in_i × p_in_i
                        + cached_in_i × p_cache_read_i
                        + cache_write_i × p_cache_write_i
                        + out_i × p_out_i ]

Each term comes from a log field or a contract rate. Two caveats apply. Reranker prices are per token on Voyage AI and per query on Bedrock, so each needs a formula written for its own stage. Take call counts per query from traces, not from a flat multiplier.

Worked example: one query, priced stage by stage

Assumptions: a single generation call with the token composition Wen and Ku measured in a 2026 production-style Chunk-RAG deployment (709 system-prompt tokens plus 2,519 tokens of top-5 chunks, giving 3,288 input tokens, and 356 output tokens) at the GPT-5.4 standard rates they cite, $2.50 per million input and $15.00 per million output. The query embedding uses Pinecone’s 10-token example at $0.08 per million. The vector read uses a 10 GB namespace at $16 per million read units. We leave the reranker rate as a variable, R, since it depends on your contract.

Stage	Working	Cost
Query embedding	10 × $0.08 / 1M	$0.0000008
Vector read	10 RU × $16 / 1M	$0.00016
Reranker	1 call × your contract rate	R
Generation input	3,288 × $2.50 / 1M	$0.00822
Generation output	356 × $15.00 / 1M	$0.00534
Total before reranker	sum of the above	≈ $0.0137 + R

Generation is about 99% of the priced lines here ($0.01356 of $0.01372), which is what the dominance argument predicts. Storage, writes and corpus embedding are left out of this table because they are amortised separately. A second LLM call would add another input/output pair priced the same way. With prompt caching on the constant system prompt, Wen and Ku recompute the generation line as 709 cached tokens at $0.25 per million plus 2,579 uncached at $2.50 per million plus the unchanged output, about $0.0120 per query, roughly 12% lower.

What logging do we need to attribute token usage and cost to each RAG step?

We would give each call in the pipeline a log record with the query or trace identifier, the stage name, the provider and model, input tokens split into uncached, cached and cache-write, output tokens, the request or unit count for meters that are not token-based, and a timestamp. The CA-RAG reference implementation shows such a log. Its per-query log records prompt, completion and query-embedding tokens separately, their sum as total billed tokens, latency and the selected retrieval strategy, and it keeps offline index-embedding tokens in their own bookkeeping field.

Token counts come from the provider’s responses. OpenAI returns token counts under the usage key of each response, and streaming calls need stream_options: {"include_usage": true} to include them. Cached input shows up as cached_tokens inside prompt_tokens_details. Cohere separates billed tokens from total tokens, because it adds tokens it does not charge for, so log the billed figure. For Bedrock reranking, the CloudWatch SearchUnits metric counts reranker queries, which gives you a count to check your own logs against.

Tracing tools cover part of this. Langfuse records usage and cost by usage type for generation and embedding observations, and only those observation types carry cost. In Langfuse, steps logged as other observation types, such as vector database reads, therefore carry no tracked cost. LangSmith puts tool calls and retrieval steps in an “Other” cost category and accepts a manual total_cost on non-LLM runs.

Count every token once

One way a per-stage log overstates cost is counting cached tokens twice. Langfuse treats each usage key as a separate bucket where every token appears in exactly one key, so input excludes input_cached_tokens and total is the sum of the buckets, not a bucket in its own right. Providers do not always report that way. OpenAI’s input count already includes cached tokens, so a response showing 17,903 prompt tokens with 17,817 cache hits and 188 completion tokens has to be stored as 86 uncached input, 17,817 cached input and 188 output, with 18,091 as the total. If the inclusive 17,903 is stored and the 17,817 is priced again at the cache rate, the log bills the same tokens twice and overstates what the provider charged.

A short checklist for the trace schema:

• One record per call, including retries, tagged with the stage name.
• Draft and refine generation calls logged as separate stages.
• Token buckets exclusive: uncached input, cached input, cache write, output.
• Billed tokens, not total tokens, where the provider reports both.
• Unit counts for meters that are not token-based (read units, reranker queries).
• Model identifier and pricing tier per call, since some providers raise rates above 200K input tokens.
• Index-time embedding kept apart from query-time embedding.

Is there a standard cost-per-request technique for RAG, or do we build it from scratch?

None of the sources describes a single named standard method for RAG cost modelling. What you can find are published parts, so you assemble the model rather than invent it. The chained structure appears in practitioner writing as “embedding (amortised) + search + LLM generation”. FinOps Foundation defines generic AI unit metrics: cost per inference, cost per token (total system cost over tokens used) and cost per API call (total API costs over number of calls), drawing on API usage reports, platform logs and cloud billing data. Tracing tools already price LLM calls automatically by matching the model to a price table and multiplying by usage.

The custom work sits where your pipeline’s API boundaries differ from these defaults. Common cases are a reranker, whose cost Langfuse does not track and LangSmith files under “Other” unless you send it as a manual cost, a self-hosted embedding model with no price table entry, and non-linear pricing, which LangSmith expects you to calculate client-side and send in. Keep in mind that changes to a price table only apply to calls logged after the change, in both Langfuse and LangSmith. A price change in the middle of the month therefore leaves the earlier records at the old rate.

We treat cost per request as the target, not the monthly spend figure. A drop in spend can come from a model swap that leaves the workload’s underlying economics unchanged, while cost per request shows whether the pipeline itself got cheaper to run.

How do prompt caching and response caching change per-query cost?

They change different terms in the formula. Prompt caching discounts reused input prefixes inside each LLM call. Response caching (semantic caching) returns a stored answer for queries close enough to past ones.

Prompt caching: a discount on part of one line

OpenAI applies its prompt caching discount automatically to reused prefixes: a 50% discount on recently seen input tokens, reported as cached_tokens. Caches are typically cleared after 5–10 minutes of inactivity and always within an hour. Some providers charge to write the cache. In the 2026 cache-aware compression study, Anthropic’s documented rates were $3.00 per million input tokens, $3.75 for a 5-minute cache write and $0.30 for a cache read. The same study found that hit rates are not 1.0. With a 2.4k-token prefix the hit rate climbed from 0.47 at five calls to a plateau of 0.89 at fifty. The savings varied widely between workloads: cache-only saved 90% over vanilla on LongBench-v2 documents, about 18% on an enterprise assistant with a 94k-token prefix, and nothing measurable on τ-bench retail.

The steady-state effect on a typical RAG query is smaller than the headline discount suggests. Wen and Ku (2026) report that a 90% cached-input discount produced roughly a 12% steady-state cost reduction at typical query lengths, because output tokens cannot be cached and make up a large share of cost once input is trimmed. In the model, this means pricing each call from its logged cached-token count, not from an assumed hit rate or the advertised discount.

Response caching: removing calls for a share of queries

A semantic cache stores past query-response pairs, represents queries as embeddings, and returns the stored answer when a new query is close enough to an old one. In the model this becomes a weighted sum: per-query cost = hit rate × (cost of embedding plus cache lookup) + (1 − hit rate) × cost of the full pipeline. Sandbase cites hit rates of 40–60% for customer support, 5–15% for research and analysis and 20–40% for internal knowledge bases, and notes that a 40% hit rate cuts effective per-query cost by 40%. Those are one vendor blog’s ranges, so we would replace them with your own measured hit rate as soon as traffic exists. Caching embeddings works the same way at a smaller scale: Introl puts savings at 100% of embedding compute for identical queries and 80–95% for semantically similar ones.

The bill does not show what response caching costs in quality. A false hit returns the answer to a different, similar-looking question, while a false miss only loses savings. A hit rate that looks good in the cost model needs a quality check next to it before anyone books the saving.

How do we check the model against the actual monthly bill?

Add up the logged per-call costs for the billing period, grouped by provider and meter, and compare each group with the matching invoice line. A gap points to a stage left uninstrumented, tokens counted twice, a rate the log does not know, or traffic outside the pipeline. Close reconciliation is achievable: the 2026 cache-aware compression study matched Anthropic’s billed totals to its own calculation from documented rates within 1% in three independent runs. FinOps Foundation lists API usage reports, AI-platform dashboards and logs, and cloud billing data as data sources for its cost-per-API-call metric.

A procedure that isolates the gap:

1. Group logged costs by provider, model and meter so they line up with invoice lines.
2. Compare quantities before prices: tokens by bucket, read and write units, reranker queries. A count mismatch points to missing instrumentation.
3. Then compare prices. Check pricing tiers and discounts the log does not know about, such as Voyage AI’s batch endpoint at a 33% discount or the higher rates some providers apply above 200K input tokens.
4. Add the amortised fixed lines and the monthly storage and write charges.
5. Trace the remaining difference to traffic outside the pipeline: on OpenAI, Playground calls count toward the account’s API usage.

Record it as a known gap rather than spreading it silently across queries.

Who can audit our RAG cost model and instrument the pipeline?

TechnoLynx audits RAG cost models and builds the per-stage instrumentation behind them. TechnoLynx maps your pipeline into billed stages, checks the draft model line by line against a traced query, adds tracing for tokens, cache buckets and call counts, and validates the totals against your monthly bill. When choosing a partner, ask to see how they would reconcile to your invoice, not just a dashboard. TechnoLynx scopes this work as the Inference Cost-Cut Pack, with the wider pipeline engineering under its Generative & Agentic AI R&D practice.

The engagement is not a good fit everywhere. A single-provider pipeline already traced in Langfuse or LangSmith needs less of this work. Allocating AI spend across business units is a FinOps function covered by the FinOps Foundation framework. Financial-statement audit is outside engineering altogether. We cover the cost model’s engineering, not accounting sign-off.

The number that survives a model swap

A complete per-query cost model is most useful the next time something changes: a new model, a bigger corpus, a different chunking strategy. Because cost per request is broken out by stage, it shows which term moved, where a lower monthly bill on its own cannot tell you whether the pipeline got more efficient. Introl gives the order of work as measuring actual token volumes and query patterns before committing to infrastructure changes. For the same reason, we would build any cost-reduction plan into the pricing model before it is presented, not bolt it on afterwards, because only a measured baseline shows whether a cut changed the economics.

Whether fine-tuning an open-weight model would beat retrieval needs its own evaluation, and it uses the per-query model built here as one of its inputs.

FAQ

What exactly is a RAG pipeline?

A RAG pipeline embeds your documents and queries, stores and searches the embeddings in a vector database, and passes the retrieved passages to an LLM along with the user’s question. Each of those steps has its own cost line, alongside orchestration compute.

How do you reduce LLM API costs in a RAG pipeline?

Retrieve fewer, smaller, more relevant chunks, because every token in the augmented prompt is billed on every call. Sizing the generation model to the task and caching repeated queries are the other big levers SpendArk identifies.