The typical rag ai example is a two-step recipe: embed some documents, retrieve the top-k chunks, paste them into a prompt, then judge the whole thing by whether one demo question comes back with a clean answer. It works on the demo. That is exactly the problem. A retrieval-augmented generation system is not a two-step recipe. It is a multi-stage pipeline — chunking, embedding, retrieval, reranking, generation — where each stage has its own failure mode, and where a better-looking answer on one hand-picked query tells you almost nothing about the questions the system will actually field in production. When teams optimize the generation step in isolation and read a single good answer as proof the pipeline works, they are optimizing the demo, not the system. How should you think about a RAG AI example actually in practice? Retrieval-augmented generation puts a lookup step in front of the language model. Instead of asking the model to answer from its parametric memory, you retrieve relevant passages from a corpus you control and hand them to the model as context. The model then generates an answer grounded in that retrieved text rather than in whatever it happened to memorise during pre-training. That is the concept. The mechanism is where the naive example and the production pipeline diverge. In practice, “retrieve relevant passages” decomposes into a chain of decisions: how you split documents into chunks, which embedding model you use to represent them, how many candidates you pull, whether you rerank them, and how you assemble the final prompt. Each of those decisions moves an answer. A demo tuned to a single impressive response gets better at that response. A pipeline tuned against a faithful retrieval-and-answer evaluation gets better at the distribution of questions it will see. Those are different objectives, and they pull in different directions more often than most teams expect. The reason this matters is subtle: a better answer can mask worse retrieval. If your demo question happens to sit near the top of the embedding space, you can degrade chunking or retrieval — pull noisier context, drop the relevant passage entirely — and still get a plausible answer because the model filled the gap from parametric memory. The answer looks better. The system got worse. Without an evaluation structure that holds the corpus, the query set, and the scoring fixed, you cannot tell those two situations apart. What are the stages of a RAG pipeline, and where does each one fail? Treat the pipeline as five stages, because each one fails differently and each one needs its own measurement. Stage What it does Its characteristic failure What to watch Chunking Splits source documents into retrievable units Chunks too large (dilute relevance) or too small (sever context) Whether the answer-bearing text survives intact in a single chunk Embedding Maps chunks and queries into vector space Domain mismatch — a general embedding model that can’t separate your jargon Retrieval recall on domain queries vs. generic ones Retrieval Pulls top-k nearest chunks for a query Relevant passage ranks below the top-k cutoff and never reaches the model Recall@k on a labelled query set Reranking Re-scores retrieved candidates with a stronger model Reranker over-trusts lexical overlap and demotes the right passage Rank of the gold passage before vs. after reranking Generation Produces the answer from retrieved context Model ignores context and answers from memory, or hallucinates over gaps Answer faithfulness scored against the retrieved evidence The stages compound. A chunking choice that severs an answer across two chunks looks like a retrieval failure downstream. A weak embedding model looks like a generation problem, because the model is doing its best with irrelevant context. This is why a single blended score — “did the demo answer well?” — is diagnostically useless. It tells you the pipeline produced one good answer; it does not tell you which stage is carrying the system and which one is quietly broken. Choosing the embedding stage well starts with understanding what a bi-encoder is and when to use it in retrieval evaluation, because the retriever’s representation model sets the ceiling on everything downstream. In practice, the tooling for each stage is mature and well-known. Chunking and orchestration typically run through LangChain or LlamaIndex; embeddings come from sentence-transformers, OpenAI, or Cohere models; retrieval sits on a vector store like FAISS, Milvus, or pgvector; reranking often uses a cross-encoder such as a Cohere reranker or a bge-reranker. The frameworks are not the hard part. Knowing which stage produced which delta is the hard part. How do you evaluate a RAG system so retrieval and faithfulness are measured separately? The single most important structural decision in a RAG example is refusing to collapse the evaluation into one number. Retrieval quality and answer faithfulness are different properties, they fail independently, and they must be scored independently. Retrieval quality asks: did the pipeline surface the passages that actually contain the answer? This is measured on a labelled set — questions paired with the gold passages that answer them — using recall@k and, where ordering matters, mean reciprocal rank. It is a retrieval-only measurement; the generator is not involved. Answer faithfulness asks: is the generated answer supported by the retrieved context, without invented claims? This is scored against the retrieved evidence, not against a global truth. An answer can be factually correct and still unfaithful — if the model produced it from parametric memory while the retrieval failed, the pipeline is broken even though the demo looked fine. Keeping these apart is what lets you attribute a change to a stage. If faithfulness improves but retrieval recall is flat, the generation prompt got better. If recall improves but faithfulness is flat, you fed the model better context and it still ignored it — a generation problem masquerading as a retrieval win. This is the same discipline that separates a real model gain from a measurement artefact in any evaluation, which is why we treat RAG optimization as a case of how machine-learning model optimization works and what it measures rather than as prompt-fiddling. Which run conditions must stay fixed across optimization runs? A RAG delta is only real if the conditions behind it held constant. Change two things at once and you have measured nothing — you have two confounded runs and a number you cannot trust. Hold the following fixed across every comparison, and vary exactly one when you want to attribute a delta: Corpus version. The exact document set and its version hash. Re-ingesting a corpus silently changes what “correct retrieval” even means. Query set. A stable, representative set of questions — not the demo question, and not a set you edit between runs. This is the population the pipeline is optimized against. Top-k. The retrieval cutoff. Widening k trades recall for prompt noise; that trade must be a deliberate, isolated change, not a side effect. Embedding model and version. Swapping embeddings re-shapes the entire vector space. It is a legitimate experiment, but it invalidates any comparison to prior runs that used a different model. Reranker configuration and scoring rubric. The rerank model, its score threshold, and the faithfulness scoring definition must not drift between runs. The discipline here is the same one an evaluation spec enforces across any AI system: the number is only comparable if the task, dataset, scoring, and run conditions are pinned together. We describe that structure in detail in how an evaluation spec links task, dataset, scoring, and run conditions. RAG is not an exception to that rule; it is one of the sharpest cases of it, because it has five moving stages instead of one. How do you tell a genuine RAG improvement from a lucky demo? This is the divergence point the naive example never reaches. A genuine improvement moves the aggregate metrics on the fixed query set: recall@k rises, or answer faithfulness rises, across the distribution — not on one question. A lucky demo moves one answer while the aggregate stays flat or drops. The diagnostic is mechanical once the evaluation is in place. Run the full query set before and after the change under matched conditions. Look at the two metrics separately. If the demo question improved but recall@k on the set fell, the change made retrieval worse and the demo simply got lucky on context the model could paper over from memory. That is the trap: the thing you were staring at got better while the thing you shipped got worse. We see this pattern regularly in early RAG work — a team celebrates a visibly better answer, ships it, and grounding rates fall in production because the “improvement” was a retrieval regression the single-query demo could not surface. (This is an observed pattern across engagements, not a published benchmark rate.) The only defence is an evaluation that scores the population, not the anecdote. Which stage should you tune first, and how does the framework tell you the bottleneck? Tune retrieval before generation, and let the metrics — not intuition — say which stage is the bottleneck. The logic is that generation cannot recover what retrieval never surfaced. If the answer-bearing passage is not in the top-k, no prompt engineering saves the answer; the model is guessing. So the ordering follows the data flow. Establish the retrieval ceiling. Measure recall@k on the labelled set. If the gold passage isn’t being retrieved, fix chunking, the embedding model, or k first. Everything downstream is capped by this number. Add reranking if the candidate pool is right but the ordering is wrong. If recall@(larger k) is high but recall@(final k) is low, the relevant passage is in the pool but ranked too low — a reranker’s job. Only then tune generation. Once retrieval is surfacing the right context, work on the prompt and faithfulness scoring. Gains here are real because the input is now sound. The framework tells you the bottleneck by which metric is farthest from its achievable ceiling. Low recall@k with high faithfulness-on-retrieved-context means retrieval is the ceiling. High recall with low faithfulness means the model is ignoring good context. The bottleneck is always the stage whose metric is lagging, and the separated scores make that unambiguous. A monitoring harness that tracks these deltas over time — rather than metrics captured once at launch — is what keeps the attribution honest as the corpus grows, a concern we cover in what a machine-learning model monitoring framework misses after deployment. When does further RAG optimization stop paying off? Optimization stops paying off when the lagging-stage metric flattens against its structural ceiling and the marginal gain per iteration falls below what the change costs to ship and maintain. The evaluation framework signals this directly: plot recall@k and faithfulness across runs, and when successive changes move them by amounts inside the noise band of the query set, you have reached the plateau for the current corpus and stack. At that point the honest move is to stop tuning and either accept the grounding rate or change something structural — a different corpus, a domain-adapted embedding model, a hybrid lexical-plus-vector retriever. Continuing to fiddle with prompts past the plateau produces demo-level noise, not production gains. The framework’s job is to make that plateau visible so the decision to stop is evidence-based rather than a loss of patience. For teams building this into a broader serving stack, our work on production AI infrastructure for AI-infrastructure SaaS treats the RAG evaluation loop as a first-class part of release readiness, not an afterthought. FAQ How does rag ai example actually work? A RAG example puts a retrieval step in front of a language model: relevant passages are pulled from a corpus you control and handed to the model as context, so the answer is grounded in retrieved text rather than parametric memory. In practice this decomposes into chunking, embedding, retrieval, reranking, and generation — a pipeline, not a two-step recipe. A demo tuned to one impressive answer improves that answer; a pipeline tuned against a faithful evaluation improves the distribution of questions it will actually field. What are the stages of a RAG pipeline, and where does each one fail? The five stages are chunking (chunks too large or too small break relevance), embedding (a general model can’t separate domain jargon), retrieval (the relevant passage ranks below the top-k cutoff), reranking (the reranker demotes the right passage), and generation (the model ignores context or hallucinates over gaps). The stages compound: a chunking failure looks like a retrieval problem downstream, and weak embeddings look like a generation problem. That is why each stage needs its own measurement rather than one blended score. How do you evaluate a RAG system so retrieval quality and answer faithfulness are measured separately? Score retrieval quality on a labelled set — questions paired with gold passages — using recall@k and mean reciprocal rank, with the generator not involved. Score answer faithfulness against the retrieved evidence, checking that the answer is supported without invented claims. Keeping them apart is what lets you attribute a change to a stage: if faithfulness rises but recall is flat, the prompt improved; if recall rises but faithfulness is flat, the model is ignoring good context. Which run conditions must stay fixed across optimization runs so RAG deltas are comparable? Hold the corpus version, the query set, top-k, the embedding model and version, and the reranker configuration and scoring rubric fixed across every comparison, then vary exactly one to attribute a delta. Change two things at once and the runs are confounded — the number means nothing. This is the same evaluation-spec discipline that pins task, dataset, scoring, and run conditions together for any AI system. How do you tell a genuine RAG improvement from an answer that looked better on a demo query but retrieved worse context? Run the full fixed query set before and after the change under matched conditions and read the aggregate metrics separately. A genuine improvement moves recall@k or faithfulness across the distribution; a lucky demo moves one answer while the aggregate stays flat or drops. If the demo question improved but recall@k fell, retrieval got worse and the model papered over the gap from memory — a regression disguised as a win. When you optimize a RAG pipeline, which stage should you tune first, and how does the evaluation framework tell you where the bottleneck is? Tune retrieval before generation, because generation cannot recover context that retrieval never surfaced. Establish the retrieval ceiling with recall@k first, add reranking when the candidate pool is right but the ordering is wrong, and only then tune the generation prompt. The framework identifies the bottleneck as the stage whose metric is farthest from its achievable ceiling — low recall means retrieval is the limit, high recall with low faithfulness means the model is ignoring good context. When does further RAG optimization stop paying off, and how does the evaluation framework signal that? It stops when the lagging-stage metric flattens against its structural ceiling and successive changes move recall@k and faithfulness by amounts inside the query set’s noise band. At that plateau, continued prompt-fiddling produces demo-level noise, not production gains. The honest move is to accept the grounding rate or change something structural — a different corpus, a domain-adapted embedding model, or a hybrid retriever — and the framework’s job is to make that plateau visible so the decision to stop is evidence-based. Where the evaluation has to live A RAG example that ends at a single good answer is a screenshot, not a system. The moment the corpus changes, the retrieval settings drift, or someone swaps the embedding model, that screenshot is worthless — you have no way to tell whether the pipeline still grounds its answers or just still looks good on the one question you remember. The retrieval and grounding deltas you claim are only defensible when the conditions behind them are captured, which is why a RAG pipeline that has to prove itself for procurement needs its evaluation conditions fixed in an evidence pack: our [production AI monitoring harness](Production AI Monitoring Harness) exists to pin the corpus, query set, scoring, and run conditions so the deltas across retrieval and generation are real and not measurement artefacts. The naive example asks whether one answer is good. The right question is narrower and harder: on the questions you will actually field, which stage is your bottleneck — and how will you know when tuning it has stopped paying off?
