How to debug a RAG pipeline that returns wrong answers
Your RAG app worked in testing and now returns wrong or fabricated answers in production. The reflex is to open the prompt and start rewriting it. That is almost always the wrong first move.
Debug it by isolating the stage that failed before you change anything. A RAG answer can go wrong in three places, and the fix for each is different, so guessing wastes your time. Inspect what was actually retrieved first, then test generation in isolation with known-good context, then work outward to chunking, embeddings, filters, and index freshness. The reason the prompt is the wrong place to start: if retrieval never fetched the right chunk, no amount of prompt wording will conjure it.
The four stages where a RAG answer goes wrong
Every wrong answer traces to one of four stages:
- Retrieval: the right chunk was never fetched. The model answered from nothing, or from the wrong context.
- Context assembly: the right chunk was fetched but then truncated, buried in the middle of a long context, or stripped of the attribution the model needed.
- Generation: the context was good and the model still ignored it or fabricated over it.
- Stale index: the source document changed and the index did not, so retrieval faithfully returns an old answer.
The whole reason to localize is that RAG inherits faults from both an information-retrieval system and an LLM. The experience report Seven Failure Points When Engineering a RAG System (Barnett et al., 2024) makes this concrete across three real deployments and lands on a blunt conclusion: a RAG system’s robustness “evolves rather than [is] designed in at the start,” and validation “is only feasible during operation.” You find these failures in production. So you need a method built for production, because testing will not have caught everything.
One caveat before you inherit a number you will see everywhere: the claim that “70% of RAG failures are retrieval, not generation” is a widely-repeated heuristic with no primary source behind it (the same figure floats around as 60% elsewhere). The direction is well-supported by the failure taxonomy above and by broad practitioner consensus: retrieval and upstream-data faults are the most common and most under-diagnosed cause, and people wrongly blame the LLM first. Treat it as a rough direction. Do not cite it as a measured statistic.
Step 1: inspect what was actually retrieved
Do not read the answer. Read the retrieval. For the failing query, pull the exact chunks the retriever returned and their similarity scores, and check four things in order:
- Is the answer even in your corpus? Search the store directly. If the fact was never ingested, no retrieval tuning will help. The problem is upstream in your data, and no RAG tuning will touch it.
- Did retrieval return the relevant chunks at all?
- Were they ranked high enough to be used? A chunk retrieved at position 15 when you only pass the top 5 to the model is, functionally, not retrieved.
- Is the context complete, or is the answer split across chunks so each one is a fragment?
This one inspection usually tells you whether you are dealing with a retrieval problem or need to move to Step 2. If the right chunks are not there, stop, you have localized it, and the prompt was never the issue.
Step 2: test generation in isolation
This is the single cleanest fault-localization test in RAG. Manually hand the model known-good context and run generation.
- If it now produces a good answer, your generation is fine and retrieval is the problem.
- If it still fails with perfect context in front of it, generation is the problem: the model is ignoring the context, misreading it, or overriding it with its training data.
That one experiment splits your entire search space in half without touching the prompt or the retriever config. Do it before anything else in this section.
Step 3: walk outward to root causes
Once you know which side failed, the fixes are well-catalogued. (The technique names below are the general approaches; frameworks like LlamaIndex ship named implementations, but they are only examples of many valid ways to do this.)
| Symptom | Likely cause | Fix |
|---|---|---|
| Chunks share keywords but are off-topic | Semantic-similar-but-wrong retrieval | Add a reranker; raise top_k and let the reranker prune; hybrid (semantic + keyword) search |
| Partial answers; works when you paste the full text | Poor chunking (info split across chunks) | Larger or overlapping chunks; sentence-window or hierarchical parsing |
| Quality dropped suddenly after a code or dependency change | Config / embedding-model drift (query and docs embedded differently) | Pin the embedding model name and version; if you change it you must rebuild the whole index |
| Pulls from the wrong document set (last year’s report) | Missing or too-loose metadata filtering | Structured metadata at ingest; metadata filters at query time; separate namespaces |
| Different answers on different days; contradictory chunks | Stale or fragmented index | Doc-id tracking, scheduled re-index, a freshness check |
| Good chunks verified, answer still wrong | Synthesis failure | Instruct “use only the provided context”; stronger model or lower temperature; a refine synthesizer |
A note on context assembly specifically, since it is the easiest to miss: even with the right chunks retrieved, ordering matters. Liu et al., Lost in the Middle (TACL 2023) found models use information best when it is at the beginning or end of the context and significantly worse when it sits in the middle, a U-shaped curve that held even for long-context models. So burying your best chunk in position 8 of 15 can produce a wrong answer from correct retrieval. Rerank so the most relevant chunk lands first (or last), and do not stuff chunks in arbitrary order.
Doing this in production, not on your laptop
Everything above works beautifully on one query at your desk. In production the hard part, per the RAG-debugging tool paper RAG Without the Lag (Romero Lauro et al., 2025), is exactly that retrieval and generation are “intertwined, making it hard to identify which component(s) cause errors in the eventual output.” Their answer, drawn from a study of 12 engineers, is to inspect the intermediate states interactively before changing any config. You cannot debug what you cannot see.
That is the layer we use Latitude for. Every request is captured as a trace: the transformed query, the exact chunks retrieved and their scores, the assembled prompt, and the final answer, with the failing step localized in the trace. So instead of reproducing a bad answer by hand, you replay the actual failing request and see which stage broke, which is Step 1 and Step 2 above but on real production traffic rather than a laptop repro.
For recurring wrong answers, semantic session search clusters the same wrong-answer class into a Behavior or a Signal with an occurrence count over a window, so “it sometimes answers strangely” becomes “this specific wrong-answer pattern happened 12 times over the last 12 days, on 4% of traces, detected automatically on every trace without a separate evaluation.” And when a fix is warranted, Latitude can hand the failure to your own coding agent over MCP to open a PR against the retrieval or chunking bug and promote the failing traces into a regression set. A human reviews and merges; Latitude drives the agent and opens the PR, it does not fix your RAG for you.
The honest tradeoff: capturing full traces with retrieved chunks on every request is real storage and instrumentation overhead, and clustering runs an LLM over your sessions, which costs money. You are buying the ability to localize a fault in seconds instead of reproducing it by hand, and for a laptop-scale project that trade may not be worth it yet.
A regression query set so it doesn’t come back
Whatever you fix, capture it so it cannot silently return. Keep a golden query set: each entry is a query, the chunks that should be retrieved for it, and the answer that should come out. Re-run it after every pipeline change and every deploy. A failing query becomes a new golden entry; a passing baseline query is what you diff against when the next thing breaks. This is what turns a one-off debugging session into a system that stays fixed.
FAQ
Why shouldn’t I just fix the prompt first? Because most wrong RAG answers are decided before the prompt ever runs. If retrieval fetched the wrong chunks, prompt wording cannot recover information that was never in the context. Localize the failing stage first; the prompt is often innocent.
How do I tell a retrieval failure from a generation failure? Hand the model known-good context manually and run generation. Good answer means retrieval was the problem; still-wrong answer means generation is the problem. It is the fastest single test in RAG debugging.
My retrieval looks fine but answers are still wrong. Now what? Two usual suspects. Context assembly: the right chunk is retrieved but buried in the middle of a long context or truncated (see lost-in-the-middle), so reorder and rerank. Or synthesis: the model is ignoring good context, so constrain it to answer only from the provided context and consider a stronger model.
Is bad chunking really that common? Yes, and it is under-diagnosed. When answers are partial but paste-the-full-text works, chunking split the information. Tune chunk size and overlap, or use sentence-window or hierarchical parsing so a complete thought survives in one chunk.
It works locally but returns random answers in production. Why? Usually an index or deployment problem: a half-built index, an environment variable pointing at the wrong index, or a stale vector store. Treat the index as a versioned artifact and health-check it with a known-good query right after each deploy.