Note
How to Structure a RAG Pipeline You Can Test Stage by Stage
How should a RAG pipeline be structured so each stage can be tested and replaced independently?
Split it into four stages with explicit contracts — embedding (text in, vectors out), storage (vectors and their source passages), retrieval (question in, ranked passages out) and generation (question plus passages in, cited answer out) — and test each stage against its own fixed inputs. Then a weak stage shows up on its own scorecard instead of hiding behind a fluent final answer.
The failure a monolithic pipeline hides
When a RAG system gives a wrong answer, the cause is usually upstream: the right passage was never retrieved, or was chunked so badly it lost its meaning. But if retrieval and generation live in one function, the only thing you can evaluate is the final answer — and a capable model will write a confident, fluent answer from the wrong passage. The bug looks like a model problem and isn't.
Four stages, four contracts
- Embedding — text in, vectors out. Testable on its own: the same text gives the same vector, and near-duplicate passages land near each other.
- Storage — vectors plus the passage, source and metadata they came from. Testable on its own: everything you indexed can be found again with its provenance intact.
- Retrieval — question in, ranked passages out. Testable on its own with a fixed set of questions whose correct source passages you know.
- Generation — question plus passages in, answer with citations out. Testable on its own by feeding it known-good passages, so you measure the model rather than the retriever.
Why 'replaceable' matters as much as 'testable'
In Sarthi, Shreyansh's AI relationship-manager copilot for banking, the pipeline is organised as these separate modules. That made the prototype honest about its own maturity: its retrieval stage could start simple — the first version answered from the bank's policy corpus with straightforward keyword retrieval — behind the same interface a vector retriever implements, so upgrading retrieval doesn't touch generation, and the generation tests keep passing while retrieval improves.
The same separation lets you swap embedding models or vector stores without re-validating the whole system, and makes cost and latency attributable to a stage.
Grounding is a generation-stage contract
Citations belong in the generation contract: the model is given numbered passages and must reference them. In Sarthi's chat, answers cite the bank's own policy identifiers rather than general knowledge — which is only checkable because the passages that went in are known.