AI · 7 MIN READ
Your RAG system is failing at retrieval, not generation
When an assistant gives a bad answer, the instinct is to rewrite the prompt. Usually the correct passage was never fetched in the first place.
AB19 Innovations
In short
Most retrieval-augmented generation failures are retrieval failures. Evaluate retrieval separately from generation, use hybrid search with reranking, and chunk along document structure rather than fixed sizes.
There is a predictable sequence to debugging a retrieval-augmented system. The answer is wrong. Someone rewrites the prompt. It is still wrong, differently. Someone tries a larger model. It is now expensively wrong. Nobody has yet checked whether the passage containing the answer was ever retrieved.
It usually was not. Generation is the part of the stack that modern models have largely solved: give a capable model the right context and it will write a good answer. Deciding what the right context is remains an ordinary information-retrieval problem, and it is where the errors concentrate.
Measure the two halves separately
The single most useful change you can make is to score retrieval on its own. Build a set of questions with the passages that should be returned for each, then measure whether they were — before any generation happens.
- Recall@k: was the correct passage anywhere in the top k results?
- Precision: how much of what was retrieved was actually relevant?
- Per query class: identifier lookups, conceptual questions and comparisons fail differently, so never average them together.
Once retrieval is scored independently, a bad answer becomes diagnosable. Either the passage was missing, in which case no prompt will help, or it was present and the generation step mishandled it, which is a different and much easier fix.
Fixed-size chunking destroys structure
Splitting documents every 500 tokens is the default in every tutorial and it is wrong for most real material. It severs tables from their headers, separates a clause from the definition it depends on, and produces fragments that are individually meaningless.
Chunk along the document’s own structure instead: sections, headings, table boundaries, list groupings. Carry the heading path into each chunk so a fragment still says what it belongs to. This one change routinely moves retrieval quality more than any embedding model swap.
Pure vector search misses exact terms
Embeddings capture meaning, which is exactly wrong when someone searches for a part number, an error code or a policy reference. Semantically, `SKU-40192` and `SKU-40196` are nearly identical. Practically, returning the wrong one is a complete failure.
Run lexical and vector search together and rerank the merged set. Hybrid retrieval is more moving parts, and it is the difference between a system that demos well and one that survives the queries people actually type.
Make "I don’t know" a tested behaviour
If retrieval returns nothing relevant, the correct answer is to say so. This has to be an explicit, evaluated path — with cases in your test set where the expected output is a refusal — or the model will fill the silence with something plausible.
A system that confidently answers when it should abstain has not failed gracefully. It has failed invisibly, which is worse.
None of this is exotic. It is ordinary search engineering, applied before the interesting part. The teams whose assistants work are rarely the ones with the best prompts — they are the ones who took retrieval seriously.