Building RAG Systems That Actually Answer Correctly

Retrieval-Augmented Generation has become the default pattern for enterprise AI — and with good reason. But most enterprise RAG deployments underperform significantly, and in almost every case, the problem isn't the language model. It's the retrieval layer.

Why Retrieval Is the Hard Problem

A language model can only reason about the context it's given. If you give it the wrong context — content that contains relevant keywords but not relevant meaning — it will either hallucinate the answer it expects to find, or produce a response that's technically grounded in the retrieved text but practically incorrect.

The most common retrieval failure mode: fixed-size chunking that breaks documents at arbitrary character limits. The second: embedding models trained on general text that perform poorly on domain-specific terminology. The third: no query preprocessing, so the raw user query hits the vector store directly.

Hierarchical Chunking

Replace fixed-size chunking with structure-aware chunking. Preserve document hierarchy: section headers, subsections, paragraph boundaries. A paragraph from the "Exceptions" section of a policy document means something completely different from the same paragraph in "General Rules" — and flat chunking loses that context entirely.

Query Rewriting and Expansion

Before sending a user query to your vector store, process it. At minimum, use the conversation history to rewrite an ambiguous query into a more complete one. "What's the limit?" becomes "What is the annual claim limit under policy type B for employees hired after 2022?" The retrieval precision improvement is significant.

Re-ranking After Retrieval

Bi-encoder embedding models optimise for retrieval speed, not relevance precision. After retrieving your top-k candidates, use a cross-encoder re-ranker to score each candidate against the original query. In our experience, adding a re-ranking step improves end-to-end accuracy by fifteen to twenty-five percent with no changes to the underlying model.

Evaluate Retrieval Independently

Build a test set of questions with known correct answer locations in your document corpus. Measure precision@k and recall@k for your retrieval layer independently of the LLM output. If your retrieval precision is sixty percent, no amount of prompt engineering will get you to ninety percent end-to-end accuracy.

Related Insights