RAG Architecture Fundamentals

The five failure modes of naive RAG

"Naive RAG" is the approach you get from following a LangChain tutorial: split documents into fixed-size chunks, embed them with a single model, do a simple top-K vector search, and pass the results straight to the LLM. It works surprisingly well in demos. It fails predictably in production. Here are the five ways it breaks.

Failure 1: Semantic dilution. Fixed-size chunks (say, 512 tokens) routinely split meaningful content across chunk boundaries. A contract clause that says "The vendor shall deliver within 30 days of the purchase order date, subject to force majeure provisions outlined in Section 14.2" might get split at "subject to" -- and the second half, without the first, is meaningless for retrieval. The embedding of the truncated chunk captures "force majeure in Section 14.2" but not "30-day delivery obligation," destroying the semantic connection between the two halves.

Failure 2: The lost-in-the-middle problem. When you pass 5-10 retrieved chunks to the LLM, research from Liu et al. (2023) demonstrated that LLMs pay more attention to information at the beginning and end of the context, and less to information in the middle. If your most relevant chunk happens to land in positions 3-7, the LLM may underweight it. This is not a retrieval failure -- it is a generation failure caused by naive context assembly.

Failure 3: No negative signal. Naive RAG always returns the top-K results, even when none of them are actually relevant. If you ask about a topic that is not in your corpus, the system still returns the K "least irrelevant" chunks, and the LLM gamely synthesises an answer from them. There is no mechanism to say "we do not have information about this."

Failure 4: Single-hop only. Naive RAG handles questions that can be answered from a single chunk or a handful of related chunks. It fails on multi-hop questions: "Which projects led by managers who joined after 2023 have exceeded their budget by more than 20%?" This requires retrieving manager records, filtering by join date, cross-referencing project assignments, and then checking budget data. Vector similarity search alone cannot do compositional reasoning.

Failure 5: No query understanding. The user's question is embedded as-is and used for retrieval. But user questions are often ambiguous, underspecified, or phrased differently from how the information is stated in the documents. "What's our liability exposure?" could refer to legal liability, financial liability, insurance liability, or regulatory liability. Naive RAG does not expand, disambiguate, or reformulate the query.

Patterns that solve the failure modes

Each naive RAG failure mode has a corresponding advanced pattern. Here are the four most impactful.

Query expansion addresses the "no query understanding" failure. Instead of embedding the user's raw question, you first use the generative LLM to produce 3-5 reformulated queries. "What's our liability exposure?" becomes:

• "What legal liabilities does the organisation currently face?"
• "What is the estimated financial exposure from pending litigation?"
• "What regulatory compliance risks could result in fines or penalties?"

Each reformulation is embedded and searched separately. The results are merged using Reciprocal Rank Fusion (RRF), which combines rankings from multiple retrieval signals. Query expansion typically improves recall by 15-30% because it casts a wider semantic net.

HyDE (Hypothetical Document Embeddings) takes a different approach. Instead of reformulating the query, you ask the LLM to generate a hypothetical document that would answer the query. This hypothetical answer is then embedded, and you search for real documents similar to the hypothetical answer rather than similar to the question.

Why does this work? Because questions and answers often use different vocabulary. The question "What's our parental leave policy?" and the document "Employees are entitled to 16 weeks of paid family leave..." are semantically related but lexically different. HyDE bridges this vocabulary gap by converting the question into answer-space before searching.

Multi-hop retrieval addresses compositional questions. Instead of a single retrieval step, the system breaks the question into sub-questions, retrieves for each, and chains the results. "Which projects led by managers who joined after 2023 have exceeded budget?" becomes:

1. Retrieve: managers who joined after 2023 (returns names)
2. Retrieve: projects led by [those specific managers]
3. Retrieve: budget status for [those specific projects]
4. Synthesise: combine the chain into a final answer

This requires an orchestration layer that uses the LLM to decompose questions and plan retrieval steps. It is slower (3-4 retrieval round-trips instead of one) but handles questions that single-hop retrieval literally cannot answer.

Self-RAG addresses the "no negative signal" problem. In self-RAG, the generative model is trained (or prompted) to emit special tokens that indicate whether it is using retrieved context, whether the retrieved context is relevant, and whether its own response is supported by the context. This gives the system a built-in quality signal: if the model flags low relevance or low support, the system can return "I don't have enough information to answer that" instead of hallucinating.

Two models, two jobs

This is one of the most common points of confusion in enterprise RAG, and getting it wrong leads to architectural mistakes that are expensive to fix.

Embedding models convert text into fixed-size numerical vectors. They are small (140M to 1.5B parameters), fast (thousands of chunks per second on a single GPU), and produce output of fixed dimensions (768 to 4096 floats). Their job is to create a numerical representation of meaning so that similar texts produce similar vectors. They do not generate text. They do not understand questions. They do not reason. They are a very sophisticated similarity function.

Generative models (Gemma 4, Llama, GPT-4o, Claude) take text as input and produce text as output. They are large (2B to 400B+ parameters), slower (tens to hundreds of tokens per second), and their output is variable-length natural language. Their job is to read context and produce a thoughtful, structured response.

In a RAG system, you need both. The embedding model handles stages 3, 5, and part of 6 (embed chunks, embed queries, similarity search). The generative model handles stages 7 and 8 (reranking and generation). They are different tools for different jobs, like a library's catalogue system (embedding model) versus the librarian who reads the books and answers your questions (generative model).

You cannot use Gemma 4 27B as your embedding model. It is a generative model -- it produces text, not fixed-size vectors. You could technically extract internal representations from it, but that would be absurdly expensive (running a 27B model for every chunk embedding when a 140M model produces better embeddings) and architecturally wrong.

The correct architecture: a small, fast embedding model (GTE-Qwen2-1.5B, Nomic Embed v2, or similar) running on a modest GPU for embedding and retrieval, plus Gemma 4 for reasoning and generation. The embedding model might process 2,000 chunks per second. The generative model might produce 80 tokens per second. They have completely different performance profiles and hardware requirements.

What each stage costs in milliseconds

One of the most practical things you can do as an architect is establish a latency budget for your RAG pipeline. Here are realistic numbers for an on-premises deployment.

The critical insight: retrieval is fast; generation is slow. Everything from query embedding through reranking typically completes in under 200 ms. The generation step dominates the end-to-end latency. This means optimising retrieval latency from 50 ms to 20 ms is irrelevant if your generation step takes 5 seconds.

Where to invest your optimisation effort depends on your use case. For interactive chat (where users see streaming tokens), time-to-first-token (TTFT) matters most -- and TTFT is dominated by the generation model's startup time. For batch processing (where you process thousands of queries), throughput matters most -- and throughput is dominated by how many concurrent requests your generation infrastructure can handle.