Generation with Open Models

The RAG prompt template

The prompt you send to the generative model is the interface between your retrieval pipeline and the user-facing answer. Getting it right is the difference between a system that users trust and one they abandon.

Here is a production RAG prompt template:

<system>
You are an internal knowledge assistant for {organisation_name}.
Answer the user's question using ONLY the information provided
in the context sections below. Follow these rules strictly:

1. If the context does not contain enough information to answer
   the question, say "I don't have enough information in the
   available documents to answer this question" and explain
   what information would be needed.
2. Never use information from your training data. Only use the
   provided context.
3. Cite your sources using [Source N] notation after each claim,
   where N corresponds to the context section number.
4. If different context sections contain contradictory information,
   acknowledge the contradiction and present both positions with
   their sources.
5. Keep your answer concise and directly relevant to the question.
</system>

<context>
[Source 1: {document_title_1}, {document_date_1}]
{chunk_text_1}

[Source 2: {document_title_2}, {document_date_2}]
{chunk_text_2}

[Source 3: {document_title_3}, {document_date_3}]
{chunk_text_3}
</context>

<question>
{user_query}
</question>

Key design decisions in this template:

System prompt sets the grounding rules. The instruction to use "ONLY the information provided" is the primary defence against hallucination. Without this, the model may supplement retrieved context with knowledge from its training data, producing answers that sound authoritative but cite information not in your documents.

Source metadata is included. Document title and date help the model (and the user) assess recency and authority. An answer citing a 2019 policy when a 2024 revision exists is a failure your users will catch.

Citation format is explicit. The [Source N] notation creates a verifiable link between each claim and its source document. Users can click through to verify. This is non-negotiable in enterprise RAG -- unsourced answers are unacceptable in legal, financial, and medical contexts.

Contradiction handling is specified. Enterprise document corpora invariably contain contradictions (policy revisions, regional variations, superseded documents). The prompt tells the model how to handle this rather than letting it silently choose one version.

Grounding, attribution, and self-verification

Hallucination in RAG takes a specific form: the model generates claims that are not supported by the retrieved context. This is different from general LLM hallucination (making up facts) -- in RAG, the model has access to the correct information but either ignores it, misrepresents it, or supplements it with unsupported content.

Three techniques reduce RAG hallucination:

Grounding through prompt design. The system prompt instructions from the previous section are the first layer. Beyond the basic "use only provided context" instruction, you can strengthen grounding with:

• "Begin your answer by identifying which source sections are most relevant to the question."
• "For each claim, quote the specific phrase from the source that supports it."
• "If you are uncertain whether the context supports a claim, do not include it."

These instructions force the model to explicitly connect its output to the input context, making it harder to inject unsupported claims.

Attribution verification (post-generation check). After the model generates a response, run a second pass that checks each cited claim against the cited source. This can be done by the same model (with a different prompt) or a smaller model:

Given the following claim and the source text it cites,
does the source text support the claim?

Claim: {claim}
Cited source: {source_text}

Answer YES if the source directly supports the claim,
NO if it does not, or PARTIAL if the source partially
supports the claim. Explain briefly.

Claims that fail verification are flagged or removed. This adds one more LLM call per response (100-300 ms) but catches hallucinated citations -- a particularly insidious failure mode where the model cites a real source for a fabricated claim.

Self-verification (self-RAG pattern). Train or prompt the model to emit confidence signals alongside its response: "Based on Source 2, I am confident that the delivery deadline is 30 days. I am less certain about the penalty amount -- Source 3 mentions liquidated damages but does not specify a percentage." This gives users a nuanced understanding of answer reliability rather than a binary "here is the answer" that may or may not be fully supported.

When retrieved context exceeds the window

Modern LLMs have context windows of 128K tokens (Gemma 4) or even longer, so context overflow is less common than it was in 2023. But at enterprise scale, it still happens. A query about a major multi-year project might retrieve 20 relevant document sections totalling 50,000 tokens. Here is how to handle it.

Strategy 1: Truncate by relevance. After reranking, you have results ordered by relevance score. Include chunks starting from the most relevant until you hit your context budget (leave room for the system prompt, the question, and the generated response). Typical budget: 60-70% of the context window for retrieved content.

Strategy 2: Summarise low-ranked chunks. Instead of including the full text of every chunk, summarise chunks ranked 5-10 into a condensed form. Include full text for the top 4 chunks (highest relevance, most likely to contain the direct answer) and summaries for the rest (providing breadth of context without consuming as many tokens).

Strategy 3: Iterative retrieval. For complex queries, do not try to fit everything into a single generation call. First, use the LLM to generate an initial answer from the top-5 chunks. Then, identify gaps ("What information is missing?"), retrieve additional chunks for those gaps, and generate a more complete answer. This is more expensive (2-3 LLM calls) but handles complex multi-aspect queries better.

The "lost in the middle" mitigation. As discussed in Module 2, LLMs underweight information in the middle of the context. Two practical mitigations:

1. Put the most relevant chunks first and last. Sandwich the less-critical chunks in the middle. The model pays more attention to the beginning and end of the context.
2. Reduce total chunks. Passing 5 highly relevant chunks is often better than passing 15 chunks of mixed relevance. The model processes fewer chunks more thoroughly.

Streaming responses

For interactive RAG applications, streaming the response (showing tokens as they are generated) is critical for user experience. A 500-token response takes 5-10 seconds to generate fully. Without streaming, the user stares at a blank screen for 5-10 seconds. With streaming, they see the first token in 100-500 ms and can start reading while the rest generates.

vLLM supports streaming natively via its OpenAI-compatible API. Your application receives Server-Sent Events (SSE), with each event containing one or more tokens. The implementation is straightforward -- the complexity is in the frontend:

• Progressive rendering. Render tokens as they arrive. Markdown formatting (headers, lists, code blocks) should be applied incrementally.
• Citation rendering. When the model outputs [Source 3], your frontend must resolve this to the actual source document and render it as a clickable link. This can happen inline or after the response completes.
• Error handling. If generation fails mid-stream (GPU OOM, timeout), show what was generated so far with a clear error indicator, not a blank screen.

Handling "I don't know."

One of the most important capabilities of an enterprise RAG system is knowing when it does not know. If a user asks about a topic not covered in the corpus, the system must say so clearly rather than hallucinating an answer.

The prompt template from earlier includes this instruction, but implementation requires more:

1. Retrieval confidence scoring. If the highest reranker score for any retrieved chunk is below a threshold, flag the query as potentially unanswerable before generation even starts. You can show a warning: "Limited relevant documents found -- answer may be incomplete."

2. Generation confidence. If the model's response includes qualifiers like "I could not find specific information about..." or "The available documents do not address...", surface these prominently rather than burying them in the response.

3. Fallback paths. When the system cannot answer, offer constructive alternatives: "This topic may be covered in [Department X's] knowledge base" or "You might consult [Subject Matter Expert] for this type of question." These fallbacks require a knowledge map of what your RAG system covers and what it does not.

The worst outcome is a confident-sounding answer that is wrong. A clear "I don't know" preserves user trust and avoids the downstream consequences of acting on incorrect information -- which in enterprise contexts can mean regulatory violations, financial losses, or legal liability.