Retrieval-Augmented Generation (RAG) Guide 2026

A practical RAG guide — what retrieval-augmented generation is, the ingest → chunk → embed → store → retrieve → generate pipeline, chunking, embeddings, hybrid search, reranking, evaluation, and RAG vs fine-tuning vs long context.

XCODX TeamJuly 29, 2026 · 7 min read

Retrieval-augmented generation (RAG) grounds a large language model in your own data: instead of relying only on what it learned in training, the model retrieves relevant documents at query time and answers from them. That reduces hallucination, uses fresh and private data without retraining, and gives traceable citations. This guide walks the full RAG pipeline for 2026 and 14 practical decisions along the way, each with the reasoning.

1. Understand what RAG is and when to use it

RAG injects retrieved, relevant context into the prompt so the model answers from your corpus rather than parametric memory. Use it when answers must reflect private, fresh or large bodies of data, need citations, or must minimize hallucination — without the cost and staleness of retraining the model.

// the core idea, in one line
answer = LLM(system + retrieve(query) + query)

2. Ingest and clean your sources

Parse source documents (PDF, HTML, Office, DB rows) into clean text while preserving structure (headings, tables) and attaching metadata (source, date, section, permissions). Layout-aware parsing and good metadata are what later enable filtering and citations; silently losing tables and figures is a top quality killer.

// keep structure + metadata, not just raw text
{ "text": "...", "source": "handbook.pdf", "section": "Refunds",
  "date": "2026-05-01", "acl": ["staff"] }

3. Chunk deliberately

Chunk size drives retrieval quality. A 512–1024 token baseline is a common default; smaller (256–512) suits factoid lookups, larger suits analytical/narrative content. Structure-aware and semantic chunking (splitting on topic/heading boundaries) generally beat naive fixed-size splits — at higher indexing cost. Overlap helps some workloads and not others, so test it rather than assuming.

# pragmatic default: recursive splitter, ~800 tokens, small overlap
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, chunk_overlap=100)
chunks = splitter.split_text(doc)

4. Embed with a current model

Embeddings turn each chunk into a vector so similar meanings sit close together. In 2026 strong options include OpenAI’s text-embedding-3-large (up to 3072 dims, truncatable), Google’s Gemini embeddings, Cohere and Voyage, plus open self-host models. Leaders cluster closely on benchmarks — pick for cost/latency/hosting and benchmark on your data, not a leaderboard.

vec = openai.embeddings.create(
    model="text-embedding-3-large", input=chunk
).data[0].embedding

5. Store in a vector database

A vector database indexes embeddings for fast approximate-nearest-neighbor search. Choose by scale and ops: pgvector if you already run Postgres; Pinecone for zero-ops managed; Weaviate for native hybrid search; Qdrant for fast filtered search; Milvus for billions of vectors; Chroma for prototyping.

// store chunk vectors + metadata for retrieval
db.upsert(id, vector=vec, metadata={"source":"handbook.pdf",
  "section":"Refunds", "acl":["staff"]})

6. Retrieve with approximate nearest-neighbor search

At query time, embed the question and fetch the top-k most similar chunks via ANN search (usually HNSW) over cosine or dot-product similarity. ANN trades a little recall for large latency wins, which is what makes retrieval fast enough to run per-query at scale.

q = embed(query)
hits = db.search(q, top_k=8)   # nearest chunks by similarity

7. Add hybrid search (semantic + keyword)

Pure vector search drifts on exact terms — product codes, entity names, rare technical words. Combining dense vectors with lexical BM25 and fusing the scores (for example Reciprocal Rank Fusion) catches both meaning and exact matches. Adding BM25 is often the single highest-impact upgrade to a vector-only system; treat hybrid as the baseline.

// fuse dense + sparse results
dense  = vector_search(query, k=20)
sparse = bm25_search(query, k=20)
results = reciprocal_rank_fusion(dense, sparse)

8. Rerank the top candidates

Retrieval is recall-oriented and noisy. A cross-encoder reranker (or a hosted reranker like Cohere/Voyage) re-scores the top-N candidates against the query and keeps the best few, restoring precision. Anthropic’s Contextual Retrieval reported large drops in failed retrievals when hybrid search and reranking were added.

candidates = retrieve(query, top_k=40)
top = reranker.rank(query, candidates)[:6]  # precision pass

9. Filter by metadata for correctness and security

Constrain retrieval by structured metadata — tenant, date, document type, access-control list — alongside vector similarity. This is how you enforce per-user permissions and recency at retrieval time, rather than leaking then post-filtering. It’s a correctness and security requirement, not an optimization.

hits = db.search(q, top_k=8, filter={
    "tenant": user.tenant, "date": {"$gte": cutoff},
    "acl": {"$in": user.roles}
})

10. Augment the prompt and generate with citations

Insert the retrieved chunks into the prompt with an instruction to answer only from the context and cite sources. Explicit grounding plus citations is what converts retrieval into low-hallucination, auditable answers — and lets users verify claims against the source.

Answer using ONLY the context below. Cite the source for each
claim. If the context does not contain the answer, say so.
<context>{{ retrieved_chunks }}</context>
Question: {{ query }}

11. Fix the common failure modes

When RAG underperforms, diagnose the stage: bad chunking -> try semantic/structure-aware chunks; retrieval misses -> add hybrid search, reranking and query rewriting; "lost in the middle" from stuffing too much -> pass fewer, better chunks, not more; stale answers -> schedule re-indexing. More context is not the fix — better retrieval is.

// symptom -> fix
// wrong/partial facts  -> chunking + reranking
// misses exact terms   -> add BM25 (hybrid)
// ignores mid-context  -> fewer, higher-quality chunks

12. Evaluate retrieval and generation separately

Measure the two stages independently: retrieval with context precision/recall, generation with faithfulness/groundedness and answer relevancy — the metrics popularized by RAGAS. Tools like RAGAS, DeepEval and Langfuse make this repeatable. The minimum bar is at least one retrieval metric and one generation metric, tracked on every change.

# score both stages on a gold set
scores = ragas.evaluate(dataset,
    metrics=[context_recall, faithfulness, answer_relevancy])

13. Choose RAG vs fine-tuning vs long context

The 2026 consensus is "RAG-first, tune-second." Big context windows didn’t kill RAG — they don’t solve ranking, salience or noise. Rule of thumb: if the whole knowledge base fits comfortably (~under 200K tokens), full-context prompting with caching can beat building retrieval infra; above that, use RAG. Fine-tune for behavior (format, style, policy), not to add facts.

// decision sketch
// small, static corpus    -> full context + prompt caching
// large / fresh / private -> RAG
// change behavior/format  -> fine-tune (PEFT/LoRA), + RAG for facts

14. Consider agentic RAG for hard questions

Agentic RAG lets an agent decide whether, what and how to retrieve — issuing multiple retrieval steps, rewriting queries and self-correcting — instead of a single fixed lookup. It’s the notable 2026 trend for complex, multi-hop questions, at the cost of more latency and tokens. See how to build AI agents.

// agent loop: retrieve -> assess -> retrieve again if needed
while not enough_context(state):
    q = agent.next_query(state)
    state += retrieve(q)

RAG checklist

Ingest with structure + metadata; chunk deliberately (test overlap).
Embed with a current model; store in a vector DB matched to your scale.
Retrieve with ANN; add hybrid (BM25 + vector) and a reranker.
Filter by metadata for permissions and recency.
Ground generation in retrieved context and require citations.
Diagnose failures by stage; better retrieval beats more context.
Evaluate retrieval and generation separately; RAG-first, fine-tune for behavior.

Frequently asked questions

What is retrieval-augmented generation (RAG)?

RAG is a technique where an LLM retrieves relevant documents from an external corpus at query time and answers from them, instead of relying only on its trained-in knowledge. It reduces hallucination, lets you use fresh or private data without retraining, and supports citations. The pipeline is: ingest → chunk → embed → store in a vector DB → retrieve → augment the prompt → generate.

Does a 1M-token context window make RAG obsolete?

No. Large context windows help but don’t solve ranking, salience or noise — stuffing everything in causes "lost in the middle" and high cost. The 2026 rule of thumb: if your whole knowledge base fits comfortably (~under 200K tokens), full-context prompting with caching can win; above that, RAG is more accurate and cheaper per query.

What’s the best chunk size for RAG?

A 512–1024 token chunk is a common baseline: smaller (256–512) suits factoid lookups, larger suits analytical or narrative content. Structure-aware and semantic chunking generally beat naive fixed-size splitting, at higher indexing cost. Overlap helps some workloads and not others, so measure on your own data rather than following a fixed rule.

RAG or fine-tuning — which should I use?

Use RAG to add or update knowledge (facts, docs, private data) and fine-tuning to change behavior (output format, style, tool-use policy, classification). They’re complementary and often combined: fine-tune for how the model behaves, RAG for what it knows. Start with RAG — it’s cheaper, updatable, and doesn’t risk baking in stale facts.

Prototype a RAG pipeline in the browser with XCODX Studio — npm packages for embeddings and vector search, no setup. Continue with how to build AI agents and how to build with LLM APIs.