📚 2.02 – RAG Deep Dive

📑 On this page

RAG Architecture Overview
Chunking Strategies
Embedding Models
Retrieval Strategies
Reranking
RAG Evaluation
Exercises

🏗️ RAG Architecture Overview

📌 Complete RAG Pipeline

     ┌─────────────────────────────────────────────────────────────┐
     │                    OFFLINE (Indexing)                       │
     │  Documents → Chunking → Embeddings → Vector Database        │
     └─────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
     ┌─────────────────────────────────────────────────────────────┐
     │                    ONLINE (Query)                           │
     │  User Query → Embeddings → Similarity Search → Top-K Chunks │
     │                                                             │
     │  Prompt: Context + Query → LLM → Response                   │
     └─────────────────────────────────────────────────────────────┘

💡 Key Insight

RAG solves the "knowledge cutoff" problem. The LLM doesn't need to know your data – it just needs to read relevant chunks at query time.

✂️ Chunking Strategies

📌 Why Chunking Matters

Documents are too long to fit in context windows. Chunking determines what information is retrieved. Bad chunking = irrelevant retrieval = bad answers.

Strategy	Description	Best for	Pros/Cons
Fixed-size chunking	Split every N characters/tokens	Simple documents, consistent format	✅ Simple ❌ Might break sentences
Semantic chunking	Split at sentence boundaries, paragraphs, or semantic meaning	Prose, articles, books	✅ Respects natural boundaries ❌ More complex
Recursive chunking	Try paragraph → sentence → character if too long	Mixed content (markdown, code, prose)	✅ Adaptive ❌ Slower
Document-specific	Use document structure (headers, lists, tables)	HTML, Markdown, PDF with structure	✅ Preserves context ❌ Format-specific

Chunking Example (Python with LangChain):

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Recommended for most documents
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # characters per chunk
    chunk_overlap=200,      # overlap between chunks (preserves context)
    separators=["\n\n", "\n", ".", "!", "?", ",", " ", ""]
)

chunks = text_splitter.split_text(document)

💡 Chunk Size Guidelines:

Small (256-512 tokens): High precision, may miss context
Medium (512-1024 tokens): Good balance (recommended)
Large (1024-2048 tokens): More context, but might include noise
Overlap: 10-20% of chunk size

🔢 Embedding Models

📌 Popular Embedding Models Comparison

Model	Dimensions	Context Length	MTEB Score	Cost per 1M tokens
text-embedding-3-small	1536	8192	62.3	$0.02
text-embedding-3-large	3072	8192	64.6	$0.13
BAAI/bge-large-en	1024	512	63.7	Free (local)
intfloat/e5-mistral-7b	4096	32768	66.0	Needs GPU

Creating Embeddings (OpenAI):

from openai import OpenAI
client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Your text to embed",
    dimensions=1536  # Can reduce dimensionality to save cost
)

embedding = response.data[0].embedding

🎯 Retrieval Strategies

📌 Beyond Simple Similarity Search

Vanilla Similarity: Cosine similarity of query vs chunk embeddings
Hybrid Search (BM25 + Vector): Combines keyword matching with semantic similarity
Multi-Query Retrieval: Generate multiple query variations, retrieve for all, deduplicate
Contextual Retrieval: Include surrounding chunks for better context

Hybrid Search Implementation (Weaviate):

# Combined vector + keyword search
response = client.query.get(
    "Document", ["text", "source"]
).with_hybrid(
    query="What is the refund policy?",
    alpha=0.5  # 0 = pure keyword, 1 = pure vector
).with_limit(10).do()

Multi-Query Retrieval:

from langchain.retrievers.multi_query import MultiQueryRetriever

retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(),
    llm=ChatOpenAI(model="gpt-4"),
    prompt="Generate 3 different versions of this question..."
)

# Automatically generates multiple queries and combines results
docs = retriever.get_relevant_documents(query)

📊 Reranking

📌 Why Rerank?

Initial retrieval (embedding similarity) is fast but can be noisy. Reranking uses a more expensive but more accurate model to reorder results.

Cross-Encoder Reranking:

from sentence_transformers import CrossEncoder

# Load cross-encoder model (more accurate but slower)
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

# Initial retrieval (fast)
initial_results = vectorstore.similarity_search(query, k=20)

# Rerank pairs (query, document)
pairs = [(query, doc.page_content) for doc in initial_results]
scores = reranker.predict(pairs)

# Sort by reranking score and get top k
top_docs = sorted(zip(initial_results, scores), key=lambda x: x[1], reverse=True)[:5]

Method	Speed	Accuracy	Cost
Vector similarity	Very fast	Good	Low ($)
Cross-encoder rerank	Slow	Excellent	Higher (but only on top results)
LLM-as-reranker	Very slow	Best	High ($$$)

📈 RAG Evaluation

📌 Key Metrics for RAG Systems

Hit Rate: % of queries where relevant document is in top-k
Mean Reciprocal Rank (MRR): Position of first relevant result
Faithfulness: Does the answer contradict the retrieved context?
Answer Relevance: Does the answer address the question?
Context Relevance: How relevant are retrieved chunks?

Evaluation with RAGAS:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_relevancy

# Prepare dataset with questions, answers, contexts, ground_truth
dataset = Dataset.from_dict({
    "question": questions,
    "answer": generated_answers,
    "contexts": retrieved_contexts,
    "ground_truth": ground_truths
})

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])

print(result)  # Scores between 0-1

💡 RAG Optimization Checklist

✅ Chunk size: 512-1024 tokens with 10-20% overlap
✅ Embedding model: text-embedding-3-small for cost/performance balance
✅ Retrieval: Hybrid search (BM25 + vector)
✅ Reranking: Cross-encoder for production systems
✅ Evaluation: Track hit rate and faithfulness

✍️ Exercises

Exercise 2.02.1 – Chunking experiment

Take a long document (e.g., a news article). Try chunk sizes of 256, 512, and 1024 tokens. Ask the same question to each version. Which chunk size gave the best answer?

There's no single correct answer – it depends on your document and question. Generally:
- Smaller chunks (256) → higher precision but may miss context
- Larger chunks (1024) → more context but may include noise
- 512 with overlap is a good starting point

Exercise 2.02.2 – Embedding choice

You're building a RAG system for 1 million short support tickets (average 200 words each). Which embedding model would you choose and why?

Recommendation: text-embedding-3-small
Reasons: 1M tickets × ~200 words = ~200M tokens. At $0.02/1M tokens = $4 total embedding cost. Small is 2x cheaper than large, only 5% less accurate. Perfect for this scale.

Exercise 2.02.3 – Build a simple RAG system

Using LangChain, Chroma, and OpenAI, build a RAG system that answers questions from a PDF of your choice.

from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# Load PDF
loader = PyPDFLoader("your_document.pdf")
documents = loader.load()

# Split
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
docs = text_splitter.split_documents(documents)

# Embed and store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings)

# Create QA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4"),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4})
)

# Ask
answer = qa_chain.run("What is the main topic of this document?")
print(answer)

📌 Key Takeaways

Chunk size and overlap dramatically affect retrieval quality
Hybrid search (keyword + vector) often beats pure vector search
Reranking improves accuracy but adds latency
Evaluate with hit rate, MRR, and faithfulness metrics

📘 Next Module: 2.03 – LangChain & LlamaIndex →