Understanding retrieval systems before adding infrastructure is essential. This article walks through building a local RAG system in Python using BM25, embeddings, and intelligent fusion.
The core insight: hybrid retrieval (combining lexical and semantic search) outperforms either method alone.
The Problem
BM25 excels at exact token matching but misses synonyms and vague intent. Query "family adventure with fuzzy protagonist" has no lexical overlap with Paddington's metadata, so BM25 fails.
Embeddings capture semantics beautifully but struggle with rare named entities. Query "that movie where leo gets attacked by a bear" may not reliably point to Leonardo DiCaprio in embedding space, so dense search fails.
They fail in complementary ways!
The solution: run both in parallel and fuse results.
Architecture Overview
Raw documents flow through two parallel indexing pipelines:
Documents
│
▼
Parse & Chunk
┌────┴────┐
│ │
▼ ▼
BM25 Index Vector Index
│ │
▼ ▼
Keyword Search Semantic Search
└────┬────┘
▼
Hybrid Fusion (RRF)
▼
Cross-Encoder Reranker
▼
Context Builder
▼
LLM Generation
Caching is built in. Indexes are pickled, embeddings stored as .npy files. Subsequent runs reuse these instantly.
Building BM25: Lexical Search
BM25 is a probabilistic ranking function from information retrieval. It's decades old, battle-tested, and still essential.
Tokenization and Preprocessing
Text flows through three steps:
- Stopword removal: Strip high-frequency words ("the", "a", "to")
- Stemming: Reduce words to root form ("traveling" → "travel")
- Tokenization: Split into tokens
text = "Paddington travels to London"
# After preprocessing: ["paddington", "travel", "london"]
The BM25 Formula
For each document, score query terms using:
score = Σ IDF(term) * (TF(term) * (K1 + 1)) / (TF(term) + K1 * (1 - B + B * doc_length / avg_doc_length))
Where:
- TF: Term frequency in the document
- IDF: Inverse document frequency (log(total_docs / docs_with_term))
- K1, B: Tuning parameters (defaults: 1.5, 0.75)
Why it works: TF captures importance within a document. IDF prevents common words from dominating. Length normalization keeps longer documents from unfairly winning.
This project stores postings, term frequencies, document lengths, and computes BM25 scores on the fly. No external libraries. Code is inspectable.
Dense Retrieval: Semantic Search
Dense retrieval learns embeddings from data. This project uses all-MiniLM-L6-v2, a 22M-parameter model producing 384-dimensional vectors.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
query_vec = model.encode("animated bear in london")
doc_vec = model.encode("Paddington travels to London...")
# Relevance = cosine similarity
similarity = np.dot(query_vec, doc_vec) / (np.linalg.norm(query_vec) * np.linalg.norm(doc_vec))
Cosine similarity is cheap (dot product), differentiable, and rotation-invariant.
Document vs. Chunk Embeddings
The system maintains two indexes:
- Document-level: Embed full
title: description. Fast retrieval of complete documents. - Chunk-level: Split descriptions into passages, embed each chunk. Finer-grained context for LLM prompts.
Embeddings are cached as NumPy arrays. Regenerating takes seconds; cached lookup takes milliseconds. This dramatically improves iteration speed during development.
Chunking: A Subtle Lever
Chunking—splitting documents into passages—is often overlooked but high-impact.
Fixed-Size vs. Semantic
Fixed-size (200 words, 1-word overlap):
def fixed_chunks(text, size=200):
words = text.split()
return [" ".join(words[i:i+size]) for i in range(0, len(words), size-1)]
Simple but splits sentences arbitrarily.
Semantic (4 sentences, 1-sentence overlap):
def semantic_chunks(text, max_sentences=4):
sentences = split_sentences(text)
return [" ".join(sentences[i:i+max_sentences]) for i in range(0, len(sentences), max_sentences-1)]
Respects sentence boundaries, cleaner for LLMs.
Why it matters: Better chunking → better retrieved context → better LLM answers. A 1000-word document becomes 5 chunks. The query matches the specific relevant chunk. You retrieve 200 words, not 1000.
Hybrid Search: Fusing Results
Now we have two ranked lists. How to combine them?
Reciprocal Rank Fusion (RRF)
RRF avoids score normalization issues by operating on ranks:
def rrf(bm25_results, semantic_results, k=60):
scores = {}
for rank, (doc_id, _) in enumerate(bm25_results, 1):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
for rank, (doc_id, _) in enumerate(semantic_results, 1):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)
For each result, add reciprocal rank: 1/(k + rank). A document ranked #1 by both methods gets 2/(k+1). Ranked #1 by one, #10 by the other? Closer scores.
Why RRF wins: It's tuning-free. No score normalization. No alpha parameter to fiddle. Empirically, it often outperforms weighted fusion without any tweaking.
Reranking: The Second Pass
Retrieval gives candidates. Reranking refines using a stronger model.
Cross-encoder (fast): Score [query, document] pairs directly.
model = CrossEncoder("ms-marco-MiniLM-L-12-v2")
scores = model.predict([
["bear movie leo attacked", "The Revenant..."],
["bear movie leo attacked", "Brother Bear..."]
])
LLM reranking (slow, accurate): Use an LLM to score candidates. Expensive but highest quality.
Trade latency for accuracy. For interactive search, skip reranking. For RAG generation where quality matters most, add it.
Retrieval-Augmented Generation
Retrieved chunks become LLM context:
context = "\n\n".join([f"[{i}] {chunk}" for i, chunk in enumerate(top_chunks)])
prompt = f"""Answer using only the provided context.
Context:
{context}
Question: {question}
Answer:
"""
Why this works: LLMs hallucinate. Grounding them in retrieved facts forces accuracy. A smaller retrieval system + large LLM often beats the large LLM alone.
Evaluation: Measuring What Matters
Without metrics, you're optimizing by intuition.
Create a golden dataset:
[
{"query": "animated bear in london", "relevant_docs": [0, 5]},
{"query": "leo attacked by bear", "relevant_docs": [2]}
]
Measure:
- Precision@5: Of top 5 results, how many are relevant?
- Recall@5: Of all relevant docs, how many appear in top 5?
- F1: Harmonic mean
Example: Query "animated bear in london", relevant docs [0, 5], top 5 results [0, 3, 5, 7, 9]:
- Precision@5 = 2/5 = 0.4
- Recall@5 = 2/2 = 1.0
- F1 ≈ 0.57
Run evaluation against golden dataset:
uv run cli/evaluation_cli.py --limit 5
This reveals what's actually working.
Design Decisions
Python + NumPy: Readable, modifiable code. Every developer understands it.
Local caching: No distributed complexity. Debugging is straightforward.
CLI, not web service: Clean interfaces. Add REST API later if needed.
No ORM, no frameworks: Raw Python. Queries are explicit. No magic.
This intentional simplicity serves learning. Before scaling with Weaviate or Pinecone, understand the basics.
Scaling the System
Current limitations are features, not bugs:
- Vector databases: For 10M+ documents, swap NumPy arrays for Weaviate/Pinecone.
- Incremental indexing: Currently all-or-nothing. Add streaming ingestion for real-time updates.
- REST API: Wrap in FastAPI for HTTP clients.
- Larger datasets: Beyond 10K documents, infrastructure above becomes necessary.
The roadmap is clear. Start simple. Scale when you understand the system.
Lessons
Retrieval engineering is underrated. Better retrieval + smaller model often beats larger model alone.
Hybrid systems win. BM25 + embeddings > either alone. This principle applies beyond search.
Chunking is crucial. It touches encoding, retrieval quality, and prompt size. Underestimating chunking is common.
Caching changes iteration speed. 5 seconds to regenerate vs. 100ms cached lookup is the difference between experimenting and waiting.
Start local, scale later. Understanding a system on your laptop beats guessing about distributed infrastructure.
Repository: https://github.com/phraakture/rag-search-engine
Clone, run uv sync, and experiment. The codebase is designed to be explored and understood.