A Multi-Index RAG pipeline

You have two search systems with complementary strengths. VectorIndex handles meaning; BM25Index handles exact strings. Running them both by hand for every query is tedious and the results come back on incompatible scales — how do you even compare a cosine distance of 0.71 to a BM25 score of 12.4? You don't. You use a simple trick called reciprocal rank fusion to merge the two rankings without ever looking at the raw scores, and you wrap everything behind a single Retriever class that queries all indexes in parallel.

One interface, many indexes

Both VectorIndex and BM25Index were designed with deliberately similar APIs:

• add_document(doc) — add a document to the index
• search(query, k) — return top-k matches

That consistency is what makes this lesson easy. Any search layer that implements those two methods can be plugged into a multi-index pipeline without changes. In type-system terms, they both satisfy a SearchIndex protocol:

from typing import Protocol, Dict, Any, List, Tuple

class SearchIndex(Protocol):
    def add_document(self, document: Dict[str, Any]) -> None: ...
    def search(self, query: str, k: int) -> List[Tuple[Dict[str, Any], float]]: ...

As long as a class implements that interface, the Retriever doesn't care what's inside — vector search, BM25, a graph-based search, a SQL full-text search, whatever.

Reciprocal Rank Fusion

The challenge with merging two search results is that they use different scoring systems. Vector search returns cosine distances; BM25 returns term-weighted scores. You can't just add them up, because the scales are incompatible and neither one is calibrated.

Reciprocal Rank Fusion (RRF) solves this by ignoring the raw scores entirely and using only the rank of each document in each search. The formula:

RRF_score(d) = Σ (1 / (k + rank_i(d)))

For each document d, loop over every search result list i, and add 1 / (k + rank_i(d)) where rank_i(d) is the position of d in that list (or some large number if it wasn't returned). Sum across all lists and you get a score that rewards documents for consistently placing well across every search method.

The constant k is usually 60 (the value most papers use), though smaller values like 1 make the numbers more dramatic for illustration.

A worked example

Query: "INC-2023-Q4-011"

VectorIndex.search returns: Section 2 (rank 1), Section 7 (rank 2), Section 6 (rank 3) BM25Index.search returns: Section 6 (rank 1), Section 2 (rank 2), Section 7 (rank 3)

Using k = 1 for clarity:

• Section 2: 1 / (1 + 1) + 1 / (1 + 2) = 0.5 + 0.333 = 0.833
• Section 6: 1 / (1 + 3) + 1 / (1 + 1) = 0.25 + 0.5 = 0.75
• Section 7: 1 / (1 + 2) + 1 / (1 + 3) = 0.333 + 0.25 = 0.583

Final ranking: Section 2 > Section 6 > Section 7.

Notice what happened. Section 2 wasn't the top result in either individual search — BM25 put Section 6 first. But Section 2 was strong in both (first in vector, second in BM25), while Section 6 was strong in BM25 but only third in vector. Consistency across both indexes won the day. That's exactly the property we want — documents that look relevant to multiple search methods are more likely to actually be relevant than documents that only one method liked.

The k constant damps this effect: larger k flattens the score differences between early and late ranks, making rank differences less punishing. k = 60 is the standard; tune it if you have a reason, but you usually don't.

Implementing the Retriever class

from typing import Any, Dict, List, Tuple

class Retriever:
    def __init__(self, *indexes: SearchIndex):
        if len(indexes) == 0:
            raise ValueError("At least one index must be provided")
        self._indexes = list(indexes)

    def add_document(self, document: Dict[str, Any]) -> None:
        for index in self._indexes:
            index.add_document(document)

    def search(
        self, query_text: str, k: int = 3, k_rrf: int = 60
    ) -> List[Tuple[Dict[str, Any], float]]:
        all_results = [idx.search(query_text, k=k) for idx in self._indexes]

        rrf_scores: Dict[int, float] = {}
        seen_docs: Dict[int, Dict[str, Any]] = {}

        for results in all_results:
            for rank, (doc, _) in enumerate(results):
                doc_key = id(doc)  # in practice use a stable content-based key
                rrf_scores[doc_key] = rrf_scores.get(doc_key, 0.0) + 1.0 / (k_rrf + rank + 1)
                seen_docs[doc_key] = doc

        ranked = sorted(rrf_scores.items(), key=lambda kv: kv[1], reverse=True)
        return [(seen_docs[key], score) for key, score in ranked[:k]]

Three things this does:

• add_document fans out — every document goes into every index. Adding to the Retriever is the only call site, so you can't accidentally forget to index somewhere.
• search queries all indexes in sequence, collects the rank-ordered results, and applies RRF.
• k_rrf is the RRF constant. Default to 60 unless you know otherwise.

The one production subtlety: use a stable content-based key for deduplication, not id(doc). Two separate dict objects referring to the same chunk should deduplicate to a single entry. In the course code that's handled by deriving a hash from the content field; in your own system you'd use the document's ID.

Using the hybrid retriever

Wire it up:

vector_store = VectorIndex()
bm25_store = BM25Index()
retriever = Retriever(vector_store, bm25_store)

for chunk in chunks:
    retriever.add_document({"content": chunk})

results = retriever.search("What happened with INC-2023-Q4-011?", k=3)

One call to add_document now populates both indexes. One call to search queries both, merges via RRF, and returns a clean ranked list.

Run the same incident-ID query that confused both individual search methods:

• Section 10: Cybersecurity Analysis (highest RRF — the actual incident report)
• Section 2: Software Engineering — Project Phoenix Stability (second — the section that references the incident in context)
• Section 5: Legal Developments (third — downstream consequences of the incident)

All three are genuinely relevant. The off-topic financial section that plagued the vector-only search is gone, because BM25 didn't rank it and RRF only rewards documents that rank well in at least one of the merged lists.

Extensibility: just implement the protocol

The real value of this architecture is that it scales to more indexes without touching the Retriever. Want a graph-based search? A SQL full-text index? A specialised medical-terminology index? A colour-keyword index for a design asset library? As long as the new class implements add_document and search, you pass it to Retriever(...) alongside the existing indexes and RRF handles the merging automatically.

This is what production RAG pipelines actually look like — a small number of specialised indexes, each good at a different kind of query, merged through a rank-fusion layer that doesn't care about their internal scoring. You can start with the two you built in this module and grow the system organically as failure modes demand new indexes.

What you now have

A complete, extensible RAG pipeline:

• Chunking — structure-based when you have it, size-based with overlap when you don't.
• Vector search — semantic retrieval via VoyageAI embeddings and a vector index.
• Lexical search — BM25 for exact-token and rare-term queries.
• Hybrid retrieval — a Retriever wrapper that queries both and merges results with reciprocal rank fusion.
• Final prompt construction — slot the top-k results into an XML-tagged context block and hand the whole thing to Claude.

In the next module you'll start using Claude's advanced features — extended thinking, image and PDF support, prompt caching, citations — which build on top of everything you've done so far. Tool use gives Claude capabilities; RAG gives Claude knowledge. Together they are the foundation of any Claude application that has to deal with the real world.