BM25 lexical search

A user searches your support knowledge base for INC-2023-Q4-011. This is an incident ID — a specific, exact string that appears in exactly one section of your documentation. Semantic search returns a plausible-looking cybersecurity section that does contain the ID, plus a finance section that doesn't mention it at all but happens to use similar language. The right answer is mixed in with the wrong one. This is a failure mode embeddings inherit from the fact that they measure meaning, not exact matches, and the classical fix is BM25.

Semantic search isn't enough on its own

Embedding-based search excels at understanding meaning — "car" matches "automobile," "login" matches "authentication" — but that strength is also a weakness. When the query is something like:

• INC-2023-Q4-011 — an incident ID
• HTTP 418 — a status code
• claude-sonnet-4-0 — a model identifier
• amoxicillin — a specific drug name
• func_process_order — a function name

…the important thing is the exact token, not its semantic neighbourhood. Embeddings will happily return chunks that are "similar in meaning" to the query's context, which is not what you want when a user needs the paragraph that literally contains the identifier.

The fix isn't to abandon semantic search — it's to run a classical keyword search alongside it and merge the results. That classical piece is BM25.

What BM25 is

BM25 (Best Match 25) is a lexical search algorithm that ranks documents by how many rare, query-relevant terms they contain. It's been around since the 1990s, it's what search engines were built on before neural embeddings existed, and it's still the right tool when exact tokens matter.

BM25 processes a query in four conceptual steps:

1. Tokenise the query. Break the user's input into individual terms. "What happened with INC-2023-Q4-011?" becomes something like ["what", "happened", "with", "inc", "2023", "q4", "011"] after lowercasing and punctuation handling.

2. Count term frequency across the corpus. How often does each term appear in the whole collection of documents? "what" probably appears thousands of times. "inc-2023-q4-011" probably appears once.

3. Weight terms by inverse document frequency. Rare terms get high weight; common terms get low weight. This is the core insight that makes BM25 work. The word "what" is almost useless for finding relevant documents because it appears everywhere. The token inc-2023-q4-011 is almost uniquely identifying because it appears nearly nowhere. BM25 biases its rankings accordingly.

4. Score each document by summing the weighted term matches it contains. Documents with more instances of the high-weighted terms rise to the top; documents with only common-word matches sink.

The result: when you search for an incident ID, the document that actually contains that exact string shoots to the top because nothing else in the corpus has that token.

Implementing BM25 search

The course provides a BM25Index class with the same shape as VectorIndex — add_document and search methods — so you can use it as a drop-in alternative search layer. Setup looks almost identical to the vector index:

# 1. Chunk the text by section
chunks = chunk_by_section(text)

# 2. Create a BM25 store and add documents
store = BM25Index()
for chunk in chunks:
    store.add_document({"content": chunk})

# 3. Search the store
results = store.search("What happened with INC-2023-Q4-011?", 3)

for doc, score in results:
    print(score, "\n", doc["content"][:200], "\n----\n")

A few differences from the vector flow:

• No embedding step. BM25 works directly on raw text. No VoyageAI API call, no precomputed vectors — just tokenisation.
• add_document takes a metadata dict directly, no separate embedding parameter. The BM25 index tokenises the content field internally.
• Search takes a query string, not an embedding. BM25 does its own tokenisation at query time.

Run the search for the incident ID and you'll see the sections that actually contain it rise to the top — the cybersecurity section, the software engineering section. No more off-topic finance-section noise that semantic search produced for the same query.

Why BM25 works where embeddings fail

BM25 is blind to meaning but exquisitely sensitive to rare tokens. It doesn't know that "car" and "automobile" are related, and it doesn't care. What it does know is that if an unusual string appears in your query and also appears in a document, that document is probably relevant — and if the string is rare enough, that single match is probably more diagnostic than any amount of fuzzy semantic resemblance.

This makes BM25 strong at:

• Technical terms that don't have semantic synonyms
• IDs and codes — incident IDs, SKUs, error codes, version strings
• Rare proper nouns — unusual names, niche product references
• Specific phrases that the user literally copy-pasted from somewhere

And weak at everything embeddings are good at:

• Synonym matching
• Paraphrased queries
• Conceptual similarity without shared vocabulary
• Natural-language questions where the important content words don't match the document's wording

The strengths and weaknesses are complementary. Semantic search handles meaning; BM25 handles exact strings. Neither one alone is good enough for production RAG. Both together, merged correctly, give you a retrieval layer that's robust to the full range of real user queries.

The next step: merging results

You now have two search systems answering the same question differently. The next lesson builds a hybrid retriever that queries both in parallel and combines their results using reciprocal rank fusion — a simple scoring method that rewards documents for ranking highly in both indexes. That hybrid layer is the production-grade RAG pipeline we've been building toward across the whole module.