Implementing the RAG flow

The conceptual tour is over. Time to write the code that turns a Markdown document into a working semantic search pipeline. By the end of this lesson you'll have five concrete functions stitched together — chunk, embed, store, query, retrieve — and a real example query returning real relevant chunks.

The five steps, in code

The pipeline lines up with the flow from the last lesson:

1. Chunk the text by section.
2. Generate embeddings for each chunk.
3. Create a vector store and add each embedding plus its source text.
4. Generate an embedding for the user's question.
5. Search the store for the top-k most similar chunks.

Step 1 — Chunk the text

Load the source document and split it into sections using the chunk_by_section function from the chunking lesson:

with open("./report.md", "r") as f:
    text = f.read()

chunks = chunk_by_section(text)

chunks is now a list of strings — one per section of the Markdown report. Printing chunks[2] should show you the table of contents or whatever the third section happens to be, confirming the chunker is running.

For this example we're using structure-based chunking because the source is well-formed Markdown. In a production system you'd validate that every document you ingest has the expected structure, or fall back to size-based chunking when it doesn't.

Step 2 — Generate embeddings for every chunk

Extend the generate_embedding helper from the previous lesson to handle a list of strings, not just one. VoyageAI's API supports batching — you can pass a whole list and get a whole list of embeddings back in a single call:

def generate_embedding(text, model="voyage-3-large", input_type="document"):
    if isinstance(text, str):
        text = [text]
    result = client.embed(text, model=model, input_type=input_type)
    return result.embeddings  # list of embeddings, one per input

Two changes from the single-string version:

• Normalises to a list whether you pass a string or a list of strings.
• Returns the full list of embeddings rather than just the first one.
• Defaults input_type to "document" because you'll call this during indexing far more often than for queries.

Now generate embeddings for the chunks in one call:

embeddings = generate_embedding(chunks)

One API call, one embedding per chunk. Batching matters at real scale — serialised per-chunk calls get expensive and rate-limited fast.

Step 3 — Create a vector store and populate it

Use a VectorIndex class — the course's minimal in-memory vector database that supports add_vector and search. In a production system you'd swap this for a real vector database (pgvector, Qdrant, Pinecone), but the API is intentionally similar.

store = VectorIndex()

for embedding, chunk in zip(embeddings, chunks):
    store.add_vector(embedding, {"content": chunk})

For each chunk, add the embedding alongside a metadata dict containing the original text under the "content" key. Two details worth pausing on:

Store the original text. When you query later, the database gives you back embeddings and whatever metadata you stored with them. The embeddings themselves are unreadable floats — without the original text, retrieval results are useless. Always carry the source text as metadata so you can include it in the eventual Claude prompt.

Metadata is flexible. {"content": chunk} is the minimum, but you can attach anything you'd want to filter or display: source document ID, section number, author, timestamp, page number. Whatever you put here is what you can reference at retrieval time.

Step 4 — Process a user query

When a query comes in, embed it — this time as a query, not a document:

user_embedding = generate_embedding(
    "What did the software engineering dept do last year?",
    input_type="query",
)[0]

Two notes:

• input_type="query" — VoyageAI's embedding model uses different internal paths for queries and documents. Matching the mode to the actual usage improves retrieval quality.
• [0] — because our batched generate_embedding returns a list, unwrap the first (and only) element.

Step 5 — Search the vector store

results = store.search(user_embedding, 2)

for doc, distance in results:
    print(distance, "\n", doc["content"][:200], "\n")

store.search(query_embedding, k) returns the top-k matches as tuples of (metadata_dict, distance). Lower distance means closer match. Loop through the results and print the distance alongside a preview of the chunk content.

For a query about software engineering, you'd typically see something like:

0.71
## Section 2: Software Engineering — Project Phoenix Stability Enhancements
...

0.72
## Methodology
...

The software engineering section is the top match (distance 0.71), with a methodology section as a close runner-up. Both have cosine distances below 0.75, which for real-world queries is a reasonable relevance threshold.

What the final prompt looks like

With the retrieval step wired up, composing the final Claude call is a matter of joining the retrieved chunks into a context block and calling client.messages.create():

retrieved = store.search(user_embedding, 3)
context = "\n\n".join(doc["content"] for doc, _ in retrieved)

prompt = f"""
Answer the user's question based on the report sections below.

<user_question>
What did the software engineering department do last year?
</user_question>

<report_sections>
{context}
</report_sections>
"""

messages = []
add_user_message(messages, prompt)
answer = chat(messages)

This is the full RAG loop: chunk offline, embed offline, store offline, embed query, retrieve top-k, compose the prompt, and ask Claude. The prompt contains only three chunks — not the whole document — and Claude has exactly the context it needs to answer correctly.

When the basic approach breaks down

This pipeline works well for a lot of cases. It also fails in some predictable ones:

• Exact-term queries — a user searches for INC-2023-Q4-011 (a specific incident ID) and semantic search returns sections that are conceptually related but don't actually contain the ID. Embeddings are great at meaning and bad at exact strings.
• Proper nouns and rare terms — technical vocabulary, product names, identifiers, codes — anything where the exact token matters more than semantic similarity.
• Short queries — "Q4 2023" on its own doesn't have enough semantic content for the embedding model to find the right section.

The next lesson covers BM25 — a classic lexical search algorithm — and then the lesson after that combines it with the vector search you just built into a hybrid retriever that's robust to both kinds of queries.