Introducing Retrieval Augmented Generation

You have an 800-page financial document and a user who wants to ask it questions. The naive instinct is to paste the whole thing into a prompt and let Claude sort it out. That instinct is wrong for three reasons: the document is often too long to fit at all, it costs real money to process every token of it on every query, and the model's effective attention degrades when the prompt is dominated by a giant document that's 95% irrelevant to the actual question. The solution is retrieval-augmented generation.

The problem with "just stuff the prompt"

Here's what the naive approach looks like:

Answer the user's question about the financial document.

<user_question>
What risk factors does this company have?
</user_question>

<financial_document>
{the entire 800 pages}
</financial_document>

This works, kind of, until it doesn't. Four failure modes, in order of how quickly they bite:

• Hard token limits. Even with long-context models, there's a ceiling. Push past it and the API rejects the request outright.
• Cost. You pay for every token on every call. Processing 800 pages of text for a question that only needed 200 words of context is expensive.
• Latency. Larger prompts take noticeably longer to process. A user-facing chat feels slow; a batch job runs longer and costs more.
• Effective attention degrades. Even within the context window, models don't pay equal attention to everything. Drowning the real question in irrelevant pages reduces answer quality.

These are real problems for any non-toy application. The question is what to do instead.

What RAG is

Retrieval-augmented generation is the pattern of preprocessing documents into smaller chunks, storing them in a searchable index, and — at query time — retrieving only the chunks most relevant to the user's question before composing the prompt.

There's a preprocessing phase that happens once:

1. Chunk the document into smaller passages (paragraphs, sections, fixed-size windows).
2. Index the chunks in a way that supports "find me what's relevant to this question" lookups — usually vector embeddings, sometimes keyword search, often both.
3. Store the index.

Then there's a retrieval-and-generation phase that happens on every query:

1. Search the index for chunks relevant to the user's question.
2. Select the top few matches.
3. Construct a prompt that includes the question plus just those chunks as context.
4. Call Claude with the slimmed-down prompt.

Instead of 800 pages going into every prompt, maybe three paragraphs do — the three paragraphs that actually contain the answer. The prompt is small, the cost is low, Claude's attention is focused on exactly the relevant material, and you can answer questions about documents that would never have fit in context to begin with.

A concrete example

A user asks: "What risks does this company face?"

Without RAG, the whole document goes into the prompt. With RAG:

1. The question gets converted into a search query.
2. The retrieval step finds three chunks from the "Risk Factors" section — maybe an overview paragraph, a paragraph about cybersecurity exposure, and one about supply chain.
3. Those three chunks are inserted into the prompt alongside the question.
4. Claude answers based on the retrieved content.

The user sees the same crisp answer they would have gotten from a full-document prompt (probably better, because the model isn't being pulled in 800 different directions). You paid for maybe 2% of the tokens and got the answer in a quarter of the time.

The benefits

• Scalable to very large documents. RAG can handle documents that don't fit in any context window, because the model only ever sees the retrieved subset.
• Scalable to many documents. A single index can cover a whole document corpus. The same retrieval step that found the right paragraph in one document can find the right paragraph across thousands.
• Cheaper and faster per query. Smaller prompts cost less, run faster, and reduce rate limit pressure.
• Focused attention. Claude does better work when the relevant material is concentrated in the prompt instead of diluted.

The trade-offs

RAG is not free. The costs:

• A preprocessing step that has to run once per document and again whenever the document changes. This is infrastructure you have to build and maintain.
• A search mechanism that genuinely finds "relevant" chunks. Bad retrieval produces bad answers, even if the underlying documents are perfect.
• Context loss. A chunk selected by the retrieval step might not carry the surrounding context Claude needs to answer the question correctly. If the relevant information is split across two chunks and only one is retrieved, the answer will be incomplete.
• Chunking is a design decision. How you break up the document shapes what the retrieval step can return. The rest of this module is mostly about making that decision well.

When RAG is worth it

RAG adds complexity. Reach for it when:

• The documents are too large to fit reliably in a prompt.
• You have many documents and can't pick the relevant one ahead of time.
• You're cost-sensitive or latency-sensitive and want to stop paying for irrelevant tokens on every call.
• You need to update the knowledge base more often than you can redeploy, and an index is a cleaner abstraction than shipping new prompts.

Skip RAG when the document fits in context, is small enough that cost doesn't matter, and doesn't change often. A single-file system prompt with 2,000 tokens of product documentation is usually fine without a RAG layer.

The key trade-off is simplicity versus scalability. RAG costs you upfront engineering for preprocessing, indexing, and retrieval, in exchange for the ability to work with document collections that simply wouldn't fit any other way. The rest of this module is the practical how-to: chunking strategies, text embeddings, vector search, lexical search, and a hybrid pipeline that combines them.