Text chunking strategies

You have a document. Before you can embed it, index it, or retrieve from it, you have to break it into chunks. How you break it up is the first real design decision in a RAG pipeline, and a bad one will silently ruin the whole system downstream. A user asks about software engineering bugs; your retrieval returns a medical research paragraph that happened to contain the word "bug" in the sense of a pathogen; Claude confidently answers the wrong question. That failure traces directly back to the chunking step.

Why chunking matters so much

Retrieval quality is bounded by chunk quality. The retrieval step can only return chunks that exist — it can't reach into the middle of a chunk and pull out a cleaner piece. If your chunks mix unrelated topics, no retrieval algorithm will cleanly untangle them. If your chunks cut off mid-sentence, relevant passages will lose meaning. If your chunks are too short, you lose the surrounding context that would have disambiguated the match.

Four strategies are worth knowing, each with different trade-offs.

Size-based chunking

Size-based chunking splits the document into fixed-length strings. The simplest possible approach and, for that reason, the most reliable production default.

def chunk_by_char(text, chunk_size=150, chunk_overlap=20):
    chunks = []
    start_idx = 0

    while start_idx < len(text):
        end_idx = min(start_idx + chunk_size, len(text))
        chunks.append(text[start_idx:end_idx])

        start_idx = (
            end_idx - chunk_overlap if end_idx < len(text) else len(text)
        )

    return chunks

Two parameters:

• chunk_size — how big each chunk is. Measured in characters here; in production you'd often measure in tokens.
• chunk_overlap — how many characters each chunk shares with its neighbours. Overlap protects you from cutting important phrases in half — if the relevant sentence lives on a chunk boundary, the overlap ensures both neighbouring chunks contain it.

The upside: it works on anything. Code, prose, CSVs, concatenated logs, PDFs converted to messy text. It doesn't care about document structure and can't be confused by missing markers.

The downside: it ignores meaning entirely. A chunk boundary can land mid-sentence, mid-word, mid-function, mid-paragraph. The overlap helps, but it doesn't fully compensate.

Structure-based chunking

Structure-based chunking splits on a document's natural boundaries — headers, paragraphs, sections. When your document has reliable structural markers, this is the cleanest option.

import re

def chunk_by_section(document_text):
    pattern = r"\n## "
    return re.split(pattern, document_text)

For Markdown, split on header markers. For HTML, split on <h2> or <section>. For a structured JSON log file, split per entry.

The upside: each chunk is a complete, meaningful unit. A "Risk Factors" section stays together as one chunk. A "Executive Summary" section is its own chunk. Retrieval becomes semantically aligned with document structure.

The downside: you need guarantees about the structure. Real-world documents — uploaded PDFs, scraped HTML, customer-submitted files — often don't have clean markers. A structure-based strategy that assumes \n## as the section delimiter will produce one giant chunk on any document without those markers. You need either a controlled corpus or a robust fallback.

Semantic chunking

Semantic chunking splits the text into sentences, then uses NLP to group related sentences together into chunks based on meaning. Consecutive sentences that are semantically close get bundled into one chunk; a shift in topic triggers a chunk boundary.

The upside: the chunks track meaning, which is exactly what you want for retrieval. You don't have the arbitrary boundaries of size-based chunking or the format dependence of structure-based.

The downside: it's computationally expensive (you're running similarity models on every sentence boundary) and complex to implement correctly. You need a sentence embedder, a similarity threshold, and some tuning to decide when a shift is "big enough" to warrant a split. For most production systems the complexity isn't worth the marginal retrieval improvement over well-configured size-based chunking.

Sentence-based chunking

Sentence-based chunking is a practical middle ground: split into sentences, then group N sentences into each chunk with optional overlap.

import re

def chunk_by_sentence(text, max_sentences_per_chunk=5, overlap_sentences=1):
    sentences = re.split(r"(?<=[.!?])\s+", text)

    chunks = []
    start_idx = 0

    while start_idx < len(sentences):
        end_idx = min(start_idx + max_sentences_per_chunk, len(sentences))
        chunks.append(" ".join(sentences[start_idx:end_idx]))

        start_idx += max_sentences_per_chunk - overlap_sentences

    return chunks

The upside: chunks always end on sentence boundaries, so nothing is cut off mid-thought. Good for free-form prose where sentences are meaningful units but there are no section headers to split on.

The downside: it struggles with content where "sentence" isn't a clean unit — code, structured data, documents where sentence boundaries are unreliable (bullet points, tables, etc.). The regex-based sentence splitter also has edge cases (Dr. Smith, e.g., quoted dialogue) that it doesn't handle gracefully.

Choosing a strategy

| Strategy | Best for | Main risk | |---|---|---| | Structure-based | Well-formatted Markdown, HTML, or any corpus you control | Fails silently on documents without expected markers | | Sentence-based | Free-form prose where sentence boundaries are reliable | Struggles with code and structured content; regex edge cases | | Semantic | Cases where retrieval quality justifies complexity | Expensive to compute and tune | | Size-based with overlap | Anything else — the reliable fallback | Arbitrary boundaries, but overlap compensates |

Size-based chunking with overlap is usually the right production default. It's simple, cheap, fast, and works on any document type including messy ones. It won't give you the most semantically perfect chunks, but it consistently produces reasonable ones — and it never silently produces one giant "chunk" that covers the whole document.

The rule of thumb: start with size-based chunking with overlap, measure retrieval quality on realistic queries, and only move to a more sophisticated strategy if the data proves it's worth the complexity. Most RAG systems don't need anything fancier.

There is no single best chunking strategy. The right approach depends on your documents, your queries, and the trade-offs you're willing to make between implementation complexity and chunk quality. What stays constant is the discipline of evaluating retrieval quality after you chunk — the test, as always, is whether real queries return real answers.