The 5 Levels of Text Splitting for Retrieval

0:00One of the most effective ways to improve a language-model application is to split your data into smaller chunks — giving the model only the information it needs and nothing more. Greg Kamradt's 69-minute tutorial walks the five levels of text splitting, from naive character counts to an LLM-driven "agentic chunker," plus a bonus round on alternative representations. It's a foundational piece on RAG, and still a useful mental model even as the tooling has moved on since its January 2024 release.

Why chunk at all

3:41Two reasons you can't just feed your whole corpus to the model. One: context limits — an upper bound on how much data you can pass in. Two: signal-to-noise ratio — as Chroma co-founder Anton puts it, "distracting information in the model's context window does tend to measurably destroy the performance of the overall application." So instead of handing over the kitchen sink, you prune the fluff. Chunking is "the process of splitting your data into smaller pieces so you can make it optimal for your task and your language model." The framing to carry through every level: don't ask "how should I chunk my data?" — ask "what's the optimal way to pass the data my model needs for this task?" There is no single right answer, and evaluation (he points to RAGAS) is what tells you if you're improving.

Level 1 — Character splitting

6:54The most basic form: split by a fixed, static character length. Pros: extremely simple. Cons: rigid, and it doesn't respect the structure of your text — it cuts words mid-way ("this is the text I would like to chu…"). 7:21This level introduces the two concepts that carry through everything: chunk size and chunk overlap. With a chunk size of 35 and overlap of 4, the last four characters of chunk one reappear at the head of chunk two — a hedge against cutting a thought in half. He demos LangChain's CharacterTextSplitter (a one-liner) and LlamaIndex's SentenceSplitter, noting the object-model difference: LangChain returns Document objects (page content + metadata), LlamaIndex returns Node objects with IDs and relationships (previous/next/source) for traversal. His aside: "I don't know anybody that does this in production."

Level 2 — Recursive character splitting

16:03Instead of a static count, you give the splitter a list of separators and let it recurse: try double-newlines (paragraphs) first, then single newlines, then spaces, then characters. Any chunk still too large after one pass drops to the next separator. The payoff: it "snaps" to word boundaries, and better, to paragraph boundaries — grouping text the way humans actually write it. The hypothesis is that a paragraph holds semantically similar content. He shows this interactively in chunkviz.com, a tool he built for the video: at chunk size 35 the character splitter produces 26 ragged chunks; the recursive splitter cleanly respects paragraph breaks. 20:38His verdict: this is his go-to splitter — a one-liner, no extra processing, "the ROI for your energy is pretty awesome." If you're starting a project, start at level 2.

Level 3 — Document-specific splitting

21:01When your data has structure, exploit it. For markdown, split on headers — a newline followed by one-to-six pound signs (H1–H6) — because headers announce what a section is about (MarkdownHeaderTextSplitter). For code, split on classes, functions, and indentation (PythonCodeTextSplitter, RecursiveCharacterTextSplitter.from_language(JS)) — a whole class stays enveloped in one chunk, which is exactly what you want. 24:52For PDFs with tables, he reaches for the unstructured library: partition_pdf extracts elements, and tables come out as HTML — because tables are easy for humans to read but hard for models, and models were trained on HTML. 28:20For images, he extracts them, then generates a text summary of each with GPT-4 Vision, and embeds the summary — so semantic search can surface the image by meaning without needing a text-and-image joint embedding (he acknowledges CLIP exists but finds it underperforms). The takeaway: your chunking strategy should adapt to your data formats, because the goal is to group similar items together.

Level 4 — Semantic splitting

32:09Levels 1–3 all keyed off physical position — like shelving books by size, not genre. Semantic splitting asks what the text means. His method: embed every sentence, then measure the distance between consecutive sentences. 35:39He first tries hierarchical clustering with a positional reward (so a short "you know" sentence glues to the long sentence before it), but finds it messy and hard to tune intuitively. The cleaner method: compute embedding distance from sentence group N to N+1 (with a one-sentence buffer on each side to reduce noise), plot the distances, and treat the 95th-percentile outliers as break points — a big distance means the text changed topic. On Paul Graham's essay this yields ~17 chunks that align with genuine thematic shifts ("these two are actually getting split up at a spot where there's a big breakpoint"). His honest caveat: it's experimental, more expensive, and slower — but "as compute gets better, there's no way we're going to do physical-based chunking anymore."

Level 5 — Agentic splitting

48:02The deep end. His question: "what if a human were to do the chunking?" You'd walk the document top to bottom, deciding per piece whether it belongs in the current chunk or starts a new one — which is pseudo-code for an agent. He's careful with the word "agent," calling it an "agent-like system" (decision-making via the model with an unbounded path). 50:27The first ingredient is propositions — standalone, self-contained sentences — drawn from LangChain's proposition-based retrieval (the "Dense X Retrieval" work). "He likes walking" is ambiguous; "Greg likes walking" stands alone. 55:06His agentic_chunker: each chunk gets a UUID, a summary, and a short title; for each new proposition, the model decides whether it belongs to an existing chunk (returning that chunk's ID) or starts a new one; summaries and titles optionally re-generate as a chunk grows. On the essay it produced clean thematic chunks — "date and times," "personal history," "returns performance and returns relationship" — the last one's title shifting as new propositions arrived. Verdict: slow and expensive, but "if you bet that language models will speed up and get cheaper, this starts to come into play."

Bonus — Alternative representations

1:02:37Chunking is only part of retrieval — the next step is embedding, and you can embed an alternative representation of your text instead of the raw chunk. Three tactics: multi-vector indexing — embed a summary (or hypothetical questions, ideal for Q&A bots) and return the raw document (LangChain's MultiVectorRetriever over Chroma); parent document retrieval — match on a small chunk (good semantic search) but return its parent for context (LlamaIndex's hierarchical node parser at 2048/512/128); and graph extraction — pull entities and relationships into a graph document via the Diffbot Transformer, so "Greg lives in New York" becomes person/entity nodes and a relationship edge, answerable by structure rather than by raw text.

The chunking commandment

The through-line, repeated at every level: your goal is not to chunk for chunking's sake — it's to get your data into a format where it can be retrieved for value later. A splitting strategy that doesn't serve the downstream task is pointless regardless of how clever it is. Start with recursive character splitting (level 2), escalate to structure-aware splitting when your data has structure (level 3), and reserve semantic and agentic chunking for when physical structure stops correlating with meaning — and always measure with evaluation, not vibes.

Claims checked

Sources & further reading