Chunking Strategy and Contextual Retrieval

A chunk is a standalone claim, not a slice of a document

The RAG architecture lesson gave you the working baseline: split at 300 to 800 tokens with 10 to 20 percent overlap, embed each piece, index it. That baseline is a starting point, and underneath it sits the decision that sets your ceiling. A chunk plays two roles at once. It is the retrieval unit, so it has to be findable by a query written by someone who has never seen the document. It is also the context-budget unit, so eight of them have to fit in a prompt with room left over for an answer. Those two roles pull opposite ways: retrievability wants each chunk to carry enough surrounding detail to identify itself, and the budget wants each chunk small and dense. Every technique in this lesson is a different way to buy the first without paying for it in the second.

The orphaned claim

Here is the failure, on one real-shaped paragraph.

document: "Q3 2025 investor letter, Northwind Logistics" (heading, page 1)

  ... freight volumes recovered through the summer as port
  congestion eased. Revenue grew 3% that quarter, and operating
  margin held at 11.2% ...

fixed split at 600 tokens, no overlap:

  chunk 41 ends   "... as port congestion eased."
  chunk 42 begins "Revenue grew 3% that quarter, and operating
                   margin held at 11.2%."

query: "how much did Northwind revenue grow in Q3 2025"

  chunk 42 contains no "Northwind", no "Q3", no "2025".
  its embedding lands in the region of the space where every
  company's revenue sentence lands, and nothing in the chunk
  pulls it toward this company or this quarter.

The chunk is not badly written and it is not too long. It is unretrievable, because the words that identify it are in the heading four hundred tokens above it. Call this an orphaned claim: a true statement that no query can reach, because the query has to name the subject and the chunk does not.

Overlap is a guess, and a narrow one

Overlap exists to survive a boundary that lands mid-thought. It duplicates the last N tokens of each chunk into the front of the next, so a sentence cut in half appears whole in at least one chunk. That is worth having and it is cheap in engineering. It is not cheap in index size: 20 percent overlap is 25 percent more chunks, 25 percent more vectors, and 25 percent more candidates competing for your top-k, because the stride between chunk starts drops to 80 percent of the chunk size and 1 / 0.8 is 1.25. And it fixes only local damage. The orphaned claim is not a boundary problem. The context that would have made chunk 42 findable was never adjacent to it.

Structure-aware splitting: split where the document already splits

The next lever costs nothing at query time. Documents already carry their own boundaries, and a splitter that reads them makes better chunks than one counting tokens. Split on headings, and every chunk can inherit its heading path as a prefix. Keep a fenced code block whole, because half a function is not a smaller function, it is a syntax error with an embedding. Element-based chunking (splitting on the structural elements a document-understanding model annotates, rather than on paragraphs or a token count) is measured to improve RAG results on financial reports, and it reaches a good chunk size without tuning one.

Tables deserve their own rule, and it is not the obvious one. A table split across two chunks is worse than a table truncated at a chunk boundary. Truncated, you lose rows and keep the header binding, so what survives is still true. Split, the second chunk is a grid of numbers whose column meanings are in a chunk it will never be retrieved with, and the model that reads it will confidently attach the wrong header to the right number. Prefer to keep a table whole, and when it will not fit, repeat the header row into each piece.

Contextual retrieval: spend a generation at ingestion

The 2024 answer to the orphaned claim was more overlap. The current answer is to write the missing context onto the chunk before embedding it. At ingestion, for each chunk, you send the whole document plus that chunk to a model and ask for one or two sentences situating the chunk in the document. You prepend the result to the chunk text, then embed the combined string and index it in BM25 as well.

the ingestion-time prompt, run once per chunk, with the whole document in view:

  <document>
  {{WHOLE_DOCUMENT}}
  </document>
  Here is the chunk we want to situate within the whole document
  <chunk>
  {{CHUNK_CONTENT}}
  </chunk>
  Please give a short succinct context to situate this chunk within the
  overall document for the purposes of improving search retrieval of the
  chunk. Answer only with the succinct context and nothing else.

what gets embedded, before:

  "Revenue grew 3% that quarter, and operating margin held at 11.2%."

what gets embedded, after (the generated context is 50 to 100 tokens):

  "This chunk is from Northwind Logistics' Q3 2025 investor letter, in
   the section reporting consolidated results for the quarter ending
   September 30, 2025.
   Revenue grew 3% that quarter, and operating margin held at 11.2%."

The chunk is now a standalone claim. The query names Northwind, Q3 and 2025, and so does the text being embedded. Anthropic published the measurement on a top-20 retrieval evaluation, and the arithmetic is worth doing rather than reading, because it tells you which stage to buy next.

top-20 chunk retrieval failure rate, same corpus, same eval set

  baseline: embeddings + BM25                   5.7%
  contextual embeddings                         3.7%   (5.7 - 3.7) / 5.7 = 35% fewer failures
  contextual embeddings + contextual BM25       2.9%   (5.7 - 2.9) / 5.7 = 49% fewer failures
  the same, then rerank top 150 down to top 20  1.9%   (5.7 - 1.9) / 5.7 = 67% fewer failures

what the ingestion pass costs, at the published $1.02 per million document tokens
(the whole document rides in the prompt for every chunk, so prompt caching is what
makes that figure attainable rather than a per-chunk re-read)

  200,000 documents x 4,000 tokens = 800,000,000 document tokens
  800 x $1.02 = $816 once, plus a re-run for each document that later changes

Two things fall out of that table. The reranker and the chunking work are not competing; they stack, and the last row is the first three techniques together. And the cost is a one-time ingestion charge measured per million document tokens, which means it is a capital expense you can compute exactly before committing, unlike a query-time technique whose bill grows with traffic forever.

Late chunking: one forward pass instead of one call per chunk

There is a cheaper way to get a chunk embedding that has seen the whole document, and it changes the order of two operations you already know. A transformer embedding model produces one vector per token and then pools them into a single vector. Naive chunking splits first and pools within each chunk, so no token ever attends outside its own chunk. Late chunking runs the long-context model over the whole document first, then pools per chunk afterward.

naive chunking
  split -> embed(chunk 1), embed(chunk 2), ...
  the forward pass for chunk 42 sees 600 tokens and nothing else

late chunking
  embed(whole document) -> token vectors t1 ... tN     one forward pass
  pool(t1    ... t600)  = chunk 1 vector
  pool(t601  ... t1200) = chunk 2 vector
  ...
  every pooled vector is built from token states that attended
  to the heading on page 1, so "revenue grew 3%" is already
  colored by "Northwind" and "Q3 2025"

One forward pass per document instead of one generation per chunk, and no extra training. The constraint is that the model has to be a long-context embedding model, and the document has to fit its window; beyond that window you are back to splitting, just at a coarser grain.

Table

The middle column is the one that predicts retrieval failure. The rows are ordered by how much of the document a single chunk embedding is allowed to know about, and the cost column is what each step of that ladder is priced at.

Approach	Ingestion work per document	What one chunk embedding has seen	What it costs
Fixed-size split	one pass, no model call	its own tokens only	cheapest; orphaned claims survive
Overlap	one pass, no model call	its own tokens plus an adjacent margin	index grows by the overlap fraction
Structure-aware split	one pass plus layout parsing	its own tokens plus its heading path	parser quality becomes a dependency
Contextual retrieval	one generation per chunk	a generated summary of the whole document	$1.02 per million document tokens
Late chunking	one long-context forward pass	every token of the document	one embedding call; document must fit the window

What the measurements actually say

Both advanced strategies are real, and neither dominates. A 2026 evaluation compared them directly against fixed-size chunking and found contextual retrieval better at holding a document's meaning together but substantially more expensive in compute, while late chunking was the more efficient of the two and gave back some relevance and some completeness in exchange. That is the honest shape: a trade, measured, not a winner.

This matters more than it looks, because chunking is the stage where teams adopt a technique on reputation. Semantic chunking, which splits at points where the embedding of consecutive sentences shifts, is the upgrade most teams reach for first, and the published comparisons in this area are between fixed-size splitting, late chunking and contextual retrieval. Adopt a chunker because you measured it on your corpus, at your k, against your queries.

Interview nuance: name the measurement, not the technique. "We would use semantic chunking" is a preference. "Our top-20 failure rate is 5.7%, contextual retrieval takes it to 3.7% for a one-time $816 on this corpus, and a reranker on top takes it to 1.9%" is an engineering answer, and the second half of it, that these stack, is what shows you have run the experiment rather than read the blog post.

Recap: a chunk is both a retrieval unit and a context-budget unit, and the orphaned claim is what happens when you optimize only the second. Overlap fixes boundaries and nothing else. Structure-aware splitting is free and keeps tables and code intact. Contextual retrieval buys the largest measured drop in retrieval failure for a one-time per-million-document-token charge, late chunking buys most of the same effect for one forward pass, and the two are measured trades rather than a ranking.

Sources: Anthropic, Contextual Retrieval · Late chunking · Financial report chunking · Evaluating advanced chunking strategies