Document Chunking: The Evidence Unit That Determines Retrieval Systems

## Chunking is the resolution of retrieval

Document chunking must answer a preliminary question: what happens when the correct answer happens to span two chunks? The answer is that the retriever can only return units established beforehand. It does not have "half a sentence from the original", only chunks; if chunking cuts a sentence in half, separates a heading from its body, or splits a table into two chunks, then no chunk is complete, and even if retrieval hits, it retrieves only incomplete evidence. Conversely, if an entire document is treated as one chunk, the embedding vector averages out the document's themes, diluting any local details, and retrieval likewise cannot find that specific answer.

The causal relationship behind this is: chunk size and boundaries directly determine the smallest evidence unit the retrieval system can see, that is, the retrieval "resolution". If the resolution is set incorrectly, no matter how good the embedding model or how strong the reranker, the upstream cannot obtain complete evidence, and the downstream cannot make an independent judgment. Hitting a keyword only means that a certain word appears in this chunk; it does not mean that the chunk contains all the conditions needed for independent judgment—for example, a rule chunk for "refund within 7 days" hits "refund", but if the "list of non-refundable items" is cut into another chunk, drawing a conclusion from the former alone will inevitably be wrong.

Therefore, the input to chunking includes not only the original document structure, but also typical queries, answer spans, and the embedding budget; the output is retrievable evidence chunks with original-text positions. When establishing a chunking benchmark, you should first annotate the "minimum sufficient evidence span" from real tasks: what proportion are single-sentence facts, what proportion are conditions and exceptions within the same subsection, what proportion are associations across table rows and columns, and what proportion are multi-hop reasoning across sections. Then count the "complete evidence hit rate" under different chunking methods, rather than only counting the "chunk hit rate containing answer keywords". The latter masks problems—chunks that hit keywords but lack conditions are exactly the typical manifestation of chunking failure.

The cost of chunking should also be tiered. If measurement finds that 80% of queries need only single-segment evidence and 20% need parent-level context, then retrieval can first hit small chunks, and expand only for this 20% along adjacency or heading paths, rather than letting all requests bear the cost of large chunks. The benefits and costs of a chunking strategy are therefore not abstract trade-offs, but concrete proportions determined by the query distribution.

Chunking must also be traceable. After modifying the parser or chunk size, chunk id, original-text offset, and embedding should constitute a new version: compare the two sets of indexes offline, confirm that citations can still locate the same original text, then switch traffic. Directly overwriting the old index will make regression samples, caches, and user feedback point to chunks that no longer exist, creating an observational break that is hard to reproduce. For frequently updated documents, ensure that child chunks are invalidated together when the parent chunk changes, otherwise a combination of new heading and old body will appear. After a chunking version change, the new id, offset, and index must be retained, and the old index must not be directly overwritten, which would invalidate citations.

## Four Types of Strategies

Is "fixed 500 tokens" the default answer for all documents? This question deserves a direct answer: no. The choice of chunking strategy depends on the reliability of document structure, whether topic boundaries are clear, and whether context needs to be backfilled. There are four commonly used strategies, each with its applicable conditions and costs.

Fixed-length chunking is the simplest and most controllable: chunk according to a budget, with a consistent token count per chunk, suitable for continuous text lacking reliable structure and with fuzzy topic boundaries. Its cost is that boundaries are unrelated to semantics; sentences may be cut in half, headings may be separated from body text, and whether a chunk's content is complete depends entirely on luck. It solves the problem of "still being able to build an index when there is no better structural basis," not the problem of "evidence completeness."

Structural chunking works the other way: chunk according to the document's inherent boundaries such as headings, paragraphs, functions, etc. Chunk references built this way are stable and interpretable—you can point to a chunk and say, "It corresponds to Chapter 3, Section 2." The cost is that chunking quality inherits the quality of the parser: a bad parser inherits a bad structure, heading level recognition errors, and tables being split apart all enter the index as-is. Structural chunking is suitable for documents with reliable heading hierarchies or code structures.

Semantic chunking looks for topic boundaries based on changes in the representation of adjacent sentences: when the embedding similarity of consecutive sentences shows a significant jump, that is likely the boundary between two topics. It can adapt to the flow of the discourse and does not depend on whether the format is regular. The cost is that threshold, model, and language changes all lead to version drift—for the same document, if the embedding model or threshold is changed, the boundaries shift, and chunk IDs and references from the previous version no longer correspond.

Parent-child chunking combines small chunks with large chunks: small chunks are responsible for recall and localization, large chunks are responsible for backfilling context, striking a compromise between localization precision and evidence completeness. It solves the problem of "small chunks hitting but evidence being incomplete": in the retrieval stage, a small chunk is hit; in the generation stage, the corresponding parent chunk is retrieved as complete context. The cost is deduplication—multiple child chunks may point to the same parent chunk, and if deduplication is not performed, the same large context segment will be brought back repeatedly, wasting budget and diluting the signal.

These four types of strategies can be combined. A common combination method: first establish uncrossable hard boundaries by chapter to ensure that chunks are never concatenated across chapters; then split overly long chapters internally by paragraph or semantic boundaries; finally, inherit heading paths for each short chunk so that chunks separated from the original text can still indicate where they come from. The order of combination is meaningful: hard boundaries ensure evidence is not misattributed, internal splitting controls chunk size, and heading inheritance compensates for the context lost by small chunks.

The input to strategy selection is document structure reliability, topic boundaries, and backfill needs; the output is fixed-length, structural, semantic, or parent-child chunking and their combinations. When evaluating any chunking strategy, always preserve the chunker version and original text offsets; otherwise, you cannot explain whether a change in a particular hit comes from content changes or boundary changes. There is only one final standard for comparison: measure evidence completeness against real queries. No strategy is the default answer for all documents; claiming that a particular strategy is universal itself violates the first principle of chunking—chunking must serve evidence, not habit.

## Benefits and Costs of Overlap

Adjacent chunks often retain a segment of overlapping content. This has only one purpose: to ensure that a sentence that happens to fall exactly on a chunking boundary remains complete in at least one of the chunks. Fixed-length chunking is most likely to cut a sentence in half at the boundary; overlap gives the boundary sentence a second chance—the first half is incomplete in chunk A, but chunk B begins with the complete sentence. The benefit is therefore specific: the probability of boundary truncation decreases, and the probability that cross-boundary evidence is completely contained in some chunk increases.

The costs are equally specific, and there are three of them. First, the index becomes larger: the duplicated text is embedded twice or more, and both storage and retrieval computation expand accordingly. Second, duplicate recall: the same sentence appears in multiple chunks, and a single query may rank several semantically almost identical chunks in the results, crowding out positions that should belong to other evidence. Third, context competition: the tokens available to fill the generation context after recall are limited, and duplicated content is charged repeatedly, squeezing out effective evidence.

Precisely because the benefits and costs are entangled, overlap handling must follow a sequential principle: deduplication and adjacency merging should be performed after retrieval, not by deleting duplicates at the indexing stage. Keeping overlap in the index is to give boundary sentences a chance to be matched in full; it is the duplicates in the retrieval results that need to be removed. So the flow is: overlap enters the index → retrieval recalls multiple chunks → deduplicate the results and merge them by adjacency → then calculate the context budget. If the order is reversed and duplicate text is deleted at ingestion time, it is equivalent to deleting the only complete copy of the boundary sentence yourself.

The overlap ratio is not a number to be decided off the top of your head; it should be determined through experiments on answer span: measure the proportion of answers to real queries that cross boundaries, use experiments to select the overlap window, and separately report the number of effective evidence items before and after deduplication. Before deduplication, look at the hit rate—whether overlap really causes more complete evidence to be recalled; after deduplication, look at efficiency—how much non-duplicate effective evidence remains and how many tokens it costs. Only by reporting the two numbers separately can you distinguish "overlap truly brought complete evidence" from "overlap merely repeated itself".

One easily overlooked detail: if multiple matched chunks come from the same boundary (for example, chunk 3 and chunk 4 both hit the overlap region between chunks 3 and 4), they should be merged before calculating the context budget. Otherwise the system will treat the same content as two independent pieces of evidence, doubling the budget consumption, when there is actually only one piece of evidence.

Overlap design therefore has three inputs: window length, answer cross-boundary ratio, and context budget; the outputs are the repetition range for adjacent chunks and the merging rules after retrieval. Finally, its boundary must be made clear: overlap can only repair the single type of damage called "boundary truncation". It cannot repair reading order problems—when evidence chunks are arranged incorrectly, overlap is of no help; nor can it repair permission errors—content that should not be seen still should not be seen even if overlapped. Overlap is compensation for boundary loss, not an overall remedy for chunking errors.

## Preserving Metadata and Hierarchy

To determine whether a chunk is qualified, you can use a simple test: After a paragraph leaves its heading, does it still express the same meaning? If the answer is "no", and this paragraph is cut into an isolated chunk for storage, then every time the retrieval system retrieves it, it is treating a sentence stripped of context as a fact. The heading carries exactly part of the context—a sentence under the heading "Application Conditions" and the same sentence under the heading "Exceptions" mean completely different things.

Therefore, a chunk must carry metadata, not just text. Metadata inputs include: document ID, heading path, page number or line number, version, time, and permissions. The output is a locatable, filterable, and auditable evidence unit. Locatable means the chunk can point back to the original coordinates, answering "where does this sentence come from"; filterable means the chunk can be filtered out by document, time, and permissions, answering "can this evidence still be used"; auditable means the chunk can participate in auditing, answering "what was the conclusion based on at the time".

Handling headings has a detail worth distinguishing: when necessary, you can concatenate the short heading into the embedding text to help the embedding vector understand the chunk's content, but the reference must still point back to the original location. In other words, heading concatenation is for the embedding model to "read", not for the reference to "point to". The embedding text can include "Refund Policy / Exceptions: digital goods not applicable", but the source coordinates recorded for the chunk are still the actual position of that paragraph in the original document. The two have different purposes and must not be confused—if the concatenated text is treated as the original coordinates, the reference will point to a piece of text that does not exist.

This leads to a direct constraint: a paragraph whose meaning changes after being detached from its heading cannot be used as a context-free fact. Such a paragraph must either retain sufficient hierarchical information or be retrieved together with its parent. Permissions and versions must also travel with the chunk—when the chunk moves between indexes, stays in a cache, or is cited in user feedback, its permission constraints and version number must follow it. An outdated version carrying new permissions, or a new version losing permission restrictions, are both errors caused by broken metadata propagation.

Metadata does not solve the problem of chunking itself; it solves the interpretability of a chunk after it leaves the original text: letting each isolated chunk still know its position in the document, the section it belongs to, the time it takes effect, and the readers allowed. Without these, the retrieval system returns only a pile of text fragments; with these, what it returns is evidence that can be verified.

## Non-plain text requires specialized parsing

Can tables, code, and PDFs be hard-cut character-by-character? No. The information in these formats is not in character sequences but in structural relationships: a table's meaning depends on the correspondence between row headers and column headers, code's meaning depends on function signatures and dependencies, and a PDF's meaning depends on visual reading order. Hard-cutting breaks these structural relationships in half; the retrieval failure that occurs afterwards actually happens before indexing.

Chunking tables requires preserving row and column headers. A cell "30" has meaning only under the row/column headers such as "Refund amount / within 7 days"; if the table is split row-by-row and the header remains in another chunk, each numeric chunk becomes a meaningless isolated value. The correct approach is to use the whole table or a row group with its header as the unit, so that each chunk carries the structure needed to interpret the numbers.

Code is chunked by function or class, and dependency signatures are retained. A function is the smallest understandable unit of code; splitting in the middle of a function is like splitting in the middle of a sentence. The dependency signature is the source of a function's meaning—parameter types and return types explain how the function is used. A chunked code block can serve as complete evidence of "what this code does" only if it carries the signature.

The trouble with PDFs comes before chunking: first fix the reading order and page headers. The internal text flow order of a PDF is not necessarily the visual order; two-column layouts, page headers and footers, and footnotes can all be mixed into the body text flow. Without fixing the reading order, the chunks produced are themselves out of order. If page headers are mixed into a chunk, they also contaminate the embedding vectors, adding irrelevant noise to every chunk.

The general process is therefore two-stage: the input to a dedicated parser is PDF layout, table rows and columns, a code syntax tree, or dialogue turns, and the output is chunks that preserve their respective atomic structures. The parser first restores reading order, table headers, function dependencies, or referring entities, and then chunks. Restoration comes before chunking—this order cannot be reversed, because chunking can only be performed on the basis of the restored structure.

An important diagnostic note: parsing errors are often mistaken for embedding or model failures. When retrieval effectiveness deteriorates, you should first check upstream—whether the table headers are still present, whether the reading order has been restored, and whether the function signatures have been preserved. If the chunks themselves are incomplete, even the best embeddings cannot retrieve semantics that never entered the index. Character-level hard cutting is therefore suitable only for one type of situation: continuous text whose structure is unimportant and has already been verified. Applying hard cutting to tables, code, and PDFs is not a matter of strategy preference but a fundamental error of treating structured information as a character stream.

## How to Tune Chunk Size

When tuning chunking parameters, the first question to ask is not "how large should the chunk be?" but "should we optimize Recall@k or answer accuracy?". The two cannot substitute for each other: Recall@k only tells you whether the chunk containing the answer keyword was retrieved; answer accuracy tells you whether the system ultimately answered correctly. Answer accuracy itself depends on the generation model's own capabilities and prompt design; using it alone to evaluate chunking mixes chunking issues with issues from other components.

The correct order is to first measure evidence hit and citation, then measure end-to-end answer, token, latency, and duplication rate. Chunking's direct responsibility is the evidence layer: whether the necessary evidence has completely entered the candidate chunks, and whether citations can be traced back to the original text. Once this layer passes, then test downstream—whether the answer is correct, how many tokens the context used, how much retrieval latency there is, and how much duplication exists in the retrieved results. If upstream metrics pass but downstream fails, the problem lies in generation or other components; if upstream does not pass, chunking is responsible first.

Evaluation must be bucketed by query type: fact lookup, concept summarization, and multi-hop reasoning have completely different requirements for chunk size. Fact lookup needs the minimal sufficient chunk—an accurate number or condition; concept summarization needs multiple chunks covering the entire topic; multi-hop reasoning needs the ability to retrieve two causal chains. Mixing the three query types together to compute a single average metric makes any chunking approach look good, when in fact each type may fail. Bucketed results often point to multiple indexes: use a small-chunk index for fact lookup and a large-chunk index for concept summarization, routing by query type, rather than using one compromise chunk size to serve everyone simultaneously.

The inputs for chunking hyperparameter tuning therefore have four parts: annotated queries, the necessary evidence for each query, the candidate index, and the generation task; the outputs have six: evidence recall, citation localization, answer, token, latency, and duplication rate. The first criterion for judgment is "first ensure that the necessary evidence completely enters the candidates"—when necessary evidence is incomplete, no matter how high Recall is, it is still unqualified, because what is retrieved are chunks containing keywords, not chunks containing complete evidence. After the evidence is complete, then look at the context budget and the final task performance.

As for the claim that "512 tokens is the optimal chunk size": there is no universal optimal value. 512 tokens is only a starting hyperparameter, a default starting point for the first round of experiments. The real chunk size is determined by the evidence span distribution, and this distribution varies by document and by query type. Treating the starting value as a conclusion is equivalent to ending hyperparameter tuning before measuring evidence spans.

## How should a refund policy be chunked

Use a concrete scenario to tie together the principles from the previous sections. Suppose a refund policy has length L = 120 tokens and contains two rules: a "30-day general limit" and a "quality issue exception." If the general limit and the quality exception happen to be split by a fixed window, what will the retriever see? It will see two incomplete chunks: one chunk contains only "can be refunded within 30 days" but lacks the exception, and the other contains only "quality issue" but is detached from the deadline condition. For the query "What about quality issues at 35 days?", no single chunk can independently determine the answer—the answer must use both rules.

The same policy behaves completely differently under different chunking methods. The point of comparison in Figure 1 is: fixed-length chunking treats syntactic and rule boundaries as incidental positions, and where the boundary falls depends on the window number rather than the rule structure; structured parent-child chunking, by contrast, uses small units to improve hit rate, and after a hit backfills the complete parent segment to restore conditions and exceptions.

Compare three approaches. Option 1: 40-token window, no overlap, gives 3 chunks, indexing about 120 tokens; for the query "35-day quality issue", the exception may fall exactly across a chunk boundary; the cost is low but fragile. Option 2: 40-token window, 10-token overlap, the number of chunks becomes 4, indexing about 150 tokens; the boundary sentence is more likely to fall intact within some chunk; the cost is about 25% index bloat and easy duplicate recall. Option 3: structured subchunks plus an 80-token parent segment, 3 subchunks, indexing about 120 tokens; after a subchunk hit the complete rule is returned; the cost is needing to preserve hierarchy and control the backfill budget. The three approaches spend a similar amount of index, and the differences lie in fragility, duplication rate, and backfill complexity—this is exactly why chunking needs experimental comparison rather than choosing by feel.

The numbers for Option 2 can be derived directly from the window formula. The step size of an overlapping window is w − o, and the approximate number of chunks is n = ⌈(L − w)/(w − o)⌉ + 1. Substituting L = 120, w = 40, o = 10: the step size is 40 − 10 = 30, n = ⌈(120 − 40)/30⌉ + 1 = ⌈80/30⌉ + 1 = 3 + 1 = 4. Four full chunks would embed 160 tokens, but the last chunk is less than 40, so in practice about 150; compared with the no-overlap 120, this is exactly about a 25% increase. The formula gives the approximate number of chunks—it assumes the full text is evenly chunked, and a last chunk that is not full is the norm, so the actual number of indexed tokens must be calculated based on the true final chunk length.

Is this 25% index bloat worth it? The answer cannot be fixed by experience; it can only be determined by data: what proportion of annotated answers cross boundaries? How much does the hit rate of cross-boundary evidence improve? How much better is the final answer? If cross-boundary answers are rare, this 25% is pure overhead; if conditions and exceptions often appear in pairs in rule-based documents, overlap buys back complete evidence.

Finally, give the acceptance criterion for this case. The evidence needed for the query "35-day quality issue" is the complete "30-day general rule + quality issue exception + current version". Hitting only the half-sentence containing "quality issue" does not count as success—that is the typical failure of keyword hit but incomplete evidence. A qualified evidence chunk must allow the reader to independently determine the applicable conditions and be able to return to the original location for verification. The output of chunking is thus determined: input document length L, window w, overlap o, and rule structure; output number of chunks n, indexed tokens, and complete evidence.

## How to Handle Boundary Cases Separately

Why can the same character splitter simultaneously break PDFs, tables, code, and conversations? Because the common point of these four types of material is: meaning is not in the character sequence, but in the structure outside the characters. The splitter does not recognize these structures, so it will "quietly" make a clean cut at the structural break — the resulting text is even syntactically fluent, but the semantics have already been lost. This type of failure does not produce an error, so it is called a silent failure. The atomic structures to preserve, common silent failures, and validation methods differ for each type of material:

Table failures are the most visible: a "30" that leaves the "refund days" column name is just a number. Code failures are the most hidden: the chunked function has complete syntax and correct indentation, but no one knows its parameter types and import sources; the dependency context is already lost. Conversation failures are the most common: after the pronoun "it" is separated from the speaker and the previous turn, its reference is completely dangling. The validation methods correspond to the repair methods — comparing against the rendered page can detect interleaved double columns, restoring row-column semantics can verify whether headers are still present, AST parsing and compilation checks can confirm dependency completeness, and entity binding can catch dangling references.

Parsing quality should have a separate gate before embedding. Gate metrics include: empty chunk rate, abnormally short chunk rate, title inheritance rate, table header coverage, and source text localization success rate. These metrics can be calculated at index construction time, without waiting for end-to-end performance to degrade before tracing back. Without this gate, "retrieval failure" looks like an embedding model problem, but the actual cause is that correct sentences never entered the index in a complete, understandable form — the error occurs before retrieval, but is only noticed after retrieval.

Finally, delineate the boundaries of remedial measures. Overlap can only mitigate accidental boundaries: if a sentence happens to fall at the window edge, overlap gives it a chance to remain intact in another chunk. But overlap cannot fix incorrect reading order — text with interleaved double-column order remains out of order no matter how many times it overlaps; cannot fix permission cross-library contamination — content that should not appear will only expand leakage when repeated; and cannot fix version mixing — content generated after overlapping old and new text is still self-contradictory. A larger overlap also has a side effect: the same evidence crowds the top-k with multiple near-duplicate chunks, requiring deduplication based on document id and adjacency relationships. Therefore, handling boundary materials has two outputs: first, evidence chunks that can independently restore meaning; second, parsing quality metrics. Only by confirming that the structure is not lost through sampling comparison, header coverage, AST checks, and entity binding can we rule out the possibility that "retrieval failure is actually because the index was never built."

## Connecting the Causal Chain

The entire content of chunking can be threaded through a causal chain from problem to practice. The problem lies at the start of the chain: a retrieval system can only return chunks it has previously created; if evidence is fragmented, cut incorrectly, or lost, no downstream step can recover it. Starting from this problem, each step is a transformation of the previous step's output, and each step has a corresponding way to be verified.

Step one, parse the document to recover structure. The reading order of PDFs, table row and column headers, code syntax trees, and dialogue coreference relations all need to be recovered at this step. If structure is not recovered before chunking, the error occurs before indexing and will be misattributed to an embedding or model failure. The output of this step is structurally reliable text and boundary markers.

Step two, choose the unit according to query granularity. Fact lookup, concept summarization, and multi-hop reasoning require different chunk sizes; the basis for choosing the unit is the distribution of annotated evidence spans, not a fixed token count. If the granularity is chosen incorrectly, the resolution will not match the query.

Step three, add necessary overlap and hierarchical metadata. Overlap keeps sentences that happen to fall on a boundary intact in at least one chunk; title paths, document IDs, versions, times, and permissions propagate with the chunks, so that chunks separated from the original text remain locatable, filterable, and traceable. The overlap ratio is determined by the proportion of answers that cross boundaries; if metadata is missing, chunks cannot be verified.

Step four, embed and build the index. Only at this point do chunks become vector units that retrieval can hit. The index stores the output of this step; any change in chunking parameters creates a new version, keeping new chunk IDs and original text offsets, rather than overwriting directly.

Step five, post-recall adjacency expansion and deduplication. Overlap is a benefit in the index but a cost in retrieval results: after retrieval, merge matched chunks that come from the same boundary according to adjacency relationships, deduplicate by document ID, and only then calculate the context budget. The order must not be reversed—deduplicating before indexing would delete the only complete copy of boundary sentences.

Step six, tune parameters using both evidence and answer metrics. Evidence hits and citation localization come before end-to-end answers: first ensure that the necessary evidence is complete in the candidates, then look at answers, tokens, latency, and duplication rate. High Recall with incomplete evidence still means the chunking is substandard; wrong answers with complete evidence mean the problem lies after chunking. Both levels jointly determine parameters, rather than blindly trusting any single metric.

Each link in this chain presupposes the previous link, and the verification points are exactly distributed on the output of each link: whether structure is recovered, whether units match granularity, whether metadata propagates with chunks, whether the index can trace back versions, whether deduplication is performed after recall, and whether metrics are stratified. Chunking is not a one-time parameter setting, but a process in which every change along this chain requires re-validation—this is also why chunking versions must be traceable: once observation breaks, it is impossible to determine whether a failure at a certain link is caused by content changes or boundary changes.