Denshin / Blog / AI

RAG chunking strategies that actually improve answers

Most bad RAG is a retrieval problem, not a model problem. A practical guide to fixed size, recursive, structure aware and semantic chunking, overlap that pays for itself, metadata worth keeping, handling tables, code, PDFs and scans, parent document retrieval, hybrid search, reranking, and measuring recall@k first.

Denshin Engineering · Engineering Team · 27 August 2026 · 8 min read

When a retrieval system gives bad answers, the first instinct is to blame the model and go shopping for a bigger one. In our experience that is almost never where the fault is. The model can only reason over the text you handed it, so if the right three paragraphs never made it into the prompt, no amount of model upgrade will save the answer. Bad RAG is a retrieval problem, and retrieval starts with how you cut documents into pieces. This is the unglamorous work that decides whether the feature is useful.

What chunking is, and why it decides answer quality

Chunking is the process of splitting source documents into passages small enough to embed and retrieve individually, but large enough to still make sense on their own. Every chunk is a bet that a future question will be answerable from that passage alone. Get the boundaries wrong and you retrieve a paragraph whose subject was named two paragraphs earlier, or a definition severed from the condition that qualified it.

The practical consequence is that chunking is not preprocessing you do once and forget. It is a design decision with the same weight as your schema design, and like a schema it is expensive to change later because every change means re-embedding the corpus.

Four chunking strategies, ranked by how often they are the right call

Structure aware chunking

Split on the document's own structure: headings, sections, list boundaries, slide breaks, table rows. This is the default we reach for, because the author already did the semantic segmentation for you when they wrote the headings. A section of a policy document or a subsection of an API reference is a coherent unit of meaning by construction. Where source documents have real structure, this beats everything else and costs the least to implement.

Recursive character splitting

Split on a priority list of separators: paragraphs first, then sentences, then words, only falling back to a hard character cut when a piece is still too big. This is the sensible general default when your documents are unstructured prose. It respects natural boundaries where they exist and degrades gracefully where they do not.

Fixed size chunking

Cut every N tokens regardless of content. It is trivial to implement and it will slice a sentence in half, separate a heading from the paragraph it introduces, and split a table row from its header. Use it as a baseline to beat, or for genuinely uniform machine generated text such as log lines. Do not ship it as your answer for a document corpus.

Semantic chunking

Embed sentences, then start a new chunk where consecutive sentences drift apart in embedding space. Conceptually elegant, and it does produce good boundaries on flowing narrative text. It is also the most expensive option to run over a large corpus and the hardest to debug when it misbehaves, because the boundaries are not inspectable by a human reading the source. Try it after the cheaper options have plateaued, not before.

Strategy	Best for	Main risk	Cost to run
Structure aware	Docs, wikis, API references, policies	Sections that are far too long	Low
Recursive	Unstructured prose	Boundaries ignore meaning	Low
Fixed size	Uniform machine text, baselines	Cuts mid sentence and mid table	Very low
Semantic	Long narrative text	Opaque, hard to debug	High

Overlap: useful up to a point, then it wastes budget

Overlap means repeating the tail of one chunk at the head of the next, so a fact that straddles a boundary survives in at least one piece. A modest overlap, roughly ten to fifteen percent of chunk length, is cheap insurance for prose. Beyond that you are paying storage and embedding cost to store the same sentences several times, and you make duplicate retrieval more likely: the top results all contain the same overlapping passage, so your top five is really a top two wearing a disguise.

If you already chunk on structure, you often need no overlap at all, because your boundaries are meaningful rather than arbitrary. Overlap is a patch for bad boundaries. Fix the boundaries instead where you can.

Metadata that earns its keep

A chunk without metadata is a floating string. Attach fields you will actually use in filtering, ranking or the citation you show the user. Anything else is noise you will maintain forever.

Source identity: document id, title and a link, so the answer can cite something a human can open.
Section path: the heading breadcrumb, for example "Billing / Refunds / Partial refunds". Prepending this to the chunk text before embedding is one of the cheapest quality wins available, because it restores the context the split removed.
Date: created and last updated. Essential for boosting current material over superseded material, and for spotting stale corpus rot.
Permissions: the visibility group or owner. Filter on this before retrieval, never after generation. Retrieval is your access control boundary, which is one of the reasons retrieval beats training for anything with per-user document access, as we argue in RAG vs fine-tuning.
Type: policy, FAQ, code, changelog. Useful for routing and for filters the user can set in the interface.

{
  "id": "billing-refunds-partial-0003",
  "text": "Billing / Refunds / Partial refunds\n\nPartial refunds are issued when ...",
  "doc_id": "billing-policy-v7",
  "section_path": ["Billing", "Refunds", "Partial refunds"],
  "updated_at": "2026-07-14",
  "visibility": "staff",
  "type": "policy"
}

Documents that break naive chunkers

Four categories cause most of the pain, and each needs a deliberate decision rather than a default.

Tables. A row split from its header is meaningless. Either keep a whole table in one chunk with its caption, or serialise each row into a sentence that repeats the column names. Both work. Cutting a table by character count does not.
Code. Split on function or class boundaries using a parser, not on blank lines. Keep the file path and the imports summary in metadata so a retrieved function is identifiable.
PDFs. Multi column layouts routinely extract in the wrong reading order, interleaving two columns line by line. Always eyeball the extracted text for a few documents before trusting the pipeline. Headers, footers and page numbers should be stripped or they pollute every chunk.
Scanned documents. These need OCR before anything else, and OCR quality varies enormously with scan quality. Budget time to check the output, and consider excluding documents below a confidence threshold rather than poisoning the index with garbled text.

Small to big: retrieve precisely, generate with context

Here is the trick that resolves the central tension of chunk sizing. Small chunks match questions precisely because they are focused. Large chunks answer questions well because they contain surrounding context. You do not have to choose.

Index small chunks, but when one is retrieved, pass the larger parent unit to the model: the full section, or the chunk plus its neighbours. Retrieval precision comes from the small piece, answer quality comes from the big one. This pattern goes by parent document retrieval or small to big retrieval, and it is usually a bigger improvement than any amount of fiddling with token counts.

Hybrid search and reranking, the highest leverage upgrades

Pure vector search has a known weakness: it is fuzzy about exact strings. Error codes, product SKUs, surnames, version numbers and internal jargon are precisely the terms users search for, and precisely what similarity is worst at. Run keyword search alongside vector search and merge the result sets. The cheap and effective merge is reciprocal rank fusion, which combines by position rather than by score and therefore avoids the problem of two scoring systems on incompatible scales. If you want the mechanics of the vector half, we cover it in embeddings and vector databases explained.

Then rerank. Retrieve generously, perhaps the top thirty candidates, and pass them through a cross encoder reranker that scores each candidate against the query directly rather than comparing precomputed vectors. Keep the top five. In our experience this is the single highest leverage change available to a mediocre RAG system, because first stage retrieval optimises for recall and reranking supplies the precision. It costs latency, so measure that trade rather than assuming it is free.

How to measure it before touching the generation prompt

Almost every team tunes the generation prompt first, because it is the visible part. That is backwards. If the right passage was never retrieved, the prompt cannot fix it, and you will spend a week rewording instructions to compensate for a retrieval failure.

Hand build fifty questions from real user needs. For each, record which document and section contains the answer. This is a day of work and it is the highest value day in the project.
Measure recall@k: for what fraction of questions does the correct source appear in the top k retrieved chunks? Track k at five and at twenty.
Change one thing at a time: chunking strategy, then section path prefixing, then hybrid search, then reranking. Re-run after each. Keep the numbers in a file in the repo.
Only when recall@5 is high do you start tuning the generation prompt. Now failures are genuinely generation failures, and treating them as such actually works.

Building that question set is the same discipline as writing evals for anything else you cannot unit test, a subject we go into in evals that catch real failures.

A debugging loop that works

When a specific answer is wrong, do not guess. Log the retrieved chunks for every request in development, and read them. Then walk the ladder: was the correct chunk in the index at all? If not, it is an ingestion or parsing bug. Was it in the index but not in the top k? That is an embedding or search problem, so try hybrid search and a section path prefix. Was it in the top k but ranked low? That is a reranking problem. Was it retrieved and in the prompt, but the answer still ignored it? Only now is it a generation problem, and only now should you edit the prompt.

What to do next

Write fifty questions with known answer locations before changing any code.
Switch to structure aware chunking and prepend the heading breadcrumb to each chunk before embedding.
Add keyword search next to vector search and fuse the rankings.
Add a reranker over a generous candidate set and measure the latency cost honestly.
Log retrieved chunks in development so every bad answer can be traced to a rung on the ladder above.

If you have a retrieval feature that demos well and disappoints in production, that gap is usually diagnosable in a couple of days with the loop above. Talk to us if you would like a second pair of eyes on it.

Tags: RAG, Chunking, Vector Search, Embeddings, LLM

All posts · Work with Denshin