Chunking clinical guidelines — where recall is won or lost

Here's the uncomfortable truth the evaluation surfaced: the single biggest weakness across every retriever is recall — the passage that best answers a question often just isn't in the retrieved set. And retrieval can only return chunks that exist. If the right sentence got split across two chunks, or buried in a 2,000-token wall of unrelated text, no retriever recovers it. Recall is a chunking problem before it's anything else.

PDF to clean text

Clinical guidelines are hostile source material — multi-column layouts, tables, figures, footnotes, cross-references. The documents go through Docling for OCR and layout parsing, which turns each PDF into structured markdown with the headings, lists and tables preserved rather than flattened into a soup of text. Getting this stage right matters more than it looks: everything downstream inherits its mistakes.

Split by section, then by size

A good chunk is about one thing. Guidelines hand you that structure for free — numbered sections and headings — so the splitter follows the document's own boundaries first, keeping a section together as a unit. Only when a section runs long does it sub-split, with a cap of 2,000 tokens per chunk and a 200-character overlap between consecutive chunks so a fact that straddles a boundary still appears whole in at least one of them. Splitting by structure beats splitting by a fixed window, because a fixed window happily cuts a dosing table in half.

The 512-token catch

An honest wrinkle I'll name rather than hide. The embedding model truncates its input at 512 tokens — but chunks can be up to 2,000. So a long chunk is embedded on roughly its first quarter, and the tail contributes nothing to whether it's retrieved. That mismatch is almost certainly part of the recall problem, and it's near the top of the fix list: either bring the chunk size down toward the embedding window, or embed a long chunk as several sub-windows and pool them. It's the kind of quiet defect that a metric points at but a demo never shows.

The duplicate corpus I didn't know I had

Late in the project I found the store held two ingestion passes — 43 document names for roughly 22 guidelines, because an earlier, coarser pass had never been cleaned out. Every retriever comparison had quietly been running over duplicated text. De-duplicating brought it down to 27 clean documents and changed the retriever ranking. A later pass found one more near-duplicate pair — two documents overlapping by 88% — which settled the corpus at 24 documents, 1,768 chunks. The lesson repeats itself across this series: fix the data before you trust the leaderboard.

A tree on top of the chunks

Leaf chunks aren't the only thing in the store. RAPTOR builds a hierarchy above them — clustering related chunks, summarising each cluster with an LLM, and repeating — so a broad question can match a summary node instead of hunting for one perfect leaf. That's covered in the retriever deep-dive; the point here is that even the summaries are only as good as the chunks they're built from.

The other half of this is the pipeline overview, and the knobs that tune all of it live in the parameters. Getting the unglamorous data layer right — where most RAG projects quietly lose — is what I do at twentytwotensors. Get in touch.