How RAPTOR actually works — and the three ways my code quietly wasn't it

I have a corpus of clinical guidelines — NICE, KDIGO, UKKA — and they have an annoying property for retrieval: the same fact is spread across all of them, rephrased a dozen ways. Plain semantic search over that is unstable. Ask the same question twice and it lands on a different paraphrase each time, because twenty near-identical chunks are all a similar distance from the query.

RAPTOR (Sarthi et al., Stanford, ICLR 2024) is a clean answer to exactly that. And I implemented it, and it worked — until I read the paper properly and found three places my "RAPTOR" had quietly drifted from the real thing. This is what it actually does, and what those drifts cost.

The idea: a tree of summaries over the untouched source

Most retrieval indexes your chunks and stops. RAPTOR keeps the chunks (the leaves — you still want them for detail and for citing a source), then builds a consolidation layer on top. The build is bottom-up and recursive:

• Chunk the documents into small ~100-token pieces, never splitting a sentence.
• Embed them.
• Cluster the embeddings — reduce dimensions with UMAP, then fit a soft Gaussian Mixture Model (a chunk can belong to more than one cluster), using BIC to choose how many clusters there are.
• Summarise each cluster with an LLM (the paper gets ~72% compression), then embed those summaries and recurse — cluster the summaries, summarise them — until you can't cluster any further.

What you end up with is a tree: raw passages at the bottom, tighter and tighter summaries above. Those scattered paraphrases of one fact get pulled into a single summary node. That's the whole point for a redundant corpus — not a benchmark bump, but a stable place for a diffuse idea to live.

The retrieval trick most people miss

There are two ways to query the tree, and the difference matters. Tree traversal starts at the root and walks down, top-k at each layer. Collapsed tree throws every layer into one big pool — leaves and summaries together — and just retrieves the most similar nodes up to a token budget (~2000 tokens, ~20 nodes).

Collapsed tree wins consistently, and it's the paper's default. The reason is lovely: it lets the granularity of the answer float. A broad question surfaces a high summary node; a specific one surfaces a leaf. You don't decide the altitude up front — the query does. In the paper's own results, 18.5–57% of retrieved nodes come from the summary layers, which is the evidence that the summaries are earning their keep rather than just padding the index. (For the headline-chasers: QuALITY jumps to 82.6% with GPT-4, up from a 62.3% prior best. But the granularity story is the part you'll actually feel.)

Then I compared my code to the paper

Here's the honest half. "I implemented RAPTOR" and "I implemented RAPTOR faithfully" turned out to be two different claims, and the gap between them is where every bug lived. Three of them.

1. The cluster with no size limit

My summariser concatenated all the text in a cluster and sent it to the LLM in one go. Fine for a small cluster. But a populous layer-1 cluster could pull together 60k+ tokens of member text — well past the model's limit — which meant silent truncation and a corrupt summary node sitting quietly in the tree. The paper's clusters are bounded by its recursive small-cluster structure, so its ~72% compression assumes a digestible input. The fix is the obvious one once you see it: map-reduce — pack members into budget-sized batches, summarise each, then summarise the summaries.

2. Single-stage clustering instead of global-then-local

I did one UMAP+GMM pass. The paper does two: a coarse global clustering, then a local clustering within each global cluster. I'd skimmed right past it — but that global stage is exactly the thing that groups paraphrases scattered across different documents. For my redundant corpus, it wasn't a detail; it was the highest-leverage fidelity fix in the whole method. The paper didn't just validate my approach, it named the two things I'd gotten wrong.

3. A reasoning model eating its own budget

The one that actually hurt. After the fixes, a build finished cleanly — and 508 of 647 summary nodes were zero bytes. Eighty percent empty, reported as success.

The cause: I capped summary generation at 1024 tokens, and my summariser was a reasoning model that emits a long <thought> block before its answer. The 1024 tokens were entirely consumed by thinking; the model hit the cap mid-thought, produced no answer, and my code — which strips the thought block — was left with an empty string, dutifully cached as a 0-byte "summary." A raw-API repro confirmed it: 1024 tokens in, pure thought out, nothing after. Raising the cap to 4096 and refusing to cache empty results fixed it (and I made truncation log loudly instead of silently, so the next one can't hide). The deeper lesson was model choice: a small reasoning model (MedGemma-4B) was simply unfit — it truncated, looped, and leaked planning tokens — and I landed on a non-reasoning Gemma variant that just writes the summary.

Why bother telling you the drifts?

Because "we use RAPTOR" is on a hundred architecture diagrams, and the interesting question is never whether you use it — it's whether the tree you built is the one the paper describes. Mine wasn't, in three specific ways, and each one degraded retrieval invisibly: no crash, no error, just quietly worse answers. The method is elegant. The fidelity is the work.

If you're building retrieval over a messy, redundant corpus and want it done faithfully — and debugged down to why a node came back empty — that's the kind of work I do at twentytwotensors. Get in touch.