The retriever zoo — three bets scored, two that didn't make it

In a RAG system, the retriever is the part that fails quietly. If it hands the model the wrong passage, no amount of clever prompting recovers — the model either says nothing useful or, worse, fills the gap from memory. So after the simple baseline, most of my effort went into how to find the right chunk of guideline text. I benchmarked three retrievers end-to-end — flat, tree and RAPTOR — behind a single factory flag so any level could swap one in, plus two more that never earned a place in the comparison. This is each one, and what the numbers said.

Flat — plain, and hard to beat

The default. Embed the query, cosine-search every chunk in the store, keep the top few above a similarity threshold. Two small touches earn their keep: it expands medical terms so "CKD" also searches "chronic kidney disease," and it drops genuinely irrelevant chunks rather than padding the context to a fixed size. It's the least clever option and, as it turned out, the hardest to beat on precision — when it returns something, it's usually on-topic.

Tree — routing by structure (and the bug that hid for months)

Clinical guidelines have structure — numbered sections, headings, sub-headings — and the tree retriever tries to use it. Two phases: first match the question against a small index of section headings, then pull chunks only from the sections that matched. The bet is that narrowing to the right section before searching beats searching the whole corpus flat.

Here's the honest part. For a long time this retriever was silently broken. Its section-heading index had never been populated, and even after I fixed that, a metadata mismatch meant phase two joined on the wrong fields and found zero chunks on every query. Each time, it quietly fell back to flat retrieval — and because the fallback still returned five results, nothing ever looked wrong. "Returns results" is not the same as "works." It took evaluating the retriever in isolation to notice that "tree" and "flat" were producing identical routing. Fixed the join (document name plus section heading) and rebuilt the index, and it finally does what it says.

RAPTOR — summarise your way up

RAPTOR builds a tree on top of the chunks. Cluster the leaf chunks by meaning, have an LLM write a summary of each cluster, then cluster and summarise those summaries, and repeat — so you end up with layers of increasingly abstract nodes above the raw text. A query can land on a leaf (a specific fact) or on a summary node (a theme that spans several documents). That makes it good at broad, synthesising questions that no single chunk answers. I built the ~2,400-node tree with Gemma-4 E4B writing the summaries on a GPU box; it came out nominally top of the three, by a margin I'll argue below is too small to bank.

Hybrid and contextual — one that failed its gate, one that was never built

Two more were meant to join the comparison and didn't. Hybrid fuses dense semantic search with BM25 keyword search via reciprocal rank fusion, which should help when the answer hinges on an exact term (a drug name, an eGFR threshold). I rebuilt it from a stub into a genuinely working retriever and it failed its benchmark gate — I'd rather report that than quietly drop it. Contextual retrieval — prepending a short LLM-written sentence of context to each chunk before embedding — is designed but not built: there's no collection behind the flag, so the option cannot run at all. The zoo, honestly, is three animals and two enclosures.

The scoreboard

Scored with RAGAS on the de-duplicated corpus, Gemma-4 E4B as the generator, the tightened grounding prompt, ten CKD questions. Higher is better; the average is over the four metrics.

Retriever	Faithfulness	Answer rel.	Ctx precision	Ctx recall	Average
Flat	0.718	0.541	0.706	0.296	0.565
Tree	0.699	0.604	0.692	0.333	0.582
RAPTOR	0.760	0.695	0.842	0.259	0.639

RAPTOR leads on faithfulness and context precision — but at n=10 a gap of this size in the average is noise, and I'm not going to call it a win. The more useful result is the split, from a separate retrieval-only benchmark: RAPTOR wins natural-language questions, flat wins keyword and factual ones. Both score 0/4 on adversarial out-of-corpus queries — neither knows how to abstain. Flat and tree each have a personality too: flat holds strong precision, tree edges the others on recall (routing does surface a bit more of the right material). Two honest caveats. Answer relevancy is jumpy — a known RAGAS quirk where safe, hedged clinical phrasing gets read as evasive — so I lean on the other three. And context recall sits around 0.3 for everyone: the single passage that best answers the question often isn't retrieved at all. That's the ceiling on the whole system, and no choice of retriever fixes it.

Where the real gain is left — recall — sends you straight back to the pipeline: chunking, and what actually makes it into the store. The full, unvarnished scorecard is in the evaluation. Building and honestly measuring retrieval over dense domain text is what I do at twentytwotensors. Get in touch.