It's easy to ship a RAG demo that looks great on three hand-picked questions. The whole reason to evaluate is to find out where it actually breaks — so this part is about measuring the CKD assistant honestly, and being straight about what the numbers say.
How I measured it: RAGAS
I used RAGAS, which scores a RAG pipeline without a hand-labelled golden dataset by using a cheap external model as a judge. Four metrics, each answering a plain question:
- Faithfulness — did the answer only say things the retrieved text supports? (the anti-hallucination metric, and the one that matters most for anything clinical)
- Answer relevancy — did it actually address the question?
- Context precision — were the retrieved chunks actually useful?
- Context recall — did retrieval find everything it needed?
The results, unvarnished
Two lenses, because they disagree in a useful way.
The first is a source-match confusion matrix: for each question, did retrieval pull from the right document? Aggregate precision 0.53, recall 0.27, F1 0.36 — not a number I'd put on a marketing page, which is exactly why it's here.
The second lens scores the whole answer with RAGAS, comparing the retrievers on a de-duplicated corpus with the tightened grounding prompt. Higher is better; the average is over the four metrics.
| Retriever | Faithfulness | Answer rel. | Ctx precision | Ctx recall | Average |
|---|---|---|---|---|---|
| Flat | 0.718 | 0.541 | 0.706 | 0.296 | 0.565 |
| Tree | 0.699 | 0.604 | 0.692 | 0.333 | 0.582 |
| RAPTOR | 0.760 | 0.695 | 0.842 | 0.259 | 0.639 |
RAPTOR came out nominally ahead, with flat and tree close behind — close enough that at ten questions I read the gap as noise rather than a verdict. The full breakdown is in the retriever deep-dive. But getting to a table I'd trust meant fixing two things I'd assumed were fine. The store quietly held duplicate copies of several guidelines — a second ingestion nobody had cleaned up — which muddied every comparison until I de-duplicated it. And the tree retriever had been silently falling back to flat for months. Both only surfaced once I measured the components in isolation.
One change clearly helped the generation side: tightening the prompt to answer only from the retrieved text and to refuse when it can't. On the same model and data that lifted faithfulness across every retriever — RAPTOR's went from 0.70 to 0.76 — for the price of the assistant honestly saying "the guidelines don't cover this" a little more often.
Why it's hard, and where the fix is
This is the honest lesson of the whole series. Across every retriever the consistent weak spot is context recall — around 0.3: the single passage that best answers a question often isn't in the retrieved set at all, and if it wasn't retrieved, no bigger model or extra agent recovers it. RAG over dense clinical guidelines is genuinely hard — the text is long, cross-referential and precise — so finding exactly the right passage is most of the battle. The next effort doesn't go into a bigger model or more agents; it goes back to the pipeline — better chunking, better recall.
What I'd take to a real build
Measure before you believe. Treat a mediocre score as the most valuable output you have — it points straight at the thing to fix. And in a high-stakes domain, a system that scores honestly and refuses when unsure beats one that scores impressively on a curated demo and quietly hallucinates in the field.
That's the series — from why three levels through the pipeline, the three levels themselves, and now the honest scorecard. Building grounded, evaluated, privacy-aware RAG — and telling you where it's weak — is what I do at twentytwotensors. Get in touch.