Evaluating a clinical RAG — the honest results

It's easy to ship a RAG demo that looks great on three hand-picked questions. The whole reason to evaluate is to find out where it actually breaks — so this part is about measuring the CKD assistant honestly, and being straight about what the numbers say.

How I measured it: RAGAS

I used RAGAS, which scores a RAG pipeline without a hand-labelled golden dataset by using a cheap external model as a judge. Four metrics, each answering a plain question:

• Faithfulness — did the answer only say things the retrieved text supports? (the anti-hallucination metric, and the one that matters most for anything clinical)
• Answer relevancy — did it actually address the question?
• Context precision — were the retrieved chunks actually useful?
• Context recall — did retrieval find everything it needed?

The results, unvarnished

Two lenses, because they disagree in a useful way.

The first is a source-match confusion matrix: for each question, did retrieval pull from the right document? Aggregate precision 0.53, recall 0.27, F1 0.36 — not a number I'd put on a marketing page, which is exactly why it's here.

The second lens scores the whole answer with RAGAS, comparing the retrievers on a de-duplicated corpus with the tightened grounding prompt. Higher is better; the average is over the four metrics.

Retriever	Faithfulness	Answer rel.	Ctx precision	Ctx recall	Average
Flat	0.718	0.541	0.706	0.296	0.565
Tree	0.699	0.604	0.692	0.333	0.582
RAPTOR	0.760	0.695	0.842	0.259	0.639

RAPTOR came out nominally ahead, with flat and tree close behind — close enough that at ten questions I read the gap as noise rather than a verdict. The full breakdown is in the retriever deep-dive. But getting to a table I'd trust meant fixing two things I'd assumed were fine. The store quietly held duplicate copies of several guidelines — a second ingestion nobody had cleaned up — which muddied every comparison until I de-duplicated it. And the tree retriever had been silently falling back to flat for months. Both only surfaced once I measured the components in isolation.

One change clearly helped the generation side: tightening the prompt to answer only from the retrieved text and to refuse when it can't. On the same model and data that lifted faithfulness across every retriever — RAPTOR's went from 0.70 to 0.76 — for the price of the assistant honestly saying "the guidelines don't cover this" a little more often.

Why it's hard, and where the fix is

This is the honest lesson of the whole series. Across every retriever the consistent weak spot is context recall — around 0.3: the single passage that best answers a question often isn't in the retrieved set at all, and if it wasn't retrieved, no bigger model or extra agent recovers it. RAG over dense clinical guidelines is genuinely hard — the text is long, cross-referential and precise — so finding exactly the right passage is most of the battle. The next effort doesn't go into a bigger model or more agents; it goes back to the pipeline — better chunking, better recall.

Update, September 2026 — the agentic and multi-agent levels got their first real numbers, and a metric bug hid behind a "bad model" score

When this post first shipped, Level 2 (agentic) and Level 3 (multi-agent) had no evaluation numbers attached at all — the honest scorecard above only ever covered Level 1's retrievers. A September session finally ran RAGAS end-to-end on both, and found two real bugs along the way, which is exactly the kind of thing this series keeps finding when it actually measures instead of assumes.

The first run looked damning: Level 3's citation score came back 0.000. Every other quality signal was fine — routing precision was 1.0, recall 0.95 — so a flat zero on citations specifically was a smell, not a verdict. It traced to a reference-corpus mismatch in the eval harness (the evaluation was scoring citations against the wrong grounding text) plus two further citation-formatting bugs a code audit turned up. Fixed and re-run live: citation score 0.000 → 0.718, in line with Level 2's citation score on the same corpus.

The second bug was worse, because it was silent in a way that actively lies. Level 2's generator has a token budget (MODEL_MAX_NEW_TOKENS) that must be set explicitly for a self-hosted model — leave it unset and the default alone exceeds what the local server allows, so every generation call fails. The code's fallback path on that failure was an 80-character canned apology, which then got scored by RAGAS as if it were a real answer — producing plausible-looking 0.000 faithfulness/relevancy numbers that read exactly like "the model is bad" rather than "the model never ran." The signature, for anyone chasing the same bug: every response exactly 80 characters, sub-second elapsed time. Once the budget was set correctly and Level 2 was re-run for real:

Metric	Level 2 (agentic)	Level 3 (multi-agent)
Faithfulness	0.667	n/a — RAGAS not yet wired for the multi-agent path
Answer relevancy	0.681	n/a
Context precision	0.515	n/a
Context recall	0.474 (was 0.192)	n/a
Citation score	0.541	0.718 (was 0.000)

The recall jump is the real result here. Context recall had been stuck around 0.19–0.3 for the whole series — flagged repeatedly as the system's honest ceiling. Once the eval references were re-grounded to match the actual corpus, Level 2's recall more than doubled to 0.474. That's not the retriever suddenly finding more — it's the measurement finally asking a question the corpus could actually answer. Worth remembering next time a number looks like a modelling problem: check what you're comparing it against first.

One case from this re-run is worth naming rather than filing away quietly: a metformin-dosing query where the model gave a correct, appropriately cautious answer and the metric still penalised it — a case of the evaluation harness itself, not the system, being wrong. Flagged as an open limitation rather than "fixed," because it wasn't.

That's the series — from why three levels through the pipeline, the three levels themselves, and now the honest scorecard. Building grounded, evaluated, privacy-aware RAG — and telling you where it's weak — is what I do at twentytwotensors. Get in touch.