Agentic Evaluation Framework — LLM-as-a-Judge for a Multi-Tier RAG System

The problem this solves

Most agent evaluation still comes down to one number — pass/fail on a test set, or an aggregate LLM-judge score. That’s fine for a single RAG call. It falls apart the moment the system routes — decides which specialist to call, retrieves from multiple sources, or hands off between agents — because a single score can’t tell you which layer is broken. A 0.7 average can hide a perfectly-grounded answer sent to the wrong sub-agent, or a well-routed query answered from the model’s own weights instead of the retrieved context.

I built this framework to evaluate exactly that: a 3-tier agentic RAG system (simple RAG → LangGraph orchestrated agent → multi-agent specialist system) for chronic kidney disease guidance, using the same harness across all three tiers so the numbers are actually comparable.

How the harness is layered

1. LLM-as-a-Judge (RAGAS) — an independent LLM (Gemini 2.0 Flash) scores faithfulness, answer relevancy, context precision, and context recall against the retrieved chunks. This is the layer most teams stop at. I found it isn’t enough on its own: RAGAS silently zero-fills context_recall when a query has no reference answer, which quietly drags down the aggregate and makes healthy queries look like failures. Fixed by excluding unscoreable queries from the average instead of silently counting them as zero — a small change, but it’s the difference between an honest number and a broken one you don’t notice.

2. Custom domain metrics — citation accuracy, stage-appropriateness, disclaimer presence, actionability, and hard safety flags (e.g. NSAID-avoidance, dosing checks) — things a generic LLM judge doesn’t know to check for, scored programmatically rather than by the judge.

3. System-level metrics — intent-routing accuracy and PII-detection accuracy at the single-agent tier; agent-routing precision/recall at the multi-agent tier. This is the layer that actually catches orchestration bugs.

What the layering caught

Running all three together on the ship-snapshot evaluation (17 queries) surfaced a bug that either layer alone would have missed:

Metric	Agentic (single-agent)	Multi-Agent
Intent routing accuracy	0.882	—
PII detection accuracy	1.000	—
Agent routing precision	—	0.598
RAGAS faithfulness (LLM judge)	0.448	—
Disclaimer present	0.765	0.529

The multi-agent tier’s routing precision (0.598) was the real find: the system was over-fanning-out to the RAG agent on 9 of 17 queries — a concrete, traceable cause (a too-permissive 0.3× primary confidence threshold), not a vague “the agent got confused.” The RAGAS/LLM-judge layer wouldn’t have shown this at all — it only sees the content of an answer, not whether the right agent was asked in the first place. You need the system-metrics layer to see routing failures, and the judge layer to see grounding failures. Neither substitutes for the other.

Faithfulness at 0.448 (LLM-judge score) was the headline weakness — the model reasoning beyond retrieved context more than it should. Reported as-is, because the point of building this harness was to stop failures from hiding behind an aggregate.

Why this matters beyond one project

The pattern generalises: any agent that routes, retrieves, or hands off needs evaluation at every layer it operates on, not just the final text output. An LLM-as-a-Judge score alone will tell you an answer sounds right; it won’t tell you the routing was wrong, the retrieval was thin, or a safety check was silently skipped. This harness — and the NaN-handling, threshold-tuning, and metric-layering decisions in it — is reusable for any multi-agent or multi-tier RAG system, not just this one.

Full breadcrumb trail, including the corpus de-duplication and retriever comparison work that fed into these numbers, is in the ongoing MedGemma CKD series.

If you’re shipping an agent and need to know which part of it is actually working, get in touch.