twentytwotensors — Build in Public

Why my AI agent was billing my Claude subscription as 'extra usage'

Sun, 16 Aug 2026 00:00:00 GMT

I run a terminal AI agent called Hermes Agent for most of my coding and automation work. It's configured to use my Claude subscription via the same OAuth credential as the claude CLI. One day it started failing with a specific error while the CLI itself worked perfectly:

HTTP 400: You're out of extra usage. Add more at claude.ai/settings/usage and keep going.

That error is Anthropic's way of saying your purchased extra-usage credits are empty. My first thought was the obvious one: I must be out of quota. But the claude CLI still worked — same subscription, same week. And when I checked my usage page, my Claude Code weekly allowance looked untouched. What had actually happened was worse and more interesting: every request from my agent had been billed to extra-usage credits instead of the subscription quota my plan already includes. I'd burned through paid top-ups without touching the allowance I was already paying for. This is the story of how I proved that, and the one-line root cause that explained all of it.

Same token, same account — so why different billing?

Both the CLI and the agent authenticate the same way: with the Claude Code OAuth token stored in my Mac's keychain. I verified this byte-for-byte by pointing the CLI at a local logging proxy and comparing the token it sent against the one the agent uses. Identical. Same account, same token, same model, same API endpoint.

Yet the CLI's requests drew from the subscription, and the agent's drew from extra usage. The only remaining difference was the request itself — the headers, the URL, and the payload. Anthropic's OAuth billing classifier doesn't just look at who you are. It looks at what client you're presenting as, and routes the request to a billing lane based on that fingerprint.

The fingerprint that was wrong

I captured the real CLI's request through a local proxy and diffed it against what my agent was sending. Four things stood out:

The user-agent. The CLI sends claude-cli/2.1.233 (external, sdk-cli). The agent sent claude-code/2.1.74 (external, cli) — a stale product prefix from an old naming scheme, an outdated version, and a different entrypoint label.
Missing query param. The CLI hits /v1/messages?beta=true. The agent hit /v1/messages without it.
A shorter beta list. The CLI sends ten anthropic-beta headers (thinking, context management, effort, etc.). The agent sent two.
Two missing markers entirely. The CLI's payload starts its system prompt with a billing-routing header (x-anthropic-billing-header: cc_version=…; cc_entrypoint=sdk-cli;) and includes a metadata.user_id carrying the account UUID. The agent sent neither.

Any one of those might be enough to look like a third-party client to the classifier. Together they guaranteed it. The agent was impersonating a slightly-off version of the official client — close enough to authenticate, not close enough to bill like one.

Why the version was stale — a debugging side-quest

There was a second, smaller bug hiding behind the first. The agent tries to detect the installed Claude Code version so it can put a current version in its user-agent. On my machine that detection silently failed, and it fell back to a hardcoded 2.1.74 — from early 2026 — while my actual CLI was 2.1.233.

The cause was a broken duplicate install: /usr/local/bin/claude is a global npm copy that crashes under the current Node version, and it shadows the working binary at ~/.local/bin/claude that my shell actually uses. The version probe hit the broken one, got empty output, and fell back to the stale constant. The fix was updating that constant — and the deeper lesson was already on my to-do list: uninstall the duplicate.

The fix

The agent is open source and installed from source on my machine, so I patched its Anthropic adapter to mirror the CLI's fingerprint exactly:

User-agent → claude-cli/<version> (external, sdk-cli)
The full ten-beta header list
?beta=true on the messages URL
A session-id header
The x-anthropic-billing-header block at the start of the system prompt
metadata.user_id with the account UUID read from ~/.claude.json

Then I ran the test suite, and then the real test:

$ hermes chat -q "Reply with exactly: OK"
→ OK

Log line from that run: API call #1: model=claude-sonnet-5 provider=anthropic latency=2.0s, finished normally, no billing error. The same token that had been draining extra usage all day was now drawing from the subscription. I've been using it since, and the extra-usage balance has stopped moving.

What I'd tell anyone running a tool on a Claude subscription

If you use Claude Code's OAuth credential from anything other than the official CLI — another agent, an IDE extension, a script — and you see "out of extra usage" while the CLI still works, this is the likely cause. A few things that helped me:

Check the billing lane, not just the balance. If the CLI works and the usage page says your weekly allowance is untouched, you're being billed somewhere else.
Capture and compare. Point the working client at a local logging proxy, capture its request, and diff it against the failing client's. The difference is the bug.
Same token isn't the same request. Anthropic's OAuth routing looks at the full client fingerprint — user-agent, beta headers, query params, and payload markers.

I've written up the full diagnosis and the patch so the next person (or future me) doesn't have to re-derive it — the fix is specific to the version of the agent I'm running, but the method applies to any tool using Claude Code's OAuth.

Debugging my own tooling so my infrastructure bills to the plan I already pay for — that's the kind of thing I enjoy getting to the bottom of. Get in touch if you've got a system doing something similar.

How to set up Hermes Agent with your Claude subscription

Sun, 16 Aug 2026 00:00:00 GMT

Hermes Agent is an open-source AI agent that runs in your terminal — same family as Claude Code, but provider-agnostic: it can use OpenRouter, OpenAI, Google, local models, and more. What most people don't realise is that it can also run on a Claude Pro/Max subscription you already pay for, using the same OAuth login as Claude Code — no Anthropic API key, no per-token API billing. Here's exactly how I set it up, and the one gotcha that cost me a day.

What you need before starting

A Claude subscription (Pro or Max) with Claude Code available on it
Claude Code CLI installed and logged in at least once (claude login), on macOS or Linux
An empty afternoon — setup itself is ten minutes, but the gotcha at the end is worth knowing about

The reason Claude Code needs to have been logged in is that Hermes doesn't do its own Anthropic OAuth. It reads the credential Claude Code already stored — that's the whole trick, and also the source of the one failure mode I'll get to.

Step 1 — Install Hermes

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

That installs the launcher, a Python environment, and the agent itself. Then confirm it runs:

hermes doctor

Step 2 — Point it at the Claude subscription

Run the setup wizard and pick the Anthropic provider:

hermes setup

Or set the pieces directly — the default profile config lives in ~/.hermes/config.yaml:

model:
  default: claude-sonnet-5
  provider: anthropic

The key part: do not set an ANTHROPIC_API_KEY. If you do, Hermes will use API billing. Leave it out and Hermes automatically discovers the Claude Code OAuth credential instead — on macOS from the Keychain entry "Claude Code-credentials" and/or ~/.claude/.credentials.json.

Step 3 — Verify what it's actually using

hermes auth list

You want to see the anthropic provider holding an oauth credential sourced from claude_code — that's the subscription route:

anthropic (1 credentials):
  #1  claude_code          oauth   claude_code ←

Then a live smoke test:

hermes chat -q "Reply with exactly: OK"

If you get OK back, you're running on your subscription.

Step 4 — The gotcha: 'out of extra usage' while the CLI works

This is the one that bit me. Hermes authenticates fine and answers queries — but you notice your purchased extra-usage credits draining, while your subscription's Claude Code weekly allowance sits untouched. And when the extra usage runs out, Hermes fails with:

HTTP 400: You're out of extra usage. Add more at claude.ai/settings/usage

…while claude -p "hi" works perfectly.

Same token, same account — different billing lane. Anthropic's OAuth billing classifier routes requests by the client fingerprint (user-agent, beta headers, query params, payload markers), not just by who you are. If the tool presents a slightly-off version of the official Claude Code fingerprint, Anthropic bills it as a third-party app drawing from extra usage instead of the subscription quota.

I wrote up the full diagnosis in Why my AI agent was billing my Claude subscription as 'extra usage' — including the exact request diff and the patch. If you hit this error, that's the post to read.

Keeping it healthy

Credentials go stale — if Hermes starts failing auth, run claude -p hi once. That re-syncs the OAuth credential Hermes auto-discovers.
Check the lane, not just the balance — if you see extra usage moving while the CLI works, your tool is being billed as a third party. Fix the fingerprint, don't just top up.
Shell hygiene — if you use helper functions that export ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN (e.g. to point Claude Code at another provider), those exports persist in the shell after the function exits. Launch Hermes from a fresh shell or you'll silently route it somewhere else.

A cheap fallback worth having

Hermes supports profiles, so I keep a second one on DeepSeek — a separate API-billed provider that's cheap enough to treat as a spare. If the Claude lane is ever down, I switch with a single flag rather than being blocked:

alias hermes-ds="hermes -p deepseek"

Subscription for the main work, a cheap API as the spare. That combination has been rock solid since I fixed the fingerprint issue.

Getting the most out of the tooling you already pay for — and debugging it when it misbehaves — is what I do. Get in touch if you're wrestling with something similar.

The score is about connections, not cells

Mon, 20 Jul 2026 00:00:00 GMT

Here is a way to fail at cell tracking.

You find every single cell. Every nucleus, in every frame, perfectly. And you still score close to zero.

Because nobody asked you where the cells are. They asked which cell became which. And the answer to that is not made of cells. It is made of the connections between them.

That sentence is the whole of this series. Everything after it is a consequence — including both of the things I got wrong.

A real recording: a zebrafish embryo over a hundred timepoints, tens of thousands of nuclei.

The competition

The CZ Biohub cell tracking challenge gives you light-sheet microscopy recordings of a developing zebrafish embryo and asks for the lineage: which nucleus in this frame is the same nucleus as that one in the next, and where a cell divided into two. It runs as a Kaggle code competition — your notebook is re-run on roughly 199 hidden volumes, inside twelve hours, with no internet access. Whatever you build has to be fast enough to survive that.

This series is a working log of getting to 0.881 on the public leaderboard over about three weeks, what actually moved the number, and the two experiments that moved it the wrong way. I'll be specific throughout about which parts were mine and which were not, because that distinction turns out to matter for the last post.

What the metric actually counts

The score is adjusted_edge_jaccard + 0.1 · division_jaccard. Strip the names away and it means: mostly, you are graded on edges — the link between a cell in frame t and the same cell in frame t+1 — with a small bonus for correctly spotting divisions.

Two details in that definition end up driving every design decision I made:

Nodes are matched by proximity, not identity. A predicted nucleus counts as the ground-truth one if its centroid lands within about 7 µm. Inside that radius, closer is not better. This sounds like a technicality. It is the reason one of my later "improvements" scored exactly zero.

Only consecutive frames count. An edge from t to t+1 scores. An edge from t to t+2 — jumping over a frame where you missed the cell — scores nothing at all. You cannot skip a hole in your own detections. You have to fill it.

What the metric counts: edges between consecutive frames, with nodes matched by proximity.

You are graded on a fraction of what you can see

This is a real recording: a zebrafish embryo, a hundred timepoints, tens of thousands of nuclei. But the ground truth is sparse. Only about 6% of the cells are labelled — roughly two thousand, out of the thirty thousand you can plainly see.

The labelled cells lighting up against everything else visible in the same volume.

In a single frame that ratio is startling: eighteen labelled cells among more than a hundred obviously-visible ones. You are graded on a fraction of what is in front of you, and you have no way of knowing in advance which fraction.

The practical consequence is that you cannot naively score yourself. If you compute a normal tracking metric against sparse labels, every correct detection of an unlabelled cell looks like a false positive, and your local number collapses in a way the leaderboard does not. Building a scorer that understood sparsity was the first real work of the project, and it's the next post.

Depth is not like width

One more property of the data, because it shapes where the errors live. Look at the volume face-on and the nuclei are round. Look at the same volume from the side and they go blocky.

The microscope samples depth about four times more coarsely than width — the same 104 µm of tissue, a quarter of the detail. This is anisotropy, and it is not a rendering artefact; it is the actual resolution of the data. Every distance you compute, every neighbourhood you search, has to be expressed in physical microns rather than voxels, or it silently means something different along z than it does along x.

Depth is where the detector is least sure. It is worth knowing that before you start trusting it.

The same volume face-on and from the side: the microscope samples depth four times more coarsely.

Why the combinatorics are brutal

So: connections. How hard can that be?

Each frame holds around three hundred cells, and any one of them could connect to any of three hundred in the next. Across the whole recording, that is roughly nine million possible links. About thirty thousand are real.

One in three hundred.

Candidate links exploding between two frames: about nine million possible, roughly thirty thousand real.

And you cannot tell them apart by looking. These are featureless blobs — no texture, no colour, no shape to distinguish one from another. They all look identical. They drift between frames. They divide, so one becomes two. And every so often the detector blinks and a cell simply is not there for a frame.

That last failure mode sounds minor. It is the single most expensive thing that happens in the whole pipeline, and it gets its own post.

What's in the rest of this series

Six parts, each landing one idea:

Part	The idea
1 · this one	The score is about connections, not cells
2	Build a scorer you trust before you build a model
3	Finding cells is the easy half — and the biggest win wasn't mine
4	Scoring links one at a time is a trap
5	One missed cell costs two links and one lineage
6	Two failures, two different axes — and where the ceiling is

A note on honesty before we start. The detector and the linker at the heart of this pipeline are public work, not mine — I'll say so plainly when we get there. My own contribution is a graph post-processing stage worth +0.014, and a local scorer that predicted the leaderboard to within 0.008. Public notebooks in this competition reach 0.902, so 0.881 sits at the public floor rather than ahead of it.

That framing is not modesty. It is the reason the final post is worth reading: when you know exactly which part you built, you know exactly which part to distrust.

Build a scorer you trust before you build a model

Sun, 19 Jul 2026 00:00:00 GMT

A Kaggle code competition gives you five submissions a day, and each one costs a full re-run over roughly 199 hidden volumes. That is not an iteration loop. It is a rationing system.

So the first thing I built was not a detector or a tracker. It was a copy of the scoreboard.

Why you can't just use an off-the-shelf metric

The obvious move is to grab an existing tracking metric and point it at your predictions. It does not work here, and the reason is the sparse ground truth from Part 1.

Only about 6% of the visible nuclei are labelled. A standard metric assumes the annotation is complete, so every cell you correctly detect that nobody labelled is counted against you as a false positive. Run that on this data and a genuinely good tracker scores terribly — not because it is wrong, but because the metric is measuring a question nobody asked.

Worse, the error is not a constant offset you could ignore. It scales with how many unlabelled cells you happen to find, which means it is largest exactly when your detector improves. A metric that punishes you for getting better is not a slow guide. It is an actively misleading one.

What the local scorer had to get right

Three things, all of them consequences of how the real metric is defined:

Score only against labelled cells. Match predicted nodes to ground-truth nodes by centroid within the physical radius, then evaluate edges only where both endpoints matched something real. Predictions in unlabelled regions are neither rewarded nor punished — they are simply out of scope.

Match in microns, not voxels. Because of the anisotropy, a radius expressed in voxels is a squashed ellipsoid in real space — far more permissive in z than in x. Every distance in the scorer is physical.

Count consecutive edges only. An edge that spans a gap is worth nothing. If the scorer credits it, the scorer will happily tell you that a bad idea is a good one.

Did it work?

The test of a local scorer is not whether it looks sensible. It is whether it predicts. So the first two real submissions were calibration, not attempts to win.

Submission	Local score	Leaderboard	Gap
Classical detector + greedy linking	0.066	0.147	—
Blob detector + greedy linking	0.7535	0.746	0.008

That second row is the one that mattered. A local number of 0.7535 landed at 0.746 on the leaderboard — within 0.008. From that point on I could try an idea, score it in minutes on my own machine, and know roughly what the board would say without spending a submission to find out.

Nearly everything in the rest of this series was found that way. The whole project runs on that one number being trustworthy.

The part I got wrong

Here is the honest caveat, and it is the seed of the final post.

"Within 0.008" tells you the scorer is well calibrated in absolute terms. It does not tell you the scorer can resolve a difference of 0.002 between two candidate configurations. Those are completely different claims, and I conflated them for about a week.

A scorer can be accurate on average and still be too noisy to rank two near-identical options — or biased in a specific direction that happens to reward a change the real metric does not care about. Mine turned out to be both, in different experiments. Building the scorer was the right first move. Believing it was precise because it was accurate was not.

The discipline I'd state now, having learned it the expensive way, is not "measure locally." It is: know your measurement's noise floor, and refuse to ship inside it.

The infrastructure underneath

Briefly, because it is the unglamorous half. Offline iteration only helps if running an experiment is cheap, so the scorer sits on top of a small amount of throwaway platform work: Terraform-provisioned GPU boxes on AWS (T4 and L4 spot lanes, with a CPU lane for the linker), S3 and EBS for persistence so a terminated spot instance costs nothing but time, an isolated MLflow instance for run tracking, and a self-terminating job pattern so a forgotten box does not quietly bill overnight.

There is also a wrinkle specific to code competitions: the notebook runs with no internet, so every dependency has to be packaged as offline wheels and installed from a Kaggle dataset. And the GPU you get is a lottery — a P100 has a different compute capability than a T4, which means a build that works in testing can fail on submission. That one costs you a day if you find it late.

None of this is interesting on its own. It exists so that the loop in the middle — change something, score it, know within minutes — actually spins.

With a scoreboard I trusted, the question became where the score actually comes from. The answer was lopsided enough to be worth its own post: detection is the lever, and the biggest single improvement in the project was not my work.

Finding cells is the easy half — and the biggest win wasn't mine

Sat, 18 Jul 2026 00:00:00 GMT

Detection is the easy half of tracking. You are only asked where the nuclei are in a single frame — no identity, no history, no divisions. And yet detection turned out to be where almost all of the score lives.

This post is the part of the project I am least able to take credit for, so let me put the conclusion at the top: the largest single improvement I made was deleting my detector and using someone else's.

The runtime budget rules everything

Before any accuracy discussion, the constraint that eliminated most options. The notebook re-runs over roughly 199 hidden volumes inside twelve hours. That leaves, very roughly, 3.6 minutes per volume for the entire pipeline — detection, linking, post-processing, writing output.

This single number decided more of the architecture than any accuracy result did.

What I tried first: a hand-written blob detector

Nuclei in this data are, visually, blobs. No texture, no distinguishing features — just roughly spherical bright regions. That is close to the textbook case for a multiscale Laplacian-of-Gaussian detector, which is old, boring, and has no learned parameters at all.

I implemented it as 3D convolutions in PyTorch so it ran on GPU, with the filter scales chosen in physical microns and non-maximum suppression done in an anisotropic neighbourhood — the same micron-not-voxel discipline the scorer needed.

It scored 0.7535 locally, 0.746 on the board, in well under the budget.

What happened when I tried the obvious deep models

The expectation is that a pretrained segmentation model beats a hand-tuned classical filter. On this data, the two I tried did not.

Detector	Local edge-Jaccard	Runtime / volume	Verdict
Classical threshold + connected components	0.066	fast	Baseline only
Multiscale LoG blob (mine)	0.7535	fast	Shipped
Cellpose-3D	0.7245	~85 min	Rejected — ~23× over budget
StarDist-3D (pretrained, out of the box)	0.3978	—	Rejected — worse than the blob

Cellpose was slightly less accurate than the blob detector and roughly twenty-three times over the entire per-volume budget on its own. StarDist out of the box scored about half.

It would be easy to read that as "deep learning is overrated here." That is the wrong lesson, and the next section is why. The right lesson is narrower: a pretrained model carries its training domain with it. These are anisotropic light-sheet volumes of densely packed embryonic nuclei. That is not what those checkpoints saw. The architecture was never the problem; the domain gap was.

What a U-Net does: encode down, decode back up, with skip connections carrying detail across.

The thing that actually worked, which I did not build

Partway through the competition, a support pack was published under CC0 containing a trained temporal 3D U-Net — a 3D U-Net with per-voxel attention across several consecutive frames, plus a shared backbone feeding both a detection head and an edge-scoring transformer.

Reading several consecutive frames at once: a nucleus ambiguous in one frame is often obvious with its neighbours.

That temporal window is the whole idea, and it is a good one. A nucleus that is ambiguous in one frame — dim, or overlapping a neighbour — is usually obvious once you can see the frames either side of it. The model gets to use the same context a human would.

Swapping it in:

Detector	Local	Leaderboard	Runtime
My LoG blob + greedy linking	0.7535	0.746	fast
Public temporal U-Net + ILP linking	0.9423	0.867	~1.7 min

+0.113 on the leaderboard. For comparison, every idea of my own across the entire project added up to +0.014. The single biggest lever was adopting public work, and it was not close.

I want to state that plainly rather than bury it, for two reasons. The first is simple credit. The second is that it sets up the honest reading of my final score: reaching 0.867 was reproduction. It got me to the pack, not ahead of it. Public notebooks in this competition reach 0.902.

Detection output: nuclei found per frame, before any linking.

What I learned by trying to improve it, and failing

Having adopted the detector, the obvious next move was to train a better one. Several attempts, all negative, and the negatives are more useful than another table of wins:

A longer temporal window did not help. If two frames of context are good, four should be better. My first test said yes — but that test changed three variables at once. Re-running it as a proper matched comparison flipped the result: window-2 scored 0.9623 against window-4's 0.9599, and window-4 lost on four of five volumes. The pilot was confounded; the control was the experiment.

A longer-trained checkpoint did not help. A 350-epoch checkpoint scored 0.9439 against the 50-epoch model's 0.9423 — a difference well inside noise. The detector had converged long before I got to it.

Architecture swaps are a dead end here. Between StarDist's 0.3978 and Cellpose's runtime, the evidence says the remaining headroom is not in choosing a different architecture. It is in in-domain training — and doing that properly means a from-scratch run at full scale, which is where the last post ends up.

The shape of the result

Detection contributed +0.113. Everything else in the pipeline — linking strategy, graph repair, parameter tuning — contributed less than +0.02 combined.

That lopsidedness is worth internalising before optimising anything. It is very easy to spend a week on the clever part of a pipeline when the boring part in front of it holds an order of magnitude more score. I did spend that week, and the next two posts are what I found there. They are genuinely interesting — but they are the edge, not the lever.

Next: why scoring links one at a time poisons everything downstream.

Greedy is a trap: linking cells with an integer program

Fri, 17 Jul 2026 00:00:00 GMT

You have detections in every frame. Now you need the connections — and from Part 1, the connections are the entire score.

Each candidate link gets a score: how close are these two nuclei in physical space, how well does the motion fit, what does the model's edge head think. Then the obvious thing is to sort by score and take the best ones first.

The obvious thing is a trap.

Why taking the best link first loses

Greedy assignment makes each choice in isolation, and every choice removes options from the choices after it. One confident wrong link early on poisons everything downstream of it.

The clearest version is small enough to hold in your head. Suppose the greedy step takes a link scoring 0.95 — genuinely the highest-scoring single edge available. But taking it consumes a cell that two other links needed, and both of those are now forced onto worse alternatives. Total: 1.45. Solve the same set jointly, decline that 0.95, and the total comes to 2.55.

The greedy answer is not just worse. It is worse because it was locally optimal.

Greedy takes the 0.95 link, blocks cell B, and forces two bad links — 1.45 total. Solved jointly, the same candidates give 2.55.

The global version

The fix is to stop choosing one at a time and describe the whole frame-to-frame assignment as a single optimisation. Every candidate link becomes a binary variable — take it or don't. The objective is the total score of everything taken. The constraints encode what a lineage is allowed to look like:

a cell has at most one parent in the previous frame;
a cell has at most one child in the next — unless it divided, in which case exactly two;
appearances and disappearances are allowed, but they cost something, so the solver doesn't use them to wriggle out of hard cases.

That is an integer linear program. The "integer" part is not decoration: you cannot take 0.6 of a link. A cell either connects to that other cell or it does not, and it is precisely that integrality which makes the problem combinatorial rather than a matter of solving a linear system.

Variables, objective, constraints — and why it must be integer: you cannot take 0.6 of a link.

Given all candidates at once, the solver returns the highest-scoring set of links that is simultaneously legal — no cell with two parents, no accidental three-way division. Not locally greedy. Globally consistent.

Linking across frames: every cell gets one parent, unless it divided, in which case two children.

The first implementation was correct and unusable

My first working version used ultrack with an open-source solver. It produced good tracks and found 108 divisions, which was more than anything I'd managed before.

It also took 6.79 minutes per volume, against a budget of roughly 3.6. That is 2.3× over — a correct answer that would fail the submission re-run.

The obvious escape was parallelism, and it did not work either: running multiple workers deadlocked. I spent a while there, found and fixed two real bugs in my own usage along the way, and then parked the whole approach. A correct pipeline that cannot finish is not a pipeline.

The version that shipped uses the public pipeline's ILP formulation via tracksdata with a SCIP backend, which solves the same problem inside the budget. This is the second time in the series that the practical answer was "use the public thing" — and, as in Part 3, I'd rather say so than imply the formulation was mine.

The part that transferred, and the part that didn't

Something worth noticing about the two linking approaches. Greedy linking on my blob detector scored 0.746. ILP linking on the public U-Net scored 0.867. It is tempting to attribute that gap to the linker.

Most of it is the detector. The two changed together, and I never ran the clean factorial — blob-plus-ILP and U-Net-plus-greedy — that would separate them. I am fairly confident the linker is worth real points, because the failure mode it fixes is visible in the output. But "fairly confident" is not "measured," and after what happens in Part 6 I am wary of asserting effect sizes I did not isolate.

Which is a small example of a bigger habit worth having: when you change two things and the number moves, you have learned that the pair helps. You have not learned which one.

What the ILP still can't fix

The solver can only choose among the links you gave it. If the detector missed a nucleus entirely in frame t, there is no candidate edge into it and no candidate edge out of it — nothing for the optimisation to select, however global it is.

That gap is not a small local blemish. It is the single most expensive event in the pipeline, and repairing it was the largest improvement I made myself.

Next: one missed cell, two lost links, one broken lineage.

One missed cell, two lost links, one broken lineage

Thu, 16 Jul 2026 00:00:00 GMT

The tracks coming out of the linker are good. They are not clean. And this is where the idea from Part 1 — that the score is about connections, not cells — finally earns its keep.

Flattening space onto a plane and giving the vertical axis to time: a cell's life becomes a curve, a division a fork.

The asymmetry that makes gaps expensive

Watch what happens when the detector blinks: one cell, missed, for one frame.

You do not lose one connection. You lose two — the edge coming in, and the edge going out. And because the track is now severed, it snaps into two fragments, so the family tree is wrong from that point onward.

One missed cell. Two lost links. One broken lineage.

A track intact, with its two incident links lit. The detector blinks for one frame — and both edges die at once.

That asymmetry is the whole insight. A missed detection costs double, because edges are what the metric counts and every interior node carries two of them. Which flips the obvious priority: the most valuable repair is not finding more cells. It is healing the ones that broke.

You cannot leap the hole

There is a catch, and it comes straight from the metric definition. The score only counts links between consecutive frames.

So the tempting shortcut — notice the gap, draw an edge from t straight to t+2, move on — earns exactly nothing. That edge spans two frames and the metric does not see it.

You have to fill the hole instead: invent a node at t+1 where the detector missed one, interpolate its position, and thread two consecutive edges back through it. Do that and you recover both of the lost links plus the continuity of the lineage.

This is a case where reading the metric definition closely was worth more than any modelling idea. The difference between the version that scores and the version that scores nothing is entirely in a detail of how edges are counted.

Filling the hole: a synthetic node at t+1 and two consecutive edges threaded back through it.

The calibration stage

All of this lives in one module that takes (nodes, edges) from the linker and returns a repaired (nodes, edges). It touches no imagery and runs in seconds. Four transforms:

Filter short tracks. Fragments a few frames long are usually noise, and their edges are usually wrong. Dropping them is a small clean win. The public pipeline used a minimum length of 7; my own sweep found 4 was better on this data, and 7 measurably hurt. A tuned constant, but a tuned constant with a measurement behind it.

Add safe divisions. The division term is worth 10% of the score, and most submissions score close to zero on it. Where a track ends near the start of two others, that is a plausible missed division — so insert a conservative second child, gated by per-frame and global caps so it cannot run away.

Close gaps. The repair above: synthesise the missing node and two consecutive edges. This was the single largest lever in the stage.

Line-fit smoothing. Least-squares smoothing of positions along each track. Hold that thought — it is the subject of Part 6, and not for a good reason.

Filtering short fragments before the other repairs run.

The order matters more than any single transform

This surprised me and is probably the most transferable thing in the post.

These transforms are not commutative. Run them as filter → divisions → gap and the local score is 0.7691. Run the same three as gap → filter → divisions and it is 0.7636 — and, worse, the division count collapses to zero.

The reason is that gap-closing creates short synthetic fragments, which the filter then eats, which removes exactly the track endings that the division detector was going to pair up. Each transform is individually correct. Composed in the wrong order, one of them silently destroys another's input.

No single transform is buggy. The pipeline is. That failure only shows up if you measure the composition rather than the parts.

What it was worth

Configuration	Local	Leaderboard
Public U-Net + ILP, no calibration	0.9423	0.867
+ graph calibration	0.9571	0.881

+0.014 on the leaderboard, and it transferred from local almost exactly — 0.0148 local against 0.014 on the board. For a stage that never touches a pixel and runs in seconds on top of someone else's detector, that is the piece of this project I am happiest with.

It is also, honestly, an order of magnitude smaller than the +0.113 that came from swapping the detector in Part 3. The detector was the lever. This was the edge.

On a real embryo: every line a cell that was actually labelled, tracked through a hundred frames.

A caveat I want on the record

Almost all of this was tuned on one volume. Later multi-volume validation across five volumes — spanning roughly 6,900 to 73,000 cells — held up, with the shipped configuration gaining about +0.009 mean over baseline on the three usable ones. But those volumes come from a single embryo.

Cross-embryo generalisation is untested. Given that the hidden test set is ~199 volumes, that is the largest methodological risk in the whole project, and I would rather name it than let a leaderboard number imply more robustness than I actually demonstrated.

Next, and the reason this series exists: two experiments that made things worse, in two completely different ways.

Two failures, two different axes

Wed, 15 Jul 2026 00:00:00 GMT

By this point the pipeline was working. A public temporal U-Net for detection, a global ILP for linking, my own graph calibration on top: 0.881.

And then everything stopped working.

Failure one: the improvement worth exactly zero

Line-fit smoothing was the obvious next step. Fit a line through each track's positions and snap the points onto it, so cells travel along clean trajectories instead of jittering. Visually it is a clear improvement — the tracks stop wobbling.

Fit a line, snap the points: the track gets visibly cleaner while every link stays identical.

My local scorer agreed: +0.0064. Small, but well outside anything I'd have called noise.

The leaderboard moved by 0.000. Not a little. Not within error. The score before was 0.881 and the score after was 0.881.

The two failures plotted against the noise band — deliberately on opposite sides of it, because they are not the same failure.

In hindsight the reason is sitting in the metric definition from Part 1. A predicted node matches a ground-truth node if its centroid falls within about 7 µm. Inside that radius, closer is not better. Matching is a threshold, not a gradient.

My smoothing moved points by fractions of a micron. Every point it moved was already inside the radius and stayed inside it. Not one match flipped, so not one edge changed, so the score could not move. I had spent a day polishing positions — and the score is about connections. I was polishing cells.

One honesty note, because I want the claim to be exactly as strong as the evidence. The measured facts are +0.0064 local and +0.000 on the leaderboard. The explanation — that sub-threshold polish is invisible to a radius-matched metric — is a well-supported reading of the metric definition, not something I isolated in a separate experiment. It fits, and I believe it. It is reasoning, not a measurement.

Failure two: the improvement too small to see

The second one is quieter and, I think, more instructive.

I swept the detection confidence threshold across its full range, looking for a better operating point. The local curve was flat — every value within about ±0.001 of every other. A reasonable reading is "this parameter doesn't matter."

Instead I shipped the value that looked best, 0.98, at +0.0005 local.

It lost 0.002 on the board, taking 0.881 down to 0.879. I reverted it.

Nothing here was a modelling mistake. The mistake was treating the highest point on a flat curve as a signal. A +0.0005 difference was smaller than my scorer's ability to resolve, so "best" was indistinguishable from "arbitrary" — and I picked an arbitrary value and paid for it.

Why these are not the same failure

It is tempting to file both under "the local score didn't transfer." That reading is wrong, and it throws away the useful part.

	Local	Leaderboard	What it says about the scorer
Line-fit smoothing	+0.0064	+0.000	Biased — measured a real change the true metric ignores
Detection threshold 0.98	+0.0005	−0.002	Noisy — the difference was below its resolution

The smoothing result was big enough to measure — +0.0064 sits well outside a ±0.002 band. It measured the wrong thing. That is bias: my scorer was rewarding position accuracy that the real metric, with its match radius, is structurally incapable of caring about.

The threshold result was too small to measure at all. That is noise: nothing wrong with what the scorer values, just no ability to resolve a difference that fine.

Different diagnoses, different fixes. Bias means your metric disagrees with the real one about what counts, and no amount of extra data helps — you have to fix the metric. Noise means your metric agrees but cannot see finely enough, and more volumes or more repeats would help.

A number you compute for yourself can be wrong in both of those ways. Mine was wrong in each, exactly once.

The discipline I'd actually state

Part 2 ended on a local scorer that predicted the leaderboard to within 0.008, which felt like the project's foundation. It was — but I drew the wrong conclusion from it.

Being accurate to 0.008 in absolute terms says nothing about being able to rank two options that differ by 0.002. I treated those as the same property for about a week.

So the rule is not "measure locally." It is:

Know your measurement's noise floor, and refuse to ship inside it.

Had I plotted my own scorer's variance before trusting its third decimal place, both of these would have been caught before they cost submissions. The threshold sweep would have been visibly flat rather than deceptively peaked, and the smoothing gain would have been visibly the wrong kind of gain.

Other things that did not work

For completeness, since negative results are most of the actual work:

Motion-gated gap recovery. Only close a gap when the motion is consistent — clearly smarter than closing every gap. Measured worse at every setting, 0.9571 down to 0.9544. It stayed in the codebase, defaulted off. Testing it prevented a regression, which is a real return on a negative result.
The division term is effectively dead. Worth 10% of the score, and division Jaccard came in at ≤0.009 on the one volume with enough labelled divisions to measure, and zero elsewhere. I put real effort here for no return. Judging by public scores, so did everyone else — which makes it one of the two places genuine headroom might still be hiding.
Longer temporal windows and longer training — both flat, both covered in Part 3.

The ladder: naive blob, then the learned detector, then healing the graph.

Where the ceiling is

0.881. I am fairly confident that is the ceiling for this approach — a public detector plus my own graph calibration — because every cheap lever I could find is exhausted and the remaining ideas all measured flat or negative.

Beating it means training a detector from scratch, in-domain, at full scale. Days of compute, on a hunch, with no positive signal that it would help. That is a pure appetite bet, and I have not taken it.

Two caveats keep the number honest. Public notebooks in this competition reach 0.902, so 0.881 sits at the public floor, not ahead of it. And nearly everything was tuned on a single volume from a single embryo, so cross-embryo generalisation is untested — the risk I flagged in Part 5 and have not retired.

The competition is still open, so there is no ranking to report and I am not claiming one.

What I will claim is the smaller, more portable thing. I built a measurement, trusted it further than it deserved, and got caught twice — once by bias and once by noise. Finding the edge of your own instrument is not a detour from the work. On this project it was the work.

0.881 — the ceiling for a public detector plus my own graph calibration.

Knowing where the ceiling is, is also a result.

What it actually costs to self-host a coding model

Fri, 10 Jul 2026 00:00:00 GMT

I pay for an AI coding assistant every month, and one day I did the obvious thing and asked: what if I just ran the model myself? The open-weight coders are good now. A spot GPU is a few pence an hour. It felt like it should be cheaper.

So I priced it out — properly, across the models I'd actually use (Qwen3.5, GLM 4.7 Flash, DeepSeek V3.2), for my real workload of roughly 12M tokens a month with a heavy cache-hit rate. The conclusion isn't the one I set out to prove. At my volume, self-hosting is not the way to save money. It's the way to buy something an API won't sell you.

The cheapest option is an API — by a mile

Here's the number that ended the "to save money" version of this project before it started. DeepSeek V3.2 Chat, at my usage, costs about $2.84 a month — and it scores 72–74% on SWE-bench Verified, which is genuinely strong for writing and fixing code. There's no server, no GPU, nothing to babysit. GLM 4.7 Flash has a free tier that costs literally $0.

Against that, my best self-hosted plan comes in around $11 a month. So the very first honest thing to say is: if the only axis you care about is cost, and your usage is modest, you stop here. You do not spin up a GPU to replace a $3 API bill. I nearly talked myself out of it on this line alone.

So when does self-hosting actually win?

The answer is a crossover. An API bill is variable — it scales with every token. A self-hosted spot box is a fixed cost — roughly the same whether you send it 12M tokens or 100M. So the more you use it, the better fixed looks.

Monthly usage	DeepSeek Chat (API, variable)	Qwen3.5-27B on a GCP L4 spot (fixed)
12M tokens	~$2.84	~$11.20
50M tokens	~$11.83	~$11.20
100M tokens	~$23.66	~$11.20

The lines cross at roughly 50M tokens a month. Below that, the API is cheaper and less hassle. Above it, the fixed box pulls ahead and keeps pulling. But notice what that framing quietly admits: at 12M tokens, cost is the wrong reason to self-host. The real reasons are the ones that don't show up in the table — no rate limits, nothing leaving your own hardware, and open weights (Apache 2.0) you fully control. Those are worth paying a small premium for. "Saving money" is not why you'd do it at my scale.

The best self-hosted coder — and the catch I only found by measuring

On paper, the pick is easy. Qwen3.5-27B scores 72.4% on SWE-bench Verified — level with DeepSeek — fits on a single 24GB L4 at a 6-bit quant, and costs about $11 a month on GCP spot. Same accuracy as the API, on hardware I own, for pocket change. Done, surely.

Then I put a stopwatch on it. On the L4, that 27B dense model generates at around 13 tokens per second. For a chat-style question that's fine. For an agentic coding loop — the kind that re-reads files, calls tools, and takes several turns to land a change — 13 tok/s is painful. The spec sheet sold me a model; the stopwatch sold me a much slower one. This is the whole lesson of the project in one line: never trust a throughput number you didn't measure yourself.

There was a second trap underneath it. Qwen3.5's hybrid architecture has known KV-cache bugs in llama.cpp right now (and therefore in Ollama), which make multi-turn sessions reprocess the whole prompt each turn. The workaround is real and worth knowing: serve it with vLLM, which handles the architecture properly.

The trade you're really making: speed vs. reasoning

Once you accept that a slow-but-accurate model is frustrating to code with, the interesting options are the fast Mixture-of-Experts ones. Qwen3.5-35B-A3B runs several times faster than the 27B; GLM 4.7 Flash does 100–145 tok/s and is built for running a few agents at once. More speed means more attempts in the same minute, and for iterative coding, more attempts often beats a single more-accurate one.

The catch — there's always a catch — is that the fast MoE models keep only a few billion parameters "active" per token, so their reasoning on hard, tangled problems is noticeably weaker. Which points at the setup that actually works: a fast local model for the 80% that's iteration, and a strong API reasoner (DeepSeek, or Claude Sonnet when it has to be right) for the 20% that's genuinely hard. Not either/or. Both, on purpose.

The transferable bit

Two habits travel far beyond this one project. First: separate the reason from the number. "It's cheaper" and "it's unlimited/private/mine" are different goals, and only one of them survives contact with a $2.84 API bill — be clear which you're buying. Second: measure the thing that matters to you (here, tokens per second in a real agent loop), because the headline benchmark and the number that decides whether a tool is pleasant to use are rarely the same number.

Where I actually landed

Start on DeepSeek Chat at ~$3 a month — validate that the quality is good enough for your work at essentially zero cost and zero risk. Reach for a self-hosted Qwen3.5 on a GCP L4 spot the day you actually need what the API can't give you: unlimited tokens, or your code staying on your own box. Serve it with vLLM, expect to pair it with an API reasoner for the hard problems, and go in knowing you're buying control, not a discount.

If you're weighing build-vs-buy on AI infrastructure and want the honest version — costs, crossovers, and the caveats the spec sheets leave out — that's the kind of work I do at twentytwotensors. Get in touch.

A weekly adequacy number from data you already log

Thu, 09 Jul 2026 00:00:00 GMT

I ended this series on the readiness score, and meant it as a close. But one tile kept nagging at me — I've sketched it and deleted it more than once — because it's the most useful number I don't currently have, and also the one I'm most afraid of getting wrong. So this is a bonus part: not something I've shipped, but a plan, and an honest account of why I haven't pressed the button yet.

The question I can only answer a few times a year

Every week I'd like to know one thing: am I actually getting enough treatment? The clinic answers that with a standard adequacy measure, taken from a blood test every few months. I wanted a read every week — and ideally from data I already log, without another needle. The catch is that a number about my own health is exactly the kind of thing you must not fake, so the whole design turned into an exercise in scoping honestly.

Why it was confusing — and how I made it simple

That measure isn't one number, and that's the trap. There's the clinic's version, which needs a blood marker measured before and after each session — data I don't have. And there's a volumetric version, which needs exactly the things the app already records: how much treatment fluid a session used, how long it ran, and an estimate of my body water.

The honest move was to stop trying to reproduce the clinic's figure. That was the source of all the ambiguity — chasing a number my data can't support. Drop that goal, scope to the version my logged data can actually compute, and frame it plainly as a personal weekly trend rather than a clinical result. The moment I let it be a different (well-defined) number instead of a worse copy of the clinic's, it became buildable.

Turning a week of sessions into one verdict

The shape is simple. A one-time profile — height, age, sex — feeds a standard formula for body water. Each session I log becomes a small adequacy figure. The week's sessions roll up into a single standardised weekly number, checked against the published target for that measure. On the home screen it's one tile you glance at — an illustrative week might read "above target" — tappable into a per-session breakdown so a weak session is visible instead of hidden in the average.

Everything it needs is already in the app. No new sensor, no new blood draw — just arithmetic over numbers I already record several times a week.

The one gate I won't skip

Here's the honest part, and it's the whole reason this tile isn't live. Buried in that calculation is one approximation I don't yet trust — roughly how saturated the used treatment fluid is with what's being cleared. I've penciled in a reasonable value, but it's a guess, and it sits right in the middle of the result. Get it wrong and the tile doesn't fail loudly; it shows a confident, wrong number about my health. That's worse than showing nothing.

The transferable bit

A number that's confidently wrong is worse than no number — especially in health, and especially when it looks tidy. So the rule for shipping this is firm: validate the one shaky approximation against ground truth (a real week checked against a known-good figure from the machine or the clinic) before it goes anywhere near the home screen. And when the data is thin — a missing session, less than a week logged — the tile says "not enough data yet," never a guess dressed up as a fact. Design the honest fallback before the happy path.

That's the whole philosophy of this app in a single tile: surface a genuinely useful signal from data you already have — but never a confident guess wearing the costume of a measurement. Whether this one ships comes down entirely to that validation week. If it checks out, it's a real weekly answer to a question I currently guess at. If it doesn't, it stays a sketch, and that's the right outcome too.

If you're building health or safety tooling where a wrong number is worse than a missing one — and you want the version that's honest about its own uncertainty — that's the kind of work I do at twentytwotensors. Get in touch.

Backing up everything that matters, for pennies a month

Thu, 09 Jul 2026 00:00:00 GMT

Everything I'd genuinely hate to lose is text. An Obsidian vault I think in, and a folder of code repositories. Not photos, not video — just the stuff I've written and built. Which is good news, because it means the backup I actually needed is small, cheap, and dull. I built it in one afternoon: an encrypted, versioned, daily, offsite copy of everything irreplaceable, for roughly 2p a month. Here's how, and — more usefully — the three things that make a backup real, none of which the tool does for you.

First I decided what NOT to back up

I started where most people start: "back up the whole machine." Then I actually looked into a bootable, bare-metal restore on macOS and concluded it basically isn't a thing without a local Time Machine drive — and I wanted the copy off the machine, in the cloud. So I stopped chasing the heroic version and asked a better question: what's genuinely irreplaceable here?

The answer was small. The vault, and the source of my repos — code and git history only. Everything else is re-downloadable: the OS, the apps, node_modules, Python venvs, the Flutter SDK, Terraform state, model weights, big data folders. All excluded. That one decision — naming what you're deliberately not protecting — is what shrank the job from "image a 500GB laptop" to a 1.22 GiB first snapshot. Cheap and trustworthy start with scope.

restic, into a Google Cloud Storage bucket

The tool is restic, backing up to a GCS bucket. Three properties made it the pick over an rsync copy or a nightly tarball:

Client-side encryption — Google only ever stores ciphertext. The data is readable on my laptop and nowhere else.
Deduplication — a daily run on a vault that barely changed stores almost nothing new, so "daily" is nearly free.
Snapshot history — point-in-time restore. It's a Time Machine for the cloud, not a single overwritten mirror. rsync and tarballs can't give you that, so they were out.

The retention I wanted — keep 7 daily, 4 weekly, 6 monthly restore points and prune the rest — is a single restic forget policy. A launchd agent runs it every day at noon (and on wake if the laptop was asleep). At GCS Standard prices, 1.22 GiB is a rounding error: a couple of pence a month, all in.

The one secret that has to outlive the machine

Client-side encryption has an uncomfortable flip side, and it's worth saying plainly: if you lose the passphrase, the backup is gone. Not "phone support" gone — mathematically gone. So the passphrase lives in two places, on purpose. In the Mac's Keychain, so the daily automated run can read it unattended. And escrowed in my password manager — the copy that survives if the laptop is lost, stolen, or dead. That second copy is the whole point: in a real disaster the machine is gone, and the entire value of the backup collapses onto one string of characters stored somewhere else. Guard it like it's the backup, because it is.

A backup you've never restored from is just a hope

The last step is the one everyone skips: I ran a restore drill. Pulled the vault and a repo back down from the cloud and diffed them against the originals — byte-identical. That drill is the entire difference between "I have backups" and "I have restores," and only one of those is worth anything at 2am.

It matters because backups fail in stupid, silent ways. Building this, a code review caught a real one: an empty-array expansion in the backup script that crashes under set -u on the ancient bash 3.2 that ships with macOS — the kind of bug that quietly turns a "daily backup" into nothing. So the job also emits a macOS notification if a run errors or the last success is more than two days old. A backup that fails quietly is identical to no backup at all.

The transferable bit

A backup is only ever as good as three things the tool won't decide for you: what you're deliberately not protecting (scope is what keeps it cheap and simple), the one secret that must outlive the hardware (and where a second copy lives), and proof that a restore actually works (a drill, not a promise). The software — restic, a bucket, a scheduled job — is the easy 20%.

It runs at noon every day and I've stopped thinking about it, which was the entire goal. If you want infrastructure that's boring on purpose — the kind you can forget because it's actually correct — that's the sort of work I do at twentytwotensors. Get in touch.

Three ways to OCR a document, and what each one costs

Thu, 09 Jul 2026 00:00:00 GMT

I wanted something modest: OCR some PDFs, cheaply, without babysitting a server. And because there's a good open-weight vision-language OCR model now — GLM-OCR, 0.9B parameters, top of the OmniDocBench leaderboard — my instinct was to self-host it. That instinct cost me an afternoon and taught me the actual lesson, which is about the question, not the model.

The self-hosting rabbit hole

Serving a 0.9B vision model is a GPU job, and GPUs don't like being idle-and-cheap at the same time. I walked the usual ladder — spot VM, then a container pipeline, then Cloud Run for scale-to-zero — and hit the tax at every rung. On Cloud Run, a cold start is 30–60 seconds before the first page is even read. AWS Lambda, the natural "cheap and serverless" pick, is out entirely for a model like this: 15-minute timeout, a 6 MB response cap, billing for init. I was engineering hard to make an expensive shape fit a cheap budget.

The question I'd skipped: do I even need a vision model?

Then I asked the thing I should have asked first. There are really two OCR jobs, and they get conflated constantly:

Text + coordinates — "what words are here, and where?" A classic detect-then-recognise pipeline. Plain text out, bounding boxes out.
Document understanding — tables reconstructed, reading order, structured Markdown or JSON. This is what a vision-language model buys you, and it's genuinely harder.

I'd assumed I needed the second and was paying for it. I needed the first. And once you only need text and coordinates, the model gets absurdly small. PP-OCRv6 (PaddleOCR) is a detect→recognise pipeline whose small tier is ~31 MB and its tiny tier ~6 MB — hundreds of times smaller than the vision model. A 6 MB CPU model doesn't just save money, it deletes the entire problem I'd been solving: no GPU, no meaningful cold start, and Lambda and the Cloud Run free tier become trivially sufficient. The infrastructure pain was self-inflicted by the model choice.

The plot twist: sometimes you should just pay the API

If you do need real document understanding — tables, structure, clean Markdown — there's a third option that beats self-hosting anything: a hosted OCR API. Mistral's OCR runs about $2 per 1,000 pages (roughly half that in batch mode), returns structured Markdown with reconstructed tables, and has a zero-second cold start because it's someone else's always-warm GPU. For comparison, AWS Textract is around $65 per 1,000 pages — a hosted API isn't automatically the expensive option; this one is ~30× cheaper than the incumbent. When your volume is a few thousand pages, "pay pennies per page and write no infrastructure" is very hard to beat.

Reading the numbers honestly

Two honesty notes, because this is where these comparisons usually mislead. First: don't reflexively pick the smallest model. For personal batch work, latency is irrelevant — nobody's waiting — so you should optimise accuracy, not size. Going from PP-OCRv6 tiny to small lifts recognition accuracy from ~73.5% to ~81.3% (about +8 points) for a cost that's still effectively nothing. The tiny tier is for when milliseconds matter, which for a batch job they don't.

Second, and I want to be straight about it: I haven't run the bake-off yet. The vision-model CPU throughput figures are estimates, the hosted-API accuracy numbers are the vendor's own benchmarks (not independently verified), and the real test — quality on my actual documents — is still to do. This is a scoping-and-costing exercise, not a validated result, and the honest next step is to run the same twenty pages through all three and look.

The transferable bit

Before you optimise how you deploy a model, check you've picked the right class of model. Half the infrastructure problems in an ML project are self-inflicted by reaching for a bigger model than the task needs — I spent an afternoon fighting GPU cold starts for a job a 6 MB CPU model does on a free tier. The cheapest deployment is the one you delete by asking "what do I actually need out of this?" first.

What I'd actually run

Model first, then deployment. For plain text and positions: PP-OCRv6 small, self-hosted on CPU — a spot VM for batches, or Cloud Run's free tier for the occasional job. For real structured document understanding: the Mistral OCR API — pennies a page, zero infrastructure. Self-host the vision model only in the one case that actually forces it: you need document understanding and privacy rules out sending pages to an API. That's a real case — but it's the exception, not the default I started with.

If you've got a document-processing pipeline to cost out — and you want the honest split between "self-host this" and "just pay for that" — it's the kind of work I do at twentytwotensors. Get in touch.

How RAPTOR actually works — and the three ways my code quietly wasn't it

Thu, 09 Jul 2026 00:00:00 GMT

I have a corpus of clinical guidelines — NICE, KDIGO, UKKA — and they have an annoying property for retrieval: the same fact is spread across all of them, rephrased a dozen ways. Plain semantic search over that is unstable. Ask the same question twice and it lands on a different paraphrase each time, because twenty near-identical chunks are all a similar distance from the query.

RAPTOR (Sarthi et al., Stanford, ICLR 2024) is a clean answer to exactly that. And I implemented it, and it worked — until I read the paper properly and found three places my "RAPTOR" had quietly drifted from the real thing. This is what it actually does, and what those drifts cost.

The idea: a tree of summaries over the untouched source

Most retrieval indexes your chunks and stops. RAPTOR keeps the chunks (the leaves — you still want them for detail and for citing a source), then builds a consolidation layer on top. The build is bottom-up and recursive:

Chunk the documents into small ~100-token pieces, never splitting a sentence.
Embed them.
Cluster the embeddings — reduce dimensions with UMAP, then fit a soft Gaussian Mixture Model (a chunk can belong to more than one cluster), using BIC to choose how many clusters there are.
Summarise each cluster with an LLM (the paper gets ~72% compression), then embed those summaries and recurse — cluster the summaries, summarise them — until you can't cluster any further.

What you end up with is a tree: raw passages at the bottom, tighter and tighter summaries above. Those scattered paraphrases of one fact get pulled into a single summary node. That's the whole point for a redundant corpus — not a benchmark bump, but a stable place for a diffuse idea to live.

The retrieval trick most people miss

There are two ways to query the tree, and the difference matters. Tree traversal starts at the root and walks down, top-k at each layer. Collapsed tree throws every layer into one big pool — leaves and summaries together — and just retrieves the most similar nodes up to a token budget (~2000 tokens, ~20 nodes).

Collapsed tree wins consistently, and it's the paper's default. The reason is lovely: it lets the granularity of the answer float. A broad question surfaces a high summary node; a specific one surfaces a leaf. You don't decide the altitude up front — the query does. In the paper's own results, 18.5–57% of retrieved nodes come from the summary layers, which is the evidence that the summaries are earning their keep rather than just padding the index. (For the headline-chasers: QuALITY jumps to 82.6% with GPT-4, up from a 62.3% prior best. But the granularity story is the part you'll actually feel.)

Then I compared my code to the paper

Here's the honest half. "I implemented RAPTOR" and "I implemented RAPTOR faithfully" turned out to be two different claims, and the gap between them is where every bug lived. Three of them.

1. The cluster with no size limit

My summariser concatenated all the text in a cluster and sent it to the LLM in one go. Fine for a small cluster. But a populous layer-1 cluster could pull together 60k+ tokens of member text — well past the model's limit — which meant silent truncation and a corrupt summary node sitting quietly in the tree. The paper's clusters are bounded by its recursive small-cluster structure, so its ~72% compression assumes a digestible input. The fix is the obvious one once you see it: map-reduce — pack members into budget-sized batches, summarise each, then summarise the summaries.

2. Single-stage clustering instead of global-then-local

I did one UMAP+GMM pass. The paper does two: a coarse global clustering, then a local clustering within each global cluster. I'd skimmed right past it — but that global stage is exactly the thing that groups paraphrases scattered across different documents. For my redundant corpus, it wasn't a detail; it was the highest-leverage fidelity fix in the whole method. The paper didn't just validate my approach, it named the two things I'd gotten wrong.

The transferable bit

When you implement a paper, the dangerous bugs aren't the parts you found hard — they're the parts you found easy and skimmed. I "knew" my leaves were roughly the right size, so I never measured them; I assumed they were a uniform 2000 tokens (a 20× mismatch with the paper). Measuring 2,650 real chunks showed a median of ~190 tokens — close to the paper — but a spread from 1 to 2,084, with 24% over 800. The variance was the problem, and it's precisely what detonated bug #1. Measure the thing you assumed, especially when the assumption felt too obvious to check.

3. A reasoning model eating its own budget

The one that actually hurt. After the fixes, a build finished cleanly — and 508 of 647 summary nodes were zero bytes. Eighty percent empty, reported as success.

The cause: I capped summary generation at 1024 tokens, and my summariser was a reasoning model that emits a long <thought> block before its answer. The 1024 tokens were entirely consumed by thinking; the model hit the cap mid-thought, produced no answer, and my code — which strips the thought block — was left with an empty string, dutifully cached as a 0-byte "summary." A raw-API repro confirmed it: 1024 tokens in, pure thought out, nothing after. Raising the cap to 4096 and refusing to cache empty results fixed it (and I made truncation log loudly instead of silently, so the next one can't hide). The deeper lesson was model choice: a small reasoning model (MedGemma-4B) was simply unfit — it truncated, looped, and leaked planning tokens — and I landed on a non-reasoning Gemma variant that just writes the summary.

Why bother telling you the drifts?

Because "we use RAPTOR" is on a hundred architecture diagrams, and the interesting question is never whether you use it — it's whether the tree you built is the one the paper describes. Mine wasn't, in three specific ways, and each one degraded retrieval invisibly: no crash, no error, just quietly worse answers. The method is elegant. The fidelity is the work.

If you're building retrieval over a messy, redundant corpus and want it done faithfully — and debugged down to why a node came back empty — that's the kind of work I do at twentytwotensors. Get in touch.

The hard part of a personal AI agent isn't the agent — it's the memory

Thu, 09 Jul 2026 00:00:00 GMT

I wanted a specific, unglamorous thing: a daily chat companion that knows who I am, remembers across sessions, and can wear a different hat depending on what I'm doing that day. Not a task agent, not a content factory — a companion I'd actually use. Let me be upfront before you read on: this is a design sketch, not a shipped system. Nothing here is built yet. But the design is where all the interesting decisions live, and where I nearly talked myself in circles.

The conversation is easy. Remembering is the problem.

A single turn — you say something, the model replies — is a solved problem. The hard part of a personal agent is everything between the turns: what does it carry from today into next Tuesday, and how does it do that cheaply enough to run every day without a growing bill or a context window that eventually explodes? "Memory" sounds like one feature. It isn't. It's at least four, and they behave differently.

Four tiers, and why they're different jobs

The design splits memory into four tiers, each answering a different question:

Working — the last few turns, held in the model's context window. No storage; it evaporates when the chat ends.
Session — the full transcript of the current conversation, one row per message. The verbatim record.
Summary — a short, embedded recap written after a session ends, searchable later. One row per conversation.
Semantic — the distilled, durable facts: "prefers terse replies," "is learning React," "works on a consulting brand." Atomic notes, each typed as a preference, fact, or decision, tagged and embedded.

A small honesty flag lives in that list. I first called the third tier "episodic" — until I realised that in the established agent-memory vocabulary, "episodic" means something specific (past action sequences replayed as examples). I was reading the wrong map and using a word that would mislead anyone fluent in it, so I renamed it "Summary." Minor, but it's the kind of terminology drift that quietly makes your design illegible to everyone else in the field.

Modelling memory like an Obsidian vault

The part I liked most: the semantic tier is modelled like a notes vault. Atomic notes, explicitly linked to each other — the same instinct as an Obsidian graph, where the connections carry as much meaning as the notes. Except it isn't markdown files; it's real Postgres, with a note_links table for the graph and pgvector for similarity search. A memory-writer — one cheap background model call at the end of each session — reads the transcript and returns structured JSON: a summary to embed, any new facts to file (with their type), and links to notes that already exist.

Retrieval at each turn is hybrid: ordinary SQL filters (which role, which tags, how recent) to narrow the field, then vector similarity to rank what's left, stitched into the prompt. The role itself is session-scoped — picked when the conversation starts and fixed for its duration. New hat, new session. That one rule sidesteps a whole category of bugs around swapping tools and stale system prompts mid-conversation. The whole thing is designed API-first, on Cloud Run, with the model doing turns and a cheaper model doing the background summarising.

The uncomfortable part: I might be over-building it

Now the honest twist, and it's the reason I'm writing this as a sketch and not a launch. While researching, three different tools each independently told me I was hand-rolling a worse version of something that already exists:

One minimalist agent's whole philosophy is "keep a small file of high-signal facts and prune it" — suggesting my four tiers might collapse into a single well-curated list.
A coding agent I use every day stores its memory as markdown files on disk, not a vector database — a strong hint that my "notes vault" instinct was right but my Postgres-and-pgvector machinery was overkill.
And a purpose-built memory library made the point bluntly: I'd essentially be writing a worse version of it, by hand.

Underneath sits a contradiction I couldn't resolve on paper. My pull toward "simple, linked, plain-text notes" points at files. My pull toward "serverless and SQL" points the exact opposite way. Both can't be the priority. When I reviewed my own notes, the diagnosis was embarrassing and familiar: analysis paralysis — six coherent, mutually exclusive architectures, no decision criterion to choose between them, and a goal that had quietly drifted from "a companion I actually use" to "understand agent memory deeply." Those are different projects.

The transferable bit

A design can be internally elegant and still be the wrong thing to build. The tell isn't a flaw in the diagram — it's when the ecosystem keeps handing you the same feedback from different directions ("this is simpler than you're making it"), and you keep finding reasons the diagram is special. The fix is a kill criterion and a shipping goal: build the cheapest version that runs, put it in daily use, and let real usage — not more research — decide which tiers earn their keep. Most won't.

What's built, and what's next

Nothing's built — and that's the honest headline. The real next move isn't more architecture; it's to stop researching, pick the single goal ("a companion I use"), ship the smallest thing that runs, and let a week of actually talking to it tell me whether I need four tiers or one good list. My bet, writing this, is that reality deletes at least two of the four. That's not a failure of the design — it's the design's job to be pruned by use.

If you're building an agent that has to remember its users — and you want the version that ships and gets pruned by real usage, not the one that's perfect on a whiteboard — that's the kind of work I do at twentytwotensors. Get in touch.

Where does dbt fit in three clouds?

Thu, 09 Jul 2026 00:00:00 GMT

Every time I start a data project I re-Google the same thing. "Is Glue the AWS Dataproc?" "Where does dbt actually run on Databricks?" So I finally drew the map once, properly, and the act of drawing it surfaced the one idea that makes the whole comparison simple. I'll get to that — but first, the frame that makes any of these stacks legible.

Everything is one of three layers

Strip the branding off and every cloud data stack is the same three layers: ingest (get data in), transformation (shape it), and storage & query (keep it, ask it questions). GCP, AWS and Databricks each have a full set of tools for all three — Pub/Sub or Kinesis or Auto Loader for streaming ingest; BigQuery or Redshift or Databricks SQL for the warehouse; and so on. Once you sort tools into those three buckets, the "which cloud" noise drops away and you can actually compare like for like.

The idea that makes it click: dbt isn't in any of them

Here's what I'd never quite articulated. dbt isn't an ingest tool, isn't storage, and isn't a compute engine. It's a SQL compiler, test runner and dependency graph that runs inside whichever warehouse already owns the compute. It doesn't move a single byte of data or spin up a cluster of its own — it turns your SELECTs into a managed, tested, documented DAG and hands them to BigQuery, or Redshift, or Databricks SQL to execute.

Which is why it plugs into all three clouds through a thin per-platform adapter:

Warehouse	dbt adapter	Runs on
BigQuery	`dbt-bigquery`	BigQuery slots
Redshift	`dbt-redshift`	Redshift cluster / Serverless
Athena (lake on S3)	`dbt-athena`	Athena / Trino over S3
Databricks	`dbt-databricks`	Databricks SQL warehouse
Snowflake	`dbt-snowflake`	Snowflake (the most common pairing of all)

Follow that thread and a nice conclusion falls out: the warehouse is the only real variable. Swap warehouses and you swap one adapter; your ingest (Fivetran, Airbyte), your transforms (dbt models), and your orchestration (Airflow) all travel with you. Choosing a stack is mostly choosing where the compute lives.

Where dbt wins — and where the native tool beats it

The honest part, because "just use dbt everywhere" is lazy advice. Each platform has a native transform layer that can genuinely beat dbt for its home turf:

Dataform (GCP) — now free inside BigQuery, Git-in-the-console, zero setup. For a BigQuery-only shop it's less to run.
Delta Live Tables (Databricks) — streaming-first, with a built-in data-quality expectations framework dbt doesn't have natively.
AWS Glue Workflows — not even a dbt competitor; it's a different layer (Spark/compute orchestration), and conflating them is a common mistake.

So the rule of thumb I settled on: dbt wins on portability, community, testing ergonomics and auto-generated docs. Native tools (Dataform, DLT) win on zero setup and streaming. Reach for dbt when the team already knows it, or when there's any real chance you'll switch warehouses. And know its edges: dbt doesn't move data, doesn't do heavy Spark/Python work, and doesn't schedule itself in production — it still needs Airflow, dbt Cloud, or Databricks Workflows to run on a cadence.

Three default stacks (and the hybrid everyone actually runs)

At a readable altitude, the out-of-the-box paths are:

GCP: GCS → Dataflow / BigQuery Data Transfer → BigQuery → dbt or Dataform → Looker.
AWS: S3 → DMS / Firehose / Glue → S3 + Glue Catalog (lakehouse) or Redshift → dbt or Glue Spark → QuickSight.
Databricks: object store → Auto Loader / DLT → Delta Lake on Unity Catalog → dbt or DLT or notebooks → Databricks SQL.

But the stack most teams actually run isn't any single vendor's poster — it's the hybrid: Fivetran or Airbyte for SaaS ingest, dbt for the SQL transforms, Airflow for orchestration, pointed at whichever warehouse they picked. Which is the earlier point again, from the other direction: the warehouse is the variable, everything around it is portable.

So which do I pick?

The short heuristics that survive contact with a real project:

Cheapest to start, small data: BigQuery — pay-per-query, no cluster idling. S3 + Athena a close second.
ML-adjacent, notebook-first: Databricks — the only one where notebooks are a first-class production artifact.
Already on Kafka: AWS (MSK) or Databricks (Structured Streaming).
Best governance out of the box: Unity Catalog on Databricks.
Most vendor-neutral: dbt + Iceberg + object store — avoids lock-in, at the cost of writing more glue yourself.

The transferable bit

When a tool seems to fit everywhere, it's usually because it's solving a different problem than the ones it sits next to. dbt feeling cloud-agnostic isn't magic — it's because it deliberately owns only the transform-as-code layer and delegates compute, ingest and scheduling to whatever's around it. Find the one layer a tool actually owns, and both its portability and its limits stop being surprising.

Two honest caveats before you take any of this as gospel: I haven't put concrete pound figures on a real workload here (a 1 TB/day medallion build would tell a sharper cost story), and the Iceberg-everywhere and dbt-Fusion stories are both moving fast enough in 2026 that the adapter picture may shift under us. Treat this as the decision map, not the final invoice.

If you're choosing a data stack and want the version grounded in your workload and budget rather than a vendor diagram, that's the kind of work I do at twentytwotensors. Get in touch.

Chunking clinical guidelines — where recall is won or lost

Wed, 08 Jul 2026 00:00:00 GMT

Here's the uncomfortable truth the evaluation surfaced: the single biggest weakness across every retriever is recall — the passage that best answers a question often just isn't in the retrieved set. And retrieval can only return chunks that exist. If the right sentence got split across two chunks, or buried in a 2,000-token wall of unrelated text, no retriever recovers it. Recall is a chunking problem before it's anything else.

PDF to clean text

Clinical guidelines are hostile source material — multi-column layouts, tables, figures, footnotes, cross-references. The documents go through Docling for OCR and layout parsing, which turns each PDF into structured markdown with the headings, lists and tables preserved rather than flattened into a soup of text. Getting this stage right matters more than it looks: everything downstream inherits its mistakes.

Split by section, then by size

A good chunk is about one thing. Guidelines hand you that structure for free — numbered sections and headings — so the splitter follows the document's own boundaries first, keeping a section together as a unit. Only when a section runs long does it sub-split, with a cap of 2,000 tokens per chunk and a 200-character overlap between consecutive chunks so a fact that straddles a boundary still appears whole in at least one of them. Splitting by structure beats splitting by a fixed window, because a fixed window happily cuts a dosing table in half.

The 512-token catch

An honest wrinkle I'll name rather than hide. The embedding model truncates its input at 512 tokens — but chunks can be up to 2,000. So a long chunk is embedded on roughly its first quarter, and the tail contributes nothing to whether it's retrieved. That mismatch is almost certainly part of the recall problem, and it's near the top of the fix list: either bring the chunk size down toward the embedding window, or embed a long chunk as several sub-windows and pool them. It's the kind of quiet defect that a metric points at but a demo never shows.

The duplicate corpus I didn't know I had

Late in the project I found the store held two ingestion passes — 43 document names for roughly 22 guidelines, because an earlier, coarser pass had never been cleaned out. Every retriever comparison had quietly been running over duplicated text. De-duplicating brought it down to 27 clean documents and changed the retriever ranking. A later pass found one more near-duplicate pair — two documents overlapping by 88% — which settled the corpus at 24 documents, 1,768 chunks. The lesson repeats itself across this series: fix the data before you trust the leaderboard.

A tree on top of the chunks

Leaf chunks aren't the only thing in the store. RAPTOR builds a hierarchy above them — clustering related chunks, summarising each cluster with an LLM, and repeating — so a broad question can match a summary node instead of hunting for one perfect leaf. That's covered in the retriever deep-dive; the point here is that even the summaries are only as good as the chunks they're built from.

The chunking lesson

Recall is decided at chunk time. Before a retriever can find the right passage, that passage has to be a retrievable chunk — cleanly extracted, sensibly bounded, and fully embedded. A context-recall score around 0.3 doesn't send you to a bigger model; it sends you straight back here.

The other half of this is the pipeline overview, and the knobs that tune all of it live in the parameters. Getting the unglamorous data layer right — where most RAG projects quietly lose — is what I do at twentytwotensors. Get in touch.

See it answer — grounded, cited, and willing to say "no"

Wed, 08 Jul 2026 00:00:00 GMT

The rest of this series is about how it's built and how it scores. This one is just: what does it actually say? Below are real questions run through the Level-1 assistant. Two things to watch for — every clinical claim is followed by the [Source: …] it came from, and the last example shows the behaviour I care about most in a research prototype over clinical text: it refuses when the guidelines don't cover the question, rather than inventing an answer. One caveat worth stating plainly — these are illustrative examples, not a measured abstention rate. On a separate adversarial set the retrievers scored 0/4 on out-of-corpus questions, so refusal is designed for and visible here, but it is not yet demonstrated to be reliable.

What are the dietary potassium restrictions for CKD stage 3?

For CKD stage 3, dietary potassium is generally limited to around 2,000–3,000 mg per day to reduce the risk of high blood potassium (hyperkalaemia), with tighter limits if your potassium runs persistently high [Source: NICE NG203].

There isn't a single blanket restriction for everyone — it's guided by your blood results, and where potassium is persistently raised the focus shifts to lower-potassium food choices and reviewing medicines that push potassium up [Source: UKKA Hyperkalaemia].

Educational information drawn from clinical guidelines — not a substitute for professional medical advice.

SourcesNICE NG203UKKA Hyperkalaemia

Can I take ibuprofen for a headache if I have CKD?

Generally, no. NSAIDs such as ibuprofen are best avoided in CKD — they reduce blood flow to the kidneys and can cause a further fall in kidney function or acute kidney injury, especially alongside blood-pressure medicines like ACE inhibitors [Source: NICE NG203].

Paracetamol is usually the preferred option for pain. If you take an NSAID regularly, check with your GP or pharmacist rather than stopping or continuing on your own.

Educational information drawn from clinical guidelines — not a substitute for professional medical advice.

SourcesNICE NG203

When are ACE inhibitors recommended in CKD?

An ACE inhibitor (or an ARB) is recommended when CKD comes with significant protein in the urine — for example an ACR of 30 mg/mmol or more with diabetes, or 70 mg/mmol or more without — because it slows the decline in kidney function [Source: NICE NG203].

Kidney function and potassium are checked after starting and each time the dose is increased.

Educational information drawn from clinical guidelines — not a substitute for professional medical advice.

SourcesNICE NG203

What kind of exercise is recommended if I have CKD?

Physical activity is encouraged. The guidance is to build up gradually toward around 150 minutes a week of moderate aerobic activity, combined with muscle-strengthening exercise, adapted to your CKD stage and how you feel [Source: UKKA Exercise and Lifestyle in CKD].

Regular activity supports blood pressure, cardiovascular health and general wellbeing.

Educational information drawn from clinical guidelines — not a substitute for professional medical advice.

SourcesUKKA Exercise & Lifestyle

Which brand of phosphate binder should I buy?

The provided guidelines do not cover this. They describe when and why phosphate binders are used, but they do not recommend specific products or brands — that's a decision for your kidney team based on your bloods and other medicines.

Sourcesnone returned

That last answer is the one that matters. A tightened prompt tells the model to answer only from the retrieved guideline text and, when that text doesn't contain the answer, to say so in as many words — "The provided guidelines do not cover this." In a clinical setting, a system that declines cleanly beats one that fills the gap with a confident guess.

What you're seeing

Retrieve → answer → cite, with grounding enforced by the prompt: no claim without a source, and a refusal when the guidelines are silent. How well it holds up under measurement — and where retrieval still misses — is the honest scorecard.

The rest of the series covers how this is built — the three levels, the pipeline, and the retrievers that decide whether any of it works. Building grounded, cited, privacy-aware RAG is what I do at twentytwotensors. Get in touch.

The knobs that move the numbers (and the ones that don't)

Wed, 08 Jul 2026 00:00:00 GMT

It's tempting to treat every constant in a config file as a lever to pull. In practice, only a few change the outcome — and knowing which is most of the skill. Here are the real values behind this CKD assistant, grouped by where they live, with the ones that actually matter called out.

Retrieval — the ones that matter most

These decide what the model even gets to see, so they move recall and precision directly. Top-k of 5: how many chunks come back per query — raise it and you catch more of the right passage (recall) at the cost of diluting the context with near-misses (precision). Similarity threshold of 0.3: chunks below it are dropped rather than padded in, so a weak query returns three good chunks instead of five mediocre ones. If I could only tune two numbers, it would be these.

Chunking

Chunk size 2,000 tokens, overlap 200 characters. Covered in the chunking deep-dive — and worth re-flagging that the embedding model only reads the first 512 tokens of a chunk, so these two interact more than they look.

Embeddings

768 dimensions. EmbeddingGemma supports Matryoshka representation learning, so the same model gives you 128, 256, 512 or 768-dimension vectors from one pass. I run the full 768 for quality; dropping to 256 is the lever if storage or latency ever bites, with a graceful quality trade rather than a model swap.

The retriever internals

Retriever	Parameter	Value	What it does
Tree	section-k	8	Section headings matched in phase one
Tree	chunks-per-section	3	Chunks pulled from each matched section
RAPTOR	max depth	3	Levels of summary above the leaves
RAPTOR	cluster dim	10	UMAP target dimensions before clustering
RAPTOR	min cluster prob	0.1	Soft-assignment threshold for the GMM
Hybrid	semantic / BM25	0.7 / 0.3	Weighting of dense vs keyword search — and this one is a bug, not a default. The fusion maths says the BM25 weight has to exceed (1+K)/(11+2K), which is 0.466 at K=60, or a document found only by BM25 can never reach top-k. The configured 0.3 was below every threshold, which is part of why hybrid failed its benchmark gate.

Generation — and two that caught me out

Temperature 0.7. Reasonable for answers, but it made the evaluation non-deterministic: the same retriever produces different answers on re-runs, which adds noise to already-small score differences. For a reproducible eval you want this near zero; for a friendly assistant you want it warm. Those are different jobs.

Max output tokens. Set generously at 32,768 for the reasoning model — which then blew up against a self-hosted server whose total context window is 32,768, since input plus output can't exceed it. A parameter that's fine on one backend and a hard error on another is a good argument for testing on the target you'll actually deploy to.

Repetition and frequency penalties (1.5 and 0.4) round out the generation config, there to stop the smaller models looping — a real failure mode on a 4B model that a 26B one rarely hits.

The parameters lesson

Don't tune everything — measure which knob moves your number. Here it's top-k, the similarity threshold and chunk size (all recall levers); nearly everything else is a sensible default you set once. And watch the parameters that behave differently across backends — temperature's effect on eval reproducibility, and an output-length that's valid on a hosted API but a 400 error on your own GPU.

These knobs sit on top of the retrievers and the pipeline, and their effect is what the evaluation measures. Knowing which parameter earns its keep — and which is just a number in a file — is what I do at twentytwotensors. Get in touch.

The retriever zoo — three bets scored, two that didn't make it

Wed, 08 Jul 2026 00:00:00 GMT

In a RAG system, the retriever is the part that fails quietly. If it hands the model the wrong passage, no amount of clever prompting recovers — the model either says nothing useful or, worse, fills the gap from memory. So after the simple baseline, most of my effort went into how to find the right chunk of guideline text. I benchmarked three retrievers end-to-end — flat, tree and RAPTOR — behind a single factory flag so any level could swap one in, plus two more that never earned a place in the comparison. This is each one, and what the numbers said.

Flat — plain, and hard to beat

The default. Embed the query, cosine-search every chunk in the store, keep the top few above a similarity threshold. Two small touches earn their keep: it expands medical terms so "CKD" also searches "chronic kidney disease," and it drops genuinely irrelevant chunks rather than padding the context to a fixed size. It's the least clever option and, as it turned out, the hardest to beat on precision — when it returns something, it's usually on-topic.

Tree — routing by structure (and the bug that hid for months)

Clinical guidelines have structure — numbered sections, headings, sub-headings — and the tree retriever tries to use it. Two phases: first match the question against a small index of section headings, then pull chunks only from the sections that matched. The bet is that narrowing to the right section before searching beats searching the whole corpus flat.

Here's the honest part. For a long time this retriever was silently broken. Its section-heading index had never been populated, and even after I fixed that, a metadata mismatch meant phase two joined on the wrong fields and found zero chunks on every query. Each time, it quietly fell back to flat retrieval — and because the fallback still returned five results, nothing ever looked wrong. "Returns results" is not the same as "works." It took evaluating the retriever in isolation to notice that "tree" and "flat" were producing identical routing. Fixed the join (document name plus section heading) and rebuilt the index, and it finally does what it says.

RAPTOR — summarise your way up

RAPTOR builds a tree on top of the chunks. Cluster the leaf chunks by meaning, have an LLM write a summary of each cluster, then cluster and summarise those summaries, and repeat — so you end up with layers of increasingly abstract nodes above the raw text. A query can land on a leaf (a specific fact) or on a summary node (a theme that spans several documents). That makes it good at broad, synthesising questions that no single chunk answers. I built the ~2,400-node tree with Gemma-4 E4B writing the summaries on a GPU box; it came out nominally top of the three, by a margin I'll argue below is too small to bank.

Hybrid and contextual — one that failed its gate, one that was never built

Two more were meant to join the comparison and didn't. Hybrid fuses dense semantic search with BM25 keyword search via reciprocal rank fusion, which should help when the answer hinges on an exact term (a drug name, an eGFR threshold). I rebuilt it from a stub into a genuinely working retriever and it failed its benchmark gate — I'd rather report that than quietly drop it. Contextual retrieval — prepending a short LLM-written sentence of context to each chunk before embedding — is designed but not built: there's no collection behind the flag, so the option cannot run at all. The zoo, honestly, is three animals and two enclosures.

The scoreboard

Scored with RAGAS on the de-duplicated corpus, Gemma-4 E4B as the generator, the tightened grounding prompt, ten CKD questions. Higher is better; the average is over the four metrics.

Retriever	Faithfulness	Answer rel.	Ctx precision	Ctx recall	Average
Flat	0.718	0.541	0.706	0.296	0.565
Tree	0.699	0.604	0.692	0.333	0.582
RAPTOR	0.760	0.695	0.842	0.259	0.639

RAPTOR leads on faithfulness and context precision — but at n=10 a gap of this size in the average is noise, and I'm not going to call it a win. The more useful result is the split, from a separate retrieval-only benchmark: RAPTOR wins natural-language questions, flat wins keyword and factual ones. Both score 0/4 on adversarial out-of-corpus queries — neither knows how to abstain. Flat and tree each have a personality too: flat holds strong precision, tree edges the others on recall (routing does surface a bit more of the right material). Two honest caveats. Answer relevancy is jumpy — a known RAGAS quirk where safe, hedged clinical phrasing gets read as evasive — so I lean on the other three. And context recall sits around 0.3 for everyone: the single passage that best answers the question often isn't retrieved at all. That's the ceiling on the whole system, and no choice of retriever fixes it.

Two lessons worth keeping

First: "returns results" isn't "works" — a broken retriever hid behind a fallback for months, and only isolated measurement caught it. Second: the fanciest retriever only pulls ahead once the corpus beneath it is clean. I'd been comparing on a store that quietly held duplicate copies of several guidelines; de-duplicating it changed the ranking. Fix the data before you read the leaderboard at all — and at ten questions, read it gently.

Where the real gain is left — recall — sends you straight back to the pipeline: chunking, and what actually makes it into the store. The full, unvarnished scorecard is in the evaluation. Building and honestly measuring retrieval over dense domain text is what I do at twentytwotensors. Get in touch.

Where the model runs — local, hosted, or a GPU box

Wed, 08 Jul 2026 00:00:00 GMT

A RAG system has two model calls: embed the text, and generate the answer. The trick to keeping deployment sane is to treat those two very differently. Embeddings are small, cheap and private, so they run locally, always. Generation is the expensive, swappable part — so it talks to whatever's cheapest or fastest in a given environment, through one interface.

The models

EmbeddingGemma 300M — embeds both the query and the chunks, at 768 dimensions. It's small enough to run on a laptop CPU, so it never leaves the machine. Patient text doesn't get shipped to an API just to be turned into a vector.
The generator — Gemma-4 E4B locally, or the larger Gemma-4 26B on AI Studio. (MedGemma 1.5 4B was the original and still names the project; nothing current runs on it.) None of these are hard-wired: the code just points at an OpenAI-compatible endpoint and asks for a model by name.

One endpoint, any provider

The generator is configured with two environment variables — a base URL and a model id. That's the whole abstraction. Point them at a local vLLM server and it self-hosts; point them at Google AI Studio's OpenAI-compatible API and it's fully managed; point them at anything else that speaks the same protocol and it just works. Embeddings stay local regardless. So moving from a hosted model to a self-hosted GPU is a config change, not a code change — which is exactly what you want when "where does this run" has a different answer on a laptop, in an eval loop, and wherever it might eventually be deployed.

Three places I actually ran it

Local (a Mac). Embeddings on the CPU, generation pointed at a hosted endpoint. No GPU, no server, cheapest possible dev loop. This is where most of the day-to-day work happened.

Hosted (AI Studio). Generation via Gemma-4 26B on AI Studio's API — zero infrastructure. The catch is honest: it's a reasoning model, so it's slow and emits long internal "thinking" before the answer, and the free tier throttles you on tokens-per-minute. Great for correctness, awkward for a fast eval loop.

A GPU box (EC2 g6, an L4, with vLLM). Self-hosting Gemma-4 E4B for full control and no rate limit. Two things bit me here. E4B disables FlashAttention, so vLLM falls back to a slower attention path — fine for a batch job, sluggish for interactive use. And spot capacity in my volume's availability zone was simply gone, so a one-off run meant paying on-demand. When I did rebuild the retrieval index on it, the move that mattered was ordering: stop the LLM server, use the freed GPU to pre-compute every embedding at once, then restart the server so the summariser model had the GPU to itself.

What I'd deploy

Embeddings local, always — it's cheaper and it keeps the sensitive text on your own hardware. Generation behind the provider-agnostic endpoint, chosen per environment: a hosted model for the dev and eval loop, and a warm GPU only if production latency actually demands it. What I would not do is stand up a GPU for a bursty, occasional workload — the spin-up time and the spot-capacity lottery cost more than they save.

The setup lesson

Decouple the generator endpoint from day one. "Where the model runs" should be a deployment decision you can change with an environment variable — never something baked into the application. Get that boundary right and you can move between a laptop, a managed API and your own GPU without touching a line of code.

The pieces this runs on — the pipeline, the retrievers, and the parameters — are the rest of the series. Building deployable, privacy-aware RAG that isn't chained to one vendor is what I do at twentytwotensors. Get in touch.

Building an in-app knowledge base your past self will thank you for

Thu, 02 Jul 2026 00:00:00 GMT

You learn a lot treating yourself at home. Not from a course — from things going slightly wrong and figuring them out. What a specific alarm actually means. The order of steps that clears it. The little procedure a nurse showed me once that isn't written down anywhere.

The trouble with hard-won knowledge is that it fades. I'd solve something, feel clever, and three weeks later be back to square one, scrolling old texts trying to remember what worked. So I gave it somewhere to live.

My own notes — in my words, one search away. (Synthetic data.)

Writing one down is a title and a body. Nothing else to fill in.

The version in my own words

The manual is a binder. What I needed was the version that says "when you see this, do that" — mine, phrased the way I'd explain it to myself at 6am. So each entry is just a title and free text. The moment a notes app asks me for tags and categories, I stop writing in it; the whole value is that capturing a note costs nothing.

And it quietly helps the assistant

There's a bonus I didn't plan for at first. These notes aren't only for me to read — the AI assistant can pull them in as context. So when I ask "what does this alarm mean?", the answer leans on my own documented fix instead of a generic one. The knowledge I keep for myself makes the assistant sound like me.

Next: the data I wasn't tracking at all — sleep, heart rate, the state of my body between treatments.

Bringing body data into a health app: fitness trackers + dialysis

Thu, 02 Jul 2026 00:00:00 GMT

My treatment log is careful. Blood pressure, weight, fluid, the whole panel. But it's all about the sessions — and the days I struggle with aren't always the session days. The gap between "how I feel" and "what my tracker knows" was the state of my body between treatments. My sleep. My resting heart rate. How recovered I actually was.

My watch had been quietly recording all of it the whole time. I just never brought it anywhere useful.

Steps, resting HR, sleep, SpO₂, HRV — in one glanceable place. (Synthetic data.)

Bringing the body into the picture

So the app grew a fitness tab: steps, resting heart rate, sleep, oxygen saturation, heart-rate variability — synced in and laid out as plain insight tiles. Nothing clever on the surface. The point isn't the dashboard; it's that for the first time my "how I feel" has numbers sitting next to it.

The quiet foundation for what's next

On its own, this tab is just a nicer way to see body data. But it's really the groundwork for the thing I most want: a daily readiness score that looks at last night's sleep and recovery and tells me, honestly, what kind of day my body is set up for — so I can plan around how I actually am, not how I hope to be. You can't build that without the data first. This is the data first.

Next: the small settings that quietly make the whole app fit one specific person — me.

Personalized settings in a health app: one number changed everything

Thu, 02 Jul 2026 00:00:00 GMT

There's a single number my whole treatment revolves around: my dry weight — where my body should sit with no excess fluid. Get it wrong and the fluid goal each session is wrong, which is not a rounding error when it's your blood pressure and how you'll feel that evening.

A generic app would bury that in a profile screen and move on. In mine it's the first thing settings asks for, because almost everything downstream is derived from it.

The few knobs that make the app fit one body. (Synthetic data.)

My usual session length — so the countdown starts right, every time.

Small settings, big reach

There aren't many of them, and that's deliberate. My dry weight drives the fluid goal. My default treatment duration drives the live countdown and pre-fills the post-treatment summary, so I'm not setting the same length every single session. Each setting is small; each one quietly removes a decision I'd otherwise repeat forever.

My keys, my data

Two more live here for a reason. The AI assistant runs on my own key, so the intelligence is under my control, not rented from behind someone's paywall. And export is one tap — my treatments and blood tests as CSV or PDF whenever I want them, to hand a clinician or just to keep. It's my health data; leaving with it should be trivial.

That's the theme of this whole project, really. Not an app for patients in the abstract — an app that bends to one patient. The settings page is just where that bending is most obvious.

Next: the assistant that reads all of this — the sessions, the trends, the supplies, the notes, the baseline — and answers back in plain words.

Tracking treatment consumables without thinking about it

Thu, 02 Jul 2026 00:00:00 GMT

Treating at home means the storeroom is a spare bedroom, and the stock manager is me. Every session eats the same set of things — bags, a cartridge, needles, saline. Miss an order and it's not an inconvenience later, it's a missed treatment now. For a long time I "managed" this by opening the cupboard and hoping.

The question I actually needed answered was never "how many bags do I have?" It was quieter than that: how many more sessions can I run before I have to reorder? So I taught the app to answer it for me.

Green, amber, red. I only have to notice the colour. (Synthetic data.)

I told it once how much a session uses, and what buffer I want.

When a delivery lands, I tick off what actually arrived.

It counts so I don't have to

I set one thing up: how much each item a session uses, and how much I like to keep spare. That's it. From there the app quietly does the arithmetic I used to do on the cupboard floor — turning stock into a plain "about this many sessions left," and a coloured dot so I can tell in a glance whether I'm fine, should think about ordering, or need to act today.

Trust, but verify the delivery

The part that keeps the count honest over months is small and unglamorous: when supplies arrive, I don't assume the order came in full. I confirm what actually turned up and adjust anything short or over. Only then does the stock go up. It's the difference between a number I glance at and a number I believe.

Since supplies are eaten by treatments, logging a session can nudge the count down on its own — so the number stays live without me tending it. The whole feature is designed to be forgettable. I don't want to think about bags. I just want the cupboard to never surprise me again.

Next: the notes I've learned the hard way — the alarm at 6am, and what to do about it.

A local-first community health app: why I'd keep data off the cloud

Tue, 23 Jun 2026 00:00:00 GMT

The community build keeps everything on-device — and exports to CSV and PDF. (Synthetic data.)

My own version uses the cloud — sync, the AI assistant, a Sheet my clinical team can read. That suits me. But it doesn't suit everyone, and it shouldn't have to. Some people don't want an account; some don't trust their health on someone else's server; some just want to open an app and start. All completely reasonable. So the same Flutter codebase builds a second flavour for them.

What it lets go of

Behind a build flag, the community edition is fully local-first: data lives on the device, there's no cloud, no sign-in, no account to create. You install it and start logging. Nothing leaves your phone unless you decide to send it.

What it keeps

It keeps the part that actually matters — the daily tracking loop — and adds on-device export to CSV and PDF, so you can hand a clean record to your own clinical team without any of my infrastructure in the middle. There's a distributable APK and a web build.

You can try the web build right now: homehd-community.web.app. Nothing you type leaves your browser.

One codebase, two products, switched by a flag. The real work is keeping the boundary honest: the cloud features sit behind interfaces, so the local build simply doesn't wire them up — no forked code to maintain.

Why bother

Partly engineering discipline — if a feature can be cut by a flag, your boundaries are honest. Mostly, though: I had to build the tracking part anyway, and giving it to other people in the same situation costs me a build target, not a rewrite. That's a good trade.

A readiness engine for any condition (not just wearables)

Tue, 23 Jun 2026 00:00:00 GMT

That's the series. Across 14 posts I've shown a real system I built and use every single day: the tracking loop, the supplies and the hard-won notes, the data and trends, the AI assistant and its action layer, the body data, the £0 architecture, the community edition, and the readiness score I'm building next. None of it is a mock-up. It's the thing I run my life on.

Still on the roadmap

A skills layer — more things the assistant can actually do, not just describe.
On-device speech and a small local model, for when there's no signal.
And the readiness score itself, turned from a plan into a real number.

The bigger point

Strip away the specifics of my condition and what's left is a pattern: take someone's own structured data, make it trustworthy and legible, put an assistant on top that reasons over it, and help them plan around it. I built it for one condition. It transfers to many — any illness where your capacity changes day to day — and to plenty of problems that have nothing to do with health.

If this is your problem too

Patients: if you live with something similar, try the local-first community edition — it runs entirely in your browser at homehd-community.web.app.
Researchers, clinicians, commissioners, founders: if you want this readiness-engine pattern built for a specific condition, I've already built the hard, real version. Let's talk.

Building production AI, data, and cloud systems — grounded in real use, not slideware — is what I do at twentytwotensors. If any of this is useful to you, get in touch.

ADK vs LangGraph: one AI support system, two frameworks

Tue, 23 Jun 2026 00:00:00 GMT

I built a customer support agent for a fictional UK transport company, UKConnect. The 8-part series builds it on Google's Agent Development Kit (ADK).

Then I built the whole thing again on LangGraph.

Same Gemini model. Same SQLite database and the same policy documents. Same two specialist agents. The only thing I changed was the framework. So it's a fair comparison — and it answers the question I get asked most: which one should I use?

The system is the same shape

Both versions work the same way. Something reads the message and decides what it needs. Then a specialist handles it — one answers from policy documents, one acts on the booking database.

Same system, two frameworks

All of that comes down to one thing: who decides where a message goes, and how much of that I write myself.

ADK decides for you

In ADK, the Master Agent is an LlmAgent, and the two specialists are registered under it. I describe each agent and let the framework hand the work over. The model picks who should answer; ADK does the routing. Guardrails sit on before/after callbacks. State lives in the ADK session.

The good part is how little I had to write. I described the agents and the tools, and the framework wired the conversation together. If you've already picked Gemini and Google's stack, that's a lot of plumbing you don't have to own.

LangGraph makes you draw it

LangGraph turns the flow into a graph I draw myself. A low-temperature Gemini call in a router node reads each message and tags it — policy, ticket, or general. Then conditional edges send it to the right node. The specialists are ReAct agents. Policy search runs on FAISS. A MemorySaver checkpointer keeps the state, so a conversation can be resumed or replayed.

I wrote more code than with ADK. In return I can see the routing, log it, and test it edge by edge. And the state is something I can save and replay.

What actually changed

Routing

This is the whole decision. ADK routes for you — less code, and you trust the model to get it right. LangGraph makes you write the routing — more code, but you can inspect it and test it. When something goes wrong, being able to point at the exact edge that fired is worth a lot. For anything regulated or high-stakes, I want that.

Retrieval

The ADK build uses plain scikit-learn cosine similarity. No extra infrastructure, and fine for a few hundred policy chunks. The LangGraph build uses FAISS, which is worth it once the document set grows. Pick for your data size, not the logo.

State

ADK session state is enough for one conversation. LangGraph's checkpointer makes the state durable — save it per session, resume it later, replay a run to debug it. That matters more in production than in a demo.

Lock-in

ADK is Google-native. It gives you the most for the least when you're all-in on Gemini. LangGraph sits in the wider LangChain ecosystem, so swapping models or reusing tools is easier. Both of mine run on Gemini on purpose — I wanted the framework to be the only thing that differed.

So which one?

Use ADK if you're committed to Google and Gemini and want to write as little orchestration as possible.
Use LangGraph if you want routing you can see and test, state you can replay, or room to move across models and tools.
Most of the time, someone eventually asks "why did it do that?" — and that's when the explicit version pays for itself.

Both are open source

You can read and run both:

ADK version: production-ai-customer-support
LangGraph version: production-ai-customer-support-langchain

For the full ADK walkthrough — architecture, RAG, the database layer, orchestration, testing, and an honest look at what it's worth — start with the series.

Need this built?

Picking the right framework and shipping it to production — with guardrails, evaluation, and a flow you can actually explain — is what I do at twentytwotensors. Get in touch and tell me about your use case.

"Why am I itching this week?" — and it actually knew

Tue, 23 Jun 2026 00:00:00 GMT

I ask in plain words; it answers from my own numbers. (Synthetic data.)

A generic chatbot can tell you what a blood marker means. It can't tell you why yours moved this month, because it has never seen your numbers. Living with a condition, that second question is the only one that matters — and it's the gap I wanted to close.

It reads my situation before every answer

The idea is simple, and the whole thing hangs on it: before each reply, the app builds a fresh prompt from my live data —

my current or most recent session,
a summary of my latest blood results and how they've moved,
my supplies and what's running low,
and the relevant notes from my knowledge base.

So the model isn't recalling facts about the world. It's reasoning over a snapshot of my situation, refreshed on every message. Ask the same question next week and the answer changes, because the numbers underneath have.

The itch, explained

That late-night question worked because retrieval is keyed to symptoms, not just keywords. Say "I've been itchy," and the assistant knows which markers tend to drive it, pulls those from my recent results, and points at the one that actually moved. The question is fuzzy; the answer is grounded in my own data. That combination is the part worth engineering.

Voice, because my hands are busy

During a treatment my hands aren't free, so I can just talk to it — audio goes straight to the model, which for this app beats on-device speech-to-text on accuracy and effort. I speak, it reads my data, it answers.

The pattern, generalised

Strip out the health specifics and this is a function-calling assistant whose context is rebuilt from live application state every turn, with retrieval keyed to intent. That pattern works for any product sitting on structured user data — support, finance, operations. The data changes; the architecture doesn't.

Building assistants that reason over a business's own live data — safely, grounded in real records — is exactly the kind of work I do at twentytwotensors. If that's a system you need, get in touch.

Next: giving this assistant hands, so it can act in the app instead of only talking.

How to build a daily readiness score from health data

Tue, 23 Jun 2026 00:00:00 GMT

That morning guess is the quiet tax of a fluctuating condition. My capacity genuinely changes day to day, I can't see it coming, so I plan the day blind. Everything else in this series is shipped and in daily use — but this post is different. This is the part I'm still building, and I'm going to be clear about what's real and what's not.

The foundation, already live: daily body data syncing in. (Synthetic data.)

What already works

The groundwork is done: the app pulls my body data in every day — sleep, heart-rate variability, resting heart rate, and more — and lands it right next to my treatment history and blood trends. One place. That part works today.

What I'm building on top of it

The next leap is a single daily readiness score: fuse those signals into one honest number for how my body actually is this morning, then let the assistant help me plan around it. Demanding things on the good days. Pull back on the rough ones. Replace the gamble with a read.

That score doesn't exist yet. I'm not going to show you a finished dashboard I haven't built — that's exactly the overclaim this whole series avoids. What I have is the data and the plan. I'll show the score when it's real.

Why it matters beyond me

A tracker for one condition is niche. A readiness engine isn't. The hard problem of any fluctuating-capacity illness is the same one I have every morning: capacity changes, you can't see it coming, so you plan badly and lose time. Long COVID, ME/CFS, heart failure, recovery after treatment — different conditions, same daily gamble. Same engine, different thresholds.

The thesis

Quantify your daily readiness from your own body data, then plan your day around it to reclaim time. The shipped app is the proof I can build it. The readiness score is the reason it matters beyond me.

If you're a research group, a clinician, or a founder working on chronic-illness self-management — this is the conversation I most want to have. The pattern is general; I've already built the hard, real version of it. Get in touch.

How I built a system to run my life around a chronic illness

Tue, 23 Jun 2026 00:00:00 GMT

The home screen. Every treatment I log shows up here. (Synthetic data.)

I have a chronic illness.

For years I planned my whole life around it. When to rest. When I could work. When I had energy and when I didn't. Most days I guessed wrong.

So I built software to help me.

A lot of numbers, every week

My condition means a lot of numbers to track. Blood pressure. Weight. Fluid. A blood-test panel every month. Supplies that run out if I stop counting. And the treatments themselves, several times a week.

For a long time that lived in a Google Sheet I hated opening. The sheet worked, but it didn't help. It stored numbers. It never told me anything.

What I built instead

Now it's an app. I built it myself, on my own time, for my own use. It does three things so far.

It logs each treatment — before, during, and after.
It reads my blood tests and shows the trends, two years of them.
It has an AI assistant that looks at my real data and answers questions. I can ask "why am I itching this week" and it points at the marker that actually moved.

Logging a live session — readings come in while the treatment runs. (Synthetic data.)

None of this is a demo. I use it every week. The screenshots here use synthetic data, but the app is the one I actually run my life on.

I'm not a content creator

I'm a patient who got tired of guessing. I happen to build software for a living, so I built the thing I wished existed.

That's the honest version. I'm not selling an app. There's no waitlist. I wanted to stop opening a spreadsheet I hated, and I wanted answers I could trust — because they came from my own numbers.

What's next

Over the next few weeks I'll show how each part works: the tracking, the data, the AI assistant, and the architecture (it runs on £0).

And the part I'm building next: a daily readiness score from my own body data — sleep, heart rate, the rest — so I can plan my day around how my body actually is, not how I hope it is. That part isn't built yet. I'll show it honestly as I go.

If you live with something similar, follow along. I think a lot of us have the same problem.

About this series

This is part 1 of Home HD — a series on building a health system solo, from the daily tracker to the AI assistant and the cloud architecture behind it. Building production AI and data systems is also my day job at twentytwotensors; if that's useful to you, get in touch.

What does a factory's electricity actually cost?

Tue, 23 Jun 2026 00:00:00 GMT

Here's the question I set out to answer: for a factory that runs around the clock and buys electricity at the prevailing wholesale price (no fixed deal), what is its price exposure for a single delivery day?

The answer I landed on, for a Tuesday in June 2026: a base case of ~10.8 p/kWh for a flat 24/7 load, with a range of ~10.2 to 10.8 p/kWh depending on the load profile. And — the part I care about most — it matched the day's realised traded price to within 0.5%.

The hard part was scoping, not maths

An electricity bill has several parts: the wholesale energy, network charges, policy levies, and supplier margin. Most of those are largely fixed and known in advance. Only one is genuinely set by the market on the day — the wholesale price. So that's the part that's actually "exposed," and that's what I estimated. Quoting a big all-in number would have been easy and wrong; the question was about exposure, and only the wholesale bit is exposed.

Second decision: this is something I can read, not something I have to forecast. The GB day-ahead auction publishes each day's hourly prices the afternoon before delivery — for the 16th, it cleared at 09:57 the day before. So I'm reading a settled, money-backed market price, not predicting one. Exposure for the day is then just the consumption-weighted average of those hourly prices.

The load profile is the biggest source of uncertainty, and I didn't have it — so I carried it as an explicit assumption and reported a range across plausible profiles, rather than hiding a guess inside a single number. A flat 24/7 load is the top of the range; daytime- or overnight-weighted loads are cheaper, because they miss or partly miss the evening peak.

Free data only — and a validation step

The actual auction price is free to view but its API is paid, so I read the published cleared prices directly. The one free machine-readable GB series is Elexon's Market Index Data — the volume-weighted price of trades actually done in each half-hour. It isn't a day-ahead price and isn't published in advance, which makes it perfect for one thing: checking my estimate after the fact against what really traded.

The two lines track closely all day. The daily average gap was +0.5%. That's the difference between "here's a number" and "here's a number, and here's the evidence it was right."

Why the level is what it is

On a weekday like this, gas plants are usually on the margin — they're the last unit dispatched, so they set the price. That's why the level sits where it does and why the evening peak is pronounced. It's worth saying out loud in the writeup, because a number with no explanation is hard to trust.

The transferable bit

Most of the value in a data task like this isn't the calculation — it's the scoping ("what is genuinely the exposed quantity?"), the honesty about the biggest uncertainty (the load profile, carried as a range), and validating the answer against ground truth. Those three habits travel to almost any estimation problem.

If you've got a measurement or estimation problem like this — where the answer needs to be defensible, not just plausible — that's the kind of work I do at twentytwotensors. Get in touch.

Giving an AI assistant hands: MCP in practice

Tue, 23 Jun 2026 00:00:00 GMT

It fills the form; the amber fields show what it touched; I confirm. (Synthetic data.)

Once the assistant could read my data (part before this one), the obvious next step was letting it do things. But "an AI that can act on my medical data" is exactly the sentence that should make you nervous. So I built its hands in two deliberate layers, both designed to be trustworthy before they're clever.

The layer that acts — and then stops

This one drives the app: open a screen, pre-fill a form, start a flow. Crucially it doesn't poke at the interface directly — every action goes through a small state machine, so "fill the pre-treatment form" is a defined, allowed move, not the model clicking around hopefully. And it always stops short of committing. It pre-fills; I confirm. The amber highlight you can see above is a deliberate tell: it shows me exactly what the AI touched before I sign off.

The layer that only looks

The second layer is a read-only /api/mcp endpoint that exposes my data to a model over the Model Context Protocol — read-only, by design. A model can look; it can't write to my records over the wire. Anything that changes data has to come back through the on-device path, where I'm in the loop.

The rule: separate "look" from "touch"

Reading is low-risk and can be broad. Acting is higher-risk and should be narrow, explicit, and confirmable. Splitting them into two layers — a read-only data context and a constrained, human-confirmed action layer — is how you get an agent that's genuinely useful without being scary. That split holds for almost any agent in production.

This is the agent question every team hits eventually: not "can the model call tools," but "what is it allowed to do unsupervised, and how do I prove it." When it's your own health on the line, you answer that question honestly.

Home dialysis saves the NHS thousands — and it's the better treatment

Tue, 23 Jun 2026 00:00:00 GMT

For my condition, treatment can happen two ways: in a hospital unit several times a week, or at home. When I was offered home, it was equal parts freedom and terror — you're running the whole thing yourself, no nurse in the room. I said yes anyway. Most people, completely understandably, don't.

That "most people don't" stuck with me. Because home isn't the compromise option — it's cheaper for the NHS and better for patients, and both of those are in the published evidence, not my opinion. So I went and found the numbers, because a real cost case is worth more than a good story.

The numbers

£9–12k

saved per patient, per year, treating at home vs. in a unit

16%

treated at home now — against the NHS's own 20% minimum

£28.8–52.3m

per year saved if every centre hit that target

~4.4%

uptake for my specific home treatment

The per-patient saving — about £9,000 to £12,000 a year — is from Roberts et al. (2022), the most rigorous recent UK cost analysis. The policy target and the system-wide saving come from the NHS's own Getting It Right First Time (GIRFT) programme (2022): it recommends a minimum of 20% of patients treated at home, the national rate sits around 16%, and closing that gap would save tens of millions a year.

One honest caveat. You'll see bigger figures quoted — £15k to £20k a year. Those are real, but they apply to a different home modality, or to older tariff models. For a clean like-for-like comparison, £9–12k is the number I'd defend.

The gap isn't money, and it isn't safety

Here's the part that matters, and it's the part I felt myself. Treating at home is already cheaper, and outcomes are at least as good — better blood pressure control, fewer hospital stays, better quality of life in the NICE evidence. So the barrier to more people doing it isn't cost, and it isn't safety.

It's confidence.

Running the whole thing yourself, several times a week, with no nurse in the room — that is genuinely daunting, and I remember exactly how daunting. People stay in the unit because home feels unsupported, not because it's worse.

The opportunity, in one line

If the blocker is confidence and support rather than cost or outcomes, then better software — something that tracks the treatment, watches the trends, and answers questions from your own data — is exactly the kind of infrastructure that moves the number. That's what I'm building.

Why I'm writing this down

This is the most concrete part of the whole project. Not "here's a nice app" — "here's a measurable saving, an NHS target the country is missing, and a tool aimed at the actual blocker: the feeling I had on day one."

If you work on this from the other side — a commissioner, a research group, a founder, a clinical team — that's the conversation I want to have. Turning lived experience and data into systems people can actually use is also what I do for a living. Get in touch if it's useful.

A leakage-free fraud feature store, built with dbt

Tue, 23 Jun 2026 00:00:00 GMT

This project takes five raw tables — transactions, cards, users, merchant-category codes, and fraud labels — and turns them into something a fraud model can actually train on. It's a standard dbt project on BigQuery, with a staging layer and a marts layer. The interesting part is the feature store at the end, and one rule it has to obey.

The shape: staging → marts

Nothing exotic in the layering, on purpose. Staging models are views that rename, cast and lightly clean each source — parse dates, trim merchant fields, normalise error values, add a couple of derived fields like debt-to-income. The marts are tables: an enriched fct_transactions fact (incremental, so it only processes new rows), a dim_users dimension with lifetime spend and fraud rate, a daily rpt_fraud_summary, and the feature store.

Every staging model carries tests — uniqueness and not-null on keys, and accepted_values on fields like card brand and type — so a bad assumption about the source data fails the build instead of quietly poisoning a model downstream.

The rule that matters: no leakage

Here's the part that's easy to get wrong and expensive to discover late. The feature store computes velocity, baseline and deviation features — things like "how much has this card spent in the last hour" or "how far is this amount from the user's normal." The temptation is to compute those with a plain window over the whole table.

That leaks. If a feature for a transaction includes the transaction itself — or anything after it — the model gets to peek at the future during training. It looks brilliant in evaluation and falls apart in production, because in production the future genuinely isn't available yet.

The one constraint

Every feature uses data strictly before the current transaction's timestamp — never the row itself, never anything later. Get this right once, in the transformation layer, and every model trained on the feature store inherits it for free. Get it wrong and you'll chase a phantom accuracy you can never ship.

Doing this in dbt rather than in notebook code matters: the point-in-time logic lives in one tested, versioned place that every consumer reads from, instead of being re-implemented (and re-broken) in each data scientist's feature script.

Why a fact table and a feature store, not one big query

fct_transactions is the honest record of what happened — every transaction, enriched with its dimensions and label. The feature store is a derived view of that, shaped for one purpose: training. Keeping them separate means the fact table stays reusable for reporting and analysis, while the feature store can be opinionated about leakage and time windows without distorting the source of truth.

Building clean, tested data layers that machine learning can actually trust — leakage-free, versioned, documented — is a big part of what I do at twentytwotensors. Get in touch if your feature pipeline needs that.

Logging a dialysis session: the UX that makes people actually do it

Tue, 23 Jun 2026 00:00:00 GMT

The routine is always the same. I weigh myself, set up, and sit down for a few hours. Partway through, I take readings. Then it ends and there's a tidy-up. Three steps — and for years the record of them lived in a notebook and a spreadsheet I updated later, badly, from memory.

So the heart of the app just is those three steps. It doesn't reinvent my evening. It sits alongside it.

Before: vitals and the goal for tonight. (Synthetic data.)

During: a timer, and readings as they happen.

After: the summary is mostly written already.

Before I begin

I put in my weight and a couple of vitals, and the fluid goal works itself out from my dry weight. But every field stays editable, because real days don't match the formula — some evenings I already know the number should be different. The medications I take mid-treatment carry over from last time, so I'm not re-typing the same thing several times a week.

While it runs

The middle is the part that simply has to work. A reading goes in and it's saved — no spinner, no "are you sure", no chance of losing it if my phone hiccups. Hooked up, mid-treatment, is the worst possible moment to fight an app, so the writes land on the device instantly and sync later. The engineering is there entirely so I never have to notice it.

When it's done

By the end, the summary has mostly filled itself in — duration, totals, final vitals. I glance at it, fix anything that's off, and it joins the history on the home screen.

That was the one rule I set myself: auto-fill everything, but never lock me out of correcting it. A tool that argues with you when you know your own body better doesn't get used twice.

Next: what happens when you keep two years of these numbers — turning them into trends I can actually read.

Level 2 — Agentic RAG with LangGraph

Tue, 23 Jun 2026 00:00:00 GMT

Level 1 answers questions. It doesn't check what's in the question, doesn't notice when a question isn't really a retrieval question, and doesn't know if its own answer was any good. Level 2 fixes those three gaps by turning the flow into an explicit LangGraph state machine — each step a node, the state passed between them.

The graph

Every query runs through the same path: check for PII → classify intent → branch. A normal clinical question goes retrieve → generate → evaluate. But not everything should hit the retriever, so the router has four exits:

Retrieval — a real clinical question; do the full RAG path.
Direct — a simple conversational turn that needs no documents.
Clarification — too vague to answer; ask for more.
Out of scope — not a CKD question; decline politely.

Making that routing explicit (rather than hoping one big prompt handles every case) is the whole point of Level 2: you can see which path fired, log it, and test each branch on its own.

PII first, always

The first node runs Microsoft Presidio with custom recognizers for things like NHS numbers, so anything identifying is caught and redacted before the query is logged or sent anywhere. In a health context this can't be an afterthought bolted on later — it's the first thing that happens, by design.

The system scores itself

On the retrieval path, an evaluation node can run RAGAS on the response — faithfulness, context precision, recall — so quality is measured per answer, not assumed. I also added CKD-specific checks: did it cite a source, did it include the right disclaimer, was the advice appropriate to the stage. Those custom metrics encode "what a good answer looks like here," which generic metrics can't.

Why a graph, not just more prompt

The value isn't the LangGraph library — it's that the control flow becomes a thing you can inspect, test edge by edge, and extend. "PII is always checked first" and "out-of-scope questions are declined, not answered" stop being hopes and become structure. That's exactly the routing-control trade-off I wrote about in the ADK vs LangGraph comparison.

Next: when one assistant isn't focused enough — splitting it into specialists. (Part 5)

From 24 clinical PDFs to a vector store

Tue, 23 Jun 2026 00:00:00 GMT

Everyone wants to talk about the model. But for a RAG system, the answer is only ever as good as what you retrieved, and what you retrieve is decided here — in the pipeline that turns source documents into vectors. For the CKD assistant (Part 1) that source is 24 clinical guideline documents from NICE, KDIGO, the UK Kidney Association and Kidney Care UK.

OCR that respects tables

Clinical guidelines are full of tables — dosing thresholds, staging tables — and a naive text extract turns those into mush. I used IBM's Docling with table-structure recognition on, so a table comes out as a table, not a scrambled run of numbers. It exports both markdown (readable) and a structured JSON tree. Some pages are scanned, so OCR is on too.

Splitting out the parts that aren't clinical content

A guideline PDF is maybe 60% clinical content and 40% noise for my purposes — title pages, conflict-of-interest declarations, a long bibliography, abbreviations. If you embed all of it, your retriever happily returns a citation list when someone asks about potassium. So each document is split into main_text, references, and front/end matter, and only the clinical content goes forward.

I did the classification with a hybrid approach: MedGemma classifies each heading, with a regex heuristic as a fallback when the model isn't available. The fallback matters — it means the pipeline still runs without a GPU, just a bit more bluntly.

Cleaning and block-aware chunking

Then a manual cleaning pass for OCR artifacts (I tracked each document's review status — pass, needs-check, or delete; one KDIGO guideline was superseded by a newer edition and dropped). Finally, chunking. The rule I care about: don't slice through a coherent block. A chunk that ends mid-table or mid-recommendation retrieves badly, so the chunker is block-aware rather than a blind fixed-size window.

The clean chunks are embedded with EmbeddingGemma into 768-dimension vectors and stored in ChromaDB under one collection. That collection is the shared foundation every level in this series reads from.

The point

Most "the RAG isn't working" problems are pipeline problems — bad OCR, references polluting the index, chunks cut through the middle of a table. Spend your effort here before you reach for a bigger model. I found this out the hard way in the evaluation: nearly all the failures were retrieval, which traces straight back to this stage.

Next: the simplest system that uses this store — Level 1, plain RAG. (Part 3)

Evaluating a clinical RAG — the honest results

Tue, 23 Jun 2026 00:00:00 GMT

It's easy to ship a RAG demo that looks great on three hand-picked questions. The whole reason to evaluate is to find out where it actually breaks — so this part is about measuring the CKD assistant honestly, and being straight about what the numbers say.

How I measured it: RAGAS

I used RAGAS, which scores a RAG pipeline without a hand-labelled golden dataset by using a cheap external model as a judge. Four metrics, each answering a plain question:

Faithfulness — did the answer only say things the retrieved text supports? (the anti-hallucination metric, and the one that matters most for anything clinical)
Answer relevancy — did it actually address the question?
Context precision — were the retrieved chunks actually useful?
Context recall — did retrieval find everything it needed?

The results, unvarnished

Two lenses, because they disagree in a useful way.

The first is a source-match confusion matrix: for each question, did retrieval pull from the right document? Aggregate precision 0.53, recall 0.27, F1 0.36 — not a number I'd put on a marketing page, which is exactly why it's here.

The second lens scores the whole answer with RAGAS, comparing the retrievers on a de-duplicated corpus with the tightened grounding prompt. Higher is better; the average is over the four metrics.

Retriever	Faithfulness	Answer rel.	Ctx precision	Ctx recall	Average
Flat	0.718	0.541	0.706	0.296	0.565
Tree	0.699	0.604	0.692	0.333	0.582
RAPTOR	0.760	0.695	0.842	0.259	0.639

RAPTOR came out nominally ahead, with flat and tree close behind — close enough that at ten questions I read the gap as noise rather than a verdict. The full breakdown is in the retriever deep-dive. But getting to a table I'd trust meant fixing two things I'd assumed were fine. The store quietly held duplicate copies of several guidelines — a second ingestion nobody had cleaned up — which muddied every comparison until I de-duplicated it. And the tree retriever had been silently falling back to flat for months. Both only surfaced once I measured the components in isolation.

One change clearly helped the generation side: tightening the prompt to answer only from the retrieved text and to refuse when it can't. On the same model and data that lifted faithfulness across every retriever — RAPTOR's went from 0.70 to 0.76 — for the price of the assistant honestly saying "the guidelines don't cover this" a little more often.

Why it's hard, and where the fix is

This is the honest lesson of the whole series. Across every retriever the consistent weak spot is context recall — around 0.3: the single passage that best answers a question often isn't in the retrieved set at all, and if it wasn't retrieved, no bigger model or extra agent recovers it. RAG over dense clinical guidelines is genuinely hard — the text is long, cross-referential and precise — so finding exactly the right passage is most of the battle. The next effort doesn't go into a bigger model or more agents; it goes back to the pipeline — better chunking, better recall.

What I'd take to a real build

Measure before you believe. Treat a mediocre score as the most valuable output you have — it points straight at the thing to fix. And in a high-stakes domain, a system that scores honestly and refuses when unsure beats one that scores impressively on a curated demo and quietly hallucinates in the field.

That's the series — from why three levels through the pipeline, the three levels themselves, and now the honest scorecard. Building grounded, evaluated, privacy-aware RAG — and telling you where it's weak — is what I do at twentytwotensors. Get in touch.

Level 3 — A multi-agent clinical assistant

Tue, 23 Jun 2026 00:00:00 GMT

By Level 2 the assistant is safe and self-aware, but it's still one generalist. The problem with one generalist over CKD is that the domains pull in different directions: "how much potassium can I have" wants dietary thresholds and a calculation, "is ibuprofen safe for me" wants kidney-safe medication rules, and both want different slices of the guidelines. Level 3 makes each its own specialist.

An orchestrator and four specialists

A router reads the query and hands off to the right agent, then a final step synthesises the result into one reply:

Knowledge (RAG) agent — general evidence-based answers from the guidelines, with citations.
Diet agent — dietary guidance with actual calculation for potassium, phosphorus, sodium and protein, not just prose.
Medication agent — kidney-safe medication guidance and interaction warnings.
Lifestyle agent — exercise, blood pressure, smoking, stress.

Each specialist is narrower, so it can have a sharper prompt and pull the right part of the knowledge base instead of competing with everything else for the model's attention. The orchestrator's job is just classification and synthesis — the same coordinator-plus-specialists shape I used in the customer-support system, in a different domain.

When the extra layer earns its place

Honestly? Not always. For a plain factual lookup, Level 1 answers just as well and Level 3 is overhead. Where the specialists pull ahead is the questions that genuinely span domains, or where a domain needs to do something — like the diet agent computing against a limit rather than describing it. That's the rule: add an agent when there's a distinct job with distinct logic, not because multi-agent sounds impressive.

The honest caveat

More agents don't fix bad retrieval — they just route to it more precisely. If the chunk the Diet agent needs isn't being retrieved, a dedicated Diet agent still can't answer. Which is exactly why the last part of this series is about measuring retrieval, not admiring the agent graph.

Next, the part that keeps all of this honest: evaluation, and results I'm not going to dress up. (Part 6)

MedGemma CKD: building a clinical RAG assistant three ways

Tue, 23 Jun 2026 00:00:00 GMT

This is an eleven-part walkthrough of a retrieval assistant for chronic kidney disease (CKD), grounded in 24 real clinical guideline documents — NICE, KDIGO, the UK Kidney Association, Kidney Care UK. I built it for the Kaggle MedGemma Impact Challenge, and I built it three times on purpose.

Most "agentic AI" demos skip the boring question: do you actually need the agents? So I started with the simplest thing that could work and only added machinery when a number told me to. That gives three levels.

The three levels

Level 1 — Simple RAG. Retrieve the relevant guideline text, answer from it, cite the source. The honest baseline. (Part 3)
Level 2 — Agentic RAG. The same retrieval wrapped in a LangGraph state machine that redacts PII, routes by intent, and scores itself. (Part 4)
Level 3 — Multi-agent. A router hands off to Diet, Medication, Lifestyle and Knowledge specialists. (Part 5)

The shared foundation

All three levels sit on exactly the same base, which is what makes the comparison fair. The guideline PDFs are turned into clean, chunked text and embedded with EmbeddingGemma into a ChromaDB vector store; and a Gemma-4 model writes the answers from whatever was retrieved. (MedGemma 1.5 4B was the original generator and it's where the project's name comes from, but it was replaced by Gemma-4 — E4B locally, 26B on AI Studio — and none of the numbers in this series come from it.) Only the orchestration changes between levels — never the model or the knowledge base. Building that foundation is its own job, and it's the next part of the series. (Part 2)

And an honest ending

I'll also show the evaluation, and it isn't flattering. RAG over dense clinical guidelines is genuinely hard, and the scores say so. I'd rather walk through why than pretend otherwise — that's Part 6.

Why this is worth reading

The pattern — start simple, earn each escalation with a measurement, and be honest when the numbers are middling — is how I'd approach any RAG build, clinical or not. The CKD domain just makes the stakes (and the difficulty) obvious.

Grounded, evaluated, privacy-aware RAG is a chunk of what I do at twentytwotensors. Get in touch if you're building one.

Level 1 — Simple RAG, the honest baseline

Tue, 23 Jun 2026 00:00:00 GMT

Level 1 is deliberately plain: take the question, embed it, search the vector store, hand the top chunks to the generator, and answer from them with a citation. No agents, no routing. The reason to build it first is that it sets the bar — if the fancier levels don't beat this, they're just complexity.

Embeddings: one model, several sizes

Queries and chunks are embedded with EmbeddingGemma. It supports Matryoshka representation learning, so the same model gives you 128-, 256-, 512- or 768-dimension vectors from one pass — you can trade retrieval quality against speed and storage without swapping models. I ran at 768 for quality, with optional caching so repeated queries don't re-embed.

The retriever is where the work is

The default retriever does two unglamorous but important things. It expands medical terms — "CKD" also searches "chronic kidney disease," and so on — so a query phrased one way still finds text phrased the other. And it matches on word boundaries, so "AKI" doesn't spuriously match the middle of another word. There's a configurable similarity threshold so genuinely irrelevant chunks are dropped rather than padded in.

I also built more elaborate retrievers — a hybrid one using reciprocal rank fusion, a tree retriever that routes by section, and RAPTOR-style and contextual variants — and kept them behind a single factory so a level can pick one. But the lesson from the evaluation was sobering: clever retrieval helps less than getting the chunks right in the first place.

Generation, and saying "I don't know"

The model answers only from the retrieved context, and cites it. The prompt's job is to keep it honest — answer from the chunks, and if the chunks don't cover the question, say so rather than improvising. For a clinical assistant that refusal behaviour is a feature, not a failure.

The baseline lesson

For a lot of direct factual questions, simple RAG is genuinely enough. Build it, measure it, and only add machinery where it falls short. Everything in the next two parts is an answer to "where does this baseline break?" — not decoration on top of a working system.

Next: wrapping this in a LangGraph workflow that adds the safety and routing a clinical assistant needs. (Part 4)

Running a health app on zero pounds (all-free-tier GCP)

Tue, 23 Jun 2026 00:00:00 GMT

I built this for myself, on my own time, with no intention of paying a monthly bill to keep my own health data alive. So the whole system runs on Google Cloud free tiers — not "cheap," actually £0. The pieces fit like this:

Firebase Hosting serves the web build.
Cloud Run runs a small Hono server for the AI calls and the /api/mcp endpoint.
Firestore holds the treatment and health data.
Cloud Storage holds files and exports.
A weekly Google Sheet sync gives the clinical team a read-only summary — no new app, no login.

The moment that must never stall

Free tiers have one sharp edge: Cloud Run scales to zero, so when it wakes it cold-starts. A few seconds of "please wait" is fine for a chat reply. It is not fine when I'm mid-treatment, hooked up, trying to save a reading before I forget it. That scenario is the whole reason the app exists — it can't be the one place it fails.

So the write path that has to be instant skips the server entirely: treatment data is written client-side, straight to Firestore, with a custom auth token. No Cloud Run in the loop. The server only handles the things that can tolerate a pause — the AI, the exports. The critical path got its own route, on purpose.

The transferable idea

Don't put your whole app behind one tier of infrastructure. Find the one path that must never stall, and architect that path for it — even if it takes a different route than everything else. Free tiers are generous if you respect where their sharp edges are.

Designing cloud architecture that's both cheap and dependable — knowing which corners are safe to cut and which aren't — is a big part of what I do at twentytwotensors. Get in touch if that's useful.

Turning blood tests into trends: lab data you can act on

Tue, 23 Jun 2026 00:00:00 GMT

Six months of one marker, filtered to the home phase, with the latest values below. (Synthetic data.)

Two years of monthly panels is a lot of numbers over a lot of dates. Any single result barely means anything on its own — what I actually care about is the direction. A folder of PDFs holds all the data and answers none of that. So the real work wasn't drawing a chart; it was getting the data into a shape where the direction became obvious.

The boring schema that made it easy

I store results in long format — one row per marker, per datetime — not a wide table with a column per marker. It sounds like a detail, but it's the whole trick: adding a new marker never changes the schema, and every chart becomes a simple filter and sort. Boring, and exactly why it keeps working.

One bad month shouldn't erase the rest

Here's the thing that nearly ruined the charts. Every result is tagged with a phase — at home, in a unit, or during a hospital admission — because a rough spell in hospital throws out extreme values. Plot those on the same axis as steady months and the storm flattens everything else into a flat line, and you learn nothing. Tagging the phase lets me filter the storm out, or look at it alone, so the at-home trend stays honest. Those filters sit at the top of the screen above.

Where I deliberately didn't use AI

The results arrive clean and structured, so reading them is a regex job, not an AI job. I only reach for a model when the input is genuinely messy or the task genuinely needs judgement — and a well-formatted result line is neither. That restraint is a habit worth keeping: use the simplest thing that works, and save the model for the part that actually thinks.

Next: the other thing that quietly piles up when you treat at home — the supplies — and the night I nearly ran out of them.

LLM Fine-tuning Guide 2024

Sun, 15 Dec 2024 00:00:00 GMT

Large Language Models (LLMs) have revolutionized how businesses handle text-based tasks, from customer service to content generation. However, off-the-shelf models often fall short when it comes to domain-specific requirements. This is where fine-tuning becomes essential.

💡 Key Takeaway

Fine-tuning allows you to adapt powerful general-purpose LLMs to your specific use case, achieving better performance than generic models while being more cost-effective than training from scratch.

What is LLM Fine-tuning?

Fine-tuning is the process of taking a pre-trained language model and adapting it to perform better on specific tasks or domains. Instead of training a model from scratch (which requires massive computational resources), you start with a model that already understands language and teach it your specific requirements.

Types of Fine-tuning

Task-specific fine-tuning: Adapting a model for specific tasks like classification or summarization
Domain adaptation: Training on domain-specific data (legal, medical, financial)
Instruction tuning: Teaching models to follow specific instructions or formats
RLHF (Reinforcement Learning from Human Feedback): Aligning model outputs with human preferences

When Should You Fine-tune?

Not every use case requires fine-tuning. Consider fine-tuning when:

Domain-specific language: Your industry uses specialized terminology
Consistent output format: You need structured, predictable responses
Performance gaps: General models don't meet your accuracy requirements
Cost optimization: You want to use smaller, more efficient models
Data privacy: You prefer on-premise deployment

Data Preparation: The Foundation of Success

Quality data is the most critical factor in successful fine-tuning. Here's how to prepare your dataset:

Data Collection

Gather 1,000-10,000 high-quality examples (depending on task complexity)
Ensure examples represent real-world scenarios
Include edge cases and challenging examples
Maintain consistent annotation guidelines

# Example data format for instruction tuning
{
"instruction": "Summarize this financial report for executives",
"input": "Q3 revenue increased 15% year-over-year...",
"output": "Key highlights: 15% revenue growth, strong market position..."
}

Data Quality Checks

Remove duplicates and inconsistencies
Validate input-output pairs
Check for bias and fairness issues
Split data appropriately (80% train, 10% validation, 10% test)

Fine-tuning Strategies

Full Fine-tuning

Updates all model parameters. Most effective but requires significant computational resources.

Parameter-Efficient Fine-tuning (PEFT)

Techniques like LoRA (Low-Rank Adaptation) that update only a small subset of parameters:

LoRA: Adds trainable rank decomposition matrices
Adapters: Inserts small neural networks between layers
Prompt tuning: Optimizes soft prompts while keeping model frozen

🚀 Pro Tip

Start with LoRA fine-tuning - it requires 90% less memory than full fine-tuning while achieving similar performance for most tasks.

Technical Implementation

Choosing the Right Base Model

Consider these factors when selecting a base model:

Model size: Balance performance vs. computational requirements
License: Ensure commercial usage rights
Domain relevance: Some models perform better on specific domains
Architecture: Encoder-decoder vs. decoder-only models

Hyperparameter Optimization

# Key hyperparameters for fine-tuning
learning_rate = 5e-5  # Start lower than pre-training
batch_size = 16       # Adjust based on GPU memory
epochs = 3-5          # Avoid overfitting
warmup_steps = 500    # Gradual learning rate increase
weight_decay = 0.01   # Regularization

Evaluation and Monitoring

Metrics to Track

Task-specific metrics: BLEU, ROUGE, F1-score, accuracy
General quality: Perplexity, human evaluation scores
Business metrics: User satisfaction, task completion rates
Safety metrics: Toxicity, bias, hallucination rates

Preventing Overfitting

Monitor validation loss throughout training
Use early stopping when validation performance plateaus
Apply regularization techniques (dropout, weight decay)
Validate on held-out test set

Deployment Considerations

Model Serving Options

Cloud APIs: Easy to scale but higher latency
On-premise deployment: Better privacy and control
Edge deployment: Reduced latency for real-time applications
Hybrid approaches: Combine cloud and edge for optimal performance

Optimization Techniques

Quantization: Reduce model size with minimal performance loss
Distillation: Create smaller student models
Pruning: Remove unnecessary parameters
Caching: Store common responses for faster serving

Cost Management

💰 Budget Planning

I'm not going to quote you a price band, because the honest answer depends entirely on model size, data volume and where you run it. What I can give you is one real anchor: my Mistral 7B clinical-notes LoRA run took 17.9 minutes on a single GPU. A 4-bit LoRA on a 7B model is a pounds-not-thousands job; what actually costs money is the serving afterwards, and the data preparation before.

Cost Optimization Strategies

Use parameter-efficient methods (LoRA, adapters)
Leverage spot instances for training
Implement gradient checkpointing to reduce memory usage
Consider smaller base models for simpler tasks

Common Pitfalls and How to Avoid Them

Data-Related Issues

Insufficient data: Start with at least 1,000 quality examples
Data leakage: Ensure proper train/validation/test splits
Annotation inconsistency: Develop clear guidelines and validate annotations

Technical Challenges

Catastrophic forgetting: Use lower learning rates and fewer epochs
GPU memory issues: Implement gradient accumulation and mixed precision
Slow convergence: Adjust learning rate schedule and warmup

Future Trends in LLM Fine-tuning

The field is rapidly evolving with new techniques emerging:

Multi-modal fine-tuning: Adapting models for text + image tasks
Few-shot fine-tuning: Learning from minimal examples
Federated fine-tuning: Training across distributed data sources
Automated hyperparameter optimization: AI-driven tuning processes

Conclusion

LLM fine-tuning is a powerful technique for adapting general-purpose models to specific business needs. Success depends on quality data, appropriate technique selection, and careful evaluation. While the initial investment can be significant, the long-term benefits in terms of performance and cost efficiency make it worthwhile for many applications.

🎯 Ready to Get Started?

If you're considering LLM fine-tuning, this is genuinely my ground — LoRA on Llama 3 for a brand-voice notification model, and again on Mistral 7B for clinical-notes summarisation, where it moved ROUGE-1 from 0.381 to 0.423. Happy to think through whether fine-tuning is even the right answer for your problem, since often it isn't. Get in touch.

MLOps Best Practices

Thu, 05 Dec 2024 00:00:00 GMT

Machine Learning Operations (MLOps) has emerged as a critical discipline for organizations looking to scale their AI initiatives effectively. While building ML models is challenging, deploying and maintaining them in production environments presents an entirely different set of complexities.

🎯 What You'll Learn

This comprehensive guide covers the essential MLOps practices that enable reliable, scalable, and maintainable machine learning systems in production environments.

Understanding MLOps

MLOps combines machine learning, software engineering, and DevOps practices to standardize and streamline ML workflows. It addresses the unique challenges of ML systems, including data drift, model degradation, and the experimental nature of ML development.

Why MLOps Matters

Scalability: the practices that let a team manage many models without the process collapsing
Reliability: Ensure consistent model performance
Reproducibility: Recreate results and debug issues
Compliance: Meet regulatory and audit requirements
Collaboration: Enable seamless teamwork between data scientists and engineers

The MLOps Lifecycle

A mature MLOps pipeline encompasses the entire machine learning lifecycle:

Data Management

Version control for datasets, data validation, and feature engineering

Model Development

Experiment tracking, model versioning, and reproducible training

Model Validation

Automated testing, performance evaluation, and bias detection

Deployment

CI/CD pipelines, containerization, and infrastructure as code

Monitoring

Model performance tracking, data drift detection, and alerting

Governance

Model lineage, audit trails, and compliance reporting

Data Management Best Practices

Data Versioning

Treating data as code is fundamental to reproducible ML workflows:

# Example using DVC (Data Version Control)
dvc init
dvc add data/training_set.csv
git add data/training_set.csv.dvc
git commit -m "Add training dataset v1.0"
dvc push

Data Validation

Implement automated checks to ensure data quality:

Schema validation: Verify column types, names, and constraints
Statistical validation: Check distributions, ranges, and correlations
Freshness checks: Ensure data is recent and complete
Drift detection: Monitor changes in data distributions

# Example data validation with Great Expectations
import great_expectations as ge

df = ge.read_csv("data/new_batch.csv")

# Define expectations
df.expect_column_values_to_not_be_null("customer_id")
df.expect_column_values_to_be_between("age", 18, 100)
df.expect_column_mean_to_be_between("purchase_amount", 10, 1000)

# Validate
validation_result = df.validate()

Feature Store Implementation

Centralized feature management ensures consistency across teams:

Feature discovery: Catalog of available features
Feature lineage: Track feature transformations
Point-in-time correctness: Prevent data leakage
Online/offline consistency: Same features for training and serving

Model Development and Experiment Tracking

Experiment Management

Systematic experiment tracking enables better model development:

# Example using MLflow
import mlflow
import mlflow.sklearn

with mlflow.start_run():
# Log parameters
mlflow.log_param("learning_rate", 0.01)
mlflow.log_param("max_depth", 5)

# Train model
model = train_model(learning_rate=0.01, max_depth=5)

# Log metrics
mlflow.log_metric("accuracy", 0.95)
mlflow.log_metric("f1_score", 0.92)

# Log model
mlflow.sklearn.log_model(model, "model")

Model Versioning Strategy

Implement semantic versioning for models:

Major version: Breaking changes in API or significant architecture changes
Minor version: Backward-compatible improvements
Patch version: Bug fixes and minor updates

🔧 Versioning Example

Model v2.1.3 indicates: Major version 2 (new architecture), Minor version 1 (feature enhancement), Patch 3 (third bug fix).

Automated Testing for ML Systems

Types of ML Tests

ML systems require testing beyond traditional software testing:

Data tests: Validate input data quality and consistency
Model tests: Verify model behavior and performance
Infrastructure tests: Ensure deployment environment reliability
Integration tests: Test end-to-end pipeline functionality

Model Testing Framework

# Example model testing with pytest
import pytest
import numpy as np

class TestModel:
def test_model_accuracy(self, trained_model, test_data):
"""Test model meets minimum accuracy threshold"""
predictions = trained_model.predict(test_data.X)
accuracy = accuracy_score(test_data.y, predictions)
assert accuracy >= 0.85, f"Model accuracy {accuracy} below threshold"

def test_prediction_latency(self, trained_model, sample_input):
"""Test model inference time"""
start_time = time.time()
prediction = trained_model.predict(sample_input)
latency = time.time() - start_time
assert latency < 0.1, f"Prediction latency {latency}s too high"

def test_model_invariance(self, trained_model):
"""Test model behavior on edge cases"""
# Test with edge cases, noise, etc.
pass

CI/CD for Machine Learning

ML-Specific CI/CD Pipeline

Traditional CI/CD must be adapted for ML workflows:

# Example GitHub Actions workflow for ML
name: ML Pipeline

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.9

- name: Install dependencies
run: pip install -r requirements.txt

- name: Data validation
run: python scripts/validate_data.py

- name: Run model tests
run: pytest tests/test_model.py

- name: Train and validate model
run: python scripts/train_model.py

- name: Deploy to staging
if: github.ref == 'refs/heads/main'
run: python scripts/deploy_staging.py

Deployment Strategies

Choose the right deployment strategy based on your requirements:

Blue-Green Deployment: Zero-downtime deployment with instant rollback
Canary Deployment: Gradual rollout to subset of traffic
A/B Testing: Compare model performance with controlled experiments
Shadow Deployment: Run new model alongside production without affecting users

Model Monitoring in Production

Key Monitoring Metrics

Comprehensive monitoring covers multiple dimensions:

Model Performance: Accuracy, precision, recall, latency
Data Quality: Missing values, outliers, distribution changes
Data Drift: Changes in input feature distributions
Concept Drift: Changes in the relationship between features and target
Infrastructure: CPU, memory, disk usage, error rates

Alerting Strategy

# Example monitoring setup with Evidently AI
from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab, NumTargetDriftTab

def monitor_model_drift(reference_data, current_data):
dashboard = Dashboard(tabs=[DataDriftTab(), NumTargetDriftTab()])
dashboard.calculate(reference_data, current_data)

# Extract drift metrics
drift_report = dashboard.tabs[0].info['metrics']

if drift_report['dataset_drift']:
send_alert("Data drift detected in production model")
trigger_retraining_pipeline()

Model Governance and Compliance

Model Registry

Centralized model management ensures governance and compliance:

Model metadata: Training data, hyperparameters, performance metrics
Lineage tracking: Data sources, feature transformations, model ancestry
Approval workflows: Model review and sign-off processes
Access control: Role-based permissions for model access

Audit and Compliance

Maintain comprehensive audit trails for regulatory compliance:

Model decisions: Log all model predictions with timestamps
Data lineage: Track data sources and transformations
Model changes: Document all model updates and reasons
Human oversight: Record human interventions and overrides

Infrastructure and Scalability

Containerization

Docker containers ensure consistent deployment environments:

# Example Dockerfile for ML model
FROM python:3.9-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy model and code
COPY models/ models/
COPY src/ src/

# Set environment variables
ENV MODEL_PATH=/app/models/model.pkl
ENV PORT=8080

# Health check
HEALTHCHECK --interval=30s --timeout=10s \\
CMD curl -f http://localhost:$PORT/health || exit 1

# Start API server
CMD ["python", "src/api.py"]

Orchestration

Use workflow orchestration tools for complex ML pipelines:

Apache Airflow: Python-based workflow orchestration
Kubeflow: Kubernetes-native ML workflows
MLflow: End-to-end ML lifecycle management
Prefect: Modern workflow orchestration platform

Cost Optimization

Resource Management

Optimize computational costs without sacrificing performance:

Auto-scaling: Scale infrastructure based on demand
Spot instances: Use discounted cloud instances for training
Model optimization: Quantization, pruning, distillation
Caching: Cache predictions for common inputs
Batch processing: Group predictions for efficiency

Team Organization and Culture

Cross-functional Collaboration

Successful MLOps requires collaboration between diverse teams:

Data Scientists: Model development and validation
ML Engineers: Pipeline development and optimization
DevOps Engineers: Infrastructure and deployment
Data Engineers: Data pipeline and quality
Product Managers: Business requirements and metrics

Establishing MLOps Culture

Shared responsibility: Everyone owns model success
Continuous learning: Regular training and knowledge sharing
Experimentation: Encourage controlled experiments
Documentation: Maintain comprehensive documentation
Feedback loops: Regular retrospectives and improvements

Common MLOps Challenges and Solutions

Technical Debt

Challenge: ML systems accumulate technical debt quickly.

Solution: Regular refactoring, code reviews, and automated testing.

Model Drift

Challenge: Model performance degrades over time.

Solution: Continuous monitoring, automated retraining, and champion-challenger frameworks.

Reproducibility

Challenge: Difficulty reproducing model results.

Solution: Version control for code, data, and environments; comprehensive experiment tracking.

🚀 Getting Started with MLOps

Start small with experiment tracking and basic CI/CD, then gradually add monitoring, automated testing, and advanced deployment strategies. Focus on solving real pain points rather than implementing everything at once.

Future of MLOps

The MLOps landscape continues to evolve with emerging trends:

AutoML Integration: Automated model selection and hyperparameter tuning
Federated Learning: Distributed training across multiple parties
Edge ML: Model deployment to edge devices and IoT
Explainable AI: Built-in interpretability and explainability tools
DataOps Integration: Closer integration between data and ML operations

Conclusion

MLOps is essential for organizations serious about scaling their machine learning initiatives. By implementing these best practices, teams can build reliable, maintainable, and scalable ML systems that deliver consistent business value.

Success in MLOps requires a combination of technical practices, cultural changes, and organizational commitment. Start with the fundamentals—experiment tracking, basic CI/CD, and monitoring—then gradually build more sophisticated capabilities as your team matures.

🎯 Need MLOps Implementation Support?

Worth being straight about where I sit on this: MLOps is not the deepest part of my experience. Two of the systems I've built genuinely ran in production for years; most of my work has been R&D, and heavy infrastructure is not what I'd sell you. If you want to think through pipeline design or what a sane ML lifecycle looks like for your team, get in touch — and if you need someone to own production infrastructure end to end, you want a specialist, not me.

I built a customer support agent as a team, not one bot

Sat, 03 Aug 2024 00:00:00 GMT

This is the first of eight posts where I build a complete customer support agent — a database, company policies, guardrails and tests, the parts a demo usually skips. I build it for a fictional UK transport company, UKConnect, which means every piece is inspectable and also that nothing here has faced a real customer.

This part is about the architecture: how the pieces fit, and why I didn't build it as one big agent.

Why bother — what's wrong with normal support

Start with the problem. Phone-and-email support has the same issues almost everywhere:

❌ Long wait times (average 8 minutes)
❌ Inconsistent service quality across different agents
❌ High operational costs (£40K+ per agent annually)
❌ Limited availability (business hours only)
❌ Scaling difficulties (linear cost increase)

How I built it: one coordinator, two specialists

Here's the shape. One coordinator, two specialists. Each agent does one job well, and the coordinator decides who handles what.

The three agents

The Master Agent — the coordinator

The Master Agent reads the question and decides what to do with it. It doesn't answer policy questions or touch the database itself — its job is to analyse the query, route it to the right specialist, and turn the result into a reply. It's a Gemini model with careful prompting, nothing more.

The Policy Agent — answers from documents

The Policy Agent handles anything that lives in the company's documentation: refund rules, terms and conditions, fares. It retrieves the relevant policy text first, then answers from that — grounded in what the company actually says, not guessed. Under the hood that's a vector database and RAG, which I cover properly in part 3.

The Ticket Agent — does the work

The Ticket Agent does the actual work on bookings — search, book, change, cancel — by calling tools against the database. This is the agent that changes things, so it's the one I keep on the tightest leash.

Three things this design gets right

Splitting the work this way isn't decoration. It buys three things.

Routing. The Master Agent reads intent and sends each request to the specialist that can actually handle it, so the customer never has to know there's more than one agent behind the reply.

Context. The customer's state and the conversation history are kept across every hand-off, so a multi-step request — cancel this, then tell me the refund — doesn't fall apart halfway through.

Focus. Each agent has one job. That makes it simpler to build, easier to test, and more accurate than one big agent trying to do everything at once.

What it actually buys you

An honesty note first: this is a project I built for a fictional company, so I can't put a real cost figure on it. I haven't run it for an actual support desk and measured the savings, and anyone who quotes you a tidy "60% cost reduction" off a demo is guessing. What I can talk about is what the architecture genuinely gives you.

The answers are consistent — the same question gets the same answer every time, which a room full of people can't promise. There are no wait times, it handles as many conversations at once as arrive, and it's available around the clock. And every interaction is logged, which a phone line never gives you, so you can see what customers actually ask and where the system struggles. Those are properties of the design, not numbers I made up.

The stack

Nothing exotic. The intelligence is Google's Gemini, with a vector database and embeddings for semantic search over the policy documents. The backend is Python, with SQLite for the operational data here (you'd want PostgreSQL if this ever ran for real) and FastAPI for the API. The agents are wired together with Google's ADK; data wrangling is pandas and NumPy, and the early vector search uses scikit-learn.

If you're weighing frameworks, I later rebuilt the whole thing on LangGraph — the ADK vs LangGraph comparison walks through what changed and why.

What it can actually do

In practice it handles four kinds of request: policy questions (refunds, terms, fares), booking operations (search, book, modify, cancel), general service (accounts, complaints, problems), and mixed requests that need a policy lookup and a database change in one go — those are the ones a single-purpose bot usually fails.

It also adapts how it talks: formal and detailed for a corporate client, casual for a younger audience, short for someone reading on a phone. Same facts, different register.

What a request actually looks like

Two examples make the routing concrete. A simple one stays with a single specialist:

A simple policy question

Customer: "What's your refund policy?"
↓
Master Agent: [Analyzes] → Identifies as policy-related query
↓
Policy Agent: [RAG Search] → Retrieves relevant refund policy sections
↓
Master Agent: [Synthesizes] → Delivers formatted, personalized response

A harder one needs both specialists, in order — the database does the cancellation, the Policy Agent works out the fee, and the Master Agent stitches it into one answer:

A booking change that needs both agents

Customer: "Cancel my booking UKC005 and tell me the refund amount"
↓
Master Agent: [Analyzes] → Identifies booking + policy requirements
↓
Ticket Agent: [Database] → Cancels booking, calculates base refund
↓
Policy Agent: [RAG] → Applies refund rules and applicable fees
↓
Master Agent: [Synthesizes] → "Booking cancelled, £67.50 refunded"

Why this design would hold up

Three reasons, all of them practical rather than theoretical. Because the agents are separate and stateless, I can scale the busy ones independently and run them across several servers without sharing memory. Because each agent owns one concern, I can update or fix one without taking the others down, and I can test each in isolation. And because adding a capability means adding an agent — not rewriting the existing ones — the system grows without a redesign. Moving it to a new industry is mostly new prompts and new tools, not new architecture.

How fast and how accurate

These are from my own testing on the project's scenarios, not a production deployment. Simple questions come back in under a second; multi-step operations that touch the database take one to three seconds, and the heaviest multi-agent flows two to five. Routing lands on the right agent the large majority of the time, and policy answers come back grounded in the right document. I'd trust it — but I'd measure it properly against real traffic before claiming anything firmer.

It's not just for transport

I built this for a transport company — rail and bus bookings, fares, refund rules. But the shape is industry-agnostic. Swap the policies and the tools and it's a healthcare desk doing appointments, patient queries and insurance; or e-commerce doing orders, returns and shipping; or hospitality doing reservations and guest services. The coordinator-plus-specialists pattern doesn't change. Only the documents and the database behind it do.

Next in the series

Next I build the data layer the agents run on: the database schema, the hybrid storage, and the RAG setup with the vector database and embeddings. That's where the Policy Agent gets something real to retrieve from.

That's the foundation. Everything in the rest of the series sits on top of it. The point of splitting the work across specialist agents isn't elegance — it's that each piece stays small enough to build, test, and trust on its own.

This is the kind of system I build at twentytwotensors. If you want one for your business, get in touch.

The data layer: a database and a vector store

Sat, 03 Aug 2024 00:00:00 GMT

In part 1 I laid out the architecture — one coordinator, two specialists. This part builds the data layer underneath it. The agents are only as good as what they can read and write, and a support system needs two very different kinds of storage.

Two kinds of data, two kinds of store

Bookings, customers, transactions — that's structured, relational data. You want exact answers, foreign keys, and transactions you can trust. SQLite in development, PostgreSQL in production.

Policies, FAQs, help articles — that's unstructured text, and customers never ask for it using the words it's written in. That needs semantic search, which is what a vector database is for. I cover the retrieval side properly in part 3; here I'm just standing both stores up.

┌─────────────────┐    ┌─────────────────┐
│ Relational DB   │    │ Vector Database │
│ (SQLite/PG)     │    │ (Embeddings)    │
├─────────────────┤    ├─────────────────┤
│ • Customers     │    │ • Policy Docs   │
│ • Bookings      │    │ • FAQ Content   │
│ • Transactions  │    │ • Knowledge Base│
│ • Train Data    │    │ • Help Articles │
└─────────────────┘    └─────────────────┘

Each store does the thing it's good at: the relational database for precise operations, the vector database for "find me the policy that answers this." The system reaches for whichever the question needs.

The relational schema

Four tables. Nothing fancy — the point is that they're clean and the agents can reason about them.

1. Customers Table

CREATE TABLE customers (
customer_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
address TEXT,
phone TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Design rationale: Simple customer profiles with essential contact information. The address field will be used for location intelligence in later tutorials.

2. Bookings Table

CREATE TABLE bookings (
booking_id TEXT PRIMARY KEY,
customer_id TEXT,
departure_station TEXT,
arrival_station TEXT,
departure_datetime TIMESTAMP,
ticket_type TEXT,
price DECIMAL(10,2),
status TEXT,
FOREIGN KEY (customer_id) REFERENCES customers(customer_id)
);

Key features:

Human-readable booking IDs (e.g., "UKC001")
Flexible status field for booking lifecycle management
Precise pricing with decimal support

3. Transactions Table

CREATE TABLE transactions (
transaction_id TEXT PRIMARY KEY,
booking_id TEXT,
amount DECIMAL(10,2),
transaction_type TEXT,
payment_method TEXT,
status TEXT,
processed_at TIMESTAMP,
FOREIGN KEY (booking_id) REFERENCES bookings(booking_id)
);

Supports: Payment processing, refund tracking, financial reporting

4. Fares Table

CREATE TABLE fares (
route_id INTEGER PRIMARY KEY,
departure_station TEXT,
arrival_station TEXT,
distance_miles INTEGER,
standard_price DECIMAL(10,2),
flexible_price DECIMAL(10,2)
);

Enables: Dynamic pricing, route planning, availability checking

Database Initialization

Setting it up is two commands — one to create the schema, one to fill it with sample data:

# Navigate to the project directory
cd production-ai-customer-support

# Create database schema
python -c "from utils.create_schema import create_database_schema; create_database_schema()"

# Populate with realistic sample data
python -c "from utils.populate_data import populate_data; populate_data()"

This creates a SQLite database with complete sample data including 15 realistic customers and 25+ varied booking scenarios.

Vector Database for RAG Implementation

The Challenge of Knowledge Retrieval

Traditional keyword search fails when customers ask questions like:

"Can I get my money back if I cancel?" (instead of "refund policy")
"What happens if my train is delayed?" (instead of "compensation terms")

Solution: Semantic search using vector embeddings that understand meaning, not just keywords.

Vector Database Architecture

Our RAG system follows this pipeline:

Policy Documents → Text Chunking → Embeddings → Vector Storage → Semantic Search

Implementation Details

1. Document Processing

We start with company policy documents stored in database/UKConnect_policy.txt:

# Document contains:
- Refund policies and procedures
- Terms & conditions
- Fare rules and regulations
- Compensation policies
- Special circumstances handling

2. Intelligent Chunking Strategy

# From utils/chunking_data.py
chunk_config = {
"chunk_size": 500-1000,  # characters per chunk
"overlap": 100,          # character overlap between chunks
"preserve_boundaries": True  # Don't break sentences/paragraphs
}

Why chunking matters:

Manageable size: Fits LLM context windows efficiently
Overlap: Ensures context continuity across chunk boundaries
Boundary preservation: Maintains semantic coherence

3. Embedding Generation

# From database/vector_db.py
import google.generativeai as genai

def generate_embeddings(text_chunks):
embeddings = []
for chunk in text_chunks:
result = genai.embed_content(
model="models/embedding-001",
content=chunk
)
embeddings.append(result['embedding'])
return embeddings

4. Vector Storage

For development, we use pickle storage. Production systems would use dedicated vector databases like Pinecone, Weaviate, or ChromaDB.

# Initialize vector database
python -c "from database.vector_db import setup_vector_database; setup_vector_database()"

Configuration Setup

Environment Variables

Create a .env file with your configuration:

# API Configuration
GOOGLE_API_KEY=your_google_api_key_here

# Database Configuration
DATABASE_URL=sqlite:///database/ukconnect_rail.db

# Vector Database
VECTOR_DB_PATH=database/vector_db.pkl

Project Structure

production-ai-customer-support/
├── database/
│   ├── ukconnect_rail.db          # SQLite operational database
│   ├── vector_db.pkl              # Vector embeddings storage
│   ├── UKConnect_policy.txt       # Source policy documents
│   ├── ukconnect_qa_pairs.json    # Pre-processed Q&A pairs
│   └── ukconnect_rag_chunks.json  # Processed text chunks
├── utils/
│   ├── create_schema.py           # Database schema creation
│   ├── populate_data.py           # Sample data generation
│   └── chunking_data.py           # Document processing

Sample Data Population

Realistic Customer Profiles

Our sample data includes diverse, realistic customer profiles:

# Example customer data
{
"customer_id": "CUS001",
"name": "James Thompson",
"email": "james.thompson@email.co.uk",
"address": "45 Baker Street, London, W1U 7EW",
"phone": "+44 20 7946 0958"
}

15 customers covering:

Different geographic locations (London, Manchester, Birmingham, Glasgow)
Various demographics and communication preferences
Mixed booking histories and support interactions

Complex Booking Scenarios

# Example booking data
{
"booking_id": "UKC001",
"customer_id": "CUS001",
"departure_station": "London Euston",
"arrival_station": "Manchester Piccadilly",
"departure_datetime": "2024-08-15 09:30:00",
"ticket_type": "Standard",
"price": 85.50,
"status": "Active"
}

25+ bookings including:

Various routes and pricing tiers
Different booking statuses (Active, Cancelled, Modified)
Edge cases for testing (same-day changes, refunds, etc.)

RAG System Deep Dive

The Retrieval Process

When a customer asks a question, our RAG system follows these steps:

Query Analysis: Extract semantic meaning from the customer's question
Vector Search: Find the most relevant policy documents using cosine similarity
Context Preparation: Combine retrieved information with query context
Response Generation: LLM creates a response using the retrieved knowledge

Example RAG Flow

# Customer query: "Can I cancel my ticket and get a refund?"

# 1. Generate query embedding
query_embedding = embed_content("Can I cancel my ticket and get a refund?")

# 2. Search vector database
relevant_chunks = vector_db.search(query_embedding, top_k=3)
# Returns: [
#   "UKConnect Refund Policy: Tickets can be cancelled up to...",
#   "Cancellation fees apply as follows: Standard tickets...",
#   "Refund processing typically takes 3-5 business days..."
# ]

# 3. Generate contextual response
response = llm.generate(
query="Can I cancel my ticket and get a refund?",
context=relevant_chunks,
customer_data=customer_profile
)

Performance Optimization Strategies

1. Chunk Overlap Management

overlap_config = {
"size": 100,  # characters
"purpose": "context_continuity"
}

Ensures important information isn't lost at chunk boundaries.

2. Relevance Filtering

similarity_threshold = 0.7  # Only return highly relevant results
filtered_results = [r for r in results if r.similarity > similarity_threshold]

3. Caching Strategy

# Cache frequent queries for faster response times
cache_config = {
"ttl": 3600,  # 1 hour cache lifetime
"max_size": 1000,  # Maximum cached queries
}

4. Batch Processing

# Process multiple documents efficiently
batch_size = 32
embeddings = process_embeddings_batch(text_chunks, batch_size)

System Testing and Validation

Database Health Verification

# Test database connectivity and data integrity
python -c "
from database.database import DatabaseManager
db = DatabaseManager()
print('Customers loaded:', len(db.get_all_customers()))
print('Bookings loaded:', len(db.get_all_bookings()))
print('Database health: OK' if db.health_check() else 'ERROR')
"

Vector Database Testing

# Verify vector search functionality
python -c "
from database.vector_db import VectorDatabase
vdb = VectorDatabase()
results = vdb.search('refund policy', top_k=3)
print('Vector search working:', len(results) > 0)
print('Sample result:', results[0][:100] + '...' if results else 'No results')
"

End-to-End Integration Test

# Run a complete system test with sample customer interaction
python run_test_scenarios.py --session 1

This executes a full customer support scenario, testing both database operations and RAG retrieval.

Monitoring and Maintenance

Database Health Monitoring

def check_database_health():
"""Monitor database performance and integrity"""
checks = {
"table_integrity": verify_table_schemas(),
"data_consistency": check_foreign_key_constraints(),
"query_performance": measure_average_query_time(),
"storage_usage": get_database_size_metrics()
}
return all(checks.values())

Vector Database Maintenance

def update_vector_database():
"""Maintain vector database accuracy and performance"""
tasks = [
reprocess_modified_documents(),
regenerate_outdated_embeddings(),
update_search_indices(),
cleanup_unused_vectors()
]
return execute_maintenance_tasks(tasks)

Production Deployment Considerations

Database Scaling

For production environments, consider:

PostgreSQL: Replace SQLite for concurrent access and advanced features
Connection pooling: Manage database connections efficiently
Read replicas: Distribute read queries for better performance
Backup strategy: Automated backups and point-in-time recovery

Vector Database Scaling

Production vector storage options:

Pinecone: Managed vector database with excellent performance
Weaviate: Open-source with GraphQL interface
ChromaDB: Simple, powerful embeddings database
Qdrant: High-performance vector search engine

Common Troubleshooting Issues

Issue: Vector Search Returns Poor Results

Solutions:

Verify embedding model consistency
Check chunk size and overlap settings
Adjust similarity thresholds
Review document preprocessing quality

Issue: Database Connection Timeouts

Solutions:

Implement connection pooling
Optimize query performance with indexes
Consider read/write splitting
Monitor concurrent connection limits

Issue: RAG Responses Lack Context

Solutions:

Increase chunk overlap
Include more context in retrieval
Improve document chunking boundaries
Enhance query preprocessing

Next in the series

The data layer is in place — precise operational data, and a knowledge base the system can search by meaning. Next I build the first agent that uses it: the Policy Agent, which retrieves from this vector store and answers from what it actually finds, not what it half-remembers.

Designing data layers that AI can reason over — not just store — is what I do at twentytwotensors. Get in touch.

The Policy Agent: RAG that answers from real documents

Sat, 03 Aug 2024 00:00:00 GMT

In part 2 I built the data layer — a relational database for bookings and customers, and a vector store for the policy documents. This part builds the agent that uses that vector store: the Policy Agent. It answers policy questions — refunds, terms, fares — and the whole point is that every answer is grounded in the company's actual documents, not the model's memory.

Why policy questions are hard

Policy is exactly where human support struggles, and where a grounded AI earns its place. People forget the details or answer inconsistently. The documents are long and change often. The awkward questions need two or three sections combined and read carefully. And for anything compliance-related, "roughly right" isn't good enough — you need the real wording, and ideally a citation back to it.

That's what RAG is for. Retrieval-augmented generation means the model retrieves the relevant policy text first and answers from that — it doesn't get to improvise. For policy, that's the difference between a system you can trust and one you can't.

How the Policy Agent works

The flow is straightforward: take the question, find the policy chunks that actually match it, hand those to the model as context, and have it answer from them — with a citation back to the source.

Customer Query
↓
Query Embedding (Vector Generation)
↓
Vector Similarity Search
↓
Document Chunk Retrieval (Top-K)
↓
Re-ranking & Filtering
↓
Context Assembly
↓
LLM Prompt Construction
↓
Response Generation
↓
Citation Addition
↓
Final Response

The critical insight for enterprise AI consulting is that each stage requires careful optimization. A poorly implemented RAG system suffers from irrelevant retrievals, context overflow, or hallucinated responses. Professional-grade implementation demands attention to chunking strategy, embedding model selection, retrieval algorithms, and prompt engineering.

Key architectural decisions:

Vector Database Choice: a managed store like Pinecone is what you'd reach for at scale; Chroma or Weaviate suit smaller deployments. The UKConnect build uses neither — one policy document fits in a pickled vector store, as Part 2 covered
Embedding Model: OpenAI's text-embedding-3-large (3072 dimensions) provides excellent semantic understanding for policy documents, though sentence-transformers offer cost-effective alternatives
Chunking Strategy: Semantic chunking with 500-token overlap preserves context across chunk boundaries, critical for policy documents with interconnected clauses
Re-ranking Pipeline: a cross-encoder re-ranker should recover relevant chunks that pure vector similarity ranks too low — worth reaching for when compliance accuracy matters, though I haven't measured the gain on a corpus this small

If you're picking a vector database for something real, I'd prioritise query latency and filtering capability over raw storage capacity. Policy systems tend to need filtering by document version, department, or regulatory jurisdiction during retrieval, and that requirement narrows the field faster than scale does.

Document Processing & Chunking Strategy

The foundation of any RAG implementation is document processing. Poor chunking destroys retrieval quality - chunks that are too large dilute semantic meaning, while chunks that are too small lose critical context. Enterprise policy documents present unique challenges: legal language spans multiple paragraphs, cross-references connect distant sections, and hierarchical structures (sections, subsections, clauses) must be preserved.

The chunking strategy uses semantic boundaries rather than arbitrary token counts:

import tiktoken
from typing import List, Dict
import re

class PolicyDocumentProcessor:
def __init__(self, chunk_size: int = 500, overlap: int = 100):
self.chunk_size = chunk_size
self.overlap = overlap
self.encoding = tiktoken.encoding_for_model("gpt-4")

def extract_hierarchy(self, text: str) -> Dict[str, str]:
"""Extract document structure metadata."""
hierarchy = {
'section': None,
'subsection': None,
'article': None
}

# Match section headers (e.g., "Section 3.1: Refund Policy")
section_match = re.search(
r'Section\\s+(\\d+(?:\\.\\d+)?):?\\s*([^\\n]+)',
text,
re.IGNORECASE
)
if section_match:
hierarchy['section'] = section_match.group(2).strip()

# Match article numbers for legal documents
article_match = re.search(
r'Article\\s+(\\d+)',
text,
re.IGNORECASE
)
if article_match:
hierarchy['article'] = article_match.group(1)

return hierarchy

def semantic_chunk(self, document: str, metadata: Dict) -> List[Dict]:
"""Chunk document at semantic boundaries with context preservation."""
chunks = []

# Split on double newlines (paragraph boundaries)
paragraphs = re.split(r'\\n\\n+', document)

current_chunk = []
current_tokens = 0
current_hierarchy = {}

for para in paragraphs:
para_tokens = len(self.encoding.encode(para))

# Extract hierarchy for this paragraph
para_hierarchy = self.extract_hierarchy(para)
if any(para_hierarchy.values()):
current_hierarchy.update(para_hierarchy)

# Check if adding this paragraph exceeds chunk size
if current_tokens + para_tokens > self.chunk_size and current_chunk:
# Create chunk with metadata
chunk_text = '\\n\\n'.join(current_chunk)
chunks.append({
'text': chunk_text,
'metadata': {
**metadata,
**current_hierarchy,
'token_count': current_tokens
}
})

# Start new chunk with overlap
# Keep last paragraph for context continuity
if len(current_chunk) > 1:
current_chunk = [current_chunk[-1], para]
current_tokens = len(self.encoding.encode(
'\\n\\n'.join(current_chunk)
))
else:
current_chunk = [para]
current_tokens = para_tokens
else:
current_chunk.append(para)
current_tokens += para_tokens

# Add final chunk
if current_chunk:
chunks.append({
'text': '\\n\\n'.join(current_chunk),
'metadata': {
**metadata,
**current_hierarchy,
'token_count': current_tokens
}
})

return chunks

def process_policy_document(
self,
filepath: str,
doc_metadata: Dict
) -> List[Dict]:
"""Process a complete policy document into searchable chunks."""
with open(filepath, 'r', encoding='utf-8') as f:
content = f.read()

# Remove excessive whitespace while preserving structure
content = re.sub(r'\\n{3,}', '\\n\\n', content)

# Chunk the document
chunks = self.semantic_chunk(content, doc_metadata)

# Add chunk IDs for citation
for idx, chunk in enumerate(chunks):
chunk['chunk_id'] = f"{doc_metadata['document_id']}_chunk_{idx}"

return chunks

This approach preserves semantic coherence while maintaining manageable chunk sizes. The overlap strategy ensures that concepts split across paragraphs remain retrievable. I haven't benchmarked this against fixed-size chunking on the UKConnect corpus — it's one policy document, which is too small for the comparison to mean anything.

Critical implementation considerations:

Token counting accuracy: Use the same tokenizer as your embedding model to avoid chunk size mismatches
Metadata enrichment: Capture document version, effective date, and regulatory references during processing
Structure preservation: Maintain hierarchical context (section numbers, article references) in metadata for filtering
Update handling: Implement version control to deprecate outdated policy chunks when documents are updated

For organizations with complex policy frameworks spanning hundreds of documents, professional AI consulting ensures chunking strategies align with document structure and retrieval requirements. We've seen enterprises waste months on RAG implementations that fail due to poor document processing fundamentals.

Vector Embedding Implementation

Vector embeddings transform text into high-dimensional numerical representations where semantic similarity corresponds to geometric proximity. This mathematical foundation enables semantic search - the ability to find relevant content even when query terms don't exactly match document language.

Scaled up, an embedding pipeline needs batch processing, caching, and error recovery. Here's the shape of one — note this sketch uses managed services rather than the Gemini embeddings and pickle store the UKConnect repo actually runs:

import openai
from typing import List, Dict
import numpy as np
from pinecone import Pinecone, ServerlessSpec
import hashlib
import json

class EmbeddingPipeline:
def __init__(
self,
openai_api_key: str,
pinecone_api_key: str,
index_name: str = "policy-embeddings"
):
self.openai_client = openai.OpenAI(api_key=openai_api_key)
self.pc = Pinecone(api_key=pinecone_api_key)
self.index_name = index_name
self.embedding_model = "text-embedding-3-large"
self.embedding_dimension = 3072

# Initialize or connect to Pinecone index
self._initialize_index()

def _initialize_index(self):
"""Create Pinecone index with optimal configuration."""
existing_indexes = [idx.name for idx in self.pc.list_indexes()]

if self.index_name not in existing_indexes:
self.pc.create_index(
name=self.index_name,
dimension=self.embedding_dimension,
metric='cosine',  # Cosine similarity for text embeddings
spec=ServerlessSpec(
cloud='aws',
region='us-east-1'  # Choose region near your application
)
)

self.index = self.pc.Index(self.index_name)

def generate_embeddings(
self,
texts: List[str],
batch_size: int = 100
) -> List[List[float]]:
"""Generate embeddings with batching and error handling."""
embeddings = []

for i in range(0, len(texts), batch_size):
batch = texts[i:i + batch_size]

try:
response = self.openai_client.embeddings.create(
model=self.embedding_model,
input=batch,
encoding_format="float"
)

batch_embeddings = [
item.embedding for item in response.data
]
embeddings.extend(batch_embeddings)

except Exception as e:
print(f"Embedding generation failed for batch {i}: {e}")
# Retry individual texts in failed batch
for text in batch:
try:
response = self.openai_client.embeddings.create(
model=self.embedding_model,
input=[text],
encoding_format="float"
)
embeddings.append(response.data[0].embedding)
except Exception as inner_e:
print(f"Failed to embed text: {inner_e}")
# Add zero vector as placeholder
embeddings.append([0.0] * self.embedding_dimension)

return embeddings

def upsert_chunks(self, chunks: List[Dict], namespace: str = "policies"):
"""Upload document chunks with embeddings to Pinecone."""
# Extract texts for embedding
texts = [chunk['text'] for chunk in chunks]

# Generate embeddings
embeddings = self.generate_embeddings(texts)

# Prepare vectors for Pinecone
vectors = []
for chunk, embedding in zip(chunks, embeddings):
# Create unique ID from content hash
content_hash = hashlib.md5(
chunk['text'].encode()
).hexdigest()[:8]

vector_id = chunk.get('chunk_id', content_hash)

# Prepare metadata (Pinecone has size limits)
metadata = {
'text': chunk['text'][:1000],  # Truncate for storage
'full_text_hash': content_hash,
**{k: v for k, v in chunk['metadata'].items()
if v is not None}
}

vectors.append({
'id': vector_id,
'values': embedding,
'metadata': metadata
})

# Upsert in batches
batch_size = 100
for i in range(0, len(vectors), batch_size):
batch = vectors[i:i + batch_size]
self.index.upsert(
vectors=batch,
namespace=namespace
)

def query_similar(
self,
query: str,
top_k: int = 5,
namespace: str = "policies",
filter_dict: Dict = None
) -> List[Dict]:
"""Find most similar document chunks to query."""
# Generate query embedding
query_embedding = self.generate_embeddings([query])[0]

# Query Pinecone
results = self.index.query(
vector=query_embedding,
top_k=top_k,
include_metadata=True,
namespace=namespace,
filter=filter_dict  # e.g., {'section': 'refund_policy'}
)

# Format results
matches = []
for match in results.matches:
matches.append({
'id': match.id,
'score': match.score,
'text': match.metadata.get('text', ''),
'metadata': {
k: v for k, v in match.metadata.items()
if k != 'text'
}
})

return matches

This sketch covers the features embedding generation needs before you'd trust it with real volume:

Batch processing: Reduces API calls and improves throughput for large document sets
Error recovery: Handles individual embedding failures without losing entire batches
Metadata management: Preserves document context while respecting Pinecone's metadata size limits
Content hashing: Enables deduplication and change detection for document updates
Namespace organization: Separates different policy domains (HR, legal, technical) for targeted retrieval

My working assumption — from the retrieval work I've done since, on clinical guidelines rather than rail policies — is that embedding quality matters more than retrieval algorithm sophistication. Modern embedding models give good zero-shot performance on domain content without fine-tuning. For organisations with highly specialised terminology, fine-tuning the embedding model on domain data is the next lever to reach for, though I can't put a number on the gain for your corpus and neither can anyone who hasn't measured it on your corpus.

Search & Retrieval Optimization

Vector similarity search provides a strong baseline, but enterprise AI systems require additional optimization layers. Raw vector search suffers from several limitations: it can't filter by document metadata efficiently, doesn't understand query intent, and treats all semantic similarity equally regardless of document importance.

The retrieval pipeline implements a three-stage process:

from typing import List, Dict, Optional
import cohere

class PolicyRetrieval:
def __init__(
self,
embedding_pipeline: EmbeddingPipeline,
cohere_api_key: str
):
self.embedding_pipeline = embedding_pipeline
self.cohere_client = cohere.Client(cohere_api_key)

def hybrid_search(
self,
query: str,
top_k: int = 20,
rerank_top_k: int = 5,
filter_metadata: Optional[Dict] = None
) -> List[Dict]:
"""Three-stage retrieval: vector search, re-ranking, filtering."""

# Stage 1: Vector similarity search with over-retrieval
# Retrieve more candidates than needed for re-ranking
vector_results = self.embedding_pipeline.query_similar(
query=query,
top_k=top_k,
filter_dict=filter_metadata
)

if not vector_results:
return []

# Stage 2: Re-ranking with cross-encoder model
# Cohere's re-ranker uses bidirectional attention
documents = [result['text'] for result in vector_results]

rerank_response = self.cohere_client.rerank(
model='rerank-english-v3.0',
query=query,
documents=documents,
top_n=rerank_top_k,
return_documents=True
)

# Map re-ranked results back to original metadata
reranked_results = []
for result in rerank_response.results:
original_result = vector_results[result.index]
reranked_results.append({
**original_result,
'rerank_score': result.relevance_score,
'original_rank': result.index
})

# Stage 3: Apply business logic filters
filtered_results = self._apply_business_filters(
reranked_results,
query
)

return filtered_results

def _apply_business_filters(
self,
results: List[Dict],
query: str
) -> List[Dict]:
"""Apply domain-specific filtering logic."""
filtered = []

for result in results:
metadata = result.get('metadata', {})

# Filter out deprecated policy versions
if metadata.get('status') == 'deprecated':
continue

# Ensure minimum relevance threshold
if result.get('rerank_score', 0) < 0.5:
continue

# Check document recency for time-sensitive policies
# (Implementation would check effective_date in metadata)

filtered.append(result)

return filtered

def explain_retrieval(self, query: str, results: List[Dict]) -> Dict:
"""Generate retrieval explanation for debugging and transparency."""
explanation = {
'query': query,
'num_results': len(results),
'top_scores': [r.get('rerank_score') for r in results[:3]],
'source_documents': [
{
'id': r['id'],
'section': r['metadata'].get('section', 'unknown'),
'score': r.get('rerank_score')
}
for r in results
]
}
return explanation

This hybrid approach significantly improves retrieval quality:

Over-retrieval strategy: Fetch 20 candidates for re-ranking to ensure relevant documents aren't missed by vector search alone
Cross-encoder re-ranking: Cohere's re-ranker evaluates query-document pairs jointly, capturing subtle semantic relationships that vector similarity misses
Business logic filters: Implement domain-specific rules (document freshness, deprecation status, regulatory requirements)
Transparency mechanisms: Provide retrieval explanations for debugging and compliance auditing

Hybrid retrieval should beat pure vector search on queries where an exact term matters — a policy code, a station name — because lexical matching catches what embeddings smooth over. I've not measured that here. What I'd say with more confidence is that retrieval optimisation is iterative: you need retrieval quality metrics and real user queries before any threshold you pick is more than a guess.

LLM Integration & Prompt Engineering

The final stage of our RAG pipeline combines retrieved documents with sophisticated prompt engineering to generate accurate, well-cited policy responses. This is where many enterprise AI implementations fail - poor prompts lead to hallucinations, missing citations, or overly generic responses that don't leverage the retrieved context effectively.

The prompt engineering strategy follows these principles:

from typing import List, Dict
import openai

class PolicyAgent:
def __init__(self, openai_api_key: str, retrieval_system: PolicyRetrieval):
self.client = openai.OpenAI(api_key=openai_api_key)
self.retrieval = retrieval_system
self.model = "gpt-4-turbo-preview"

def construct_prompt(
self,
query: str,
context_chunks: List[Dict]
) -> str:
"""Build sophisticated RAG prompt with grounding instructions."""

# Format retrieved context with citations
context_sections = []
for idx, chunk in enumerate(context_chunks, 1):
metadata = chunk['metadata']
section_ref = metadata.get('section', 'Unknown Section')

context_sections.append(
f"[Source {idx}: {section_ref}]\\n{chunk['text']}\\n"
)

context_text = "\\n".join(context_sections)

prompt = f"""You are a Policy Assistant for an enterprise customer support system. Your role is to provide accurate, helpful answers to customer policy questions based ONLY on the provided policy documents.

RETRIEVED POLICY CONTEXT:
{context_text}

CUSTOMER QUESTION:
{query}

INSTRUCTIONS:
1. Answer the question using ONLY information from the provided policy context
2. If the context doesn't contain enough information to answer fully, say so explicitly
3. Cite specific sources using [Source N] notation for all factual claims
4. Use clear, customer-friendly language while maintaining accuracy
5. If policies conflict or have exceptions, explain them clearly
6. Do not make assumptions or add information not present in the context
7. For time-sensitive policies, mention any effective dates if present

RESPONSE FORMAT:
- Start with a direct answer to the customer's question
- Provide supporting details with source citations
- End with any relevant caveats or additional information the customer should know

Your response:"""

return prompt

def generate_response(
self,
query: str,
temperature: float = 0.1,
max_tokens: int = 800
) -> Dict:
"""Generate policy response with full RAG pipeline."""

# Retrieve relevant policy chunks
retrieved_chunks = self.retrieval.hybrid_search(
query=query,
top_k=20,
rerank_top_k=5
)

if not retrieved_chunks:
return {
'response': "I apologize, but I couldn't find relevant policy information to answer your question. Please contact our support team for assistance.",
'sources': [],
'confidence': 'low'
}

# Construct prompt with retrieved context
prompt = self.construct_prompt(query, retrieved_chunks)

# Generate response with low temperature for consistency
completion = self.client.chat.completions.create(
model=self.model,
messages=[
{
"role": "system",
"content": "You are a precise, helpful policy assistant. Always ground your responses in the provided context and cite sources."
},
{
"role": "user",
"content": prompt
}
],
temperature=temperature,
max_tokens=max_tokens
)

response_text = completion.choices[0].message.content

# Extract source citations from response
import re
cited_sources = re.findall(r'\\[Source (\\d+)\\]', response_text)
unique_sources = list(set(cited_sources))

# Build source references
source_refs = []
for source_num in unique_sources:
idx = int(source_num) - 1
if idx < len(retrieved_chunks):
chunk = retrieved_chunks[idx]
source_refs.append({
'source_number': source_num,
'section': chunk['metadata'].get('section', 'Unknown'),
'document_id': chunk.get('id', 'Unknown'),
'relevance_score': chunk.get('rerank_score', 0)
})

return {
'response': response_text,
'sources': source_refs,
'confidence': self._assess_confidence(retrieved_chunks, cited_sources),
'retrieval_explanation': self.retrieval.explain_retrieval(
query,
retrieved_chunks
)
}

def _assess_confidence(
self,
retrieved_chunks: List[Dict],
cited_sources: List[str]
) -> str:
"""Heuristic confidence assessment based on retrieval quality."""
if not retrieved_chunks:
return 'low'

top_score = retrieved_chunks[0].get('rerank_score', 0)
num_citations = len(set(cited_sources))

if top_score > 0.8 and num_citations >= 2:
return 'high'
elif top_score > 0.6 and num_citations >= 1:
return 'medium'
else:
return 'low'

This implementation incorporates several advanced prompt engineering techniques:

Explicit grounding instructions: Prevents hallucination by constraining the LLM to retrieved context
Structured citation format: Makes source tracking programmatically verifiable for compliance
Customer-friendly language directive: Balances accuracy with accessibility
Caveat handling: Encourages the model to surface edge cases and exceptions
Low temperature setting: Reduces variability for consistent policy responses
Confidence scoring: Provides quality indicators for routing low-confidence queries to human agents

Prompt engineering is an empirical discipline: test prompts against diverse query types, measure citation accuracy, and iterate. In a later RAG project of mine, query expansion alone was worth 0.030–0.035 factual F1 on a measured benchmark — a real number, from a different corpus, offered here only to show the scale these adjustments move at rather than to promise you the same.

Testing & Quality Assurance

Enterprise RAG systems require rigorous testing to ensure accuracy, consistency, and compliance. Unlike traditional software where test cases are deterministic, LLM-based systems exhibit variability that demands statistical evaluation approaches.

The testing framework covers multiple dimensions:

import pytest
from typing import List, Dict
import json

class RAGTestSuite:
def __init__(self, policy_agent: PolicyAgent):
self.agent = policy_agent
self.test_cases = self._load_test_cases()

def _load_test_cases(self) -> List[Dict]:
"""Load curated test cases with ground truth answers."""
return [
{
'query': 'What is the refund window for cancelled bookings?',
'expected_sources': ['refund_policy'],
'expected_facts': ['48 hours', 'full refund'],
'category': 'refund'
},
{
'query': 'Can I change my booking date after confirmation?',
'expected_sources': ['modification_policy'],
'expected_facts': ['24 hours notice', 'one-time change'],
'category': 'modification'
},
# ... more test cases
]

def test_retrieval_precision(self):
"""Test that correct policy sections are retrieved."""
results = []

for test_case in self.test_cases:
retrieved = self.agent.retrieval.hybrid_search(
test_case['query'],
rerank_top_k=3
)

retrieved_sections = [
chunk['metadata'].get('section', '')
for chunk in retrieved
]

# Check if expected sources appear in top 3
precision = any(
expected in str(retrieved_sections)
for expected in test_case['expected_sources']
)

results.append({
'query': test_case['query'],
'precision': precision,
'retrieved_sections': retrieved_sections
})

precision_rate = sum(r['precision'] for r in results) / len(results)
assert precision_rate >= 0.90, f"Retrieval precision {precision_rate} below 90% threshold"

def test_factual_accuracy(self):
"""Test that generated responses contain expected facts."""
results = []

for test_case in self.test_cases:
response = self.agent.generate_response(test_case['query'])
response_text = response['response'].lower()

# Check if expected facts appear in response
facts_present = [
fact.lower() in response_text
for fact in test_case['expected_facts']
]

accuracy = sum(facts_present) / len(facts_present)

results.append({
'query': test_case['query'],
'accuracy': accuracy,
'missing_facts': [
fact for fact, present in
zip(test_case['expected_facts'], facts_present)
if not present
]
})

avg_accuracy = sum(r['accuracy'] for r in results) / len(results)
assert avg_accuracy >= 0.85, f"Factual accuracy {avg_accuracy} below 85% threshold"

def test_citation_completeness(self):
"""Test that responses include proper source citations."""
for test_case in self.test_cases:
response = self.agent.generate_response(test_case['query'])

assert len(response['sources']) > 0, \\
f"No sources cited for query: {test_case['query']}"

# Verify citations appear in response text
import re
citations = re.findall(r'\\[Source \\d+\\]', response['response'])
assert len(citations) > 0, \\
f"Sources provided but not cited in text: {test_case['query']}"

def test_consistency(self, iterations: int = 5):
"""Test response consistency across multiple generations."""
for test_case in self.test_cases[:3]:  # Test subset for speed
responses = []

for _ in range(iterations):
response = self.agent.generate_response(test_case['query'])
responses.append(response['response'])

# Check that key facts appear consistently
for expected_fact in test_case['expected_facts']:
appearances = sum(
expected_fact.lower() in resp.lower()
for resp in responses
)
consistency_rate = appearances / iterations

assert consistency_rate >= 0.8, \\
f"Fact '{expected_fact}' inconsistent across generations"

def test_hallucination_detection(self):
"""Test queries designed to trigger hallucinations."""
hallucination_queries = [
"What is the refund policy for Mars trips?",  # Non-existent product
"How much does a premium subscription cost?",  # If not in docs
]

for query in hallucination_queries:
response = self.agent.generate_response(query)

# Low confidence or explicit "I don't know" expected
assert (
response['confidence'] == 'low' or
"don't have" in response['response'].lower() or
"couldn't find" in response['response'].lower()
), f"Potential hallucination for query: {query}"

This testing framework validates several critical quality dimensions:

Retrieval Precision: are the correct policy sections found? 90%+ is the bar I'd set before trusting it — a target, not a result I measured
Factual Accuracy: do generated responses contain the expected factual elements? Again 85%+ is a threshold I'd want to clear, not a score I recorded
Citation Completeness: Validates that all responses include proper source attribution
Response Consistency: Tests that the same query produces semantically consistent answers across multiple generations
Hallucination Resistance: Evaluates system behavior on queries that should trigger "I don't know" responses

Anything that actually ships needs continuous evaluation against real user queries: user satisfaction ratings, escalation rates to human agents, citation accuracy audits. I'd expect demonstrable testing rigour to help with stakeholder buy-in and regulatory sign-off too — that one is reasoning from how these conversations tend to go, not something I've carried through an approval process myself.

Production Deployment Considerations

Moving from prototype to production requires addressing scalability, monitoring, and maintenance:

Latency optimization: cache frequent queries and pre-compute embeddings for common policy sections if you're chasing sub-2-second responses
Cost management: Monitor embedding and LLM API costs; implement query batching and result caching to reduce expenses at scale
Version control: Track policy document versions in vector database metadata to enable rollback and audit trails
Monitoring dashboards: Instrument retrieval precision, response latency, citation accuracy, and user satisfaction scores
Human-in-the-loop: Route low-confidence responses to human review; use feedback to improve retrieval and prompts

Most of the hard work in a real deployment is here, not in the core RAG. A system that's technically clever but slow, expensive, or unmonitored doesn't survive contact with production. This is the part I spend the most time on with clients.

Next in the series

The Policy Agent answers questions. Next I build its opposite number — the Ticket Agent, which doesn't just answer but acts: searching, booking, modifying and cancelling against the database.

Production-ready RAG — grounded, cited, fast enough, and monitored — is what I build at twentytwotensors. Get in touch if you need it.

The Ticket Agent: books, changes and refunds

Sat, 03 Aug 2024 00:00:00 GMT

In part 3 I built the Policy Agent — the one that answers from documents. This part is its opposite number: the Ticket Agent, the one that actually changes things. Search a route, make a booking, modify it, cancel it, work out a refund. Real database operations, driven by plain language.

The problem it solves

Booking is where support gets messy. A real request rarely maps to one clean database call. Someone wants to change a trip — that means finding their booking, checking what's available, cancelling one thing, creating another, and keeping it all straight across a few messages. They use informal names, like "the London train", that don't match a station code. And when a step fails halfway through, you can't leave the data in a broken state.

So the Ticket Agent isn't a chatbot with database access bolted on. It's an agent with a fixed set of tools — each one a specific, safe operation — and the judgement to pick the right one and recover when something fails.

How it works: tools, not free-form SQL

The agent doesn't write SQL. It calls tools I wrote and tested — named operations — and its only job is to choose which tool to call and with what arguments. That's the safe pattern: the model decides, the tools constrain what can actually happen to the data.

Customer request
→ Ticket Agent picks a tool:
• Booking search        — what's available
• Customer lookup       — their active bookings
• Book / modify / cancel — the transactional work
• Payment + refund calc  — logged every time
• Location mapping       — "London" → the right station
→ Response

Two of those are worth a note. Location mapping turns the informal place name a customer actually types into the station the database understands — I go into that in part 6. And state is kept across the conversation, so "actually, make it the later train" works without starting over.

The full implementation — every tool, the schema, the transaction handling — is in the open-source repo. I'd rather point you at code that runs than paste two hundred lines here.

Next in the series

Now I have both specialists — Policy and Ticket. Next I build the part that decides which one to use: the Master Agent, and the routing and context-handling that ties the whole thing together.

Building transactional AI that touches real systems — and doesn't break them — is what I do at twentytwotensors. Get in touch if you need it.

The Master Agent: routing and context

Sat, 03 Aug 2024 00:00:00 GMT

I've now got two specialists — the Policy Agent and the Ticket Agent. This part is the piece that decides which one to use: the Master Agent. It reads the customer's message, sends it to the right specialist, and turns whatever comes back into one clean reply.

The problem it solves

Coordinating agents is harder than it sounds, because the hard cases aren't the clean ones. The agent has to work out what a message is even asking for. It has to carry the conversation's context across a hand-off, so the specialist knows what came before. When a request needs both specialists — cancel this booking, and tell me the refund rule — it has to call them in order and merge the two answers. And when neither specialist can resolve something, it needs to escalate rather than make something up.

So the Master Agent's whole job is judgement: read intent, route, hold context, synthesise. It never touches the database or the documents itself.

How it works: route, hold context, synthesise

Customer message
→ Master Agent reads intent
→ routes to Policy Agent  (documents)
→ and/or Ticket Agent     (database)
→ holds conversation history + customer state
→ synthesises one reply

The two things that make this feel natural rather than robotic are context and synthesis. Context means the customer never repeats themselves — the history and their state travel with the conversation. Synthesis means that even when two specialists were involved, the customer gets a single, coherent answer, not two stitched-together fragments.

The full orchestration code is in the open-source repo. If you're weighing how explicit this routing should be, the ADK vs LangGraph comparison is the post to read — it's exactly the trade-off this layer turns on.

Next in the series

The system works, but it still assumes the customer says exactly what the database expects. Real people don't. Next I add location intelligence — turning "the London train" into the station the system actually knows.

Multi-agent orchestration that stays explainable in production is what I do at twentytwotensors. Get in touch if you need it built.

Location intelligence: 'the London train' into a station

Sat, 03 Aug 2024 00:00:00 GMT

Up to now the system has quietly assumed customers say exactly what the database expects. They don't. Someone books "the London train" — but there are several London stations, and the database wants a specific one. This part fixes that small, constant friction.

The problem it solves

People describe places the way people do, not the way a database does. They name a city when the system wants a station. They say "near Manchester". They don't know — and shouldn't have to know — the station code. If the agent just passes that straight through, the search fails or, worse, books the wrong thing.

There's also a quieter win hiding here. I already know where each customer lives, because it's on their profile. So I can default their departure station sensibly instead of asking every single time.

How it works: a mapping, not magic

This is deliberately not clever. It's a curated mapping from the places customers actually type to the stations the system knows, plus a default departure station derived from the customer's address on file. No IP geolocation, no guessing — I have the address, so I use it.

"the London train"  →  resolve against known stations  →  London Euston
customer address     →  default departure station        →  pre-filled, editable

The point is to do the obvious resolution before the Ticket Agent runs its tools, so the booking operation gets a real station, not a vague phrase. When it's genuinely ambiguous, the agent asks — it doesn't quietly pick one and hope.

The mapping and the lookup are in the open-source repo. It's a small file doing an unglamorous job, which is exactly why it works.

Next in the series

That's the whole system built — two specialists, a coordinator, and the location handling. Next I do the part most tutorials skip: testing it, and proving it actually behaves before it goes anywhere near a customer.

Building AI that meets customers where they are — plain language, real data — is what I do at twentytwotensors. Get in touch.

Testing an LLM system you can't unit-test

Sat, 03 Aug 2024 00:00:00 GMT

Most tutorials stop at "it works on my machine." This is the part that decides whether you'd actually let it talk to a customer. And testing an LLM system is genuinely different from testing normal software, so it's worth doing properly.

Why you can't test it the normal way

The usual move is to assert that output equals an expected string. That doesn't work here. Ask the agent the same question twice and you get two correct answers worded differently. A string-equality test would fail both. So the question isn't "did it return exactly this text?" — it's "was the answer correct, grounded in the real policy, and appropriate?"

Part of the system is deterministic, though, and I test that the boring, reliable way. The database tools — search, book, cancel, refund maths — have exact right answers. Those get ordinary unit tests. It's only the model's judgement that needs a different approach.

How I test it: scenarios, plus a judge

Two things do the heavy lifting.

Realistic scenarios. I wrote 15 end-to-end customer scenarios — the simple policy question, the booking change that needs both agents, the ambiguous location, the request that should be refused. They run the whole system, not one piece, because the bugs that matter live in the hand-offs between agents.

LLM-as-a-judge. Instead of matching strings, I have a separate model grade each response against a rubric: is it correct, is it grounded in the actual policy, is the tone right? That turns a fuzzy "seems fine" into a score I can track and fail a build on.

15 scenarios  →  run the full system  →  capture each response
↓
LLM judge grades vs. rubric
↓
correct? grounded? appropriate?  →  report

The scenarios and the evaluation harness are in the open-source repo. The judge isn't perfect — but it catches regressions a human reviewer would miss on the hundredth run, and it never gets bored.

Next in the series

The system is built and tested. The last part is the honest one: this is a project, not a client deployment — so what can I actually say it's worth, and what would I refuse to claim?

Evaluating LLM systems so you can trust them in production — that's a big part of what I do at twentytwotensors. Get in touch.

The honest part: it's a project, not a case study

Sat, 03 Aug 2024 00:00:00 GMT

For most of this series I wrote like a consultant pitching a business case. I want to drop that for the last part and be straight with you.

UKConnect isn't a real company. I made it up. I built this system to learn, and to show how the pieces actually fit together — not for a paying client. So there are no real customers, and there is no real ROI. I'm not going to invent one.

Why this post used to have numbers in it

The first version of this post quoted a "60% cost reduction" and a tidy per-agent saving. I put those there because I thought they'd get clicks, and they probably would. But a confident ROI figure for a fictional company is exactly the kind of thing that falls apart the moment someone asks a real question — and it's the opposite of why I write any of this. So I took it out. This is the honest version.

What I can actually stand behind

Plenty, just not a savings percentage. The system is real and it works. The architecture genuinely gives you consistency, round-the-clock availability, scaling without new headcount, and a logged record of every conversation — those are properties of the design, not numbers I made up. I also know what it costs to run, because I run it: a Gemini API bill and some cheap infrastructure, not a salaried team.

What I can't tell you is what it would save your support desk, because I've never pointed it at a real one. Anyone who quotes you a clean percentage off a demo is guessing.

How you'd find the real number

If you did deploy something like this, the honest way to learn its worth is to measure it, not to trust a slide. Take a baseline first — what a resolved contact costs you today. Then run the system on a slice of real traffic and watch cost per resolved contact, how many contacts it handles without a human, escalation rate, and customer satisfaction. The ROI is whatever those numbers say after a couple of months. It'll be specific to you, and it might be great — but it's earned, not assumed.

One thing I'd say with confidence: it won't take your team to zero. You still want people for the hard, sensitive, and angry cases, and someone watching the system. The realistic shape is "handle the bulk, escalate the rest."

The proof is the system, not a stat

So here's what I'll claim instead of a number: it's a complete, working, open-source build that you can read, run, and pull apart yourself. That's the actual evidence I can offer — more honest than any figure I could put on a fictional company, and more useful too.

That's the series

Over eight parts I've built a real customer support system — the architecture, the data layer, the Policy Agent, the Ticket Agent, the orchestration, the location handling, and the testing. It's all open source, and there's a LangGraph version too if you're choosing a framework.

If I had one piece of advice: start with a pilot, measure everything against a real baseline, and expand on proven value rather than a slide deck.

If you want a system like this built for your business — and rolled out without breaking your support team — that's what I do at twentytwotensors. Get in touch.