Every prior post in this series produced a number computed against pre-soak, offline labels — real, honestly measured, but static. This post is about what happened when three detectors ran live, at the same window, on the same clips, for real — and about a decision to end the test 13 days earlier than planned, made deliberately and reported plainly rather than absorbed quietly into the write-up.
Three arms, and the rule that governs all of them: tag, never gate
The trial ran three detectors against every incoming clip simultaneously: the DSP heuristic from Part 2 (unchanged, frozen), the on-chip CNN from Part 5, and — as a benchmark rather than a deployment candidate, since Part 4 already ruled the cloud tier out on cost — a cloud-scored version for comparison purposes only. None of the three ever decided whether a clip got uploaded or discarded. All three scored every clip; nothing was ever suppressed by disagreement. That distinction matters: a design where one arm's score could silently veto another arm's data would make every subsequent comparison unfair by construction.
What gets frozen, and why touching it mid-trial would be worse than a bad result
At the moment the trial started (T0), several things were locked and stayed locked for the entire run:
- Model weights, all arms. No mid-flight retraining, however tempting a bad week might make it look.
- Operating thresholds, all arms. Chosen out-of-fold, before T0, exactly as described in Part 5 — never re-picked after seeing how the soak was going. Re-tuning a threshold after seeing results is precisely the failure that put the DSP detector's original 90/90 claim at 0.921 in-sample and 0.812 held-out in Part 2.
- The trigger's own sensitivity margin. A known issue — the trigger runs too hot overnight, flooding on low-value clips — was left deliberately unfixed. Changing it mid-soak would change the input distribution partway through and make before/after numbers incomparable. The fix is queued for after the trial; the finding itself gets reported as a result, not silently acted on.
Because everything meaningful was frozen at T0, every label collected during the soak is genuinely held-out for every arm — which is the property that makes the eventual verdict mean anything at all.
The pre-registered exit rule, written before any data came in
If any arm reaches ≥90% precision at ≥90% recall — estimated with the inverse-probability-weighted method from Part 3, at that arm's frozen threshold, with the 90% lower confidence bound on recall at or above 85% — that arm ships and the trial ends. If none do, the trial still ends, written up as a null result: hand-designed and learned approaches both plateau on this problem.
Three distinct outcomes were defined up front, deliberately kept from collapsing into two: an arm can clear the bar, fail it, or be not measurable at this sample size — a point estimate that passes while its confidence interval doesn't quite. That third outcome is the easiest one to misreport as a pass, and naming it in advance was the only defence against doing that later under the pull of a result that looks almost good enough.
The decision to stop at 17 of 30 days
The trial was pre-registered for a 30-day window. It ran for 17. That's a real deviation from the plan, not a rounding error, and it's recorded as one rather than quietly writing the log as if 30 days had always been optional.
The justification: the daily trend — chip clearly ahead, DSP's recall gap structurally present — had held identically on every single day since the trial began, with no wobble, no crossing point, nothing that suggested more time would change the ranking. Stopping early was a judgement call about diminishing information, made and confirmed explicitly rather than assumed. The honest cost is smaller data than a full 30 days would have produced — fewer total labels, and a narrower confidence interval than the pre-registration was designed to deliver. That's stated plainly rather than smoothed over, because the whole point of pre-registering an exit date in the first place was to stop a trial from picking a convenient stopping point after seeing how the numbers were trending — and ending 13 days short is exactly the kind of thing that discipline exists to catch.
The final numbers
Over all post-T0 labels — 1,594 total, 183 confirmed sirens — at each arm's T0-frozen threshold:
| Arm | Threshold | Precision | Recall | Detail |
|---|---|---|---|---|
| DSP | 0.40 | 94.7% | 68.9% | 126/183 caught, 57 missed, 7 false alarms |
| Chip (on-device) | 0.65 | 98.2% | 90.7% | 166/183 caught, 17 missed, 3 false alarms |
Verdict: the chip model clears the pre-registered bar (P98.2 / R90.7). The DSP does not — its recall sits roughly 22 points short, at 68.9%. That gap wasn't a one-off bad week; it was present, in roughly the same 20–30-point range, on every single day of the soak from the first to the last. This settles, with real live data rather than an offline replay, the question this whole project opened with back in Part 2: does this actually need machine learning? Yes.
Retraining, and an honest negative that got reported instead of buried
With the soak concluded, the obvious next move was retraining the chip model on the newly collected post-T0 labels, folded in alongside the original pre-T0 training set. The first attempt held out two full days (60 sirens) as a fresh test set — standard hygiene, don't fold new labels straight into training and declare victory without checking against something the retrained model has never seen.
The result was, honestly, inconclusive — and in the specific way that a smaller test set produces rather than a real regression. The retrained model scored worse on that held-out pair of days than the original, frozen model: 93.5% precision versus 98.3%, at matching recall. Both comfortably cleared 90/90; nothing was actually broken. But "retraining made it better" — the assumption going in — simply wasn't true on this particular measurement, and that got surfaced as a genuine open question rather than re-run quietly until a more flattering number turned up.
The fix wasn't to pick a better-looking result. It was to change the measurement itself: with only 60 positive examples in a single held-out slice, the comparison was too noisy to trust a one-shot delta between two very similar models. Cross-validation — train fresh models on rotating subsets, average the result — answers a different, more reliable question: not "is this exact retrained model better than that exact frozen model" but "does this modelling approach perform well across many splits of the available data." The trade is explicit: CV validates the method, but the model that actually ships is trained on all the data with no held-out test of the specific final artefact.
Retrained on the full 1,898-label corpus (six times the original 308-label training set the T0 freeze had used), evaluated with honest 4-fold day-grouped cross-validation — a fresh model trained per fold, scored only on days that fold never saw during training:
| Fold | n test | Precision | Recall | F1 |
|---|---|---|---|---|
| 0 | 449 (71 pos) | 95.9% | 98.6% | 0.972 |
| 1 | 497 (69 pos) | 93.2% | 98.6% | 0.958 |
| 2 | 572 (70 pos) | 93.0% | 94.3% | 0.936 |
| 3 | 380 (69 pos) | 98.4% | 89.9% | 0.939 |
| mean | 95.1% | 95.3% | 0.951 ± 0.015 |
Every fold individually clears 90/90 on six times the labelled data the original model had. The final deployed model was trained on 100% of the corpus — no held-out set for the artefact that actually ships, with the cross-validation above standing in as the honest out-of-sample proxy for what a self-reported accuracy figure can't claim on its own.
Deploying it, with a trap the firmware now catches automatically
Quantised to int8 — 9.6 KB, 46.6% of the flash budget from Part 5 — and reflashed to the device. A verification script confirmed the on-device frontend and the training path score identically on real audio: zero disagreement across 15 spot-checked clips.
One easy-to-miss trap: swapping in a retrained model changes its int8 quantisation scale and zero-point, and the firmware had those values hard-coded from the original model. Forgetting to update them wouldn't have crashed anything — it would have quietly fed the new model inputs calibrated for the old one, scoring garbage while looking like it was working. The firmware now asserts these values against the model's own tensor metadata at boot, so a forgotten update fails loudly and immediately rather than shipping a device that silently mis-scores every clip.
The pipeline, simplified now that the question is answered
With a verdict in hand, the three-arm comparison infrastructure — the disagreement/stratified labelling queue, the paired scoring, the coverage correction — stopped earning its keep, and came out the same day the model shipped:
- The upload gate moved onto the device itself. Clips only leave the window if the chip model scores them above its frozen threshold. Everything the trigger detects is still timed and counted locally; it just doesn't all travel to the cloud anymore. If the chip model somehow fails to initialise, the gate fails open rather than closed — a broken model uploads everything rather than silently uploading nothing, so a failure is loud and visible instead of a quiet, permanent blackout nobody would notice.
- The corpus moved to a short-retention live bucket plus a cold archive. Audio used to be kept indefinitely for labelling and retraining. Now the live bucket holds 48 hours as a working buffer, and everything durable moves to a colder, cheaper archive tier — a directional shift from "keep everything, just in case" to "keep what's needed, cheaply."
- The dashboard shrank to a single number. A daily siren count, because every clip that reaches the bucket has already passed the device's own gate — there's nothing left to compute. The labelling UI, the disagreement queue, the coverage badges — all of it existed to support a comparison that's now finished.
What this series was actually about
Six parts, and the pattern underneath all of them is the same one, applied three times: don't assume ML is needed before proving it (Part 2); don't trust a comparison until the labels behind it are honest (Part 3); don't trust an accuracy number until you've checked whether it can actually ship (Part 4); and don't trust a modelling decision until it's tested where it will really run, live, with a rule for stopping written down before you know how it'll turn out (Part 5, Part 6). Every one of those checks reversed a decision that looked settled without it — the 90/90 claim in Part 2, the "best" cloud model in Part 4, the "retraining always helps" assumption in this post.
The chip model shipped. The DSP detector, hand-tuned and fast to build, still does real work as the baseline every learned approach had to clear, and as a documented cautionary tale about what an in-sample number is worth on its own. Neither of those things is a footnote — they're the actual result.