With a labelling design in place (Part 3) and a DSP baseline whose real, held-out performance was now known rather than assumed (Part 2), the next question was whether a learned model could actually beat it — and if so, which one. This post is about the first of two ML tiers tested: a model that would run in the cloud, scoring uploaded clips after the device had already decided to send them.
Siren detection is secretly an image problem
The working observation underneath this whole stage is that a siren's structure is visually obvious on a spectrogram — a diagonal sweep with parallel harmonic stripes running alongside it, the exact shape the hand-tuned harmonic-sum score in Part 2 was built to detect algebraically. That suggests treating detection as image classification: render each clip as a mel-spectrogram, and let a vision model tell sirens from everything else.
That reframing opens up a genuine choice, and it's worth resisting the obvious answer. The default move is "fine-tune an ImageNet backbone" — ResNet-18, EfficientNet-B0, whatever's convenient. But there's a specific argument against assuming ImageNet transfer is the strongest starting point here: YAMNet, a model pretrained on Google's AudioSet, already has label classes for Siren, Emergency vehicle, Ambulance (siren), Police car (siren) and Fire engine (siren) baked into its training data. It has, in a meaningful sense, already seen the target concept — where ImageNet has never seen a spectrogram at all.
That also opens a baseline nothing else on the table offers: a zero-shot score straight from YAMNet's own siren-class posteriors, with no training at all. With only a few dozen labelled positives on hand at the time, an untrained model that already knows what a siren sounds like was worth putting on the table alongside anything trained from scratch.
So the choice of backbone was decided by measurement, not argument — three candidates, scored on the same folds, the same labels, the same evaluator:
| Candidate | Backbone | Trained? |
|---|---|---|
| ImageNet CNN | ResNet-18 / EfficientNet-B0 on mel image | Head only |
| AudioSet embedding | YAMNet, frozen | Head only |
| AudioSet zero-shot | YAMNet siren-class posteriors | No training |
The fold design, and a bug the evaluator refused to hide
One siren pass triggers several near-duplicate captures in quick succession, so a naive random train/test split leaks — adjacent clips of the same physical event end up on both sides of the split, and every candidate benefits from the leak equally, which keeps the ranking plausible while making the absolute numbers fiction. Folds were grouped by day instead, and capped at three: with only nine labelled days available at the time and between 1 and 27 positives per day, five folds wouldn't have held enough positives per fold to mean anything.
Two folds turned out not to be a safe fallback at all — the evaluation code was changed to refuse it outright after measuring what it actually did: the same DSP baseline scored 0.921 F1 at three folds and 0.480 at two. At two folds the training side collapses to a single fold, so the inner validation loop has nothing genuinely held out to score against, and the reported threshold ends up chosen on scores that are effectively all zero. That's not a worse model — it's a broken split wearing the costume of one.
The container budget, decided before the bake-off ran
The cloud model doesn't run in a vacuum — it has to fit inside the same lightweight Cloud Run container the device already depends on staying alive and responsive. That container had already had a dependency (scipy) deliberately stripped out to keep it light, specifically because the device's uploads depend on it never falling over. An embedding or vision model is heavier again, and cold-start latency compounds the problem: a slow-starting container is a device staring at a timeout.
The budget was fixed before any candidate was scored, not chosen afterward to justify a preferred answer: dependencies at or under 246 MB (three times the measured 82.1 MB baseline the shim already runs at) and latency at or under 500 ms per clip. Both were measured per candidate during the bake-off itself, not checked once on whichever model happened to win — a backbone that tops the accuracy table while blowing the container budget hasn't actually won anything, because it can never ship into the container the device is depending on.
The result: every learned model wins, and none of them ship
All four candidates — the DSP baseline included as a row in its own comparison, not exempted from it — were scored by one shared evaluator on the frozen day-grouped folds:
| Candidate | F1 mean ± sd | Deps (MB) | ms/clip | In budget? |
|---|---|---|---|---|
| dsp_v3 | 0.871 ± 0.033 | 82 | 2 | yes |
| yamnet_zeroshot | 0.934 ± 0.030 | 2084 | — | no |
| yamnet_embed | 0.965 ± 0.014 | 2230 | — | no |
| melimage (ResNet) | 0.959 ± 0.014 | 5005 | 73 | no |
Every learned model beat the DSP baseline on F1 — by a meaningful margin, 0.93 to 0.97 versus 0.87. The spectrogram-as-image premise wasn't wrong; it was right, measurably. And not one of them fits the container: 8 to 20 times over the 246 MB cap. Latency for the two YAMNet variants was never even measured, because the dependency gate alone disqualified them before latency could matter.
The DSP wins by disqualification, not by being better. This is the pre-registered rule doing exactly what it was built to do — refusing to let a model that scores well on a spreadsheet quietly become "the winner" when it can't actually run where it needs to run. The learned models' accuracy advantage is real, and it's not deployable into this particular deployment, and the difference between those two claims is the entire point of measuring the container cost as a gate rather than a footnote.
The transferable bit
"Best model" and "best model I can actually ship" are different questions, and only one of them is usually asked before code gets written. Measuring deployment cost during a bake-off, on every candidate, rather than once on whichever candidate wins on accuracy, is what turns a leaderboard exercise into a real engineering decision. The losers here aren't discarded footnotes — they're the actual answer to "does this deployment need ML," which turns out to be a more interesting question than "which model is most accurate."
One caveat, stated plainly
Because each candidate is trained and evaluated with its own best-fit architecture rather than a shared one, this bake-off answers "what do I actually get if I deploy each of these" — not "what does quantisation or platform cost in isolation, holding the model fixed." Those are different, equally legitimate questions, and conflating them would overstate what got measured here.
What's next
The cloud tier is settled: no learned model ships there, at least not yet — a smaller or more efficient embedding model could reopen this later, but nothing tested here clears the bar. That's not the end of the ML question, though. A completely different constraint applies on-device: no container to fit into, but a flash budget measured in kilobytes rather than megabytes, and an inference budget measured in milliseconds on a microcontroller rather than a GPU-backed container. Part 5 is that attempt — and this time, the outcome is different.