A Kaggle code competition gives you five submissions a day, and each one costs a full re-run over roughly 199 hidden volumes. That is not an iteration loop. It is a rationing system.
So the first thing I built was not a detector or a tracker. It was a copy of the scoreboard.
Why you can't just use an off-the-shelf metric
The obvious move is to grab an existing tracking metric and point it at your predictions. It does not work here, and the reason is the sparse ground truth from Part 1.
Only about 6% of the visible nuclei are labelled. A standard metric assumes the annotation is complete, so every cell you correctly detect that nobody labelled is counted against you as a false positive. Run that on this data and a genuinely good tracker scores terribly — not because it is wrong, but because the metric is measuring a question nobody asked.
Worse, the error is not a constant offset you could ignore. It scales with how many unlabelled cells you happen to find, which means it is largest exactly when your detector improves. A metric that punishes you for getting better is not a slow guide. It is an actively misleading one.
What the local scorer had to get right
Three things, all of them consequences of how the real metric is defined:
Score only against labelled cells. Match predicted nodes to ground-truth nodes by centroid within the physical radius, then evaluate edges only where both endpoints matched something real. Predictions in unlabelled regions are neither rewarded nor punished — they are simply out of scope.
Match in microns, not voxels. Because of the anisotropy, a radius expressed in voxels is a squashed ellipsoid in real space — far more permissive in z than in x. Every distance in the scorer is physical.
Count consecutive edges only. An edge that spans a gap is worth nothing. If the scorer credits it, the scorer will happily tell you that a bad idea is a good one.
Did it work?
The test of a local scorer is not whether it looks sensible. It is whether it predicts. So the first two real submissions were calibration, not attempts to win.
| Submission | Local score | Leaderboard | Gap |
|---|---|---|---|
| Classical detector + greedy linking | 0.066 | 0.147 | — |
| Blob detector + greedy linking | 0.7535 | 0.746 | 0.008 |
That second row is the one that mattered. A local number of 0.7535 landed at 0.746 on the leaderboard — within 0.008. From that point on I could try an idea, score it in minutes on my own machine, and know roughly what the board would say without spending a submission to find out.
Nearly everything in the rest of this series was found that way. The whole project runs on that one number being trustworthy.
The part I got wrong
Here is the honest caveat, and it is the seed of the final post.
"Within 0.008" tells you the scorer is well calibrated in absolute terms. It does not tell you the scorer can resolve a difference of 0.002 between two candidate configurations. Those are completely different claims, and I conflated them for about a week.
A scorer can be accurate on average and still be too noisy to rank two near-identical options — or biased in a specific direction that happens to reward a change the real metric does not care about. Mine turned out to be both, in different experiments. Building the scorer was the right first move. Believing it was precise because it was accurate was not.
The discipline I'd state now, having learned it the expensive way, is not "measure locally." It is: know your measurement's noise floor, and refuse to ship inside it.
The infrastructure underneath
Briefly, because it is the unglamorous half. Offline iteration only helps if running an experiment is cheap, so the scorer sits on top of a small amount of throwaway platform work: Terraform-provisioned GPU boxes on AWS (T4 and L4 spot lanes, with a CPU lane for the linker), S3 and EBS for persistence so a terminated spot instance costs nothing but time, an isolated MLflow instance for run tracking, and a self-terminating job pattern so a forgotten box does not quietly bill overnight.
There is also a wrinkle specific to code competitions: the notebook runs with no internet, so every dependency has to be packaged as offline wheels and installed from a Kaggle dataset. And the GPU you get is a lottery — a P100 has a different compute capability than a T4, which means a build that works in testing can fail on submission. That one costs you a day if you find it late.
None of this is interesting on its own. It exists so that the loop in the middle — change something, score it, know within minutes — actually spins.
Next
With a scoreboard I trusted, the question became where the score actually comes from. The answer was lopsided enough to be worth its own post: detection is the lever, and the biggest single improvement in the project was not my work.