A version number tells you which training run produced a checkpoint. It tells you nothing about whether the model is safe to ship. That gap is where perception teams lose the thread on versioning — and where long-tail regressions slip into production unnoticed.

Walk into most automotive perception teams and versioning looks like a filename convention. detector_v3.pt, then detector_v4.pt, then a -final suffix nobody trusts. The version is tagged to a training run — new data went in, weights came out, increment the counter. Then someone asks the only question that matters at release time: is v4 better or worse than v3, and in which conditions? A bare version number cannot answer that. And so the team re-runs the headline mAP, sees it ticked up two points, and ships. Three weeks later a heavy-rain, low-sun corner case that v3 handled fine starts dropping pedestrians. The version number was never the problem. The problem was what the version number failed to carry.

What machine learning model versioning actually binds

Versioning in practice is not a label on a checkpoint. It is the mechanism that makes robustness comparable across releases. A useful version binds a set of weights to the scenario-class evidence it was validated against — the weather, lighting, edge classes, and sensor-mounting variance it actually held accuracy across. Strip that binding away and you have a checkpoint with a name, not a version you can defend.

The distinction is easiest to see through a contrast. Software versioning inherited from Git conditioned a whole generation of engineers to treat a version as a pointer to a code state, reproducible by construction: the same commit compiles to the same binary. Machine learning breaks that assumption in two places. First, the artifact is a function of data, hyperparameters, random seeds, and the training stack — reproduce all of those and you might reproduce the weights. Second, and more importantly, the thing you actually care about is not the weights but the behaviour, and behaviour is only observable through validation. So a perception model version has to point at two things at once: the inputs that produced it (data snapshot, config, code, seed) and the evidence that characterises it (per-scenario-class accuracy against a defined validation distribution).

Tools like DVC, MLflow, and Weights & Biases have made the first half routine. You can pin a data hash, log the training config, and reproduce a run. That is necessary and it is not sufficient. The reproducibility layer answers “how was this built.” It does not answer “is this safe to ship relative to the last thing we shipped,” and in our experience with production perception teams, that second question is where releases actually get stuck.

Why a version number on a checkpoint is not enough to defend a release

Consider what a release reviewer is being asked to accept. They are not being asked whether v4 exists, or whether it trained cleanly. They are being asked to sign off that v4 is at least as safe as the last defensible release across the conditions the vehicle will encounter. A single headline metric collapses that question into a scalar and hides exactly the information a reviewer needs.

Aggregate mAP is an average over a validation set. If that set is dominated by clear-daylight, well-lit highway frames — as most are, because those are the cheap frames to collect — then a two-point mAP gain can coexist with a five-point drop on night-time occluded pedestrians. The average moved the right way. The safety case moved the wrong way. This is the mechanism behind most post-release perception surprises we see: the regression was always present in the evidence, but the versioning discipline threw the evidence away and kept only the average. We unpack the trap of trusting a single aggregate figure in what each machine learning model performance metric actually proves, which is the companion to this piece on the metrics side.

The correct framing: a version’s defensibility lives in its scenario-class decomposition, not its headline. The number that matters is the accuracy delta per scenario class between the incoming version and the last shipped release. Positive across every class you care about, you have a clean improvement. Negative on even one safety-relevant class, you have a regression you must either justify or block — regardless of what the aggregate did.

How do you tie a model version to its scenario-class robustness evidence?

The binding is a record, and the record has to be part of the version — not a report that lives in someone’s inbox next to it. A defensible version record contains the training inputs, the validation distribution it was scored against, the per-scenario-class metrics, and the deltas against the prior release. Anything less and the next reviewer is re-deriving the safety case from scratch.

A version record a release reviewer can accept

Record field	What it must contain	Why the reviewer needs it
Build provenance	Data snapshot hash, training config, code commit, seed	Answers “how was this built” and makes the run reproducible
Validation distribution	The scenario-class taxonomy and frame counts per class it was scored on	Prevents an average from hiding a sparse but safety-critical class
Per-class metrics	Accuracy / recall per scenario class, not just aggregate	The behaviour characterisation the release actually rests on
Baseline reference	The version ID of the last defensible release	Deltas are meaningless without a fixed comparison point
Per-class delta	Signed change per scenario class vs. baseline	The single view that tells a reviewer improve-or-regress
Accepted-evidence pointer	Link to the evidence pack the prior reviewer accepted	Lets each release build on the last instead of re-litigating it

The scenario-class taxonomy in row two is the load-bearing part, and it is upstream of versioning. If you have not defined the classes — the specific weather bands, lighting conditions, edge object types, and sensor-mounting variance you require accuracy across — then there is nothing for the version to bind to. Getting that specification right is its own discipline; we cover it in writing a robustness specification for an automotive perception model. Versioning consumes that specification. It does not replace it.

How do you compare two versions to detect long-tail regressions?

Version-to-version comparison is where the whole discipline pays for itself. Once each version carries per-scenario-class evidence scored against the same validation distribution, comparing two releases is a subtraction, not an investigation. You lay the incoming version’s per-class metrics next to the baseline’s, compute the signed delta per class, and read off exactly where the model improved and where it degraded.

Long-tail regressions are precisely the ones a headline hides, because the long tail is, by definition, sparse in any aggregate. A scenario class that is 0.4% of your validation frames — say, low-sun glare with a partially occluded cyclist — contributes almost nothing to mean accuracy but everything to a specific fatality mode. Scored per class, a drop there is a red flag the reviewer sees immediately. Averaged away, it is invisible until the field. The comparison surface below is the operational form of this.

Per-scenario-class release gate (worked example)

Assume a detector where the last defensible release is v3 and the incoming candidate is v4. The numbers here are illustrative, to show the shape of the comparison, not measured results:

Scenario class	v3 recall	v4 recall	Delta	Verdict
Clear day, highway	0.94	0.96	+0.02	Improved
Night, urban, lit	0.88	0.90	+0.02	Improved
Heavy rain, dusk	0.81	0.83	+0.02	Improved
Low-sun glare, occluded VRU	0.79	0.71	−0.08	Regression — block or justify
Aggregate mAP	0.87	0.89	+0.02	Misleading

The aggregate says ship. The per-class view says the opposite: v4 traded a broad, cheap gain for a narrow, expensive loss on the exact class where a miss is a vulnerable-road-user collision. This is an observed-pattern we see repeatedly across perception engagements — not a benchmarked rate, but a recurring shape. The versioning discipline is what turns that shape from a post-mortem finding into a pre-release gate.

Versioning against the production driving distribution and sensor-mounting variance

A version is only as good as the validation distribution it was scored against, and that distribution has to track the fleet, not the lab. Two things make this harder than it sounds. The production driving distribution drifts — the vehicle encounters conditions the validation set under-represents, and a version validated against last year’s distribution is defending against last year’s world. Monitoring the live distribution and feeding drift back into the scenario-class taxonomy is what keeps versioning honest; we treat the monitoring side in depth in machine learning monitoring for provenance-preserving compliance.

Sensor-mounting variance is the subtler one. The same model behaves differently across a fleet because the cameras are not mounted identically — small differences in extrinsics change the effective input distribution per vehicle. A version validated on one mounting geometry is not automatically defensible on another. Practically, this means the version record’s validation distribution has to span the mounting variance the fleet actually exhibits, and a version that only ever saw one rig’s geometry carries a quiet caveat the reviewer should know about. The failure modes this produces — the ones that survive clean benchmarks and only appear in the field — are worth understanding directly, and we catalogue them in the perception failure modes that survive benchmarks.

When does a new version justify a release?

A new version justifies a release when its versioned evidence shows it improved rather than regressed on every scenario class the safety case depends on — or when a per-class regression is explicitly accepted with a documented rationale. “The mAP went up” is not a justification. “Per-class deltas are non-negative across all safety-relevant classes, and the one flat class was already at ceiling” is. The whole point of binding evidence to the version is that the release decision becomes reviewable by someone who was not in the room for training.

This is also the connection to the broader safety-evidence picture. Versioning does not exist to make model management tidy; it exists so that each release stands on the accepted evidence of the last one, and the audit trail compounds instead of resetting. How perception models meet automotive safety demands, from ASIL evidence down to the validation pack, is the wider frame this sits inside — we develop it in how perception models meet safety evidence, and versioning is the release-comparison layer that makes that evidence portable across releases.

If you are building the monitoring and evidence layer that makes this comparison possible, our work on the Production AI Monitoring Harness and our broader computer vision engineering practice is where the version-bound evidence pack becomes an operational artifact rather than a spreadsheet.

FAQ

What’s worth understanding about machine learning model versioning first?

In practice, a version binds a set of model weights to two things: the inputs that produced them (data snapshot, config, code commit, seed) and the evidence that characterises their behaviour (per-scenario-class accuracy against a defined validation distribution). It is the mechanism that makes robustness comparable across releases, not a filename on a checkpoint. Tools like DVC and MLflow handle the reproducibility half; the evidence-binding half is what makes a version defensible.

Why is a version number on a checkpoint not enough to defend a perception model release?

A bare version number tells a reviewer that a checkpoint exists and trained cleanly, but not whether it is at least as safe as the last defensible release across real conditions. A single headline metric collapses that question into a scalar and hides the per-class information a reviewer actually needs — a two-point aggregate gain can coexist with a five-point drop on a safety-critical class. The defensibility lives in the scenario-class decomposition, not the number.

How do you tie a model version to the scenario-class robustness evidence it was validated against?

You make the evidence part of the version record, not a report that lives next to it. The record contains build provenance, the validation distribution and its scenario-class taxonomy, per-class metrics, the baseline version ID, per-class deltas against that baseline, and a pointer to the evidence pack the prior reviewer accepted. The scenario-class taxonomy is upstream of versioning — if the classes are not defined, there is nothing for the version to bind to.

How do you compare two versions to detect long-tail regressions before they reach production?

Once both versions carry per-scenario-class metrics scored against the same validation distribution, comparison is a subtraction: lay the incoming metrics next to the baseline and read the signed delta per class. Long-tail regressions are exactly the ones a headline hides, because a sparse class contributes almost nothing to a mean but everything to a specific failure mode. Scored per class, a drop there is an immediate red flag rather than a field surprise.

What should a version record contain so a release reviewer can accept it?

Build provenance (data hash, config, code commit, seed), the validation distribution with its scenario-class taxonomy and per-class frame counts, per-class accuracy metrics, the baseline release’s version ID, the signed per-class delta against that baseline, and a pointer to the evidence the prior reviewer accepted. The last two turn each release into an increment on an accepted safety case rather than a fresh litigation of a headline score.

How does versioning relate to the production driving distribution and sensor-mounting variance across a fleet?

A version is only as strong as the validation distribution it was scored against, and that distribution has to track the fleet rather than the lab. The production driving distribution drifts over time, so a version validated against last year’s data is defending last year’s world. Sensor-mounting variance means the same model behaves differently across vehicles, so the version record’s validation distribution should span the mounting geometries the fleet actually exhibits.

When does a new version justify a release, and what versioned evidence demonstrates it improved rather than regressed?

A new version justifies a release when its per-class deltas are non-negative across every scenario class the safety case depends on — or when a specific regression is explicitly accepted with a documented rationale. “The mAP went up” is not a justification; “per-class deltas are non-negative across all safety-relevant classes” is. The versioned evidence that demonstrates it is the per-scenario-class delta table against the last defensible release, not the aggregate score.

Versioning is not the interesting engineering problem — it is the boring discipline that keeps the interesting problems from reaching the road. The question worth carrying forward is narrower than “which version is best”: for the one scenario class where a miss is a collision, can you show, release over release, that the number went the right way? If your versioning cannot answer that in one subtraction, it is still a filename convention.