A perception team reports a MOTA figure and a mean IoU, staples them to the validation pack, and calls tracked segmentation validated. The number looks strong. The trouble is that it fuses two entirely different failure surfaces into one score a reviewer cannot take apart.

That fusion is the mistake worth catching early. Segmentation tracking — the task of producing consistent, identity-stable object masks across a video stream — degrades in two structurally distinct ways, and the safety argument that consumes its output needs to reason about each of them separately. Treat the two as one accuracy problem and you ship evidence that a functional-safety reviewer cannot map to hazard behaviour. Tracked segmentation is an input the safety argument consumes, not a safety property in itself.

What’s worth understanding about segmentation tracking first?

At the mechanical level, segmentation tracking layers two operations. First, a per-frame segmentation model — a Mask R-CNN head, a transformer-based mask decoder, or something in the Segment Anything Model family — assigns a pixel-level mask to each object instance in the current frame. Second, an association step links masks across frames so the same physical object carries a stable track identity over time. The association can be geometric (mask IoU or centroid distance between consecutive frames), appearance-based (a re-identification embedding), or motion-model-driven (a Kalman filter over the mask centroid). Most production stacks blend all three.

What “in practice” adds to that textbook description is the operational context. In an automotive pipeline the segmentation runs at the camera frame rate, the association budget is a few milliseconds, and the scene is adversarial: objects occlude each other, leave and re-enter the frame, and change appearance under lighting and motion blur. The per-frame model and the association logic fail under different triggers, and those triggers correlate with different driving situations. A pedestrian stepping behind a parked van stresses the association step, not the mask quality. Heavy rain on the lens degrades mask quality without necessarily breaking identity continuity.

The practical meaning, then, is that “segmentation tracking works” is never a single statement. It is a claim about two coupled subsystems whose weaknesses surface under different conditions.

What is the difference between per-frame segmentation error and cross-frame association error?

Per-frame segmentation error is a spatial error inside a single frame. The mask boundary is wrong, a thin object is missed, two adjacent instances are merged, or a shadow is segmented as an object. This is what mean IoU, boundary F-score, or panoptic quality measure. It is the failure surface your segmentation benchmark — COCO-style panoptic evaluation, or a domain dataset like the automotive segments of a driving corpus — actually characterises.

Cross-frame association error is a temporal error. The masks in each individual frame may be perfectly accurate, and the tracker still fails: it swaps two pedestrians’ identities as they cross (an ID switch), it drops a track for several frames and re-acquires it as a new object (fragmentation), or it loses a track entirely behind an occluder and never recovers it (track loss). None of these are visible in a per-frame IoU. They live only in the temporal dimension, and they are measured by association-oriented metrics — IDF1, ID switch count, fragmentation rate, and the association component of HOTA.

Each matters separately because each maps to a different hazard reasoning path. A boundary error on a distant object mask may be tolerable if the downstream planner only needs a coarse occupancy estimate. An ID switch between a pedestrian and a cyclist at close range can invalidate a trajectory prediction that the planner relies on — the object’s motion history now belongs to the wrong entity. The two errors are not interchangeable, and a system that is strong on one can be weak on the other. In configurations we have looked at, a segmentation model with excellent per-frame IoU paired with a naive IoU-only associator produces frequent ID switches under crossing motion precisely because the per-frame quality gives no temporal protection (observed pattern across perception-review engagements; not a benchmarked rate).

Why a single fused tracking score does not demonstrate safe temporal behaviour

MOTA and mean IoU are useful engineering summaries. They are not safety evidence on their own, for a specific structural reason: they compress the two failure surfaces into one scalar, and the compression is lossy in exactly the dimension the safety argument cares about.

MOTA, for instance, sums false positives, false negatives, and ID switches into a single normalised figure. Two systems can post an identical MOTA where one accrues its errors as scattered false negatives on distant, low-hazard objects and the other accrues them as ID switches on close-range vulnerable road users. The scalar cannot distinguish these, but the hazard analysis must. Mean IoU has the opposite blind spot — it says nothing about temporal identity at all, so a tracker that reboots every object’s identity each frame can still report a strong mean IoU.

This is why decision-grade validation separates the axes before it reports anything. As we argue in what each machine-learning performance metric actually proves, a metric’s value is bounded by what it can and cannot distinguish; a fused score that cannot separate spatial from temporal failure cannot, by construction, evidence temporal safety behaviour.

Which segmentation-tracking metrics feed the functional safety argument?

Not every metric a perception team tracks belongs in the safety argument, and the ones that do belong there for different reasons. The table below separates what a validation pack surfaces from what the safety case can actually consume.

Decision table: what each metric evidences

Metric	Failure surface	What it evidences	Safety-argument role
Mean IoU / panoptic quality	Per-frame spatial	Mask accuracy on a static frame	Feeds spatial-accuracy sub-claim; says nothing temporal
Boundary F-score	Per-frame spatial	Mask edge fidelity	Refines spatial claim for thin/small objects
IDF1	Cross-frame association	Identity-preservation quality over time	Directly feeds temporal-continuity sub-claim
ID switch count / rate	Cross-frame association	Frequency of identity swaps	Maps to trajectory-misattribution hazards
Fragmentation rate	Cross-frame association	Track drop-and-reacquire frequency	Maps to intermittent-detection hazards
Track loss under occlusion	Cross-frame association	Recovery behaviour after occluder	Maps to occlusion-blindness hazards
MOTA (fused)	Both, compressed	Aggregate tracking health	Engineering summary only — not a standalone safety sub-claim

The evidence classes here are observed-pattern for how the metrics degrade in operational scenes and benchmark for the metric definitions themselves; the point of the table is the mapping, not any single figure. The pack’s job is to state, per metric, which condition it was measured under and which hazard behaviour it informs — so the reviewer can trace each number to a specific line in the hazard analysis rather than to a fused headline.

How do these failure modes map to hazard behaviour the safety argument must address?

The mapping is the reason the decomposition matters at all. An ID switch is not just a metric regression — it is a mechanism by which the planner inherits the wrong motion history for a road user, which is a distinct hazard from simply failing to detect that user. Fragmentation produces intermittent detection: an object that flickers in and out of the track set forces the downstream fusion and tracking-at-the-object-level logic to decide whether to trust a freshly re-acquired track, and that decision has its own failure path. Track loss under occlusion is the occlusion-blindness hazard directly: the system stops reasoning about an object it can no longer see and may not recover it in time.

These are, precisely, the temporal reliability behaviours the system-level reliability discipline reasons about when perception evidence feeds the safety argument. The perception team’s obligation is to characterise how each behaviour degrades under the operational conditions the system will actually face — crossing pedestrians, dense urban occlusion, low-light re-identification — and to report the degradation curve, not a single benchmark point. A rate measured on a clean daytime highway sequence does not evidence behaviour in a rainy junction, and a pack that implies otherwise creates re-review rounds when the reviewer catches the scope gap.

How should a team characterise degradation without overclaiming?

The discipline is honest scope. State the condition the metric was measured under, state the metric class, and stop there. The pack should read as “under condition C, association error class E occurred at rate R, which informs hazard behaviour H” — a claim a reviewer can check — rather than “tracked segmentation achieves score S,” which a reviewer cannot map to anything.

This mirrors the reasoning we apply when comparing detectors: as in the discussion of what a validation pack must show when choosing between DETR and YOLO, the useful evidence is the conditional behaviour under named operating conditions, not a leaderboard rank. Decomposing the segmentation-tracking evidence this way reduces reviewer clarification cycles because the reviewer no longer has to ask how the fused number behaves under occlusion — the pack already answers it per failure surface.

FAQ

How does segmentation tracking work in practice?

It layers two operations: a per-frame segmentation model assigns pixel-level masks to each object instance, and an association step links masks across frames so each physical object keeps a stable identity over time. Association can be geometric, appearance-based, or motion-model-driven, and production stacks usually blend all three. In practice it means two coupled subsystems whose weaknesses surface under different driving conditions, so “it works” is never a single claim.

What is the difference between per-frame segmentation error and cross-frame association error, and why does each matter separately?

Per-frame error is spatial — wrong mask boundaries, missed or merged instances within one frame, measured by IoU or panoptic quality. Cross-frame error is temporal — ID switches, fragmentation, and track loss that occur even when every individual mask is accurate, measured by IDF1 and switch/fragmentation rates. They matter separately because each maps to a different hazard: a boundary error may be tolerable to a coarse planner, while an ID switch on a close-range road user corrupts the trajectory history the planner depends on.

Which segmentation-tracking metrics surface in a validation pack, and which of them feed the functional safety argument?

Per-frame metrics (mean IoU, boundary F-score) feed the spatial-accuracy sub-claim; association metrics (IDF1, ID switch rate, fragmentation rate, track loss under occlusion) feed the temporal-continuity sub-claims. MOTA and mean IoU are engineering summaries, not standalone safety sub-claims. The pack’s role is to state, per metric, the condition it was measured under and the hazard behaviour it informs, so each number traces to a line in the hazard analysis.

Why does a single fused tracking score not, on its own, demonstrate safe temporal behaviour?

Because MOTA compresses false positives, false negatives, and ID switches into one scalar, two systems can post identical scores while one accrues errors on distant low-hazard objects and the other on close-range vulnerable road users — a distinction the hazard analysis must make but the scalar erases. Mean IoU has the reverse blind spot and says nothing about temporal identity. A fused score that cannot separate spatial from temporal failure cannot, by construction, evidence temporal safety behaviour.

How do segmentation-tracking failure modes map to hazard behaviour the safety argument must address?

An ID switch is the mechanism by which the planner inherits the wrong motion history for a road user — distinct from a plain missed detection. Fragmentation produces intermittent detection that forces downstream logic to decide whether to trust a re-acquired track. Track loss under occlusion is the occlusion-blindness hazard directly, where the system stops reasoning about an object it can no longer see and may fail to recover it in time.

How should a perception team characterise segmentation-tracking degradation under operational conditions so the evidence supports the safety case without claiming to be it?

Report degradation curves per failure surface under named operating conditions — crossing pedestrians, dense occlusion, low-light re-identification — rather than a single benchmark point. Frame each claim as “under condition C, error class E occurred at rate R, informing hazard behaviour H,” which a reviewer can check. A rate from a clean daytime sequence does not evidence behaviour in a rainy junction, so the scope must be stated explicitly.

What scope claims about safety should a team avoid making from segmentation-tracking metrics alone?

Avoid any claim that the tracked-segmentation output is itself safe or that a fused score demonstrates safe temporal behaviour. The metrics are inputs the safety argument consumes, not safety properties; they evidence specific sub-claims under specific conditions and nothing broader. Never let a single figure imply behaviour under conditions it was not measured against.

The sharper question for any perception team is not “what is our tracking score?” but “which segmentation-tracking behaviours does our validation pack actually evidence, and under which conditions — and which does it silently leave unaddressed?” That is the line between a pack a reviewer can consume and one that generates another round of clarifications.