A part on a moving line does not appear once. It enters the field of view, travels across it, and leaves — showing up in dozens of consecutive frames. The naive inspection pipeline treats each of those frames as an independent event: run the detector, run segmentation, make a defect call, repeat. That approach works in a demo and quietly falls apart in production, because it has no concept of which physical part it is looking at. Multi-object tracking exists to fix exactly that gap. It is the layer that carries a part’s identity — and its mask association — across frames, so a defect gets attributed to one physical part rather than double-counted, smeared, or lost. The distinction that trips teams up is subtle but decisive: tracking is a data-association problem layered on top of detection and segmentation, not a substitute for either. You still need a detector to find candidate parts and a segmentation model to isolate them precisely. Tracking answers a different question — “is this the same part I saw one frame ago?” — and getting that answer right is what lets an expensive, higher-precision segmentation variant sit behind a real-time line without re-segmenting every single frame. How does multi-object tracking work in practice? At its core, a tracker maintains a set of tracks, each representing one physical object moving through the scene. For every new frame, the detector produces a set of fresh detections. The tracker’s job is to decide which of those detections belong to which existing tracks, which represent new objects entering the frame, and which existing tracks have exited. That matching step is the data association. Two ingredients drive the match. The first is motion prediction — a track carries a predicted position for where the object should be in the next frame, classically via a Kalman filter that models constant velocity. On a conveyor moving parts at a fixed line speed, motion is close to deterministic, which is why simple motion models work remarkably well in this setting. The second is appearance similarity — an embedding or feature descriptor of the object’s visual content, used to disambiguate when two parts are close together or when a part is briefly occluded. Trackers such as SORT rely almost entirely on motion; DeepSORT and ByteTrack add appearance and better handling of low-confidence detections to reduce identity errors. The association itself is usually solved as a bipartite matching problem — the Hungarian algorithm assigns detections to tracks by minimizing a combined cost of position mismatch and appearance distance. This is a well-understood, cheap computation compared with running a segmentation network. That asymmetry is the whole point. Why not just re-run detection and segmentation on every frame? The honest answer is: you can, if your latency budget allows it and you do not care about identity. But two things usually make that impractical. First, segmentation is expensive. A high-quality mask from a Segment Anything–class model is far heavier than a bounding-box detection. Even lightweight variants — the ones we cover in our explainer on Mobile SAM as a lightweight Segment Anything for on-line inspection — trade accuracy for speed precisely because full segmentation on every frame at line rate is often infeasible. If you can segment on keyframes only and carry the result forward with tracking, you free up throughput headroom for a slower, more accurate variant when it matters. Second, and more fundamentally, re-segmenting every frame still gives you no identity. You would produce a fresh mask each frame with no guarantee it corresponds to the same physical part as the previous frame’s mask. Per-part defect attribution — the thing your quality system actually needs — is an identity problem, and identity is what tracking supplies. This is the divergence point where teams stop treating tracking as an optimization and start treating it as a correctness requirement. It is a distinct concern from the frame-level detection accuracy we discuss in how an image detection model works in industrial inspection; a model can be accurate per frame and still corrupt your part counts without a tracker. What is an ID switch, and how does it corrupt per-part defect attribution? An ID switch is when the tracker mistakenly assigns the identity of one physical part to a different one — track 7’s history gets attached to what is actually a new part, or two parts swap identities as they pass each other. On an inspection line, the consequences are concrete and expensive. Suppose part A has a scratch detected on frame 40, and part B is clean. If an ID switch occurs around frame 55, the scratch evidence accumulated under one identity can migrate to the wrong physical part. Now your system either rejects a good part (false reject) or passes a defective one (false accept). Worse, because line-side counting often aggregates per-track, a single ID switch can double-count one part and drop another — corrupting yield statistics that feed downstream statistical process control. If you are feeding those counts into SPC, read our companion piece on Re-ID for tracking parts across the line for SPC, which covers the re-identification angle that keeps identity stable across gaps and camera hand-offs. ID-switch rate — the count of identity errors normalized over the number of tracked parts crossing the field of view — is therefore the single most decision-relevant tracking metric on a line. It is an observed-pattern in our experience that ID-switch rate correlates more directly with defect-attribution errors than any raw detection metric; a detector with excellent per-frame mAP can still produce unusable line-level attribution if association is weak. For the detection-side metrics that sit beneath this, our explainer on what mAP@50 means for defect detection covers the frame-level side of the story. How keyframe segmentation plus tracking lets you use a slower, more accurate model Here is the architecture the naive approach never reaches. You do not need a mask on every frame. You need a mask once per part, associated correctly, plus the tracker maintaining that association through the frames in between. The pattern: Detect on every frame (or every few frames) with a fast detector to keep tracks alive and predict motion. Segment on keyframes — the first solid detection of each part, or a re-segment when appearance or pose changes materially. Carry the mask association forward via the tracker, so the keyframe mask is attributed to the same physical part across the rest of its transit. Because segmentation now runs on a small fraction of frames, the per-keyframe compute budget expands dramatically. That headroom is what lets you swap a fast, lower-fidelity segmenter for a slower, higher-precision Segment Anything variant on keyframes — the mask quality that matters for a borderline defect call — without blowing the real-time budget. The relationship between the mask model and the pipeline is covered further in our explainer on instance segmentation models for manufacturing inspection. Tracking is the mechanism that makes a variable-latency segmentation model viable behind a fixed-rate line. Decision surface: per-frame segmentation vs. keyframe-plus-tracking Dimension Segment every frame Keyframe segmentation + tracking Segmentation compute Full model × every frame Full model × keyframes only Headroom for a higher-precision SAM variant Little to none at line rate Substantial — freed by skipped frames Per-part identity None (frame-independent masks) Maintained by the tracker Double-count / lost-part risk High — no identity to dedupe against Low if ID-switch rate is controlled Dominant failure mode Latency overrun, no attribution ID switches under occlusion / clutter What to measure FPS, mask accuracy ID-switch rate, keyframe skip fraction, FPS headroom The right-hand column is not always the answer — if parts are sparse, well-separated, and segmentation is cheap enough to run every frame, tracking adds complexity you may not need. The decision hinges on line speed, part density, and how expensive your target mask quality is. That trade-off is exactly the kind of profiling the readiness assessment below is built to do. What metrics tell you whether tracking is working on your line? Three numbers, watched together, tell you almost everything: ID-switch rate across the field of view — the correctness metric. Rising ID switches mean identity is degrading and defect attribution is becoming unreliable. Keyframe skip fraction — the fraction of frames on which you can skip full segmentation while preserving identity. This is the efficiency dividend tracking buys you. Throughput headroom (FPS) freed by that skip fraction — the budget you can reinvest into a higher-precision segmentation variant on keyframes. Standard tracking benchmarks report MOTA and IDF1 (benchmark-class metrics, as defined by the MOT Challenge evaluation protocol), and they are worth understanding, but on a production line the operationally relevant figures are the three above. They map directly to money: false reject/accept rates and yield-statistics integrity. If you want the broader treatment of the moving-object side of this, our article on real-time object tracking for line-side inspection and the deeper walkthrough in tracking multiple objects on a manufacturing line cover complementary angles — the former on real-time constraints, the latter on the multi-object association mechanics end to end. Tracking is one component of the larger computer vision inspection systems we build for production lines, where the detector, segmenter, and tracker have to be tuned together against a single latency budget. FAQ How does multi-object tracking actually work? A tracker maintains a set of tracks, one per physical object, and for each new frame decides which fresh detections belong to which existing tracks. It combines motion prediction (often a Kalman filter, which works well on a fixed-speed conveyor) with appearance similarity, then solves the assignment as a bipartite matching problem. In practice it means each physical part carries a stable identity across all the frames it appears in. How does tracking relate to the detector-plus-SAM segmentation pipeline on an inspection line? Tracking sits on top of detection and segmentation, not in place of them. The detector finds candidate parts, the segmenter isolates them precisely, and the tracker answers “is this the same part as last frame?” — carrying each part’s identity and mask association forward so segmentation does not have to re-run on every frame. Why not just re-run detection and segmentation on every frame instead of tracking? You can, but segmentation is expensive and running it at line rate on every frame is often infeasible, and — more fundamentally — re-segmenting each frame still gives you no identity. Per-part defect attribution is an identity problem, and only a tracker supplies the continuity that links masks across frames to the same physical part. What is an ID switch, and how does it corrupt per-part defect attribution? An ID switch is when the tracker assigns one physical part’s identity to a different part, or swaps identities as two parts pass each other. It can migrate defect evidence to the wrong part, causing false rejects or accepts, and it can double-count one part while dropping another — corrupting the yield statistics that feed statistical process control. How does carrying identity across frames let you use a slower, higher-precision SAM variant on keyframes only? If the tracker preserves identity between segmentations, you only need a mask once per part rather than on every frame. Segmenting on keyframes frees most of the per-frame compute budget, and that headroom lets you swap in a slower, higher-precision Segment Anything variant for the keyframe masks that decide borderline defect calls. What throughput and accuracy metrics tell you whether tracking is working on your line? Watch three numbers together: ID-switch rate across the field of view (the correctness metric), the fraction of frames you can skip full segmentation on while preserving identity, and the throughput headroom (FPS) that skip fraction frees. These map directly to false reject/accept rates and yield-statistics integrity — the figures your quality system actually cares about. Tracking is where an inspection pipeline stops being a per-frame classifier and becomes a system that knows what it is counting. If your line is at the point where double-counting and ID switches are eroding attribution accuracy, the association layer is the thing to profile against your latency budget — the failure class our Production CV Readiness Assessment is built to surface, by tuning segmentation cadence against how fast parts actually move through the frame.
