A tracker on a moving inspection line is not one model that “follows objects” frame to frame. It is a pipeline: a detector proposes objects in each frame, an association step links their identities across frames, and heavier per-frame work — like a segmentation mask — runs only where the budget allows. Treating it as a single black box is where most line-side tracking projects quietly go wrong.

The failure looks harmless in a demo. You wire up a detector, add a “tracking” flag, and watch bounding boxes follow a part down a video clip. It works. Then you put it on the real line, running at line speed on the target industrial computer, and the tracker starts stalling — identities swap between adjacent parts, boxes lag behind the belt, and the segmentation you added “for accuracy” pushes the whole thing past the frame interval. Nothing was broken. The math just never fit.

How does real-time video tracking actually work?

The pattern that holds up under real throughput is called detect-then-associate. It splits the job into two responsibilities that people tend to blur together.

First, a detector runs on the current frame and proposes where objects are — boxes, classes, confidences. On an inspection line this is usually a single-stage detector like a YOLO family model, chosen because real-time object detection on the production line needs to finish inside a tight budget rather than maximize a leaderboard score. The detector has no memory. Every frame is a fresh question: what is in view right now?

Second, an association step takes this frame’s detections and last frame’s tracks and decides which is which. Part #7 in frame 100 and a box in frame 101 that overlaps heavily, sits where a motion model predicted, and carries similar appearance features — those are almost certainly the same object, so they get the same identity. This is where “tracking” as a reader imagines it actually lives. The detector sees; the associator remembers.

Classic associators like SORT and DeepSORT make this concrete. SORT uses a Kalman filter to predict each track’s next position and the Hungarian algorithm to match predictions to detections by overlap. DeepSORT adds an appearance embedding so two parts that briefly overlap on the belt do not swap identities when they separate. The detail matters on a line because parts often look nearly identical, move at a known speed, and pass through occlusions from fixtures or robot arms. Motion prediction plus appearance is what keeps identities stable through those moments — the same machinery we describe in multi-object tracking on an inspection line.

The reason this framing matters is not academic. Once you see tracking as detect-plus-associate, the cost structure becomes visible, and the cost structure is what determines whether the system survives contact with the line.

Why is running detection and segmentation on every frame the wrong default?

The naive instinct is to run everything, everywhere, every frame: detect, then segment the whole frame, then track. It feels safe — more computation, more accuracy. In practice it spends your entire budget on frames where nothing needs that resolution.

Consider what a mask actually costs. A SAM-variant segmentation model produces pixel-accurate masks, which is genuinely useful for measuring a defect’s shape or area. But running a full-frame segmentation pass on every frame is expensive, and most of the frame is belt, fixtures, and empty space you already know contains no part. You are paying to segment background thirty times a second.

The expert move is to invert the flow. Let the tracker tell segmentation where and when to run. The detector and associator maintain identities cheaply on every frame; segmentation is invoked only on tracked regions that need it, and only at the cadence the defect logic requires — perhaps once per part rather than once per frame. This is exactly the role a lightweight variant plays, which is why teams reach for Mobile SAM’s lightweight Segment Anything rather than the full model on line-side hardware. In configurations we’ve tested, restricting segmentation to tracked regions rather than full frames recovers a substantial share of the per-frame headroom the naive approach burns (observed pattern across inspection-line engagements, not a published benchmark).

That recovered headroom is the whole point. It is what lets you hold a stable frame rate and still segment the parts that matter.

How does a per-frame latency budget decide whether tracking holds line speed?

Here is the piece that a single model’s benchmark number will never tell you. On a moving line, the operationally relevant cost is not the detector’s latency alone. It is the sum:

per-frame cost = detection + association + (segmentation, where invoked)

And that sum has to fit inside the frame interval. If your line camera runs at 30 FPS, the interval is roughly 33 milliseconds. Everything the pipeline does for one frame has to finish before the next frame arrives, or the pipeline falls behind and either drops frames or lags — and a lagging tracker is a tracker that loses identities.

Worked example: does a 30 FPS line survive the pipeline?

Assume a 30 FPS line (≈33 ms budget) on a target industrial GPU. The numbers below are illustrative placeholders to show the accounting, not measured device specs — you must measure your own stack.

Stage	Illustrative per-frame cost	Runs when
Detection (single-stage)	~12 ms	every frame
Association (Kalman + Hungarian + embedding)	~4 ms	every frame
Segmentation (full-frame SAM-variant)	~40 ms	every frame (naive)
Segmentation (tracked-region, lightweight)	~6 ms	once per tracked part

Run detect + associate + full-frame segmentation every frame and you are at roughly 56 ms — well past the 33 ms budget. The tracker stalls. Run detect + associate every frame (~16 ms) and invoke lightweight segmentation only on the frames where a tracked part needs a mask, and you stay comfortably inside budget while still catching the defect. Same models, same hardware — the difference is where the segmentation runs. This is a budgeting exercise, not a modeling one.

The discipline is simple to state and easy to skip: profile each stage on the target hardware, add them up under the worst-case frame (the frame where the most parts are in view and segmentation fires), and confirm the sum fits the interval with margin. Skip this accounting and the tracker looks fine in the lab and stalls on the line. We treat this latency read as a first-class part of scoping any moving-line computer vision inspection system, because it is the number that predicts production behavior — not any single model’s leaderboard result.

What metrics show whether a real-time tracker is production-ready?

A tracker that “looks like it works” and a tracker that is production-ready are separated by three measurements, taken at line speed on the target hardware.

• Sustained FPS at line speed. Not peak, not a burst on a short clip — the frame rate the pipeline holds continuously under real throughput, including the frames where segmentation fires. If sustained FPS drops below line speed, the tracker is falling behind regardless of how good the boxes look.
• Identity-switch rate. How often a tracked part’s identity swaps or is lost across frames. On a line where parts look alike and occlude each other, this is the number that tells you whether association is actually holding. High switch rate breaks any downstream logic that assumes a part keeps one identity — the same concern that motivates Re-ID for tracking parts across the line for SPC.
• Defect-catch rate under the segmentation policy. The rate at which real defects are caught when segmentation is applied only to tracked regions. This closes the loop: it confirms that spending less compute (segmenting selectively) did not cost you the accuracy the segmentation was there to provide.

Read together, these three tell you whether the latency budget held and whether it held without sacrificing the inspection’s job. A tracker can pass FPS and fail identity-switch; it can pass both and still miss defects if the segmentation policy is too aggressive. All three are the acceptance test.

FAQ

How should you think about real time video tracking in practice?

Real-time video tracking is a detect-then-associate pipeline, not a single model that follows objects. A detector proposes objects in each frame, an association step links those detections to identities from the previous frame, and heavier per-frame work like segmentation runs only where the budget allows. In practice it means the detector sees the current frame while the associator carries memory across frames.

How does the detect-then-associate pattern link object identities across consecutive frames on a moving line?

The associator takes this frame’s detections and last frame’s tracks and matches them using motion prediction and appearance. Classic methods like SORT use a Kalman filter to predict where each track should be and the Hungarian algorithm to match by overlap; DeepSORT adds an appearance embedding so parts that briefly overlap do not swap identities. On a line with known part speed and predictable occlusions, motion plus appearance is what keeps identities stable.

How does a per-frame latency budget determine whether tracking can hold line speed on target hardware?

The operationally relevant cost is detection plus association plus optional segmentation, summed per frame, and that sum must fit inside the frame interval — roughly 33 ms at 30 FPS. You profile each stage on the target hardware, add them under the worst-case frame, and confirm the total fits with margin. If it does not, the tracker stalls or lags, and a lagging tracker loses identities.

Where does a SAM-variant segmentation stage fit into a tracking pipeline without blowing the frame budget?

Segmentation runs only on tracked regions the tracker points it at, and only at the cadence the defect logic needs — often once per part rather than once per frame. Full-frame segmentation on every frame spends budget masking background you already know is empty. A lightweight SAM variant applied selectively recovers the per-frame headroom the naive approach burns.

Why is running detection and segmentation on every full frame the wrong default for real-time tracking?

It spends your entire budget on frames and regions that never needed that resolution, most of which are belt and fixtures containing no part. That pushes the per-frame sum past the frame interval and stalls the tracker. Letting the tracker decide where and when segmentation runs keeps the frame rate stable while still segmenting the parts that matter.

What metrics show whether a real-time tracker is production-ready — FPS, identity-switch rate, and defect-catch rate?

Sustained FPS at line speed shows whether the pipeline holds throughput continuously, not just in a burst. Identity-switch rate shows whether association is holding when parts look alike or occlude each other. Defect-catch rate under the selective-segmentation policy confirms that spending less compute did not cost the accuracy the segmentation was there to provide. All three, measured at line speed on target hardware, are the acceptance test.

The honest framing is that real-time tracking is a budgeting problem wearing a modeling costume. If you can name your frame interval, profile each stage on the hardware you will actually ship, and prove the worst-case sum fits with margin, the tracker will hold the line — the same latency-budgeted, instrumented discipline we bring to a Production CV Readiness Assessment for any moving-line inspection stage.