A tracker that “follows objects across frames” sounds like one component. Treat it that way and you inherit a debugging problem you cannot decompose: when an object’s ID flips mid-scene, you have no way to say whether the detector missed a frame, the association logic mishandled an occlusion, or the pipeline simply fell behind its real-time budget and dropped a frame. The single most useful thing you can do with real-time object tracking is stop treating it as a monolith. It is three stages — detection, association (including re-identification), and state estimation — and each one is independently testable, independently replaceable, and owns a distinct class of failure. That framing is not academic hygiene. It is the difference between chasing a flaky tracker for three days and pointing at a metric that names the guilty stage in an afternoon. Why “tracking” is not a single operation The intuitive mental model is that a tracker watches the video and keeps its eye on things. That is the wrong abstraction for a production system. What actually happens, every frame, is a pipeline: A detector runs inference on the current frame and emits a set of bounding boxes with class labels and confidence scores. An association step matches those fresh detections to the set of tracks it already maintains — deciding which new box is the same object as an existing one. A state estimator (commonly a Kalman filter or a learned motion model) predicts where each track should be, smooths noisy detections, and carries a track forward through frames where the detector produced nothing usable. Detection answers what is in this frame. Association answers which of these are the same objects I saw before. State estimation answers where should this object be even when I can’t see it clearly right now. These are genuinely different questions, and the reason a detector alone cannot follow objects across frames is that a detector has no memory — it re-derives the scene from scratch on every frame, with no notion that box #3 this frame is the same forklift as box #1 last frame. Identity is manufactured by the association and state-estimation stages, not by detection. We cover the detection side of that boundary in more depth in how object recognition models feed a tracker, and the association handoff itself in how an object tracker works: detection-to-track association in practice. The classic algorithmic families make this separation concrete. SORT pairs a plain IOU-based association with a Kalman filter and nothing else. DeepSORT adds a learned appearance embedding to the association step so it can re-identify an object after an occlusion. ByteTrack keeps the low-confidence detections that most pipelines throw away and uses them in a second association pass. Notice what varies across those three: the detector is interchangeable, and what actually differentiates them is the association strategy. That is the tell that association is its own stage with its own design space. How do the stages map onto a modular CV architecture? Once you accept the three-stage decomposition, the architecture question becomes concrete rather than philosophical. Each stage is a component with a defined input and output contract, and you can instrument the seam between them. Stage Input Output Owns this failure Primary metric Detection Raw frame Boxes + class + confidence Missed / spurious objects Recall, mAP, per-frame inference latency Association / Re-ID Current boxes + existing tracks Box-to-track assignments ID switches, mismatches after occlusion ID-switch rate, IDF1 State estimation Assigned detections over time Smoothed track positions, gap-filling Track fragmentation, drift during gaps Track continuity, fragmentation count The value of the table is that it tells you where to look. A rising ID-switch rate with stable detection recall is an association problem — the detector is finding the objects, the matcher is losing track of which is which. Detection recall collapsing under load with unchanged association logic is an inference problem, usually a latency-induced frame drop. Tracks that drift smoothly and then snap back are a state-estimation tuning problem, not a detection or association problem at all. Without the decomposition, all three present identically as “the tracking is flaky,” which is why monolithic trackers cost days to debug. This is the same modularity principle we apply across pipeline design — the pipeline is the architecture decision, and the tracker is a facet of it, not a black box you bolt on. What is the difference between detection and tracking? This is the question that trips up most teams first, so it is worth being blunt. Detection is stateless. Run a detector on frame 100 and frame 101 and it will happily give you two independent sets of boxes with no claim whatsoever that they describe the same objects. Tracking is the machinery that imposes identity and continuity on that stream of independent detections. A detector alone cannot follow objects across frames for a structural reason, not a quality reason. Even a perfect detector — 100% recall, zero false positives, per NVIDIA’s published TensorRT throughput figures running comfortably above 30 FPS — produces no track identities. It has no persistent state, so it literally has nowhere to store the fact that “this is object 7 again.” Improving the detector never fixes an identity problem; it just gives the association stage cleaner input. Teams that respond to ID switches by retraining the detector are optimizing the wrong stage, and the decomposition is what makes that visible. The related question of running detection fast enough to feed a tracker at all is covered in real-time object detection and what throughput really costs. How do trackers handle occlusion and ID switches? Occlusion is where the stage ownership pays for itself. When a person walks behind a pillar and reappears three seconds later, three things can happen, and each belongs to a different stage. The detector stops emitting boxes while the person is hidden. That is correct detector behaviour — the object genuinely is not visible. The state estimator must carry the track forward through the gap, predicting motion so the track does not die the instant the object disappears. Tune the maximum coasting time too short and tracks fragment; too long and you accumulate ghost tracks. This is a state-estimation parameter, owned by the Kalman-style predictor. When the person reappears, the association step must recognise them as the same track rather than spawning a new ID. Pure motion-based association (SORT) will usually fail here because the predicted position has drifted. Appearance-based re-identification (DeepSORT’s learned embedding) is what recovers the identity. So re-identification is an association-stage capability. An ID switch is therefore almost never a single-cause event, but it is always attributable once the stages are observable. If detection recall held through the occlusion window and the ID still switched, the association embedding failed. If detection dropped out and the track died before reappearance, the state estimator’s coasting horizon was too short. Naming which stage owns the failure is the entire point — and it is exactly what an A2 pipeline assessment checks: whether detection, association, and state estimation are independently testable and observable, or whether they are fused into one component that can only fail opaquely. What keeps tracking inside a real-time latency budget? Real-time is a budget, not a feeling. At 30 FPS you have roughly 33 milliseconds per frame for the entire pipeline — decode, detection inference, association, state update, and any downstream consumers. The dominant cost is almost always detection inference; association and state estimation are typically single-digit milliseconds on modern hardware. Miss the budget and the failure is insidious: the pipeline falls behind, frames queue or drop, the detector effectively sees a lower frame rate, motion between processed frames grows, and association degrades — presenting as ID switches that look like an association bug but are really a latency bug upstream. This is the divergence point the whole decomposition exists to catch. A monolithic tracker degrades silently under load; a modular one lets you attribute the miss. A stage-by-stage latency and continuity checklist Use this before declaring a tracking pipeline production-ready: Measure detection inference latency in isolation, at production resolution and batch size, on the target hardware. If this alone exceeds the frame budget, no amount of association tuning helps. Consider FP16 or INT8 quantization via TensorRT, or a lighter detector. Log the end-to-end per-frame latency and its p99, not just the mean. Real-time budgets are broken by tail latency, not average latency. Track ID-switch rate and IDF1 separately from detection recall so you can tell an association regression from a detection regression (observed pattern across the CV pipelines we work on; not a benchmarked constant). Report MOTA and fragmentation count as your end-to-end continuity metrics, but never debug with them alone — they aggregate all three stages and hide which one moved. Correlate ID-switch spikes with latency spikes. If they co-occur, your “association bug” is a frame-drop symptom and belongs upstream. Verify the state estimator’s coasting horizon against the longest occlusion your scene actually produces, not a default value. The ROI here is measurable rather than rhetorical. In our experience, teams that instrument these three seams cut ID-switch debugging from days to hours, because “tracking is flaky” becomes “association IDF1 dropped 8 points at loads above 25 concurrent tracks while detection recall held” — a stage-level statement you can act on (observed-pattern; not a published benchmark). When should state estimation live in the same service as inference? This is a real architecture decision, not a default. State estimation is cheap and stateful; detection inference is expensive and stateless. Those opposite profiles create a genuine trade-off. Keep state estimation co-located with inference when latency between stages matters more than independent scaling — a single-camera edge device with a tight budget benefits from avoiding any network hop, and the state update cost is negligible next to inference. Split state estimation into a separate stage when you need to scale detection independently (for example, batching inference across many camera streams on a GPU while each stream keeps its own lightweight tracker), or when you want the estimator to survive a detector restart without losing every track. The decision mirrors the broader stateless-vs-stateful split you see across production ML services; the same reasoning shows up in how temporal CV pipelines for video are structured. FAQ How does real-time object tracking work? Real-time object tracking runs a per-frame pipeline of three stages: a detector emits boxes for the current frame, an association step matches those boxes to existing tracks, and a state estimator predicts and smooths track positions over time. “Real-time” means the whole pipeline completes inside a fixed per-frame budget — roughly 33 ms at 30 FPS. In practice it means identity and continuity are manufactured by the association and state-estimation stages, not by detection. What are the distinct stages of an object-tracking pipeline, and how do they map onto a modular CV architecture? The three stages are detection (what is in this frame), association/re-identification (which detections are the same objects seen before), and state estimation (where each object should be, including through gaps). Each is a component with a defined input/output contract, so you can instrument the seam between them. Mapping them as separate components is what lets you attribute a failure — detection recall, ID-switch rate, and track continuity are each owned by exactly one stage. What is the difference between detection and tracking, and why can’t a detector alone follow objects across frames? Detection is stateless: it re-derives the scene from scratch every frame and makes no claim that a box this frame is the same object as last frame. Tracking is the machinery that imposes identity and continuity on that stream of independent detections. A detector alone cannot follow objects because it has no persistent state to store “this is object 7 again” — even a perfect detector produces no track identities. How do trackers handle occlusion, re-identification, and ID switches, and which stage owns each failure? During occlusion the detector correctly stops emitting boxes, the state estimator coasts the track forward by predicting motion, and on reappearance the association step must re-identify the object as the same track. Coasting horizon that is too short causes fragmentation (state-estimation failure); a re-identification miss on reappearance spawns a new ID (association failure). Once stages are observable, every ID switch is attributable to one of them rather than to “tracking” in general. What determines whether object tracking stays within a real-time latency budget, and how do you measure it per stage? The dominant cost is detection inference; association and state estimation are usually single-digit milliseconds. Measure detection latency in isolation at production resolution on the target hardware, then log end-to-end per-frame latency including p99 — tail latency, not the mean, breaks the budget. When the pipeline falls behind, frames drop and association degrades, so correlate ID-switch spikes with latency spikes to catch a latency bug masquerading as an association bug. Which tracking metrics matter in production, and how do you observe them by component? Track ID-switch rate and IDF1 for the association stage, detection recall and mAP for the detector, and fragmentation count plus track continuity for state estimation. Report MOTA as an aggregate end-to-end continuity number, but never debug with it alone because it hides which stage moved. Observing metrics per component is what turns “tracking is flaky” into a stage-level statement you can act on. When should tracking state estimation live in the same service as inference versus a separate stage? Co-locate state estimation with inference when inter-stage latency matters more than independent scaling — typically a single-camera edge device with a tight budget. Split it out when you need to scale detection independently (batching many streams on a GPU while each keeps its own lightweight tracker) or when tracks must survive a detector restart. The decision mirrors the stateless-vs-stateful split common across production ML services. Before you sign off on this approach The uncomfortable truth is that most “tracking problems” are misdiagnosed because the tool used to observe them is a single aggregate score. If your only signal is MOTA, every failure looks the same and every fix is a guess. The question worth carrying into your next tracking build is not “which tracker should we use” but “can I, right now, tell whether an ID switch was a missed detection, a failed re-identification, or a dropped frame?” If the answer is no, the tracker is not the problem — the pipeline’s observability is. That is precisely the modular-testability question an A2 pipeline assessment is built to answer.
