A highlighted lead vehicle that flickers, swaps its label with the car beside it when the two cross, or drifts off a pedestrian who steps behind a van for half a second — that is not a rendering bug. It is a tracking failure that the overlay is faithfully drawing. An automotive AR overlay is only as trustworthy as the tracks it draws on, and the fastest way to make a heads-up display distract instead of inform is to feed it per-frame detections and call that tracking. The distinction matters because detection and tracking answer different questions. A detector answers “what objects are in this frame, and where?” A tracker answers “which of the objects in this frame is the same object I saw in the previous frames, and where will it be next?” The naive pipeline collapses these two questions into one: run a detector every frame, draw a box around each result, done. That works in a static demo. It falls apart the moment the ego vehicle and the tracked objects are both moving at speed. What does tracking multiple objects actually mean in practice? Multi-object tracking (MOT) is the stage that sits between per-frame detection and anything downstream that needs a stable notion of this object over time. Its job is to maintain a persistent identity and a predicted position for every object across the video stream, even when the raw detections are noisy, briefly missing, or overlapping. In the tracking-by-detection paradigm that dominates real deployments, MOT is built from two coupled loops running every frame: Data association — match this frame’s detections to the set of existing tracks. Which new box belongs to track #7, which is a genuinely new object, and which existing track got no detection this frame? State estimation — maintain a motion model (position, velocity, sometimes acceleration) for each track, predict where it should be in the next frame, and correct that prediction when a matching detection arrives. Classic trackers such as SORT and DeepSORT wire these together with a Kalman filter for the motion model and the Hungarian algorithm for the assignment, with DeepSORT adding a learned appearance embedding so two objects that cross paths can be told apart by how they look, not just where they are. Newer methods like ByteTrack push association further by also using low-confidence detections instead of discarding them. The architecture is what matters here more than the specific tracker: detection is one stage, association-and-prediction is a separate stage, and the overlay consumes the output of the second, never the raw output of the first. This is the same discipline that separates a good detection metric from a good tracking metric. If you have not yet locked down detection quality, the [email protected]:0.95 detection metric behind automotive AR perception is the upstream concern; MOT assumes those detections already exist and asks what you do with them across time. Detecting each frame versus tracking across frames The clearest way to see why per-frame detection is not tracking is to watch what each one does to an object’s identity. Behaviour under motion Per-frame detection only Multi-object tracking Object identity Re-assigned independently every frame Persistent ID carried across frames Two objects crossing Boxes may swap which is “lead vehicle” Association + appearance keeps IDs separate Brief occlusion (object behind another) Box vanishes, then a “new” object appears Track coasts on prediction, re-associates on reappearance Overlay stability Flicker, jitter, label jumps ID-stable annotation locked to the object Missed detection (one bad frame) Annotation drops out entirely Predicted state fills the gap A detector treats every frame as an independent problem. That is exactly what you want for measuring detection accuracy, and exactly what you do not want when a driver is looking at the result. The overlay renders whatever it is handed, so if the “lead vehicle” label is computed fresh each frame with no memory, the label has no reason to stay on the same car — and it won’t. The temporal consistency that a driver reads as trustworthy has to be manufactured by the tracking stage; it does not fall out of good detection alone. How does MOT keep identities stable through occlusions and crossings — and what causes ID switches? An ID switch is the failure the driver actually notices: track #3 was the motorcycle on the left, and now #3 is the sedan it just passed. The identity jumped from one object to another. In a driver-facing overlay, that is the annotation lying to the driver about which object is which. Two situations produce most ID switches. The first is crossings, where two objects’ bounding boxes overlap in image space. If association relies only on position and overlap (IoU), the tracker has no way to decide which detection continues which track when they occupy nearly the same pixels — so it can guess wrong and swap them. Appearance embeddings exist precisely to break this tie: a learned feature vector per object gives association a second, position-independent signal. The second is occlusion, where an object disappears behind another for several frames. Here the motion model earns its place. A Kalman filter’s prediction lets a track coast through the frames with no detection, holding a plausible position so that when the object reappears, association can reconnect it to the same ID instead of spawning a new one. Tune the coast window too short and every occlusion becomes an ID switch; tune it too long and the tracker holds ghost tracks for objects that have genuinely left the scene. This is an observed trade-off in the tracking configurations we have worked through, not a setting with a universally correct value — it depends on frame rate, typical object speed, and how cluttered the scene is. Which metrics tell you whether a tracker is good enough for a driver-facing overlay? Detection metrics like mAP say nothing about identity over time, so MOT has its own family. Three carry most of the signal for an overlay: MOTA (Multi-Object Tracking Accuracy) — a single aggregate that combines false positives, missed detections, and ID switches into one number. Useful as a headline, blunt as a diagnostic, because a good MOTA can hide a bad ID-switch count if detection is strong. IDF1 — measures how consistently each ground-truth identity is assigned to the same predicted track across its whole lifespan. This is the metric that most directly reflects what a driver experiences, because it penalises identity fragmentation even when frame-level detection looks fine. ID-switch count — the raw count of identity jumps. For a driver-facing overlay this is close to a safety-relevant number: every switch is a moment the annotation could point the driver at the wrong object. These are the standard MOTChallenge metrics; reported as benchmark figures against a public dataset such as MOT17, they are comparable across trackers, but a benchmark number is not your number. The measurement that matters for a specific overlay is on your own footage, at your own frame rate, under your own lighting and occlusion mix — treat a MOTChallenge leaderboard score as a starting screen, not a deployment guarantee. A quick way to read the three together If MOTA is high but IDF1 is low, your detector is good and your association is bad — you are catching objects but losing track of which object. If both MOTA and IDF1 are high but the ID-switch count spikes in specific scenarios (dense traffic, low sun), you have a localised association weakness worth fixing before it reaches the HUD. And if MOTA itself is low, the problem is upstream in detection, and no amount of association tuning will rescue the overlay. How does the tracker’s motion prediction feed the AR HUD? This is where MOT stops being a computer-vision nicety and becomes structural to the overlay. An automotive AR HUD has to lock its annotations to the road frame while the car is moving, which means it cannot afford to render an object where the object was when the camera captured it — by the time the pixel lights up on the windshield, the object has moved and so has the ego vehicle. The tracker’s state estimate is what closes that gap. Because each track carries a motion model, the pipeline can predict where the object will be at render time, not just where it was detected. That predicted state is the input to the pose-prediction the HUD uses to place the annotation correctly. Without a stable track carrying a velocity estimate, there is nothing to predict from — you are back to drawing on stale detections. Prediction is also what carries a track through the frames where detection is momentarily uncertain, so the overlay holds steady instead of dropping out. The rendering side of this locking problem — fusing the GPU passes so the overlay stays frame-locked — is its own engineering concern, covered in fusing GPU passes for frame-locked AR overlays. MOT supplies the ID-stable, motion-predicted tracks that make that frame-locking meaningful; the two stages have to agree on a latency budget or the overlay drifts regardless of how good either one is alone. Where does tracking latency fit in the end-to-end budget? Tracking is not free, and it sits on the critical path from camera to windshield. The end-to-end budget runs: capture → detection inference → tracking (association + state update) → pose prediction → render. Every stage spends milliseconds, and the HUD’s sub-frame latency target is the sum, not any single stage. The detection stage is usually the heaviest GPU consumer — a YOLO-class detector’s throughput and batching behaviour dominate the frame-time story, which is why how YOLO inference on GPU works and what batching really changes is worth understanding before you optimise the tracker. Association and state update are comparatively cheap in raw FLOPs but can still blow the budget if the appearance-embedding network runs per detection per frame without batching, or if the Hungarian assignment scales badly in dense scenes. The tracking latency is a line item, and in the GPU audit and safety review that an overlay passes before it ships, both track latency and ID stability are checked explicitly — a tracker that is accurate but too slow fails the same review as one that is fast but drops identities. If you want the broader picture of how these latency and safety constraints interact, the challenges of perception, latency, and safety in autonomous and assisted driving sets the context. The GPU engineering practice that tunes this pipeline treats the tracker’s latency as a budget commitment, not an afterthought. How is automotive AR tracking different from a generic MOT demo? A pedestrian-tracking demo on a fixed CCTV feed and a driver-facing overlay share an algorithm and almost nothing else. In the demo, the camera is static, so the only motion is the objects’. In a car, the camera is moving fast, which means the apparent motion of every tracked object is a combination of its real motion and the ego vehicle’s — the motion model has to account for a moving reference frame or its predictions drift. Frame rates are higher and jitter tolerances are tighter, because the output goes to a driver in real time rather than to an analyst reviewing footage. And the failure cost is different in kind: a swapped ID in a surveillance montage is an annoyance; a swapped ID on a HUD can point a driver’s attention at the wrong vehicle at the wrong moment. Those differences are why the pose-estimation and keypoint work that supports driver-facing AR — see keypoint estimation for driver-facing AR — is held to a stricter bar than a generic demo would suggest. In our experience the gap between “works in the demo video” and “holds up on a moving vehicle at motorway speed” is where most of the real tracking engineering lives, and it is not visible in a clip. FAQ What should you know about tracking multiple objects in practice? Multi-object tracking maintains a persistent identity and a predicted position for every object across video frames. In the common tracking-by-detection approach it runs two coupled loops each frame: data association matches new detections to existing tracks, and state estimation predicts each track’s next position and corrects it when a detection arrives. In practice it is the stage that turns independent per-frame boxes into stable objects an overlay can annotate over time. What is the difference between detecting objects each frame and tracking them across frames? A detector answers “what is in this frame and where?” and treats every frame as an independent problem with no memory. A tracker answers “which of these is the same object I saw before, and where is it going?” — carrying a persistent ID and a motion model across frames. Per-frame detection produces flicker, label jumps, and dropouts under motion; tracking produces ID-stable annotations that survive occlusions and crossings. How does multi-object tracking keep object identities stable through occlusions and crossings — and what causes ID switches? Crossings are handled by adding an appearance embedding so association can tell two overlapping objects apart by how they look, not just by position. Occlusions are handled by a motion model (typically a Kalman filter) whose prediction lets a track coast through frames with no detection and re-associate when the object reappears. ID switches — where a track’s identity jumps to a different object — come mostly from position-only association during crossings and from coast windows that are too short for the occlusion. Which metrics (MOTA, IDF1, ID switches) tell you whether a tracker is good enough for a driver-facing overlay? MOTA is a headline that combines false positives, misses, and ID switches into one number but can hide a bad switch count behind strong detection. IDF1 measures how consistently each real identity keeps the same track over its lifespan and most directly reflects what a driver sees. ID-switch count is close to safety-relevant for a HUD, since each switch is a moment the annotation could point at the wrong object. Read them on your own footage, not just a leaderboard. How does motion prediction in the tracker feed the pose-prediction the AR HUD needs to stay locked to the road frame at speed? Each track carries a motion model, so the pipeline can predict where an object will be at render time rather than where it was detected. That predicted state is the input to the HUD’s pose prediction, which places the annotation correctly on a moving object seen from a moving car. Without a stable track carrying a velocity estimate there is nothing to predict from, so the overlay would draw on stale detections and drift. Where does the tracking latency fit inside the automotive overlay’s end-to-end latency and safety budget? The end-to-end path runs capture → detection → tracking → pose prediction → render, and the HUD’s sub-frame target is the sum of all stages. Tracking is comparatively cheap in FLOPs but can blow the budget if appearance embeddings run unbatched or assignment scales badly in dense scenes. In the GPU audit and safety review an overlay must pass before shipping, both track latency and ID stability are explicit line items. How does object tracking for automotive AR differ from generic multi-object tracking demos? The camera is moving fast, so the motion model must account for a moving reference frame rather than only object motion. Frame rates are higher and jitter tolerances tighter because the output goes to a driver in real time. And the failure cost differs in kind: a swapped ID in a surveillance clip is an annoyance, while one on a HUD can direct a driver’s attention to the wrong vehicle at the wrong moment. The honest closing question for any automotive AR team is not “is our tracker accurate?” but “which identities does it lose, under what motion, and can the overlay’s latency budget absorb the prediction we need to hide those gaps from the driver?” Locked, ID-stable tracks are the prerequisite for a HUD that informs; the ID-switch count and the track-to-render latency are the two numbers the safety review will ask for, so they are the two worth measuring first.
