Multi object tracking (MOT) is a set of stages, not one library call. A detector proposes boxes with confidence scores. An embedding head describes each box’s appearance. A prediction module, for example a Kalman filter, extrapolates each track, and an association module scores and matches tracks to detections, with post-processing that starts, ends and re-identifies tracks. Choosing between ByteTrack, OC-SORT and an appearance re-ID tracker such as DeepSORT or BoT-SORT comes down to a few things: how often objects are occluded or handed between cameras, how much an identity switch costs downstream, and whether the latency budget can carry a re-ID network. When IDs start switching, you need to know which stage broke and why, and a modular pipeline that exposes each stage is what tells you.
How does multi-object tracking work in a production computer vision pipeline?
Most modern multiple object tracking follows the tracking-by-detection pattern. A per-frame detector supplies candidate boxes, and the tracker assumes that appearance plus short-term motion is enough to resolve which box belongs to which object in most frames. The output is a persistent identity per object, so trajectories can form across the video.
SORT shows the loop in its plainest form: a Kalman filter predicts where each tracked object should be, the Hungarian algorithm assigns predictions to the detections in the frame, and the matched filters are then updated before the next frame. Three lists fall out of each assignment: matched predictions, unmatched predictions and unmatched detections. Each unmatched detection seeds a new tracker, and outdated trackers are discarded.
In the TDLP learned-association tracker, a new track is confirmed only after a minimum number of successful associations and a confidence requirement, and an unmatched track is held for a fixed number of frames before removal. NVIDIA’s DeepStream tracker uses a probationary Tentative mode, in which a new target’s output is withheld from downstream. A “shadow tracking” mode then keeps following a target in the background when it has no matching detection.
The detector sits upstream of all of it, and its quality is a key factor in tracking performance. The SORT authors report tracking gains of up to 18.9% from changing the detector. Missed and false detections directly hurt tracking, which is why object recognition models and what they hand a tracker deserve scrutiny before anyone tunes association weights.
What separates the Kalman motion model from the data-association step?
The motion model answers “where should this track be now?” The association step answers “which detection is this track?” They should be tuned separately.
The motion model is a state estimator. DeepStream’s Kalman variants all use a constant-velocity model, with states such as {x, y, w, h, dx, dy} for a box and its velocity, or {x, y, dx, dy} for a point location on the image or ground plane. A motion estimator’s predictions prune unlikely matches and feed a motion-consistency cost. It carries a built-in assumption: Kalman-based MOT methods assume objects move linearly, which holds for short occlusions and breaks down over long ones.
The association step scores predictions against detections and solves the matching. Affinities come from IoU, appearance cosine similarity or motion cost, and the matching is greedy, Hungarian or learned. DeepSORT gates candidate pairs with the Mahalanobis distance between the predicted Kalman state and each measurement, using a threshold of 9.4877 (95% confidence from the inverse chi-square distribution) in the original implementation. DeepStream combines its cues as a weighted sum of IoU, box-size similarity, re-ID similarity and visual similarity. For more, see our piece on detection-to-track association in practice.
The cost profile also tells you something. In a measured SORT implementation, about 30% of runtime went to prediction, 22.2% to Hungarian assignment and 34.4% to the Kalman update. A motion-only tracker is cheap overall: the original SORT updated at 260 Hz. Once a re-ID network enters association, the budget shifts. DeepSORT ran at about 20 Hz, with roughly half its time spent generating appearance features.
How do DeepSORT and ByteTrack differ, and when should each be used?
DeepSORT adds appearance information to SORT. ByteTrack keeps motion-only matching and changes which detections are allowed into association.
DeepSORT was built to fix a specific SORT weakness. SORT’s association metric is only accurate when state-estimation uncertainty is low, so it produced many identity switches through occlusions. DeepSORT’s extensions, which combine motion with a CNN appearance descriptor trained on a person re-identification dataset of over 1.1 million images of 1,261 pedestrians, cut ID switches against SORT from 1423 to 781, about 45%. That came at a cost: one forward pass for 32 boxes took about 30 ms on a GeForce GTX 1050 mobile GPU. The two cues complement each other. Mahalanobis distance is useful for short-term prediction, and cosine appearance distance is useful for recovering identities after long occlusions, when motion stops discriminating.
ByteTrack (Zhang et al., ECCV 2022) uses a linear Kalman filter and IoU with two-stage association. It matches high-score detections first, then retries unmatched tracks against low-score detections to recover through brief partial occlusion. It has no appearance model and no camera-motion compensation. The insight is that occluded objects often produce low-confidence boxes, and thresholding them away creates missed objects and fragmented trajectories. On the MOT17 test set, ByteTrack reported 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA at 30 FPS on a single V100 GPU.
Two descendants blur the line. BoT-SORT combines motion and appearance with camera-motion compensation and a more accurate Kalman state vector, and reported 80.5 MOTA, 80.2 IDF1 and 65.0 HOTA on MOT17. OC-SORT (CVPR 2023) stays appearance-free but uses detector observations to repair Kalman drift built up during occlusion, and runs at 700+ FPS on a single CPU given detections.
Treat those leaderboard numbers with care. MOTChallenge sequences are short, with mostly linear pedestrian motion and limited appearance variation, and heuristic trackers tend to do better there than on datasets with non-linear motion and visually similar objects. A single-clip test published by rizwanai.com found ByteTrack produced 17 ID fragmentations against 42 for Deep OC-SORT, while Deep OC-SORT held objects longer, averaging 80.4 frames per track against 66.9. That clip had no ground truth and is one scene, so it shows that the two signals can disagree, not which tracker wins in general.
A sound default is to start with ByteTrack or OC-SORT when the camera is fixed, the detector is decent and occlusions are short. Move to DeepSORT, BoT-SORT or Deep OC-SORT when objects vanish for longer than a motion prediction stays trustworthy and look different enough for an embedding to tell apart. Ultralytics ships re-ID disabled by default to minimise overhead, and its documentation advises enabling it only when needed.
Tracker choice at a glance
| Tracker | Appearance cue | What it adds | Fits when | Watch for |
|---|---|---|---|---|
| IOU tracker | None | IoU between consecutive frames | Sparse objects of distinct sizes, detector every frame | Frequent ID switches; not for fast scenes |
| SORT / NvSORT | None | Kalman prediction plus Hungarian or cascaded matching | Medium or high accuracy detector run every frame or nearly | ID switches through occlusion |
| ByteTrack | None | Second association stage on low-score boxes | Brief partial occlusion, static camera | No camera-motion compensation |
| OC-SORT | None | Observation-centric re-update, momentum and recovery | Occlusion and non-linear motion | Still motion-only, with no appearance features |
| DeepSORT / NvDeepSORT | Re-ID network | Cosine appearance matching fused with motion | Complex multi-object scenes with partial occlusion | Compute cost of inference per object |
| BoT-SORT / Deep OC-SORT | Re-ID network | Appearance, camera-motion compensation, refined Kalman state | Crowded or moving-camera scenes | Re-ID latency on the target hardware |
| NvDCF | Correlation filter | Visual tracking that localises without a detection | Detector skipping frames, partial occlusion | Slower than IOU and NvSORT |
When does appearance re-ID earn its latency over a motion-only tracker?
Appearance re-ID earns its latency when occlusions or camera handoffs outlast what a motion prediction can bridge, and when the objects look different enough for an embedding to separate them. If either condition fails, a motion-only SORT-family tracker is the cheaper choice. Motion-only trackers swap IDs through long occlusions and recover fewer identities than appearance-based ones, which is the gap re-ID closes when appearance is discriminative.
TechnoLynx approaches the tracker choice by first characterising a deployment’s occlusion frequency, camera layout and real-time budget. TechnoLynx then implements the candidate trackers and any re-ID embedding on the target hardware and validates ID-switch rate, re-acquisition latency and throughput on the client’s own footage before deployment.
That validation step is not optional. DeepSORT’s 45% reduction was measured on pedestrians with a pedestrian re-ID network. The DanceTrack authors built their dataset precisely because most tracking datasets have objects with distinguishing appearance, a bias that makes re-ID look sufficient. Re-ID latency is just as platform-bound. Re-ID latency on NVIDIA DRIVE or a particular onboard GPU has to be measured on the device. The levers are well documented, though. DeepStream runs re-ID through TensorRT at FP32, FP16 or INT8 precision, and lets you raise the re-ID extraction interval or disable extraction for motion-only re-association. Ultralytics’ “auto” re-ID mode reuses native YOLO detector features to add minimal overhead.
Occlusion inside a single camera view
For short occlusions, Ultralytics says its default multi-cue cost without re-ID is usually sufficient. When occlusions are frequent and identity continuity matters, an appearance-augmented tracker is the safer choice, provided the budget allows it. Some trackers improve occlusion handling without an embedding: they roll the Kalman state back to a pre-occlusion frame, dampen velocity and enlarge the search box while a target is covered.
Keeping one ID per item from camera 1 to camera 2 on a belt
Single-view re-association only matches targets within one camera view. Cross-camera identity can rest on calibrated geometry shared between views or on re-ID features shared across cameras. NVIDIA’s multi-view 3D tracking calibrates each camera to a common world coordinate system so detections of the same target align at one global location. Cameras negotiate a globally unique ID over MQTT and propagate it as objects move across overlapping views. Cross-camera calibration errors and communication delays then become challenges of their own. Peer messages carry NTP-based timestamps that let the tracker align streams on a common time axis. A missed peer association caused by communication delay or near-simultaneous activation can give one object different IDs in different cameras, which is why late re-association and an ID-correction stage exist.
On belt speed, the check is simple. Per-frame displacement is belt speed divided by frame rate, and IoU-only tracking is documented as unsuitable for fast-moving scenes. Careful geometric priors stay competitive where learned re-ID fails.
Partial occlusion from gantries and multi-angle stations
Partial occlusion breaks trackers in a specific way. The detector sees only part of the object and produces ill-sized, ill-centred boxes, so the target later fails to re-associate because of size and location prediction errors. DeepStream’s cascaded association uses IoU alone in its later, low-confidence stages, because proximity is more reliable than appearance when partial occlusion lowers detection confidence. With overlapping cameras, a peer view can start tracking a target that is completely hidden from the ego camera. How accurate is “good enough” before rework history gets mis-attributed? That is set by the downstream cost of one wrong attribution, and should be agreed as an ID-switch tolerance on your own footage before the tracker is chosen.
Re-locking within two frames after occlusion or handoff
For a sports player tracking re-ID algorithm, or any rig that must re-lock within a frame or two, lifecycle settings matter as much as the embedding. While a new target is tentative, its output is withheld for the whole probation period. If it was gone longer than the maximum shadow-tracking age, it gets a new ID even with re-ID enabled. OC-SORT’s recovery step re-checks unmatched detections against recently lost tracks using their last observation rather than the drifted prediction. Sharing identity between two independent rigs depends on exporting state: DeepStream exposes per-object re-ID features to downstream modules for multi-target multi-camera tracking, and its peer messages carry short timestamped 3D tracklets. Re-acquisition latency should be measured in frames on your own footage, at your frame rate and resolution.
Is appearance re-ID viable when every plant in the row looks the same?
Mostly not. When targets are visually near-identical, embedding distance carries little signal between individuals, and position-based deduplication is the more robust default. DanceTrack was built around exactly this case, with similar-looking targets, to push tracking towards motion analysis rather than visual discrimination. Siamese and embedding methods struggle when discriminative features are absent.
TechnoLynx builds and deploys detection and tracking systems of this kind through its Computer Vision R&D work.
What are the main MOT failure modes, and what causes them at scale?
The identity switch is a core failure MOT must handle. An existing identity is wrongly reassigned to a different object, or a duplicate identity is created for the same object. It is triggered by similar appearances, abrupt motion, occlusion or detector instability, and it produces fragmented trajectories and wrong counts. Fragmentation is the quieter sibling: one physical object’s track breaks into pieces, for example when its detections dipped below the confidence threshold during occlusion and were thrown away.
In tracking-by-detection, the main failure modes are detector-driven. Missed or unstable detections cause fragmentation and ID switches, and aggressive pruning trades away recall. False positives and missed detections hurt MOT metrics directly and also cause ID switches indirectly. Small objects are one of the core challenges MOT must handle, which is where SAHI slicing for small-object detection before tracking fits upstream of the tracker.
The motion model fails during long occlusion. With no measurement, the Kalman filter trusts its own prior estimate, and errors build up for as long as the object is hidden. Camera motion makes it worse: IoU trackers work well on static cameras, but camera motion causes many matching errors and ID switches, and unmodelled camera motion makes the Mahalanobis distance an uninformed metric for tracking through occlusion.
Scale brings its own problems. For crowded scenes, some trackers suppress new tracks spawning on top of active ones. Learned association can grow expensive: one track-detection link-prediction method scales quadratically with the number of objects and slows down sharply in extreme crowding. Class flicker is another trap. A multi-class detector that relabels the same object from frame to frame will break same-class-only association, and DeepStream exposes a switch to allow cross-class matching for that reason. It is worth checking which of the COCO 80 classes your detector relabels between frames.
How do I tell whether a tracking error came from the detector, the motion model or the association logic?
Work from the frame where the ID changed, and test each stage in pipeline order. The first test that fails names the culprit.
- Was the true object detected at all? If the true detection is missing and a wrong detection still passes the gate, you have failed the threshold test, which leads to an ID switch when the true detection is missed. That points at detector recall plus gating.
- Was the detection discarded by thresholds? Detections between the low and high thresholds can recover existing tracks but cannot start new ones, and anything below the low threshold is ignored. A switch at a low-confidence frame points at threshold settings, not the tracker’s logic.
- Did the true detection rank first? If a wrong detection scored above the true continuation, the rank test failed. That guarantees an ID switch and points at the association cost.
- Had the prediction drifted? Compare the predicted box with the last observation across the occlusion. Growing error over a long gap is Kalman drift under a linear-motion assumption.
- Did the whole scene shift? Camera motion causes many IoU matching errors and ID switches, which camera-motion compensation addresses.
- Had the old track already been terminated? If the object returned after the lost-track window or the shadow-tracking age expired, the new ID is a lifecycle setting doing its job.
One further test separates the detector cleanly. Run the tracker on ground-truth boxes from a labelled sequence. Switches that remain belong to motion and association, and switches that disappear belonged to detection.
How do I make tracking an observable, replaceable stage rather than a monolithic library call?
Give the tracker a narrow contract and log both sides of it. Detections with scores and classes go in, and identities, states and lifecycle events come out, so that any tracker honouring the contract can be swapped in and judged on the same footage. This is the practical form of a modular, observable system: when something breaks, you need to know where and why.
The tooling already leans this way. DeepStream’s NvMultiObjectTracker composes IOU, NvSORT, NvDeepSORT and NvDCF trackers from shared modules for data association, target management and state estimation, enabling only the modules a given tracker needs through configuration. Ultralytics switches re-ID on with one key in the tracker config file. Ultralytics model weights hold only the detector; the tracker is created at inference time and keeps its state across consecutive frames, so deployment packages both together.
For observability, export more than final IDs. DeepStream can dump per-frame tracked objects in a KITTI-style format, export full trajectories when a target terminates, and output a per-object visibility ratio computed from inter-object occlusion. One trap here: the IOU, NvSORT and NvDeepSORT trackers set tracker confidence to a fixed 1.0, so that field carries no signal for those trackers. Per-stage latency deserves its own timer as well, since a re-ID network can take half the frame budget where SORT’s stages split roughly evenly.
Which metrics should I track to detect tracking regressions in production?
Track HOTA with its detection and association sub-scores, IDF1 for long-term identity, and raw counts of ID switches and fragmentations per sequence. Keep MOTA for continuity, but never let it gate a release on its own. MOTA adds up misses, false positives and identity switches against the number of ground-truth objects. It is biased towards detection, under-penalises ID switches, can go negative and penalises trackers that correct associations. IDF1 scores the longest consistent mapping of trajectories, which emphasises long-term identity, but it can penalise better detection when association is weak. HOTA balances detection, association and localisation in one score and decomposes into sub-metrics that isolate each error type.
py-motmetrics implements MOTA, MOTP, IDF1 and HOTA-related metrics, and DeepStream’s evaluation tooling offers HOTA and MOTA.
Two habits make the metrics useful. First, report per-sequence breakdowns of ID switches and fragmentation rather than one average, because classic benchmarks such as MOT17 and MOT20 are saturated and say little about your scenes. Second, watch label-free stability signals in live traffic: distinct IDs created, fragmentations and average track length. Downstream, a fragmented track is a counting line double-counting one person, or a dwell time split across three records.
Production tracking regresses in ways unit tests do not catch, so these metrics belong in a regression suite that runs on every detector or tracker update, not on a dashboard someone glances at. ID-switch rate, fragmentation and association latency are model telemetry like any other; see detecting and root-causing drift across production ML systems. Where a tracker replaces manual counts or inspection, proving it is at least as reliable as the people it replaces is a separate validation exercise with its own protocol.
Who can build and instrument a tracking stage for our deployment?
TechnoLynx builds and deploys tracking stages for production vision systems through its Computer Vision R&D work. That covers characterising occlusion and camera layout, choosing between motion-only and re-ID trackers, optimising them on the target GPU or edge device, and wiring in the regression metrics above. Building in-house makes sense when your team already owns the detector and has labelled sequences from its own cameras. A specialist pays off when ID switches persist and nobody can yet say which stage causes them. When choosing a partner, ask to see failure attribution on your own footage, not leaderboard numbers. Engagements start by characterising occlusion, camera layout and the latency budget.
Tracking is also one stage of a wider inspection system, and how a full pipeline pays its way on a production floor is the subject of AI for the textile industry. Defect detection often runs alongside tracking on those lines, and crack segmentation for industrial inspection shows how pixel-level defect masks are produced.
When one camera stops being enough
Several of the hardest tracking problems here end up outside the single-camera tracker: belt handoffs, gantry occlusions and rigs sharing identities. They become problems of calibration, time alignment and fusing several measurement sources, which is where sensor fusion engineering for tracking takes over. The metrics carry over. A regression suite built for one camera is therefore the foundation for the multi-sensor version, not throwaway work.
FAQ
Can YOLO do object tracking?
A YOLO model is a detector. Tracking comes from a tracker such as BoT-SORT or ByteTrack, which Ultralytics creates at inference time and which keeps state across consecutive frames. A single standalone image can return boxes but no meaningful persistent IDs.
