Ask most engineers how an object tracker works and you get a comfortable answer: it follows objects around the frame. That description is intuitive, and it hides the one thing that actually decides whether tracking holds up in production — how detections get matched to existing tracks, frame after frame. A tracker does not watch objects. It solves a matching problem every frame, and the quality of that match is bounded by the geometry of the boxes it is handed. That distinction matters because the naive mental model treats a tracker as a black box you bolt onto any detector. Feed it boxes, get IDs. It works right up until the detector’s box geometry starts breaking the association step, and then identity switches multiply for reasons that look like a tracking bug but originate one stage upstream. What is the difference between object detection and object tracking? Detection answers a per-frame question: what objects are in this image, and where? It has no memory. Run a detector on frame 100 and it produces a set of boxes with class labels and confidence scores, knowing nothing about frame 99. Tracking adds the temporal dimension. Its job is to assign a persistent identity to each object so that the box around a given forklift in frame 99 and the box around that same forklift in frame 100 carry the same ID. Everything a tracker does — motion models, appearance embeddings, association logic — exists to answer one recurring question: which of the new detections is the continuation of which existing track? That reframing is the whole article. Detection is a spatial problem solved independently per frame. Tracking is an association problem solved across frames, and it inherits every weakness of the detections it receives. If you want the detector side of this story, how object detection with YOLO works covers the per-frame stage that feeds the tracker; this article picks up where those boxes leave the detection head. How does a tracker associate detections across frames, and where does IoU come in? The dominant paradigm is tracking-by-detection, and the canonical example is the SORT / DeepSORT family (and its descendants like ByteTrack and OC-SORT). The loop is straightforward to describe and subtle in practice: Predict. For every existing track, a motion model — usually a Kalman filter — predicts where that object should appear in the current frame based on its recent trajectory. Match. Compare each new detection against each predicted track position, build a cost matrix, and solve the assignment (typically with the Hungarian algorithm) to pair detections with tracks. Update. Matched tracks absorb their new detection and update the motion model. Unmatched detections spawn new tracks. Unmatched tracks age out after a few frames. The cost matrix is where Intersection-over-Union earns its keep. The most common association cost is 1 − IoU between a predicted track box and a candidate detection box: high overlap means low cost means likely the same object. Some trackers add an appearance term — a learned embedding distance, as DeepSORT does — but geometric overlap remains the backbone of the match in most production pipelines. Here is the mechanism that the black-box view misses: IoU-based association assumes the boxes it compares are tight around the objects. The moment a box captures a lot of background alongside its object, its IoU with neighbouring boxes becomes unreliable, and the assignment problem gets ambiguous. That ambiguity is not a tracker defect. It is geometry the tracker was handed. Why do rotated or densely packed objects cause identity switches in tracking? Consider a conveyor carrying rectangular parts at a 40-degree angle, or a warehouse shelf photographed at a slant where boxes are packed edge to edge. An axis-aligned bounding box (AABB) drawn around a rotated object is inflated: it has to be large enough to contain the tilted shape, so a substantial fraction of its area is background. When two such rotated objects sit close together, their AABBs overlap each other even though the objects themselves do not touch. Now run the association step. Two predicted track boxes and two new detection boxes, all overlapping because of the background padding, produce a cost matrix where several cells have similar low costs. The Hungarian solver picks an optimal assignment, but the margin between the correct pairing and the wrong one is thin. A little motion noise, a slightly different confidence ranking, and the two identities swap. That is an identity switch, and it is the signature failure of AABB geometry in dense or rotated scenes. Non-maximum suppression (NMS) compounds it. NMS deduplicates overlapping detections by IoU threshold. When neighbouring objects’ AABBs overlap heavily, NMS can merge two genuine detections into one or suppress the wrong one, so the tracker loses a detection entirely and fragments the track. In our experience working on line-side inspection and logistics pipelines, dense-packing scenes are where “the tracker is broken” tickets almost always trace back to detection geometry rather than the association code — an observed pattern across engagements, not a benchmarked rate. How does the choice between axis-aligned and oriented bounding boxes affect track stability? An oriented bounding box (OBB) hugs a rotated object with a tilted rectangle, so it captures far less background and does not spuriously overlap its neighbours. Feed OBBs into the association step and the cost matrix sharpens: the correct detection-to-track pairing sits at a clearly lower cost than the alternatives, the assignment margin widens, and identity switches drop. The decision of where box geometry is fixed happens in the model, not the tracker — the detection head is where AABB vs oriented box geometry is decided. The tracker inherits that choice. So the practical lever for track stability often lives upstream: if your scene has rotation or dense packing, an orientation-aware detector will do more for your MOTA and IDF1 than any amount of tracker tuning. That does not make OBB free. Oriented detection heads cost more to train and run, and they need orientation-labelled data. The right question is a trade-off, laid out below. Decision rubric: AABB or OBB detections for a tracker? Scene / condition Geometry that helps association Why Objects mostly upright, well-spaced AABB Padding is small; IoU already discriminative — no OBB cost justified observed-pattern Objects rotated (conveyor angle, tilted parts) OBB AABB inflates with background, blurring the cost matrix Dense packing (shelf, bin, crowd) OBB Neighbouring AABBs overlap and confuse both NMS and association Oblique / high viewing angle OBB Perspective tilts objects; axis-aligned boxes over-cover Fast motion, sparse scene AABB + strong motion model Association is dominated by prediction, not overlap Rotated and dense OBB (highest payoff) Both failure mechanisms stack; this is the AABB worst case Use the rubric to decide whether the association ambiguity you are seeing is worth paying for oriented detection. If the scene is upright and sparse, spend your effort on the motion model, not on box geometry. What metrics measure tracking quality, and how do you tell a detection problem from a tracking problem? Tracking quality is not detection quality. A detector can post excellent precision, recall, and mAP and still feed a tracker that produces garbage identities. The standard tracking metrics are: MOTA (Multiple Object Tracking Accuracy) — aggregates false positives, misses, and identity switches into one number. Sensitive to detection quality overall. IDF1 — the F1 score of identity assignments. This is the metric that punishes identity switches specifically; a tracker with clean detections but poor association shows a low IDF1 at high MOTA. Track fragmentation count — how often a single ground-truth trajectory gets broken into multiple track segments. Rises sharply when NMS drops detections in dense scenes. The diagnostic question — is this a detection problem or a tracking problem? — has a practical test. Freeze the association logic and inspect the raw detections in the frames where identities swap. If the boxes are loose, overlapping, or missing before association even runs, the problem is geometry upstream. If the detections are clean and tight but IDs still swap, the problem is in the motion model or the appearance term inside the tracker. That split tells you whether to tune the detector’s box geometry or the tracker’s association parameters — and the ROI is measurable: quantify the identity-switch rate before and after tightening geometry, then weigh it against the added detection cost. FAQ How does object tracker actually work? An object tracker assigns a persistent identity to each detected object across frames. In practice it runs a per-frame loop: a motion model (often a Kalman filter) predicts where each existing track should appear, a cost matrix compares new detections against those predictions, and an assignment solver pairs them. It does not “watch” objects — it solves a matching problem every frame, and the match quality is bounded by the geometry of the detection boxes it receives. What is the difference between object detection and object tracking? Detection is a per-frame spatial problem with no memory: it finds what objects are in a single image and where. Tracking adds the temporal dimension, linking detections across frames so the same object keeps the same ID. Tracking inherits every weakness of the detections it is handed, which is why tracking quality is bounded by detection geometry upstream. How does a tracker associate detections across frames, and where does IoU come in? The tracker predicts each existing track’s position, builds a cost matrix comparing predictions against new detections, and solves the assignment (typically with the Hungarian algorithm). The most common association cost is 1 − IoU between predicted and candidate boxes — high overlap means low cost means likely the same object. Some trackers add a learned appearance embedding, but geometric overlap remains the backbone of the match. Why do rotated or densely packed objects cause identity switches in tracking? An axis-aligned box around a rotated object is inflated with background, and neighbouring inflated boxes overlap even when the objects do not touch. That produces a cost matrix with several similar low-cost cells, so the assignment margin between the correct and wrong pairing is thin — a little noise flips the IDs. Non-maximum suppression compounds it by merging or dropping genuinely distinct detections in dense scenes, fragmenting tracks. How does the choice between axis-aligned and oriented bounding boxes affect track stability? Oriented boxes hug rotated objects tightly, capturing less background and avoiding spurious overlap with neighbours, so the association cost matrix sharpens and identity switches drop. The geometry choice is fixed in the detection head, not the tracker, so an orientation-aware detector often does more for MOTA and IDF1 than tracker tuning. The cost is more expensive detection and orientation-labelled data, so it is only worth it when the scene is rotated or densely packed. What metrics measure tracking quality, and how do you tell a detection problem from a tracking problem? MOTA aggregates false positives, misses, and identity switches; IDF1 specifically scores identity assignment quality; track fragmentation counts how often one trajectory breaks apart. To tell the problems apart, freeze the association logic and inspect the raw detections where IDs swap: loose or overlapping boxes point to geometry upstream, while clean boxes with swapping IDs point to the motion model or appearance term inside the tracker. If you are scoping a system where parts move across frames on a line — say, tracking components across a conveyor for line-side inspection — the failure to plan for is treating association tuning as the fix when the real divergence is production geometry the off-the-shelf detector never assumed. That same divergence between production rotation, packing density, and viewing angle versus off-the-shelf assumptions is exactly what an A2-style geometry assessment is built to surface before it propagates downstream into your track identities. For the full tracking-model picture, how a tracking model works from oriented detections to persistent object IDs carries the argument the other direction — from geometry into identity.
