Pull down a COCO-pretrained detector, point it at your production scene, and watch it fire confidently on people, cars, and chairs — while your actual target object never triggers a single box. The class list looked complete. It wasn’t complete for you. This is the failure that the COCO taxonomy quietly sets up. Microsoft’s Common Objects in Context dataset ships with 80 object categories, and because almost every off-the-shelf detector — YOLO variants, RT-DETR, Faster R-CNN reference weights — is pretrained on exactly those 80, the class list has become a de facto assumption about what “object detection” means. That assumption is safe for demos and generic benchmarks. It breaks the moment your scene contains an object COCO never labelled, or lumps two things you need to distinguish into one coarse class. The right way to read the 80 classes is not as a feature list. It is a design constraint. The taxonomy biases what a detector will and won’t fire on, and that bias propagates downstream: a tracker can only associate and follow objects the detector actually reported. When tracks fragment or targets vanish, the reflexive move is to blame the tracker and start retraining. Often the real problem is upstream — COCO simply doesn’t represent the object at all. What are the 80 COCO classes, and how are they grouped? COCO organizes its 80 “things” classes into 12 supercategories. The grouping is not cosmetic; it tells you where the dataset’s labelling effort was concentrated and where it thins out. Here is the coverage at a glance. Supercategory Example classes Class count (approx.) Person person 1 Vehicle bicycle, car, motorcycle, airplane, bus, train, truck, boat 8 Outdoor traffic light, fire hydrant, stop sign, parking meter, bench 5 Animal bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe 10 Accessory backpack, umbrella, handbag, tie, suitcase 5 Sports frisbee, skis, snowboard, sports ball, kite, baseball bat/glove, skateboard, surfboard, tennis racket 10 Kitchen bottle, wine glass, cup, fork, knife, spoon, bowl 7 Food banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake 10 Furniture chair, couch, potted plant, bed, dining table, toilet 6 Electronic tv, laptop, mouse, remote, keyboard, cell phone 6 Appliance microwave, oven, toaster, sink, refrigerator 5 Indoor book, clock, vase, scissors, teddy bear, hair drier, toothbrush 7 (The exact per-supercategory counts sum to 80; the numbers above are the canonical COCO 2017 grouping. Segmentation-oriented variants add “stuff” classes like sky and grass, but the 80 detection categories are the ones a pretrained bounding-box model reports.) Two things jump out when you read this as a constraint rather than a catalogue. First, the dataset is heavily weighted toward everyday indoor and street scenes — kitchen items, furniture, household electronics, common animals, road vehicles. That reflects where the images came from: consumer photos and web imagery, not factory floors or warehouse aisles. Second, the granularity is uneven. car, truck, and bus are separate, but every kind of industrial part, tool, or packaged product collapses into nothing at all — there is no box, no pallet, no component, no SKU-level distinction. Why the class taxonomy matters for a tracker downstream A detector’s job is to answer “what is here and where.” A tracker’s job is to answer “which detection in frame t+1 is the same object as this detection in frame t.” Those are separate problems, and the boundary between them is exactly where COCO class coverage decides your fate. If COCO doesn’t contain your target class, the detector produces no box for it — not a low-confidence box, not a mislabelled box, simply nothing. The tracker never sees the object, so it can’t follow it. From the outside this looks like a tracking failure: the target “disappears.” But there was never anything to track. This is the class-scene fit problem, and it is why understanding what feeds a tracker versus what the tracker itself does has to come before any tracker tuning. The distinction has a measurable signature, which is what makes it useful. A genuine tracking bug shows up as ID switches — the object is detected consistently, but the tracker reassigns identities, fragments one trajectory into several, or swaps two nearby objects. A class-coverage problem shows up as detection recall gaps — the object is simply absent from the detection stream, so there are no IDs to switch in the first place. Reading which signature you have tells you which stage to fix. The mechanics of that association step are worth understanding directly if you’re diagnosing this class of failure, and we cover them in how an object tracker turns detections into persistent IDs. This is not a subtle diagnostic once you know to look for it. In our computer vision engagements, the most expensive detour we see is a team spending weeks tuning association thresholds, Kalman-filter noise parameters, and re-identification embeddings — when the detector never fired on the object because the class wasn’t in its training vocabulary. The tracker was working exactly as designed on an empty input. What common objects does COCO not cover? The gap is largest precisely where production computer vision lives. COCO was built to be general, and generality is the opposite of what most deployed systems need. A few recurring gaps we run into: Industrial and warehouse objects — pallets, totes, cartons, machine parts, tools, fixtures. None are COCO classes. A logistics pipeline built on COCO weights will detect the forklift operator (person) and maybe the forklift (nearest class: none — it’s not truck) but not the load. Retail SKUs — COCO has bottle, cup, banana, apple. It has no notion of product identity. Distinguishing one branded bottle from another, or counting facings on a shelf, is a fine-grained problem COCO’s coarse classes cannot express. Dedicated dense-shelf datasets exist for this reason; if you’re in retail, Open Images V7’s broader label set and where it still falls short for retail CV is a better starting reference than COCO. Domain-specific safety and inspection targets — PPE items beyond a coarse person, defect regions, cracks, weld seams, specific vehicle sub-types in an automotive scene. COCO’s car/truck/bus split is far too coarse for perception systems that need to distinguish an emergency vehicle from a delivery van. Anything below its granularity floor — where COCO says chair, you may need “operator seat vs. visitor chair”; where it says bird, an ecology project needs species. When you hit one of these gaps, the fix is not a tracking change. It is a data decision: fine-tune the detector on labelled examples of the missing class, relabel to your own taxonomy, or add a fine-grained classifier stage behind a coarse COCO detection. Which one depends on how far your target sits from COCO’s vocabulary — a topic we return to below. The annotation mechanics of doing this correctly are covered in COCO labels and the bounding-box annotation format; the format is reusable even when the class list isn’t. How to tell a class-coverage problem from a tracking bug Before committing any retraining spend, run this diagnostic. It scopes the fix to the right stage and takes an afternoon, not a sprint. Diagnostic checklist — is your tracking failure actually a detection class problem? Freeze the tracker. Log raw per-frame detections directly out of the detector, before any association. This is the single most informative step. Count detections on the failing object. Over a labelled clip where the target is clearly visible, how many frames produced any box on it? Near zero → class-coverage problem. COCO doesn’t represent the object. Stop tuning the tracker. High and consistent → the detector sees it; the failure is downstream. Check the assigned class label. When boxes do appear, what class does the detector call the object? A COCO detector will force everything into one of its 80 buckets — a mislabel here (your part called remote) is a coverage symptom, not a confidence-threshold issue. Separate the failure signature. Are trajectories fragmenting/swapping (ID switches → tracker) or absent (recall gap → detector)? These are diagnostically distinct. Only then look at association parameters. If and only if detections are present, consistent, and correctly classed, is tuning the tracker the right move. The value of this ordering is that it makes the detection-vs-tracking boundary measurable rather than intuited. A missing class shows up as a recall gap in step 2 — an observed pattern across our vision engagements, not a benchmarked rate, but a reliable one: the teams that log raw detections first almost never end up retraining the tracker for a coverage problem. When to use COCO-pretrained as-is versus fine-tune There is no universal answer, only class-scene fit. The decision rests on how well the 80 classes represent your actual targets. Situation Class-scene fit Recommended action Targets are COCO classes (people, vehicles, common objects) High Use pretrained detector as-is; invest effort in tracking and deployment Targets are COCO classes but need finer granularity Partial Coarse COCO detection + a fine-grained classifier stage behind it Targets resemble COCO classes but aren’t (industrial variants) Low Fine-tune on labelled domain data; keep COCO backbone Targets absent from COCO entirely (pallets, SKUs, defects) None Relabel to your own taxonomy; treat COCO only as a pretrained backbone for transfer The economic point behind this table is the ROI anchor: knowing your class-scene fit up front lets you decide between a pretrained detector and a fine-tuning or relabelling effort before you commit engineering time. A missing class is a detection recall problem with a known fix — collect and label data. Misattributing it to the tracker converts a scoped data task into an open-ended tuning slog. We’ve watched that conversion cost teams weeks, and it is entirely avoidable by asking the class-coverage question first. If your targets are fine-grained retail items, the practical path usually runs through building an SKU dataset rather than stretching COCO. It’s also worth knowing where COCO sits historically. It succeeded the older, 20-class PASCAL VOC dataset that established the object-detection label format; COCO widened the vocabulary to 80 and added segmentation masks and dense scene context. That lineage explains both its strengths — well-curated, widely benchmarked, excellent for general objects — and its blind spots. It was never meant to be the taxonomy for your factory or your shelf. FAQ How does coco dataset classes work? COCO defines a fixed vocabulary of 80 object categories that a detector is trained to recognize. In practice this means a COCO-pretrained model will only ever produce boxes for those 80 classes — anything outside the list is invisible to it. Treat the class list as a constraint on what your detector can see, not as a complete description of your scene. What are the 80 COCO dataset classes, and how are they grouped into supercategories? The 80 classes are organized into 12 supercategories: person, vehicle, outdoor, animal, accessory, sports, kitchen, food, furniture, electronic, appliance, and indoor. The grouping reflects COCO’s origin in everyday consumer and street imagery, so coverage is dense for household objects, common animals, and road vehicles, and sparse-to-absent for industrial, retail-SKU, and inspection targets. Why does the COCO class taxonomy matter for a detector that feeds a tracking pipeline? A tracker can only follow objects the detector reports. If COCO doesn’t contain your target class, the detector produces no box, so the tracker never sees the object — and the failure looks like a tracking problem when it is really a detection-coverage gap. The class list therefore directly bounds what your tracking stage can ever associate and follow. What common production objects are not covered by COCO’s 80 classes, and what do you do about the gap? Pallets, totes, cartons, machine parts, individual retail SKUs, defect regions, PPE beyond a coarse person class, and fine-grained vehicle sub-types are all absent or too coarse in COCO. The fix is a data decision: fine-tune on labelled domain examples, relabel to your own taxonomy, or add a fine-grained classifier behind a coarse COCO detection — not a tracker change. How do I tell whether a tracking failure is really a COCO class-coverage problem at the detector? Log raw per-frame detections before any tracking and count how many frames produced a box on the failing object. Near-zero detections point to a class-coverage problem; consistent detections with swapped or fragmented identities point to a genuine tracking bug. A coverage gap shows up as detection recall loss, a tracking bug shows up as ID switches. When should I use a COCO-pretrained detector as-is versus fine-tune or relabel for my scene? Use it as-is when your targets are COCO classes and standard granularity is enough. Add a fine-grained classifier when the class is right but you need finer distinctions, fine-tune when your objects resemble but aren’t COCO classes, and relabel to your own taxonomy when the objects are absent from COCO entirely. The deciding factor is class-scene fit, assessed before any retraining spend. Before you touch a single tracker parameter, ask the prior question: does COCO even represent the object you’re trying to follow? If the answer is no, no amount of association tuning will help — you have a class-coverage gap at the detector, and the artifact to reach for is a detection-stage class-coverage check against your deployment scene, not a tracking rewrite.
