“COCO labels” gets treated as a solved thing: download a dataset, point a detector at the JSON, and assume the box format is settled. That assumption holds right up until your production geometry stops looking like the upright, well-spaced objects the format was built around. Here is the core fact people skip. The COCO annotation bbox is a four-value axis-aligned rectangle — [x, y, width, height] — with no angle term at all. It cannot describe a rotated object. If your targets tilt on a conveyor, pack densely on a shelf, or lie at arbitrary orientations in an aerial frame, the stock format will faithfully record a box that is partly background and partly your neighbour’s object. The format is not wrong; it is answering a narrower question than the one you have. Treating the label format as a deployment-condition decision — not a download default — is what separates a project that trains the right model from one that re-annotates three months in. How do COCO labels work in practice? COCO — Common Objects in Context — is both a dataset and an annotation schema, and the schema is the part that outlives the dataset. When you say a model “expects COCO-format labels,” you are talking about a specific JSON structure that most object-detection toolchains read natively. A COCO annotation file is one JSON document with a few top-level arrays. The three that matter for detection are images, categories, and annotations. Each entry in images is an image record: an integer id, a file_name, plus width and height. Each entry in categories maps an integer category_id to a human-readable name (and an optional supercategory). Each entry in annotations ties one object instance to one image: it carries an image_id, a category_id, the bbox, an area, an iscrowd flag, and — for segmentation tasks — a segmentation field holding polygon points or a run-length-encoded mask. The design is relational on purpose. Categories are declared once and referenced by ID everywhere, so a category rename is a single edit rather than a find-and-replace across thousands of boxes. That indirection is also the first place projects break: if you remap category IDs during a dataset merge and the box geometry no longer lines up with the label, every downstream metric quietly reports the wrong thing. We see this pattern regularly when teams stitch two public datasets together without reconciling their category tables. What does a COCO annotation actually contain? It helps to look at a single annotation record rather than the abstract schema. A cat sitting in the middle of a 640×480 image might be encoded like this: { "id": 1042, "image_id": 17, "category_id": 17, "bbox": [220, 140, 180, 210], "area": 37800, "iscrowd": 0 } Read that literally. The object’s box starts at pixel (220, 140) — the top-left corner — and extends 180 pixels wide and 210 pixels tall. category_id: 17 points into the categories array, which in the standard 80-class set is the cat class. (For the full class list and what each covers, see the 80 COCO categories and their coverage.) area is the box area in pixels, used by the evaluation code to bucket objects into small, medium, and large. iscrowd: 0 says this is a single distinct instance; a value of 1 marks a region of many overlapping instances annotated as one blob, and it changes how matching is scored. Nothing in that record encodes orientation. There is no rotation angle, no corner list, no polygon that would let the box tilt. The rectangle is locked to the image axes. That is the schema working exactly as specified — and exactly where the trouble starts for non-upright scenes. How does the COCO bbox format store a box, and why is it axis-aligned? The bbox convention is worth stating precisely because two common formats look almost identical and are not interchangeable. COCO stores [x_min, y_min, width, height] in absolute pixels. The PASCAL VOC convention, by contrast, stores [x_min, y_min, x_max, y_max] — two corners, not a corner plus extents. YOLO’s text format normalises differently again: [x_center, y_center, width, height] scaled to [0, 1]. Feed VOC corners into code expecting COCO extents and every box silently balloons; the training loop runs, the loss decreases, and the model learns nonsense. If you are moving between these conventions, our walkthrough of the PASCAL VOC annotation format lays out the corner-versus-extent distinction that trips up most conversion scripts. Why axis-aligned at all? Because an axis-aligned bounding box (AABB) is the cheapest useful localisation primitive. Four numbers, trivial to store, trivial to compute intersection-over-union on, and it matches the assumption baked into most detector heads: predict box centre and extents relative to a grid cell or anchor. For the natural-image objects COCO was assembled from — people, cars, furniture photographed roughly upright — an AABB is a tight enough envelope. The box-to-object area ratio stays reasonable, so the box mostly contains the thing you care about. That reasonableness is a property of the scene, not the format. Rotate the object and the axis-aligned envelope has to grow to enclose it, dragging in background. This is decided architecturally in the model too, not only in the labels — the choice of AABB versus oriented geometry lives in the detection head, where AABB versus oriented-box geometry is set. What happens to COCO labels when objects rotate or pack densely? Consider a diagonal object — a wrench lying at 45 degrees, a ship in an overhead image, a component tilted on a moving belt. The tightest axis-aligned box around a rotated rectangle can be close to twice the area of the object itself. Everything inside that extra area is background or, worse, part of an adjacent object. Two things degrade at once. First, the model learns a loose association between “box” and “object,” because a large fraction of the box’s pixels are not the target — this is an observed-pattern effect: the tighter the true object fits its axis-aligned envelope, the cleaner the supervision signal, and diagonal objects break that. Second, in dense scenes the axis-aligned boxes of neighbouring objects overlap heavily even when the objects themselves do not touch. Non-maximum suppression, which uses box IoU to decide which detections are duplicates, then merges or suppresses genuinely separate instances. On a packed retail shelf or a full pallet, you lose objects not because the model failed to see them but because the box geometry cannot tell them apart. The rotated bounding box detection discussion goes deeper on exactly when orientation crosses from nuisance to blocker. This is the divergence point that matters. The label format is fine for the data COCO shipped. Your production geometry is a separate question, and the honest answer is often that a stock COCO annotation cannot represent the object you need to detect. Decision surface: can standard COCO labels represent your objects? Before you annotate a single frame, run your scene against this rubric. It resolves one facet — whether the axis-aligned schema can even encode your targets — and it is deliberately conservative. Scene property Standard COCO AABB is fine You need oriented boxes or a converted format Object orientation Roughly upright, small tilt Arbitrary rotation, 45°+ common Object aspect ratio Compact, near-square Long and thin (blades, tools, ships, text) Packing density Well-spaced instances Objects touch or overlap in the frame Box-to-object area ratio Box mostly contains the object Box drags in >30–40% background when rotated NMS behaviour Neighbours separable by box IoU Neighbour boxes overlap despite separate objects Downstream use Presence / counting is enough Pose, orientation, or tight measurement needed If your scene lands in the right-hand column on even two rows, decide the label format before annotation, not after. A dataset annotated as standard axis-aligned COCO boxes cannot train an oriented detector — the angle term is simply not in the data, and no amount of model tuning invents it. The measurable payoff of getting this right upfront is annotation cost avoided: you annotate once, in a format the target model can consume, instead of paying for a second full pass to add orientation. How do you extend or convert COCO annotations for oriented boxes? There is no single blessed oriented-COCO standard, which is part of why this decision gets deferred until it hurts. In practice, teams take one of a few routes. The most common extension keeps the COCO JSON structure and replaces the four-value bbox with a rotated representation — either [x_center, y_center, width, height, angle] or an explicit four-corner polygon [x1, y1, x2, y2, x3, y3, x4, y4]. Oriented-detection toolchains built around aerial and document benchmarks read one of these variants. The corner-list form has an advantage: it degrades gracefully, because an axis-aligned box is just a special case of a rectangle whose corners happen to line up with the axes. A second route derives oriented boxes from segmentation. If your annotations already carry a segmentation polygon or mask, you can fit a minimum-area rotated rectangle to the mask and emit that as the oriented box — OpenCV’s minAreaRect does exactly this. That is why some teams annotate masks first and generate both AABB and oriented boxes from a single source of truth. Promptable segmentation models make this cheaper than it used to be; our note on SAM-derived oriented boxes from promptable masks covers the workflow. The conversion direction that does not work is the one people hope for: you cannot upgrade axis-aligned boxes to oriented boxes after the fact. The information — the angle — was never captured. This is the whole reason the format is a deployment-condition decision. Getting it wrong means re-annotation, which is the cost the upfront choice exists to avoid. For the manufacturing case specifically, deciding whether a stock COCO label set can represent rotated conveyor parts is part of the line-side inspection feasibility work; our computer vision practice treats that as a scoping question, not an implementation detail. How are COCO category IDs and box geometry used in IoU and mAP? The label format is not just training input — it is also the ground truth every evaluation metric compares against. Two fields do the work: category_id and bbox. Intersection-over-union (IoU) is computed between a predicted box and a ground-truth box of the same category. It is the overlap area divided by the union area — a pure geometric ratio, which is exactly why axis-aligned versus oriented geometry matters here too. For rotated objects, axis-aligned IoU can report a “good” overlap between two loose boxes that both mostly contain background, flattering the model. Mean average precision (mAP) then thresholds IoU (commonly at 0.5, or averaged across 0.5–0.95) to decide which detections count as correct, and averages precision across categories. If category IDs are mismapped, mAP silently scores predictions against the wrong ground truth. If box geometry is loose because the object was rotated, mAP under-reports localisation quality even when the model found every object. Our breakdown of precision, recall, mAP and IoU for inspection works through how the threshold choice changes what the number means. The practical implication: correctly mapped category IDs and appropriately tight box geometry feed directly into whether mAP tells you the truth about localisation. Both come from the label format. Get the format wrong and the metric that is supposed to tell you the model works becomes unreliable in ways that are hard to notice. FAQ What should you know about coco labels in practice? COCO labels are a JSON annotation schema — not just a dataset — that most object detectors read natively. In practice it means three linked arrays: images (image records), categories (integer IDs mapped to class names), and annotations (each tying one object to one image via image_id, category_id, and a bbox). The format is the interface your toolchain expects, so its limits become your project’s limits. What does a COCO annotation actually contain — images, categories, and bounding boxes? Each annotation record carries an image_id, a category_id referencing the categories table, a four-value bbox, an area (used to bucket objects by size), and an iscrowd flag; segmentation tasks add a segmentation polygon or mask. Categories are declared once and referenced by ID everywhere, which makes renames cheap but makes ID remapping during dataset merges a common source of silent errors. How does the COCO bbox format store a bounding box, and why is it axis-aligned by default? COCO stores [x_min, y_min, width, height] in absolute pixels — a top-left corner plus extents, with no angle term. It is axis-aligned because an AABB is the cheapest useful localisation primitive: four numbers, trivial IoU, and it matches the assumption in most detector heads. That is tight enough for the roughly-upright natural images COCO was built from, but the reasonableness is a property of the scene, not the format. What happens to COCO labels when your objects are rotated or densely packed? The tightest axis-aligned box around a rotated object can enclose nearly twice its area, dragging in background and weakening the box-to-object association the model learns. In dense scenes, neighbouring objects’ axis-aligned boxes overlap even when the objects do not, so non-maximum suppression merges or suppresses genuinely separate instances — you lose objects to geometry, not to a model failure. How do you extend or convert COCO annotations for oriented (rotated) bounding box detection? Most teams either replace the four-value bbox with [x, y, w, h, angle] or a four-corner polygon, or derive oriented boxes by fitting a minimum-area rotated rectangle to an existing segmentation mask (OpenCV’s minAreaRect). The conversion that does not work is upgrading axis-aligned boxes to oriented ones after the fact — the angle was never captured, so the only fix is re-annotation. How are COCO category IDs and box geometry used in IoU and mAP evaluation? IoU is the geometric overlap between a predicted and ground-truth box of the same category_id; mAP thresholds IoU (at 0.5, or averaged 0.5–0.95) and averages precision across categories. Mismapped category IDs score predictions against the wrong ground truth, and loose boxes from rotated objects make mAP under-report localisation quality — so both fields of the label format directly determine whether the metric is trustworthy. Do off-the-shelf detectors expect COCO-format labels, and what changes if you need oriented boxes? Yes — most stock detection toolchains read COCO-format JSON natively, which is why it became the default assumption. If you need oriented boxes, you leave that default: you adopt an oriented-detection toolchain, switch the annotation to a rotated representation, and choose a detection head that predicts an angle. That is an architectural and data decision made before annotation, not a switch flipped at training time. Where the decision actually lands The mistake is rarely misreading a JSON field. It is assuming the format is neutral — that “COCO labels” describes any scene you point it at. The schema encodes a worldview: objects are upright, spaced out, and well-approximated by an axis-aligned rectangle. When your production geometry disagrees, the format does not warn you; it just quietly captures background and lets your metrics look fine while your model learns the wrong association. So the question to ask before you annotate is not “what format is the dataset in” but “does the axis-aligned box describe the object I need to detect.” If the answer is no, the label format is the first line item in the feasibility assessment, not a download default — and resolving it early is what keeps a rotated-target project from paying for annotation twice.
