How are rotating bounding boxes different from standard bounding boxes? A standard bounding box (the axis-aligned box most detectors ship with) is a rectangle whose sides stay parallel to the image edges. A rotating, or oriented, bounding box adds one angle so the rectangle can turn with the object. That one parameter brings new challenges, such as feature misalignment and angle periodicity, and needs its own CUDA operators for IoU and NMS. Rotation tends to pay off when objects sit at arbitrary angles, pack tightly or are long and thin. Aerial imagery, retail scenes and industrial inspection are among the scenes where oriented detection is preferred. For upright, well-spaced objects we would keep the axis-aligned box.
How is a rotating bounding box different from a standard, axis-aligned one?
PyTorch’s torchvision NMS operator expects axis-aligned boxes as two corners, (x1, y1, x2, y2). An oriented bounding box (OBB) is most often written as (x, y, w, h, θ): the horizontal box plus an angle. The alternative is four corners in clockwise order, which is how the DOTA aerial benchmark annotates objects. Ultralytics YOLO uses both: labels are four normalised corner points, while losses and outputs are processed internally as xywhr (centre, width, height, rotation).
A horizontal box drawn around a rotated object also takes in background, and that makes it harder for the network to pick out the features it needs for classification and regression. The authors of Oriented R-CNN put it bluntly: a horizontal box may contain more background or several objects, so classification confidence and localisation accuracy stop agreeing. In dense arrangements the IoU between neighbouring horizontal boxes can be very large, especially for objects with extreme aspect ratios, and non-max suppression then causes missed detections. The DOTA authors describe cases where the overlap between two horizontal boxes is so large that detectors cannot tell the objects apart.
Worked example: a 10:1 part lying at 45 degrees
Take a rectangle w long and h wide, rotated by θ. Its axis-aligned box is (w·cosθ + h·sinθ) wide and (w·sinθ + h·cosθ) tall. For a part 10 units long and 1 unit wide at 45°, each side comes to 11/√2 ≈ 7.78, so the box covers about 60.5 square units while the part itself covers 10. An oriented box of the same part covers exactly 10 units, so the object fills all of it. A second identical part lying parallel one unit away would fall largely inside the first part’s axis-aligned box, and the two boxes would overlap heavily. This is the dense-arrangement overlap that makes NMS delete real detections.
How should an oriented box be parameterised, and why does the angle destabilise training?
There are two families. Five-parameter protocols add an angle θ to the horizontal box, and eight-parameter protocols regress the four corners (x1, y1 … x4, y4) as a quadrilateral, which only works if the corners are sorted in a fixed order beforehand. The same physical box gets noticeably different parameter values under different representations.
| Tool or method | Representation | Angle convention |
|---|---|---|
OpenCV minAreaRect |
RotatedRect | Angle falls in [-90, 0); past a right angle the next edge is used |
| OpenCV-style definition (survey) | (x, y, w, h, θ) | Acute angle to the x-axis, θ in (0, π/2]; w may be the shorter side |
| RRPN | (x, y, h, w, θ) | h is the short side, w the long side, θ from the x-axis to the long side |
| Oriented R-CNN | (x, y, w, h, θ) | θ in [-π/2, π/2], measured to the longer side |
| Ultralytics YOLO OBB | Four-corner labels, xywhr internally | Defined modulo 180°; labels canonicalised to long-edge form, predictions not |
Detectron2 RotatedBoxes |
(x_center, y_center, width, height, angle) | Five values in that fixed order |
| O²-DETR (2026 paper) | (cx, cy, w, h, θ) | θ in [0, π) relative to the x-axis |
| DOTA annotations | Four vertices | Clockwise; first point marks the “head” where the object has one |
Boundary discontinuity is the sharp jump in loss at the boundary, caused by how the angle and edges are parameterised. The angle difference between two boxes increases sharply as the angle approaches 0 or π/2. The prediction’s width may also be the short edge while the ground truth’s width is the long edge, which produces a large width and height loss as well. Near-square objects have their own version: if width and height are swapped, the angles differ by about π/2 and the loss is large even when the rotated IoU is close to 1. The corner representation does not escape this: there the boundary problem comes from corner ordering. The authors of Circular Smooth Label (CSL) show that the root cause of the boundary problem is that the ideal prediction can fall outside the defined range.
Each remedy changes something different. CSL turns angle prediction into classification over discrete bins, with labels that wrap around so neighbouring angles stay close. Gaussian methods (GWD, KLD, KFIoU) convert the box into a 2-D Gaussian, encoding the angle through trigonometric functions that the periodicity does not affect. Under the GWD loss the OpenCV and long-edge definitions become equivalent, so the method does not depend on which box definition you chose. H2RBox-v2 uses a “snap loss” that keeps the angle difference within π/2. Its authors report that without boundary handling the loss fluctuated widely and sometimes failed to converge. The 2026 YOLO26 analysis describes a dedicated angle loss for the same purpose, meant to hold up even on near-square objects.
A practical risk is a convention mismatch between the annotation tool, the training framework and the evaluation script. Ultralytics warns, for example, that result.obb.xywhr can report a width smaller than the height, with the angle measured from the short edge, while the corner output describes the same rectangle either way. Pick one convention per pipeline and convert explicitly wherever data crosses a boundary.
Why can’t standard NMS be reused for rotated boxes?
Standard NMS assumes axis-aligned geometry. torchvision’s nms only accepts boxes as (x1, y1, x2, y2) with x1 < x2 and y1 < y2, and it discards any box whose IoU with a higher-scoring box exceeds the threshold. You could pass it the axis-aligned box around each rotated prediction, but that brings back exactly the inflated overlaps that rotated boxes were meant to remove. Rotated IoU needs the intersection of two rotated boxes. OpenCV’s rotatedRectangleIntersection returns the vertices of that intersecting region. The DOTA toolkit ships OBB IoU in CPU and GPU versions, and MMRotate lists IoU and NMS for OBBs among the CUDA operators that rotated detectors use.
Rotated IoU is also sensitive to small angle errors. In the RRPN paper, an anchor with a 1:8 aspect ratio and an angle error of π/12 scores an IoU of only 0.31, far below a 0.7 threshold, even though it could reasonably count as a positive match. The CSL authors make the same point: a slight angle deviation causes a notable IoU drop, especially for large aspect ratios. For that reason, IoU-only NMS with a 0.7 threshold is insufficient for oriented proposals. Oriented R-CNN runs a cheap horizontal NMS at IoU 0.8 on first-stage proposals to save time, then a polygon NMS at 0.1 on the final oriented boxes.
Rotated NMS needs both a rotated IoU threshold and a confidence threshold. It adds latency, and because oriented boxes are sensitive to IoU changes, the results depend on the threshold you set. Training is affected too. IoU-based losses such as GIoU and DIoU do not transfer directly, because rotated IoU is not differentiable. Computing it involves custom operations, such as edge intersection and vertex sorting, whose derivatives standard deep learning frameworks have not implemented. Evaluation changes in the same direction: oriented detection is scored with rotated IoU instead of IoU. OBB mAP usually comes out slightly below HBB mAP for the same algorithm, because the oriented task demands tighter localisation. Comparing an OBB score with an AABB score as if they measured the same thing will make rotation look worse than it is.
What extra cost does oriented detection add in annotation, training and inference?
Annotation. Rotated-box and mask labels cost more to produce than horizontal boxes. Moving existing datasets to rotated boxes has meant labour-intensive re-annotation: DIOR-R relabelled the aerial DIOR dataset (192K instances) and SKU110K-R did the same for the SKU110K retail dataset (1,733K instances). DOTA annotators had to pick the corner that marks an object’s “head” for categories that have one, and default to the top-left point for categories that do not.
Training. Backbones built on axis-aligned convolutions produce features misaligned with oriented objects. Horizontal anchors overlap poorly with elongated rotated objects, so label assignment struggles to find enough positive samples. Rotated anchors fix the overlap at a price: rotated RPN places 54 anchors per location (3 scales × 3 ratios × 6 angles), which its successors describe as causing massive computation and memory use. RRPN reports nearly six times the proposals of Faster R-CNN and about twice the runtime. The CSL angle fix adds parameters and computation of its own. CSL-based detectors added about 82.96% to GFLOPs and 45.63% to parameters, against 3.24% and 0.92% for the denser DCL encoding. Rotation augmentation also matters: in the DOTA v2 experiments, scale and rotation augmentation improved results by a large margin even with a feature pyramid network and RoI Transformer already in place.
Inference. YOLO26 is designed so oriented boxes add no significant latency penalty. Rotated NMS adds latency. The YOLO26 analysis credits its NMS-free head with a 43% inference speedup over heuristic rotational NMS baselines; that figure is the authors’ own comparison. Real-time oriented detection has been reported: the O²-DFINE-s model in the same 2026 transformer paper reaches 297 FPS with 10M parameters at 1024×1024 input on an NVIDIA GPU. On your hardware, measure the real cost rather than assume it.
Can off-the-shelf detectors output oriented boxes, or do you need a custom model?
Oriented architectures are available off the shelf. Ultralytics YOLO has a dedicated OBB task that outputs rotated boxes with class labels and confidence scores. MMRotate implements 18 rotated detection algorithms, supports the three most common angle definitions and publishes trained models and benchmarks. Support has not always been there: the DOTA v2 authors had to add an OBB head to MMDetection because the original lacked oriented detection modules.
The weights are where the gap opens. Ultralytics describes its OBB datasets as tailored to aerial and satellite analysis. We would not expect a model pretrained on aerial and satellite data to know the parts on your production line. One route is fine-tuning an existing OBB architecture on your own rotated labels, not designing a new detector. One study did exactly this kind of repurposing, using YOLOv11-OBB for robotic grasp detection. Whether fine-tuning is enough, or the problem needs a model built around it, is the broader question of custom vs off-the-shelf computer vision. Where the angle is predicted inside the network is covered in our explainer on the detection head and where AABB vs oriented box geometry is decided.
Can you derive oriented boxes from masks or polygons you already have?
Yes, if you already hold instance masks or polygon annotations. OpenCV’s minAreaRect returns the minimum-area rotated rectangle around a point set, and boxPoints gives that rectangle’s four vertices, which is already the four-corner layout of the YOLO OBB label format. One catch: minAreaRect reports its angle in [-90, 0), so normalise it to your training framework’s convention before writing labels. Datasets in the COCO format often store segmentation polygons alongside their axis-aligned boxes. Our guide to COCO labels and the annotation format behind bounding-box detection explains where those polygons live.
Generating masks just to get rotated boxes is a different matter. The H2RBox authors argue that an HBox-to-mask-to-RBox pipeline can be complex and expensive when objects are dense and backgrounds noisy, and that segmentation can be harder than rotation detection. Whether promptable segmentation models change that trade-off is worked through in our piece on SAM models for segmentation-derived oriented boxes. If you only have horizontal boxes, weak supervision is an option. H2RBox-v2 learns rotated boxes from horizontal labels using object symmetry, and on DOTA it came within 0.13% of a rotated-box-supervised FCOS baseline without multi-scale or rotation augmentation. It relies on symmetry, though, and occlusion makes that symmetry harder to exploit. In the other direction, DOTA’s horizontal ground truth was computed as the axis-aligned boxes around its oriented annotations.
Which production conditions make rotated object detection worth it?
Overhead and aerial views. Seen from above, objects can take any orientation. In DOTA, object angles are spread evenly across [-π, π]. In gravity-bound domains such as scene text and faces, most angles fall in a narrow band. Aerial frames are large, and the O²-DETR authors crop DOTA images into 1024×1024 patches with 200-pixel overlap. Our guide to patching large images for detection covers the mechanics.
Dense packing. Faster R-CNN OBB beats horizontal Faster R-CNN by 8 AP on large vehicles in DOTA-v1.0, because the horizontal model’s NMS suppresses tightly parked neighbours. DOTA patches range from a single instance up to about 1,000. If a downstream object tracker associates detections by box overlap, we would expect heavily overlapping axis-aligned boxes in a packed scene to weaken association too.
Elongated objects, with a caveat. The rotated IoU of long objects such as bridges and ships is highly sensitive to orientation error. The original DOTA paper also found these objects hard to regress in OBB form, while their axis-aligned versions had ordinary aspect ratios and looked fairly good. Long, self-similar objects can also attract several predictions on a single instance. Rotation helps elongated objects, but the angle has to be learned well before that benefit appears.
Conveyors and picking. One study used the centre of each YOLOv11-OBB box as the grasp position and its angle as the grasp orientation, all from one forward pass, and pointed to sorting on conveyor belts as the industrial case. Industrial inspection is also on the H2RBox-v2 authors’ list of complex scenes where oriented detection is preferred, alongside autonomous driving, which our explainer on autonomous car perception and detection covers. Where the question is only whether a known component is rotated correctly, as in component orientation checks in PCB AOI, a pose formulation can fit better than a general detector.
Burrs, tool marks and pits: building an orientation-aware defect taxonomy
On machined and rotating parts, we would give oriented measurement to the elongated defect classes that can appear at any angle to the part’s axis: burrs, tool marks and shavings. We would keep axis-aligned boxes for compact, roughly round defects such as pits and inclusions, whose axis-aligned box changes little with rotation. The geometry explains the split. An oriented box reports the scratch’s own length, width and angle. For a near-square pit box, swapping width and height gives a large loss even when the rotated IoU is about 1. Forcing those defects into oriented boxes adds labelling work and training noise without adding information.
Pass/fail thresholds follow from the same split. Once elongated defects are measured as oriented boxes, we would set thresholds on the oriented dimensions: maximum length, maximum width, and where it matters, angle relative to the part’s own reference axis (an edge, a bore, the turning direction) rather than the image axis. Measuring angle against the camera frame makes the threshold depend on fixturing, which defeats the point. Compact classes keep simple thresholds on area or maximum dimension.
We would build three caveats into the protocol from the start. Borderline defects that are only slightly elongated will be classified inconsistently between the two conventions unless the taxonomy sets an explicit aspect-ratio cut-off. Oriented annotation costs more than horizontal boxes. We would treat published surface-defect class lists as a starting point to extend for cylindrical parts, not a one-to-one map. One published study comes close to this problem: Peng et al. (2026) on measurement-oriented micro-burr inspection compared YOLOv8 axis-aligned detection against segmentation, deriving the detection boxes from masks for a fair comparison. YOLOv8n-seg reached 0.969 [email protected] against 0.894 for YOLOv8n-det. We would take a mask-first route for burrs, fitting a minimum-area rectangle afterwards to get oriented length, width and angle. The same study reports, for its segmentation pipeline, 18.2 ms per image on an RK3588 NPU at 320×320, rising to 120.4 ms at 640×640, a reminder that resolution choices drive line-side latency. How we judge whether an inspection task is ready for a pilot at all is covered in when industrial computer vision inspection actually works. The inspection-specific case for rotation is argued further in rotated bounding box detection for inspection.
When should you use an oriented bounding box instead of an axis-aligned one?
Use an oriented box when orientation carries information, or when packing density makes axis-aligned overlap ambiguous. Otherwise keep the axis-aligned default.
| Deployment condition | Box geometry | Reason |
|---|---|---|
| Upright, well-spaced objects (gravity-bound scenes) | Axis-aligned | Most angles fall in a narrow band |
| Overhead or arbitrary orientation (aerial, top-down conveyor) | Oriented | Horizontal boxes absorb background that weakens features |
| Dense packing, especially of elongated objects | Oriented | Horizontal-box NMS suppresses real neighbours |
| The angle is the output (grasp pose, defect direction) | Oriented | Location and orientation come out of one forward pass |
| Compact, near-square objects | Axis-aligned | Angle regression is unstable on square-like shapes |
| Pixel-accurate shape needed | Segmentation, oriented box derived after | A mask head converges more easily and scores better, but costs more compute |
Before committing, measure both options on your own data: an axis-aligned baseline and an oriented model, compared with a rotated-IoU-aware metric and a latency figure that includes rotated NMS on your target hardware. If the oriented model only wins on headline mAP, keep in mind that OBB mAP usually runs slightly lower than HBB mAP for the same algorithm. The test that matters is whether downstream measurement, counting or picking actually improves. When the answer is yes, the project becomes a question of fine-tuning versus building your own. TechnoLynx’s Computer Vision R&D team builds and deploys object detection systems.
