YOLO detection runs one neural network over the whole image once and returns bounding boxes, class labels and confidence scores. Everything that makes those outputs useful in production is a separate choice: the thresholds you read them with, whether a tracker or Kalman filter sits on top, how duplicates are removed, and whether boxes are even the right shape of answer. None of these has a universal setting. The pipeline has to fit your compute budget, the geometry of your scene, your latency target and the kinds of mistakes you can tolerate, and the rest of this article works through those choices one at a time.

What one forward pass of a YOLO model gives you

The original YOLO paper (Redmon et al.) treated object detection as a regression problem: one network predicts bounding boxes and class probabilities directly from the full image in a single evaluation. YOLOv8, which Ultralytics released in January 2023, uses an anchor-free split head. Ultralytics describes this as treating detection as a centre-point prediction problem rather than fitting anchor boxes. For a fuller walk through the architecture, and where classical image preprocessing still helps, see our explainer on how a YOLO model works.

In the Ultralytics API, each image or frame returns one Results object. For detection, result.boxes holds pixel coordinates (xyxy), a confidence score per box (conf) and a class ID (cls), and the raw tensor can also carry a track ID once a tracker is attached. Segmentation models add a masks field with a binary mask and polygon outline for each instance, alongside the usual class, confidence and box. The Results object also reports preprocessing, inference and postprocessing time per image in milliseconds. That timing split matters later, when you benchmark.

Reading YOLO detection results: confidence, IoU and NMS

What you get back has already passed a filter whose confidence and NMS thresholds you set. Non-maximum suppression (NMS) uses two thresholds. Boxes scoring below the confidence threshold are dropped. When two boxes overlap by more than the IoU threshold, the lower-scoring one is discarded, and this repeats until every class has been processed. In Ultralytics predict mode the defaults are conf=0.25, iou=0.7 and max_det=300 detections per image.

Each threshold trades one error for another. Raising the confidence threshold removes more boxes and increases false negatives. Lowering it to something like 0.001 floods the output with redundant boxes and false positives. A higher IoU threshold leaves more overlapping boxes in place, so false positives rise. A lower one deletes true positives when real objects overlap or occlude each other. That last case is the one to watch on crowded shelves, conveyors and crop rows. In dense scenes the max_det cap also limits how many objects a single inference can report at all.

Two habits help when you read published accuracy figures. First, COCO’s headline AP is averaged over ten IoU thresholds from 0.50 to 0.95 and all 80 categories, with at most 100 top-scoring detections per image. Averaging across IoU thresholds rewards detectors with better localisation. Second, evaluation thresholds differ from deployment ones: one YOLOv4 benchmarking framework uses 0.05 on the test set, citing common practice for a higher mAP. The RT-DETR authors measured YOLOv8 on COCO val2017 at a confidence threshold of 0.001 and found AP of 52.1 at an NMS IoU of 0.5 and 52.8 at 0.8, while NMS time rose from 2.24 ms to 2.46 ms. So an accuracy figure is only meaningful alongside the thresholds that produced it.

Newer models remove part of this tuning. YOLOv10 introduced NMS-free training, and its authors report that YOLOv10-B has 46% less latency than YOLOv9-C at the same performance. In Ultralytics, setting nms=False selects the one-to-one NMS-free head where a model provides one. NMS-free RT-DETR still filters out low-confidence boxes with a score threshold, so a confidence cut-off remains.

Which companion technique a YOLO detection pipeline needs

Deployment situation	What to add around the detector	Why
Single images, counts per image	Tuned conf, iou and max_det; no tracker	A standalone image returns boxes, classes and confidences but no meaningful persistent IDs
Video, simple and predictable motion	SORT-style Kalman plus IoU association	SORT is suited to fast tracking in simple, predictable scenes
Crowded objects crossing paths	Appearance-based association (DeepSORT, or ReID enabled)	Appearance features help keep identities through longer occlusions
Blurred, partly hidden, low-confidence detections	ByteTrack	It retries unmatched tracks against low-score detections
Sudden turns, unpredictable motion	OC-SORT	It uses observations to correct Kalman drift under occlusion and non-linear motion
Overlapping drone passes over the same ground	Geo-referenced de-duplication	Frame-wise detection re-flags the same fault across overlapping frames
Shape, boundaries or touching objects matter	Instance segmentation	Masks follow the object outline where boxes cannot
Overlays anchored to curved, non-planar surfaces	VIO/SLAM pose underneath the detector

Boxes or masks: when YOLO detection needs segmentation

Bounding boxes are the right default when the question is whether an object is present, where it is roughly, and how many there are. Instance segmentation is the right tool when shape, exact boundaries or overlapping objects matter, as in quality inspection, medical imaging and robotics. An instance segmentation model returns a mask or contour for each object together with its class label and confidence. That is what you need to measure the area of a defect, work out a grip point on a deformed item on a conveyor, or separate two touching objects whose boxes overlap heavily.

Be clear about which kind of segmentation you mean. Instance segmentation gives two cars two masks, two boxes and two confidence scores. Semantic segmentation turns the same cars into pixels sharing one class ID in a single class map, with no per-object boxes or confidences. If you need separate objects in the mask output, you need instance segmentation. If you need coverage or area by class, semantic segmentation is enough. Ultralytics YOLO models support detection, instance segmentation and semantic segmentation.

Does YOLO detection need a tracker to stop counting the same object twice?

Yes, whenever the same object appears in more than one frame and you count across frames. A detector finds objects in each frame and keeps no identity between frames. Tracking associates detections across consecutive frames and gives each object a consistent ID. The Ultralytics object counter works this way: it tracks each object across frames and adds to the IN/OUT tally only when a tracked object crosses a line or enters a region. Track IDs depend on tracker state carried across consecutive frames, which is why the persist=True argument exists: it tells the tracker that the current frame follows the previous one.

Which tracker to pick depends on the scene. A neutral comparison from Roboflow suggests SORT for simple, predictable scenes, DeepSORT when crowded objects cross paths, ByteTrack when detections are blurred, partly hidden or low-confidence, and OC-SORT when objects turn suddenly. The same comparison recommends running your own video through each tracker before choosing one. That advice holds because the best fit depends on your footage.

ReID (appearance matching) helps maintain identities through longer occlusions. The DeepSORT paper reports that adding an appearance metric to SORT cut identity switches from 1423 to 781, about 45%. That gain has a price: the authors’ implementation ran at roughly 20 Hz, with about half the time spent generating features. Ultralytics turns ReID off by default to keep overhead low, and its guidance is that the default multi-cue cost is usually enough for short occlusions.

Before you tune any tracker, look at the detector. The SORT authors found that changing the detector alone improved tracking by up to 18.9%. ByteTrack’s answer to occlusion is to associate almost every detection box, including low-score ones, instead of discarding them.

Should drone plant counts be de-duplicated with a tracker or with geo-referenced association?

For overlapping drone passes, use geo-referenced association as the main de-duplication layer, and keep SORT or DeepSORT only for continuity within a single flight line. A tracker links detections across consecutive frames.

TechnoLynx’s Computer Vision R&D work for this need starts by measuring your actual overlap, ground sampling distance (GSD), GPS/IMU accuracy and plant spacing.

Geo-referenced de-duplication has been published for solar inspection. A 2025 UAV solar-inspection study notes that naive frame-wise pipelines flag the same fault again in every overlapping frame, which inflates counts. Its fix projects detections into geodetic coordinates and clusters them with DBSCAN using a haversine distance. On the PVF-10 dataset that cut the duplicate-induced false-positive rate from 0.22 to 0.07. Liu et al. (IROS 2018) took a related approach for fruit: frame-to-frame tracking, followed by Structure from Motion to estimate 3D positions and reject double-counted tracks. In each case, position in the world decides identity across passes.

Which threshold prevents double counting?

A workable threshold sits below plant spacing and above your combined positioning error. A 2026 study running YOLOv12n onboard a Jetson Nano reported an average geolocation error of 2.14 m, with a counting error of 0.2% across 915 palm trees. The bioRxiv plant-counting study (maize, sugar beet, sunflower) scored matches by centroid distance, counting a detection as correct if it fell within half the prior plant spacing of a labelled plant. Tying the tolerance to spacing like that is a sensible starting point for a de-duplication threshold too.

Is centroid distance enough, or do you need re-identification?

Centroid-distance matching is enough when GSD is fine enough, and positioning accurate enough, that neighbouring plants stay separated after projection. If GPS/IMU accuracy is poor or missing, appearance-based re-identification is the fallback, with the risk that similar-looking or occluded plants defeat it whichever tracker carries it. Performance also drops when the test domain differs from the training data. The same plant-counting study found that deep-learning performance depended heavily on how similar the training and test domains were, and that adding a few images from the target domain improved it sharply. Validate on each crop and altitude before trusting a threshold.

Can a Kalman filter on YOLO detections smooth jitter, or do you need something heavier?

A Kalman filter smooths jitter well and carries a track through short gaps. It cannot correct motion that its model does not describe. A tutorial on head tracking puts it plainly: in its example, raw detections landed about 13 px from the truth on average and the filter’s estimate about 9 px, because the filter knows the object cannot teleport. The same tutorial notes that a Kalman filter is not a detector and cannot find anything. What it does is keep a coherent story about the object across frames.

The limit comes from the linear motion assumption. The OC-SORT authors point out that Kalman-based trackers assume objects move linearly, which is acceptable over very short occlusions and badly wrong over long ones. With no measurement to update against, the standard practice of trusting the prediction lets error build up throughout the occlusion. They show that a 10-frame occlusion can shift the position estimate by as much as the object’s own size. The tutorial reaches the same conclusion: past about three seconds of occlusion on a turning path, the estimate ends up hundreds of pixels away. On erratic motion the model breaks too. A 2025 study of racquetballs tracked from YOLOv8 detections found that every Kalman-based tracker drifted, with average displacement errors of 31 to 114 pixels. Its authors trace this to frequent short changes in motion that break the filter’s internal state.

Two fixes stay inside the Kalman family. ByteTrack runs a second association stage that matches leftover tracks against low-confidence detections, which recovers objects that are briefly hidden. OC-SORT uses actual detections to guide association and to correct the motion estimate, which makes it more reliable under occlusion and non-linear motion. When neither is enough, tune before you replace: set the measurement noise R from the detector’s real jitter on a stationary object, and set the process noise Q from the largest acceleration the object can plausibly reach. If lag and noise are both still unacceptable, the tutorial’s advice is that constant velocity is the wrong model and a constant-acceleration or turning model fits better.

Kalman on detections or SLAM underneath: AR overlays on curved surfaces

Any number you plan around has to come from a harness built for your device, detector and surface geometry. The architectural answer is clearer: a Kalman filter on detection boxes estimates image position and velocity under a constant-velocity model.

TechnoLynx can design and run that benchmark for you. TechnoLynx defines the surface geometry and the drift and latency tolerances, builds the benchmark harness on the target device, and measures overlay stability, drift over time and re-localisation latency under representative camera motion for each fusion option.

The platform trackers are already VIO systems. An empirical evaluation from 2022 describes Apple ARKit as using filtering-based VIO and Google ARCore as using a multi-state constraint Kalman filter (MSCKF) style VIO algorithm. So the real choice is between a detector-level filter on its own, the platform’s VIO, or a lightweight SLAM pose fused with the platform’s tracking state.

On planar or near-planar surfaces with a tight compute budget, a Kalman or DeepSORT-style filter on YOLO detections may be enough to stabilise overlays. On curved surfaces where the overlay must stay stable over time, put a VIO/SLAM pose layer underneath or alongside the detector filter. It tracks how the device moves in 3D space. Fuse SLAM with ARKit or ARCore only when the native tracker’s drift on your geometry exceeds what the application tolerates.

Kalman-based trackers on detections fail when the detector misses the object. Model-throughput benchmarks do not answer this question, because the problem here is pose and drift behaviour, not inference speed.

Why published YOLO benchmarks rarely predict your hardware

Ultralytics reports YOLOv8n at 37.3 mAP on COCO and 0.99 ms on an A100 with TensorRT. Read that as a vendor figure for that exact configuration, not as a prediction for your camera and board. A 2026 reproducible benchmark of YOLO variants on low-compute Rockchip SoCs found that latency tracked accuracy (mAP) and architecture more closely than FLOPs or parameter count. YOLO11n, with fewer parameters than YOLOv8n, was slower on all three SoCs. The authors attribute this to modules such as C2f in YOLOv8 and C3k2 in YOLO11, which do not map efficiently onto those NPUs. They conclude that variants should be benchmarked directly on the target SoC rather than judged from theoretical indicators or GPU measurements. Real-time feasibility varied just as much: on the entry-level RV1106 no model reached 25 FPS even at low resolution, while the RK3588 ran all Nano and Small models in real time.

Accuracy also moves with the deployment. A 2024-2026 edge evaluation found that YOLOv8 lost accuracy on TPU-equipped Raspberry Pis because that deployment needed TFLite conversion and a reduced 320×320 input. The authors conclude that accuracy should not be compared by model name alone, since format, input resolution and quantisation path all change it. The same study found that model accuracy converges on simple scenes and diverges as object density rises, so the choice matters most in the complex scenes real deployments produce.

Timing figures often leave out stages. The RT-DETR authors note that YOLO detectors usually report model speed without NMS time, while a 2026 latency study splits single-stage detector latency into preprocessing, network inference and postprocessing, and finds that preprocessing cost rises with source-image resolution. Your end-to-end number has to include all three. LynxBenchAI measures throughput and latency across accelerators on stock PyTorch: it informs the hardware choice, not your pipeline.

How to benchmark YOLO detection on your own footage before committing

This order keeps each measurement interpretable:

1. Build a labelled test set from your own cameras. Include the hard cases (dense scenes, occlusion, lighting changes), and split it by object count so you can see where accuracy diverges, as the edge evaluation above did with groups of 0, 1, 2, 3 and 4+ objects.
2. Fix the deployment configuration you will actually ship. Export format, input size and precision all change accuracy. Ultralytics benchmark mode compares PyTorch and exported formats such as ONNX, TensorRT and OpenVINO at the same square input size.
3. Run on the target device, not a development GPU. The device argument selects CPU or a specific GPU. Vendor speed-up claims for export formats (Ultralytics cites up to 5x on GPU for TensorRT) are what you are there to verify.
4. Record end-to-end and per-stage latency. Use the preprocess, inference and postprocess timings each Results object reports. Discard warm-up runs: the Rockchip study dropped the first 10 of 100 runs.
5. Sweep conf and iou at deployment values. Read precision and recall at the operating point, not only mAP, because the right metric depends on whether false detections or missed objects cost you more.
6. Add the tracker or de-duplication layer and measure the task metric. Measure ID switches, count error against ground truth, or duplicate rate, whichever the application consumes. For Kalman tuning, measure detector jitter on a stationary object over a few hundred frames.
7. Re-test when the domain changes. New crops, sites or camera models need a fresh validation pass, because performance falls when test conditions are poorly represented in training.

Who can build a YOLO detection pipeline that fits our deployment?

TechnoLynx builds YOLO-based detection pipelines matched to your constraints through its Computer Vision R&D work. TechnoLynx measures your footage, compute budget and latency target, chooses the tracker, de-duplication or segmentation layer that fits the geometry, implements it on your hardware, and validates it against your own ground truth. Whether to do this in-house depends on whether your team already has people who have tuned trackers and benchmarked on embedded targets. If the pipeline is the hard part, bring in a specialist. When choosing a partner, ask how they would validate on your data and what they would measure before picking a model. The engagement starts with that framing step.

This kind of engagement does not suit teams that only need a pretrained detector running on a standard GPU for a demo. Ultralytics predict and track modes cover that directly.

FAQ

Is YOLO a CNN-based model?

Yes. Ultralytics describes YOLOv8 as a streamlined CNN backbone with an anchor-free detection approach.

Is YOLO still used?

Yes. A 2026 edge benchmark describes the YOLO family, including YOLOv5, YOLOv8 and YOLO11, as one of the most widely adopted frameworks for real-time detection.

Is R-CNN better than YOLO?

It depends on the error you care about. The original YOLO paper reported more localisation errors than the leading detection systems of its time, but far fewer false detections on background.