Segmentation tracking keeps each segmented defect mask’s identity across frames. Each mask found on one frame is linked to the masks on the frames that follow, so each object keeps one consistent ID across frames. On a PCB automated optical inspection (AOI) line, that linking has an anchor most video trackers lack: the system already aligns every capture to a reference built from CAD/Gerber data or a verified golden board. We would bind masks to that reference so that tracked defect counts stay comparable with the false-call and escape baselines the line was validated against. The parent page covers AOI feasibility and false-call and escape costing. This page covers how the tracking works, how it breaks, and which signals show it breaking first.
One caveat applies throughout. The tracking studies cited here track objects such as people and cars in video, not boards on a conveyor. We read that evidence as describing mechanisms that carry over to AOI, and we treat its numbers as belonging to their own benchmarks, not to any PCB line.
How segmentation tracking works: masks, tracklets and assignment
Deep-learning methods for mask tracking follow tracking-by-detection: objects are detected in each frame, then joined into tracks as a post-processing step. Detections matched across consecutive frames form short “tracklets”, using cues such as position, appearance and motion. The matching is set up as a cost matrix between existing tracklets and new detections, and the Hungarian algorithm solves it as a bipartite assignment that minimises total cost.
In one well-documented design, the short-term step warps the previous frame’s masks with optical flow, computes mask IoU against the new masks, runs the Hungarian algorithm on the negative IoU, and connects only the matches above a threshold. That design makes the step class-agnostic on purpose, so a classification error between similar classes does not break the track. The PCB parallel is direct: a defect the classifier calls “insufficient solder” on one frame and “bridge” on the next should not become two defects, so we would keep identity and class as separate decisions.
A second cue is appearance. MOTSNet pools features under the instance mask to give each detection an embedding vector, then solves tracking as a series of linear assignment problems in that embedding space. Pooling under the mask matters because box features mix object and background, and only the foreground helps recognise the object across frames. The two tasks also help each other: in one study, adding segmentation to a tracker reduced drift and identity switches by a large margin over tracking alone.
How does a defect mask stay tied to one pad across frames?
A defect mask stays tied to one pad when the pad’s identity in the board reference, not its pixel position in the current image, is the key the tracker keeps. AOI already checks each component’s position, orientation and outline against a tolerance window for its type. So we would use registration to give every mask a footprint and pad, and the tracker’s job is to keep that binding as the image changes. On a board we would compute association overlap in registered board coordinates rather than raw image coordinates, so that a panel-index shift moves the reference and the mask together.
Within that frame, the association cues need gates, because a learned mask propagator is sometimes wrong. McByte, a tracker that fuses propagated masks with bounding boxes, makes the gates explicit:
- Visibility: if a tracklet’s propagated mask is not visible in the current frame, it gives no guidance.
- Confidence: the mask’s per-pixel probabilities are averaged, and if the average is below a threshold, association falls back to plain IoU.
- Box coverage: the share of the tracklet’s mask that falls inside the detection box.
- Fill ratio: the share of the detection box the mask covers, where a low value points to a noisy mask or to another object’s mask overlapping the box.
The mask has the best influence on a match when both coverage values are as close to 1.0 as possible. The McByte ablation shows why the gating matters. Using the mask as the only cue lowered IDF1 against the baseline, fusing it properly with IoU gave a significant gain, and adding the confidence check improved results further. For AOI, the lesson is that geometry from the board reference should stay the primary cue, and learned mask propagation should settle ambiguous cases, not overrule registration.
The lifecycle matters too. McByte creates a mask when a tracklet starts and removes it when the tracklet ends. On a board, we would map that lifecycle to the component’s presence across the inspection sequence, so a track closes when its pad leaves the last field of view that covers it, and not just because one frame missed it.
Why does one solder bridge become two masks, or two bridges one?
Machine-vision AOI is sensitive to ambient light, which distorts captured images and raises misjudgment rates. Mask propagators fail in a similar way: masks go missing under occlusion, small visible areas, crowding or poor lighting.
Fragmentation. When an occlusion splits one track into several tracklets, a long-term step has to link them back together. One method does this with a space-time memory network. It propagates one tracklet’s masks into another tracklet’s frames and links the two if the predicted heatmap overlaps the other tracklet’s mask enough. The same paper names the limit: if one tracklet shows only a car’s hood and the other only its trunk, they share no visible parts and cannot be linked. We would expect a glare band across the middle of a solder bridge to set up the same situation, with two halves sharing no visible region. That is why we would link fragments through their shared pad binding, not through appearance alone.
Linking thresholds. How far apart two fragments can be and still merge is a real trade-off. In MOTSFusion, merging tracklets quickly and dramatically improved results over the initial tracklets. Beyond a 25-frame gap, missing-detection recovery introduced more false positives, and beyond 60 frames the merging itself began to raise identity switches. Those numbers come from that paper’s video benchmark, not from a PCB line. On a PCB line we would set the gap from the line’s own capture sequence per board.
Merging. Ambiguity shows up when tracked objects sit close together, so their boxes overlap and their IoU scores look alike. We would treat fine-pitch pads with adjacent defects as fitting that description. Research benchmarks report the effect as over-segmentation, which counts how many segments were merged to cover each ground-truth mask.
Two more effects make both failures worse. Contaminated masks and short tracklets carry little information and unstable appearance, and very short tracks usually come from spurious detections, which is why MOTSNet drops tracks below a minimum length. Errors also persist. Once a propagation-based tracker accepts a wrong prediction, that error goes into memory and is carried forward. A frame-local matcher that assigns the wrong query at one frame cannot recover later, because each step looks only at the frame before it.
Keeping false-call and escape counts aligned with the AOI validation baseline
False call rate counts good boards wrongly flagged as defective, usually in defects per million opportunities, and escape rate counts real defects that AOI misses and that reach later test or the field. Loosening a system to cut false calls risks letting real defects through. One study compared AOI machines on 1,230 components using exactly these two numbers, Defects Escaped % and False Call PPM. How that trade-off is priced and reviewed belongs to the hub on AOI false-call and escape costing.
Tracking decides whether those counts mean what the baseline meant. We would expect a fragmented bridge logged as two defects to inflate the defect count. On a good board, we would expect the ghost fragment to become a false call. We would expect merging two adjacent defects into one mask to hide the second, an escape the per-pad record never shows. Benchmark metrics reflect the same effect: in volumetric mask IoU, identity switches and fragmented tracks shrink the accumulated intersection while the union stays large, so even well-segmented frames score poorly when trajectories break.
Production AI reliability is a separate engineering discipline from model accuracy, and its deliverables are the artefacts that catch regressions (eval harnesses, regression suites, drift detection, validation packages). For the wider set of artefacts that keep a model running, see our page on industrial CV inspection production reliability. A practical checklist for buyers asks how false calls are reviewed and recorded, and whether images are linked to serial numbers or batches.
What happens to tracking after a board revision or reflow change?
We found no published study that measures how mask tracking behaves across a board revision or a reflow-profile change, so this section is our engineering judgement, not a reported result. The fixed point is that AOI compares each capture with a reference built from design data or a golden board, and that wrong package type is itself a defect class AOI looks for.
We would expect a component-package revision to change the footprint outline and pad geometry the tracker uses as its identity key. Tracks bound to the old geometry no longer have a valid home, and any coverage gate tuned on the old outlines will start rejecting good matches. We would treat a revision as a new reference: rebind pad IDs from the new design data, re-tune the coverage thresholds, and start a fresh baseline period rather than comparing counts across the change.
A reflow change acts more quietly. We would expect it to change how solder fillets look, not where pads are, so registration holds while mask confidence and fill ratio shift. The stage matters as well: pre-reflow AOI checks presence, placement and orientation but cannot assess the final solder joint. Rotation and orientation checks have their own page on pose estimation in PCB AOI.
Which signals show segmentation tracking degrading before defects escape?
The earliest signs are in how stable the association is, not in the escape count, which arrives last. The signals below come from the tracking studies cited in the table. We would log each one per tracklet and group it by component class and panel position.
| Signal | What it measures | What degradation looks like |
|---|---|---|
| Mean mask confidence per tracklet | Average per-pixel probability of the propagated mask | More tracklets fall below the gate and drop back to IoU-only association |
| Box coverage and fill ratio | How well the mask and detection box agree | Values drift away from 1.0, a sign of noisy masks or overlap from a neighbour |
| Confidence (effective-IoU) variance | How stable the confidence trace is over time | Oscillating confidence before any hard loss of the target |
| Identity switches and fragments | IDS and Frag counts | More identity switches and trajectory fragments |
| Gap versus sequence length and density | Tracking error split by condition | Error growing on longer sequences and denser regions |
The variance signal is the one a cited study tests directly as an early warning. In the Dual-Agent study, healthy propagation produced a stable confidence trace, while identity swaps or partial occlusion produced oscillation before the target was lost. In that study’s trigger ablation, adding effective-IoU variance gave the largest jump in recovery volume.
Watch the trend over time, not one snapshot. The Mind the Gap video instance segmentation analysis found that online methods’ tracking gaps grow sharply with video length and instance density. For MinVIS, the gap rose from 17 AP at low density to 29 AP at high density, and under occlusion gaps exceeded 20 AP. The same analysis found that backbone scaling left tracking gaps intact, which points to association logic, not model size, as the fix. Per-run logging of these metrics is covered in W&B logging for AOI model reliability.
When the signals move, verify before you correct. In memory-based trackers, a wrong correction changes the memory later frames read from, so one bad reset can pollute a long run of frames. Dual-Agent called its verifier on only 4.82% of unique frames and still improved long-horizon tracking. The same failure pattern applies to SAM on an inspection line, because SAM-style trackers propagate masks through memory. If the fix is a retrained detector, we would re-establish the tracking baseline along with it, as covered in re-validating a fine-tuned YOLO model.
When is segmentation tracking worth it over per-frame segmentation?
Per-frame segmentation is enough when each region of the board is captured once and registration alone assigns every mask to a pad: no second view exists, so there is nothing to reconcile. Tracking pays for itself once the same pad shows up in more than one capture (angled lighting passes, overlapping fields of view, panel-index steps) and the counts feed telemetry someone trusts.
The evidence for the gain comes from several studies, none of them on PCBs. On KITTI MOTS, MOTSFusion had 61% fewer identity switches for cars than TrackR-CNN with the same detections and segmentations (36 against 93). Adding temporal context to association, through explicit memory or decoupled refinement, gave much smaller tracking gaps than pure frame-to-frame matching, and offline methods showed smaller gaps than online ones. Where a board’s full capture sequence is available before the verdict, we would run association over it as a batch rather than force it to be causal. At research scale, 2D tracklet generation in MOTSFusion took 0.07 seconds per frame, as reported in that paper.
Where parts really move past the camera, the design looks different. That case is covered in real-time object tracking for line-side inspection, and the same mask machinery appears with heavier occlusion in segmentation tracking in automotive perception.
Who can build segmentation tracking into an existing AOI pipeline?
TechnoLynx builds tracking into computer vision systems through its Computer Vision R&D work. The work draws on Production AI Reliability artefacts, such as regression suites and drift detection, to keep tracked counts comparable with the validation baseline. In choosing a partner, we would look for one who validates against your escape and false-call baselines, not against mask IoU on a staged set.
If the AOI runs only the vendor’s rule-based inspection with one capture per region, we would treat tuning by the equipment supplier as the more proportionate route.
Identity, not mask IoU, is what the baseline measures
A tracker can post excellent mask IoU on a staged set and still drift away from the AOI baseline. The baseline counts false calls and escapes, and we hold that stable identity is what keeps those counts honest. In our view, the real test is the first production change the staged set never contained, such as a re-panelised board, a package revision or a new glare angle. The question to answer before go-live is how quickly your telemetry would show, by component class and panel position, that identity stopped holding on that day.
