Sensor fusion event triage lets cheap signals decide which footage gets analysed, and calibrated confidence decide who analyses it. The pipeline runs in four steps. First, a hardware trigger such as a PIR sensor on a fixed camera proposes an event. Pixel change then confirms the trigger, a detector scores the confirmed segment, and two confidence thresholds split events into auto-accept, human review and auto-reject. Only the band between the two thresholds reaches a person, so where you place those thresholds sets the reviewer workload. Because moving a threshold changes precision and recall, set it from those measured at each cut on your own footage.
How is sensor fusion event triage different from running a video classifier on every frame?
Running a classifier on every frame spends the most expensive model on frames where nothing has changed. NoScope showed how much a cascade saves on video. Its factor analysis put the gains at up to 30× from frame skipping, up to 3× from content-based difference detection and up to 340× from specialised models, with the full reference network consulted only when the cheaper stages are uncertain. The same authors found that the best cascade depends on the data. Differencing against the previous frame works on mostly static scenes but adds overhead on a busy highway.
Triage changes one thing about that cascade: the last stage is a person, not a bigger model. An empirical study of legal-case triage describes this shape. A cheap reader scores everything, a stronger model re-scores only the uncertain middle, and only cases still uncertain after that go to a human. The hardware trigger adds a stage in front of the cascade that never touches pixels at all. Confirmatory sensing treats a PIR trigger as a proposal to be confirmed by another modality, such as radar or pixel change. Building each stage (motion gate, detector, classifier) so it can be updated on its own is a separate architectural question. Here it is enough that each stage passes on fewer events than it receives.
Routing borderline events to reviewers instead of auto-deciding everything
Yes, borderline events can go to a human while everything else is auto-decided. Two thresholds on a calibrated score do the routing: events scoring below the lower one are auto-rejected, events above the upper one are auto-accepted, and events between them go to review.
TechnoLynx builds this routing layer as the Production AI Monitoring Harness. The engagement defines the score bands and the reviewer throughput your team can sustain. It then implements the pre-filter, event detector and routing stages on your infrastructure, calibrates the thresholds against your historical score and outcome data, checks routing accuracy and reviewer load on held-out data, and deploys with monitoring that triggers threshold re-tuning when the score drifts. The first step is fixing the bands and the target reviewer throughput.
NoScope states the mechanism precisely. Its two thresholds on the specialised model’s confidence are c_low and c_high: below c_low the system outputs “no object”, above c_high it outputs “object detected”, and anything between them is sent to the full reference network. A triage pipeline sends that middle band to a person, or to a costlier model first and then a person. Moderation funnels use the same layout and add one rule worth copying: anything flagged as high-stakes goes to review whatever its score. Hive’s documentation sets this up as rules for an example policy: images of real guns are removed automatically and lower-confidence classifications are sent for human review.
Calibration comes before any threshold
A threshold on a model’s confidence means what it claims only when that confidence is calibrated. Guo and colleagues documented systematic over-confidence in modern neural networks and showed that single-parameter temperature scaling restores calibration cheaply without changing accuracy. Expected and maximum calibration error, read off a reliability diagram, are the checks. Both are zero for a perfectly calibrated model.
In the legal triage study, naively composing a language model’s score with Bayesian-odds and Dempster–Shafer fusion more than doubled calibration error. The lesson for sensor fusion is direct. If PIR state, IMU magnitude and detector confidence are merged into one score, recalibrate the merged score on held-out events before you place a single band boundary on it.
How do we set the thresholds between auto-accept, review and auto-reject?
There is no universally correct setting, because moving a threshold in either direction turns one kind of error into the other. Chow formalised this in 1970 as the error-reject trade-off of a pattern recognition system. Reject curves show the trade-off. Precision-reject and recall-reject curves show how precision and recall change as the threshold accepts more or fewer samples, and they suit imbalanced data, where accuracy-based reject curves mislead. Camera-trap data shows the imbalance: camera traps record large numbers of empty, false-positive images.
Fennell, Beirne and Burton warn that changing the confidence threshold can significantly alter precision and recall, and can lead to different conclusions from the same data.
A procedure that holds up:
- Calibrate the fused score on a held-out set that covers every operating condition.
- Sweep candidate cuts and record, at each one, the precision of the events that would be auto-accepted and the recall lost among the events that would be auto-rejected.
- Set c_high at the lowest score where auto-accepted precision still meets your error floor, and c_low at the highest score where the misses you accept in auto-reject stay tolerable.
- Count how many events fall between the two cuts and compare that count with reviewer capacity. If the band is too wide for the team, decide which error you will accept more of.
- Repeat per class where the classes differ in cost or difficulty.
Per-class thresholds are worth the extra work. A local reject option sets one threshold per class or input region so the important classes can be made more reliable. The legal triage study chose per-class thresholds that maximise the auto-clear rate subject to a statistical upper bound on the error rate among auto-cleared cases. That result comes from legal decisions, not video, but the method carries over by analogy: set an explicit error floor for the auto-accepted band and accept whatever auto-accept rate the floor allows. The same study gives the decision-theoretic reason. The optimal threshold rises with the cost of an escaped error, and it differs between classes whenever wrongly clearing one class costs more than wrongly clearing the other. Selective classification offers the same kind of control for neural networks: pick a target error rate and a confidence level, then keep coverage as high as that guarantee allows. NoScope frames it as an optimisation, maximising throughput subject to false-positive and false-negative ceilings, with a default target of 1% for each.
How many reviewers does the uncertain middle band need?
Review capacity has to match the expected volume inside the band, not the total number of triggers or detections. That volume follows from multiplying pass-through rates, the same way NoScope estimates cost per frame: the fraction of frames left after skipping, times the fraction passing the difference filter, times the fraction whose confidence falls between c_low and c_high.
Worked example: estimating review load
Assume these quantities are measured on a representative sample of your own footage, not guessed:
| Quantity | Meaning |
|---|---|
| N | Hardware triggers per day |
| f_confirm | Share of triggers confirmed by pixel change |
| f_band | Share of confirmed events scoring between c_low and c_high |
| r | Events one reviewer clears per hour |
Daily review load is N × f_confirm × f_band events, and the reviewer-hours needed are that figure divided by r. For a sense of r, Fennell, Beirne and Burton report manual camera-trap classification at 300 to 1,000 images per hour, with a mean of 500 across five classifiers. Your reviewers’ rate depends on the clip length and the decision they have to make, so measure it.
Two corrections apply to the arithmetic. Human capacity is spent per batch: a workload balanced over a month is not balanced day by day. Arrivals also fluctuate. Research on content moderation shows that fixed admission thresholds are inefficient because they ignore uncertainty in the estimate and the time-varying pattern of arrivals and reviewer capacity, and it proposes a congestion-aware admission rule instead. The same work finds that review effort is best spent on borderline cases when all types have equal variance. When they are not, a non-borderline event with high uncertainty can deserve the reviewer’s time more than a predictable borderline one. Queue simulation is the practical way to test a proposed threshold move before committing to it: QUEST evaluates a threshold change by the extra jobs it would enqueue and its effect on output metrics, and sweeps automation thresholds to find an operating point.
| If this holds | Move the band this way |
|---|---|
| Escaped errors are expensive and reviewers have spare capacity | Widen it: lower the auto-reject cut, raise the auto-accept cut |
| Reviewer capacity is the binding constraint | Narrow it, accept more auto-decisions, re-tune against audited outcomes |
| One class is rarer or costlier than the others | Give that class its own, more cautious thresholds |
| Arrivals spike at predictable times | Simulate the queue and admit by congestion, not one static cut |
| A trigger fires on a high-stakes zone or asset | Route to review regardless of score |
Thresholds tuned once do not stay right. A system left alone after tuning gradually stops matching its policy as the content changes. Threshold tuning and alert quality are reliability work, separate from model accuracy: a model that has not regressed can still flood reviewers or let missed events through when the score distribution shifts. Sampled audits of auto-decided events help, but any borderline error outside the sampled set still gets through, so the audit sample should be weighted towards the scores just outside each cut.
Should a PIR trigger be confirmed by frame differencing, or is video gating enough?
Where missed events are costly, use both: the PIR (or IMU) is a cheap, independent wake signal, pixel change confirms it, and only confirmed events reach the detector and the triage bands. Where power and compute are tight and some misses are tolerable, frame differencing alone with per-lighting-mode thresholds is a defensible simpler design.
TechnoLynx scopes this gating stage against your edge device as part of its computer vision R&D work. The engagement sets the recall target for slow, low-contrast subjects and the compute and power envelope, parameterises MOG2 or an alternative per lighting mode, decides whether a PIR trigger is worth adding, and validates recall on your held-out dusk footage against full-frame classification.
PIR alone has documented weaknesses. It detects a heat signature, a thermal difference between the subject and its surroundings, so it stops working when that contrast disappears. One review notes it becomes ineffective above 35°C, and that the high sensitivity needed for small or fast animals increases false positives from wind-blown vegetation. Commercial camera traps also have a delay between trigger and capture that limits recording of small, fast animals, and moving foliage against a background of different temperature produces many empty images. In the WiseEye camera trap, the PIR’s detection zone was wider than the camera’s field of view. On the WiseEye platform this caused false positives when a slow-moving object triggered capture before entering the frame.
Confirmatory sensing targets these false positives. WiseEye (PLOS ONE, January 2017) compares each motion-activated image with a recent time-lapse image using background subtraction. It ran with a two-minute time-lapse interval and a user-set difference threshold below which an image counts as a false positive, and processing can be restricted to a region of interest that ignores moving clouds or vegetation. The authors report that this on-board check eliminated all the false positive images WiseEye recorded. They also made a design choice worth noting: radar activity is logged and cross-referenced instead of being required alongside the PIR, because requiring both sensors could delay capture. An IMU spike on a vehicle plays the same proposing role, and we apply the same rule to it. Let the cheap sensor open the window, let the image confirm it, and keep the sensor trace as evidence for the reviewer rather than as a hard gate.
Background subtraction methods and the parameters worth tuning
Consecutive frame differencing needs no background model and little memory, and it adapts quickly to background change. Because it uses only the previous frame, though, it may fail to detect the interior pixels of a large, uniformly coloured moving object. One comparison tested frame differencing, adaptive median, median, mixture-of-Gaussians and Kalman filtering on urban traffic video. Mixture of Gaussians produced the best results, and adaptive median filtering was a simple alternative with competitive performance. A Gaussian mixture model needs no predefined background at start-up, although most background models still struggle with sudden illumination changes and frequent repetitive motion.
OpenCV ships both MOG2 and KNN subtractors behind one interface, and both mark detected shadows as grey in the foreground mask. The parameters a practitioner tunes are history (how many frames the model remembers, which sets how quickly a stationary subject fades into the background), varThreshold for MOG2 or dist2Threshold for KNN (how far a pixel must deviate to count as foreground), detectShadows, and the learningRate passed to apply(). For slow subjects, a long history and a low learning rate stop the animal or pedestrian from being absorbed into the background before it has crossed the frame.
How do we keep recall on slow, low-contrast subjects at dusk?
Lower the gate threshold for the dusk profile and send the extra borderline gate scores into the review band instead of discarding them. Differencing against only the previous frame misses the interior of large, uniform objects, so an older reference is worth testing. NoScope uses two difference detectors, one against a fixed empty reference image and one against a frame a learned interval in the past. Its authors found that computing the difference against several frames past worked better than a single reference frame in scenes with a dynamic background.
Motion-only gating has a weakness you should test for. A 2024 comparison of six motion triggers, a CNN detector and motion-gated hybrids, run on seven static-camera scenes with 27 annotated events, found that camera jitter of up to 4 px made frame differencing store 100% of idle frames and MOG2 97.2%, while illumination steps pushed CNT to 95.0% and GSOC to 80.5%. Motion triggers stored 36–89% of idle frames, against 19–28% for triggers that used the detector. A detector run on every 8th frame gave 99.7% frame recall and caught 26 of the 27 events. In that benchmark frame recall stayed at or above 93.4% for every method and condition, so a fragile trigger cost storage and compute rather than missed events. The benchmark is small, but its threshold protocol is worth copying: thresholds were chosen leave-one-video-out to keep frame recall at or above 95% on every training scene, then applied unchanged to the held-out scene. On a mast that sways or a vehicle that vibrates, a low-rate detector may be a better gate than background subtraction.
Do day and dusk/IR footage need separate thresholds?
Yes. Detector confidence has been measured falling as the light goes. A large network-camera study tracked YOLO on a stationary black pickup: confidence fell from 0.8 at 8 PM to 0.64 at 9 PM, IOU fell from 0.67 to 0.36, and at 10 PM, under streetlight only, the truck was not detected at all. The same study found the detector struggled with dark-coloured objects and produced false positives at night. NoScope’s authors add that when a scene cycles between day and night, the training data must contain frames representative of every condition. They also observed that well-chosen thresholds differ substantially between scenes.
In practice we would use at least three profiles: daylight, dusk and IR. Each needs its own gate threshold, its own background model (or a forced re-adaptation at the IR switchover, which behaves like an illumination step), and its own c_low and c_high on the detector score. Then check recall on the footage the gate threw away, separately for each lighting stratum. A blended figure hides exactly the dusk misses this design exists to catch; the case for stratified validation across conditions, sites and hardware applies directly.
Who can build a sensor fusion triage pipeline for our cameras or fleet?
TechnoLynx frames, builds, calibrates and deploys sensor fusion event triage pipelines. A specialist pays off when the bands have to be defended against audited outcomes and kept tuned as scenes and seasons change. When choosing a partner, ask how they would measure precision and recall at each cut on your footage, how they would size the review band, and how they would detect drift after deployment.
The engagement calibrates thresholds against your historical score and outcome data. Teams whose reviewers already keep up may only need the queue simulation step.
Reviewers belong in the error budget
Sending an event to a human does not guarantee a better decision. The legal triage study notes that a prediction set’s value to a reviewer depends on how it is presented, and recommends showing reviewers the evidence trace rather than a bare score. For video triage the equivalent evidence is the sensor trace: which trigger fired, what the foreground mask showed and which lighting profile was active. A reviewer who sees why an event landed in the band can overrule it faster and more accurately. Measure the human stage the way you measure the detector, with audited outcomes per band and per lighting stratum. Otherwise the uncertain middle simply becomes the place where errors go unmeasured.
