Plan ground truth annotation before anyone picks a model, and plan it in this order. First decide what gets labelled and how: the taxonomy, the edge cases, written guidelines and a stratified sample. Next set a chance-corrected agreement gate with a stated floor, and name the person who settles disagreements that remain. Only after that, decide whether in-house, outsourced or hybrid annotators handle the volume left over once model-assisted pre-annotation has done its share. McHugh’s rule of thumb is that below a kappa of 0.60, agreement between raters is inadequate and the labels deserve little confidence. Pair model-assisted labelling with disciplined human QA.
Before model selection, we check the quality and quantity of the data, access to the processes that generate more of it, and expectations.
Ground truth annotation planning starts with what to label
The first planning question is which frames to annotate. Our answer is to stratify the sample by camera generation, capture campaign and scene conditions, so that rare classes and hardware-specific failure modes appear in the labelled set by design rather than by luck. Driving images come from recorded video sequences, so consecutive frames are often very similar. NVIDIA’s active-learning work notes that ranking images one at a time tends to score consecutive frames from the same sequence similarly high, which wastes labelling and training effort. In the same work, a pool of 2 million on-road images contained noisy and irrelevant images, so a deliberate selection step was required.
Stratification pays off most when the strata behave differently. Work on estimating a classifier’s accuracy from a limited labelled test set shows that the more the accuracy differs between strata, the more stratified sampling reduces the variance of the accuracy estimate compared with random sampling, and in some cases the reduction exceeded 65%. When every stratum has the same accuracy, proportional stratified sampling gives no gain over random sampling. Camera generations and capture campaigns are where we would expect accuracy to differ, and that is why we would use them as strata. The SHRP 2 naturalistic driving database followed similar logic for its baseline driving epochs, drawing them as a random sample stratified by participant and by share of time driven.
Guidelines should be written and tested before labelling scales. In a 2025 interview study of automotive annotation practice by Chalmers and Volvo Cars, practitioners linked clear annotation requirements to consistent labels, linked vague or evolving requirements to rework, and named edge-case coverage gaps and ambiguity among the most pressing problems. One interviewee estimated that rework takes three times the original effort. Other practitioners in the study stressed linking every annotation to a specific guideline version and tool configuration, so that a model problem can later be traced back to the labels that caused it. If you want the tooling side, we cover it in building ground-truth datasets efficiently with CVAT or Label Studio.
Should we outsource annotation or build an in-house labelling team?
For a large driving dataset, the route need not be a straight choice between outsourcing and an in-house team. Start with a pretrained model as a pre-annotation assistant, add active learning so humans see the uncertain samples first, and then decide who reviews what is left. Data sensitivity, cost and ramp-up time decide whether outsourcing or in-house wins. The pre-annotation step itself does not.
TechnoLynx builds the pre-annotation side of that hybrid. TechnoLynx defines the label taxonomy and the label quality that training needs, adapts a pretrained vision foundation model as a weak labeller, and tunes the auto-accept and flag-for-review thresholds. Before scaling, TechnoLynx measures pre-annotation precision, recall and time saved against a human-labelled reference sample. TechnoLynx’s AI engineering work for manufacturing and automotive starts by defining the taxonomy and target label quality, then checks pre-annotation against that reference sample. Once the numbers hold, the reduced, flagged queue goes to whichever annotators you choose, and label quality is monitored over time.
What pre-annotation and active learning actually save
The savings figures below each come from a different setup. An indoor object-detection study found that a two-stage method (label a small part by hand, train a detector, then correct its proposals on the rest) cut manual work by almost 90%. The lowest total workload came from annotating only 4-8% of the data from scratch. In that study, drawing a bounding box took about 10.15 seconds and correcting one about 5.20 seconds. For 3D LiDAR ground truth, a click-seeded scheme built on pre-trained networks took around 3.7 seconds per box on KITTI. The only previously published 3D annotation scheme needed roughly 114 seconds. HabileData, an annotation vendor, gives an indicative range of 30% to 70% less human time from pre-labelling with a decent model.
Pre-labelling brings a bias that the QA plan has to cover. Annotators tend to accept the model’s suggestion, mistakes included. HabileData’s countermeasure is to seed gold-set items whose pre-label is deliberately wrong and track how often annotators catch them. Active learning also has a design choice that matters with imperfect annotators. ActiveLab estimates whether it is more informative to re-label an existing example or label a new one, and its authors report that re-labelling appears vital once label noise exceeds 20%. We would budget engineering time for that loop, which a one-shot batch job does not need. We go deeper on that loop in building defensible ground truth through active learning and human-in-the-loop labelling.
What the vendor cost figures say, and how far to trust them
Treat the cost figures below as indicative estimates, not quotes. They are not measurements of your project. Intellabel’s worked model for 1 million images puts manual offshore labelling at $142K in total, foundation-model pre-labelling with human verification at $107K, and a managed service at $154K. The same vendor says AI-assisted self-serve wins on cost between 50K and 300K labels, provided you have ML engineering in-house. HabileData’s budgeting rule is to multiply the per-unit estimate by volume and then add 30% to 40% for guidelines, QA and project management. On timelines, one vendor estimates that building an in-house team takes at least 3-4 months of recruiting, hiring and training before productive labelling starts, while outsourced vendors can begin within days once guidelines and data transfer are settled.
| Situation after pre-annotation | Lean toward | Reason |
|---|---|---|
| Speed, scale and cost are the priority | Outsourcing | Vendors scale quickly and need no upfront infrastructure |
| Data is sensitive, compliance is strict, or the needed expertise is in-house | In-house | Sensitive data, strict compliance and available internal expertise favour in-house annotation |
| Bulk routine labelling plus sensitive or ambiguous edge cases | Hybrid: vendor volume, in-house guidelines and QA | HabileData advises keeping guideline and QA ownership in-house whichever model you use |
| Pre-annotation quality is poor on the reference sample | Plan for more manual correction and decide sourcing on that larger volume | Missed detections and false positives each add manual correction work |
In the LiDAR study, the main reason for missed objects was too few points in the point cloud. That is why the reference-sample check comes before any scaling decision. In the Chalmers and Volvo Cars study, practitioners said automation should assist human annotators rather than replace them, especially on complex or safety-critical edge cases.
What Cohen’s or Fleiss’ kappa is good enough to start training?
Treat a kappa of 0.60 as the floor, since McHugh counts agreement below it as inadequate; use Cohen’s kappa for two raters and Fleiss’ kappa for three or more. Set the gate before labelling starts, computing the statistic on a sample the annotators label independently. We would treat 0.60 as the floor for every category and require more for safety-critical classes such as vulnerable road users. Any lower bar for a category that is ambiguous by nature should be documented with its reasons. If the score falls short, revise the definitions, retrain the annotators and measure again before any full-scale labelling.
TechnoLynx builds this gate into the labelling standard. TechnoLynx defines what agreement means for each category, picks the statistic and threshold to suit the class balance, writes the adjudication rule into the standard, and sets up automatic agreement scoring. The gate then has to pass before labelled data goes to model training.
Chance correction matters because raw agreement overstates reliability. In McHugh’s 2012 worked example, percent agreement was 0.94 while kappa was 0.85, and the higher the agreement expected by chance, the lower the kappa. Label prevalence affects how a kappa value should be read, and driving events such as crashes are rare. A 2026 review of agreement metrics advises reporting label prevalence and confidence intervals alongside the point estimate. Kappa also has known weaknesses. McHugh notes that its assumptions about rater independence are not well supported and that it can push the estimate too low, and recommends reporting percent agreement as well.
Outside driving, the PsyGUARD suicide-risk text-labelling protocol (2024) shows the batch-gate mechanism: it sent any batch with Fleiss’ kappa below 0.6 back to its three annotators for revision.
One global kappa hides the category that will hurt you
Agreement should be reported for each homogeneous subpart of the data as well as for the whole. Artstein’s handbook chapter on inter-annotator agreement names, as a major pitfall, that a single coefficient can hide parts of an annotation effort that are reliable and parts that are not. A 2026 clinical annotation study shows the pattern clearly: pre-adjudication kappa fell from 0.69 on the simplest questions to 0.60 and then 0.57 as complexity rose. For driving data, we would break agreement down by class, by camera generation and by condition (night, rain, occlusion), so that a weak category becomes visible instead of being averaged away.
Two more conditions decide whether the number means anything. For agreement to indicate reliability, annotators have to follow written guidelines, work independently, and label a sample that represents the full material. McHugh’s rule of thumb is no fewer than 30 comparisons, because confidence intervals on small samples are wide enough to include no agreement at all. Bootstrapped interval estimates show how much this matters: in one worked example, a kappa labelled “substantial” by the Landis and Koch scale could be claimed at 95% confidence only as “fair”. A high score also proves consistency, not correctness. The annotators may still be applying the wrong construct consistently. The agreement gate certifies the labels, not the model; Northcutt and colleagues advise judging the model on correctly labelled test sets.
Who adjudicates when two annotators still disagree?
Name a person or a rule in the labelling standard before labelling starts, and log every decision as part of that standard. Protocols differ on the details. PsyGUARD resolved disagreements by majority vote first and had its three annotators discuss any instance where all three labels differed. In its crowd stage, the SafeTutors benchmark (2026) used three-way majority vote and sent no-majority cases to a senior researcher, 4.2% of that stage’s instances (152 of 3,600). In the myeloma study, a senior haematologist adjudicated every pair without direct agreement and could adopt either answer or give an independent one.
Majority vote is not always the most accurate rule. In Google’s diabetic retinopathy work, adjudication by a panel of retinal specialists who debated each disagreement until consensus was more accurate than a simple majority vote, and it surfaced subtle findings. Krause and colleagues found that even a small set of adjudicated grades substantially improved algorithm performance. That supports spending specialist time on a well-chosen adjudicated subset rather than on everything.
Driving data has its own version of this. For subjective temporal labels, the Nexar collision dataset had ten annotators mark each clip, took the median for the event time and the second-highest value for the alert time. VTTI’s four-phase QA/QC workflow for naturalistic driving data runs protocol development, reductionist training, data reduction and post-reduction, and a third reviewer makes the call on all disagreements. A single adjudicator is a single point of subjective bias, so we would sample and review the adjudicator’s decisions too, in the way VTTI checks its coders against a jointly coded gold standard.
Can we bootstrap near-collision labels from accelerometer triggers?
Yes, but only as a candidate filter. Accelerometer-triggered clips are a sensible place to look for harsh-braking and near-collision events, and every candidate still needs human review under the same agreement and adjudication rules before it trains or validates a triage model. Without a trigger stream, the set has to come from direct sampling and annotation of raw footage, with the same QA applied.
TechnoLynx designs that bootstrap pipeline. TechnoLynx defines what counts as a harsh-braking or near-collision event, pulls candidate clips from the trigger logs into a review queue, and builds the reviewer calibration and adjudication workflow. TechnoLynx then checks agreement and edge-case coverage and keeps the pipeline running as an ongoing feed for the model’s training and validation sets.
Event recorders store data on either side of an acceleration trigger, yet most of what they capture is regular driving. The NHTSA 100-Car Study concluded that purely quantitative near-crash criteria with no human verification were not feasible, because near-crash kinematics looked virtually identical to ordinary driving. In SHRP 2, a dozen trigger algorithms ran over 5,512,900 trip files, and manual validation then identified 1,549 crashes and 2,705 near-crashes. A longitudinal deceleration trigger gave the highest share of valid events. The proportion of false alarms can be severe: of 735,000 harsh driving events collected from about 123,000 vehicles over 48 hours in February 2026, manual verification confirmed only 143 as crashes, under 0.02%.
Accelerometer triggers over-fire on benign events. Three corrections follow. Review positives together with a random subsample of negatives, as VZCrash did. Use model scores to prioritise review once a first model exists; VZCrash reviewed the 1,000 events ranked highest by an ensemble of earlier models. And add targeted sampling for rare scenarios, which practitioners in the Chalmers and Volvo Cars study described as hard to cover with standard guidelines. Nexar’s acceptance rule offers a concrete gate for categorical attributes: keep a clip only if at least two annotators agreed on all of them.
Agreement drifts, so the gate has to be re-run
Particula, a vendor, describes percent agreement falling from the low 90s into the high 70s over three months as annotators met cases the original guidelines did not cover. VTTI addresses the same risk with repeated expert-rater tests and intra-rater items carried over from one test to the next, which measure whether each coder still labels the same events the same way. For a driving programme that keeps adding campaigns and camera generations, we would re-run the per-category agreement check whenever the guideline version, the annotator pool or the sensor set changes. The gate is something to repeat, not a box ticked once.
