A reviewer who asks how many readers labelled each study, and what happened when they disagreed, is not asking a bookkeeping question. They are asking whether your reported 2% error gap is a model limitation or a labelling artefact. If the answer is a CSV with one label column and no provenance, the question cannot be settled — and the clinical-grade claim the rest of the validation pack exists to defend loses its footing at exactly the moment it is being tested.
Ground truth is not a fixed input. It is a constructed artefact with its own protocol, its own error rate, and its own audit trail. Treating it otherwise is the most common way a technically sound model ends up with an unreviewable performance number.
What does ground-truth adjudication evidence mean in practice?
It means the pack can answer, for every case in the validation set, who read it, what they could see when they read it, what each of them said, and — where they disagreed — which rule produced the final label. That is four distinct records, and teams routinely ship only the last one.
The consequence is measurable in review time rather than in accuracy. Documented adjudication evidence converts a contested performance number into an adjudicable one: a reviewer can bound label noise from the reported pre-adjudication agreement statistics instead of discounting the whole result. Without it, the usual outcome is a re-read request — weeks of radiologist time, and sign-off pushed out by a procurement cycle. This is an observed pattern across clinical-imaging validation reviews we have worked on, not a benchmarked rate, and the size of the delay depends heavily on whether the site has its own reading capacity.
The second, quieter payoff is reuse. When label provenance is versioned, a model update is re-scored against the same reference standard rather than triggering a fresh labelling campaign. Ground truth becomes an asset with a lifecycle instead of a sunk cost repeated per release.
Reader count and reading conditions
Two records belong here, and they are usually conflated.
Reader count per case is a design choice, not a constant. A single reader per case is defensible only where the finding class has near-perfect intrinsic agreement or where the reader label is not the reference standard at all. Two readers plus a documented tie-break is the common working point for detection and classification tasks; three-reader majority is used where the finding is subtle and disagreement is expected to be structural rather than incidental. State which you used, per finding class, and state why.
Reading conditions determine what the label actually means. The pack should record: whether readers saw prior studies, whether clinical history or the referral indication was available, whether the full series or a selected slice range was presented, the viewing environment, and — critically — whether readers were blinded to the model output and to each other’s reads. A label produced with access to the report and the follow-up scan is a different measurement from a label produced from the images alone. Both can be legitimate reference standards; only one of them is a fair comparator for a model that sees images alone.
Where reader identity, de-identified case handling, and session audit trails are concerned, the evidence overlaps with HIPAA and GxP workflow documentation rather than sitting purely inside the performance argument — the two artefacts share that seam and should not be drafted independently.
Agreement statistics before adjudication
Report pre-adjudication agreement, not post-adjudication consistency. Post-adjudication labels agree with themselves by construction; the number carries no information about label noise.
Two figures do the work: inter-reader agreement (Cohen’s or Fleiss’ kappa, or a suitable overlap statistic for segmentation) and the raw disagreement rate. Both should be reported per finding class, not pooled — a pooled kappa on a validation set dominated by an easy finding will hide a class where readers agree barely better than chance, and that is precisely the class where a model’s reported error is uninterpretable. Where prevalence is low, report the disagreement rate alongside kappa, since kappa becomes unstable at extreme class imbalance.
This is the statistic that lets a reviewer decide how much of the model’s residual error is even attributable to the model. If readers disagreed on 8% of cases in a finding class, a 2% model-versus-reference gap in that class is not a meaningful performance claim.
How disagreements were resolved
The resolution rule must be written before adjudication starts and recorded verbatim in the pack. Three patterns cover most clinical-imaging work:
| Resolution mechanism | How it works | Fits when | What the pack must record |
|---|---|---|---|
| Tie-breaker reader | An independent third reader adjudicates only the disputed cases, usually unblinded to the two prior reads | Two-reader designs; disagreement rate is low and cases are individually resolvable | Tie-breaker’s seniority and specialty, whether they saw the prior reads, count of cases adjudicated |
| Consensus panel | Disputed cases (or all cases) are reviewed jointly until a single label is agreed | Subtle findings; disagreement is definitional rather than perceptual | Panel composition, whether consensus was forced, dissent notes, session dates |
| Composite reference standard | Reader labels combined with a non-reader source (pathology, follow-up, procedural outcome) under a fixed precedence rule | Reader labels are known to be an imperfect proxy for the clinical truth | The precedence rule, availability rate of each component, handling of cases where components conflict |
The failure mode is not choosing the wrong mechanism. It is applying one informally — a senior reader quietly overriding disputed cases with no record of which cases were touched. That leaves the reference standard un-auditable even though every individual decision may have been correct.
When a non-reader reference standard is the right choice
Where a definitive outcome exists, reader consensus is a proxy for it, and a proxy should not be presented as the truth it approximates. Histopathology for malignancy, procedural or surgical findings for anatomical claims, and interval follow-up imaging for lesion progression are all stronger reference standards than any number of readers looking at the index study.
They come with their own documentation burden. Availability is partial by nature — pathology exists only for cases that went to biopsy, which is a selected subpopulation — so the pack must state the availability rate, how cases without the definitive standard were handled, and whether that selection biases the reported metric. A composite standard that silently upgrades the easy cases and leaves the ambiguous ones on reader consensus will report a flattering number for reasons unrelated to the model.
Versioning the reference standard
Adjudication evidence is only reusable if it is versioned. In practice that means the label set carries an identifier, a date, the protocol version that produced it, and a changelog: cases added, cases relabelled, cases withdrawn, and why. Store it the way you store model weights — immutably, with the tooling and reader-assignment configuration alongside it, whether that lives in a DVC-tracked repository, an MLflow-registered artefact, or a database with append-only revisions.
The test is simple. If a model retrained six months from now cannot be scored against the exact label set the original number came from, the pack cannot support a like-for-like performance comparison, and every release re-opens the labelling question. We treat reference-standard versioning as part of the release process rather than a data-team convention, for the same reason we version evaluation code.
Adjudication evidence in the pack stops at the point where it establishes that the reference standard is sound and its noise is bounded. What a regulator asks about label provenance — traceability into an intended-use argument, risk classification, device claims — is a different document with a different audience.
The parent methodology sets out the full anatomy of the validation pack this section sits inside, in what a clinical imaging validation pack must contain beyond the benchmark report, and our wider approach to evidence that survives external review is described under production AI reliability.
Frequently Asked Questions
What does ground-truth adjudication evidence mean in practice, and what belongs in the validation pack?
A useful Ground Truth Adjudication Evidence clarification is this. It means the pack can state, for every validation case, who read it, under what information conditions, what each reader said, and which documented rule resolved disagreements. Concretely: reader count and qualifications per finding class, a reading-conditions statement, pre-adjudication agreement statistics, the written resolution rule, and a versioned label set with a changelog. Anything less leaves label noise unbounded and the performance number uninterpretable.
How many readers per case, and how should reading conditions and information access be documented?
Two readers with a documented tie-break is the common working point; three-reader majority suits subtle findings where disagreement is structural; single-reader labels need an explicit justification. Reading conditions must record whether readers saw priors, clinical history, the full series, and each other’s reads — a label produced with the report available is not a fair comparator for a model that sees images alone.
What pre-adjudication agreement statistics should be reported, and per finding class or overall?
Report inter-reader kappa and the raw disagreement rate per finding class, not pooled. A pooled figure dominated by an easy finding hides the class where readers agree barely better than chance, which is exactly where a small model-versus-reference gap stops being a meaningful claim. At low prevalence, pair kappa with the disagreement rate, since kappa destabilises under extreme imbalance.
How are disagreements resolved, and how is the rule recorded?
Through a tie-breaker reader, a consensus panel, or a composite reference standard — chosen and written down before adjudication begins, then recorded verbatim with the details the mechanism implies (tie-breaker seniority and blinding, panel composition and dissent, or the precedence rule for composite components). The real defect is applying a mechanism informally, with no record of which cases were overridden.
When is a non-reader reference standard the right ground truth, and how is that documented?
When a definitive outcome exists — histopathology, procedural findings, interval follow-up — reader consensus is only a proxy for it and should not be presented as the truth itself. Document the availability rate, since pathology exists only for biopsied cases, how cases lacking the definitive standard were handled, and whether that selection biases the reported metric.
How is label provenance versioned so a model update can be re-scored against the same reference standard?
Give the label set an identifier, a date, the originating protocol version, and a changelog of additions, relabels and withdrawals, stored immutably alongside the reader-assignment configuration. The working test: if a model retrained six months later cannot be scored against the exact label set the original number came from, like-for-like comparison is impossible and every release re-opens the labelling question.
Where does adjudication evidence in the pack stop, and what belongs instead in a regulatory submission?
The pack’s job ends once it shows the reference standard is sound and its label noise is bounded, so a site reviewer can attribute residual error correctly. Traceability into intended use, risk classification and device claims belongs to a regulatory submission — a different audience adjudicating a different decision, covered separately.
Building confidence through annotation transparency
Ground truth adjudication evidence should expose disagreement patterns, annotator qualifications, resolution protocols, and edge cases where consensus required escalation or multiple review rounds. If Ground Truth Adjudication Evidence is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
