An accuracy figure from a single-clinic demo describes only how the system did on that clinic’s notes. Treat the claim as unverified until two things are true. And you have measured the system on a stratified sample of your own notes, scored against a reference standard built from exactly what the system will see. Both steps are needed because published clinical extraction scores are averages that hide brittleness. A benchmarking study of the 2019 n2c2 shared task found normalisation accuracy of 95.3% for common concepts but 69.3% for concepts unseen in training data.

How do I judge a vendor’s clinical NLP accuracy claim from a single-clinic demo?

Start by asking what context the number leaves out. The authors of Robust Benchmarking for Machine Learning of Clinical Entity Extraction showed that headline accuracy numbers are “hugely bolstered by the existence of a few common, easier-to-map terms”, and that looking at subsets of the data can give very different metrics. In the 2019 n2c2 training data, the 100 most common concepts made up 31.5% of all annotations. A demo built on one clinic’s template will be weighted the same way, towards whatever that clinic writes most often.

What a vendor’s accuracy number must state before you can use it

The number must state	Why it matters
Which entity types it covers (diagnoses, medications, dosage, frequency)	Best-system accuracy on the 2019 n2c2 task varied by concept type, for example 93.9% for pharmacologic substances against 73.3% for body substances.
Exact or relaxed (lenient) matching	An exact match needs the field, the entity and both span offsets to agree, while relaxed matching accepts partial overlap, and the i2b2 and n2c2 challenges define an inexact match differently.
Which benchmark task and test set, by name	Without a named task, nobody can check the figure against published results.
Whose annotations served as ground truth, and how well the annotators agreed	Agreement between abstractors on the same documents sets an upper bound on achievable NLP performance.
Which model or product version produced it	One published evaluation covered only a product’s first release, which may have changed since.

An evaluation of Amazon Comprehend Medical’s first release, completed in June 2019, illustrates this kind of check. The service scored F 0.768 on the official 2009 i2b2 medication test set and 0.828 on the 2018 n2c2 Track 2 test set, which placed it lowest against the three best systems in each challenge. On 18 random outpatient notes from NYU Langone Medical Center it scored 0.753. The authors also found failures that a demo would never show, such as a medication name being cut short when a line break fell inside it. A sound check follows that template: official test sets, named comparators and a small sample of the buyer’s own notes.

Which clinical NLP benchmarks are worth mapping a vendor’s number to?

For medication and diagnosis extraction, the reference points are the i2b2 and n2c2 shared tasks, and a vendor whose numbers cannot be placed against one of them (or an equivalent public benchmark) has not given you a checkable claim. The 2018 n2c2 Track 2 challenge, summarised by Henry et al. in JAMIA in 2020, focused on adverse drug events and medication extraction and evaluated three tasks: concept extraction, relation classification and end-to-end systems. The 2009 i2b2 Medication Extraction Challenge is the other medication benchmark, and both have official test sets with gold-standard annotations. The 2019 n2c2 concept normalisation task, a different task from medication extraction, drew 108 submissions, and the top one reached 85.2% accuracy. For social determinants of health, the 2022 n2c2/UW challenge covered five factors: alcohol, drugs, tobacco, employment and living situation.

Both medication test sets consist of discharge summaries. The 2009 i2b2 test set consists of 251 de-identified discharge summaries from Partners Healthcare in Boston, and the 2018 n2c2 corpus consists of 505 discharge summaries from the MIMIC-III database, 202 of them in the official test set. A strong score on either is therefore a score on discharge summaries. It tells you much less about clinic letters, nursing notes or psychiatry assessments.

Two further points decide whether a comparison is fair. The 2018 n2c2 overview reports lenient micro-averaged F1 by concept type, so check whether a vendor’s score is lenient or strict before comparing it. Leaderboard scores also come from the best competition systems, not from off-the-shelf tools: on the dataset built in the 2019 n2c2 robustness study, MetaMap reached 59.0% accuracy and cTAKES 47.1%. The scoring schema matters as well. When only the longest span counts, 31% of the cases that study examined would have been marked wrong even though cTAKES had correctly matched a concept for part of the span.

What precision and recall should we expect on multi-specialty free-text notes?

Published figures come from other note mixes, so the honest answer is a range that narrows only once you have measured your own notes. What the literature does show is the direction of travel: accuracy falls as the notes move away from what the system was trained and tuned on. A 2020 study of BERT diagnosis models across medical specialties reported mean AUCs of 0.92 on test sets from the training specialty, 0.87 on mixed test sets and 0.83 on external ones. The same study found that performance on new corpora tracked how semantically similar the training and test sentences were.

A four-institution study of social determinants extraction found the same pattern. An instruction-tuned LLM reached micro-averaged F1 above 0.9 within its own corpora, but cross-dataset generalisation “highlighted remaining obstacles”. The weakest cross-dataset pair scored F1 0.40 for the LLM and 0.08 for ClinicalBERT. The authors attributed the drop to differences in hospital setting, practitioner preference, templates and medical specialty.

One toolkit reports stronger transfer figures. The developers of the open-source MedCAT toolkit report F1 of 0.448 to 0.738 on open UMLS datasets (against 0.429 to 0.650) and transfer above 0.94 between hospitals, datasets and concept types. Their setting included self-supervised training on about 8.8 billion words from about 17 million records at three London hospitals, plus fine-tuning on about 6,000 clinician-annotated examples. That reads as evidence that transfer can be strong after local adaptation, not as a figure to expect from an untuned product. Because the tool’s developers report it themselves, it carries less weight than the four-institution study.

LLM-based extraction adds its own evaluation complications. Small changes in prompt wording can shift F1 substantially without any change to the model, so a result belongs to one prompt rather than to the model as a whole. Hallucinated mentions can also inflate mention-level recall while lowering accuracy at the document or patient level.

Are our EHR notes clean enough for an NLP extraction pilot?

You find out by manually reviewing a representative sample of the notes, which confirms whether the target concepts are present at all. Manual chart review confirms both that the target information is recorded and that it is recorded in a form a machine can reach.

TechnoLynx can run this audit as a bounded piece of healthcare AI engineering. TechnoLynx defines which departments, note types and quality dimensions matter for your extraction goal, builds the tooling that pulls a blinded sample and scores it per department, and delivers the findings with a go/no-go recommendation, leaving deployment out of scope until a pilot is commissioned.

The audit scores structure and consistency against the extraction task you plan to run. Duplicated text is one known barrier. A University of Pennsylvania Health System study of 104,456,653 notes written from 2015 to 2020 found that 50.1% of all text had been copied from earlier text about the same patient (JAMA Network Open, 2022). Where information sits is another. In the four-institution SDoH study, one site’s social information sat mainly in social history sections, while another’s was spread across sections such as history of present illness. Abbreviation density is the third, since a term like “PE” can stand for pulmonary embolism, pulmonary oedema, physical examination and more. Concordance queries are another check. The Healthcare Quality Improvement Partnership (HQIP) describes “data quality probes” that check, for example, whether every patient recorded as diabetic has an HbA1c value.

A chart-review audit can also expose problems in the extraction pipeline. A 2025 chart-review validation study found that only 59% of vitals matched Epic, against 81% of labs, because the extraction algorithm filled in missing vitals from other encounters.

Stratify by department instead of taking a convenience sample

A feasibility signal for the whole institution needs a sample drawn at random within every department, because a purposive sample supports conclusions only about the units it covered. The MEASURE Evaluation Data Quality Audit Tool states this directly: findings from a purposive sample “cannot be used to make generalized statements”, while stratified random sampling allows inference to every unit sharing the stratification attribute. HQIP’s guide makes the same point for clinical audit: a representative sample gives every eligible case an equal chance of selection from a complete list, which a set of “typical” cases from a convenient source does not.

Variation in clinical language is another reason to stratify. Language is more consistent within strata such as clinical subdomain, locality or time than across them, so sampling within each stratum captures a wider range of examples, especially when strata differ in size. The 2019 n2c2 robustness study scored rarer terms on separate subsets, such as unseen concepts, alongside all spans. The 2025 chart-review study drew equal numbers of patients from four strata defined by outcome and an error-prone score.

How many notes per department does a feasibility audit need?

Draw enough notes per department to estimate each department’s pass rate within a margin you would act on. Under the assumptions below that means tens of notes per department, and the institution-wide figure comes out much tighter. HQIP’s guide gives the anchor: with a sample sized for 95% confidence and a 5% margin, an 84% finding means the true value lies between 79% and 89%.

Worked example: sizing a twelve-department note audit

Assumptions, stated as assumptions: each sampled note is scored pass or fail on a predefined readiness rubric, the hospital has twelve departments, and the normal approximation applies (n = z²·p(1−p)/e², with z = 1.96 for 95% confidence).

Target	Calculation	Notes needed
HQIP example precision (p = 0.84, ±5%)	3.84 × 0.84 × 0.16 / 0.0025	about 207 per estimate
Per-department estimate, p = 0.84, ±10%	3.84 × 0.1344 / 0.01	about 52 per department
Per-department estimate, worst case p = 0.5, ±10%	3.84 × 0.25 / 0.01	about 96 per department
Aggregate of 12 × 50 = 600 notes, p = 0.84	1.96 × √(0.1344 / 600)	about ±3% overall

The aggregate figure assumes the departments are weighted back to their real note volumes. With equal allocation, an unweighted average would over-represent small departments. Per-department sizing matters because a small per-department sample gives a wide margin, which can make a weak department look clean.

Before the full audit, test the rubric on a pilot. HQIP suggests 5 to 10 cases for a small local audit and 50 or more for large multi-site audits. Then set an agreement threshold for the people scoring the notes: HQIP generally sets a minimum of 85% inter-rater agreement and prefers 90% to 95%. The 2025 chart-review study reported Fleiss’ kappa of 0.90 for labs and 0.81 for vitals across four auditors. The MEASURE Evaluation tool advises access to a sampling specialist for representative designs. If the audit is also meant to seed an annotated gold-standard corpus, Juckett (J Biomed Inform, 2012) gives a method for estimating how many documents that corpus needs.

How should a blinded proof-of-concept on our own notes be set up?

Set the proof-of-concept up as a single, pre-registered test. The vendor never sees the reference labels, and the go/no-go threshold is fixed before any output comes back. Whether a vendor will run one without a contract is a commercial question. It is still a reasonable thing to ask for.

1. Write concept sheets that define each target variable, with allowed values and units, agreed by clinical and technical staff.
2. Draw the stratified sample and set aside a test set that nobody inspects during setup, because accuracy on new data is usually much lower than on a test set the system has been tuned to.
3. Build the reference standard from the same documents, in the same form and schema, that the system will receive. If annotators see extra context, measured performance will understate what is achievable and hide where the system fails.
4. Have annotators label the same notes independently, without discussion, and check their agreement against the threshold you set.
5. Fix the pass threshold at the level that matters for the decision: patient or document level if the question is whether a finding is present, mention level only if every mention counts.
6. Run a single final test on the untouched test set once all settings are fixed.
7. Analyse errors in both directions, missed mentions and invented ones, and log inputs, outputs and model version for every run.

The validation does not end at go-live. Hosted LLM providers update and replace model versions on their own schedules, and a pipeline validated against one version can produce materially different output after an update. Once a reference standard exists, rerun it to revalidate the system after model changes. Producing that kind of evidence for live systems is engineering work distinct from improving accuracy. The same reasoning covers changes a team makes itself, such as quantisation as a change-control decision backed by evidence.

Is SIFT or ORB appropriate for latent fingerprint matching?

No, not as the primary matcher. SIFT and ORB are general-purpose keypoint descriptors built for corners and textured patches in ordinary photographs. Fingerprint identity, by contrast, lives in ridge endings and bifurcations. MINDTCT locates each minutia and records its location, orientation, type and quality. BOZORTH3 then decides whether two prints come from the same finger using only minutiae location and orientation. As the NIST guide puts it, matching works on roughly 100 minutiae on a typical tenprint instead of the 250,000 pixels in the image. In a multi-matcher study, minutiae-based matchers outperformed a ridge-based matcher, consistent with findings that minutiae are more discriminative than features such as local orientation and frequency.

Matching the tool to the evidence also means measuring the evidence. MINDTCT builds a five-level quality map from low contrast, incoherent ridge flow and high curvature. NFIQ scores whole images from 1 (highest quality) to 5 (lowest). NIST states that matcher performance degrades as image quality degrades. NIST also uses the NBIS-based Verification Test Bed as a minimum baseline for matcher performance, so treat NBIS as the floor any candidate system should beat, not as a ceiling.

Examiner verification is part of the method

For casework, the prevailing method is ACE-V: analysis, comparison, evaluation and verification. In the first large-scale study of latent examiner accuracy, 169 examiners each compared about 100 latent-exemplar pairs. The false positive rate was 0.1% and the false negative rate 7.5%, and independent examination of the same comparisons caught all the false positives and most of the false negatives. Four of the five latents with false positives were on galvanised metal processed with cyanoacrylate and light grey powder, often giving tonally reversed images. The Tennessee Bureau of Investigation’s latent print SOP requires blind verification when an AFIS or NGI search returns a single identification, and on any single exclusion. Examiners are not a perfect ground truth either: decisions were unanimous on only 48% of mated pairs and 33% of non-mated pairs.

Should we combine minutiae matching with keypoint descriptors, or pick one?

Pick a minutiae matcher as the core, and add any second signal only if a fusion test on casework-like pairs shows it helps. The multi-matcher study found the best strategy combined minutiae-based and correlation-based measurements, and fusing two heterogeneous systems cut equal error rate from 1.24% to 0.88%. It also observed worse performance when fusing the NIST and UPM minutiae systems, which rely on similar strategies. Those results came from a verification dataset rather than latent casework, and none of them tested SIFT or ORB, so a keypoint add-on has to prove itself on your own evidence.

Tool-mark striation matching has no settled algorithm

Examiners still decide whether two marks share a source by subjective comparison under a light microscope. Researchers have scanned toolmarks in 3D and built comparison algorithms since 2001, using approaches as different as grey-value variance and sectioned cross-correlation of surface profiles. One recent open-source algorithm for 3D scans of striated screwdriver marks derives likelihood ratios from Beta distributions fitted to known-match and known-non-match densities. Expect to evaluate several specialised methods against your own reference marks.

The reference standard needs checking first

A vendor number, an audit score or a matcher’s error rate is only as good as the labels it is scored against. Two annotators using a reformulated clinical entity annotation framework reached a Jaccard similarity of 0.73 for entity recognition, and expert fingerprint examiners were unanimous on only 33% of non-mated pairs. Before you ask which system is most accurate, ask whether your reference standard is good enough to tell the systems apart.

FAQ

What are the data quality checks?

The MEASURE Evaluation Data Quality Audit Tool lists five verifications: describe how the source document is produced, review its availability and completeness, recount and trace reported numbers, cross-check against other data sources and, where feasible, spot-check delivery.