Designing a Defensible Evaluation Harness for Accented Speech AI Procurement

A procurement decision on a speech recognition system is defensible when the evaluation harness behind it has four parts. It needs a private, human-verified test set stratified by accent, per-accent regression gates set above the measured run-to-run noise, normalisation and tokenization artefacts removed before anything is scored, and a sign-off step that blocks cutover when a gate fails. A single average word error rate (WER) is not enough on its own. In a 2026 call-centre benchmark covering fourteen English accents, Parakeet V3 had the better average WER (9.2% against 11.2% for Canary-1B), yet its relative gap between best and worst accent was 48% against Canary-1B’s 26%. Better average accuracy does not mean better accent robustness, so a harness that reports only the average cannot support a go/no-go call for a multi-accent caller base.

Average WER hides the accents that fail

The call-centre benchmark is not an isolated result. For several models in that benchmark, the WER gap between the lowest- and highest-performing accents exceeded 10 points absolute, with Singapore, Chinese, Scottish and Indian English consistently scoring worse, while Australian and General American English tended to score better. On EdAcc, the Edinburgh corpus of video-call conversations, the best model reached 19.7% average WER, against 2.7% on clean US English read speech, and every model dropped on Indian, Jamaican and Nigerian English speakers. Meta’s Fair-Speech authors (Interspeech 2024) found significant demographic gaps in models trained on more than a million hours of data, and concluded that evaluation has to look at demographic breakdowns as well as overall accuracy.

That is why regression testing for speech systems belongs to production AI reliability rather than to model quality. Production AI systems regress in ways unit tests cannot catch, and task-specific evaluation harnesses and regression suites are the artefacts that catch those regressions before customers do. The sections below follow the order in which a harness actually gets built: data, stratification and sizing, scoring artefacts, gates, then governance.

Where do we get verified native-speaker transcripts for dialects we don’t cover yet?

For dialects you don’t cover yet, such as West African English or regional Spanish, the three channels compared here are: commission a transcription vendor, crowdsource with built-in quality control, or partner with a local broadcaster or community organisation. None of the three is best in every case. What decides it is how much audio you need, how rare the dialect is, your budget and your turnaround. Whichever channel you pick, you still have to verify the result independently before any batch becomes ground truth.

Channel	Fits when	Main risk
Commissioned transcription vendor	Dialect is rare, volume is moderate, budget allows contractual accuracy terms	Delivered transcripts are taken as correct without independent checks
Crowdsourcing with QC (dual annotation, agreement thresholds, spot audits)	Volume is high and the dialects are relatively common	Inconsistent or low-quality transcripts for low-resource dialects if QC is weak
Local broadcaster or community partner	Authentic regional usage matters most	Formal, scripted speech can crowd out conversational register

Published transcription costs differ a lot by kind of audio. Kili Technology’s 2026 annotation guide cites published figures of roughly 25 person-hours per audio hour for the Slovenian ARTUR corpus and 30 to 36 hours for Bambara field recordings, and gives roughly $6 per speech hour for crowd-collected read English. Speechmatics puts manual transcription at around $100 per hour.

A new set is still a sample, however carefully it is collected. Production traffic changes and new dialects or content types appear, so check the set against real traffic from time to time and expect to extend its coverage.

How do we check that third-party transcripts are accurate enough to be the reference?

Start by measuring how far independent transcribers agree with each other. Don’t take the vendor’s word for it. When three transcribers corrected the same ten hours of German oral-history audio, they disagreed with each other at 7.77% to 9.54% WER, which the researchers read as a human WER of about 8.7%. A reference that noisy can easily be wrong by more than the regression you are trying to detect. AssemblyAI reports that the most common cause of misleading scores is the reference transcript, not the model. In one case a newer model was penalised for correctly writing “penicillin” where the human reference said “penicillium”.

The working protocols in the sources share a few features:

Require agreement instead of assuming it. In FSD50K’s protocol, two independent raters had to agree before a label was accepted.
Run multiple passes until the transcribers converge, then score a sample. Fair-Speech ran multi-pass vendor transcription until three transcribers agreed, sent low-confidence rows to culture-specific linguists, and measured an annotation WER of 1.47% on 200 randomly selected rows.
Report agreement per class, per language and per annotator, and keep an audit trail of who labelled, who reviewed and which version of the guidelines applied.
Use native or accent-familiar annotators. The 2026 call-centre benchmark required annotators to be native English speakers or demonstrably familiar with the accent group, and ran multi-round QA.

Record what a reference has not had checked. The same call-centre benchmark computed no formal inter-annotator agreement metric and says residual transcription uncertainty may remain. CallCenterEN is a large set of real call-centre transcripts produced by a commercial ASR service, and only 0.1% of it was reviewed by humans, which its authors say is below a statistically significant sample size. A set like that is useful for research, but it is not a verified reference you could gate a release on.

Building the accent-stratified speech recognition dataset

An accent-stratified test set is useful only if it represents your callers. Public corpora can fill the gap for a while. The set a release gate can rely on is a private one built from your own traffic, with privacy handled properly. Kili Technology’s guide puts it directly: build a private, human-verified evaluation set on your own audio conditions, with agreement measured and dialects stratified, before you commit to a model. Keeping it private also protects the benchmark itself. Large open-weight models often don’t fully disclose their training data, so public test transcripts may already be in that data. The call-centre benchmark authors suggest private evaluation data or regularly refreshed test material as mitigations.

Public corpora as a stopgap, a licensed corpus, or your own call logs

Use a public corpus as a stopgap when its accents and recording conditions are close enough to your traffic. Each candidate falls short of a call centre in some way:

Common Voice is multilingual and crowd-validated: a clip counts as valid after two up-votes, and accent is an optional self-reported field.
L2-ARCTIC has 24 non-native speakers across six first languages (Hindi, Korean, Mandarin, Spanish, Arabic, Vietnamese), each recording about an hour of prompted read speech. Read speech is not how callers talk.
EdAcc has almost 40 hours of dyadic video-call conversations with a linguistic background profile for each speaker.
Fair-Speech has about 26.5K utterances of voice commands from 593 people in the United States, with demographic metadata.
Casual Conversations has 846 manually transcribed hours with age, gender and skin-tone metadata.

If you want to license a corpus rather than collect your own, there are options. The Linguistic Data Consortium catalogue lists CSLU Foreign Accented English: 4,925 telephone-quality utterances (8 kHz, µ-law) from native speakers of 22 languages, with perceptual accent ratings from three listeners. Its telephone channel is closer to a call centre than webcam or studio audio. Its accents may still not be your callers’ accents.

Channel mismatch is the caveat that matters most for every public option. Speechmatics notes that contact-centre audio is generally much noisier and harder to transcribe than broadcast audio, and that codec and cross-talk shift results significantly. When a corpus doesn’t match your accent mix, codec, noise or domain vocabulary, build from your own call logs. CallCenterEN shows one way real call audio can be handled: it was collected through authorised partnerships with BPO providers, and only PII-redacted transcripts were released, with no audio, to avoid biometric re-identification risk. Reusing customer audio for evaluation raises consent and privacy questions that need legal review in your jurisdiction.

Whatever the source, record accent and channel metadata as schema fields when the audio is annotated. You cannot report accuracy by accent, microphone or noise condition unless someone wrote down which was which at the time.

How many hours per accent before WER numbers mean something?

No fixed number of hours makes per-accent WER meaningful. The set is big enough when the confidence interval on each accent’s WER is narrower than the regression you want to catch. Published rules of thumb give you a floor to start from. Speechmatics recommends at least 10,000 words, or about one hour of continuous speech, per language when comparing engines. Applied per accent, that is a minimum, not a target. The 2026 call-centre benchmark used about 8 to 11 hours and 10 to 16 speakers per accent.

The number of speakers matters, not just the number of hours. Bisani and Ney (ICASSP 2004) found that bootstrapping WER at sentence level gives confidence intervals that are too optimistic, because it wrongly assumes utterances are independent. They therefore applied the bootstrap at speaker level as well. Liu and Peng (Interspeech 2020) showed that blockwise bootstrap, which resamples whole speakers or conversations, gave intervals about 1.5 times wider on conversation data and 2 times wider on AMI meeting data than ordinary bootstrap. Ordinary bootstrap intervals are too narrow and risk false-positive discoveries. So a set with many hours from a handful of speakers per accent is less informative than its hour count suggests.

Two more reference points help with sizing. Card et al. (EMNLP 2020) found that typical 2,000-sentence machine translation test sets have only about 75% power to detect a 1 BLEU difference. In a medical segmentation study, 20 test samples gave confidence intervals about 8 Dice points wide. Its authors note that precision depends on how variable the metric is and how big the set is. For reporting, Mozilla sizes Common Voice test sets by power analysis (99% confidence, 1% margin of error). Ma and colleagues suggest looking at the median and 90th-percentile per-utterance WER, or a population-weighted WER across speaker groups, rather than a single average.

Was the student wrong, or did the tokenizer split the word?

Our proposed design separates tokenization effects from real content errors in ESL scoring with paired test items. Take responses that human scorers judge to have equivalent content (the original misspelled or code-switched response, and a corrected or tokenization-friendly version), score both, and check whether the model’s score difference follows how much the tokenizer fragmented the text. If the score changes on a pair that humans rated equal, and the change grows with fragmentation, this design treats the tokenizer as the cause. If the score doesn’t move with fragmentation, treat the discrepancy as a real content judgement.

Two published tokenizer metrics give you the fragmentation axis. Subword fertility is the average number of tokens used per word (the ideal is 1), and the proportion of continued words is the share of words split into two or more tokens (the ideal is 0). CheckList (Ribeiro et al., ACL 2020) makes a similar case: held-out accuracy often overestimates NLP model performance, so it proposes behavioural testing.

Published robustness studies show that misspellings degrade model performance. Typos and misplaced characters can change tokenization drastically. Even large LLMs such as the Llama 3 and GPT-4 families are more sensitive to character-level changes than to subword-level ones. In the Adv-BERT study, typos in informative words did the most damage, and mistyping was the most damaging kind of error. Oversample misspellings of content-bearing words and code-switched spans, in proportion to how often they occur in your real ESL population, and reweight back to that population when you report. Code-switching also needs a written convention for where one language ends and the next begins, or two annotators will produce different references for the same text.

To check the overall result, compare score gaps between groups. Guo et al. (2025) define AI scoring disparity as the difference between the human and the model mean score gap (MSG). If the two gaps match, there is no disparity. Short science answers depend less on language proficiency than essays, so their results may understate the effect in essay scoring. Two caveats apply. Each pair needs human scorers to confirm the two versions are equivalent. Oversampling distorts the set’s composition unless you reweight.

Normalisation is the speech-side version of the same problem

ASR has its own version of this artefact: text normalisation. One study found Whisper’s non-English normaliser inadvertently strips Indic vowel signs, producing incorrect WER, and that MMS, Seamless and Hugging Face transformers adopted the routine. Transcription conventions move scores on identical audio too. At MUCS 2021, transliterating a Hindi–English test set into a consistent script gave 22.72% WER against 24.66% under conventional scoring. Normalise both transcripts before scoring and pin the normaliser version, because a change in normalisation rules changes the score.

Should a release gate be absolute WER or a delta against the last shipped model?

Use a delta against the last shipped model, measured for each accent segment, with both a relative and an absolute condition, and set it above the spread you measure when nothing has changed. A single absolute WER limit is reasonable only when baseline WER is similar across accents. When baselines vary widely, the same rise means something different for each accent.

TechnoLynx builds per-accent regression gates like this as part of its Production AI Monitoring Harness.

AssemblyAI suggests a starting point: fail the run if any slice’s WER rises by more than 0.5 points and by more than 5% relative to baseline. Requiring both stops small moves on low-WER slices from failing the build. It also says to run the suite three times with nothing changed and set the gate above the spread you see. In its own test, the same audio on the same pinned model moved 1.6 points between runs on a small slice. Pin the model version so a run measures your configuration and not a silent upgrade.

Run-to-run noise does not come from sampling temperature alone. ReasonBENCH (2025) found that the top LLM reasoning strategy won only 77% of head-to-head runs against its nearest rival, and that variance persisted under greedy decoding, so lowering the temperature does not remove it. For a fine-tuned ASR model, measure noise by repeating training and evaluation on identical data. Re-measure it whenever training data, augmentation or hyperparameters change, because the old noise estimate stops applying. Small accent segments give noisier estimates, so they need a wider threshold or more test data. The blockwise bootstrap described earlier gives intervals that account for speaker correlation.

Worked example: how the two-condition gate behaves

Assume, for illustration, two accent slices with baselines of 8% and 20% WER, using the 0.5-point and 5% relative rule above.

Slice baseline	5% relative equals	Rise needed to fail (both conditions)
8.0% WER	0.4 points	more than 0.5 points (the absolute condition binds)
20.0% WER	1.0 point	more than 1.0 point (the relative condition binds)

The arithmetic follows from the rule. On a slice where the measured run-to-run spread is wider than either figure, as with the 1.6-point swing AssemblyAI saw on a small slice, neither starting value is safe. The gate has to sit above the measured spread. Where these checks block a nightly model promotion automatically, the wiring is covered in gating model promotion with stratified per-accent WER checks. When a gate fails, listen to the files that moved. Sometimes the model has regressed, and sometimes the model is now right and the reference is wrong.

Who can build an accent-aware regression gate for our ASR models?

TechnoLynx builds evaluation and regression harnesses with per-accent slice metrics and gates calibrated against measured run-to-run noise. If you are deciding between doing this in-house and bringing in a specialist, the deciding factor is whether your team has speech-evaluation and statistics experience as well as MLOps experience. The gate logic is short; calibrating it correctly is where the work lies. When comparing partners, ask how they would measure your noise floor and how they would verify your references before any threshold is agreed. The engagement is the Production AI Monitoring Harness.

A team that already runs speech research with statistical evaluation can reasonably build this itself from the published methods.

Who signs off before cutover, and how do we stop a vendor demo overriding the verdict?

Sign-off should rest with a named approver, separate from both the vendor and the technical evaluators. That approver reviews a fixed evidence pack against pass/fail criteria agreed before any demonstration, and cutover cannot go ahead until the criteria are met. A demo or a good anecdote is not part of the evidence pack, so it cannot override the verdict.

TechnoLynx can help define objective, reproducible harness criteria and metrics before any vendor demo, structure the evidence pack so it maps directly to those criteria, and run the candidate evaluation independently to produce the verdict that feeds sign-off. The broader method for comparing candidate models is covered in designing a task-specific evaluation harness for model selection and procurement.

Similar principles appear in US federal acquisition policy, which is written for agencies. OMB memo M-24-18 (2024) tells agencies to run independent evaluations on data they define, kept out of the vendor’s reach where practicable and as close as possible to deployment data. The same memo lists contract terms requiring a new version to meet performance standards before deployment, or to roll back if it fails. M-25-21 (2025) asks agencies to check for overfitting to known test data, so vendors cannot train on the test set. It lets an agency’s Chief AI Officer waive minimum practices for high-impact AI by a written risk-based determination, a responsibility that cannot be delegated. M-24-18 tells agencies to use “show, don’t tell” approaches that require vendors to demonstrate capabilities during evaluation. In our view, a demonstration therefore belongs inside the agreed evaluation, not in place of it.

Evidence pack checklist

Pass/fail criteria and metrics, dated before the first vendor contact.
Per-accent WER with speaker-level confidence intervals, alongside the overall score.
Performance broken down by sub-group, which M-24-18 lists among required vendor metrics.
Reference verification record: agreement scores and audit trail.
Measured run-to-run spread and the gate settings derived from it.
Normaliser version and transcription conventions.
A documented exception path: who may accept risk, in writing.

The process works only under certain conditions. The approver must be insulated from sales pressure and procurement targets. The evidence format must be agreed in advance, or a vendor can argue afterwards that its demo should count. And there must be a defined exception path, or legitimate fast-track decisions stall. For a multi-language rollout such as AI versus human dubbing, the harness’s per-language verdict is what each market decision should rest on. If the procurement also includes hardware, LynxBenchAI publishes results showing how each device runs the public PyTorch stack, which gives a broad indication for custom PyTorch models. It does not run or measure your own speech workload, so the harness verdict still has to come from your audio.

The question to settle before any vendor meeting is the minimum regression on your worst-served accent that you would refuse to ship.

FAQ

Which audio transcription service is the most accurate?

It depends on your audio. Results vary significantly with background noise, codec, cross-talk, accent, age and gender. Benchmark candidates on files that look like your real traffic, not on pristine samples.

Is a sample size of 30 statistically significant?

Sample size alone does not settle it. How precise an estimate is depends on how variable the metric is and how large the test set is. ReasonBENCH chose 30 repeated runs through a power analysis for its own effect size, and notes that 30 runs support mean and moderate-tail analysis but not extreme-quantile guarantees.