Signing off a new codec, compression stage or AI-processing chain takes two kinds of evidence. The first is automated objective measurement that runs on every stream. The second is a subjective panel run to a published protocol, not a rubric written for the occasion. ITU-R BT.500 gives you most of the video half ready-made: test methods, grading scales and viewing conditions, written for laboratory work and, where possible, for operational assessments. Listening tests have their own ITU-R recommendations (BS.1534 for MUSHRA, BS.1284 for general sound-quality methods), and EBU R128 defines loudness and true-peak measurements made with a compliant meter.

BT.500 itself explains why both halves are needed. System performance may not be fully captured by objective means, so objective measurements have to be backed up by subjective ones. It works the other way too. A panel will never tell you that integrated loudness is within tolerance across a three-hour concert, and a meter will never tell you that the solo violin has started to smear.

Which method answers which sign-off question

Sign-off question	Method	Use it when	It is not enough when
How much worse is clearly impaired audio?	ITU-R BS.1534 (MUSHRA)	Impairments are audible and you need graded comparisons between systems	The chain is meant to be near-transparent and most scores land at the top of the scale
Can anyone hear a difference at all?	Blind ABX	You need to know whether listeners can tell two chains apart at all	You need a graded quality score rather than a yes/no on discrimination
Does the metric still track perception?	VMAF or PSNR plus a small subjective check	A hybrid neural codec that still behaves like conventional compression	The codec synthesises detail (generative), where metrics can mislead

Does ITU-R BT.500 give us a ready-made protocol for subjective video tests?

BT.500 already sets the test methods, grading scales and viewing conditions a home-made rubric would have to argue for. Unless the chosen method says otherwise, it asks for at least 15 observers. A limited-scope or exploratory study may use fewer, but has to be labelled “informal”. Observers can be expert or non-expert depending on the objectives, but should not have been directly involved enough in developing the system to acquire detailed knowledge of it. Before a session, they are screened for normal visual acuity and colour vision. They are trained on illustrative images that are not used in the test itself.

Session rules are just as specific. A session should not run longer than half an hour. The first session opens with about five dummy presentations, which are left out of the results, and later sessions open with about three. Presentation order is randomised (Graeco-Latin squares are the suggested source) so that tiredness and adaptation even out across sessions. For home-environment testing, BT.500 (versions 14 and 15) sets 200 lux of ambient light on the screen and a peak luminance of 70-500 cd/m². The laboratory conditions in BT.500-14 call for low room illumination and 70-250 cd/m² peak luminance. Brightness and contrast are set with PLUGE patterns per ITU-R BT.814. The design viewing distance is the distance at which two adjacent pixels subtend one arc-minute.

The part you choose is the method. BT.500 says no single method can yet be recommended over the others, and leaves the choice to the experimenter. DSIS shows the unimpaired reference and then the impaired version, and scores on a five-grade impairment scale that runs from “imperceptible” down through “perceptible, but not annoying” to “slightly annoying” and below. DSCQS shows pairs without saying which one is the reference and collects ratings on a continuous scale. It is especially useful when the test material cannot cover the full range of quality. The second choice is test material that targets the particular failures of AI processing (the sampling section below covers how). For crowdsourced or multi-lab runs, where inter-lab variability inflates variance, BT.500-15 adds an analysis that models each subject’s bias and consistency, a “soft rejection” in place of throwing out whole observers.

Panel skill level and tiredness still influence the grades. Calibrated viewing conditions only count if you can show they were measured, not just say so.

How should we document panel results and objective test logs so sign-off shows due diligence?

BT.500 asks every test report for the fullest description possible of the test configuration, materials, observers and methods.

• details of the test configuration and the test materials;
• the image source and displays, including screen size, make and model, since display size may affect results;
• the number and type of assessors, with as much detail on panel characteristics as possible;
• the reference systems used and the grand mean score;
• original and adjusted mean scores with 95% confidence intervals if any observer was eliminated.

Every test parameter is reported with its mean and 95% confidence interval. Any correction for scale-boundary effects has to be reported, because these corrections depend on assumptions of their own. For audio, BS.1284 and BS.1534 set the bar at replication: a knowledgeable person should be able to repeat the study from the report. That covers how subjects and excerpts were chosen, room dimensions and acoustics, transducers and their placement, the design, training and instructions, and the statistics.

When a complaint arrives, two records do most of the work. The first is acceptability criteria fixed before the assessment started, which BT.500 says matters in any procedure. The second is observer exclusions made by the published rule, which BT.500-15 says should be applied no more than once and restricted to small all-non-expert panels (e.g. fewer than 20). We would keep the objective logs in the same file: the meter’s conformance result, the tool version, and the full-programme loudness log for each tested stream. That way one document answers both “did you measure it” and “did anyone look”.

Which loudness measurements does EBU R128 require, and can each stream be checked automatically?

Programme Loudness (the integrated loudness of the whole programme) is normalised to −23.0 LUFS, and Maximum True Peak Level must not exceed −1 dBTP during production. Tech 3343 allows ±0.2 LU around the target for post-production programmes, so that metering tolerances do not pile up into rejections, and ±1.0 LU for live programmes and others with very short turnaround. The true-peak measurement tolerance is ±0.3 dB for signals band-limited to 20 kHz. Loudness Range, Maximum Momentary Loudness and Maximum Short-term Loudness are optional descriptors that help characterise a programme.

Two measurement rules matter for a concert broadcast. The signal is measured in its entirety, with no emphasis on speech, music or effects, so the evidence is a log of the whole concert as it leaves the new chain. The meter must comply with ITU-R BS.1770, including its gating, and with EBU Tech 3341. Gating uses an absolute threshold of −70 LUFS and a relative threshold 10 LU below the absolute-gated level, applied to 400 ms blocks with 75% overlap. Without the gate, programmes with long quiet stretches read too low and come out too loud after normalisation. We would treat orchestral programmes with pauses between movements as that case. Momentary loudness uses a 0.4 s window and short-term a 3 s window, both ungated, while integrated loudness is gated.

Tech 3343 says a strict LRA limit for every programme is neither possible nor useful. It is still a useful indicator for an AI processing chain, because if LRA after the chain is lower than before, a dynamics reduction has occurred, whether on purpose or not.

An automated R128 gate, step by step

1. Qualify the meter. Tech 3341 test case 1 feeds a 1 kHz stereo sine at −23.0 dBFS for 20 s, and an EBU Mode meter must read M, S and I at −23.0 ±0.1 LUFS. Passing the minimum tests does not prove the meter is accurate in every respect, but failing them is a strong sign it is not compliant. Reset the meter before each measurement.
2. Measure the chain’s output end to end. The FFmpeg ebur128 filter analyses a stream’s loudness, defaults to the −23 LUFS target, and can write loudness metadata into 100 ms frames for a time-series log. libebur128 is a library implementation of R128 for teams embedding the check in their own tooling.
3. Check the definitions. Momentary loudness under ITU-R BS.1771 uses an IIR filter and can differ from the EBU Mode value by up to 2 LU. In surround work, an EBU Mode meter excludes the LFE channel.
4. Compare against the thresholds and against the input. Test integrated loudness against target and tolerance and maximum true peak against −1 dBTP, then compare LRA before and after the chain.
5. Block on any breach. If any threshold fails, the release is held until the fault is fixed and the full programme has been re-measured.

Passing R128 cannot guarantee a good mix; that still takes experienced listeners. Thresholds met on one set of orchestral recordings may also fail on very dynamic or very quiet material that was never tested. We would rely on a true-peak value only from a meter that has passed the conformance signals in this release, not one that passed them once years ago.

Formal panel or a lighter blind test: what will stand up to engineering and content partners?

What stands up is a protocol taken from a published recommendation, with a panel large enough to be called formal, run alongside the objective metrics so each can check the other. The video and audio cases split on what the panel has to detect, so they get separate answers.

Video: formal BT.500 panel or crowd-sourced check

Under 15 observers, BT.500 itself calls the study informal. Crowdsourcing puts subjects in less controlled conditions, and BT.500-15 notes that its standard data-analysis methods are often poorly suited there. An Edinburgh analysis of speech-synthesis test data (2015) advises collecting significantly more data and listeners in such uncontrolled settings. For pilots, ITU-T P.913 suggests 8 to 12 subjects to show a trend, and NTIA’s 2020 analysis notes that so few subjects make significance testing unhelpful. The same NTIA report estimates that a single person ranking stimuli of similar quality gets about 36% of pairs right, and a 6-12 subject pilot with a wide quality range about 72%. Pilot-sized tests indicate trends. When a rollout faces scrutiny, the decision belongs to the formal panel.

Audio: a listening test that catches pre-echo, pumping and smearing

BS.1284 already names these artefacts. Pumping and warbling fall under periodic modulation effects, and pre- and post-echoes and smearing under temporal distortion, alongside separate classes for spatial-image distortion and missing sound. Where the defect affects the stereo image, BS.1284 says to use loudspeaker reproduction, since some faults show more clearly on loudspeakers and others on headphones.

MUSHRA is designed for intermediate quality. BS.1534-3 says scores should ideally fall between 20 and 80, that results may be invalid if most conditions score 80-100, and that the method is not intended for small impairments. If the replacement is meant to be near-transparent, the first question is discrimination (can trained listeners tell the chains apart?), which is ABX territory. MUSHRA comes in when the candidates are clearly impaired and need ranking. The Edinburgh study notes that MUSHRA has been found to yield more significant differences than a MOS test, other things being equal.

BS.1534-3 requires critical material that stresses the systems under test, sought again for each system. Without it, experiments will fail to reveal differences and be inconclusive. BS.1284 says critical material should be used whenever the system is meant to carry high-quality sound. Excerpts should run about 10 s, preferably no longer than 12 s, to limit fatigue and primacy effects. The number of excerpts is about 1.5 times the number of systems, with a minimum of five. For orchestral work, we read that as short excerpts of quiet strings and solo passages, sharp transients and full tutti, picked because they stress the chain. Continuity over a full programme is then covered by the R128 log, not by listener hours. BS.1534-3 sizes the test by excerpt count, not listening hours.

How do we design a blind listening test an artistic director can sign off?

Use a blind test with expert listeners on critical orchestral excerpts, run to an audio recommendation, with the pass criterion and significance level written down before anyone listens. In ABX, each trial routes one of the two sources at random. A listener who cannot consistently say which is which is treated as hearing no difference. A plain A/B preference test only tells you which version people liked, not whether they can reliably hear a difference. For subtle artefacts in quiet passages, forced-choice ABX is the more sensitive design.

On the panel, BS.1284 sets a minimum of ten expert or twenty non-expert listeners, and says to use experts whenever the system is meant for high-quality broadcasting. BS.1534-3 qualifies assessors by discrimination and reliability, measured with replicated items, and makes training compulsory, covering the full range of impairments and the test software. Post-screening drops any assessor who rates the hidden reference below 90 on more than 15% of items, or rates the mid-range anchor above 90 on more than 15%. Experienced listeners give more reliable results more quickly, but blinding still matters. BS.1534-3 asks for independent, double-blind data collection, and BT.500 keeps people involved in developing the system off the panel.

Worked example: the ODAQ MUSHRA panel

The Open Dataset of Audio Quality (ODAQ) shows what a full-scale MUSHRA run looks like. Each trial held a reference, two low-pass anchors and five processed samples. Fifteen listeners took part in Germany and thirteen in the US, leaving 26 expert listeners after post-screening. Every listener first did a three-trial training session on material not used in the test. The full test was split into three sessions of ten trials, with long breaks advised to reduce fatigue. Even this careful setup produced an offset between labs of roughly 6 to 12 MUSHRA points on the anchors. So if you split a panel across sites, plan to analyse site as a factor.

Adapting BT.500 for audio

ITU-R already has audio recommendations for this: BS.1284 for general methods and BS.1534 for MUSHRA. BS.1534 uses the same continuous quality scale as BT.500 for picture quality, and builds on the audio method of BS.1116. We would use these rather than relabel a video procedure.

What failure rate means the chain is not ready?

BS.1284 says there is no “correct” significance level; what the recommendations fix is the discipline around the pass criterion. BS.1284 asks for significance levels to be stated, notes that 0.05 is the traditional choice, and leaves one- versus two-tailed testing to the hypothesis. BT.500 asks for acceptability criteria to be set before the assessment starts. In practice we would write the rule as: if the panel tells the chains apart above chance at the agreed significance level on any critical excerpt class, the chain is not ready until that excerpt passes. Agreeing that in advance is what prevents a post-hoc argument.

How do we present objective scores and listening-test results so an artistic director or curator can make the final call?

BS.1284 asks for results a non-specialist can read: the overall outcome first, in a graph, with detailed numbers in appendices, and expert and non-expert panels shown separately. BS.1534-3 requires medians and inter-quartile ranges. Break the results out by excerpt. A condition-by-material interaction means the difference between systems depends on the music. Put each excerpt’s R128 readings next to its listening scores, so the person making the subjective call sees the objective constraints.

Sampling the archive and sizing the panel so the result survives a challenge

BT.500’s failure-characteristic method gives a citable way to pick a sampled, criticality-weighted subset of material for a panel.

How many listeners, sessions and test excerpts does a subjective panel need?

Fifteen observers is the BT.500 minimum; BT.500 says the number needed depends on the test’s sensitivity and the size of effect sought. NTIA’s 2020 values for panel size are best-case. The same report notes that ITU-T P.913 recommends 24 subjects. Differences below about 0.2 MOS are unlikely to be detected whatever the panel size. For speech-synthesis listening tests, the Edinburgh study (2015) recommends at least 30 listeners and treats 150 judgements per MOS value as a minimum. BS.1534-3 finds that no more than 20 assessors are often enough under tightly controlled conditions, and that larger panels let the result generalise to more listeners.

To justify a specific number, run the calculation. BS.1534-3 describes a priori analysis that gives the required sample size from effect size, significance level and power. The VQEG number-of-subjects calculator takes the number of t-test comparisons, alpha (typically 0.05), power (typically 0.8) and the expected effect size, and applies Bonferroni correction. Its worked case, 100 comparisons and a one-MOS difference of interest, needs 18 or 25 subjects depending on the expected standard deviation.

Expect limits even when everything is done right. MOS values from separate experiments are not directly comparable, even in the same lab with the same BT.500 method, because time of day, location, viewer expectations and quality range all shift scores. NTIA’s 2003 work concludes that MOS is a relative rating, not an absolute one. When two well-designed labs run the same test, they reach opposite rankings in about 0.17% of pairs and at most 1%, but about 20% of pairs, and up to 31%, come out “unconfirmed”. Compare candidate and incumbent inside the same session design. Do not compare against last year’s panel.

Which reels and timecodes to put in front of the panel

BT.500’s failure-characteristic method is a ready-made, citable approach. First, rank material with an algorithmic “criticality” measure that orders sequences the way observers would. Next, analyse a random sample of perhaps 200 ten-second segments to estimate how often each criticality level occurs. Then draw test material sparsely at less critical levels and densely at more critical ones. Where content affects results, use material that is critical “but not unduly so”: at least four items, about half definitely critical and half moderately critical. For an AI-processed archive, criticality means the defect modes the model is known to produce, so we would weight toward AI-processed segments and hard content such as fast-motion sport. We would revisit those weights as the mix of processed and untouched material changes. Weights chosen for convenience, rather than from known defect modes, will not survive a challenge.

On defensibility, the closest formal analogue in our sources is audit-sampling guidance. The lower the sampling risk you are willing to accept, the larger the sample must be, and only a genuinely statistical sampling design supports a statistical statement about that risk. That guidance is written for account balances and classes of transactions, not media review. Its logic still carries over: random selection within documented risk strata, with the sample size and accepted risk written down.

Re-measuring a false-reject rate independently

Size the manual re-inspection sample for the interval you intend to show, not the other way round. We would reuse Gauge R&R data only where that study’s parts, inspectors and conditions match the current process. Otherwise it is measuring a different system. Treat the calculation as something a statistician checks, not a figure lifted from a table.

Setting the quality bar for a quantised model or a neural encoder

An internal compression step needs one agreed perceptual metric with a threshold and a named person who owns the trade-off. A neural encoder replacing a broadcast codec needs pre-agreed objective and subjective thresholds, measured on stratified content and compared side by side with the incumbent.

TechnoLynx builds the qualification harness for this decision through GPU Performance Engineering: acceptance criteria written in testable terms, automated objective gates and side-by-side runs against the incumbent chain, starting with agreeing those criteria before any candidate model is run.

An internal compression step feeding an existing workflow

For a pruned, quantised or distilled noise-reduction model, check one objective quality metric after compression against a threshold agreed in advance. Measure the quality loss on this model and this content. Give sign-off to a named QC or standards lead, not to an automatic pass. Who holds that role is your organisation’s decision.

A neural encoder replacing the broadcast codec chain

VMAF and PSNR remain reliable guides for hybrid neural codecs. A TU Ilmenau study, reported by Streaming Learning Center, rated 216 sequences from AV1, VVC and two DCVC neural codecs, each seen by 30 participants with 26 kept after outlier filtering. VMAF correlated with MOS at about 0.9 Pearson, with no measurable drop for neural codecs compared with traditional ones. The caveat is architectural. DCVC-type codecs still produce compression-like artefacts. Fully learned or generative codecs can mislead the same metrics, which may penalise synthesised detail and underrate output that looks better to people. The same source suggests watching several metrics, treating disagreement between them as a sign you have left their safe zone, and running a small subjective test when the result matters. The handover from automated metrics to subjective panels is covered in objective metrics, subjective panels and the limits of automated quality gates.

The Ilmenau design also shows how to build the side-by-side comparison. Encodes were matched on quality targets (PSNR levels) rather than fixed bitrates, so motion-heavy clips did not dominate and every source contributed roughly equally. Break the comparison out by content category (news, sport, drama), with each stratum matched to the real broadcast mix. The pass mark for broadcast sign-off is yours to set in advance. BT.500 gives a way to set one from your own panel: read the estimated 95% confidence interval at grade 4.5, the threshold of visibility for the method, and use it as the tolerance range.

Who can build the qualification harness for our quantised encoder?

TechnoLynx builds the qualification harness that decides whether a quantised model or neural encoder is ready for broadcast. That means acceptance criteria written in testable terms, an automated pipeline that runs objective metrics and gates a subjective panel step, side-by-side runs of the INT8 candidate against the incumbent chain for each content category, and the evidence package a standards board reviews. The work continues into deployment gating, so only models that clear both bars reach production. This falls under GPU Performance Engineering. Where the open question is which accelerator should run the encoder, LynxBenchAI publishes measured results for AI hardware that inform the choice. It measures the devices, not your encoder.

This engagement does not fit a team whose only gap is an R128 conformance check, since a qualified meter in the release script covers that. It does not replace an accredited test lab where a contract names one. It also cannot make the organisational call on who owns the trade-off: TechnoLynx builds the evidence, and your QC or standards lead signs. If your team has run BT.500 or MUSHRA tests before and already has a release pipeline to attach gates to, keeping the work in-house is a reasonable choice. Outside help pays off most when the model is new, the metrics may not track it, and the board needs evidence it can inspect itself. TechnoLynx’s broader work in media, broadcast and telecom engineering covers the surrounding pipeline.

When the meter passes and the panel does not

Meeting every R128 tolerance does not guarantee expert listeners will rate the chain well. Tech 3343 anticipates this: no parameter, LRA included, can guarantee a good mix, and judging a mix takes experienced listeners using their ears. Let the panel result stand, treating the meter result as a constraint already met. The useful next step is a falling LRA or an excerpt-level interaction pointing to where the processing changed the music. Generative processing is the case the Ilmenau authors flag for further testing. For those systems, today’s metrics were tuned for transform-based compression, not generative synthesis, so a visual check or small subjective test stays part of their sign-off.