Video codec quality testing is only good rollout evidence when it is built for the specific change you are making. Classical VMAF gives the more accurate estimate when comparing versions of the same encoder, or encoders known not to inflate the score. A neural or restoration filter, film-grain synthesis, scaling, or a move to a new codec family needs a subjective test, because the metrics can reward the wrong result. We would split the test set by genre, channel, source quality and motion, and set the go/no-go gate separately for each of those groups. An average across the fleet can hide the outliers on individual channels.
Do PSNR, SSIM and VMAF settle a codec change, or do we need a viewing test?
Classical VMAF gives a more accurate estimate when comparing versions of the same algorithm. We would not trust any metric alone to judge a change that alters what the picture is made of. One study compared objective metrics against Subjectify.us viewer comparisons across 789 encoded streams from 39 codecs. It found that VMAF matched visual quality better than the other metrics, with a correlation of about 0.7 at high bitrates against 0.25 to 0.45 for the rest. The same study found that PSNR has low relevance for AV1 streams. Its explanation is that newer encoders restore object boundaries in ways that break pixel-by-pixel similarity.
The study also notes that VMAF can be gamed by preprocessing. Video specially preprocessed for VMAF can lose visual quality while the correlation turns negative, and libaom’s --tune vmaf setting was shown to reduce visual quality. Netflix developers recommend VMAF NEG for comparing codecs, and classical VMAF only for versions of the same algorithm, or algorithms known not to inflate the score. Even the metric configuration affects the ranking. Computing PSNR or SSIM on luminance only, rather than on a weighted YUV sum, can change how the compared encoders rank.
Film grain is a case where pixel-wise metrics penalise a good result. The authors of the AV1 film-grain synthesis proposal evaluated it by informal subjective comparison. Their reason was that objective metrics would not work well for a method that removes grain and re-synthesises similar grain. Netflix wrote in 2025 that it still lacks a dedicated quality model for grain synthesis, and that PSNR and VMAF penalise it because the noise lands at different pixel locations. Archive restoration has the reverse problem. A 2026 archival-film benchmark found that standard no-reference metrics reward historically inauthentic hallucination and over-smoothing, and that BRISQUE sometimes improved while perceptual quality fell.
Decision table: metric-only or subjective test
| Change under test | Objective metrics enough? | What to add |
|---|---|---|
| Parameter tuning of a trusted encoder, same codec | Usually; classical VMAF suits same-algorithm versions, and MS-SSIM beats SSIM | Watch for sharp score drops on individual clips |
| New codec family (for example HEVC to AV1) | No; PSNR in particular loses relevance on AV1 | A subjective test, with VMAF and MS-SSIM as the strongest objective metrics |
| Film-grain synthesis | No: pixel-wise metrics penalise re-synthesised grain | A subjective test on clips with varying graininess |
| Scaling, blur, restoration, super-resolution | No: use a comparison method that can register gains over the reference | CCR (comparison category rating) rather than ACR-HR |
| Neural restoration of archives with no clean reference | No: no-reference metrics can reward hallucination | Expert review against the authenticity you are aiming for |
| Encoder or preprocessing tuned towards a metric | No: correlation can invert | VMAF NEG plus a subjective check |
The subjective method should follow the study objectives. One comparison found that ACR-HR is about twice as fast and cost-effective, and suitable for general trends on good-quality sources. CCR is more sensitive for scaling, blur, restoration and fair-quality sources, and can record improvements beyond the reference, such as super-resolution. ITU-T P.910 recommends DCR when fidelity to the source is what matters. It also suggests a staged approach: run ACR or DCR with a few observers first, then use paired comparison only on the items that scored about the same. Turning a metric into a threshold you can defend is a separate task, covered in putting video quality metrics to work with thresholds and validation.
Sizing a genre-stratified video codec quality test for rollout
We would set sample size for a genre-stratified test separately for each genre. If sports, drama and animation each get their own go/no-go, then each is its own statistical decision and needs enough power on its own, not just a large enough total. The VQEG subject-count calculator makes the inputs explicit: the number of t-test comparisons, the significance level (typically 0.05), the power (typically 0.8), paired or independent design, and the expected effect size (the MOS difference divided by the standard deviation).
How many viewers and clips per genre make an ABX test defensible?
The required number depends on the comparisons planned and the difference sought. The standards and studies do set floors, and they differ by what the result has to support:
| Basis | Minimum | What the result supports |
|---|---|---|
| ITU-R BT.2095 (2016), expert viewing protocol | 9 expert viewers | A preliminary ranking only, not a replacement for a formal test |
| ITU-R BT.2095, expert viewing protocol | 15 or more expert viewers | MOS, standard deviation and confidence intervals for a more accurate ranking |
| ITU-T P.910 (1999) | At least 15 observers | A general subjective test |
| ITU-T P.1401 (2020) | 24 voters per sample or per condition | Evaluating objective models against subjective data |
| VQEG calculation, 100 videos and 100 comparisons | 18 or 25 subjects, depending on expected standard deviation | Detecting about one MOS step (for example good to fair) |
Smaller differences need more people. The VQEG figures assume only about one MOS step of difference matters. Looking for finer differences, or running every pairwise comparison, needs more subjects. For reference on an ABX design, a 2024 neural audio codec paper had at least 30 listeners rate 20 randomly selected pairs per codec comparison and reported a t-test p-value alongside the mean preference.
The choice of clips matters as well as the number of viewers. P.910 asks for at least four different scene types and test scenes that span the full spatial and temporal information (SI/TI) range of interest. With only four sequences, it suggests taking one from each quadrant of the SI/TI plane. In the 2021 codec study, each pair collected at least 10 responses, and more for clips that were hard to tell apart. We would size the clips in each genre the same way: enough clips to cover that genre’s SI/TI spread, with extra votes going to the clips that sit close to the decision boundary.
Testing many genres or channels raises the risk of false positives. VQEG notes that 100 comparisons at 95% confidence give a greater than 99% chance of at least one false positive. The Bonferroni correction divides the significance level by the number of comparisons. For three genre gates at an overall α of 0.05, that means each genre is tested at 0.05 / 3 ≈ 0.017. For 40 channel gates, each channel is tested at 0.05 / 40 = 0.00125, and the number of comparisons feeds into the required sample size. ITU-T P.1401 points to Holm’s method as just as safe and more efficient once comparisons exceed about 1000. When real-world data is limited, building statistically defensible validation under limited real-world data covers the trade-offs.
Should the go/no-go be weighted differently for sports and archive content?
Weight the genres for priority, but never let a strong genre make up for a failing one. ITU-T P.1401 allows different databases to carry different weights in an overall figure of merit. It also requires minimum thresholds both for each application and across applications. In practice that means each genre gets a floor it must pass on its own, and weighting only decides how the passing genres add up to an overall verdict. Sensitivity differs with source quality. The 2025 subjective-method comparison found that conclusions on blur and scaling changed between good-quality and fair-quality sources, while frame freezing gave much the same answer on both. We would expect the genres to fail differently and set each floor from that genre’s own observed sensitivity.
How do we stop a regression on one or two channels being averaged away?
Gate each channel on its own result: a pass means no channel regressed, not that the fleet average held up. VMAF by default reports the arithmetic mean of its per-frame scores. One study found that this mean hides badly degraded frames and so overstates quality. In the same study, Minkowski pooling with p = 8 correlated best with subjective scores on the Netflix and VQEG HD3 datasets, because it gives more weight to the most distorted frames. The codec study found the same at clip level: metrics with similar average correlations can score much worse on individual videos. Its advice is to use several metrics and watch for sharp drops on individual outputs.
ITU-T P.1401 sets out a worst-case rule that supports per-channel gates. An evaluation has to consider the worst-performing case across four views: per experiment, across experiments, per file and per test. An algorithm that wins overall but is worst in even one instance needs a deliberate decision before it is accepted. The recommendation does not endorse simply averaging metrics across experiments, because the average has no clear statistical meaning. Applying this to an archive fleet, we would treat each channel as an experiment and its worst clip as the per-file case, and set the release gate so that no channel falls below its floor.
What is a representative clip sample per channel for grain and denoising edge cases?
We would build a representative sample for 40 SD archive channels by grouping the channels by grain and noise profile, then sampling within each group. That should include the heaviest-grain and noisiest material each channel carries, not just its typical output. Grain strength can vary with signal intensity, and it should be estimated from smooth regions only, because edges and texture distort the estimate. That makes smooth, mid-tone and dark scenes good material for checking whether a denoiser is removing grain it should keep. Netflix’s grain evaluation used about 300 catalogue titles with varying graininess. It reported a smaller 10% bitrate reduction below 1080p, likely because noise is filtered out during downscaling. SD channels may therefore show different grain behaviour from HD masters of the same material.
This approach has costs and risks. We would expect a stratified sample to need more clips and viewer time than a flat one. The boundaries between genres or grain groups are judgement calls that skew the result if drawn badly. Statistical defensibility also holds only if the clips reflect the channel’s real content rather than whatever was easy to pull from the archive. ITU-T P.1401 makes the same point for evaluation data: test databases should reflect each use case in proportion to how often it occurs.
Which content exposes de-interlacing failures on legacy SD and archive channels?
We would test fast motion and sports, film-sourced 3:2 or 2:2 cadence, text over film, and fine horizontal detail first. We would test these in clean form and as degraded feeds from the subscriber path. TechnoLynx builds this kind of worst-case set for a de-interlacing or super-resolution stage. The work covers a coverage matrix of motion pattern by source degradation by genre, clips gathered or added to fill it (including synthetic fast-motion and degraded last-mile variants), and a stress harness running on the target hardware with artefact rates checked against acceptance thresholds before cutover.
Cadence handling is where we would start. Intel’s deinterlacer documentation describes detecting 3:2 cadence by matching fields of the same type. It detects 2:2 cadence by looking for the comb artefacts that appear when two consecutive fields are woven together. Its detectors are built to resist false lock, where features on adjacent fields fake a cadence that is not there. The same documentation recommends a video-over-film mode for subtitles, credits and captions added on top of film content. So a worst-case set should include cadence breaks at edits, mixed cadences, and burnt-in text over film, alongside the obvious fast-pan sports footage. Weaving fields from different moments produces combing on moving edges, and thin horizontal lines flicker between fields. We would therefore include fast motion and fine horizontal detail in each cell of the matrix. One real-time neural deinterlacer was evaluated on live broadcast, legacy movies and legacy cartoons, with sports, computer-rendered content and classic films in its data. That is a reasonable range of genres to copy.
Set a minimum coverage for each cell rather than a flat total clip count. A large set that is mostly studio talk shows proves little about sports. We would include degraded feeds from DOCSIS-hybrid last-mile paths, because those are what subscribers watch, not the clean contribution source. P.1401’s guidance that evaluation databases should reflect each use case in sufficient proportion supports the same idea. Two further limits apply. A set validated against today’s encoder may not cover the next model update, and worst-case stress testing lowers the chance of a missed failure without ruling one out.
Proving an AI denoiser keeps orchestral dynamics and quiet detail
We would validate an AI noise-reduction stage on orchestral audio with a repeatable before-and-after protocol. It combines spectrogram differencing, perceptual scores from the ViSQOL or PEAQ family, and checks on onset and attack preservation, all run on a curated set of hard passages: pianissimo strings, solo woodwind, and sudden dynamic changes. We know of no reference dataset or tool built for measuring AI-induced degradation on classical music. That means the evaluation set has to be built from your own recordings, using general-purpose quality tools.
ViSQOL v3 works on a gammatone spectrogram, aligns half-second patches and computes NSIM between them. Its verbose output reports mean NSIM per frequency band and per frame. Those per-band and per-frame figures are more useful than the overall MOS for this job, because they locate the frames and bands where quiet detail has gone. The ViSQOL authors also warn that any new use case needs careful investigation. The metric is not guaranteed to give a meaningful absolute MOS outside the cases it was designed for, and developers drew on additional metrics to confirm it applied to their problem. Set the thresholds against listening tests on your own material. A 2024 codec paper is an example of covering different quality dimensions with separate metrics: log-spectral distance for the amplitude spectrum, STOI for intelligibility, and ViSQOL for overall quality.
The results depend heavily on the difficult-passage set. The Beethoven Symphony Excerpt Dataset was built as a benchmark for orchestral transcription rather than degradation testing. It shows a useful pattern, though: 20 excerpts, each in four real recordings spanning 1929 to 2019 plus a synthetic rendition, ordered from digitised vinyl with audible noise to CD quality. Its quality ratings come from Fréchet audio distance against CD references. The authors note that these ratings are coarse and do not separate noise, clipping or compression. We would arrange a denoiser test set the same way: the same passages at several recording qualities, so that a failure can be tied to the source condition rather than the music.
We would treat AI dubbing and AI upscaling defects as specific to the vendor and the model. A review protocol should therefore start from the vendor’s own QA data on known artefact types, or from a defect-cataloguing pass over a representative sample if the vendor has none, and it needs revisiting each time the vendor updates its model.
We have found no published evidence on whether an Icecast relay’s own transcoding hides or exaggerates artefacts from a new noise-reduction stage. Until that exists, we would measure at both points: before the relay to judge the stage itself, and after the relay at the codec and bitrate listeners actually receive.
Which speech conditions break caption alignment after INT8 quantisation?
We would expect, as a hypothesis to test rather than a known failure pattern, overlapping speakers, strong accents or dialects, and fast or disfluent speech to cause INT8 alignment drift first. A parity test should report results for each of these conditions rather than one average over clean single-speaker audio. TechnoLynx builds that kind of INT8 parity harness. If the open question is which device the quantised model should run on, LynxBenchAI’s published results show how each device runs the public PyTorch stack. That is broadly indicative for custom PyTorch models, though it does not measure your own model.
We would give each edge-case category enough minutes of audio to give a meaningful parity result. A handful of overlapping-speech clips shows the condition exists, but it cannot show the model handles it.
What objective test catches combing on SD simulcast, and when can the old hardware go?
We know of no reference metric validated for combing artefacts from neural de-interlacers across a mixed SD subscriber base. The gate has to be built and calibrated locally: an objective combing measure, a pass/fail threshold set against subjective review, and a test set that includes real subscriber conditions. TechnoLynx builds this kind of gate for broadcast pipelines as part of its AI engineering for media, broadcast and telecom work, starting by defining “safe to decommission” in measurable terms for the channels at risk.
LynxBenchAI publishes how each device runs the public PyTorch stack that neural de-interlacers are built on, so the first step in choosing an accelerator for the fleet is to check its published results for your candidate devices.
We would borrow the measure from how deinterlacers detect combing. Intel’s 2:2 cadence detector works by spotting the comb artefacts that appear when two consecutive fields are woven together. We would start from a comb score computed over field-woven output, concentrated on moving edges. It only becomes a gate once its threshold has been checked against expert viewing of the same clips. We would not rely on a general metric such as VMAF for this artefact. We would keep it as a regression signal next to the comb measure, not in place of it. We would not set the threshold on a clean lab feed alone.
Pre-decommission checklist for a neural de-interlacer
Treat decommissioning as premature until every item holds:
- An objective combing measure exists with a defined pass/fail threshold, calibrated against subjective review samples.
- The threshold has been validated on real mixed subscriber feeds, not only on clean contribution source.
- Worst-case fast-motion and cadence-break clips pass the same threshold.
- Each channel passes on its own result, with the worst clip reviewed, rather than on the fleet average.
- VMAF or a similar quality measure is being monitored per channel in production, so regressions after cutover are caught.
The last item is a safety net. Production monitoring catches a regression after viewers have seen it, which is why it cannot replace validation before decommission. For monitoring, NTIA notes that continuous single-stimulus rating (SSCQE) is claimed to give more representative quality estimates. Its viewers judged quality from roughly the last 9 to 15 seconds of video.
Who can build a pass/fail gate for a neural de-interlacer before we pull the hardware?
TechnoLynx builds the validation gate: a combing measure, worst-case and mixed-subscriber test sets, an automated harness on your target hardware, and per-channel monitoring that keeps running after cutover. We treat that gate as a release-readiness deliverable rather than a dashboard. If the open question is which accelerator should run the neural de-interlacer across the fleet, LynxBenchAI publishes how each device runs the public PyTorch stack such models are built on. That is broadly indicative for a custom model, though it does not run your pipeline. The first step is defining the decommission criteria for your channels and the failure modes they must catch.
The scope has limits. If you already run a viewing room and have video engineers who can build harnesses, an expert viewing protocol under ITU-R BT.2095 run in-house may be enough for a first ranking. If the decision has to stand up to outside scrutiny, an independent subjective test lab following the ITU methods is the other realistic route.
Can a new codec or neural filter run in parallel with the legacy chain before full cutover?
Yes, and it should. One widely described 2026 deployment sequence runs offline validation, then shadow testing on copied live traffic, then a canary rollout with automatic rollback, then full promotion. For edge deployments the same source recommends canarying by site rather than by request, and keeping the previous model on the device for instant rollback. In a broadcast plant the equivalent is canarying by channel, with the hardware chain kept warm as a fallback. The statistics of the parallel run and its rollback criteria are covered in designing a defensible parallel-run validation before cutover.
Validating a pre-processor or adapted ASR pipeline against worst-case archive material
The two cases below need different tests.
Archival denoisers: test the hardest reels first
We would treat heavy grain, scratches, low light, and fast motion over fine grain as the worst cases for an archival denoiser. These are where a denoiser either removes texture that should stay or smears it over time. One recent archival-film benchmark models degradation families physically, including exposure-dependent grain, diffraction-based scratches and camera jitter. It applies them consistently across a clip rather than frame by frame. Its authors note that standardised evaluation datasets for archival film barely exist, partly because copyright restrictions keep authentic footage from being released. They also report that quantitative metrics often contradict visual quality in this area. A minimum test set for a legacy catalogue is therefore defined by coverage of each degradation family and genre, not by a clip count, and passing it needs expert review rather than a score.
Adapted ASR: prove it on the show itself
We would test an ASR pipeline adapted to fix name and place recognition on the show’s own past episodes. Hold some out from adaptation, build reference transcripts that mark every named entity, and compare the entity error rate before and after adaptation. Producing that evidence is engineering work. The decision that the error rate is low enough to avoid further QC rejections belongs to whoever owns delivery QC for the broadcaster, and it is worth agreeing that bar before the measurement starts.
The calibration gaps that remain open
We found no external reference for two parts of this design: an objective measure for combing artefacts from neural de-interlacers, and the effect of relay transcoding on audio artefacts. Both thresholds have to be calibrated locally against human review, and that calibration has to be repeated whenever the model or the distribution chain changes. That is uncomfortable, but it is also the main reason to keep the legacy chain and the per-channel gates in place until the locally calibrated measures have shown, on your own channels, that they agree with what viewers see.
