A quality bar for a new video encoder or de-noising filter holds up in front of a board only if it is written down before the first test clip is encoded. It needs six things: per-category baselines measured on the chain you already run, VMAF backed by a properly sized MOS panel, a pass line set on the worst content category rather than the average, a rule agreed in advance for when the metric and the panel disagree, a trial at production load, and an armed rollback trigger. ITU-R BT.500 says plainly that acceptability criteria should be decided before the assessment begins. Generic scores and clean lab feeds fail that test because they describe someone else’s content under someone else’s conditions.
This page covers the gate itself. The wider question of designing defensible quality validation for codec and AI pipeline changes, including model and pipeline changes that do not touch picture quality, is covered separately.
The quality gate at a glance
| Gate component | What it fixes before testing | Where it comes from |
|---|---|---|
| Per-category baseline | VMAF and MOS of the incumbent chain on each content type (news, sport, drama and so on) | Incumbent measurement under matched conditions |
| Test material | Critical but not unduly critical sequences, at least four items per case, half definitely critical and half moderately critical | ITU-R BT.500 |
| Subjective panel | At least 15 observers, with mean and 95% confidence interval reported per test parameter | ITU-R BT.500 |
| Objective suite | VMAF as the primary metric, with PSNR and SSIM computed by the same libvmaf package | Netflix VMAF repository |
| Pass threshold | Candidate meets or beats the baseline in every category, with the uncertainty of the objective score stated | Design choice, agreed in advance |
| Disagreement rule | The subjective panel decides when the metrics and the panel conflict, unless a known metric blind spot explains the gap | Agreed in advance |
| Production check | Shadow traffic, then a staged canary compared against a control population | Canary practice |
| Rollback trigger | Predefined thresholds that promote, escalate or roll back automatically | Automated canary analysis |
How do we baseline VMAF and MOS on news, sport and drama before testing a replacement?
Measure the incumbent MPEG-2 or HEVC chain on each content category, under the same conditions the candidate will face, and use those per-category numbers as the bar. A single blended average does not work as a bar. No published figure can stand in for this step, because the scores depend on your sources, your bitrates and every stage of your chain.
LynxBenchAI covers the hardware side of this baseline: when a candidate encoder or filter needs new accelerators, its published results compare them on throughput, latency and energy, so the first step is to check those results for the hardware on your shortlist; it does not score picture quality.
TechnoLynx builds this qualification harness, and its Inference Cost-Cut Pack cuts the candidate’s inference cost. TechnoLynx starts by listing your content categories and fixing the baseline protocol and a decision score that weighs VMAF, BD-rate and cost together. TechnoLynx then runs the incumbent chain and the candidate encoder under matched conditions and leaves the harness in place as a repeatable gate.
Categories should follow how your programming actually varies. BT.500 itself builds its continuous-evaluation tests around programme segments of one programme type, such as sport, news or drama, each at least five minutes long. The Alliance for Open Media common test conditions (AOM CTC) take the same approach for codec work: sources are split into categories with matched subsampling, resolution and frame count, and some classes are mandatory.
Within each category, pick critical material that stresses the system. BT.500 notes that you can allow for critical material when interpreting results, but you cannot extrapolate from non-critical material. Darker scenes are a known difficulty for both encoders and quality algorithms. We would make sure every category’s sample includes its own worst case: crowds and grass in sport, low light and grain in drama, captions and fast cuts in news.
How large and varied does the MOS panel need to be for sign-off?
BT.500 asks for at least 15 observers. An exploratory study with fewer should be labelled “informal”. For its single-stimulus continuous quality evaluation test, BT.500 asks for non-expert observers. The same clause says the number needed depends on how sensitive and reliable the test procedure is and on the size of the effect you are trying to detect. When the expected difference between incumbent and candidate is small, size the panel with a power analysis rather than stopping at the minimum.
The protocol details matter as much as the head count:
- Sessions last up to half an hour. About five dummy presentations open the first session to settle observers’ opinions, and those votes are excluded.
- Showing reference/reference pairs is a simple reliability check. Observers who understood the task score them close to the top of the scale.
- Every mean score carries a 95% confidence interval derived from the sample’s standard deviation and size.
We cannot say what a particular standards board will accept. What we can say is that BT.500 provides the assessment methodology, and that the panel design is best agreed with the board before fieldwork. If each channel or content stratum needs its own adequately sized sample, the cost multiplies, and that limits how many strata one test cycle can cover.
One pass/fail line that holds on the worst category
Set the gate so that the candidate must meet or beat the baseline in every category. If any category fails, that category is not approved, and neither is the full chain replacement. One study on codec metrics found that metrics with similar average correlation can drop sharply on individual videos in the test set.
Savings figures need the same caution. A 2022 study revisiting Bjøntegaard delta bitrate (BD-BR) recommends that comparisons use bitrates representative of real operating points, and that BD-BR be supported with RD plots unless the curves are well behaved. Bounded metrics such as VMAF and MOS saturate, especially at high bitrates, and that saturation has to be considered to get practical savings figures.
Objective thresholds should carry their uncertainty. Since v1.3.7 (June 2018), VMAF can attach a 95% confidence interval to each prediction. That interval is computed either as the bootstrap score ±1.96 times its standard deviation or from the 2.5th and 97.5th bootstrap percentiles. Netflix recommends plain bootstrapping because it is unbiased with respect to the full model, even though its intervals are wider. A defensible rule compares the lower bound of the candidate’s interval with the baseline, not just the point estimates.
A worst-case gate is stricter on a category that is simply harder to compress. That trade-off belongs to the board, recorded before testing. It should not be quietly relaxed after a result disappoints.
Metrics and thresholds for a neural de-noiser on SD archive material
To compare neural de-noising against legacy MPEG-2 grain reduction, use VMAF as the primary metric and SSIM and PSNR as supporting checks. Score everything against the legacy chain’s output, and derive thresholds from your own SD archive rather than from published values. All three metrics come out of the same libvmaf build, which also implements PSNR-HVS, MS-SSIM and CIEDE2000. In a multi-codec study from Moscow State University, VMAF variants correlated best with subjective scores and PSNR variants worst among full-reference metrics.
An SD archive has its own grain, resolution and compression history, so a number from someone else’s test set says little about your material. The gate also has a limited scope: it validates output quality, not the throughput or hardware cost of the new pipeline, and it needs revalidating whenever the archive mix or the neural model changes.
VMAF alone is not enough to decide cutover. Zvezdakova and colleagues (2019) showed that colour and contrast corrections applied to compressed video raise VMAF while barely moving SSIM, which makes the metric open to tuning in codec comparisons. Later work found that for video specially preprocessed for the metric, visual quality can drop while the correlation turns negative. In the Moscow State University multi-codec dataset, at high bitrates VMAF significantly outperformed other metrics, with a correlation of 0.7 against 0.25 to 0.45. The VMAF NEG (“no enhancement gain”) model exists to suppress the effect of sharpening and contrast changes so that only compression is measured. Netflix recommends NEG for codec comparisons, and classic VMAF when you know the algorithms do not inflate the score. A neural de-noiser does alter the image beyond compression, so we would compute both and treat a large gap between them as a reason to investigate.
When VMAF or SSIM says the new filter passes but the panel disagrees, which result decides?
By default the panel decides, under a rule written down before the test. BT.500 accepts that system performance may not be fully characterised by objective means and must be supplemented by subjective measurement. The AOM CTC uses subjective testing when objective metrics contradict one another or when a tool is expected to affect visual quality.
| Outcome | Decision |
|---|---|
| Metrics pass, panel flags artefacts | Fail. The panel outcome stands for cutover |
| Metrics fail, panel passes, failure traced to a known blind spot (for example PSNR penalising retained grain) | Gather more panel data on the affected content before deciding |
| Metrics fail, panel passes, no known blind spot | Fail until the cause is understood |
| Metrics disagree with each other | Run or extend the subjective test on the conflicting sequences |
| Metrics and panel both pass in every category | Proceed to production-load validation |
Where objective scores and panels diverge more broadly, and how far an automated gate can be trusted, is the subject of bridging objective metrics and subjective quality validation.
Which objective metrics actually track de-noising and deinterlacing artefacts on SD archive content?
No metric can be assumed to track these artefacts. VMAF is the first-pass screen we would use, but before it is trusted for bulk screening, its correlation with how people perceive grain and de-noising artefacts on your material has to be checked against panel scores. LynxBenchAI comes in at one point here: if the hardware for the neural filter is still being chosen, its published results compare candidate accelerators on throughput, latency and energy. It does not score picture quality and does not run your filter.
The sources do not settle how metrics handle grain. Bampis and colleagues note that VMAF was trained on sources containing film grain, so that it accounts for grain when predicting quality, while also admitting that grain’s effect on perceived quality is not always clear. A 2024 study on film grain coding in VVC found that PSNR and MS-SSIM penalise added grain even when perceptual quality improves, and that VMAF is not specifically trained to evaluate that case. Both points lead to the same practice: check correlation on your own archive before relying on the metric.
Two further limits apply to restoration filters. An ad hoc NTIA evaluation indicated that VMAF responds only to added compression artefacts, in line with broadcasters treating camera capture impairments as artistic intent. For a de-noiser, that means the metric judges fidelity to whichever reference you give it, not whether the noise was worth removing. For temporal artefacts, VMAF uses little temporal measurement beyond average frame differences, while ghosting and flicker deeply affect perceived quality. VMAF also does not simulate differences in frame rate. The sources here are thin on deinterlacing specifically, so the panel should carry more weight for that artefact.
A video de-noising quality gate, step by step
- Score the full content set with VMAF (and NEG) against the legacy grain-reduction output as an automated screen.
- On a representative sample per channel, run a correlation analysis between the objective scores and panel scores for this artefact type.
- Where the correlation is strong, keep VMAF for bulk screening. Where it is weak, widen the subjective sample.
- Run a power-sized panel per channel before final sign-off, and escalate to full subjective review when objective and subjective results diverge.
- Record thresholds per channel, because one global threshold may not hold across channels.
The choice of test method matters for archive material. One study comparing subjective test methods found that ACR-HR had lower sensitivity to quality drops for fair-quality sources. It recommends CCR (comparison category rating) for denoising, scaling and restoration, or whenever the source is not of good quality. We would treat SD archive content as falling in that group. Under-sizing the panel to save money is the quickest way to an indefensible gate.
How do we validate a new codec or filter at production load, not just on a clean lab feed?
Run the candidate on mirrored production traffic first, then move to a staged canary compared against a control, and ask for board approval only after both pass and a rollback threshold is armed. TechnoLynx sets this up through the Production AI Monitoring Harness. TechnoLynx starts by framing the qualification criteria and drift thresholds with your quality team, then implements the gated harness on the target hardware, validates the candidate against the incumbent chain across content categories, and tests the rollback logic before the harness moves into your production pipeline.
Shadow testing comes first. Production requests are duplicated to both the current and the new version, and outputs, latency and errors are compared offline without affecting viewers. A common pattern is to confirm stability and basic parity this way before starting a gradual canary. For load testing an encoder pipeline, the Google SRE Workbook makes a point specific to video: because encoding work units take a long time, the canary must last at least as long as one work unit. The canary must also be large and long enough to be representative. The Workbook strongly advises running only one canary at a time.
Compare canary and control separately. The SRE example shows a 5% canary with a 20% error rate appearing as only 1% in whole-service monitoring. Real subscriber conditions include the playback side as well. Before its AV1 A/B tests, Meta benchmarked phones with a small compute test, because core count, RAM and model year were poor predictors of whether a device could decode in real time. Checking that a new ladder behaves across your real client decoder population is a separate validation, and it belongs in the plan.
Both steps have limits. We treat neither shadow traffic nor content strata as covering every peak surge or production edge case.
What rollback trigger should be in place if quality drifts after go-live?
Define the trigger as acceptable-behaviour thresholds on a short list of metrics that reflect what viewers notice. Thresholds that are too strict produce false positives. Thresholds that are too loose let a bad release through. A staged rollout (for example 1%, then 5%, 25%, 50% and 100%) rolls back to the stable version at any stage where the new version underperforms. The checks against those thresholds should be automated.
Kayenta, an automated canary analysis tool, shows what a usable trigger looks like. It scores each metric statistically, combines the results into a 0 to 100 score using configured weights, and classes the result as success, marginal or failure. Success promotes the canary, marginal sends it for human approval, and failure triggers a rollback. For a quality gate, we would feed it per-category VMAF on sampled live outputs and compare against the pilot’s confidence intervals, alongside the operational signals. We would also keep the incumbent chain ready to take traffic back for the whole canary period. Expectations need regular review as the service changes.
Turning a one-off AV1 vs HEVC comparison into a reusable pre-rollout gate
A one-off codec comparison becomes a gate once its content set, metric versions, operating points, panel design and pass rules are fixed and run automatically on every future change. TechnoLynx builds that standing gate as the Production AI Monitoring Harness, drawing on its Production AI Reliability work. The deliverable is the harness, regression suite and release gate.
A reusable workflow for codec and ladder changes
Start from a fixed reference set. AOM names libvmaf as the standard metrics tool under its common test conditions, and the CTC defines mandatory sequence classes. Pin model versions too: Netflix released a new VMAF v1 model set in June 2026 while keeping the v0 models documented. Pin the model version so every run of the gate uses the same model.
Do not let one savings figure stand in for the gate. The BD-BR study found that the original BD-BR functions were designed for PSNR, and that results computed with other metrics should be interpreted with care. The BD-BR study notes that Netflix’s larger comparisons report low, medium and high quality ranges separately, because the savings can differ a lot between them. In the NTIA dataset, VMAF’s response to CRF changes differed by codec. NTIA observed that AV1’s VMAF rating drops off less steeply than AVC’s or HEVC’s as CRF rises, which does not necessarily mean AV1’s absolute quality is better. Meta’s AV1 work scored quality on a 0 to 100 FB-MOS scale and built a convex hull from 35 rate-distortion points per video. A reusable gate keeps RD curves at that level of detail rather than reducing them to one percentage.
Gating each quantisation or packing change in CI
To make “no visible quality loss” enforceable in CI, express it numerically: the candidate’s per-category VMAF confidence interval must stay within an agreed margin of the last approved build. A panel runs only when that check fails or the metrics disagree. Quantised speech or audio models need their own metrics and thresholds, which this gate does not set.
Cutting the cost of subjective testing without losing rigour
The cheapest subjective test is the one you do not need, so use objective screening to target panels at the sequences where metrics conflict, which is one of the AOM CTC’s triggers for subjective work. The choice of method matters as well. Naderi and Cutler (2025) found ACR-HR about twice as fast and cost-effective as the alternatives, but warn that the method chosen shifts saturation points and bitrate-ladder recommendations. ACR-HR suits general trends on good-quality sources, while CCR is the choice for restoration work or weaker sources. Exploratory rounds with fewer observers are acceptable if they are labelled informal.
Who can build a pre-rollout quality gate for our encoder swap, and should we do it in-house?
TechnoLynx builds the qualification harness that gates an encoder, codec or filter change. That covers per-category baselines on your incumbent chain, the objective pipeline with a power-checked panel design, shadow and canary validation, and the rollback logic that stays in place after go-live. On build versus buy: a team that already runs libvmaf at scale and has access to a subjective test lab can build this in-house. Bringing in a specialist pays off when the harness has to be designed, made statistically defensible and wired into production at the same time. When choosing a partner, ask to see how they set thresholds and handle metric/panel disagreement before any test runs. TechnoLynx offers this as the Production AI Monitoring Harness, part of its wider AI engineering for media, broadcast and telecom work.
The engagement has limits. It delivers engineering artefacts such as eval harnesses, regression suites and release-readiness reviews, and a dashboard is not the deliverable. If you need a certified subjective test, an accredited test lab is the right route.
FAQ
Is AV1 higher quality than H.265?
It depends on content and operating point, and no single metric settles it. NTIA notes that VMAF reacts differently to CRF changes for AV1 than for HEVC, and that this difference does not prove better absolute quality. Meta reported playback FB-MOS improving by about 0.6 points after moving Facebook Reels on iPhone to AV1, which is a measurement on its own content.
What is the difference between subjective and objective scoring?
Objective scoring computes a quality estimate from the video itself. VMAF, for example, fuses several quality measures with a support vector machine. Subjective scoring asks viewers to rate the video, producing MOS or DMOS with confidence intervals.
