VMAF (Video Multimethod Assessment Fusion) is a full-reference perceptual metric developed by Netflix. It fuses several elementary quality measures with a machine-learning model trained on human opinion scores, and it reports a number from 0 to 100. There is no universal “good” VMAF score. One 4K study reported about 93 as an operating point that served most of its audience. A VMAF threshold becomes a defensible quality gate only after four things: you calibrate it against subjective panel scores and your own complaint history on your content, you report it with its uncertainty, you check it against the delivery spec, and you back it with human review where the metric is known to be weak.

What is a good VMAF score, and how is it calculated?

VMAF computes elementary metrics (VIF, DLM, a motion feature and ADM) and fuses them with a support vector machine trained on mean opinion scores (MOS). It was designed around quality loss from compression and rescaling. Anything outside that, such as hallucinated texture from a neural upscaler, sits at the edge of what it was trained to see.

The training data fixes the scale. The default model’s subjective scores were collected with an absolute category rating (ACR) method on a continuous scale, cleaned with maximum-likelihood estimation and mapped so that “bad” lands at roughly 20 and “excellent” at 100. That model (vmaf_v0.6.1) predicts quality on a 1080p HDTV in a living-room setting: every distorted video was rescaled to 1080 and watched from three times the screen height (3H). A separate 4K model, added in v1.3.7 (June 2018), assumes a viewing distance of 1.5H. A phone model lets viewers choose their own distance.

Scores are produced per frame and then pooled into one figure. The pooling method can be the mean, the minimum, the maximum or the harmonic mean. By default VMAF uses the arithmetic mean, although the Netflix FAQ notes psycho-visual evidence that human opinion weighs more heavily towards the worst frames, and how best to pool is still an open question. A long, easy title with a few bad seconds can therefore score well on the mean and still draw complaints.

Resolution also matters. If you compute VMAF on a 480-line pair at native resolution, you are in effect modelling a viewing distance of about 6.75 times the picture height, which hides a lot of artefacts and inflates the score. The FAQ’s guidance is to never compare the absolute VMAF of a 1080 video with a 480 video scored natively. Instead, upsample the 480 video to 1080 and score it against the 1080 source.

Netflix’s documentation says models 0.6.2 and 0.6.3 show no material difference from 0.6.1, and that they add a bootstrapped prediction confidence interval.

The figure of 93 comes from a 4K study comparing VMAF with subjective scores. Eighteen subjects rated 4K clips encoded at a range of bitrates, using BT.500 double-stimulus testing and differential MOS. VMAF correlated with MOS at 0.948 in that study, close to Netflix’s own 0.963 and 0.939 on the NFLX-TEST and VQEGHD3 datasets. That shows a strong correlation for 4K compression of that content. It says nothing about SD archive material passed through a neural upscaler.

Checklist for reading a VMAF number

Check	Why it matters
Which model (default, 4K, phone) and version	Each model assumes a different display and viewing distance.
Resolution the score was computed at	Native low-resolution scoring hides artefacts and inflates the score.
Pooling method	Mean pooling can under-weight the worst frames that viewers remember.
Confidence interval	0.6.2 and 0.6.3 can report a bootstrapped prediction interval.
Whether the encoder was tuned for VMAF	Tuning and preprocessing can raise the score while lowering visual quality.

How do you calibrate a VMAF threshold against real viewer complaints?

You calibrate a VMAF threshold by deriving it from the outputs of the system you are gating, then scoring it against subjective ratings and against the cases viewers actually complained about, and writing down its error rates before anyone relies on it. Generic compression defaults do not carry over, because a metric’s correlation with subjective scores varies greatly with the dataset and the type of distortion.

TechnoLynx does this calibration as an engineering engagement. TechnoLynx starts by settling which of three things you need: a new operating threshold, validation of an existing threshold against complaints, or audit documentation. TechnoLynx then builds the sampling and scoring pipeline on your hardware, correlates candidate thresholds with MOS and past complaint cases, and packages the results with their sample-size basis.

Why an upscaler needs its own threshold

In the ICME 2025 video super-resolution challenge, PSNR and SSIM correlated very weakly with subjective scores on general-purpose and talking-head content, and VMAF correlated only moderately. Even LPIPS, which had a strong Pearson correlation, had a Spearman coefficient of 0.8 or below, so the organisers concluded that ranking models still needs subjective evaluation.

Two further effects push the same way. Systems optimised for cost functions aligned with VMAF or SSIM components tend to diverge when you assess them with several metrics. For videos specially preprocessed for VMAF, visual quality can drop significantly and VMAF’s correlation with it turns negative. A threshold for neural upscaling of SD archive tape has to come from your own outputs on representative clips.

Testing a candidate threshold against MOS

ITU-T P.1401 gives the statistical framework for evaluating an objective score against subjective data. It covers accuracy through prediction error (RMSE), consistency through the outlier ratio and linearity through the Pearson correlation. An outlier is a point whose prediction error exceeds the 95% confidence interval of its MOS. The recommendation also asks for minimum performance thresholds to be defined before the evaluation runs, not after.

Two details catch teams out. Any mapping from metric to MOS must be applied per experiment, before the statistics are calculated. A third-order mapping can also hide systematic misprediction, which is why P.1401 reports results after both first-order and third-order mapping. Correlation on its own can mislead, as NTIA’s comparison of subjective test methods showed when it paired correlation with RMSE.

Panel resolution sets a floor on how fine a gate can be. A MOS confidence interval is typically around 0.15, so P.1401 notes that no ranking finer than about 0.3 MOS can be made with confidence. An objective model is also not expected to predict the average opinion better than an average test subject. A gate margin narrower than the panel’s own uncertainty is therefore a number nobody can defend.

Worked method: false-pass and false-fail rates against past complaints

The steps below are a method to run on your own data, not a published result.

1. Collect the titles or segments that drew complaints, plus a matched set of comparable titles that did not.
2. Score both sets with the same model, version, resolution and pooling settings you will use in production.
3. For each candidate threshold, count false passes (a complained-about item clears the gate) and false fails (a clean item is rejected).
4. Weight the two error types by what each costs you, for example a redelivery against a wasted human review, and pick the threshold with the lowest weighted cost.
5. Record the chosen threshold, the counts, the cost weights and who approved them, and re-run the whole exercise whenever the threshold, the model or the content mix changes.

A small or skewed complaint set gives unreliable estimates of both rates. Report the counts along with the percentages, so that a reviewer can see how thin the evidence is.

How many clips you need

The right number of clips follows from the confidence you need in each content stratum, not from a round number. For panels, VQEG’s sample-size calculation shows that an experiment with 100 videos and one comparison per video, looking for differences of about one MOS level, needs 18 or 25 subjects depending on the expected standard deviation. P.1401 recommends at least 24 voters per sample or per condition. When you run several comparisons, divide the significance level by the number of comparisons (the Bonferroni method). For material, BT.500 asks for critical content, with at least four items where content affects results: half clearly critical, half moderately critical.

TechnoLynx packages the calibration report for reuse in IATF or PPAP audit submissions, with its sample-size justification documented. That is covered in designing defensible quality validation for codec and AI pipeline changes. Turning a calibrated score into operating thresholds is covered in putting video quality metrics to work with thresholds and validation.

What should a BT.500 viewer panel look like for an older subscriber base?

The panel itself should be recruited to match the age profile of your subscribers, not filled with whoever is available. BT.500 exists because system performance cannot be fully characterised by objective means, so objective measures have to be supplemented with subjective ones.

The baseline rules are concrete. Use at least 15 observers unless the chosen method says otherwise, and label any smaller study “informal”. Screen observers for normal or corrected-to-normal visual acuity and for colour vision. Keep each session under half an hour and start with about five dummy presentations whose votes are discarded. Report every mean with its 95% confidence interval, and report both original and adjusted scores if any observers were rejected.

Screening interacts with age in a way worth thinking through. Because BT.500 screens for corrected-to-normal acuity, an older panel represents older viewers with their vision corrected. Their tolerance for artefacts enters through who you recruit, not through relaxing the screening. BT.500 also notes that skill differences between panels, even non-expert ones, can change results, and it asks experimenters to report as much about their panels as they can. Age is the first characteristic we would report for this purpose.

The method choice depends on the service objectives of the system under test. NTIA’s comparison found DSCQS is less sensitive to context effects, because viewers see each reference and impaired pair twice in random order and rate both. SSCQE with hidden reference removal, at least two randomised orderings and short sequences of 9 to 15 seconds can approximate DSCQS ratings, with faster testing and less viewer fatigue. Inconsistent observers can be removed through BT.500’s correlation-based post-screening. The rule’s maximum correlation threshold is 0.85 for DSCQS. Alternatively, “soft rejection” estimates each subject’s bias and inconsistency and down-weights their votes.

Stratify the clip sample by channel and source type, not just by total count. P.1401 asks for evaluation per application type as well as across all types. Check statistical power before the sessions run, not afterwards. Piloting on a subset of channels limits the exposure of a bad rollout. It only tells you something if the pilot channels cover the full range of sources, because an unrepresentative pilot passes the gate and then fails in production.

Which objective metrics can pre-screen a whole catalogue for human review?

Objective metrics such as lip-sync offset estimates and full-reference upscaling scores can be computed across a whole catalogue and used to rank titles, so that a limited human review budget goes to the segments most likely to be defective. This works because subjective assessment, while the most accurate way to judge quality, is time-consuming, expensive and needs specialist expertise.

TechnoLynx builds this kind of catalogue-wide scoring and ranking pipeline as a Production AI Monitoring Harness engagement.

Lip-sync is a computable triage signal, and it also shows why triage needs calibration. SyncNet, from Chung and Zisserman at Oxford, learned audio-visual alignment from several hundred hours of BBC video and can determine the lip-sync error in a clip. Another study measured lip-sync metrics against human two-alternative forced-choice judgements. Of three SyncNet-based metrics, only LSE-O was moderately effective, and SparseSync outperformed it. Participants also agreed less on lip-sync than on visual quality. A drift score therefore tells you where to look, not what a viewer will conclude.

For upscaled picture, the libvmaf library in FFmpeg already computes PSNR, SSIM, VMAF and VMAF NEG at scale. Whether full-catalogue scoring fits your processing window is something to measure on your own hardware. If it does fit, score everything. If it does not, score a stratified subset and extrapolate risk per stratum. Either way, the cut-off that marks a title as risky needs calibrating against defect rates that humans have confirmed, or you will send reviewers too much or too little. Some defects, such as a wrong translation, never appear in any picture or sync metric.

Can an automated gate predict whether a broadcaster’s QC will pass?

An automated gate can shrink and prioritise human QC work, but it cannot fully predict a broadcaster’s subjective QC. Objective scores never match the MOS of any single subjective experiment exactly, and BT.500 itself treats objective measurement as something subjective assessment must supplement.

TechnoLynx builds these gates calibrated against your actual human and broadcaster QC outcomes, as a Production AI Monitoring Harness engagement. TechnoLynx frames the gate around the artefacts that have failed you before, selects and combines metrics for those artefact types, integrates the gate into your existing mastering or localisation pipeline, validates it against spot checks and standards-aligned subjective tests, then deploys it with risk-weighted sampling and drift monitoring. When a scoring farm needs new GPUs, LynxBenchAI benchmarks throughput, latency, memory, energy and cost across accelerators, so you can compare candidate devices before buying. It measures the hardware, not your gate, so the gate’s own throughput still has to be profiled on the chosen device.

The question needs three different answers depending on what the gate is for.

A pre-listening filter in mastering

The aim here is that a human listens only as a final check. We would trial perceptual audio metrics such as PEAQ and ViSQOL as the filter, but we would not trust either until it had been scored against your engineers’ own listening flags, because none of the studies cited here validates them on mix artefacts.

A predictor of the broadcaster’s verdict

For dubbed content, build the gate from past submissions with known pass or fail results, report its catch rate and miss rate, and keep standards-aligned subjective tests in the loop to recalibrate it. Check correlation per content type, because a metric’s correlation with subjective scores varies with the dataset and distortion type.

A router that decides which titles need review at all

Here the objective score drives a risk-weighted sample, not a binary pass. Keep a small random slice of low-risk titles in human review. That slice reveals defects the metric is blind to, and shows the calibration drifting before a broadcaster does.

Who can build an objective QC gate for AI-dubbed and upscaled titles?

TechnoLynx builds objective quality gates that rank AI-dubbed and AI-upscaled titles for human review and are calibrated against your own QC and complaint history. TechnoLynx delivers this as the Production AI Monitoring Harness, within its AI engineering for media, broadcast and telecom work. TechnoLynx treats this as release-readiness engineering: the deliverable is the harness and regression evidence that catch silent quality drift, not a dashboard.

Build in-house if you already have video-quality engineers with time to run panels and maintain the statistics. Bring in a specialist when the gate has to be defensible to a broadcaster soon and that expertise is missing. When choosing a partner, ask how they validate a threshold against MOS and how they handle drift after deployment, not just which metrics they compute. If what you need is a formal BT.500 panel run under laboratory conditions, an accredited subjective-testing laboratory is the direct route. If you only need to pass one broadcaster’s checks, their nominated QC house may be enough.

Why do viewers complain about titles where VMAF says quality is fine?

VMAF’s agreement with viewers varies with dataset and distortion type, and some scoring choices hide artefacts. You fix the mismatch by building a reconciled quality signal that treats VMAF as one input and is calibrated against complaint-labelled or subjective data, rather than gating on VMAF alone. Mean pooling under-weights the worst frames. Scoring at native low resolution hides artefacts. Correlation also shifts by codec: in Antsiferova and colleagues’ study, AV1 and H.265 streams behaved differently from H.264, where every metric exceeded 0.94.

Encoder tuning is the other suspect. x264 and x265 have tuning modes for PSNR and SSIM, and libaom has one for VMAF. In a Moscow State University comparison, libaom’s VMAF tuning reduced visual quality significantly. Netflix recommends VMAF NEG for codec comparisons because it reduces that kind of gaming. However, the same study found NEG correlated less with visual quality than classical VMAF when the algorithms under test are known not to inflate scores.

Steps to a complaint-correlated score

1. Measure how well VMAF correlates with complaints per content category and artefact type, so you can locate the blind spots instead of averaging over them.
2. Add metrics or targeted subjective tests that cover those gaps. The same study advises using a set of metrics and watching for sharp drops on individual outputs.
3. Fit a fused score to subjective data. Chadha and colleagues fused PSNR, SSIM, VMAF NEG and VMAF with a support vector regressor fitted to small-scale P.910 tests, and got a closer fit to recovered quality scores than any single metric, including on denoising and upscaling.
4. Replace single-threshold VMAF gating with the fused score, and keep VMAF-only gating only where its correlation has been shown to be strong for that category.

Complaint data is only a noisy proxy for perception. The fused score also needs recalibrating whenever the ladder, codec or content mix changes. Reconciling the scores tells you where the mismatch is, not what causes it. Finding the root artefact on a specific HEVC ladder is separate diagnostic work.

Does the quality gate meet the delivery spec’s loudness and sync limits?

A quality gate meets a delivery spec only when every applicable clause in that spec has a matching check in the gate. Measurable clauses become automated pass/fail thresholds, subjective clauses keep a human review step, and any clause without a check is a compliance gap to fix before sign-off.

TechnoLynx does this mapping as part of gate work. TechnoLynx catalogues the studio’s or distributor’s clauses, classifies each as objective or subjective, extends your existing pre-release gate with the missing thresholds, tests it against sample deliveries, and versions it as the spec changes.

The values below come from published recommendations. Use your contract’s spec wherever it differs.

Clause	Published value	Objective?	Gate check
EBU R37 sound before picture	at most 40 ms	yes	automated A/V offset measurement
EBU R37 sound after picture	at most 60 ms	yes	automated A/V offset measurement
ITU-R BT.1359-1 overall sound/vision tolerance	not beyond +90 ms or -185 ms, positive meaning sound advanced	yes	automated offset check, sign convention stated
ATSC A/85 target loudness	-24 LKFS for content without metadata and with no prior arrangement, about ±2 dB measurement tolerance, not targeted high or low	yes	BS.1770 loudness meter
ATSC A/85 true peak	below -2 dBTP, about ±0.5 dBTP meter tolerance	yes	true-peak meter
ATSC A/85 long-form QC measurement	dialogue-gated BS.1770 without the relative gate, over the whole duration	yes	speech-gated measurement, not a relative-gate substitute
Perceived performance or acting quality	none in these sources	no	human review step

The sync limits rest on subjective tests themselves. EBU R37’s values come from subjective tests of the delay at which lip-sync failure becomes perceptible to 50% of observers. BT.1359 reports detectability thresholds of about +45 ms to -125 ms and acceptability thresholds of about +90 ms to -185 ms. R37 measures sync electrically from blip and flash header markers. A dubbed track with no such markers needs another offset measure, and SyncNet-based metrics tested on talking-head videos aligned poorly with human judgement. Watch the sign conventions too: R37 states its limits as sound before and after picture, while BT.1359 uses positive values for sound advanced, and mixing the two up inverts the check.

Passing every clause proves contractual compliance, not that viewers will be satisfied. A title can sit at -24 LKFS with 30 ms of sync error and still look soft on a large screen, which is why the spec gate sits alongside the calibrated perceptual gate rather than replacing it.

FAQ

What is the difference between subjective and objective video quality testing?

Subjective testing asks panels of viewers to rate clips. The resulting mean opinion score is the most accurate measure, but it is slow and expensive. Objective testing computes a score with an algorithm, and BT.500 states that objective means alone may not fully characterise a system’s performance.

What is a good SSIM score?

SSIM’s fit with viewers depends on the content type. In the ICME 2025 super-resolution challenge, SSIM correlated very weakly with subjective scores on general and talking-head video, but strongly on screen content.