Video quality metrics such as PSNR, SSIM and VMAF only become decision tools once three things surround them. The first is a metric that suits the stage and the codec. The second is a threshold calibrated against human ratings, with its uncertainty stated. The third is a sampling plan large enough to trust on every release. PSNR is fast enough for early sweeps across thousands of encoder configurations, and SSIM and VMAF are usually tracked at intermediate stages because they take longer to compute. In a 2021 comparison, VMAF reflected visible artefacts at high bitrates much better than the other metrics. None of them comes with a number that predicts complaints. That number has to come from your own viewers and your own content.

Choosing between video quality metrics by stage, codec and display

How closely the metrics track viewers varies by codec and bitrate. A 2021 study scored 789 streams from 39 codecs, rated by several hundred viewers. On H.264/AVC streams every metric correlated above 0.94 with the visual scores. On AV1, PSNR’s correlation dropped sharply, and the authors concluded that no PSNR variant is justified for analysing AV1. Their suggested cause is neural networks in newer encoders restoring object boundaries, which breaks the pixel-by-pixel similarity PSNR penalises. At high bitrates, VMAF’s correlation was 0.7, against 0.25 to 0.45 for the other metrics.

VMAF has limits of its own. Netflix built it because PSNR and SSIM could run at scale but fell short of capturing human perception. The authors of the MSU benchmark point out that VMAF was trained only on H.264/AVC-compressed video, and that super-resolution in LCEVC and in neural encoders produces distortions that traditional metrics cannot handle. Video preprocessed to suit the metric can push VMAF up while visual quality falls, to the point where the correlation turns negative. That is why Netflix recommends VMAF NEG for comparing codecs and classic VMAF for comparing versions of the same algorithm.

Situation	Metric to lead with	Reason
Early sweeps across thousands of encoder configurations	PSNR	Cheapest to compute at volume
Intermediate checks on shortlisted settings	MS-SSIM and VMAF	Closer to perception; MS-SSIM beat SSIM in the 2021 comparison
High-bitrate ladder rungs	VMAF	Correlation 0.7 against 0.25 to 0.45 for other metrics
AV1 and other new-generation encoders	VMAF, MS-SSIM; not PSNR	PSNR correlation collapses on AV1
Codec comparisons across vendors	VMAF NEG	Resists preprocessing aimed at the metric
Neural codecs, super-resolution, upscalers	VMAF plus a learned perceptual metric, validated on your own ratings	Traditional metrics cannot handle these distortions, and VMAF was trained only on H.264/AVC
Near-visually-lossless masters	CVVDP or IW-SSIM, checked against JND tests	PSNR, SSIM, LPIPS and DISTS saturate in the high-quality range (image evidence)

A metric’s configuration also affects how compression methods rank. PSNR and SSIM can be computed on luma alone or as a weighted sum of Y, U and V, and that choice changes how codecs rank. The same study found 8:1:1 best for VMAF. We would fix the channel weighting, the VMAF model file and the pooling method in the test specification before anyone compares numbers. Two teams quoting “VMAF” from different setups are not quoting the same measurement. Metrics with similar average correlation can still collapse on individual clips, so the study’s authors recommend using a set of metrics and watching for sharp swings on individual outputs.

Can a perceptual metric track complaints better than PSNR or SSIM?

It can, provided you validate it on your own artefacts. Learned metrics such as LPIPS and DISTS capture texture misalignment, global contrast shifts and semantic inconsistencies that conventional metrics find hard to predict. They also need large rated datasets, cost more to compute and may fail outside the distortions they were trained on. On a high-fidelity JPEG AIC-3 image dataset, LPIPS, DISTS, SSIM and MS-SSIM handled contrast masking well, and VMAF did worse. That result comes from still images, so we read it as a reason to test a learned metric alongside VMAF on your upscaler’s output, not as a ranking to adopt.

The score assumes a screen and a viewing distance

Each VMAF model is trained for particular viewing conditions. The default v0.6.1 model assumes a 1080p display viewed from three times the screen height. On its scale “bad” maps to roughly 20 and “excellent” to 100, so a score of 70 reads as a vote between fair and good under those conditions. Netflix trained a phone model because viewers on small screens see less difference between 720p and 1080p, and a 4K model for a 4K TV viewed from 1.5 screen heights. Jan Ozer scored four renditions of two test clips, from 1080p down to 480p, with the phone model. All of them scored above 93 and sat within 6 VMAF points of each other, a gap most viewers would not notice. To score a 480p encode of a 1080p source, upsample the encode to 1080p. Downsampling the source instead hides the scaling artefacts.

What VMAF or SSIM score predicts that viewers will complain?

The published RealNetworks and JND figures describe visible or noticeable distortion, not complaints. A complaint threshold is therefore a judgement you calibrate on your own content, not a constant you can look up. RealNetworks validated VMAF against subjective scores at a correlation of 0.948. They concluded that encoding to a VMAF of about 93 serves most of the audience with content that is “either indistinguishable from original or with noticeable but not annoying distortion”. Jan Ozer notes that a difference of 6 VMAF points equals one just noticeable difference (JND).

The gap between “looks different” and “looks wrong” is where those numbers stop helping. A JND is the level of distortion a random observer detects half the time, and it shifts with display size, viewing distance, lighting and test method. The JND point also varies between people. The study models the JND location as a normally distributed variable shaped by differences between viewers and between content, and viewers split into easy-to-satisfy, normal and hard-to-satisfy groups. MOS-based methods cannot predict the share of viewers who are satisfied. A satisfied-user ratio of 0.75 is roughly what a MOS-style mean corresponds to.

A threshold that keeps the average viewer at “not annoying” can still generate tickets. Picture quality is also not the only driver of engagement. In a SIGCOMM 2011 study of over 2 million views, buffering ratio had the largest effect on engagement for every content type, and average bitrate mattered significantly more for live content than for video on demand.

Calibrating the threshold against MOS and past complaint cases

We would run the calibration in this order:

1. Write down the acceptability criteria before any assessment starts, as ITU-R BT.500 advises.
2. Build the clip pool around critical material. BT.500 notes that results can take critical material into account but cannot be extrapolated from non-critical material.
3. Run a subjective test to BT.500. Use at least 15 observers screened for visual acuity and colour vision, keep sessions under half an hour, and report the mean and 95% confidence interval for each test parameter. ITU-T P.913 recommends 24 subjects and ITU-T P.1401 recommends at least 24 voters per sample or per condition.
4. Fit the metric-to-MOS mapping, and check results after a first-order mapping as well as a third-order one. P.1401 warns that a third-order mapping can mask systematic misprediction.
5. Score the metric for accuracy (RMSE), consistency (outlier ratio) and linearity (Pearson correlation), as P.1401 recommends.
6. For each candidate threshold, we would build a confusion matrix of the metric’s verdicts against the panel’s verdicts. NTIA’s method classifies every pair as better, equivalent or worse. A false pass is a clip the metric passes and the panel or a past complaint fails. A false fail is the reverse.
7. Set the threshold’s margin from VMAF’s own uncertainty. Since v1.3.7 (June 2018), VMAF can report a bootstrapped 95% confidence interval, and Netflix recommends plain bootstrapping because it is unbiased against the full model.

The VMAF interval is the score plus or minus 1.96 bootstrap standard deviations under a normality assumption, or the 2.5th to 97.5th percentile range without one. Intervals are tighter at the high end of the scale, because the training data is denser there.

The panel’s size sets the smallest MOS difference it can resolve. NTIA’s 2021 analysis drew on 60 subjective tests in which 2,331 subjects rated 17,665 media files. It found that a 24-person test treats videos as equivalent if their ratings differ by less than 13% of the scale, and that 200 subjects would cut this to 5%. On the 5-level ACR scale, the NTIA report gives these lower limits for well-run tests:

Subjects	MOS difference at which 95% of pairs separate (ΔSCI)
24	0.5
15	0.7
9	1.1
6	1.5

The report adds that reaching 0.5 in practice takes 24 to about 34 subjects. P.1401 puts typical confidence intervals around 0.15 MOS, so no finer ranking than 0.3 MOS can be claimed with confidence. Small ad hoc assessments do much worse. NTIA measured an 11.4% error rate for a one-person ad hoc assessment, 8.5% with two people and 6.8% with three. In contrast, two labs running the same formal test disagree on a ranking less than 1% of the time, typically about 0.17%.

A threshold calibrated on past complaints only covers the artefacts those complaints described. A new codec, a new upscaler or a new device class can introduce failure modes the calibration set never contained. P.1401 makes the same point about new codecs and networks: they bring new kinds of degradation.

How many clips per release, and which ones?

Size each content category from the variance of its own metric scores and the interval width you need; the stratified-sampling section gives the formula. Which clips you pick matters too. Where content affects the results, BT.500 asks for material that is “critical but not unduly so” and for at least four items, half definitely critical and half moderately critical. When choosing material from a pool, it recommends sampling sparsely at less critical levels and densely at more critical ones.

One number that QA, engineering and sales all quote the same way

A single shared number works when it is a full specification rather than just a score. It names the metric and model file (for example the VMAF phone model), the channel weighting, the pooling method, the content category it applies to and its confidence interval. Netflix found that a simple arithmetic mean of per-frame VMAF correlates best with subjective scores, while a harmonic mean gives more weight to low outliers and suits optimisation that must avoid occasional bad encodes. For A/B tests, Netflix suggests tracking three aspects of quality: aggregate quality, start-of-play quality, and how often VMAF drops below a threshold relative to the preceding values.

When is an approval-rate drop a real regression, and who signs the cutover?

A drop in the share of clips that pass is a regression only when it falls outside an interval agreed before the test ran. This matters because, when two systems have similar quality, NTIA found that half of all better-or-worse conclusions are false distinctions that a subjective test would call equivalent. Approval rates are ratios, so their standard errors are usually estimated with the delta method or a bootstrap rather than read off raw counts. Comparing several candidates at once needs a correction. Under the Bonferroni method in P.1401, 10 comparisons at an overall α of 0.05 require 0.005 per comparison. P.1401 also recommends setting minimum performance thresholds in advance, based on previous experience where it exists.

For the go/no-go decision, a workable split is for QA to own the measurement, engineering to own the explanation of every failing stratum, and the commercial owner to sign for the residual risk. The evidence pack should report each content category separately, because BT.500 notes that scores depend on how critical the material is and that aggregated averages hide this. It should also include the worst case, since P.1401 recommends careful analysis of the worst-performing case. Our piece on designing defensible quality validation for codec and AI pipeline changes covers the full release evidence pack. Where automated quality gates stop and subjective panels have to take over is covered in a separate piece.

How do we prove a tamper detector is not flagging ordinary re-encodes?

We would measure a tamper detector’s false-positive rate on footage known to be re-encoded but not tampered with. That footage should come from the same DVR model and the same export path. We would run the detector over the set with fixed settings and count the flags. The resulting rate is valid for that profile and no other. A controlled test is needed because a benign transcode adds the extra compression step a double-encoding detector looks for. In Amped’s own example, one surveillance video is exported twice: stream-copied into AVI, and transcoded into MP4, which “has undergone one more compression step”. The forensic premise is that pristine video was compressed once at capture, so traces of further compression raise doubts about integrity. On a transcoded export, a double-encoding finding is therefore expected rather than incriminating.

A validation run for one DVR profile looks like this:

1. Exemplars. Record known-good footage on the same DVR model and firmware and export it through the route the evidence took.
2. Detector run. Process every exemplar with a fixed threshold and record the flags. In Amped Authenticate’s VPF double-encoding tool, the default threshold of 0.6 gave about 2% false positives on Amped’s validation dataset. In Amped’s wider testing on tens of thousands of videos without B-frames, a threshold set for about 3% false positives gave a 75% true-positive rate on average. Those are vendor figures on vendor data, so your exemplar count replaces them for your profile.
3. Preconditions. The VPF effect is only present and detectable when the first compression used a fixed GOP size, the two GOP sizes differ, and the first is not a multiple of the second. If those conditions do not hold, a missing VPF signal does not show the footage was compressed only once.
4. Corroboration. Double-quantisation analysis compares the distribution of each macroblock’s DCT coefficients with the one double quantisation would produce. Wang and Farid (2009) show it can localise tampering to regions as small as 16×16 pixels, but only when the second compression is of higher quality than the first.
5. Quality comparison. Score the suspect export with VMAF and SSIM against a known-clean re-encode of the same source. We read a match as consistency with normal re-encoding, never as proof that no tampering took place.
6. Record. Log the profile, firmware, settings, exemplar count, flag count and a binomial confidence interval on the false-positive rate.

Amped’s figures come from its own validation data, not from any specific DVR model or 10-bit HEVC export. That is the argument for requiring several independent indicators to agree before treating a flag as a finding. The protocol supports an examiner’s conclusion. What weight that conclusion carries remains the examiner’s call.

Should video frame interpolation run once server-side or on every client?

Managed server-side frame-rate conversion, such as MediaConvert’s FrameFormer add-on, is billed as an additional transcoding cost. The choice comes down to a break-even between encode cost per title and engagement gain, and the gain is the side that has to be measured.

LynxBenchAI is the measuring tool for the hardware part of that calculation. Its published results show how candidate GPUs run the public PyTorch stack an interpolation model would use, which makes them a broad indication for a custom PyTorch model. They do not run or measure your own pipeline. TechnoLynx then sets up the per-title cost model, covering GPU type, batch size, resolution and bitrate tiers, plus the engagement metric for the benefit side. It builds the server-side stage on the target GPU to get realistic cost-per-title figures.

The cost side per title

In MediaConvert, the managed route, on-demand transcoding is charged per minute of video. Each output’s resolution, frame rate, quality setting and codec combine into a multiplier that converts output minutes into normalised minutes. InSync FrameFormer frame-rate conversion is an add-on with its own additional cost. Server-side encode cost per title therefore scales with output minutes and their multipliers, plus the FrameFormer add-on. Motion-compensated interpolation is widely regarded as giving the best results, because it places moving objects where they belong in the new frames. On self-managed GPUs, the FRUC interface in NVIDIA’s Optical Flow SDK validates flow vectors with a forward-backward consistency check and rejects the ones that fail.

The encode cost is not the only cost. A higher-frame-rate rendition also adds CDN delivery cost on every view, while the encode is paid once. AWS describes a related tactic for long-tail catalogues: move a title to a more expensive transcode only once it passes a viewership threshold, with 10,000 views as its example. Applied to interpolation, a view threshold keeps encode spend away from titles nobody watches.

Jan Ozer’s multiview analysis offers a worked example of this kind of break-even. Jan Ozer found that a server-side 4K composite matched the quality of four client-side 1080p streams at 14.0 Mbps against about 17.0 Mbps, roughly 17% less bandwidth. He noted that turning this into money needs both the extra encode cost and the delivery cost. A colleague disputed the bandwidth argument for clients that fetch each stream at its displayed resolution. The lesson carries over: the bandwidth assumption decides which side of the break-even you land on, so it needs measuring, not assuming.

The benefit side, and which bitrate tiers gain

Engagement gain by bitrate tier has to be measured in an A/B test on real viewers. The quality signals Netflix suggests for A/B tests apply directly here: aggregate quality, start-of-play quality and drops below a threshold. The SIGCOMM 2011 result also tempers expectations. Buffering, not bitrate, had the largest effect on engagement, and bitrate mattered more for live than for on-demand content.

The tier decision is a judgement. If a higher-frame-rate source exists, a native encode at that frame rate beats interpolation, especially on the high tiers where the bits are available. Interpolation earns its place where native frames do not exist, or where a low tier cannot afford them and smoothness is the visible gain. Interpolated output should go through the same metric checks as any encode. VMAF includes a temporal feature, the mean co-located pixel difference against the previous frame. Even so, we would add a panel check on the worst-case motion content, because interpolation artefacts are temporal.

Who can work out whether GPU interpolation pays for itself on our catalogue?

TechnoLynx builds and costs the interpolation stage for exactly this decision. TechnoLynx implements the server-side stage on your target GPU and tests it across your bitrate and resolution tiers, so you get cost per title measured on your hardware and a cost model you keep. Building this in-house makes sense if your team already optimises GPU inference and can design the viewer A/B test. Otherwise, a specialist gets you to a defensible cost per title sooner. When choosing a partner, look for cost per title measured on your hardware rather than quoted benchmark speeds, and a cost model you keep afterwards. The optimisation work runs as the Inference Cost-Cut Pack.

This engagement does not suit every catalogue. If most of your titles sit in a long tail with few views, consider the viewership-threshold approach AWS describes for MediaConvert transcodes. If your sources already carry the frame rate you need, a native re-encode makes interpolation unnecessary. If the open question is the viewer study itself rather than the cost, a subjective-testing lab is the realistic partner.

How many clips per content category, bitrate tier and GPU architecture?

In stratified sampling, the variance of the overall mean depends only on the variances within each stratum. The same text gives a large-population approximation for the total sample: n ≈ 4·Σ(W_h·s_h²)/d². Here W_h is the stratum’s weight, s_h² its observed variance and d the desired half-width of a 95% interval. For 99% intervals, 7.08 replaces 4, and for 90% intervals, 2.79 does. The confidence level costs width. In one worked example, moving from 95% to 99% widened the half-width from 1.1157 to 1.4662 at the same sample size.

The sampling unit needs care. Consecutive frames from one shot are strongly correlated, so counting them as independent samples makes an interval look tighter than it is. The clip or segment is the honest unit. Krebs makes the related point that stratum sizes must be expressed in the same sample units as the sample itself. A design that holds up runs in this order:

1. Define strata as content category × bitrate tier × GPU architecture, the last whenever the pipeline runs upscaling or mixed-precision inference on the GPU.
2. Estimate the within-stratum variance of each metric, which the sample-size formula needs.
3. Compute the sample size for each stratum from that variance and the target interval width.
4. Set a tolerance threshold for each category from that category’s baseline distribution.
5. Repeat the exercise when the content mix or the hardware fleet changes.

Pilot variances taken from the old ad hoc set may themselves be biased if that set was not representative, so the first full run should re-estimate them. Too many strata with small effects can push sample sizes out of reach. Merging rare categories is usually the better trade than abandoning the stratification. Whether a GPU-to-GPU deviation is acceptable is a numerical-parity question, with its own tolerance band, separate from the perceptual threshold.

Should quality thresholds differ by content category?

Yes, where the categories differ in how critical they are. BT.500 notes that scores depend on how critical the test material is, and that reporting each sequence separately gives a fuller picture than aggregated averages. P.1401 asks for minimum performance thresholds both per application and across applications. Engagement also responds differently by content type: in the SIGCOMM study, buffering hit live content hardest. One global cutoff will over-flag forgiving categories and under-flag critical ones. The cost is governance. Every per-category threshold needs its own justification and an agreed definition of the category, so we would keep the number of categories as small as the variance data allows.

Every threshold expires with the content it was calibrated on

A VMAF or SSIM threshold is a claim about specific viewers, on specific screens, watching specific artefacts. It stays valid only while those three things hold. A new codec, an upscaler update or a shift towards phone viewing each calls for a re-calibration on fresh panel data, not a re-reading of old scores. The tamper protocol follows the same logic: the error rate belongs to the profile it was measured on. That discipline carries over when frames feed a detector rather than a person, which is the work TechnoLynx’s computer vision R&D practice covers.

FAQ

What is a MOS in video quality testing?

A mean opinion score (MOS) is the average of the individual ratings a panel gives one sample or test condition. It carries statistical uncertainty, which is reported as a standard deviation or confidence interval.

Should SSIM be high or low?

High. SSIM scores range from zero to one in the original formulation, and values closer to one mean the output is structurally closer to the reference. It is often expressed on a decibel scale instead, as −10·log10(1 − SSIM).