GPU benchmark verification means treating every performance number, whether a vendor’s or your own team’s, as a claim to check against instrumented evidence before anyone acts on it. The core checks are these. Confirm from a kernel trace which execution path actually ran. Confirm the profiler reports correctly on that architecture. Control clocks, thermals and warm-up so the timing repeats. Replay the arrival pattern you actually see in production. Then compare outputs and scores only on a common basis. A configuration file says what should have happened, and a trace says what did.
A GPU benchmark verification checklist
A perfectly controlled timing of the wrong kernel path is still wrong.
- Execution path. Profile at kernel level and confirm that the kernels you expected (tensor-core kernels for a reduced-precision claim) actually executed, layer by layer.
- Profiler correctness. Check that the profiler’s counters are trustworthy on this GPU generation before relying on its metrics.
- Environment. Lock or record clocks, watch for thermal and power throttling, separate warm-up from measurement and use representative input values.
- Metric definition. State which latency (kernel, model, request or service) and which throughput (offline or online) you are reporting.
- Workload. Replay your production arrival pattern rather than a shipped synthetic harness.
- Output parity. Compare outputs against a reference with explicit tolerances, relaxing the FP32 defaults for reduced precision where needed.
- Common basis. Compare systems only when hardware, software versions, batch semantics and metric definitions are aligned, and score them with one shared script.
Did the run use tensor cores, or quietly fall back to FP32?
You find out from a kernel-level trace, not from the configuration: in Nsight Compute, a non-zero sm__inst_executed_pipe_tensor_op_hmma.sum metric shows that HMMA (tensor-core) instructions executed. TechnoLynx instruments the submitted inference pipeline end to end, compares the observed kernel and instruction mix against what the claimed precision should produce, and hands back a reproducible trace and utilisation summary that either documents tensor-core engagement or flags the fallback.
One aggregate number is not enough evidence. Frameworks can fall back for specific layers or shapes, so a run can be mostly on tensor cores and still lose a hot layer to CUDA cores; only a per-kernel breakdown shows that. DLProf’s Tensor Core Report lists every eligible operation and whether a tensor-core kernel served it, and its Bad Tensor Shape Detector flags operations that missed tensor cores because of shape or data type. DLProf has a blind spot, though. It infers tensor-core use from kernel names, which identifies cuDNN kernels accurately but misses custom kernels and kernels outside cuDNN. For a fused or hand-written kernel, the instruction-level counter is the stronger evidence.
At fleet level, DCGM reports tensor activity as the fraction of cycles the HMMA/IMMA pipe was active, averaged over an interval. Some metric groups cannot be collected together on some GPUs (SM activity and tensor utilisation on V100, for example), and DCGM works around that by multiplexing and sampling statistically. Two cautions follow. Metric names differ by architecture: NVIDIA’s Nsight Compute guide says tensor work can appear under Tensor or Tensor Core pipelines depending on the chip, and advises querying the available metrics before hard-coding thresholds. And engagement is not throughput: a 2025 profiling study of concurrent vision inference on NVIDIA Jetson found SMs and tensor cores often at 15% to 30% utilisation even when GPU utilisation reached 100% with specific optimisations, and observed that higher tensor-core utilisation did not always mean higher throughput.
Silent fallback also happens at the precision level. A 2026 survey of the inference deployment stack notes that on accelerators without native INT4 support the runtime silently upcasts to FP16 and the expected throughput gain vanishes. When the trace shows tensor-core kernels but the numbers still look wrong, the next step is diagnosis rather than verification, which is the subject of diagnosing GPU bottlenecks with kernel-level profiling.
Profiling also adds overhead to what it measures. Nsight Compute replays kernels when one pass cannot collect every requested metric, and software-patched metrics carry the highest overhead. The Jetson study handled this with two phases, a lightweight pass for throughput and power and a heavier pass for low-level counters.
Check the profiler before you trust the profiler
Profilers can themselves give misleading results. The DELTA validation work argues that profiling has to be correct at every level, from counter setting through value collection to metric calculation, and that an error in any phase can produce misleading results. Its method is to run microbenchmarks such as pointer-chasing (P-chase), whose memory behaviour can be predicted in advance, and compare the profiled result against that ground truth.
The case that makes the point: DELTA’s traffic metrics passed validation on AMD’s Vega 10 GPU and failed on Vega 20, under-reporting memory traffic. Vega 20 had added more MPs for a wider interface, but the traffic metrics were still based on Vega 10, so the extra MPs escaped profiling. The authors’ conclusion is that profiling results need validating on every new architecture.
Clocks, heat, warm-up and input values decide whether a timing repeats
NVIDIA’s TensorRT benchmarking guide states that two runs of the same engine on the same GPU can disagree by tens of percent if the clock floats in one run and is locked in the other, or if thermal throttling starts halfway through. These are the controls we would put in place, and why each one matters.
Clocks. By default GPU clocks float, idling when there is no work and boosting under load. Floating clocks or throttling make each kernel run at a slightly different frequency and destabilise measurements, while locked clocks give consistent numbers at a somewhat lower average. The profilers behave differently: Nsight Compute locks SM clocks by default and Nsight Systems does not, so NVIDIA recommends locking clocks externally with nvidia-smi and passing --clock-control none to ncu when comparing results. With sufficient permissions, nvidia-smi --lock-gpu-clocks=tdp,tdp fixes the GPU at its base TDP frequency.
Heat and power. Thermal throttling starts at a device-specific slowdown temperature, readable with nvidia-smi -q -d TEMPERATURE, and shows up as the temperature climbing towards that threshold while the clock drops. On GPUs prone to power throttling, such as the T4, poor cooling can lower the stabilised clock through power throttling before thermal throttling begins, because heat raises leakage current. NVIDIA recommends capturing nvidia-smi -q before the run and logging clocks, power, temperature and utilisation with nvidia-smi dmon alongside it. Enabling persistence mode removes GPU initialisation costs from the measurement.
Warm-up. Warm-up fills caches, maps memory pages and lets clocks settle. NVIDIA’s CUTLASS guidelines observed GPC clocks oscillating for around the first 3 seconds of back-to-back GEMMs. By default, trtexec warms up for at least 200 ms and then runs at least 10 iterations or 3 seconds, whichever is longer. The 2026 DEEP-GAP study shows one way to separate warm-up from measurement: 20 warm-up iterations, then 100 timed iterations, timed with CUDA events and explicit synchronisation. MLPerf Inference also requires every run to last at least 60 seconds, so that power management and DVFS reach equilibrium.
Inputs. All-zero or NaN inputs draw less power than representative values because they flip fewer bits in DRAM and L2, which changes the timing. Models with data-dependent shapes can perform very differently on random inputs than on real ones. trtexec reports total GPU compute time beside host wall time, and NVIDIA links a large gap between them to host-side overhead or data transfers.
Which latency, which throughput: the common-basis rule
Two numbers can be compared only when they measure the same thing under the same conditions. The 2026 deployment-stack survey identifies at least four quantities reported as “latency”: kernel time, model latency, request latency (including batching wait and pre- and post-processing) and service latency (adding the network round trip). It notes that they can differ by an order of magnitude for the same model on the same hardware. Throughput splits the same way: offline throughput under saturated input and online throughput under a realistic arrival process within a latency SLA are fundamentally different quantities.
The spread is large. The same survey cites ResNet-152 on a Jetson AGX Orin running at 2.28 ms under TensorRT and 285.5 ms under ONNX Runtime, a 125× spread, with top-1 accuracy also varying by 4.6 percentage points across runtimes. Its own policy is to compare numbers literally only within one study, and to use cross-study numbers qualitatively unless workloads, hardware, software versions, batch semantics and metric definitions are explicitly aligned. Tools add their own differences. NVIDIA’s LLM benchmarking guide shows that GenAI-Perf excludes TTFT from inter-token latency while LLMPerf includes it, and that LLMPerf’s tokens-per-second can absorb harness overheads that sometimes reach 33% of the run at single concurrency.
MLPerf Inference was designed in part to address these ambiguities. It defines four scenarios (single-stream, multistream, server, offline) whose results cannot be compared across scenarios. It also requires near-FP32 quality within 1% and ships audit tests, including caching detection that compares runs with unique and duplicate sample indices. The deeper treatment of putting heterogeneous results on one footing is in making GPU benchmarks comparable with a normalisation framework.
Is TensorRT-LLM faster than vLLM on our models and traffic?
The honest answer is that it depends on your arrival pattern, concurrency, batch size and model, so the ranking has to be measured on your own nodes with your own traffic. TechnoLynx frames that comparison around your model family, shapes and traffic profile, runs vLLM, TensorRT-LLM and any bespoke kernel candidate on your target A100 or H100 nodes, and reports p99 latency alongside throughput; the Inference Cost-Cut Pack is the engagement that carries the result through to a cheaper production stack.
The published evidence shows why. PrefixBench-H100 (2026), which drove both runtimes with identical prompts and schedules on one H100 NVL, found vLLM’s first-token latency about 8 ms below TensorRT-LLM’s at concurrency 1, a gap it attributed to per-request overhead rather than caching. At concurrency 32 the leadership flipped: on a second model, vLLM’s p50 TTFT was 553.8 ms against TensorRT-LLM’s 265.0 ms. In its realistic RAG-style traces on one H100 NVL, under inter-arrival jitter, the study found TensorRT-LLM’s tail latency dominated the user-visible cost. Its authors tell operators to benchmark with the arrival pattern they actually see in production. A 2024 benchmarking study across accelerators found TensorRT-LLM gave the highest performance and lowest power on NVIDIA platforms and vLLM ran on a wider range of devices. The vLLM paper (2023) reported 2-4× higher throughput than FasterTransformer and Orca at the same latency. These are cross-study numbers, so they read qualitatively unless their workloads, hardware and software align with yours.
The rankings also expire. The 2026 deployment-stack survey says a TensorRT, CUDA or vLLM release can shift framework rankings by 10-30%. It also warns that the ranking can reverse between batch size 1 and batch size 32.
For a bespoke kernel path, the answer is to measure it against TensorRT-LLM on your model sizes, not assume it. Our decision rule: if the current stack already meets its SLAs, change nothing. Build custom kernels only when the measured gap is large enough to pay for maintaining them through every model update, since those gains are tied to specific model shapes and hardware. A tuned kernel configuration is not portable either: one tuned for an A100 can perform poorly on an H100. For shortlisting the hardware itself, LynxBenchAI publishes results showing how each device runs the public PyTorch stack, which is broadly indicative for custom PyTorch models but is not a run of your workload.
How do we check an INT8 video model gives the same output as FP32, frame by frame?
Align the two models’ outputs by frame or token index, never by wall-clock arrival, then score each aligned pair against an explicit tolerance over a sample that covers your real content mix.
LynxBenchAI publishes device results on the public PyTorch stack that are broadly indicative for shortlisting GPUs; start by checking its results for your candidate devices before building the parity harness.
Keep the two questions separate: one report on output deviation, one on latency. For scoring, NVIDIA’s Polygraphy compares outputs from two runtimes with absolute and relative tolerances. It warns that its defaults suit FP32 but may be too tight for reduced precision, and it can score by elementwise, mean, median or maximum error. Its comparison functions also offer L2 and cosine similarity, PSNR and SNR, and the perceptual LPIPS metric for image-like outputs. Pick the metric that matches what the product actually promises. For captions, that is the alignment tolerance your service already commits to, re-derived for each task rather than copied from another one.
When parity fails, find the layer. Polygraphy can compare per-layer outputs to locate where the error enters, and NVIDIA’s TensorRT accuracy guide recommends comparing layer outputs with golden values and controlling the precision of layers that misbehave in INT8. The 2020 study Integer Quantization for Deep Learning Inference found that a few layers often cause most of the accuracy loss, and proposed a one-at-a-time sensitivity analysis to decide which to leave in floating point. Using that workflow, all the networks it studied stayed within 1% of FP32 accuracy. Calibration data matters as well: NVIDIA’s TensorRT guide cautions that an engine calibrated on random data is a performance experiment, not a deployable result, and must be recalibrated on representative data and revalidated.
How much sample is enough? The sample has to span the content dimensions where INT8 can degrade, such as speech rate and scene complexity for captioning, because a small or uniform sample misses exactly those edge cases. The vbench benchmark offers a useful pattern: it chose its test videos by weighted k-means clustering over resolution, frame rate and entropy, drawn from YouTube’s production transcoding logs. Stratifying a parity set the same way, by the content dimensions your traffic actually has, is how we would size it. The harness can be reused because it is parameterised on model outputs rather than on one model version; a new quantisation scheme becomes another run.
van Baalen et al. (2023) argue that FP8 compute is at least 50% less efficient than INT8 in area and energy on dedicated hardware, and that INT8 matches or beats FP8 accuracy for most well-behaved networks. GPU support decides what you can actually use, though: a 2024 benchmarking study found FP8 gave a benefit on H100 while A100, lacking FP8 support, relied on INT8. Either way, the parity harness is what settles the question for your model.
A benchmark that reflects real video traffic: transcoding and captioning together
A useful benchmark for a shared media GPU replays sampled production traffic, with real codec mixes and real-time captioning inference running at the same time, and measures captioning latency while transcoding load rises. TechnoLynx characterises your codec and caption-model mix, defines the starvation and quality metrics, builds a reproducible combined harness across your candidate GPU or ASIC targets, validates it against production behaviour and hands over the configuration so your team can rerun it.
LynxBenchAI publishes device results that are broadly indicative for shortlisting candidate GPU targets; they do not measure your transcoding or captioning workload, so start by reviewing them for your shortlist.
NVIDIA’s Nsight Compute guide notes that other engines on the chip, including the video encoder and decoder, share L2, DRAM and PCIe with the kernels being measured and can confuse memory results. Representative content matters as much as concurrency: the vbench authors (ASPLOS 2018) showed that the choice of input videos changes microarchitectural trends and qualitative conclusions, and that a high-entropy-only corpus would overestimate the benefit of GPU transcoding. They also advise against averaging results across diverse videos, because averaging throws away significant information.
On the encode side, NVIDIA’s July 2023 benchmark notes recommend loading NVENC with at least 8 concurrent sessions, comparing matched presets, and scoring the bitrate the encoder actually produced rather than the target bitrate. On the inference side, Triton’s Perf Analyzer has a fixed-schedule mode that sends each request at a recorded timestamp, which is how a captured traffic trace becomes a replayable load.
Is captioning starved when transcoding spikes?
Measure captioning p99 latency and dropped or late caption segments while stepping transcoding load from typical to peak on each architecture; a captioning stream that holds its latency at peak load is not starved, and one whose tail grows with session count is. The Jetson concurrency study gives a sense of how non-linear this can be: execution-context duration grew by about 30× with 4 concurrent processes and about 70× with 8, compared with 1-2 processes. Controlled single-model studies do not answer the question; the DEEP-GAP authors themselves describe their results as single-tenant and leave shared-GPU behaviour to future work.
Will the optimisation hold latency under worst-case burst before a live broadcast?
Only a test at the provisioned peak, with burst arrival, can tell you. A trace-driven study of Akamai’s CDN transcoding workflow notes that the transcoding cloud is provisioned for peak workload. Burst behaviour differs from steady load: in PrefixBench-H100, vLLM’s mean queueing delay rose from 0.0 ms at concurrency 1 to 154 ms at concurrency 32 under burst arrival, yet stayed below 0.1 ms under fixed-rate arrival up to 16 QPS. Those authors also note that their fixed-rate arrivals are smoother than Poisson production traffic, which they expect to push tail latency towards the burst results. Before a live event, we would replay the worst recorded burst on the actual broadcast configuration with clocks and thermals logged.
Can we rerun the methodology in six months?
You can, if what you keep is the methodology, not just the numbers: documented traffic sampling, the parameterised harness and recorded environment metadata. Results will still shift with model, hardware and workload changes. The 2026 deployment-stack survey recommends re-profiling after any model change or hardware refresh, and warns that shifts in arrival rate or sequence-length distribution can make a previously optimal configuration suboptimal. Measurements are sensitive to driver and CUDA toolkit versions, so record them on every run.
Who can benchmark our serving stack and video pipeline on our own traffic?
TechnoLynx does this work for teams that need to know which serving stack, precision or GPU meets their latency targets on their own models and traffic. For a serving decision, TechnoLynx starts by framing the comparison around your model family and arrival pattern. TechnoLynx then runs the candidate stacks on your target nodes and supports deployment of the one that wins, through the Inference Cost-Cut Pack. For media and broadcast pipelines, TechnoLynx builds the combined transcoding and captioning harness and hands it over for reruns. For shortlisting hardware, LynxBenchAI publishes device results on the public PyTorch stack, which are broadly indicative for custom PyTorch models but are not a run of your application.
This engagement is not needed if your current stack already meets its SLAs with headroom at peak. A team that mainly needs independent credibility for a procurement sign-off may want a third-party report as well, and a reusable harness it owns is the better asset for decisions it will have to repeat.
Replacing eyeball review of rendered frames with an automated golden-frame diff
An automated diff step in the render pipeline computes per-frame error against golden references and surfaces only frames that exceed tolerance, with thresholds calibrated so that its flags agree with what supervisors would mark by eye. TechnoLynx builds this as part of its Computer Vision R&D work: choosing metrics suited to the content, including temporal or flicker measures for motion-heavy shots, integrating the check into the render-farm pipeline on your GPUs, profiling it so it does not slow the farm, and calibrating thresholds against supervisor review before deploying it as a monitored stage.
OpenImageIO’s idiff reports mean and maximum error and counts the pixels above warning and failure thresholds. Its thresholds can be set so that a comparison passes if fewer than a chosen percentage of pixels exceed a given difference, warns within a looser band, and fails beyond it. That three-way outcome is exactly what a pipeline stage needs: pass silently, queue for a human, or block. We would treat the first weeks as a calibration period, running the automated flags alongside supervisor review and adjusting thresholds until the disagreements stop being meaningful. PSNR is the familiar objective choice, but the vbench authors note there is no consensus on which perceptual metric works best, which is one more reason to calibrate against people rather than pick a metric and trust it. Profile the check like any other workload on the farm, so that it does not slow rendering.
Putting two forecast models on one grid before scoring them
Scores from an NWP ensemble on a Gaussian grid and an AI emulator on a regular latitude-longitude grid are not comparable until both outputs sit on one common grid and one lead-time convention, and the regridding choice has to be tested for whether it changes the ranking. Comparing two gridded products quantitatively requires regridding them to a unified resolution and grid placement; even products at the same 0.5° resolution can have unaligned grids.
LynxBenchAI publishes device results on the public PyTorch stack, a broadly indicative starting point for hardware choice; start by reviewing them for your candidate GPUs.
Choose the method by field type. In ESMF, bilinear interpolation takes a distance-weighted linear combination of the source values around each destination point, while first-order conservative regridding weights source cells by how much they overlap the destination cell and typically carries a larger interpolation error. We would use conservative regridding for fluxes and precipitation, and bilinear for smooth continuous fields such as temperature. ESMF also documents that its second-order method can produce values outside the source range, for example negatives from a non-negative field.
Regridding is not neutral. In one regridding study, a first-order conservative regrid of 0.1° data shifted back to its original grid reduced the mean by 15% and the standard deviation by 18% at global scale. We would regrid to the coarser grid or a neutral intermediate one rather than to either model’s native grid. We would then rescore under at least two methods across enough independent cases to show whether the ranking survives. Resolve lead-time conventions (initialisation-time versus valid-time indexing) alongside the spatial step, not instead of it.
Ensemble members can be scored against a common reference: MET’s Ensemble-Stat tool verifies members against observations, with climatology as the reference forecast.
Can RMSE, R² and MAPE from three pilots be turned into one comparable number?
Not by converting the reported figures; the reliable route is to rerun all three pilots’ raw predictions through one shared scoring script and express the result as a skill score against a common reference. The general skill score is 1 minus the model’s score divided by a reference score: 1 is perfect, 0 is no better than the reference and negative is worse; the mean squared skill score applies that form to MSE. For yield across varieties, we would score MSE as a skill score against a common baseline, such as a per-variety mean or persistence model.
LynxBenchAI publishes device results on the public PyTorch stack, a broadly indicative starting point for hardware choice; start by reviewing them for your candidate GPUs.
Two limits matter. Wheatcroft shows that the skill-score form of mean squared error can be biased, and that skill scores built on different accuracy measures cannot be compared directly even though they share a scale. So you pick one measure, one reference and one script, and apply them to all three pilots. We would not back-convert from reported RMSE, R² or MAPE without the raw predictions, and would harmonise evaluation period, population and grid first. A reference that suits one variety better than another can still tilt the comparison, so report the per-variety scores alongside the aggregate.
A trace before any tuning
What every section above shares is that the measurement has to come before the decision. Our position is that you profile before you optimise. A vendor-specific API is not faster by default, since the same code performs essentially the same, and best-practice rules only make sense in a context that comes from profiling. The same applies to verification. Whether a benchmark is believable depends on a trace of what ran, a profiler checked on that architecture, an environment that was controlled or recorded, and a scoring basis that is the same for everyone. The question you still have to settle is who will rerun that evidence when the model, driver or traffic changes, and how the result gets recorded so someone else can check it. That reporting step is covered in turning GPU performance numbers into auditable evidence.
FAQ
What is the difference between PTQ and QAT?
Post-training quantisation calibrates quantisation parameters offline by running a trained model over a sample dataset, with no further training. Quantisation-aware training inserts quantisation operations before training or fine-tuning, so the network can adapt to quantised weights and activations.
Is INT8 faster than FP8?
On dedicated hardware, van Baalen et al. (2023) found FP8 multiply-accumulate units 50% to 180% less efficient than INT8, depending on accumulator size. On a given GPU, the answer depends on which formats it supports natively: a 2024 study found FP8 helped on H100 while A100 had no FP8 support.
