To find where a restructured rendering kernel gains or loses time against an offline-baked baseline, run both versions on the same frozen scene, with clocks and power state held fixed, and capture a per-stage GPU timeline for every frame. On NVIDIA hardware that means NVIDIA Nsight Systems, with Nsight Graphics GPU Trace for detail inside the frame. On a standalone Quest headset it means Meta’s ovrgpuprofiler render-stage traces or Perfetto. Then compare the runs stage by stage, with variance, instead of comparing one averaged frame time.
TechnoLynx’s GPU Performance Engineering service sets up this comparison on a team’s own headset or GPU and workload. The first step is defining the per-stage breakdown and the matched-scene protocol for the baseline and the restructured kernel, followed by picking and wiring the on-device capture tooling; the details are on the GPU Performance Engineering page.
Why do averaged frame times hide which stage got slower?
If the restructured kernel adds work in one stage and removes it from another, the totals can match while the pipeline underneath behaves differently, and the mean does not say where to look. Pavel Šmejkal’s notes on GPU timing make a related point from the instrumentation side: timing individual passes gives a scope-based profiler, and work on async compute queues cannot simply be summed to give a frame time.
Averages also throw away when things happen. A 2026 arXiv study of XR timing (“Beyond Mean Frametime”) notes that reporting with means, standard deviations, percentiles or histograms discards the temporal ordering of a timing trace. That ordering carries variability, periodicity, transient structure and burstiness. In the study’s Beat Saber frametime data, mean and standard deviation alone gave almost no separation between headset groups in the full dataset, while time-series features kept substantial separation. Shuffling the frame order within each trace reduced, but did not remove, that separation, most strongly under content matching; the authors read this as direct evidence that sequential order contributes information the distribution does not. Their practical conclusion applies directly here: when two configurations have similar mean frametimes but different timing signatures, report the temporal properties alongside mean and SD.
There is a caveat on scope. That study measured application-level frame pacing, not component-resolved latencies such as compositing or scan-out. It shows that distribution-only summaries miss temporal structure. It does not show which stage is responsible, and that is exactly why you need a per-stage capture as well.
Run-to-run noise exists even with no code change. In an arXiv study of GPU performance modelling (2020), rendering the same frame took different amounts of time from one run to the next because memory access times varied, and frames of similar complexity varied considerably too. So a small difference between two averages is not a divergence until you know the spread.
TechnoLynx’s position is that the profile comes before the explanation: rules of thumb about what makes a shader or kernel slow only make sense in context, and profiling is what supplies that context.
NVIDIA Nsight Systems, Nsight Graphics or Nsight Compute: which one localises the divergence?
These three tools answer different questions, and using the wrong one gives you numbers that do not compare. For a fuller walkthrough of the GPU profiler workflow across Nsight Systems and Nsight Compute, see the dedicated page. Here the question is narrower: which tool shows where one version of a frame differs from another.
NVIDIA Nsight Systems is the first pass. NVIDIA describes it as putting system-wide CPU and GPU activity on a single timeline, and for graphics it profiles Vulkan, OpenGL, DirectX 11, DirectX 12, DXR and OptiX. It flags slow frames and stutter frames and reports CPU time per frame. Its Frame Health row compares each frame’s actions (shader compilation, present, memory mapping) with the median of the surrounding 19 frames. That last feature shows when a slow frame comes from an action such as shader compilation.
Nsight Graphics GPU Trace goes inside the frame. It runs on NVIDIA Turing and later GPUs and captures GPU unit utilisation throughout frame execution. It collects up to 15 consecutive frames per trace and shows work on each queue (graphics, compute, copy) as it executed. One trap: a GPU Trace collected from Live Replay profiles the replayed capture rather than the live application, without its CPU workload or pacing, so its timings may not match the live run.
Nsight Compute is for a single kernel once you know which one matters. It serialises kernel launches unless a dedicated replay mode is used, and Nsight Systems does not. It flushes GPU caches between replay passes by default and measures workload duration with a different method from Nsight Systems. Its durations therefore belong in a kernel deep-dive, not in a frame-time comparison. NVIDIA’s TensorRT guidance layers tools the same way: the built-in profiler finds the expensive layers, then Nsight Systems explains them at kernel level.
All three are NVIDIA tools. Nsight Systems targets NVIDIA Tegra devices, Arm SBSA and Arm Windows systems, and x86_64 systems with NVIDIA GPUs. Meta Quest 3 and Quest 3S use an Adreno 740v3 GPU, outside the NVIDIA platforms Nsight Systems targets.
Tool fit at a glance
| Tool | Runs on | What it shows | Use it to |
|---|---|---|---|
| Nsight Systems | NVIDIA GPUs on x86_64, Arm and Tegra | CPU and GPU timeline, per-frame CPU time, stutter frames | Find which frames and which region diverge |
| Nsight Graphics GPU Trace | NVIDIA Turing and later | Per-queue events and unit utilisation over up to 15 frames | See which pass or unit inside the frame changed |
| Nsight Compute | NVIDIA CUDA kernels | Per-kernel metrics with replay, serialised launches | Explain one kernel already localised |
| ovrgpuprofiler | Meta Quest headsets | Time per render stage per surface | Per-stage comparison on the headset |
| Perfetto (Meta Quest Developer Hub) | Meta Quest | GPU render stages and CPU events on one timeline | Tell whether a missed frame is CPU- or GPU-side |
| RenderDoc Meta Fork | Meta Quest | Single-frame tile timeline, up to 48 metrics per draw call | Inspect one representative frame |
| OVR Metrics Tool | Meta Quest | Session metrics with CSV recording | Record clock levels and frequencies alongside runs |
Capturing GPU timeline data on a standalone XR headset
On Quest, ovrgpuprofiler is a command-line tool for real-time GPU metrics and render-stage traces. The headset GPU is a tiler, and that changes what “per stage” means. Tile-based renderers batch the draw calls for a whole surface, split the surface into tiles and process the tiles in sequence. ovrgpuprofiler reports how much time each surface spent in each rendering stage. So the breakdown you define should use binning, render and store stages per surface, not the vertex/fragment split familiar from desktop GPUs.
Meta’s documented trace output shows the shape of the data. One example surface at 1216x1344 with MSAA 4 took 5.08 ms, split into Binning 0.623 ms, Render 1.877 ms, StoreColor 0.309 ms, Blit 0.002 ms and Preempt 1.286 ms. Meta notes that the 5.08 ms span includes profiling overhead and Preempt intervals and is not profiler-off GPU busy time. Render-stage tracing needs detailed profiling mode enabled first (-e). -t runs a trace with a default length of 0.1 s, -v expands each surface into its per-bin and per-stage breakdown, and -s attaches GPU counter values to specific binning, render or store stages rather than to the frame as a whole.
Profiling overhead is the caveat to state once and respect throughout. Meta says detailed GPU profiling adds instrumentation overhead to measured rendering times, that the figure of roughly 10% is not a universal correction, and that A/B comparisons should use interleaved measurements with profiling disabled, with -t and with -lt, under matching scene, GPU clock and compositor conditions. In practice that means alternating baseline, restructured, baseline, restructured, each with profiling on and off, rather than running all baseline captures first.
The other Quest tools fill gaps around that trace:
- Perfetto through Meta Quest Developer Hub records GPU render-stage traces, with a high-precision option, next to CPU scheduling and in-app markers on one timeline. It shows whether a missed frame sits on the CPU, the GPU or both.
- RenderDoc Meta Fork runs a tile-level render-stage trace of a single frame and shows it in a Tile Timeline with timing per surface and per tile. It also exposes hidden state such as a surface’s rendering mode and bin count, which matters if the restructured kernel accidentally changes how a surface is binned.
- OVR Metrics Tool records session performance data in report mode, and some of its advanced GPU statistics need ovrgpuprofiler enabled from a connected shell.
On other Android devices, Google’s Android Performance Analyzer documentation says GPU slices give very accurate per-frame GPU time: on Adreno they appear in sequence on the GPU Queue 0 track, and on Mali they split into fragment and non-fragment tracks.
A lightweight cross-check that does not depend on any vendor profiler is to put Vulkan timestamp queries around each render pass in your own build. The Vulkan samples use only top-of-pipe and bottom-of-pipe timestamps, noting that many other stage combinations give no meaningful result and that timestamps from different queues cannot be compared. Ticks convert to time through the device’s timestampPeriod.
Making both versions render the same frames
Matched traces start with matched work. The 2020 GPU modelling study found the most reliable way to collect reference data was to vary GPU frequency while freezing the workload. For this comparison, that means a fixed camera path or a static camera, fixed assets and a fixed frame count, all driven by a script and not by a person wearing the headset. The XR timing study did the equivalent with its data: it fixed song, difficulty, environment, mode and modifiers to build a content-matched subset that removed confounders.
The capture window then has to land on the same frames in both runs. A numbered procedure keeps this repeatable:
- Warm up before capturing. NVIDIA advises waiting for background shader compiles to go inactive when profiling a stable scene, because the resulting numbers are more stable and representative.
- Bracket the region in code. In Nsight Systems, NVTX ranges limit profiling to a CPU region of interest, and
--capture-rangestarts collection only when the matching start API or hotkey fires. With-c nvtx, collection starts when an NVTX range with a given message opens and stops when the capture ranges close. - Count frames, not seconds. Nsight Graphics GPU Trace accepts
--limit-to-framesto cap a trace at a set number of frames. - Capture several regions per run if needed. Nsight Systems can collect multiple regions of interest in one long-running process and defer report generation.
- Align the reports for review. When several reports load into one timeline, Nsight Systems aligns them by UTC time by default, or by vClock where the reports contain vClock correlation data.
Keep NVTX ranges coarse. NVIDIA’s guidance is to annotate carefully around code that runs in under a microsecond, and to avoid leaving many ranges open, since Nsight Systems cannot visualise reports with many unclosed ranges.
Clocks, thermals and power state
Unlocked clocks make two runs hard to compare. NVIDIA states that its GPUs adjust core and memory clocks during execution, which makes comparisons between runs difficult. Its TensorRT documentation adds that two runs of the same engine on the same GPU can disagree by tens of percent if the clock floats in one run and is locked in the other, or if thermal throttling starts partway through. On NVIDIA GPUs, nvidia-smi locks core and memory clocks to set rates before measurements, and GPU Trace locks the clock to base by default so that consecutive reports are comparable. When comparing traces, NVIDIA says to collect them under equivalent hardware, software and shader conditions. Locking does not rule out throttling: once throttling begins, the device behaves as though the clock floats, so log clocks and temperature during each run.
Headsets give you less control. On Meta Quest, CPU and GPU levels are hints and not fixed frequencies, the runtime adjusts them for thermal and power conditions, and Meta says to profile the resulting levels and frequencies on each device. The OS also lowers levels when a scene is less demanding. That creates a specific confound: a lighter restructured kernel may run at lower levels than the baseline. So record level and frequency with OVR Metrics Tool or Meta Quest Developer Hub alongside every trace, start each run from a similar thermal state, and treat any run where frequency differs between versions as unmatched.
Presenting per-stage timings so reviewers trust the comparison
Reviewers trust a comparison they could rerun and that shows its own uncertainty. Kalibera and Jones (2013) note that many evaluations lack proper variation estimates, so reported small differences between two systems are sometimes unreliable, and they set out a method for choosing repetition counts and reporting an effect-size confidence interval. Hoefler and Belli (2015) proposed a minimal set of rules for reporting performance results to make them easier to interpret. For this comparison, that means showing the difference per stage, with its interval, across repeated runs and not a single capture.
A reviewer-ready report would contain these items:
- Per-stage table, both versions. One row per stage (binning, render, store per surface on a tiler; per pass or queue on desktop), with median and interval for baseline, restructured and the difference. NVIDIA’s TensorRT guidance treats median latency as the number to compare.
- Variance flagged, not averaged away. Nsight Graphics GPU Trace’s aggregate mode averages metrics across frames and greys out values with significant frame-to-frame variation. Carry that signal into the report.
- Per-frame series for the diverging stage. Raw traces remain the place to inspect bursts, oscillations and stalls in that stage.
- Profiler-off totals. Interleaved profiler-off frame times, as Meta recommends for A/B comparisons, check the per-stage difference without instrumentation.
- Conditions recorded. Clock levels and frequencies, thermal state, build type and capture settings for every run. Nsight Compute’s guidance to keep capture settings identical and to use optimised builds, because debug builds change code generation and timings, applies to any profiler.
- Rerun instructions. The exact scene script, frame range and commands, so another engineer can reproduce the table.
Read each stage from the confidence interval of its difference: an interval that excludes zero marks that stage as diverging; one that includes zero means this repetition count has not shown a divergence there. Whether the change is acceptable is a separate question, settled later by regression validation against the original artefact or budget.
Who can set up a matched per-stage profiling comparison for our headset renderer?
TechnoLynx sets up matched per-stage comparisons of a restructured kernel against an offline-baked baseline, on the reader’s own headset or GPU. The work follows a fixed order: define the per-stage breakdown and the matched-scene protocol, choose the on-device tracing tools for that headset and GPU vendor, add low-overhead instrumentation and the capture pipeline, check that the traces are reproducible and matched frame for frame with variance bounds, and document the workflow so reviewers can rerun it. All of this sits within GPU Performance Engineering. Bringing in a specialist makes sense when the comparison has to convince someone outside the team. Building it in-house makes sense when the team already knows its headset tooling well. Either way, judge a partner by whether its traces can be reproduced.
This engagement measures the reader’s own kernel and baseline on their own workload. If the open question is which hardware to target in the first place, LynxBenchAI publishes hardware measurements for that decision, but it does not run your renderer. A team that only needs a one-off look at a single frame is better served by RenderDoc Meta Fork or GPU Trace directly than by a structured engagement.
