AI system latency budgeting has no generic answer, because how an end-to-end ceiling splits across stages depends on the stack, the hardware and the traffic the pipeline actually runs under. The method does carry over: fix the ceiling, log a timestamp at every stage boundary, measure each stage in production, and let the measured profile of your own pipeline decide which bottleneck to cut first. Where the time goes changes from one pipeline to the next. It is CPU retrieval in one RAG system, speech-to-text and the LLM in a voice agent, and memory-bandwidth-bound decode in a cloud-offloaded driving model.

At the strictest end, Yano and Azumi (2026) state that a safety-critical perception-to-actuation pipeline must bound the total delay from sensor input to actuator output even when every individual node meets its own deadline, and that because execution time shifts with scene content and with contention for shared resources on the SoC, a single average or a single worst case is rarely actionable. For that class of system we would build the decomposition as a worst-case model per camera stream, per model, per shared SoC resource and per hardware variant, re-run separately for each variant, rather than infer it from bench averages.

How do I break one latency ceiling into per-stage budgets I can measure?

To break a latency ceiling into stage budgets, log a timestamp at every stage boundary, run the pipeline under representative load, and read each stage’s p50, p95 and p99 from the result. The stage with the largest or most volatile share of the ceiling gets attention first, and every other stage gets a sub-budget with headroom drawn from its own tail.

TechnoLynx does this decomposition as the first step of GPU Performance Engineering. TechnoLynx frames the pipeline into explicit stages with your team, instruments each one (Nsight Systems for GPU work, custom spans or WebRTC getStats elsewhere), captures per-stage percentiles under realistic load, and advises which stage to reallocate or optimise. Where the decision is which device a stage should run on, LynxBenchAI publishes how each device runs the public PyTorch stack. That is broadly indicative for custom PyTorch models, but it does not measure your own pipeline.

The boundaries worth logging in a voice pipeline, per Stable Kernel’s voice-ordering guide, run from audio received through endpointing, first and final ASR transcript, retrieval and tool calls, first LLM token and first TTS audio byte to audio delivered. The guide also says to separate measurements by turn type, because averaging all turns together hides which part of the conversation is creating the problem. Pipecat’s latency observer builds a contributions timeline that names every part of the user-to-bot interval, including the parts no service measures (VAD silence, turn detection), and can sort those parts by duration to show the bottleneck.

Worked example: per-stage p50 and p99 inside a 200 ms payment-screening budget

KONTOGRAPH, an anti-money-laundering pipeline for the SEPA Instant rail described in a 2026 arXiv paper, runs under a self-imposed 200 ms p99 end-to-end budget. Its per-stage figures over 2,000 scored payments on a single-threaded CPU are:

Stage	p50 (ms)	p99 (ms)
Feature fetch	1.84	4.67
Graph assembly	0.00	0.01
Inference	0.42	1.78
Explanation	0.00	1.87
Counterfactual search	0.00	3.59
Unaccounted	0.17	0.46
Total	2.71	7.97

Adding the stage p99s gives 4.67 + 0.01 + 1.78 + 1.87 + 3.59 + 0.46 = 12.38 ms, well above the measured total p99 of 7.97 ms. The authors explain the gap: each stage’s p99 is measured independently, and the slowest feature fetch does not happen in the same request as the slowest counterfactual search. Per-stage p99s do not sum to the end-to-end p99, so the authors report them unreconciled rather than adding them up.

The table has other details worth reading closely. The explanation and counterfactual medians are zero because only alerting payments are explained (98 of 2,000), so p99 is the meaningful figure for that path. Percentiles are nearest-rank rather than interpolated, so each figure is a latency some request actually experienced. Repeated runs on the same machine varied between 3.6 ms and 8.0 ms p99 depending on background load, and the authors defend only the order of magnitude of headroom, not a specific figure. Each decision also logs its per-stage latency at decision time instead of reconstructing it later. In this table, summing the stage tails overstates the measured total p99. It is the wrong number to report as the system’s p99.

The dominant stage differs by pipeline

The TeleRAG study (2025) measured CPU-based retrieval at 41.1% and 60.5% of total latency at batch sizes 1 and 4. The VectorLiteRAG authors (2025) saw CPU retrieval raise time-to-first-token from 197 ms to 606 ms with a 128-million-vector database. In a cascade voice agent, WebRTC.ventures (2026) ranks speech-to-text and LLM inference as the two largest contributors, with transport and playout comparatively fixed and small. Stable Kernel puts a well-optimised voice ordering system without major tool calls at roughly 800 ms, and one with RAG, loyalty, pricing or POS calls at 1,000 to 1,500 ms unless those calls are parallelised or cached, which is why fast components do not automatically make a fast system. WebRTC.ventures puts it plainly: the only figures worth trusting for a real comparison are the ones you measure on your own integration.

Advice such as “move retrieval to the GPU” or “use the vendor API” only makes sense in a context, and profiling the stage supplies that context. Reading about someone else’s pipeline does not. TeleRAG’s authors report one such case. Moving vector search to the GPU made the search itself faster, but fetch time over a 32 GB/s PCIe link left end-to-end latency about 3% slower on average until the transfer was hidden. Once profiling shows that the binding stage is GPU inference, optimising GPU inference latency through batching, precision and kernel choices becomes the next piece of work.

The gap between a vendor’s published figures and what you measure under your own network conditions is often the difference that matters.

Why set stage budgets on p99 rather than the average, and what causes the tail?

Stage budgets belong on p95 or p99 because an average can stay flat while the tail moves. The Google SRE book gives an example where a typical request is served in about 50 ms, 5% of requests are 20 times slower, and average-based monitoring would show no change over the day. A percentile is read by sorting requests and taking the value at a given rank: at p95, 5% of requests run slower, p99 catches the worst 1%, and p99.9 is where rare spikes appear. High-order percentiles give a plausible worst-case value, and the higher the variance, the more the tail shapes the typical user’s experience, especially at high load where queuing makes it worse.

Common causes of tail latency include:

• Fan-out. In Dean and Barroso’s “The Tail at Scale” (CACM, 2013), the p99 for a single request finishing is 10 ms, but the p99 for all fanned-out requests finishing is 140 ms. Redis’s engineering blog works through the odds: if each server has a 1% chance of running slow, a request touching 100 servers has better than a 60% chance of hitting at least one, so each component must run at a much tighter percentile than the aggregate target.
• Chained AI calls. Agent and RAG pipelines are fan-out systems in disguise, and a common signature is p99 time-to-first-token climbing while p50 stays stable, with vector-search variance often the driver.
• Batching against variance. When retrieval runs in batches, the slowest query bounds the whole batch, so a lower average per-query latency need not improve end-to-end latency. The VectorLiteRAG authors show that this tail amplification cannot be fixed by optimising either stage in isolation.
• Storage and dependencies. Page-cache misses and synchronous reads push p99 up sharply once a working set exceeds RAM. A backend averaging 100 ms can see 1% of requests take 5 seconds, and its p99 can become the median response of a frontend that depends on several such services.

Measurement can hide the tail too. The SRE book suggests, as the simplest way to tell a slow average from a slow tail, recording request counts in latency buckets with roughly exponential boundaries rather than raw latencies. Load generators that send the next request only after the previous response arrives effectively coordinate with the server and skip measurement during slow periods, an effect known as coordinated omission. wrk2 avoids it by generating constant-throughput load and measuring latency from the time each request should have been sent. A harness with this flaw reports a healthy p99 exactly when the server is struggling, so latency testing for AI inference has to start from a constant-rate load model.

Where should the ceiling itself come from?

A latency ceiling has to come from the specific system’s failure tolerance, not from a number shared across industries, and the three cases below end up in very different places. Once a threshold is set, LynxBenchAI’s published results, which show how each device runs the public PyTorch stack but do not set the threshold or run your pipeline, are the first place to check whether the devices on your shortlist can carry its compute share.

LynxBenchAI publishes how each device runs the public PyTorch stack, so it tells you which shortlisted devices can carry a stage’s compute share once the ceiling is fixed, and the first step is to look up those devices in its results.

Subscriber QoE monitoring

ITU-T Recommendation G.1010 (2001) groups multimedia applications into eight categories by their tolerance to information loss and delay. It treats exceeding an upper boundary as unsatisfactory, while exceeding a lower boundary is still acceptable but may waste network resources. Command and control traffic, for example, sits below 250 ms with zero loss. We would only put sub-second inference at the 5G core if measured incident data shows that cloud-link degradation exceeds the SLA’s tolerance. Otherwise a cloud region with a bounded fallback for link loss is sufficient, with the caveat that real incident data may be sparse.

Forward-collision warning

The Euro NCAP AEB Car-to-Car test protocol (version 4.3.1, February 2024) tests FCW in CCRs scenarios at 55 to 80 km/h and requires manufacturers to supply performance predictions for each grid point before testing begins. The protocol steps test speeds by observed contact and speed reduction rather than prescribing a processing time. Because no millisecond figure is given, we would derive the camera-exposure-to-warning deadline from worst-case closing speed and the stopping dynamics the warning must leave room for, and hold it as a hard bound rather than a statistical target.

Live captioning

A 2026 US survey of deaf and hard-of-hearing viewers (216 validated participants, 302 responses) led its authors to call addressing caption delay “low-hanging fruit”, much more so than accuracy metrics. In their study design, ASR captions carried a mean delay of 2 s, against typical broadcast TV caption delays of 7 to 12 s. For a quantised captioning model, we would agree one numeric threshold for timestamp drift within the end-to-end budget before testing and use it as the ship gate. That gate is only as good as the representativeness of the test content behind it.

How do I build a worst-case timing model for a perception-to-actuation pipeline on a shared SoC?

A worst-case timing model bounds sensor-to-actuator delay even when every node meets its own deadline. It separates deterministic from stochastic latency terms and requires both positive deterministic slack and tail compliance. TechnoLynx can take on the engineering of such a model: building the budget tree per stage and per resource, instrumenting the pipeline with per-stage spans, emulating WAN loss and jitter or co-running workloads to extract P99.9 per stage, and verifying that the fallback triggers fire inside their own bound. The first step is agreeing with your safety team what “worst case” means operationally, in terms of percentile, stress scenarios and thermal and voltage corners.

Parsa et al. show the structure for a cloud-offloaded driving loop. End-to-end latency is the sum of nine terms grouped into capture, offload, compute and return, with deterministic per-cycle terms kept separate from stochastic communication and GPU-contention terms. Their feasibility gate needs both positive deterministic slack and a probability no larger than a target (10⁻⁵ in their study) that the stochastic sum exceeds the remaining budget. The binding term moves with conditions: at dense cell loading it is network scheduling, and in GPU-heavy scenarios it shifts to GPU queueing and inference. For the stack they calibrate (FP16, single-request, on a 2025-class B300), the deterministic floor for their vision-language-action model was 132 to 164 ms, above a 100 ms reactive budget for every offloading strategy, with decode alone around 114 ms because it is memory-bandwidth-bound. Their 300 ms deliberative tier is admissible only behind an onboard reactive fallback. The constraints bind in sequence (uplink, then latency, then cost), so whether the cloud can drive depends on model class, GPU year and budget, and there is no general yes or no.

Contention for shared resources on a SoC adds variability that a single average hides. In Bechtel and Yun’s industrial case study on a Jetson Nano (2023), a DNN’s average inference time rose from about 34 ms running alone to about 37 ms alongside SLAM, and to about 48 ms once cache-targeting denial-of-service tasks were added. Different SLAM threads were sensitive to different resources, with one barely affected, one sensitive to both DRAM bandwidth and the last-level cache, and one mainly to the cache. Their fix combined partitioned gang scheduling with iGPU and last-level-cache bandwidth throttling, with the throttling level chosen as a trade-off between the DNN keeping its 20 Hz rate and SLAM accuracy. A single slowdown factor cannot capture that per-resource sensitivity, so we model interference per resource.

Yano and Azumi (2026) argue that execution time in driving stacks should separate scene-dependent workload, which is largely fixed once the scene is known, from hardware- and runtime-induced residual variation, and that folding the former into a probability distribution mischaracterises it. The method we would use follows from that. We would run measurement-based worst-case timing with stress inputs that cover the heavy scenes, add interference modelling for the shared GPU (for example AEB and lane-centring running concurrently), and repeat the whole exercise per trim or hardware variant, because the method transfers between variants but the numeric allocations do not. Tools such as ros2_tracing, CARET and Autoware_Perf measure end-to-end and path-level latency on real systems, and TILDE detects deadline misses at runtime. A low-overhead online detector that is wired to fallback and ready for production deployment is, by the same authors’ account, still an open problem.

Several things can break the bound. Co-scheduled cache-bank-aware DoS tasks did so on the Jetson Nano, forcing SLAM to drop most incoming frames. A per-stage P99.9 also means little until each stage has enough samples behind it.

TechnoLynx measures a single model’s own latency contribution, such as an AI noise-suppression step in a call pipeline. On-device profilers such as Google’s LiteRT benchmark tooling expose traces that show, for instance, when operators are not fully delegated and part of the graph runs on the CPU instead of the intended accelerator.

How do we turn a deadline into a spec vendors are measured against?

Write the requirement as throughput, latency and concurrency measured on your own workload and sized for peak concurrent demand, before any vendor quotes a number. LynxBenchAI’s published results, which show how each device runs the public PyTorch stack but not your own model code or communication pattern, give a neutral first shortlist, so the first step is to read them for the devices you plan to invite to tender.

LynxBenchAI publishes how each device runs the public PyTorch stack, which gives every vendor the same neutral reference point before quotes arrive, and the first step is to look up the devices you plan to invite to tender.

Parsa et al. size edge capacity for peak concurrent demand rather than total registered fleet size. At 12% utilisation, sizing for the whole fleet would over-provision GPU capacity by several factors. For a small previsualisation team, how many sessions can run without queuing is the same calculation: the measure is concurrency at the crunch peak, not head count. At high load, queuing effects make long-tail behaviour worse. VectorLiteRAG’s authors re-profiled baseline search latency for every hardware configuration before reapplying their partitioning algorithm. A tender benchmark needs the same discipline. For ensemble forecasting, TechnoLynx builds it from a scaled-down but representative version of the ensemble model and its halo-exchange pattern.

For a robotic pick cell, TechnoLynx derives the perception budget with you from a measured breakdown of the real pick cycle. TechnoLynx then applies quantisation or pruning on the candidate edge board and benchmarks end-to-end latency and accuracy against that budget.

AI system latency budgeting in four concrete pipelines

The same budgeting method plays out very differently in four pipelines readers ask about: a speech-to-text ceiling with fixed telephony hops, an LLM response path, an XR frame, and an in-browser model load.

Of a 2-second speech-to-text budget, how much can GPU inference take?

The GPU inference step in a 2-second speech-to-text budget gets whatever is left after subtracting the measured tails of the fixed stages, so the honest answer is a formula fed with your own measurements, not a number.

GPU inference budget ≈ 2,000 ms
  − p99(telephony ingress)
  − p99(network hops, including de-jitter buffering)
  − p99(endpointing)
  − p99(post-processing and delivery)
  − headroom

Subtracting stage p99s reserves more than the measured total needs, as the KONTOGRAPH figures show, so it is a pessimistic reservation, not the system’s p99. For the network terms, ITU-T G.114 (2003) recommends assuming for planning that a de-jitter buffer adds half its peak delay to the mean network delay. It gives encoder/decoder delay as 2 × frame size + look-ahead. Endpointing is easy to forget: in Pipecat’s documented example, the VAD endpointing wait alone is 0.200 s of a 1.020 s turn, ahead of 0.125 s transcription, 0.336 s LLM inference and 0.359 s speech synthesis.

A defensible method logs per-stage latency with each request at the time it happens and reports nearest-rank percentiles, as the KONTOGRAPH study does. WebRTC.ventures (2026) notes there is no single correct latency budget for voice AI, since the silence a caller will tolerate shifts with the stakes of the conversation.

How do a single LLM call, a RAG lookup and an agent chain compare on p99?

Each added hop brings its own tail: a RAG path adds retrieval variance, and a multi-hop agent rolls the dice against the tail on every tool call. Latency increases in RAG systems also compound across multiple rounds of generation and retrieval.

Path	What widens the tail	How to fit it into a fixed budget
Single LLM call	Time-to-first-token and queueing under load	Choose the model by time-to-first-token, not total generation time, and stream output
RAG lookup	Retrieval variance, with the batch bounded by its slowest query	Design tiered or cached retrieval around hit-rate variance and long-tail queries
Multi-hop agent	Sequential tool calls, each with its own tail	We would cap hops with per-call timeouts and a fallback answer, or move the agent to an offline second pass

Per-call timeouts come from the per-component tail budget, which has to be tighter than the aggregate target as the number of hops grows. If the real-time step is triage, we would keep the agent path out of it and run the agent as a slower second-pass review, because no timeout policy makes a chain of uncapped tool calls predictable at p99. The actual p99 of each path on your infrastructure is a measurement, not something a published study can give you.

How should an 11.1 ms XR frame be split, and what gives when you de-rate?

At 90 Hz the frame budget is 11.1 ms, according to Meta’s Horizon OS documentation (Android’s XR guidance rounds it to 11 ms), against 13.9 ms at 72 Hz and 8.3 ms at 120 Hz. If combined CPU and GPU work overruns, the compositor re-displays the previous frame and the app falls to half rate. At 72 Hz, an 18 ms GPU frame is delivered every 27.8 ms. To recover, the GPU work has to fit inside a single refresh again, which at half rate means GPU utilisation must drop below roughly 50%. Meta advises comparing the per-frame App GPU time in milliseconds against the budget rather than relying on GPU%, which can read misleadingly low during drops. If App time is within budget but stale frames persist, the bottleneck is probably on the CPU side.

We would not fix the split in advance. We would measure shading, reprojection and compositor time per frame on the target headset, reserve the compositor and reprojection their measured worst case first because they are what rescues a late frame, and give shading what remains, with margin for thermal clock drops. When you have to de-rate, we would cut shading rate (foveation and VRS tiers) first. A frame bound by CPU submission will not be rescued by any shading change, which is why the per-frame measurement comes before the plan.

Can I time Pyodide init, package fetch, WASM compile and inference without shipping a profiler?

Yes. The browser’s own performance APIs and the runtimes’ built-in profiling flags give a stage-by-stage breakdown. Cloudflare reports that Pyodide takes about two seconds to initialise in a browser on a fast computer with a good connection, split roughly into one second of network time and one second of CPU time. On a low-end student Chromebook both halves will likely grow, which is why the timing has to come from the devices themselves.

1. Interpreter init: place performance.mark() calls before and after loading Pyodide and record the interval with performance.measure() from the W3C User Timing API.
2. Package and wheel fetch: read PerformanceResourceTiming entries, which break each fetched resource into its phases where a single timer cannot.
3. WASM compile: in a V8 trace, Liftoff baseline compilation ends at the wasm.BaselineFinished event and TurboFan top-tier compilation at wasm.TopTierFinished. Profile the TurboFan code, because Liftoff code only runs until TurboFan finishes and is not representative.
4. Inference: for ONNX Runtime Web, set enableProfiling in the session options and call endProfiling() after the run, or set ort.env.trace = true to log trace data through console.timeStamp. WebGPU profiling takes an ondata handler. For a Pyodide model.predict call, a mark and measure around the call is enough.
5. Collection: retrieve the entries with getEntriesByType() or a PerformanceObserver and send back a compact summary instead of a full trace.

Edge inference, a smaller model or a new protocol: which keeps a 2-second bound reliable?

To find which change keeps a 2-second evidentiary bound reliable, measure end-to-end tail latency for each candidate (edge device versus remote GPU, full-size versus smaller classifier, current versus alternative network protocol) under realistic venue conditions. Then pick the combination that meets the bound at the required percentile with acceptable accuracy, because for an evidentiary requirement the average is the wrong target. TechnoLynx can build and measure each candidate on its target hardware. The Inference Cost-Cut Pack covers shrinking and optimising the model stages, the AI Porting & Deployment Pack covers moving ASR and classification onto an edge device, and the first step is writing the 2-second requirement down with you as a tail-latency SLA. LynxBenchAI’s published device results help shortlist the edge hardware before any porting starts.

Model size moves ASR latency a long way. The Moonshine v2 authors (2026) measured 50 ms for their Tiny model (5.8 times faster than Whisper Tiny), 148 ms for Small and 258 ms for Medium, on the same hardware. They also frame latency and transcription quality as competing constraints on edge devices with limited memory, compute and power. They mark 250 ms as a commonly used one-way delay limit for interactive voice in private networks, citing Cisco. Moving to the edge removes the network round trip but puts the compute constraint on local hardware, which can force exactly that smaller model. TechnoLynx would validate each smaller classifier’s accuracy per legal objection category, not just on average. TechnoLynx would validate tail latency under realistic courtroom network load and keep monitoring it after deployment.

Who can decompose our latency budget and fix the stage that breaks it?

TechnoLynx instruments your pipeline stage by stage, measures each stage’s p95 and p99 under realistic load, and then optimises the stage the measurement shows to be binding, whether that is GPU inference, retrieval, batching or data movement. Whether to do this in-house or bring in a specialist depends on where the binding stage sits. A team with tracing, a constant-rate load harness and people who read Nsight traces is well placed. Specialist help pays when the dominant stage turns out to be GPU kernels, memory traffic or cross-stage batching the team has not tuned before. Whoever you choose, ask for per-stage percentile evidence from your own pipeline before any optimisation proposal, and for re-measurement after every change. The engagement is GPU Performance Engineering, and it starts with a profiling pass that frames your stages and their budgets.

For an automotive safety case we deliver the engineering (timing models, instrumentation, stress campaigns), and your safety assessors own the sign-off. A team that only needs to compare devices is better served by LynxBenchAI’s published results, which measure hardware on a public stack rather than your application. Where the bottleneck is a third-party SDK that exposes no timing, the realistic routes are the vendor’s own support or replacing the component.

FAQ

How much latency is acceptable for VoIP?

ITU-T G.114 (2003) says most applications experience essentially transparent interactivity below 150 ms one-way delay, and it recommends not exceeding 400 ms one-way for general network planning.

What does glass-to-glass latency mean?

Glass-to-glass latency is the time from light reaching a camera’s sensor to the corresponding image appearing on a display, including capture, encoding, transport, decoding and rendering.

Is lower latency always better?

Not without limit. ITU-T G.1010 (2001) says a service that exceeds an application’s lower boundary is still acceptable but may be wasteful of network resources.