Inference latency is an engineering problem, not a hardware problem A team deploys a Transformer model for real-time inference. The target latency is 20 milliseconds per request. The model runs on an A100 GPU. The measured latency is 85 milliseconds. The first proposal: upgrade to H100. The upgrade delivers 45 milliseconds — a real improvement from the hardware, still more than double the target. The latency budget is not a hardware problem. It is a software optimisation problem that happens to run on hardware. Inference latency is the sum of multiple components: data preprocessing, model execution (which itself comprises kernel launches and memory operations), post-processing, and serving infrastructure overhead. Optimising total latency requires understanding which component dominates and what interventions are available for each. In our experience across GPU inference engagements, the majority of production AI deployments exceed their initial latency targets — often by 2× or more — with model serving overhead frequently accounting for a third or more of total end-to-end latency (observed pattern, not a benchmarked industry rate). MLPerf Inference v4.0 benchmark results illustrate this directly: across submissions on the same hardware class, differences in software stack, compilation strategy, and serving configuration produce throughput variations of 2–5× (benchmark, MLPerf Inference v4.0). The hardware is identical; the software configuration determines the performance. For a benchmarking lens on why throughput and latency are different optimisation targets, not interchangeable metrics, we treat them as separate workstreams in our LynxBench AI methodology. NVIDIA’s TensorRT documentation reports 2–6× inference speedup over native PyTorch on Transformer architectures, depending on model size and GPU target (per NVIDIA’s published documentation). MLPerf Inference v4.0 also shows INT8 quantisation arriving within roughly 1% accuracy of FP32 for ResNet-50 and BERT while delivering 2–4× latency reduction on A100 and H100 hardware (benchmark, MLPerf Inference v4.0). The first three moves on any GPU inference workload Before getting to the wider sequence, three interventions deserve a quick-answer block — they are the highest-leverage starting points and most teams should apply them before considering anything else. Model compilation (TensorRT). Export the model to ONNX and compile with TensorRT to eliminate training-mode overhead, fuse kernels, and select GPU-optimised implementations. Observed improvement across our engagements: roughly 2–5× over PyTorch eager execution (observed pattern, not a benchmarked industry rate). Highest-impact, lowest-effort intervention. FP16 quantisation. Reduce weight and activation precision from FP32 to FP16 for an additional 1.5–2× latency reduction. Accuracy degradation is typically below 0.1 percentage points. Apply by default to every inference workload unless FP32 is required for numerical correctness. INT8 quantisation. Calibrate and quantise to INT8 for a further 2–4× latency reduction when FP16 alone does not meet the target. Requires a calibration dataset and accuracy validation — large language models may lose 1–3 percentage points on aggregate metrics. Apply when the latency budget is tight and the accuracy trade-off is acceptable. This is the diagnostic surface we use in the first hour of a GPU Performance Audit on an inference workload. Everything below explains why these three sit at the top of the list, and what to do when they aren’t enough. Why is model compilation the first optimisation step? Running a PyTorch or TensorFlow model in its training-time execution mode for inference is the most common source of unnecessary latency. Training-mode execution includes dynamic graph construction, eager operation dispatch, and runtime type checking — none of which are needed at inference time. Compilation eliminates this overhead before any other intervention has a chance to matter. TensorRT is NVIDIA’s inference optimisation compiler. It takes a trained model (from ONNX, PyTorch, or TensorFlow), analyses the computation graph, applies kernel fusion (combining multiple operations into single kernels to reduce launch overhead and memory traffic), selects optimised kernel implementations for the target GPU architecture, and produces a compiled inference engine. The speedup from TensorRT compilation is typically 2–5× over PyTorch eager execution (per NVIDIA’s published documentation and consistent with our own measurements), with no accuracy change for FP32 compilation. torch.compile (PyTorch 2.x) provides graph-mode compilation within the PyTorch ecosystem. It captures the computation graph, applies graph-level optimisations, and generates kernels through backends like Triton. The speedup we observe across our engagements is roughly 1.5–3× over eager execution (observed pattern, not a benchmarked industry rate) — less than TensorRT, but the workflow is simpler (no ONNX export, no separate compilation step) and the compiled model remains a PyTorch object that can be debugged and modified. ONNX Runtime offers cross-platform inference optimisation with execution providers for multiple hardware targets (CUDA, TensorRT, DirectML, OpenVINO). For workloads that need to run on heterogeneous hardware — GPU and CPU inference targets evaluated side by side — ONNX Runtime offers a single inference API with hardware-specific optimisation. Our recommendation is straightforward: compile the model with TensorRT as the first step. The effort is minimal (export to ONNX, run the TensorRT compiler) and the speedup is typically the largest single improvement available. We have seen production inference pipelines where TensorRT compilation alone brought latency from above-target to within-target, eliminating the need for further optimisation work. Quantisation: trading precision for speed Quantisation reduces the numerical precision of model weights and activations from FP32 to FP16, INT8, or INT4. The latency benefit comes from two sources: reduced memory bandwidth (the dominant bottleneck for many inference workloads — moving half as many bytes per operation directly halves the memory-bound execution time) and higher compute throughput (tensor cores execute FP16 operations at 2× the rate of FP32, and INT8 at 4× the rate, per NVIDIA’s published specifications). FP16 quantisation is nearly loss-free for most models. Accuracy degradation is typically less than 0.1 percentage points and the latency reduction is 1.5–2×. It should be applied by default for any inference workload that does not require FP32 for numerical correctness. INT8 quantisation delivers a 2–4× latency reduction but requires calibration: running representative data through the model to determine the quantisation parameters (scale and zero-point) per layer. Post-training quantisation (PTQ) applies INT8 to an already-trained model; quantisation-aware training (QAT) trains the model under quantisation constraints and tends to preserve accuracy better. The accuracy impact of INT8 varies by architecture: large language models typically lose 1–3 percentage points; image classification models lose 0.5–1.5; object detection models lose 1–2 on mAP (observed pattern across our engagements, not a benchmarked industry rate). The accuracy–latency trade-off has to be evaluated against the application’s acceptance criteria. A model that meets the latency target after FP16 quantisation with negligible accuracy loss does not need INT8. A model that requires INT8 to meet latency targets has to be validated for acceptable accuracy at INT8 precision before it ships. Batching strategy: throughput vs tail latency Batching — processing multiple inference requests in a single forward pass — improves GPU throughput by amortising kernel launch overhead and enabling more efficient memory access patterns. The cost is added latency for individual requests, because each request waits for the batch to be assembled before processing begins. Static batching collects a fixed number of requests before processing. The latency impact is bounded: maximum additional latency equals (batch_size − 1) × inter-arrival time. For high-throughput, latency-tolerant workloads (offline batch processing, search indexing), static batching with large batch sizes maximises GPU utilisation. Dynamic batching collects requests over a configurable time window and processes whatever has arrived when the window expires. This bounds the additional latency to the window duration (typically 5–20 milliseconds) while allowing batch size to vary with demand. NVIDIA Triton Inference Server, TorchServe, and most production serving frameworks support dynamic batching out of the box. Continuous batching (sometimes called inflight batching) is specific to autoregressive models — LLMs and sequence generators where different requests sit at different stages of generation. Instead of waiting for all requests in a batch to complete before accepting new ones, continuous batching inserts new requests into the batch as in-progress requests finish. This eliminates the padding waste of static batching for variable-length sequences and maintains higher GPU utilisation under realistic LLM traffic. The strategy depends on the latency target. As a planning heuristic from our GPU inference engagements (observed pattern, not a benchmarked industry rate): if the target is strict (sub-20ms per request), dynamic batching with a short window is appropriate. If the target is relaxed (sub-500ms), larger batches improve throughput and reduce per-request cost without breaching the SLA. Memory management: eliminating allocation overhead GPU memory allocation and deallocation (cudaMalloc, cudaFree) are expensive operations — typically 100–1000 microseconds per call (per NVIDIA’s CUDA documentation). In an inference pipeline that allocates and frees intermediate buffers on each request, cumulative allocation overhead can become significant relative to the inference time itself. Memory pooling pre-allocates a block of GPU memory at startup and serves individual allocation requests from the pool. Allocation cost drops from hundreds of microseconds to nanoseconds. PyTorch’s caching allocator does this automatically for PyTorch workloads; custom CUDA pipelines should use memory pools (CUDA’s cudaMemPool API or a custom pool) rather than direct cudaMalloc / cudaFree calls. Static memory planning determines the complete memory layout of the inference graph at compilation time, assigning fixed memory addresses to each intermediate tensor. TensorRT performs this automatically. For custom pipelines, static memory planning eliminates runtime allocation entirely — the layout is fixed and each inference pass reuses the same buffers. The optimisation sequence Apply the interventions described above in order, profiling after each step to determine whether the latency target has been met. # Step Typical effect Evidence class 1 Compile (TensorRT, then torch.compile if simpler workflow needed) 2–5× over eager benchmark / observed-pattern 2 Quantise to FP16 1.5–2× benchmark 3 Profile compute-bound vs memory-bound (diagnostic) — 4 Quantise to INT8 (if target not met and accuracy acceptable) 2–4× benchmark 5 Optimise batching (dynamic / continuous) throughput vs tail-latency trade observed-pattern 6 Optimise memory management (pooling, static planning) tens to hundreds of µs per request observed-pattern If the sequence does not achieve the latency target, the next level of intervention is algorithmic restructuring — changing the model architecture, pruning, or adopting a more efficient architecture (for example, replacing a dense Transformer with a linear-attention variant). The same techniques become even more critical when deploying CV models on edge devices, where compute and memory budgets are a fraction of data centre hardware. What makes inference infrastructure reliable and cost-efficient? Latency optimisation solves one half of the inference problem. The other half is keeping the serving infrastructure stable and economically viable once traffic is live. Reliability in inference serving means redundancy and observability. Production deployments run multiple model replicas behind a load balancer, with health checks that route traffic away from unhealthy replicas automatically. Serving frameworks like Triton Inference Server and KServe provide liveness and readiness probes, automatic restart on failure, and request queuing that absorbs traffic spikes without dropping requests. The observability layer tracks latency percentiles (p50, p95, p99), throughput, error rates, and GPU utilisation per replica — these metrics feed alerting rules that catch degradation before it breaches SLAs. Cost-efficiency is measured as cost-per-inference, not cost-per-GPU-hour. Every optimisation in the sequence above — compilation, quantisation, batching, memory management — reduces cost-per-inference by extracting more throughput from the same hardware. As an illustrative measurement from one of our GPU inference engagements (operational measurement on that project, not a benchmarked industry rate): a model optimised from 85ms to 18ms per request served roughly 4.7× more requests per GPU-second. That is a 4.7× reduction in per-request infrastructure cost without adding a single GPU. When serving load is variable, autoscaling replica counts based on request-queue depth keeps GPU utilisation high during peaks and reduces spend during troughs. FAQ How do I diagnose where AI inference latency is being spent — model compute, memory, batching, or transport? Profile the inference pipeline end-to-end before optimising anything. A compiled, FP16-quantised baseline plus per-component timing (preprocessing, model execution, post-processing, serving overhead) is enough to separate the four candidates. Model serving overhead often accounts for a third or more of total latency in production deployments; until you have measured it, optimising the model itself is premature. What is the most efficient GPU infrastructure for low-latency inference today? The most efficient infrastructure is the smallest configuration that meets your latency SLA after the optimisation sequence above is applied — not the largest hardware that fits the budget. A100 and H100 cover most production inference workloads when paired with TensorRT compilation and INT8 quantisation; jumping to newer hardware before optimising the software stack typically buys a smaller speedup at higher cost than the equivalent software work. When does FP8 / INT8 quantisation actually reduce serving latency, and when does it only save memory? Quantisation reduces latency when the workload is memory-bandwidth-bound or when tensor-core throughput at lower precision is the constraint. For compute-bound workloads on hardware without lower-precision tensor cores, INT8 mostly saves memory without reducing wall-clock latency. Profiling for compute-bound versus memory-bound behaviour after compilation is the right gate for whether to invest in INT8 calibration. How do batching strategies (continuous, dynamic, static) trade throughput against tail latency? Static batching maximises throughput but adds latency proportional to the inter-arrival time. Dynamic batching bounds the added latency to a configurable window (typically 5–20ms) while allowing batch size to flex with demand. Continuous batching is the right default for autoregressive LLM serving because it eliminates padding waste and keeps GPU utilisation high across variable-length sequences. When should I optimise the inference path rather than scale out to more GPUs? When the unit economics of the optimisation work — engineering days versus monthly GPU spend — favour optimisation, which is almost always for workloads above modest traffic. A 4.7× per-request cost reduction (as in the illustrative engagement above) compounds across every GPU in the serving tier; adding hardware buys linear capacity at full cost. How do I measure cost-per-inference before and after optimisation to justify the engineering work? Track requests-per-GPU-second at the target SLA before and after each optimisation step. Multiply by the GPU’s hourly cost to get cost-per-inference. The before/after delta, projected over expected production traffic, is the financial case for the engineering work; we publish this comparison alongside the latency numbers at the end of every audit. What remained imperfect The optimisation sequence above takes a serving deployment from “did not meet SLA” to “meets SLA at lower cost,” but it does not remove every operational risk. Two limitations remained even after the work landed in production. The first is tail-latency variance under heterogeneous request load: p50 and p95 numbers improved cleanly, but p99 and p99.9 remained sensitive to request-mix shifts (longer prompts, larger batches arriving together) in ways the static sequence cannot pre-empt — they need an adaptive batching policy tuned per traffic pattern, which is iterative work that continues after the initial latency target is met. The second is quantisation-driven accuracy regressions on the long tail of inputs: aggregate accuracy on the validation set was preserved within tolerance, but specific input sub-distributions (operational measurement from that project) showed accuracy drift that only surfaced after weeks of production traffic. Detecting these regressions required adding a quality-monitoring layer that compares the optimised model’s outputs against the unquantised reference on a sampled subset of live traffic — infrastructure that was not in the original optimisation scope and that any team running this sequence should plan for in parallel. Inference latency optimisation is also a critical step when moving a GenAI prototype into production — prototypes tolerate multi-second response times, production systems rarely can. If your team has an inference latency target that the current deployment does not meet, a GPU Performance Audit identifies the specific bottleneck and the optimisation sequence that addresses it across the full inference stack.
