A dynamic batching benchmark shows a policy wins only when the win holds across a sweep of batch sizes and sequence-length distributions. One good operating point does not show how the others behave, because the ranking of frameworks can reverse between batch size 1 and batch size 32. A common anti-pattern in batching comparisons is the comparison, not the measurement: a heavily tuned new system set against baselines left at default parameters.
Continuous batching vs dynamic batching: which baselines should the benchmark include?
Static batching waits for a fixed number of requests before it launches a forward pass. Dynamic batching builds batches of varying size from a request queue. Continuous batching adds and removes individual requests from the in-flight batch at every decoding step. Dynamic batching is set with two parameters: a maximum batch size the server tries to fill, and a window it will wait after the first request before running a partial batch. With a batch size of 16 and a 100 ms window, the server runs a full batch if 15 more requests arrive within 100 ms, and otherwise runs a partial batch when the window closes.
These are not interchangeable. Throughput under static batching at batch size 32 is a different quantity from throughput under continuous batching, even with the same model binary on the same GPU. One reason is that the time spent waiting for a batch to fill is part of request latency, and it is zero under continuous batching.
That decides the baseline set. Baseten’s guidance is to use continuous batching for most LLM deployments and dynamic batching for most other models. In the SSJF evaluation, continuous batching always gave lower job completion time and higher throughput than dynamic batching. The multi-bin batching study calls native continuous batching “a substantially stronger and more practically relevant baseline” than fixed static batching. Its best estimated-length variant came out 150.1% above static batching on the Dolly workload and still 43.0% below continuous batching. In vLLM and TensorRT-LLM the native scheduler already does continuous batching. For LLM serving, include continuous batching as a baseline. For image or vision models, the tuned static configuration and the framework’s own dynamic batcher are the baselines to beat.
What sweep should a dynamic batching benchmark run so one lucky configuration cannot carry the win?
The sweep should cover a grid of batch size (or concurrency) against sequence-length distribution, including the lengths you actually see in production. Adopt the new policy only if it matches or beats the tuned baselines on both throughput and p99 latency across the whole grid. If it wins only at isolated points, treat the result as one that does not generalise, and investigate before claiming an improvement. LynxBenchAI comes in one step earlier, when you choose the hardware the sweep will run on. Its published device results show how each device runs the public PyTorch stack that most teams’ models sit on, so they are broadly indicative for custom PyTorch models too, but LynxBenchAI does not run or measure your batching policy or your own workload.
The grid has to be wide because both axes change throughput by large factors. LLM-Inference-Bench (2024) measured throughput at batch size 64 that was 26.6 times higher than at batch size 1 for a 2048-token length on an A100. It also measured a 14.6-times throughput difference between an {input, output} shape of {1024, 128} and one of {128, 1024}. A policy tuned at one corner of that space has not been shown to work anywhere else. The optimal batch size depends on model architecture, hardware and application requirements, so benchmarking should cover a range of batch sizes, not one setting.
The grid does not have to be exhaustive. The FMwork study (2025) found that output size 128 with batch sizes at powers of two kept a 3.4% global relative difference from the full sweep on Llama 3.1 8B. NVIDIA’s guidance is to sweep concurrency from 1 to a value not much greater than the maximum batch size, because throughput generally saturates there while latency keeps rising. Input and output lengths should move as paired workload shapes instead of being cross-multiplied, since a full grid runs meaningless combinations such as a 2048-token input with a 64-token output.
Sequence lengths should reflect the input and output distribution of your own deployment. An anti-pattern is evaluating on a single-source dataset or a synthetic workload with limited input lengths. The authors of “On Evaluating Performance of LLM Inference Serving” (2025) cite Vidur’s finding that optimising for one workload type can cost up to 2× in performance on another. Uniform synthetic lengths are especially risky for batching work, because in Anyscale’s runs, length variance is what separated the batching policies. Published sweeps model this explicitly: DeepSpeed-FastGen used normal distributions with 30% variance on prompt and generation lengths, and Anyscale sampled generation lengths from an exponential distribution. Load level is an axis too. A change in arrival rate or length distribution moves the queuing operating point, so a configuration that is optimal at one utilisation can be suboptimal at another.
How hard should the static baseline be tuned before the comparison counts?
Tune the baseline as carefully as the proposed policy, not leaving it at defaults or arbitrary parameters. Seemingly minor configuration differences can cause significant performance variation, and comparing a tuned system against baselines with default or arbitrary parameters invalidates the result. The same paper’s checklist asks whether all performance-critical parameters are documented for both the proposed system and the baselines, and whether the baseline parameters were tuned for the evaluation setup.
NVIDIA’s Triton documentation lays out the procedure. First measure latency and throughput with the default dynamic batcher configuration. If latency stays within budget, raise the maximum batch size or set a non-zero batch delay, increasing the delay until the latency budget is exceeded, to see what throughput it buys. Triton’s guidance is explicit that the benefit of the dynamic batcher is model-specific and has to be found by experiment. Some published comparisons test baselines at several batch sizes. Sarathi-Serve evaluated vLLM at maximum batch sizes of 32, 64 and 128. The SMDP batching study compared its policy against static batch sizes of 8, 16 and 32 across load levels, and found the batch-size-8 static policy could not keep the system stable at normalised load of 0.8 and above.
Implementation effort also belongs inside fairness. NanoFlow reported a 1.91× throughput gain, and its own ablations showed that system-level engineering produced most of it, with the core algorithmic idea contributing 16% when isolated. If your new batching policy also ships with a faster tokeniser or a leaner request path, a reviewer will rightly ask which part did the work. We hold that best practices only make sense in context, and that context comes from profiling.
Worked example: how the baseline choice moves the headline
Anyscale’s continuous batching benchmark compared static baselines of very different strength. It used 1,000 requests of 512 input tokens, with generation lengths drawn from an exponential distribution with a mean of 128 and capped at 32, 128, 512 and 1536 tokens across four runs of rising length variance. At low variance, the static and naive continuous batchers performed about the same. As variance rose, naive static batching (Hugging Face Pipelines) fell to 81 tokens/s, while FasterTransformer, an optimised static implementation, nearly kept pace with naive continuous batching until the 1536-token cap. Anyscale included FasterTransformer deliberately as “a more competitive baseline” than the unoptimised implementation.
Three readings follow from the same data. Against Hugging Face Pipelines, the gap is largest. Against FasterTransformer, the gap is smaller. At the low-variance end, the gap nearly disappears. The same continuous batcher, compared three ways, gives three different headlines, and only the version against the strongest baseline across all four variance levels would convince a reviewer.
What p50, p95 and p99 each tell you in a batching benchmark
p50 is the typical request, and p95 and p99 are the tail where batch-formation wait and queueing show up first. Serving systems trade one part of the distribution against another, so several percentiles are needed. Before any of them means anything, the benchmark has to say which latency it measured. The “Optimizing AI Inference Across the Deployment Stack” survey (2026) lists four quantities commonly reported as latency: kernel time, model latency, request latency including batching wait, and service latency including the network. These can differ by an order of magnitude for the same model on the same hardware. For a batching policy, request latency is the honest choice, because the batching wait is part of what the policy changes.
The same survey’s minimum reporting standard includes mean, p50, p95 and p99 latency from at least 1,000 iterations after warmup, together with GPU model, driver, CUDA toolkit, framework version, precision and batch size. Tail percentiles need many more queries. In the MLPerf Inference design paper (Reddi et al.), a 99th-percentile guarantee at their chosen confidence and margin needed 262,742 queries, rounded up to 270K. A run with far fewer queries than that cannot support a p99 claim at that confidence.
Throughput belongs next to the percentiles because the two trade against each other. MLPerf found that every system delivered less throughput in the latency-bounded server scenario than in the offline scenario, with losses ranging from about 3% to 50% depending on the system. Its authors concluded that a performance comparison with unconstrained latency “has little bearing on a latency-constrained scenario”. The useful single number is capacity, the maximum request rate a system sustains while meeting its latency targets, or effective throughput, the throughput of requests that meet the SLA. The SMDP batching study notes that SLOs bound a latency percentile, not the average.
Did the new policy buy its p99 with a worse p50 or less throughput?
To check, compare p50, p99 and throughput at every grid cell and load level, looking for cells where one improves while another worsens. Published systems show this trade happening. Comparing Sarathi-Serve and vLLM on Arxiv Summarization, Sarathi-Serve had better P99 latency but higher latency around P90. In the Metron paper’s illustrative comparison, vLLM outperformed Sarathi-Serve between the 80th and 98th percentiles but had a longer one-second tail TBT. In the opposite direction, DeepSpeed-FastGen and vLLM had similar P50 generation latencies, while DeepSpeed-FastGen cut P95 by 3.7 times. A median-only report would have shown no difference.
Time per output token and normalised latency divide by the number of decode tokens, which can make a ten-second generation stall numerically small. The evaluation checklist asks that raw scheduling delay and token-generation stalls be examined alongside any normalised figure.
One way to show the trade is a throughput-versus-p99 frontier per workload shape. A configuration is Pareto-optimal if no other configuration in the sweep is better or equal on both throughput and p99. A new-policy point that no baseline point beats or equals on both throughput and p99 is Pareto-optimal in that sweep. If the frontiers cross, you have a trade that depends on which SLO you serve. Capacity under a stated SLA sums the comparison up in one number. The memory-aware dynamic batching paper (2025) reported a capacity increase from 5.4 to 6.6 queries per second under a 50 ms decoding SLA, against a static batch size as configured in vLLM.
Repeat runs and pin the stack before calling a difference real
A single run gives one noisy number. NVIDIA’s AIPerf documentation notes that without repeated runs “you can’t tell a 3% throughput improvement from random jitter”. The same documentation treats a run-to-run coefficient of variation below 5% as excellent repeatability, and non-overlapping confidence intervals as statistically meaningful. vLLM’s sweep tooling runs each parameter combination three times by default, and MLPerf requires five server-scenario runs, reporting the minimum.
Use the same seed so workloads are identical across runs, add cooldowns so thermals and caches settle, and use the same sweep values and run counts for both variants. FMwork discarded about the first 20% of iterations as warmup. Record software versions too: the deployment-stack survey reports that kernel library updates can shift performance by 10–30% between minor versions, so a baseline measured last month on an older CUDA toolkit is not a baseline.
Reading the sweep: adopt, investigate or reject
| What the sweep shows | What it means | What to do |
|---|---|---|
| New policy matches or beats tuned static and continuous baselines on throughput and p99 in every cell | The win generalises across the measured space | Adopt, then validate on live traffic |
| Wins only in a band of batch sizes or lengths | Result does not generalise | Check whether production traffic sits in that band before claiming anything |
| Lower p99, but higher p50 or lower throughput somewhere | A trade, not a win | Decide against the SLO that binds; report the trade |
| Confidence intervals overlap | No measured difference | Add runs or close the claim |
| Wins only against a default-configured baseline | Strawman comparison | Retune the baseline and rerun |
Before rollout, validating serving optimisations with shadow traffic and regression harnesses is the next step.
Testing batching policies against an SDXL launch-day traffic pattern
To test batching policies for an SDXL service before committing infrastructure spend, replay a simulated launch-day traffic profile through an existing benchmark harness. Then pick the policy with the lowest GPU-hours per image that keeps p99 latency under the SLA, with images per minute as the capacity check. TechnoLynx can run this comparison on your serving stack. TechnoLynx’s first step is to fix the launch-day traffic profile and the p99 target with you, before any harness is configured, under the Inference Cost-Cut Pack.
Reusing an existing harness, extended where needed, is the starting point. Tools such as vLLM’s vllm bench sweep and NVIDIA’s AIPerf already run parameter combinations, repeat runs and explore the latency-throughput trade. If the harness supports custom traffic shaping and your model backend, configure it. If it does not, write a thin wrapper around it rather than a new harness. For diffusion serving, check the backend first: in vLLM-Omni, step-wise continuous batching for diffusion is still marked experimental. Replaying launch-day traffic is closer to load testing than to model benchmarking, and NVIDIA treats the two as distinct activities that expose different problems.
wrk2 sends load at a constant rate and measures latency from when each request should have been sent, avoiding coordinated omission; in its author’s example, a 1.4 s stall made the corrected p99 200 times larger.
SDXL needs its own care because batching depends on request shape. vLLM-Omni batches only requests with compatible keys (shape, guidance, output count, LoRA identity), and its maximum-wait setting trades first-request latency for coalescing bursts. In PatchedServe’s measurements on H100, batches made entirely of high-resolution requests ran up to 68% slower than low-resolution ones, and the authors note that all combinations need profiling. Their baseline gained from batching only when requests shared a resolution. The traffic profile therefore has to carry the real resolution and parameter mix, not just a request rate. Check output parity too: vLLM-Omni’s design asks for validated output parity under the same seed and sampling parameters before a pipeline is marked as supporting step execution.
On metrics, SwiftDiffusion reported throughput as images generated per minute of GPU time. GPU-hours per image is the reciprocal: 1 ÷ (60 × images per GPU-minute). Two limits apply. Simulated or scaled-down traffic may miss the burst correlations of a real launch, and autoscaling interacts with the batching policy, so the two have to be tested together. For that coupling, see Designing batching and autoscaling strategies for inference serving.
Who can benchmark our batching policies against our own traffic before we commit spend?
TechnoLynx builds and runs that benchmark for your serving stack: a sweep and traffic replay that shows whether a new batching policy beats properly tuned baselines on your models, your hardware and your request mix. For testing against realistic traffic, TechnoLynx first defines the traffic profile and SLA targets with you, then configures an existing batching harness, or extends it, to replay that profile across the candidate policies. On harness reuse, TechnoLynx integrates the harness with your serving stack and target backend instead of building a new one. On metrics, TechnoLynx validates results against p99 latency, throughput and cost per image, iterates on policy parameters, and supports deployment with monitoring hooks for images per minute, p99 and cost per image. The engagement route is the Inference Cost-Cut Pack.
This work suits teams that serve their own models and need a defensible answer before scaling spend. It does not suit buyers who need a generic hardware ranking, which is what published device results are for, or who need a platform migration decided. Teams that already run vLLM’s sweep tooling or AIPerf with multi-run statistics in CI, and have someone who can tune the baselines without a stake in the result, have a credible in-house route. Whoever does it, ask how the baselines will be tuned before you ask how fast the new policy is.
A batching win is a claim about the whole stack
Batching gains depend on everything around the batcher. Operator fusion changes memory access patterns in ways that alter how well system-level batching works, so no single-layer optimisation can be judged in isolation. The deployment-stack survey describes a 3× speedup, measured with batch size held fixed, that disappears in production once the compiler version differs and the batch size is dynamic. A dynamic batching benchmark is exposed to the same failure in reverse: a policy tuned against one kernel library and one length distribution can lose its advantage after the next framework release.
So the last step before claiming a win is to profile where the time goes in both the baseline and the new policy: queueing, batch-formation wait, model time and post-processing. That tells you whether the gain comes from the policy or from an incidental change it carried along. This kind of measurement-led tuning is the core of our GPU performance engineering work, and it is why we profile before we optimise. The open question a benchmark cannot settle for you is which latency percentile your users and your SLA actually punish. Decide that first, and the sweep tells you whether the new policy is worth deploying.
FAQ
How do you calculate P99 latency?
Record the latency of every request in the run, sort the values, and take the value below which 99% of them fall. The estimate is only as good as the sample: MLPerf’s design needed roughly 270K queries for a 99th-percentile guarantee at its chosen confidence.
Is batch inference faster?
In LLM-Inference-Bench, throughput rose with batch size, partly because fixed costs such as kernel launches are paid once per batch. It can make individual requests slower, through the wait to form a batch and the longer time to process a bigger one.
