Choose a serving and batching architecture from measured latency budgets and from replays of your own production traffic, not from a GPU’s theoretical throughput. Offline throughput and throughput under a latency SLA are different quantities. A 2026 cross-layer deployment study (Singh) gives the example of a system that reaches 10,000 requests per second offline but sustains only 3,000 under a 50 ms p99 SLA, because the SLA limits the arrival rate the queue can absorb. The dominant levers on cost and latency sit below the model layer: batching policy, batch caps, queue delay, caching and admission control. That makes checking and tuning the serving path the step before buying capacity or replacing the model.

The MLPerf Inference designers drew the same conclusion: performance with no latency limit has little bearing on behaviour under one. For teams running AI infrastructure and production SaaS, that principle leads to six decisions: which batching mode fits which model, how to replay traffic, when to keep a warm tier, whether tenants share a prefix cache, what the fallback path looks like, and when speculative decoding is worth it.

Continuous batching vs dynamic batching: which one suits your model

Dynamic batching assembles requests from a queue into variable-size batches, waiting up to a configured delay for a batch to fill. Continuous batching (also called iteration-level or in-flight batching) rebuilds the batch at every decoding step: finished sequences leave and new ones join without waiting for the rest. Dynamic batching fits models that answer in a single forward pass. Continuous batching exists because autoregressive models run one iteration per generated token.

The authors of Orca (OSDI ‘22) found that request-level batching made finished requests wait for the slowest one in their batch, and made new arrivals queue behind the whole batch. Hugging Face’s explanation of the mechanism names the other parts: KV caching, chunked prefill so long prompts fit the token budget, and ragged batching so sequences of different lengths need no padding. Continuous batching with vLLM’s memory optimisations gave up to 23x the throughput of naive batching, continuous batching alone gave 8x, and median latency also fell.

The terms are not used the same way everywhere. The Hugging Face post uses “dynamic batching” for replacing a finished prompt with a waiting one. NVIDIA’s Triton documentation calls reusing batch slots as soon as they free up continuous batching. The modes are not interchangeable: throughput measured under static batching is not comparable with throughput under continuous batching, even for the same model binary on the same GPU. Check what a framework’s flag actually does before you compare numbers.

Triton’s dynamic batcher has two settings that matter. One is the maximum (or preferred) batch size. The other is max_queue_delay_microseconds, the longest a request may wait for others to join; when it runs out, the batch is sent as it is. NVIDIA’s tuning procedure starts by measuring the default configuration with Performance Analyzer. If latency is inside the budget, you raise the maximum batch size and the delay step by step until the budget is exceeded. In TensorRT-LLM the equivalent caps are max_batch_size and max_num_tokens. NVIDIA’s advice is to set them high enough to keep the GPU busy, but not so high that TTFT and TPOT SLOs are missed. To trust the winning configuration, the sweep needs a careful design, which is set out in proving dynamic batching wins with rigorous sweeps.

A fixed batch cap forces a suboptimal trade-off. A 2025 study on vLLM argued that a preset maximum batch size cannot adapt to changing arrival rates or mixed sequence lengths. With a 50 ms decoding-time SLA, its SLA-constrained dynamic batching raised capacity from 5.4 to 6.6 queries per second.

Text-to-image diffusion models behave differently: they are compute-bound. Because of that, latency rises sharply with batch size while throughput barely improves. SD-Tiny hit bottlenecks around batch size 4, where YOLOv5 handled 16. Argus serves each prompt at batch size 1 to optimise latency. A 2020 study of GPU inference as a service for high-energy physics found the opposite case: the biggest dynamic batching gains came from large models that naturally run at small batch sizes.

Model type	Batching mode to start from	Main settings	What to watch
Autoregressive LLM (chat, completion)	Continuous / in-flight batching	Max batch size, max tokens per batch, chunked prefill	TTFT and TPOT percentiles; KV-cache memory, which grows linearly with batch size and sequence length
Single-pass models (classifiers, detectors, encoders such as BERT)	Server-side dynamic batching	Max or preferred batch size, max queue delay	p99 request latency including queue time
Deadline-bound vision pipelines	Dynamic batching with a deadline-derived window	Window length, batch cap	Worst-case latency against the deadline
Diffusion text-to-image	Batch size 1 or very small batches	Instance count, approximation level	Latency per image; batching gains level off early

How do we test batching policies against a replay of realistic peak traffic before committing spend?

Record real production requests and replay them open-loop against the candidate configuration on the target hardware, at peak and beyond. Discard the warmup period. Accept the policy only if the measured percentiles stay inside the latency budget. In the 2026 load-testing study, disabling either warmup handling or recorded-traffic replay dropped accuracy to the level of generic load-testing tools. In the same study, JIT compilation, CUDA kernel caching and model weight loading distorted the first 2–5 minutes of throughput.

1. Capture real requests. Use service-mesh sidecars, proxies or application instrumentation so production requests gain no latency, and anonymise sensitive data before replay.
2. Instrument the latency split. Request latency is queue time, preprocessing, batch-wait time, model time and postprocessing; you need these separately to see whether the batching policy or the model is the cost. Triton, for example, reports queue duration as its own metric.
3. Warm up and discard. Run until compilation, kernel caching and weight loading have settled, and keep that window out of the results.
4. Replay open-loop. Schedule requests at the target rate whatever the response times are, so measured latency includes the full queueing delay instead of underestimating the tail.
5. Step towards the limit with care. Dynamic batching creates sharp cliffs where latency jumps suddenly, so reduce the step size as metrics near the SLO.
6. Report percentiles from enough samples. Report mean, p50, p95 and p99 over at least 1,000 iterations after warmup.
7. Push past the breaking point. Amazon’s Builders’ Library advises that a service not load-tested to the point where it breaks, and well beyond, should be assumed to fail in the least desirable way.
8. Compare with production and correct for hardware. Flag requests that are slower than their recorded production duration; one published replay engine flags anything over 1.5x. Same-hardware results reproduced closely in the 2026 load-testing study, while results on newer H100 hardware showed a 4.9% systematic bias.

Recorded traffic captures real production request patterns. To test the bursts you are worried about, vary its shape. HydraServe’s evaluation sampled the Azure Function trace with different coefficients of variation to vary burstiness. The 2026 load-testing study also warns that generative models need adjusting: variable output length and KV-cache pressure behave differently from fixed-output models. For a shared pool, the checks that go on top of replay are covered in validating batched, multi-tenant serving architectures before cutover.

At what cold-start latency can we drop the always-warm tier?

Size the buffer from your request arrival pattern and your measured cold-start time. Set the tolerance threshold and the cost per call from your own traffic, model and contracts.

LynxBenchAI publishes device results showing how each accelerator runs the public PyTorch stack, which gives you a guide to hardware before you measure your own cold-start path. LynxBenchAI’s results are broadly indicative for custom PyTorch models, but it does not measure your workload or your cold-start path. The first step is to check its results for the accelerators you are considering.

TechnoLynx profiles your inference workload through the Inference Cost-Cut Pack, measuring cold-start time and resource needs on each accelerator and backend.

Start by finding which stage dominates the cold start. In HydraServe’s serverless LLM setup, model fetching dominated, because of limited network bandwidth and contention between co-located containers. Container creation added a lot because the image was 8.31 GB, and moving weights onto the GPU took about 2 seconds. HydraServe, which overlaps these stages and parallelises model fetching, cut cold-start latency by 1.7x–4.7x against the baselines. Cutting cold-start time shrinks the warm buffer you need, so it usually pays to do it before costing the buffer.

Worked method: costing a warm buffer (symbols, not figures)

• Measure warm TTFT and cold-start time T_cold, broken down by stage. HydraServe set its TTFT SLO at five times the warm-request TTFT, which is one way to anchor a target.
• Replay recorded arrivals against warm buffers of size W. For each W, record which requests find a free warm instance and which wait for a cold start.
• p50(W) is the median latency across those requests: warm TTFT for requests served warm, cold-start latency for the rest.
• Cost per call(W) = W × hours in the period × GPU price per hour ÷ calls served in the period. Compare it with the same calculation for full always-warm capacity.

If some W meets the p50 target and costs less per call, move to optimised cold-start pricing. Keep always-warm only for customer segments whose contractual or observed tolerance is below the p50 cold starts can actually achieve.

Three caveats limit the result. Cold-start timings measured for one model size or accelerator should be re-measured for another. Customer tolerance is a business assumption that can shift with competition. A buffer sized for normal arrivals can also be overrun by a burst, which is why scaling-trigger timing has to be designed separately.

Should tenants share a prefix cache, or does each get its own?

Whether tenants share a prefix cache is a business decision with three inputs that must be settled together. The first is the GPU cost sharing actually saves on your tenant mix. The second is the engineering cost of a safe partitioned alternative. The third is whether any contract already requires you to prove isolation. Any one of the three can override the other two.

TechnoLynx settles this through the Inference Cost-Cut Pack, measuring the GPU-cost effect of shared versus partitioned caching on production-like traffic. The first step is an inventory of current cache-sharing behaviour, tenant traffic and isolation obligations.

The latency saving on a cache hit can be large. A 2026 study on an A100 with Qwen2.5-7B-Instruct and a 2,119-token shared prefix measured 149.6 ms TTFT without the cache and 32.8 ms with it. How much of that you get depends on your own hit rate. The MLPerf Inference designers left caching out of the benchmark because cache-hit rates vary so much between applications.

The risk comes from how caches are keyed. In multi-tenant engines, cache keys are computed from token content alone, with no link to the tenant that sent the request. Any tenant sharing a prefix hits the same blocks, which creates a timing side channel. In the same study, vLLM 0.26.0’s default 16-token block size brought one published attack’s success rate to zero. However, template-structured prompts shrink the search space whatever the block size. Engine defaults therefore do not settle the question for an application whose prompts follow a predictable template.

Full isolation has a measured price. In the SafeKV study, full isolation added 50.41% TTFT overhead on Qwen-235B-A22B and 118% on DeepSeek-R1. SafeKV, which isolates only sensitive paths, cut those overheads to 11.74% and 34.28%. Middle-ground designs also exist. PrefixShield, for example, charges each tenant group for the cache blocks it creates and only steps in when capacity is contested, instead of imposing hard partitions. Hardware partitioning with NVIDIA MIG gives each instance its own path through the memory system, from L2 cache banks to DRAM.

• A contract requires provable isolation: a shared cache is ruled out, whatever it saves. Partition, or give up reuse for those tenants.
• No such obligation: compare the measured savings with the one-time and ongoing cost (including test and audit work) of a partitioned scheme. Share only if the savings clearly win and the risk is acceptable.
• A mixed tenant base: use a hybrid, partitioning only the tenants whose contracts or sensitivity require it.

Whether a clause requires provable isolation is for whoever owns your enterprise contracts to read. The engineering contribution is a precise statement of what the cache shares, plus the tests and logs that would demonstrate separation. Two more cautions apply. Isolation risk includes security exposure even where no clause exists. And as tenants grow, one group’s new cache state can displace another’s reusable state.

What fallback path keeps critical operations running if shared-GPU serving can’t keep up?

A shared-GPU serving pool needs a dedicated fallback path built in from the start: per-line local inference or a reduced-functionality mode, kept warm and tested. Switch to it automatically when measured latency nears the production SLO, and return to the shared pool once headroom is back.

TechnoLynx designs this fallback through the Inference Cost-Cut Pack and benchmarks the shared and dedicated paths under worst-case batching load on the target hardware. The first step is framing your cycle-time requirement and failure modes.

On shared hosts, co-located tenants such as training jobs and I/O-heavy loaders create contention that makes inference latency jittery and causes SLO violations. A 2025 host-level controller combining dynamic MIG reconfiguration with PCIe-aware placement cut the SLO miss rate by about 32% and p99 latency by about 15%, at no more than 5% throughput cost. That reduces interference but does not remove it. Queueing also magnifies small slowdowns: near 90% utilisation, a 1 ms reduction in service time cuts mean response time by about 100 ms.

Base the rollback trigger on measured latency against the production SLO, not on a tuning deadline. A batching configuration that misses its target in production is a reason to fail over to the previous, versioned configuration or the dedicated path, not a reason to keep tuning live. Shed load before the queue grows: near saturation, a 5% cut in the admitted request rate can reduce p99 latency by 30% or more. Amazon’s Builders’ Library describes load shedding as rejecting excess requests so the accepted ones still finish within latency. It also recommends capping how long a request may sit in a queue. Google’s SRE guidance rejects requests by criticality as utilisation rises, and returns a degraded response rather than a failure where one is useful. One configuration trap: if load shedding fires at the same CPU threshold that drives reactive autoscaling, it can starve the autoscaler of its signal.

The fallback has costs of its own. The dedicated hardware and model must stay synchronised and warm, which costs money every hour the pool is healthy. The switch itself must not open a gap in the cycle. And if the fallback runs a lighter model, that model needs separate accuracy validation.

Sizing batch size and wait time against a production line’s cycle time

The largest batch size and wait time you can use are the values at which worst-case per-item latency (queue wait plus execution) still fits inside the cycle time with every line at peak. Measure them on the target GPU pool and check them again with one GPU removed.

TechnoLynx sizes these values through GPU Performance Engineering, designing batch size, maximum wait and queueing policy around the line’s real rhythm. The first step is capturing per-line cycle-time limits and N-1 GPU tolerance as explicit SLOs.

The calculation follows from how batching latency is made up. With batching on, an item’s latency is its execution time plus the time it queued waiting for later items to join its batch. So the batching window must never make an early item miss its deadline. DeepRT (2021) sets each category’s window to half the shortest deadline in that category. It takes execution times from an offline profile that records the worst-case run for each model, input shape and batch size. LazyBatching (2020) expresses the same constraint as slack: the SLA target minus wait time minus execution time. The same DeepRT measurements found that running several model instances at once on one GPU barely improves throughput. It raises latency and can make worst-case latency hard to estimate.

Can you simulate full twelve-line peak load before cutover? Yes. Use the replay procedure above, with all line streams merged in their real timing relationships, so that bursts that hit several lines at once show up in the test. If one GPU in a four-GPU pool fails, can the other three keep every line within cycle time? Run the same replay with one GPU removed. If cycle-time violations appear, add capacity or reduce batch size and wait time until headroom returns. For choosing the pool’s hardware, LynxBenchAI’s published device results are a guide to how each accelerator runs the public PyTorch stack, not a measurement of your inspection model.

Simulation still has limits. Real plant variation, such as camera jitter and network latency, can exceed what the simulation assumed. A batch size tuned for average load can fail under correlated bursts. And headroom should cover more than one GPU degrading at once. The load-testing study found that model size and co-location density were the strongest predictors of capacity-estimation error.

Speculative decoding on top of continuous batching: test the tail and the fallback path

Combining speculative decoding with batching can help or hurt depending on batch size, hardware, model and request mix. TurboSpec’s authors observed that adding it naively, without tuning, can degrade serving performance. A configuration is safe only once it has been benchmarked at p50, p95 and p99 across your mix of short and long completions, with a route that serves long or high-load traffic without speculation.

TechnoLynx benchmarks speculative decoding configurations through GPU Performance Engineering at p50, p95 and p99 across your request mix. The first step is replaying your short and long completions with and without speculation.

Is there a known interaction problem with SGLang when request lengths vary widely?

Published evidence on this exact combination under mixed-length workloads is thin and indirect, and it does not all point the same way. A study of batch speculative decoding measured the production frameworks: SGLang with EAGLE ran slower than non-speculative generation, and vLLM’s speculation underperformed its own baseline at high concurrency. The study also cites independent work in which EAGLE’s speedup fell from 1.73x to 1.21x as batch size grew from 1 to 128. In TurboSpec’s measurements on an L40S at 16 queries per second, even one speculative token degraded performance by 14%, from 13.7 s to 15.9 s. TurboSpec therefore switches back to standard decoding at that load. A long-context study found the opposite regime. Beyond roughly 4,000 tokens on 8xA100, loading the KV cache dominates, and speculative decoding achieved speedups that grew with batch size. Taken together, short sequences at large batch sizes are where speculation tends to lose. Long contexts can favour it, but only measurement on your own traffic settles which regime you are in.

SGLang provides the controls. Its adaptive speculative decoding adjusts the number of draft steps at runtime from a smoothed average of accepted draft length. Its documentation notes that narrow step ranges (for example [1, 2] or [1]) often do better at high batch sizes, and recommends benchmarking static step counts for each batch-size range to build a per-batch-size config. Speculation also uses extra GPU memory for the draft tree, CUDA graphs and verification buffers. If speculation helps median latency but hurts the tail for longer completions, disable it for that traffic or route requests by predicted completion length. Re-validate after each SGLang upgrade, because version changes can change how speculation and batching interact.

What numerically bounded test should gate a merge so a CPU fallback regression is caught?

Gate every change to the GPU path (fusion, precision or batching) on a test that runs a fixed input set through the CPU fallback path. The test compares outputs and latency percentiles with a stored baseline within explicit tolerances, and blocks the merge if either drifts. NVIDIA’s Polygraphy shows the shape of the output check. It takes absolute and relative tolerances (--atol, --rtol) and an error statistic (--check-error-stat set to elementwise, mean, median or max). Its documentation warns that the default tolerances suit FP32 but may not suit reduced precision. The output check matters because runtimes change numerics. In the 2026 cross-layer study, ResNet-152 reached 76.6% top-1 accuracy under TensorRT and 72.0% under ONNX Runtime on the same hardware. Tolerances need care in both directions. Set too loose, they let real regressions through; set too tight, they block valid optimisations. Choose them per output against what the downstream decision can tolerate. Building the shadow-traffic and reconciliation harness around such a gate is covered in validating serving optimisations with shadow traffic and regression harnesses.

Adopting a proven serving stack instead of writing your own batcher

Continuous batching and paged KV-cache management are now standard production infrastructure, not experimental techniques. That makes adopting an established inference server the default, and writing your own batching scheduler the exception that needs a reason. Triton lets you configure scheduling and batching algorithms per model, and publishes the latency, throughput and health metrics a Kubernetes deployment needs. Its TensorRT-LLM backend adds in-flight batching, paged attention and selectable scheduler policies. For embedding models, which answer in one forward pass, Triton-style dynamic batching is the relevant mode, not LLM-style continuous batching.

The stacks differ, so pick one by workload rather than reputation. A comparison found vLLM gave 2–24x higher throughput than Hugging Face TGI depending on concurrency and model size, while TGI gave 1.3–2x lower TTFT at low concurrency. The authors suggest routing interactive and batch traffic to different engines where both matter. Offline advantages also shrink online. The LMSYS team reported up to 3.1x SGLang throughput over vLLM on Llama-70B offline, an advantage that narrows under online serving with latency SLAs.

On engineering cost, we know of no published comparison between adopting vLLM or TensorRT-LLM and writing a batcher. Our judgement is that writing your own scheduler is rarely worth it unless a trial of the existing stacks fails to meet your isolation, latency or cost needs. Tenant isolation is the part you cannot take on trust. Hardware partitioning such as MIG gives each instance isolated memory paths and fault isolation. Test both under representative load. Managed inference endpoints take more of the work off your hands but bring lock-in and recurring cost. Published benchmarks may use synthetic workloads that do not reflect your production traffic.

Who can help us choose and validate a serving architecture before we commit GPU spend?

TechnoLynx profiles your serving path to find which change holds your latency budget, before you buy more GPUs or replace a model. Whether to do this in-house depends on what you already have. A specialist is worth it when the decision is expensive to reverse, such as a GPU contract, a plant cutover or a multi-tenant isolation commitment. Whoever you hire, ask for their replay method, the percentiles they report and how they test failover, not a peak-throughput chart. TechnoLynx packages this work as the Inference Cost-Cut Pack, which starts with a profile of the production stack.

There are realistic alternatives. Teams that would rather not run inference infrastructure at all can use a managed endpoint and accept the lock-in.

A validated configuration has a shelf life

A replay result describes the traffic you recorded, not the traffic you will have next quarter. Changes in arrival rate, sequence length or batch-size distribution move the queue’s operating point. The 2026 cross-layer study notes that a configuration that was best at 50% utilisation may not be at 80%, where queue sensitivity is 6.25x larger. The 2026 load-testing study recommends building load testing into deployment pipelines for ongoing benefit. The question left to settle is not which configuration wins today. It is which change in traffic, model or engine version triggers another replay, and who on your team notices when that change happens.