Average requests per second does not size an inference fleet. Peak concurrency does, and the peaking factor (the ratio of peak rate to average rate) is the single most important unknown in the sizing exercise. Request rate also hides what each request costs: token distribution, the streaming share, the model mix and the region all change the footprint at the same request rate. The reliable method is to characterise the workload mix, then size from per-replica throughput measured on your own infrastructure under a representative load rather than assumed.
TechnoLynx treats that measurement layer as the starting point for capacity and autoscaling decisions on AI infrastructure and production SaaS platforms.
Why break request rate down by tenant, region and time of week?
An aggregate request-rate panel blends a few heavy, bursty clients with many stable ones. In ServeGen’s characterisation of a production cloud LLM serving service (NSDI 2026), the top 29 of 2,412 clients sent 90% of requests, and most bursts and length shifts were caused by those top clients while most other clients stayed stable and predictable.
TechnoLynx builds this segmentation layer as part of the Production AI Monitoring Harness: it first agrees which segments drive your scaling and capacity decisions, then adds tenant, region and time-of-week labels and token-length histograms to your Prometheus and Grafana stack.
Time of week is the second axis. In BurstGPT, a 121-day trace of Azure OpenAI GPT services, conversation traffic peaked on weekdays and working hours and dropped at night and at weekends, while API traffic followed an aperiodic, bursty pattern over the same intervals. DynamoLLM’s week of production traces, measured in tokens per second, showed daily coding peaks with much lower weekend load, and conversation load peaking at 1.7 times its average and 3.3 times its valley. Hour-of-day and weekday or weekend labels help tell a predictable peak from a genuine burst.
Adding the segments to an existing Prometheus and Grafana setup
Once the labels exist on the series, Grafana’s Filter and Group by feature splits an aggregated query such as sum(your_metric) by whichever dimensions you select, dashboard-wide, without a variable per dimension. Hour-of-day and day-of-week do not need to be stored as labels at all, because PromQL’s hour() and day_of_week() functions derive them at query time. Prometheus recording rules then evaluate the expensive segmented queries at a fixed interval and store the result as a new series.
The cost to watch is cardinality. Tenant attributes multiply against model attributes, and in one AI gateway’s sizing guidance the response-model label is the largest contributor because providers report dated model builds that change without notice. Keeping an explicit tenant label for the clients that dominate load and folding the long tail into a single bucket keeps the series count bounded while preserving the segments that move capacity.
For token length, a defensible starting scheme is DynamoLLM’s nine buckets (short, medium and long input crossed with short, medium and long output), with thresholds set at the 33rd, 66th and 100th percentiles of a production trace. Set the percentiles from your own trace, not theirs.
| Dimension | What it reveals for sizing | Typical source |
|---|---|---|
| Tenant or client | Skewed load and the origin of most bursts | Gateway or router label |
| Region | Each region must be sized for its own share of peak | Deployment label |
| Hour of day, weekday or weekend | Periodic conversation peaks versus aperiodic API bursts | Derived at query time |
| Input and output token buckets | Different compute and memory demand per request type | Serving-engine histograms |
| Streaming share | Streaming holds a connection for the whole generation | Gateway label |
| Burstiness | In ServeGen’s production trace, short-term arrivals were often bursty, with a coefficient of variation above 1 | Computed from arrival counts |
Burstiness needs a deliberate window. BurstGPT counted how often request volume doubled from one five-minute window to the next, while FineServe measured both five-minute variation and per-second fluctuation and found small dense models jittery at second level but MoE models dominated by large-amplitude bursts. The window you pick should match how fast your autoscaler can act.
How do you get vLLM or TGI to report token counts per request without slowing it down?
vLLM already exports token counts on its Prometheus /metrics endpoint: vllm:request_prompt_tokens and vllm:request_generation_tokens histograms, alongside the vllm:prompt_tokens_total and vllm:generation_tokens_total counters. Text Generation Inference exposes a per-request tgi_request_input_length histogram on its /metrics endpoint.
TechnoLynx wires these server-side histograms into the Production AI Monitoring Harness, adds tenant and region labels at the router, checks the counts against the model’s tokenizer, and benchmarks the added latency before anything reaches production.
llmperf, for example, deliberately counts tokens with the Llama tokenizer whatever API it tests, which keeps prompts consistent across APIs but means its counts are not the target model’s counts. On TGI, the documented per-request histogram is input length.
vLLM can also return per-request timing metrics in the API response, which is useful for billing, SLA monitoring and per-request latency analysis. Its documentation warns that at high concurrency this can add non-negligible CPU overhead and says to benchmark your workload before enabling it in production. Those metrics are also suppressed for requests with n > 1.
Payload size belongs in the capacity model because input and output lengths strongly affect batching and computational load. In Microsoft’s Splitwise traces from two Azure services, the coding service had a median prompt of 1,500 tokens and a median output of 13 tokens, while the conversation service had medians of 1,020 and 129. Time to first token grew almost linearly with prompt size, and the provisioned cluster differed accordingly: more prompt machines for coding, more token machines for conversation.
Is a 40x spike a one-off or a pattern you can plan around?
Separate predictable peaks from genuine bursts before deciding: a known peak is one you can pre-warm for, while a genuine burst needs deliberately overestimated resources. The BASE autoscaling work argues that predictable peaks should not be mistaken for bursts, and flags only deviations outside the forecast confidence interval as bursts.
There is no universally accepted formal definition of a burst, so detection relies on statistical properties of the time series rather than a fixed threshold. Measures the BASE work lists include the auto-correlation function for repeating patterns, the index of dispersion for over-dispersion, the Hurst exponent for long-range persistence, and deviation from a moving average for local transients.
For event-driven spikes (a severe-weather alert, a product launch, a news break), the practical test is to align the request time series with the event timestamps and compare ramp speed, peak multiplier and geographic spread across every past event. If the profile repeats, model it as a scenario; if it varies widely from event to event, plan with wider safety margins and an event-triggered scale-out rather than a forecast. The peak-to-baseline ratio sets the bar either way: a peak ten times baseline means the autoscaling has to absorb that multiplier within minutes.
Be honest about what the evidence shows here. BASE’s own evaluation included 40x demand bursts that briefly pushed response time above the SLO before converging, but no published dataset we can point to shows that a given event type recurs with a consistent ramp. That recurrence has to come from your own logs, and it is only as precise as your event timestamps. Reactive autoscalers such as the Kubernetes Horizontal Pod Autoscaler lag in bursty scenarios because they wait for delayed metric feedback, so known events are pre-warmed in advance.
How do you replay production traces at the concurrency sales is projecting?
Replay the trace’s own arrival times and token lengths, scale the timeline to the projected rate, keep the per-client mix explicit, and use the result as a gate. TechnoLynx builds this replay harness on your serving stack and hardware: it agrees the target concurrency and pass/fail SLA criteria with you, then defines the scaling method and request mix against your worst-case production segments before running it.
A load test built from naive workloads under-provisions, because real bursts come from a few top clients. In ServeGen’s provisioning case study, at a target P99 time to first token of 2.25 s and time between tokens of 0.5 s, the realistic per-client workload led to 25 instances (4% over the number actually needed) while the naive generator led to 12, a 50% under-provision. The naive workloads missed the correlation between arrival rate and token distribution that bursty top clients create. BurstGPT showed the same at equal request rate: its burstier concurrency produced higher latency and resource use than a more uniform workload.
Worked procedure: from trace to gate
- Export per-request arrival time, input tokens and output tokens; the prompt text is not needed, because the performance metrics depend only on input and output sizes.
- Segment the trace by client and pick the busiest realistic window rather than an average day.
- Scale arrival times to the target rate: BurstGPT multiplies timestamps by a factor so the scaled trace hits a given average rate, and GuideLLM’s
time_scaleof 0.5 halves the intervals between trace events. - If projected growth is a new large tenant rather than more of today’s mix, add it as its own client stream, as ServeGen composes workloads per client, instead of compressing everyone uniformly.
- Write down the mix assumptions (token distributions, request classes, cache-hit rates) and make each at least as demanding as your worst production segment.
- Run on the real stack and judge against percentile SLOs: Splitwise required P50, P90 and P99 targets for time to first token, time between tokens and end-to-end latency all to hold.
- Attribute any violation to a pipeline stage with per-request tracing across the serving pipeline before deciding whether to re-provision, reconfigure or renegotiate the SLA.
Open tools cover the generator side: GuideLLM replays JSONL, CSV or Parquet traces using each row’s timestamp and token lengths, and NVIDIA’s GenAI-Perf generates synthetic prompts with a configurable token mean and standard deviation and offers a fixed-schedule trace mode. The realistic route is one of these tools fed from your own trace, or a small generator fitted per client.
GenAI-Perf’s documentation recommends building a traffic-footprint collector into your inference service, because popular inference servers do not record replayable traffic themselves. Where you lack history, Microsoft publishes Azure LLM inference traces with input and output token counts from November 2023 and a one-week sample from May 2024. Fitted distributions are where sources disagree: BurstGPT models request length as Zipf, while ServeGen fits inputs with a Pareto and log-normal mixture and FineServe, with 1.48 billion requests, finds log-normal captures the tails better than Zipf. Fit your own data before trusting either.
Two limits stay with any replay. Scaling history to a higher concurrency is an extrapolation that breaks if new features or customer segments change the mix, and synthetic load does not reproduce real user behaviour such as retry storms.
Which signal should drive autoscaling once traffic is characterised?
Scale on request rate or queue depth measured against the saturation point you found in replay, rather than on CPU utilisation. CPU alone may not reflect user demand, and request rate or queue depth gives more responsive scaling.
TechnoLynx sets these triggers as part of the Inference Cost-Cut Pack.
vLLM’s design documentation asks how to identify the saturation point, the inflection where a higher request rate stops buying throughput and starts adding latency, so you can autoscale effectively. The raw signals already exist as vllm:num_requests_waiting and TGI’s tgi_request_queue_duration histogram. Headroom follows from queueing theory: under random arrivals in an M/M/1 queue, running at 0.25, 0.50, 0.75 and 0.90 of maximum throughput gives normalised latencies of 1.33, 2, 4 and 10.
For turning those signals into a scale-out and scale-in policy, see designing autoscaling policies for GPU inference fleets.
How should we size GPU capacity so we neither pay for idle GPUs nor miss the peak?
Size from the peak concurrency in your replayed workload divided by per-replica throughput measured on the target hardware under that workload, then add headroom for failover, rolling upgrades and traffic above the modelled peak. TechnoLynx runs these sizing experiments on your target GPUs within the Inference Cost-Cut Pack, comparing shared, dedicated and hybrid layouts on latency, accuracy and cost per unit before recommending a sizing and autoscaling policy.
Tune the serving path before reaching for a different model: batching, caching, routing, quantisation and serving configuration usually move cost and latency more cheaply than replacing the model, and profiling decides whether replacement is needed at all. The characterised traffic is what makes that tuning concrete, because different request types want different configurations: DynamoLLM splits capacity into per-request-type pools and resizes them as the distribution shifts.
Topology follows from the measured curves. We would choose shared, dedicated or hybrid capacity by whichever meets latency and accuracy targets at the lowest cost in those experiments.
Tracing comes before tuning the policy
You cannot justify a scaling decision or an SLA to a tenant without request-level visibility into which tenant or feature consumes the GPU time, so add tracing before tuning autoscaling. Trace volume scales with request rate and may need tail sampling at peak, whereas metrics are not sampled and remain the complete record. The same per-tenant data feeds cost-per-request modelling under bursty, shared load.
Session-based and real-time audio workloads
Bursty session workloads, such as live courtroom transcription with speech bursts, silences and concurrent speakers, are not covered by the traces cited on this page. Count concurrent sessions at peak, measure each session’s compute duty cycle, and treat edge-versus-remote placement as a network question that GPU-only testing cannot settle.
Who can characterise our inference traffic and size capacity from it?
TechnoLynx characterises production inference traffic and turns it into capacity and autoscaling decisions. For the measurement side, the Production AI Monitoring Harness adds tenant, region, time-of-week and token-length segmentation to your existing Prometheus and Grafana stack. The first step is framing: confirming which segments, SLOs and target concurrency drive your decisions.
Building in-house makes sense if you already have per-tenant telemetry and SRE time to maintain a replay harness; a specialist earns their place when the telemetry is missing or the replay has to be trusted for a release gate. The engagement does not suit a team that wants a per-replica throughput figure without measurement, because that figure depends on instance type, workload mix and software version and has to be measured. For comparing candidate GPUs before you own them, LynxBenchAI’s published results show how each device runs the public PyTorch stack your models run on, so they are broadly indicative for custom PyTorch models; they do not measure your own workload. Teams that only need dashboards can get far with Grafana’s group-by and recording rules alone.
Characterisation is a recurring measurement, not a one-off study
The traffic you sized against will not hold still. ServeGen saw average input length shift by up to 1.63 times and output length by up to 1.46 times, independently of each other, which moves load between prefill and decode. FineServe separates exactly this long-term drift, which can invalidate stationary workload assumptions, from second-level burstiness. Re-run the segmentation and the replay whenever a large tenant joins, a model changes or the mix moves, and treat the last capacity plan as a hypothesis until the new trace confirms it.
FAQ
What is token latency?
vLLM exports time to first token and inter-token latency (the gap between successive streamed outputs) as Prometheus histograms, along with per-request time per output token.
