We treat traffic profiling for capacity planning on an LLM inference fleet as reducing historical request-rate, token-volume and tail-latency data to a baseline, a peak-to-baseline multiplier for launch or promotional days, and the traffic tier at which p99 latency breaks. You get the fleet size by dividing projected peak load by what one instance can sustain at a target utilisation, which leaves headroom for spikes, deploys and a dead node. The same baseline then does a second job. Compare it against post-onboarding data with a change-point test and you can show leadership whether a new customer’s traffic is a structural shift in mean or ordinary variation, with a confidence level attached.
LynxBenchAI benchmarks throughput, latency and memory across accelerators, which narrows the hardware shortlist before your own load tests set the per-instance capacity in the plan; the first step is to list the accelerators you are weighing for the fleet.
What is traffic profiling for capacity planning on an LLM inference fleet?
The table below summarises the profile. The sections after it explain how to measure each row.
| Quantity | Built from | What it sets |
|---|---|---|
| Baseline | Per-second request and token rates over ordinary weeks, from monitoring history | The reference every later comparison uses, including the board’s before/after view |
| Peak multiplier | Ratio of the busiest hour to the daily average, measured separately for past launches and promotions | Projected peak load |
| p99 breakpoint | p99 latency against throughput per instance, at the inflection that defines true maximum throughput | The per-instance capacity limit in the plan |
| Change-point result | Baseline plus post-onboarding series, tested for a shift in mean with a confidence level and a confidence interval for its timing | Whether new traffic is structural and should drive a provisioning decision |
The data an LLM traffic profile needs, and why request counts are not enough
Production LLM traffic does not arrive as a constant-rate Poisson stream. It comes in sudden spikes followed by lulls, driven by how people use chatbots and content tools. Alibaba’s ServeGen characterisation (NSDI 2026) found that short-term request arrivals are often bursty, with a coefficient of variation above 1. A one-year production trace covering April 2025 to April 2026 found the same for most models, along with positive autocorrelation, meaning busy periods tend to follow busy periods instead of fluctuating at random.
Request counts alone also miss the token work. In that one-year trace, request rate stayed roughly stationary over two months while the number of active instances fell, closely tracking hourly token volume: replica count followed token work, not request arrivals. Input length mainly drives time to first token (TTFT), and output length drives end-to-end duration. In a production trace analysed for the Vertumnus serving system (2026), requests above 32K input tokens were 4% to 8% of arrivals but 32% to 54% of the day’s input tokens.
The aggregate also hides which clients cause the movement. ServeGen found that the top 29 of 2,412 clients generated 90% of requests. Most bursts and shifts in the overall workload came from a few top clients, while most clients behaved in a stable, predictable way. For that reason we would give the profile a per-tenant view of rate, token length and burstiness.
The minimum data set for the profile is:
- request rate per second, by client or tenant;
- input and output token rates, kept separate because they load prefill and decode differently;
- p99 TTFT and p99 end-to-end latency, since averages hide the tail;
- a history longer than a short window, because LLM workloads are non-stationary over yearly timescales and short measurement windows are insufficient for capacity planning.
Building the pre-onboarding baseline from Prometheus history
Build the baseline from counters you already scrape. Compute per-second rates over windows that span ordinary weeks and past promotions, then pre-compute the common series with recording rules so the comparison stays cheap to rerun. Prometheus rate() returns the per-second average rate of increase across a range vector, and the range should be at least four times the scrape interval. With a 15-second scrape interval, that means at least [1m]. rate() extrapolates between the first and last points in its range, so a sharp change just after the start of the window can be masked by the older points.
These queries come from published Prometheus guidance and use the generic http_requests_total counter. Swap in the request and token counters your serving stack exports:
# Short-term vs long-term rate: flags when the 5m rate is 20% above the 1h rate
rate(http_requests_total[5m]) / rate(http_requests_total[1h]) > 1.2
# Current rate vs the 7-day average (a peak-to-average view)
rate(http_requests_total[5m])
/ avg_over_time(rate(http_requests_total[5m])[7d:1h])
# Standard deviations from the 24h mean (z-score)
(
rate(http_requests_total[5m])
- avg_over_time(rate(http_requests_total[5m])[24h:5m])
)
/ stddev_over_time(rate(http_requests_total[5m])[24h:5m])
We would fix the baseline window’s end date before the new customer goes live and record it. That way the later comparison cannot be contaminated by onboarding traffic or argued about after the fact. The z-score query is useful as an alarm, but it judges one point at a time. That limits it as evidence, as the change-point section explains.
How much does traffic multiply on a launch day?
A published peak multiplier is an assumption, not a constant. The figure you can defend is the ratio of your busiest hour to the daily average, taken from your own metrics and raised for sales, launches and similar events. A normal day already has a diurnal shape. ServeGen observed diurnal swings, with load peaking in the afternoon and dropping sharply in the early morning. In the Vertumnus production trace, the request arrival rate changed by more than threefold between off-peak and peak. The shape also depends on the type of service. BurstGPT’s Azure OpenAI traces showed periodic weekday and working-hour patterns for ChatGPT and GPT-4 conversation services, while API traffic was aperiodic and bursty.
Launch days break the pattern. Google’s SRE book notes that launch traffic can surge and then subside within days, that its mix can differ enough from steady state to invalidate load-test results, and that some Google products saw launch spikes up to 15 times higher than first estimated. Capacity planning has to account for organic growth and also for inorganic demand from launches and campaigns.
Two limits apply here. A multiplier drawn from past promotions may not carry over to a new kind of product launch, and history cannot show patterns from a customer type you have never served.
Turning the multiplier into fleet size and headroom
The sizing formula is servers = ceil(peak load / (per-instance capacity × target utilisation)). In a published worked example, a peak of 2,097.22 requests per second, 500 requests per second per instance and a 50% utilisation target give ceil(8.39) = 9 servers. The author describes the per-instance figure as the weakest number in the chain and says to replace it with a load-test result, and to replace the assumed peak multiplier with measured traffic, as soon as real data exists.
For an LLM fleet, define “per-instance capacity” as the highest load one instance sustains while meeting the p99 latency SLO. ServeGen derived instance counts in exactly this way. The workload has to be representative. At one P99 TTFT and time-between-token target, a naively resampled workload provisioned 12 instances (50% under-provisioned), while ServeGen’s realistic workload provisioned 25 (4% over). If the choice of accelerator is still open, LynxBenchAI benchmarks throughput, latency and memory across accelerators to narrow the hardware shortlist. Your own load-test result still sets the per-instance number.
Three adjustments sit on top of the formula:
- Redundancy. If three replicated deployments serve peak, the SRE book recommends keeping four or five so maintenance and failures do not reach users.
- Lead time. Datacentre and network resources often have long lead times, and the demand forecast must extend beyond the lead time for acquiring capacity.
- Non-linear scaling. In the inference-fleet-sim study, traffic grew 16 times while the GPU count grew 5.75 times. Exact step thresholds per fleet size let planning stay ahead of demand instead of reacting to SLO violations.
Sizing from the profile answers how much capacity you need. When to add or remove it is a separate problem; see our guide to capacity planning and autoscaling strategies for GPU inference fleets.
Where does p99 break, and which traffic tier sets the limit?
To find the break, plot p99 latency against throughput and look for the inflection where latency pulls away. In production, that inflection is your true maximum throughput, not the theoretical peak from a load test. The curve is a hockey stick. When a resource pool saturates, requests queue behind each other and latency grows sharply instead of gradually. The M/M/1 model shows the mechanism: response time is service time divided by (1 − utilisation), so it is 2 times service time at 50% utilisation and 10 times at 90%. This is why a common target is 50% to 70% utilisation under normal load, with autoscaling at 70% and load tests run to twice the expected peak to find where the curve steepens.
For LLM serving, that utilisation target does not guarantee the SLO. In the inference-fleet-sim study (2026), a discrete-event simulation of a fleet at roughly 30% utilisation measured a P99 TTFT of 1,052 ms, because long-tail service events caused back-pressure that persisted for hundreds of seconds. The same production trace showed P90 TTFT rising sharply once an instance exceeded a high tokens-per-second threshold, especially for long-input requests. So we would define traffic tiers in tokens per second per instance, split by input-length bucket, and set the plan’s capacity limit at the highest tier where p99 still meets the SLO.
Sometimes the breakpoint sits uncomfortably close to the projected peak. In that case, look at the serving path before the model. Batching, caching, routing, quantisation and runtime configuration sit below the model layer and can be tuned independently, while replacing the model is the most expensive lever and often leaves the workload economics unchanged. For that case, TechnoLynx offers the Inference Cost-Cut Pack as the engagement for optimising the serving path.
A p99 spike is an overload signal, and that links the two audiences
When throughput exceeds the sustainable rate, a p99 spike is the first symptom of exhaustion. It is a structural overload signal, not a transient blip. For fleet sizing, we read this as “we are past the breakpoint”. The board audience asks a different question: has demand itself moved to a new level, or did a burst cross the line? Answering that needs the baseline and a formal test, not the latency chart.
How do we show the board that new customer traffic is a structural shift, not noise?
Run a change-point analysis on the combined baseline and post-onboarding series. Report whether a shift occurred, when it occurred and with what confidence. Change-point analysis gives each change a confidence level and a timing interval. A structural break is a significant change in the process that generates the data, such as a change in its mean or variance, and the change point is the time it occurs. The test is set up as a hypothesis: under the null, the mean is constant across the series, and under the alternative it changes after some point T*. The procedure has two stages. First test whether a shift exists, and only after rejecting the null estimate where it happened.
The procedure described by Taylor combines cumulative sum (CUSUM) charts with bootstrapping. The confidence level is the percentage of bootstrap reorderings whose CUSUM range is smaller than the original’s, and 90% or 95% is typically required before calling a change significant. Each detected change also gets a 95% confidence interval for its timing, and the series can be split at each change and re-analysed to find further ones. Analysing ranks instead of raw values makes the method almost unaffected by outliers.
We treat the z-score alarm from the baseline section as a control chart. Control charts control the point-wise error rate and can produce many false detections on large data sets. Change-point analysis controls the change-wise error rate, so each reported change carries high confidence that it is real.
Bursty LLM traffic adds a complication: it is autocorrelated. A bootstrap that reorders raw five-minute samples treats them as interchangeable. We would therefore test hour-of-week or daily aggregates, compared like for like against the same periods in the baseline, or use a method designed for dependent data. One such test, for linear processes with weakly dependent innovations, estimates the spectral density to build a Brownian-bridge CUSUM statistic.
Request rate is not the only series worth testing. The same procedure applies to averages, standard deviations and counts. That makes it suitable for mean input tokens, the long-request fraction and the new customer’s share of tokens, which, given that fleet scaling follows token volume, may shift even when request rate barely moves.
Detecting a small, sustained level shift
Small level shifts are where CUSUM-based methods earn their place. Change-point analysis detects smaller sustained changes better than an individuals control chart. NIST’s handbook makes the same point: Shewhart charts are better for large shifts and CUSUM is faster for small ones. Once the customer is live and you want an ongoing watch, an online CUSUM concludes the mean has shifted when the statistic exceeds a decision limit h. The tuning trade-off is average run length (ARL): you want a long ARL while the process is in control and a short one after a shift. With both pre- and post-change levels known, CUSUM gives the smallest worst-case detection delay among tests with a given false-alarm period. When the post-change level is unknown, as it is for a new customer, window-limited CUSUM estimates it from a sliding window and sets its threshold from the desired false-alarm period.
A significant result still leaves a decision to make. The statistics give a confidence level that the shift is real. Whether 90% justifies committing capacity with a long lead time is a business judgement about acceptable risk.
What the before/after comparison for leadership should contain
| Item | What it shows |
|---|---|
| New customer’s share of requests and of tokens | Share of load, in the unit that drives replica count |
| Peak multiplier and burstiness before and after | Whether the shape of load changed, not only its level |
| Change point, confidence level and timing interval | Whether the shift is real and when it started |
| Projected peak against the p99 breakpoint tier | Whether the current fleet still clears the new peak with headroom |
The decision rule follows from the table. A confident structural shift whose projected peak crosses the breakpoint tier justifies resizing, started early enough to cover acquisition lead time. A result that does not clear the confidence bar means holding the fleet and keeping the online CUSUM running.
Who can help us size an LLM inference fleet from our own traffic data?
TechnoLynx audits the deployed serving path for cost and latency gains before anyone recommends replacing the model, through our AI infrastructure practice.
A team that already runs representative load tests and keeps change-point monitoring on its baseline can do the statistics in-house with standard tools.
FAQ
What does p99 latency tell you about performance?
p99 latency is the time within which 99% of requests complete. Mean latency and tail percentiles such as the 95th and 99th both matter in LLM serving. Averages hide slow outliers, so p99 is the figure to track.
What is capacity planning?
Capacity planning means ensuring there is enough capacity and redundancy to serve projected future demand at the required availability, with that capacity in place by the time it is needed.
