Someone drops “LADDIS” into a serving-path review and the room reaches for a single number — the throughput-vs-latency curve, declared the verdict. That instinct is where the trouble starts. A load-and-latency benchmark tells you how a system behaves under a synthetic request mix. It does not tell you why it behaves that way, and it will never tell you that 38% of your p95 latency is sitting in pre-batch queuing rather than on the GPU. LADDIS — the load-generation methodology that grew out of the file-server benchmarking work behind SPEC SFS — is worth understanding not because you will run the original, but because its shape is the shape of every serious load-and-latency benchmark you touch in inference profiling. It generates a controlled, reproducible mix of requests, ramps the offered load, and records how response time degrades as arrival rate climbs. That is a genuinely useful instrument. The mistake is treating the instrument’s output as the diagnosis. How should you think about LADDIS in practice? The core idea is deceptively simple: drive the system under test with a synthetic workload generator that you fully control, increase the offered request rate in steps, and measure the latency distribution at each step. The benchmark owns the stimulus — request mix, arrival pattern, concurrency — so that runs are reproducible and comparable. What comes out is a curve: latency on one axis, offered load on the other, with the familiar knee where response time turns sharply upward as the system approaches saturation. In practice, “a LADDIS-style benchmark” means any harness that does this for your serving path. For LLM inference that might be a tool driving concurrent completion requests against a vLLM or SGLang endpoint; for a classic model server it might be a load generator hitting a Triton Inference Server deployment. The methodology is the same regardless of what sits behind the socket. You are producing a defensible statement of the form: “below X requests per second, p95 stays under our SLA; above it, p95 crosses.” That statement is the benchmark’s real product. Not the peak throughput headline — the operating point. And a load curve, however clean, stops exactly there. It can show you that latency degrades past a request rate. It cannot say where inside the request that latency accumulated. What a load-and-latency benchmark measures versus what a profiler measures This is the distinction that gets collapsed, so it is worth drawing sharply. A LADDIS-style benchmark is a black-box instrument: it observes the system from the outside, treating the serving path as one opaque unit that turns requests into responses at some latency. A profiler is a white-box instrument: it opens the path and attributes time and resource consumption to the stages inside it. Dimension LADDIS-style load benchmark Serving-path profiler Vantage point Black box, from outside White box, inside the path Primary output Latency-vs-load curve, saturation knee Per-stage p50/p95, utilisation gaps Answers How much load before SLA breaks? Where does the time and money go? Controls Request mix, arrival rate, concurrency Nothing — it observes a fixed point Attribution None — one opaque latency figure Queuing, batching, kernel, I/O, padding waste Right question Setting the operating point Naming the bottleneck at that point Read the table as two halves of one method rather than two competing tools. The benchmark controls the stimulus and gives you a number with no cause attached. The profiler attaches causes but needs someone to tell it where to stand. That “where to stand” is the operating point the benchmark produced. This is the same reasoning that underpins reading any production-matching LLM benchmark: the load figure only becomes actionable once you know which stage of the serving path it indicts. The unit of what you are measuring matters here too. A load curve measured against a hardware-plus-software combination is a property of that combination, not of the GPU alone — a point we make repeatedly when reading what SPEC-style benchmarks actually measure for production AI. Swap the runtime, the batching policy, or the padding strategy and the curve moves, which is precisely why the curve alone cannot be the verdict. How do I find the operating point where p95 crosses my SLA? Start from the SLA, not from the hardware. If your service contract says p95 must stay under 400 ms, then the benchmark’s job is to find the request rate at which p95 first touches 400 ms under a request mix that resembles production traffic. That crossing point — not the peak throughput the box can sustain while latency runs wild — is your operating point. A disciplined run looks like this: Fix the request mix. Prompt-length distribution, output-length distribution, and concurrency should mirror production, not a convenient synthetic average. A benchmark run on 128-token prompts tells you nothing about a workload that averages 2,000-token contexts. Ramp offered load in steps and hold each step long enough to reach steady state — transient warm-up on the first requests will otherwise pollute the tail. Record the full latency distribution at each step, not the mean. The mean hides the tail, and the tail is what your SLA is written against. Read the crossing. The request rate where p95 crosses the SLA threshold is the operating point. Everything above it is over-saturation; everything below it is headroom. The output is a defensible sentence a platform lead can put in front of a budget owner: “we can serve N requests per second within SLA on this configuration.” That is worth far more than a single averaged throughput figure, because it is bounded by the SLA the business actually promised — an approach we treat as foundational whenever mapping a serving path for performance. Why one throughput-vs-latency curve is not enough to decide optimise-or-replace Here is the decision the curve is usually invoked to settle: the operating point is lower than the business needs, so should we tune the runtime or swap the model? A load curve alone pushes teams toward the expensive answer. Latency is high, throughput is low, therefore — the reasoning goes — the model is too big and must be replaced. That reasoning skips attribution entirely. Consider a worked example, with the assumptions stated plainly. Suppose a benchmark shows p95 crossing a 400 ms SLA at roughly 45 requests per second, well short of the 120 the roadmap assumes. The naive conclusion is that the model is too heavy and needs distillation or replacement — a multi-week effort with quality risk attached. But a profiler run at that 45 rps operating point reveals that a large share of p95 is not GPU compute at all: it is time spent in pre-batch queuing while the scheduler waits to assemble a batch, compounded by padding waste where short requests are padded to the length of the longest request in the batch. (These figures are illustrative — the point is the structure of the finding, not the specific percentages.) In a case shaped like that, the fix is a batching-policy and scheduler change, not a model swap. The model was never the bottleneck. This is the divergence point the CCU-level reasoning names: a load curve can show latency degrading, but only attribution can tell you the cause — and the cause decides whether you spend a week tuning a runtime or a quarter replacing a model. The measurable payoff of getting this right is the avoided cost of an unwarranted replacement, which is often the largest single number in the whole analysis. How the operating point feeds per-stage profiling of the serving path Once the benchmark has fixed the operating point, the profiler runs there — at the request rate where the system is genuinely stressed against its SLA, not at idle and not deep in over-saturation where every number is dominated by queue collapse. Profiling at the operating point is what makes per-stage numbers comparable and actionable. At that point you decompose the path. For a GPU-served model the stages typically include request ingress and validation, pre-batch queuing, batch assembly, tokenisation, the prefill and decode phases on the GPU, and response serialisation. Tools like the PyTorch profiler, NVIDIA Nsight Systems, and CUDA event timing let you attribute p50 and p95 to each stage and expose GPU utilisation gaps — the windows where the accelerator sits idle waiting for the CPU-side path to feed it. When prefill and decode dominate for the wrong reasons, techniques like prefill/decode disaggregation become candidate fixes; when the gap is upstream, the answer is scheduling, not kernels. The GPU workload the profiler measures is itself driven by a load generator of exactly the LADDIS shape — the same synthetic-mix, ramped-load stimulus that produced the operating point in the first place. That is the through-line: one controlled generator sets the point, one profiler names what is happening at it. FAQ How does LADDIS actually work? LADDIS is a load-generation methodology — originating in file-server benchmarking behind SPEC SFS — that drives a system with a controlled synthetic request mix, ramps the offered rate in steps, and records how latency degrades toward saturation. In practice, “a LADDIS-style benchmark” means any harness that does this for your serving path, whether against an LLM endpoint or a classic model server. Its real product is not a peak-throughput headline but a defensible operating point: the request rate at which latency crosses your SLA. What does a LADDIS-style benchmark actually measure versus what a profiler measures? A load benchmark is a black-box instrument that observes the serving path from outside and returns one latency-vs-load curve with no cause attached. A profiler is a white-box instrument that opens the path and attributes p50/p95 latency and utilisation gaps to individual stages — queuing, batching, kernels, padding waste. The benchmark tells you how much load breaks the SLA; the profiler tells you where inside the request the time and money went. How do I use a load-and-latency benchmark to find the operating point where p95 crosses my SLA? Start from the SLA threshold, fix a request mix that mirrors production traffic, ramp offered load in steady-state steps, and record the full latency distribution rather than the mean at each step. The operating point is the request rate at which p95 first touches your SLA threshold. That crossing — not the peak throughput the box sustains while latency runs wild — is the bounded, defensible figure worth reporting. Why is a single throughput-vs-latency curve not enough to decide whether to optimise or replace a model? A load curve shows latency degrading past a rate but attributes nothing, so it pushes teams toward the expensive conclusion — that the model is too heavy and must be replaced. Profiling at the operating point often reveals the bottleneck is elsewhere: pre-batch queuing, scheduler waits, or padding waste, none of which a model swap fixes. Without attribution you cannot tell whether a week of runtime tuning or a quarter of model replacement is warranted. How does the benchmark’s operating point feed into per-stage profiling of the serving path? The profiler runs at the operating point the benchmark produced — where the system is genuinely stressed against its SLA, not idle and not in over-saturation where numbers are dominated by queue collapse. At that fixed point you decompose the path into ingress, queuing, batch assembly, prefill, decode, and serialisation, attributing p50/p95 to each with tools like the PyTorch profiler and Nsight Systems. Profiling at the operating point is what makes per-stage numbers comparable and actionable. Can you give an example where a LADDIS-style number looked fine but profiling revealed a queuing or padding-waste bottleneck? In an illustrative case, a benchmark shows p95 crossing a 400 ms SLA around 45 rps, well short of the 120 the roadmap assumes, prompting a push to replace the model. Profiling at that operating point instead shows much of p95 sitting in pre-batch queuing while the scheduler assembles batches, compounded by padding waste where short requests are padded to the batch’s longest sequence. The correct fix is a batching-policy change, not a model swap — and the payoff is the avoided cost of an unwarranted replacement. How do I combine load benchmarking and profiling to attribute cost-per-request to specific serving stages? Use the benchmark to fix the operating point, then profile at that point to get per-stage p50/p95 and GPU utilisation gaps, and finally map those stage times and idle windows onto the cost of the underlying compute. That produces a benchmark-anchored bottleneck map that ties cost-per-request to specific stages rather than a single averaged figure. This is exactly the method behind our [AI Inference Cost Audit](Inference Cost-Cut Pack), which runs its profiler at the operating point a load benchmark establishes. Where this leaves the number Treat a LADDIS-style load benchmark as one controlled generator of stimulus, not as a verdict. It earns its keep by producing a defensible operating point; it fails the moment anyone reads its curve as a diagnosis. The cause always lives inside the path, and only a profiler standing at the operating point can name it. That combined method — benchmark to set the point, profiler to attribute the cost — is what our [AI Inference Cost Audit](Inference Cost-Cut Pack) is built around, and it is the discipline behind the wider R&D engagements we scope to your problem. The open question for any team holding a LADDIS number is the one the number cannot answer on its own: at your operating point, which stage of the serving path is actually spending your latency budget — and does the answer justify tuning, or replacement?
