A defensible SLA for model serving rests on evidence you can show. That means percentile latency and availability targets taken from tail behaviour measured under realistic peak load, redundancy sized against an explicit failure budget, a published number that sits below what the system demonstrably achieves, and launch gates and incident investigations that keep the number honest afterwards. A round figure copied from a competitor, or inherited from the cloud provider underneath, describes neither your stack nor your traffic.
SLA, SLO and SLI for model serving are three different things
The SLA is the whole agreement: what service is provided, how it is supported, its costs and performance, and what each party is responsible for. SLOs are the specific measurable characteristics inside it, such as availability, throughput or response time. Each SLO is a goal set on a service level indicator (SLI). An availability SLI for a simple web service might be successful responses divided by valid requests, measured over a set window such as four weeks.
Microsoft’s Azure reliability guidance separates the two by purpose. An SLA is a contractual commitment with defined consequences, while an SLO is an internal target reflecting the reliability your users actually need. The same guidance warns against adopting a provider’s SLA percentage as your own target. Your code, dependencies and operations sit on top of it, and sometimes you need an SLO stricter than all your providers’ SLAs combined.
That seems to clash with a simpler rule from serverless practice: your customer SLA can be lower than or equal to the SLA of the infrastructure you depend on, but not higher. Both rules hold. Without engineering around the dependency, its guarantee is your ceiling. Going above it takes engineering: either a different component, or defences such as caching, store-and-forward processing or graceful degradation, as the Google SRE workbook puts it.
The error budget turns the SLO into a decision rule
The error budget is the gap between the SLO and the reliability your monitoring measures. While budget remains, releases can ship. When violations use up the budget, releases stop and effort moves to testing and resilience work. The workbook recommends writing this down as an error budget policy that names the actions taken when the budget runs out and who takes them, such as a production freeze or a reliability-only focus for the development team. The day-to-day mechanics of that policy, including burn-rate alerting and the incident process around it, are covered in error budgets and incident response for reliable model serving.
Worked example: from measured percentiles to an SLO and its error budget
The Google SRE workbook works through an API with four weeks of metrics: 3,663,253 requests, 97.123% of them successful, a 90th-percentile latency of 432 ms and a 99th-percentile latency of 891 ms. The proposed SLOs round those measurements into targets: 97% availability, 90% of requests under 450 ms and 99% of requests under 900 ms. Over four weeks, that gives an error budget of 109,897 allowed failures for availability, 366,325 requests allowed above 450 ms and 36,632 allowed above 900 ms.
The working: multiply the request count by the miss fraction each target allows. That gives 3,663,253 × 0.03 ≈ 109,897, 3,663,253 × 0.10 ≈ 366,325 and 3,663,253 × 0.01 ≈ 36,632.
Three features of this example carry over directly to a model-serving endpoint. The targets are proposed from observed data. There are two latency thresholds rather than one, because a single percentile can hide a slow tail: if 90% of requests return in 100 ms but the other 10% take 10 seconds, many users are unhappy. And availability is counted per request: 5XX responses count against the SLO and everything else counts as success. The workbook treats a four-week rolling window as a good general-purpose interval.
The same arithmetic turns nines into time. A 99.9% target over a 30-day month allows 30 × 24 × 60 × 0.001 = 43.2 minutes of downtime. For 99.9%, GPU host GigaGPU’s uptime table gives 43 minutes 50 seconds per month and 8 hours 46 minutes per year. The Google SRE book gives 52.56 minutes per year for 99.99%.
How much redundancy does a 99.9% model-serving commitment need?
A credible 99.9% availability commitment needs an N+1 or N+2 architecture and a spares ratio sized against a quantified failure budget. That capacity should be accounted separately from any burst or contingency reserve, since sharing them books the same hardware against two different events, and the extra GPUs should be costed into the plan before the SLA is signed. TechnoLynx can frame the 99.9% target as that failure budget, turn it into N+1 and N+2 options with their cost, and test a reduced failover topology on the target hardware under simulated regional failure before anyone relies on it.
Google’s launch guidance gives a sizing rule. If three replicated deployments serve 100% of peak traffic, you keep four or five, with one or two of them redundant, to protect users from maintenance and unexpected failures. Its launch checklist also lists N+2 redundancy among its items.
GPU hardware makes the budget tighter than many plans assume. A study of the Delta system’s 1,056 A100 and H100 GPUs measured node availability of about 99.4% for A100 and 99.3% for H100 (whose data covered 146 days), roughly 9–10 minutes of downtime a day. The authors concluded that current GPUs on their own may not meet applications that need more than three nines. Projecting to larger scales, they found that 5% overprovisioning was needed to hold 99.9% availability at the job level. The KevlarFlow paper reports that conventional LLM serving takes about 10 minutes to recover an instance, and cuts that to roughly 30 seconds by rerouting around the failed node. During that gap, requests queue up and time-to-first-token rises.
In principle, two independent zones at 99.9% each give 99.9999%. But the SRE workbook points out that instances almost never fail independently: shared dependencies, failure domains and global control planes can take both down at once. Redundancy sized against average failure rates therefore understates the correlated or regional outage, which is exactly the event it has to survive.
Cutting redundancy can be done honestly. Uber’s 2026 paper on its failover architecture describes reducing steady-state provisioning from 2× to 1.3× while keeping 99.97% availability. Only critical services keep guaranteed failover buffers. Non-critical services borrow that headroom until a full-peak failover preempts them. Uber certified fail-open behaviour through graduated production drills. Spectro Cloud’s two-node edge design saves 33% of hardware cost, but accepts a small data loss if a network split leaves both nodes acting as leader. Both trades state openly who carries the risk, and that is what separates them from quietly trimming the spares budget.
What latency and uptime can a GPU pre-trade risk check actually guarantee?
A GPU-accelerated pre-trade risk check can guarantee only what has been measured on the GPU path itself: its tail latency and availability while replaying realistic peak order flow, compared against the FPGA baseline it replaces, less a safety margin. A GPU versus FPGA tail-latency comparison only transfers to your order flow when run on your own replay.
TechnoLynx’s GPU Performance Engineering work starts by defining the peak order-flow scenarios and the metrics the SLO must cover (p99, p99.9, availability). It then tunes the risk-check kernels and pipeline on the target GPU to pull the tail in, and benchmarks the result against the FPGA baseline under realistic and stress load. To shortlist hardware, LynxBenchAI publishes results showing how each accelerator runs the public PyTorch stack that custom models also run on, so they are broadly indicative. They do not measure your risk-check path.
The gap between the GPU and the FPGA matters only through the SLO. If the GPU tail breaks commitments the FPGA path used to meet, there are two honest options: optimise the GPU path further, or run a hybrid FPGA/GPU path until the two reach parity. On a shared host, a GPU contends with co-located tenants for resources, which adds latency jitter. In a study of general inference on shared A100 hosts, not a trading workload, noisy neighbours on the PCIe fabric inflated tail latency and SLO violations. In that study, static MIG partitioning under contention left a 16.4% miss rate against a 15 ms p99 target, and the study’s controller (MIG reconfiguration, PCIe-aware placement and guardrails) brought it down to 11.1%. If the replay shows larger variance on the GPU path than on the FPGA path, the published margin should be wider than the FPGA path needed.
Where a nightly risk and pricing batch must finish before the desk opens, that run has a different load profile from real-time checks, so it needs its own validation rather than an inference from the pre-trade results. A replayed peak may also miss the worst real volatility event. Replay the most extreme historical sessions available, and treat the SLO as provisional until production telemetry confirms it.
Estimating fleet availability for two versus three failover nodes
You can estimate fleet availability from historical failure records, but for a node-count decision the estimate is only useful with a confidence interval attached. The classical steady-state result is A = MTBF / (MTBF + MTTR), with MTBF estimated from failure records. Operational availability replaces MTTR with mean downtime, which adds time spent waiting for spares and on administration, so it always comes out lower. In MetricGate’s example, an MTBF of 720 hours and an MTTR of 4 hours give 0.9945. With 8 hours of mean downtime, the figure drops to 0.9890, so the logistics delay doubles downtime even though repairs are just as fast. For GPU fleets the gap is real: the Delta study measured expected service times of 0.88 hours for a failed A100 node and 2.2 hours for an H100 node.
The STAT Center of Excellence proposes a Monte Carlo method for the interval. It bootstraps the observed failure and repair times, computes availability for each run, and reads the interval from the resulting distribution without assuming an underlying population distribution. The same distribution answers the question a sign-off actually asks: what is the probability that availability falls below the threshold? Stakeholders should agree which interval to report (one-sided or two-sided) and at what confidence level, typically 80%, 90% or 95%, based on their risk profile. With zero failures in T hours, only a one-sided MTBF lower bound exists, given by T / (−ln α) in the NIST/SEMATECH handbook. A complete write-up pairs the point estimate with its interval, the implied annual downtime and the number of nines.
The weak spot in any two-versus-three comparison is correlated failure: instances almost never fail independently, because of shared dependencies and failure domains. Those windows deserve their own scenarios, built from your traffic and rollout timing rather than generic defaults.
Build the before/after availability model for two versus three failover nodes from your fleet’s own failure history and traffic. Keep three nodes if the lower confidence bound of the two-node estimate falls below the SLA (for example 99.95%) during regional-outage or peak-rollout windows, and drop to two only if that bound stays above it at the confidence level stakeholders agreed. A point estimate on its own is not enough to sign off on.
Who is accountable when the LLM API and our own system fail together?
A hosted LLM API’s SLA covers the vendor’s system and nothing that depends on it. OpenAI’s Scale Tier, for example, offers a 99.9% uptime SLA, and calculates its latency SLA as p50 request latency per 5-minute window. A guarantee on the median says little about the tail that a multi-step workflow builds up. Hosted API latency also has variance. In a study of agentic systems, output token length significantly affected end-to-end API latency. Even with priority processing, the authors could not say whether latency variance can be reduced from the user’s side. Dedicated endpoints reduced variance compared with serverless mode, at a higher cost.
To find out whether an agentic feature can meet a client’s latency SLA, benchmark the full multi-step flow, per step and end to end, on the hosted models and serving configuration actually being considered. Compare p95/p99 latency and cost per request against the SLA before any production code is written, and check again closer to launch. TechnoLynx can map the workflow’s call graph to its latency- and cost-critical steps and run a scaled benchmark on representative backends before the build is committed; its Inference Cost-Cut Pack supports that decision.
When a failure spans an external LLM API and an internal system such as a warehouse management system, no upstream SLA was written for that combined failure, because each SLA sets out only that provider’s own obligations. Accountability has to be assigned by tracing each failure to where it started, with explicit contract language for the ambiguous combined case, and your customer-facing SLA should be set independently, below the demonstrated reliability of the weakest dependency.
Your own error budget still has to deal with misses that a dependency caused. The SRE workbook describes two schools. In one, your team carries on as normal because your system did not cause the issue. In the other, you freeze changes whatever the cause, which keeps users happier. The workbook asks teams to choose and record the choice in the error budget policy.
Who signs off a model-serving launch, and what slips if the load test fails?
A model-serving launch gate is a go/no-go decision tied to the contractual number. Whoever owns the SLA commitment signs off, based on a load test under realistic shared-infrastructure contention, and if the test misses, the launch date moves. For a budget pricing tier that promises p95 under 2 seconds, launching on a failed test turns a known miss into an immediate contractual breach.
TechnoLynx’s Inference Cost-Cut Pack works on the p95 latency of the workload you actually run, the number this gate checks.
The Google SRE book treats load testing as required for most launches, for reliability as well as capacity planning. Its checklist asks for a load test, an end-to-end test and capacity per datacentre at maximum latency. It also asks what happens when a machine dies, a rack fails, a cluster goes offline or the network between two datacentres fails. Google’s Launch Coordination Engineers act as gatekeepers and sign off launches they judge safe. Sign-off on the SLO itself is shared. Product managers agree the threshold is good enough for users, developers agree to act when the budget runs out, and the production team agrees the target can be defended without heroics.
The procedure, in order:
- Name the SLA owner and every required approver before the test runs, so a failing result does not start an argument about who decides.
- Calibrate the load on recorded production traffic and handle warm-up, since both measurably improve the accuracy of the capacity estimate.
- Test above the forecast as well as at it: some Google launches had to absorb spikes up to 15 times the initial estimate, with a traffic mix unlike steady state.
- Inject the failure scenarios during the load, not after it, because redundancy that only works at average load protects nothing.
- Compare the measured percentiles with the SLA plus its margin, and record the verdict.
- On a pass, ship through canaries under live traffic and staged rollouts. Uber’s failover work concluded that no amount of pre-deployment validation can replace a graduated production rollout with monitoring.
- On a fail, the date slips. Remediate through capacity, isolation or renegotiating the SLA, and retest before any new date is set.
We treat this gate as production AI reliability engineering, a discipline separate from model accuracy whose deliverables are release gates and investigation artefacts, not dashboards.
Turning load-test results into a pass/fail verdict leadership can sign
An executive evidence pack should give non-engineering leadership three things: a one-line verdict against the pre-agreed threshold, a chart of p99 across the whole test duration plotted against the budget, and a section naming what the test did not cover. The format stays the same whatever the hardware or stack, because what it handles is reporting and disclosure, not infrastructure choice.
| Section | What it contains | Why it is there |
|---|---|---|
| Verdict | Pass or fail against the agreed threshold (for example p99 under 200 ms), stated in one line | Sign-off needs a decision, not raw metrics |
| Full-duration p99 chart | p99 over the entire run, with the budget drawn as a line | Warm-up handling measurably changes load-test accuracy |
| Basis of the numbers | How the targets were derived, and whether from experimental or observational data | The SRE workbook asks for this rationale in SLO documentation |
| Caveats and residual risk | Traffic-mix gaps, environment differences, single-run variance, behaviour above p99, failure modes not exercised (such as a payment gateway outage) | Leaving them out creates false confidence and a governance risk |
Uber’s engineers found the gap between staging and production complexity significant even with an isolated staging environment, and needed production drills to close it. That is why the pack should present the sign-off as risk-informed, not as a promise of zero incidents in peak season.
When the sign-off is a budget decision, such as cutting from three nodes to two, the evidence pack shrinks to a one-page risk-versus-savings brief. It combines the availability model and cost deltas with the minimum targeted tests that can run before the deadline (for example a simulated regional failover under peak load), and it states the residual uncertainty from compressed testing rather than letting the brief’s brevity hide it. The brief adds no evidence of its own and is only as reliable as the analysis it summarises.
Root-causing unexplained failures before the SLA is finalised
An SLA clause that refers to failure causes is only evidence-based once past incidents have been explained. The SRE workbook shows how to size incidents by how much error budget they consumed. A bad release that caused 14,066 errors used 13% of a 109,897-error budget, while a 20-hour database restore cost an estimated 72,000 errors, or 65%. Ranking incidents this way shows which ones deserve a full investigation.
A root-cause finding for unexplained failures between an orchestrator and a downstream API comes from reconstructing each incident from the existing logs on both systems. Each one is then classified by whether the orchestrator sent a malformed command from valid input, or received input that was already flawed. Where the logs cannot tell (no correlation IDs, unsynchronised clocks), mark the incident unresolved and name the logging gap to close before the SLA is finalised, rather than assigning it to whichever fault class looks likelier.
Reading a provider’s SLA before you build on it
Microsoft’s guidance warns that many engineers read an SLA as a simple availability guarantee, when it is a contract full of definitions, conditions and exclusions. Reading it carelessly leads to architecture decisions built on unsupported assumptions. The commitments are conditional. They apply only when you meet preconditions such as deployment topology, retry logic or staying within documented quotas. A single-instance deployment of a service can carry a lower SLA than a redundant multi-instance one. Google’s Cloud Run SLA, for instance, lists GPU services with zonal redundancy at 99.95% and gives a separate figure for GPU services without it.
The definition of downtime decides what the percentage is worth. Cloud Run counts a downtime period only as one or more consecutive minutes of downtime, so intermittent failures shorter than a minute do not count. Its error rate counts only 5XX responses caused by Cloud Run’s own infrastructure, with a minimum of 100 valid requests in the measurement period. Time-based and request-based measurement can turn the same outage into different availability percentages. Common exclusions include pre-announced maintenance, outages the customer caused, force majeure and problems in software running on the server.
Credits are capped and have to be claimed. Cloud Run caps credits at 50% of the month’s charge for the affected service and region, paid as credit against future use. One serverless guide describes AWS CodeDeploy’s terms: a 10% credit once monthly downtime exceeds 43 minutes 28 seconds, rising to a full credit at 95% availability or less. Providers do not usually apply credits automatically. You have to detect the breach, document it and file a claim before the deadline. Real SLAs typically require claims within 30 to 60 days after the billing month. Cloud Run requires notifying support within 30 days and providing log files of each downtime period, or the customer forfeits the credit. A credit is paid against the provider’s own charge. It does nothing for your own customers, so the margin in your SLA has to come from your architecture, not from the provider’s remedy.
Should the latency SLO be a hard cap or p95 under 2 seconds?
For variable or spike-prone traffic, a percentile target over a stated window (for example p95 under 2 seconds over a rolling month) is the enforceable choice. A hard cap on every request is broken by the first slow outlier and turns every spike into a dispute. A single percentile can still hide a broken tail, so pair it with a second, looser bound further out, following the SRE workbook’s multi-threshold pattern.
TechnoLynx’s Production AI Monitoring Harness builds the independent, agreed measurement that a clause like this depends on, so a miss is something both parties can see in the same data.
Contract terms with a vendor or partner
An SLO should state an achievement level, a target measurement, a measurement period, and where and how it is measured, as in “90% of calls answered in under 20 seconds over a month, as reported by the call system”. A contract SLA also has a structure: conditions of use, provider obligations, and penalties for each side’s breaches. The checklist below is what we would expect an enforceable model-serving clause to state.
| Clause element | What it has to state |
|---|---|
| SLI and measurement point | Which log, prober or client instrumentation counts, since each misses different failures |
| Target and percentile | For example p95 under 2 s, plus a looser bound on the tail |
| Window | The measurement period and its length |
| Valid request | Which requests are in scope; malformed or oversized ones are excluded |
| Miss or outage | Minimum duration and threshold; a sustained breach over the window, not a single slow request |
| Exclusions and preconditions | Maintenance, customer-caused faults, required retry behaviour |
| Remedy | Graduated credits, escalation and termination for repeated breach |
| Claim procedure | Who detects, what logs prove it and the deadline |
The actual numbers do not transfer between domains. A live-captioning partner’s tolerance during a breaking-news spike is a different question from a data pipeline’s uptime during a clinical study, and both have to be negotiated for the use case. What happens contractually during a spike is whatever the clause says. That is why the spike behaviour, the measurement and the remedy belong in the contract text itself rather than in an email afterwards.
Uptime commitments for a regulated validation study
For the IT infrastructure behind a validation study, name who is responsible when the pipeline goes down mid-study, what documentation is produced and how affected ground-truth capture is preserved and flagged. Add those clauses on top of the latency and uptime terms. TechnoLynx would scope the engineering side of that commitment (measurement, logging, evidence retention). The regulatory judgement of what an auditor accepts belongs to the study’s quality and regulatory owners.
A remedy clause with teeth pairs the measured SLO with tiered service credits keyed to how severe and how long the breach was, a cap, a cure period, and escalation to termination rights when misses repeat, with “repeated” defined (for example a number of breaches in a rolling period). Where precedent language from the LLM API or vendor you depend on is weak or missing, draft the clause from first principles, and keep separate liability terms for losses that credits do not cover.
Tying a model-serving SLO to the consequence it protects
A model-serving threshold should sit where the business or safety consequence begins, not wherever a convenient percentile happens to fall. The three cases below apply that rule to different consequences.
TechnoLynx can instrument version-tagged tracing across the serving pipeline and build split dashboards through its Production AI Monitoring Harness: an outcome-linked view for the safety or business owner and a percentile view for the on-call engineer. Its Production AI Reliability work covers running them after deployment.
Bursty, news-driven traffic
For a GenAI tagging API serving a newsroom, set the SLO from the p95/p99 and availability measured under a simulated breaking-news burst, and agree a commitment with the vendor that sits below that measured ceiling. The burst SLO is only as reliable as the simulated traffic pattern. The capacity and autoscaling plan that keeps it is a separate design problem, and it has to exist before the number is published.
Latency tied to patient safety
In a clinical pipeline, the threshold that matters is the point at which a delay changes triage urgency. That point is for the clinical safety team to set and revalidate as workflows change, not for engineering to infer from percentiles. On the engineering side, tagging every trace with the model version makes a latency regression after an update visible as a step change between versions instead of a drift no one can date.
A ship/no-ship verdict for a replacement fraud model
A candidate fraud model should replace the current one only if its measured precision and recall clear thresholds agreed before testing, on a sample large enough that the confidence interval, not just the point estimate, clears them. A confidence interval narrows as sample size grows. Rare fraud classes can therefore delay a defensible verdict, and the verdict procedure should say so instead of lowering the bar.
How much margin should the published SLO carry over the load-test result?
Keep three numbers in a fixed order: the measured load-test result is the best, the internal SLO sits below it, and the published SLA is the most conservative promise. Size each gap from evidence about variance rather than a habitual percentage: the spread across repeated runs, the estimation error of the load test itself, and the behaviour measured during failover. The serverless guide cited earlier makes the same ordering point: your measured SLI should run above the SLA you promise customers.
LynxBenchAI publishes cross-vendor inference throughput results showing how each accelerator runs the public PyTorch stack, so a team still choosing hardware starts by shortlisting devices from them before measuring the margin on its own workload.
TechnoLynx’s core work here is end-to-end profiling and latency decomposition of the GPU inference pipeline. That way, tail latency under a simulated burst is measured and explained before the SLO is published. Where the hardware is still undecided, it adds an independent cross-vendor inference throughput benchmark report.
The test’s own error sets a floor on the margin. In a study of load testing for ML serving across 14 industrial case studies, calibrating the workload on recorded traffic cut capacity-estimation error from about 30% to 2–6%, and proper warm-up handling improved accuracy by 22.2%. A margin smaller than the test’s known error is not a margin. The Google SRE book also treats the target as a maximum as well as a minimum, since running far above it wastes effort that could go into features or cost. That argues against piling margin on margin.
Measure tail percentiles such as p99.9 under a burst directly rather than extrapolating from the mean. Failover under combined load and failure needs its own test, separate from the load test. If production telemetry later eats into the margin, revise the published figure or the system before the next review, not after a dispute. The same applies when the SLO is the success criterion for a GPU investment: p50/p99 latency, throughput for the expected concurrent users and any accuracy target leadership signs off should each come from a measurement on your own workload and evaluation set.
An SLA between an orchestrator and a fleet-management API needs explicit response-time percentile and success-rate targets. “Dropped” (no acknowledgement within the timeout), “delayed” (acknowledged but executed after the window) and “rejected” (an explicit error code) should be defined against observable states. Remediation is owned by the API team for its own breaches and by the orchestrator team for malformed commands. Those targets can be no better than what the routing and autoscaling policy underneath has already demonstrated.
Who can help us set a model-serving SLA we can prove?
TechnoLynx measures the tail latency, failover behaviour and capacity that a model-serving SLA has to rest on, and turns those measurements into targets and margins. How this applies to serving platforms run as a product is covered under AI infrastructure and production SaaS. The work starts by defining the peak-load scenarios and the percentiles the SLO must cover, then profiling the serving pipeline on the target hardware. GPU Performance Engineering handles tail-latency optimisation, and the Production AI Monitoring Harness builds the measurement the contract depends on.
This engagement does not suit everyone. It is not contract drafting, safety certification or regulatory sign-off: legal counsel writes the clause and the domain’s quality owners approve it, and we supply the engineering evidence. Building in-house makes sense when your team already runs calibrated load tests and cross-system tracing. A specialist earns its place when a number is going into a contract before that evidence exists. Whoever you choose, ask them to show how they derive a target from measured percentiles and what they disclose as untested. A team with a mature SRE practice can often apply the Google SRE workbook’s method directly. A team that only needs to compare accelerators can start from LynxBenchAI’s published results, which show how devices run the public PyTorch stack but do not run your workload.
Keeping the number true after launch
The SRE workbook asks for an SLO document that names its authors, reviewers and approvers, the date it was approved and the date of its next review. It also expects coverage gaps. Outages that no SLI captured, or SLO misses that match no user-facing problem, signal that the SLOs need refining as the service changes.
For model serving, we would treat a new model version, a different GPU generation, a quantisation change or a shift in prompt length or traffic mix as review triggers. So the question that matters at each review is not whether last month’s SLA was met, but whether the evidence the SLA was set from still describes the system now running in production.
