Draw a machine learning architecture diagram and most teams reach for the same three boxes: a model, a database, an API, connected by tidy arrows. The picture reassures everyone in the room. It also hides exactly where time goes. That gap matters the first time a p99 regression lands in production. The diagram on the wall says “model → API → client,” so the instinct is to blame the model. But a request that takes 40 ms at p50 and 900 ms at p99 rarely spends that extra 860 ms doing matrix multiplications. It spends it waiting — in an admission queue, behind a cold cache, on a retried network call, or stuck because a batching window filled up while a slow neighbour held the accelerator. None of those stages appear on the box-and-arrow sketch, which is precisely why the sketch cannot tell you where the latency lives. A useful architecture diagram is not a picture of what components exist. It is a map of the serving path: every hop, queue, cache, and network boundary a single inference request actually crosses, in order, under load. Once the diagram is drawn that way, it stops being a slide for stakeholders and becomes a tool for setting latency and throughput budgets — and for attributing a regression to the stage that caused it. What does a machine learning architecture diagram actually represent? The word “diagram” carries a false promise of completeness. A component inventory feels like an architecture because it names everything present. But a serving system’s behaviour under load is not a property of which components exist — it is a property of how requests flow through them and where they wait. Consider the difference between these two questions. “Do we have a feature store?” is a component question; the box-and-arrow diagram answers it. “When p99 spikes, is the feature-store lookup on the critical path, and does it share a connection pool with the retry storm from the model timeout?” is a serving-path question, and the inventory diagram is silent on it. The second question is the one that gets asked at 2 a.m. This is the same divergence that performance engineering surfaces everywhere: the structure that explains steady-state correctness is not the structure that explains tail behaviour under contention. A diagram grounded in the real inference path is drawn to answer the second kind of question. It is directional — it follows a request from ingress to response — and it is explicit about the stages where time accumulates rather than the stages where logic lives. What a production diagram includes that a box-and-arrow sketch leaves out The typical sketch omits everything that is not a “thing you build.” But most latency lives in the spaces between things you build. In our experience reviewing serving architectures, the stages that go missing from the initial diagram are almost always the ones that turn out to own the tail (an observed pattern across production-readiness reviews, not a benchmarked distribution). A serving-path diagram makes the following visible, each as a stage with its own latency and saturation behaviour: Admission and load-balancing — where a request first queues before any work begins. Connection limits and load-balancer backlogs shape p99 long before the model is touched. Request queueing at the serving process — the depth-vs-latency trade-off. A queue that absorbs bursts also converts throughput pressure directly into tail latency. Dynamic batching — the window where individual requests wait to be grouped for accelerator efficiency. Larger batches raise throughput and raise per-request latency; the batching stage is where that trade is made, and it belongs on the diagram as its own hop. Caches — embedding caches, KV caches for LLM decode, feature caches. A cache is not a box; it is a hit-path and a miss-path with radically different latencies, and both paths belong on the diagram. The model compute stage itself — prefill and decode for autoregressive models, which have distinct latency profiles and are worth splitting. Teams serving large language models increasingly separate these; our note on how prefill/decode splitting cuts inference latency walks through why the two phases saturate differently. Network boundaries — every process hop and every external call, each carrying serialization cost, a timeout, and a retry policy. Retries are latency multipliers hiding in plain sight. Downstream data dependencies — the feature store, vector index, or database read that sits on the critical path. When these saturate, the model looks slow but is innocent. The discipline is not to draw more boxes. It is to draw the request’s journey and annotate each hop with the two numbers that govern its behaviour under load: its latency contribution and the throughput at which it begins to queue. How do you represent queues, batching, caches, and network hops? The representation that works is a directed path, not a topology. Draw the request as a line that enters at ingress and exits at the response, and place each stage it crosses as a node on that line, in the order the request encounters it. Where a request can branch — a cache hit versus a miss, a fast path versus a fallback — draw both branches, because their latencies differ by an order of magnitude and the p99 almost always rides the slow branch. Three conventions make the diagram operationally useful rather than decorative: Mark every queue explicitly. A queue is any stage where a request can wait for a resource held by another request. Admission queues, batching windows, connection pools, and thread pools are all queues. They are where throughput pressure becomes latency, so they are where p99 is born. Annotate each hop with a latency budget and a saturation point. The budget is the p95 or p99 you are willing to spend at that stage; the saturation point is the request rate at which the stage starts queueing. A hop with no annotation is a hop you are not managing. Show retries and timeouts as first-class edges. A 200 ms timeout with three retries is a potential 600 ms latency edge, invisible on a diagram that draws the call as a single arrow. Under partial failure, retry amplification is one of the most common causes of correlated tail latency, and it only shows up if the diagram draws it. The data layer deserves the same treatment. When a serving path reads from a wide-column store or a vector index on the critical path, that read is a hop with its own saturation curve; how the data layer shapes reliability for production AI is often the difference between a stable p99 and a cascading one. How does the diagram help attribute p99 tail latency to a stage? This is the payoff, and it is worth being precise about the mechanism. A p99 regression is a claim about the tail of a latency distribution. To attribute it, you need to know which stage’s distribution shifted — and you can only ask that question if the diagram already decomposed end-to-end latency into per-stage contributions. The naive workflow, working from a box-and-arrow diagram, is to look at end-to-end latency, see it rose, and start guessing. The serving-path workflow is different: because each hop on the diagram is instrumented with its own latency measurement, you sum the per-stage p99s and compare against the end-to-end p99. When they diverge, the gap is queueing — time the request spent waiting between stages, which no single stage’s span captures. Worked example: attributing a p99 spike Assume a serving path with four instrumented hops and these measured contributions (illustrative figures, chosen to show the reasoning): Stage p50 (ms) p99 (ms) Saturation onset Admission / load balancer 2 8 ~1,200 req/s Dynamic batching window 5 60 ~800 req/s Model compute (prefill + decode) 30 45 ~900 req/s Feature-store read 3 12 ~2,000 req/s End-to-end 41 210 — Sum the per-stage p99s: 8 + 60 + 45 + 12 = 125 ms. The end-to-end p99 is 210 ms. The 85 ms difference is not in any stage — it is inter-stage queueing, and the batching window’s saturation onset (~800 req/s) sitting below the model’s (~900 req/s) tells you where to look. The batching stage is filling up and back-pressuring admission. Without the diagram decomposing the path this way, the same 210 ms reads as “the model got slow” — which it did not. This is the avoided-rework outcome an accurate diagram delivers: you instrument and fix the batching window instead of re-profiling a model that was never the problem. The general rule that falls out of this: latency you cannot attribute to a stage is latency spent queueing between stages, and it is the first thing a serving-path diagram makes visible. Where do latency and throughput budgets belong on the diagram? Budgets belong on the edges and nodes where the request actually spends or waits for time — not in a separate spreadsheet that drifts out of sync with the picture. The practice we find durable is to annotate each hop with three figures: a p99 latency budget, a throughput target, and the saturation point at which that hop begins to queue. The end-to-end budget is then the sum of the hop budgets plus a queueing allowance, and if the hop budgets sum above the SLO, the diagram has surfaced an infeasible design before a single line of code ships. Cost sits on the same edges. Every hop that touches an accelerator or an external service has a cost-per-inference, and the batching and caching stages are exactly where cost trades off against latency — a larger batch is cheaper per request and slower per request. Making that trade explicit on the diagram connects directly to what the inference-cost-audit methodology measures: cost-per-inference at a target latency, hop by hop. How does a serving-path diagram feed a release-readiness gate? An architecture diagram that names the stages, budgets, and saturation points is not just an analysis artifact — it is a checklist generator. Each hop on the diagram is a place where the release-readiness gate needs evidence: a measured p99 within budget, a load test that finds the saturation onset, and a retry/timeout policy that does not amplify tail latency under partial failure. This is where the diagram feeds the release-readiness decision framework that instruments AI infrastructure for production. The stages the diagram exposes become the explicit criteria the gate checks. That instrumentation is what the Production AI Monitoring Harness is built to capture, and it is a core part of how we structure our production AI reliability engagements. A diagram that omits a stage produces a gate that cannot see it — which is how features ship with an un-load-tested batching window and regress the first time real traffic arrives. How do you keep the diagram accurate as the serving stack evolves? A serving-path diagram decays the moment the stack it describes changes and the diagram does not. A new caching layer, a runtime swap, a model that adds a decode phase — each introduces or removes a hop, and an out-of-date diagram is worse than none because it directs attribution to the wrong stage. The practice that keeps it honest is to treat the diagram as derived from instrumentation, not maintained alongside it. If every hop on the diagram corresponds to a span in your tracing and a metric in your dashboards, then the diagram and the running system share a source of truth, and drift becomes visible: a stage in the traces with no box on the diagram is a hop you stopped managing. When teams adopt an LLM orchestration framework or a new serving runtime, this is exactly where undrawn hops sneak in. FAQ How does a machine learning architecture diagram work? In practice, a useful machine learning architecture diagram maps the serving path — the ordered sequence of hops, queues, caches, and network boundaries a single inference request crosses under load. It is directional rather than a component inventory, and each stage carries a latency contribution and a saturation point. That makes it a tool for setting budgets and locating regressions, not just a slide for stakeholders. What should a production ML architecture diagram include that a typical box-and-arrow sketch leaves out? It should include the stages where requests wait, which the sketch omits: admission and load-balancing queues, the dynamic batching window, cache hit and miss paths, network boundaries with their retry and timeout policies, and downstream data reads on the critical path. Most latency lives in the spaces between the “things you build,” so those spaces are the ones that must be drawn. How do I represent the serving path — queues, batching, caches, and network hops — in an architecture diagram? Draw the request as a directed line from ingress to response, placing each stage it crosses as a node in encounter order, and draw both branches wherever a request can take a fast or slow path. Mark every queue explicitly, annotate each hop with a latency budget and saturation point, and show retries and timeouts as first-class edges since a timeout with retries is a hidden latency multiplier. How does an architecture diagram help attribute p99 tail latency to a specific stage? Because each hop is instrumented with its own latency, you sum the per-stage p99s and compare against the end-to-end p99. When they diverge, the gap is inter-stage queueing — time no single span captures — and the stage with the lowest saturation onset points to where back-pressure begins. This attributes the regression to a stage instead of leaving the team guessing whether the model, the serving stack, or the load profile is at fault. Where should latency and throughput budgets be annotated on the diagram? On the hops and edges where the request actually spends or waits for time — each annotated with a p99 latency budget, a throughput target, and a saturation point. The end-to-end budget is the sum of the hop budgets plus a queueing allowance; if the hop budgets already exceed the SLO, the diagram has surfaced an infeasible design before code ships. How does a serving-path diagram feed a release-readiness gate for an AI feature? Each hop becomes a criterion the gate must check: a measured p99 within budget, a load test that finds the saturation onset, and a retry/timeout policy that does not amplify the tail under partial failure. A stage the diagram omits produces a gate blind to it, which is how features ship with an un-load-tested batching window that regresses under real traffic. How do I keep an architecture diagram accurate as the serving stack evolves? Treat the diagram as derived from instrumentation rather than maintained beside it: every hop should correspond to a tracing span and a dashboard metric. Then drift becomes visible — a stage that appears in the traces but has no box on the diagram is a hop you have stopped managing — and diagram and running system share one source of truth. Draw the diagram to answer the 2 a.m. question, not the kickoff-meeting one. When the next tail-latency regression arrives, the difference between a serving-path diagram and a box-and-arrow sketch is whether your team spends the night attributing the problem to a stage — or re-profiling a model that was never slow.
