Draw the MLOps architecture on a whiteboard and it looks clean: an orchestration layer on top, a model registry to the side, training and serving pipelines feeding off a shared data plane, observability wired through the whole thing. The diagram is correct. It is also incomplete — because none of those boxes tell you how far apart the GPUs are, what cables connect them, or what happens to your all-reduce when a marginal link degrades under a data-rate transition. That gap is the single most common way an MLOps architecture that reviews well on paper stalls in production. The stack is designed as a software concern and dropped onto whatever interconnect the hardware team standardized on. The two designs never get reconciled — they get sequenced. And a data-parallel job that shards perfectly in theory ends up bottlenecked on a physical constraint nobody in the software review was looking at. What does MLOps architecture actually mean on a GPU cluster? MLOps architecture is the set of layers that carry a model from data to a served prediction, plus the machinery that keeps that path reproducible, observable, and re-runnable. On a single workstation that description is enough. On a multi-node GPU cluster it is not, because one of those layers — the one that moves gradients and activations between ranks — runs on physical cable, not on abstraction. The useful mental model has five layers, and the honest version adds a sixth that most diagrams omit: Orchestration — the scheduler that places jobs onto nodes. Kubernetes with a GPU operator, or Slurm on bare metal. This is where rank-to-node assignment is decided. Data — the plane that feeds samples to GPUs: object storage, a caching tier, and the loaders that stage batches into device memory. Training — the distributed framework: PyTorch with torchrun, Fully Sharded Data Parallel, or DeepSpeed. This layer emits the collectives. Serving — the inference path: TensorRT, Triton Inference Server, or a custom runtime, with autoscaling in front. Observability — metrics, traces, and logs across all of the above. Fabric — the physical interconnect: NICs, switches, and the cables between them. This layer does not appear on most MLOps diagrams, and that omission is the whole problem. The first five layers are portable. You can move them between clouds, re-parameterize them, redraw them. The sixth is bolted to the building. When the software layers are designed without reference to the sixth, the reconciliation happens at runtime, during a job, as a stall — which is the worst possible place to discover it. Why the fabric bounds the software, not the other way around Here is the reframe worth internalizing: on a distributed training job, the collective communication library — NCCL in the NVIDIA world — is not a free abstraction over the network. It is a direct consumer of link bandwidth and link quality. When you shard a model with Fully Sharded Data Parallel, you are trading memory for communication: every step reconstructs full parameters via all-gather and reduces gradients via reduce-scatter. Those collectives run over the fabric. If the fabric is marginal, the collective is marginal, and step time inflates regardless of how clean the orchestration YAML looks. The physical constraint that makes this concrete is cable reach. Direct Attach Copper (DAC) is cheaper, lower-latency, and lower-power than optical, but it only carries a clean signal over a short distance — roughly 3 to 7 meters at 200G and 400G, with the ceiling shrinking as the data rate climbs (vendor-published DAC reach specifications; see the cable datasheet for your exact SKU — this is not a single universal number). Beyond that reach you need an Active Optical Cable. If your rank placement assumes a topology that fits within DAC reach but the physical racks don’t, the link either won’t come up or will run degraded, and your all-reduce pays for it. This is why the reach and signal behavior of Direct Attach Copper cabling belongs in the MLOps design conversation, not just the procurement one. A rank that the scheduler happily places two racks away is fine in the orchestration model and broken in the physical one. The scheduler doesn’t know about meters. Someone has to. How do DAC vs AOC choices change distributed training and inference? The choice between copper and optical is usually framed as a cost decision. In an MLOps context it is a topology decision, because it determines which ranks can talk to which ranks at full rate. Active Optical Cable links extend reach where copper runs out, at higher cost and slightly higher latency, and they change the failure profile — a marginal copper link degrades quietly, while an optical link tends to fail more cleanly. The table below is the reconciliation surface: what each fabric choice implies for the software layers above it. Concern DAC (copper) AOC (optical) Practical reach at 200G/400G ~3–7 m (vendor spec) Tens of meters Latency Lowest Slightly higher Power draw Lower Higher (active optics) Failure mode Can degrade quietly at marginal length Tends to fail cleaner MLOps implication Constrains rank placement to nearby racks Frees placement across rows When it bites Rate transition pushes a marginal link past reach Rarely on reach; watch cost + power budget Read the last two rows carefully. The DAC row is where paper architectures die: a link that carried 200G fine over 5 meters can fall into the AOC-required zone the moment you upgrade the NICs to 800G-class parts like ConnectX-8, because the reach ceiling moves down as the rate moves up. The cable didn’t change. The assumption baked into it did. What happens when the data rate transitions from 200G to 400G? This is the scenario that turns a healthy cluster into a support ticket. A team runs distributed training happily at 200G. Procurement approves a fabric upgrade to 400G — faster is better, obviously. The NICs and switches get swapped. And step time gets worse on some jobs, not better. The mechanism: the reach at which a given cable carries a clean signal shrinks as the data rate rises. A DAC run that was comfortably inside spec at 200G can sit right at or past the edge at 400G. The link may still train and come up, but it runs with retransmits or a reduced effective rate, and NCCL’s all-reduce inherits that degradation. The software stack is unchanged. The MLOps diagram is unchanged. The only thing that moved was the physical margin — and it moved because nobody re-checked the cable plant against the new rate. The lesson is not “avoid 400G.” It is that a data-rate transition invalidates assumptions that were never written down, because they lived in the cable rather than in the config. An MLOps architecture that treats the fabric as fixed infrastructure has no place to record “these ranks assume DAC reach at 200G” — so when the rate changes, the assumption silently breaks. Reconciling the two designs means those assumptions become explicit constraints in the topology plan, re-validated at every rate generation. Which metrics tell you whether software or fabric is your bound? The diagnostic question is: is my step time limited by orchestration and the software stack, or by the interconnect? These signals separate the two. Treat them as a checklist, top to bottom. Training step time trend — if step time inflates as you add ranks faster than the added compute should cost, communication is your bound, not compute. (observed pattern across distributed-training engagements; not a fixed threshold — the crossover point depends on model and fabric.) Collective (all-reduce) bandwidth utilization — measure achieved NCCL bandwidth against the link’s rated bandwidth. A large gap points at the fabric or the collective algorithm, not the model. GPU idle fraction during distributed jobs — high idle time with low utilization while a step is “running” is the fingerprint of GPUs waiting on communication. Link error counters and retransmits — non-zero and climbing on specific links is the smoking gun for a marginal-length cable, especially after a rate change. Rank-to-topology map — if you can’t answer “which physical cable connects rank 3 to rank 11,” you can’t diagnose a fabric bound at all. If the top four look healthy and step time still disappoints, the bound is genuinely in the software layer — the loader, the shard strategy, the framework config — and that’s a different investigation entirely. The point of the checklist is to stop you from tuning FSDP settings for a week when the real problem is a 6-meter copper run at 400G. Observability has to reach the fabric layer for any of this to work. If your dashboards stop at GPU utilization and never touch link counters, you have instrumented five of the six layers. The three pillars of observability applied to GPU utilisation still hold here — metrics, logs, traces — but the metrics have to include the NIC and the switch, not just the accelerator. FAQ What matters most about MLOps architecture in practice? MLOps architecture is the layered path a model travels from data to a served prediction, plus the machinery that keeps that path reproducible and observable. In practice on a GPU cluster it spans orchestration, data, training, serving, and observability — and, critically, the physical fabric that carries collectives between GPUs. It only performs as well as that physical layer allows, so the architecture is really a reconciliation between a software design and an interconnect design. What are the core layers of an MLOps architecture on a GPU cluster? Five software layers plus one physical layer: orchestration (scheduling and rank placement), data (storage, caching, loaders), training (the distributed framework and its collectives), serving (the inference runtime and autoscaling), and observability (metrics, logs, traces). The sixth is the fabric — NICs, switches, and cables — which most diagrams omit even though it bounds everything above it. How do cluster interconnect choices like DAC vs AOC affect distributed training and inference performance? They determine which ranks can communicate at full rate, which in turn shapes how you can place ranks and shard workloads. DAC (copper) is cheaper and lower-latency but limited to roughly 3–7 meters at 200G/400G per vendor specs; AOC (optical) extends reach at higher cost and power. A rank placement that exceeds DAC reach forces optical or runs degraded, and a degraded link slows NCCL all-reduce and inflates step time. Why must MLOps architecture be reconciled with the physical fabric topology rather than designed on top of it? Because the fabric bounds the software, not the reverse — a data-parallel job that shards perfectly on paper still runs its collectives over physical cable. If the two designs are sequenced (software first, hardware later), the reconciliation happens at runtime as a stall. Reconciling them up front makes physical constraints, like which ranks fit within DAC reach, explicit in the topology plan. How does a data-rate transition from 200G to 400G change the assumptions baked into an existing MLOps architecture? Cable reach shrinks as data rate rises, so a DAC run that was well inside spec at 200G can sit at or past the edge at 400G. The link may still come up but run with retransmits or reduced effective rate, degrading all-reduce and worsening step time — even though the software stack and diagram are unchanged. A rate transition silently invalidates reach assumptions that lived in the cable rather than in any config. What metrics tell you whether your MLOps architecture is bounded by software orchestration or by the underlying interconnect? Watch step-time growth as ranks are added (faster-than-compute growth signals communication bound), achieved all-reduce bandwidth versus rated link bandwidth, GPU idle fraction during distributed jobs, and link error/retransmit counters. Non-zero, climbing retransmits on specific links after a rate change point at a marginal cable; healthy fabric signals with disappointing step time point back at the software layer. Most teams already know how to run a distributed job on private-cloud Kubernetes for GPU AI workloads. Fewer can answer, before the job runs, which physical cable connects rank 3 to rank 11 and whether that cable holds its rated signal at next year’s data rate. When you plan a cluster buildout or a fabric upgrade, our GPU engineering work starts by mapping the MLOps stack onto the physical interconnect — because the failure class here is not a bad config, it’s an unrecorded assumption living in a cable.
