A model that still passes its accuracy checks can quietly cost you twice what it did last quarter. The predictions look fine on the dashboard, the drift detector is green, and yet the monthly inference bill keeps climbing. That gap is the whole problem with treating model monitoring as an accuracy exercise bolted on after launch. The naive setup watches prediction distributions, maybe a rolling F1 or AUC, and calls that monitoring. It answers exactly one question: is the model still right? That question matters, but it is blind to the layer where most production inference problems actually live — the compute layer. When inference slows down or costs climb, an accuracy-only view cannot tell you whether you have a data-drift problem, an underused GPU, or a compute-API regression that showed up when you moved a workload across AMD, Intel, and NVIDIA hardware. All three degrade the same metric on the finance report. Only one of them is a model problem. Effective monitoring for GPU inference tracks the full path from request to response, and it keeps model-quality signals separate from infrastructure signals so that a regression can be diagnosed rather than guessed at. That separation is what turns mean-time-to-diagnosis from days into hours. This piece explains what to instrument, why the infrastructure layer is not optional, and how monitoring data feeds back into the GPU performance work we do across accelerator fleets. Why an accuracy dashboard is not model monitoring Prediction quality and serving performance are different failure surfaces with different owners, and they fail independently. A model can hold its accuracy for a year while its per-request latency doubles because a driver update changed how a kernel dispatches. A model can also degrade in accuracy while GPU utilisation stays pinned at a healthy 85%. If your monitoring collapses both into one screen — or worse, only watches the first — you are structurally unable to distinguish the two when something goes wrong. Here is the mechanism that catches teams out. Inference cost is a product of how many accelerator-seconds a workload consumes, and accelerator-seconds are governed by utilisation and throughput, not by whether the answer is correct. A model that is 99% accurate but leaves the GPU 50% idle between batches is burning money the accuracy dashboard will never show. In configurations we have profiled, idle capacity on production inference nodes frequently sits in the 30–60% range before anyone notices (observed pattern across TechnoLynx engagements; not a benchmarked rate). The finance report eventually notices. By then the root cause — a batching change, a memory-transfer stall, a fallback to a slower code path after a runtime upgrade — is weeks old and harder to trace. The correct frame is to treat inference as a lifecycle with two independent health axes. Model quality answers is it still right. Infrastructure health answers is it still efficient. You need both, wired separately, because the value of monitoring is diagnostic — it is only useful if a green light on one axis rules that axis out when the other goes red. We develop this distinction further in ML model metrics that explain GPU bottlenecks versus metrics that mislead. Which infrastructure signals belong alongside accuracy? The infrastructure axis is not a single number. It decomposes into four families of signal, each of which fails in a recognisably different way. Watching only one of them — GPU utilisation is the usual choice, because nvidia-smi makes it trivial — gives a false sense of coverage. Signal family What it measures What its drift usually means GPU utilisation Fraction of time the compute units are busy Idle capacity, poor batching, host-side bottleneck feeding the device Kernel-level throughput Useful work per unit time (ops/sec, tokens/sec at the kernel) A code path regressed; a fused kernel silently fell back to unfused Memory pressure HBM occupancy, allocation churn, transfer volume over PCIe/NVLink Model or batch grew past a threshold; fragmentation; OOM risk Latency tails (p95/p99) Slowest requests, not the mean Contention, thermal throttling, per-workload variance the average hides The reason all four matter together is that they disambiguate each other. High utilisation with falling kernel throughput points at a regressed or fallen-back kernel — the device is busy doing less useful work. Low utilisation with healthy per-request latency points at a feeding problem: the data pipeline, not the GPU, is the bottleneck. Rising memory pressure alongside a widening p99 tail is the classic signature of a batch or context length that has outgrown the accelerator’s HBM budget. No single metric tells that story; the combination does. This is the same observability principle applied at the compute layer that we cover in the three pillars of observability applied to GPU utilisation. Latency tails deserve particular attention because averages actively hide them. A workload with a mean latency of 40ms and a p99 of 400ms is failing one request in a hundred badly enough to break an SLA, while the mean looks healthy. Consumer-grade accuracy metrics never surface this, and neither does mean latency. Per-workload p95/p99 tracking is where tail drift becomes visible before it becomes a support ticket. How monitoring separates data drift from a utilisation or API regression This is the diagnostic payoff, and it is worth stating as a decision procedure because it is where the accuracy-only approach fails hardest. When inference gets slower or more expensive, three root causes present identically on a naive dashboard. The way you tell them apart is by reading the axes together. Diagnostic rubric — inference got slower or more expensive: Did prediction-quality metrics move? If accuracy, calibration, or input feature distributions shifted, you have a data-drift or concept-drift problem. Retraining or a data-pipeline fix is the lever. Infrastructure is a red herring here. Did GPU utilisation drop while throughput held per-request? The device is idle more often — a batching, scheduling, or data-feed change. The model is fine; the serving path starves the accelerator. Did kernel-level throughput drop while utilisation stayed high? The device is busy but doing less useful work per second. This is the signature of a compute-API or runtime regression — a torch.compile graph that stopped fusing, a TensorRT engine that rebuilt with a slower plan, or a kernel that fell back to a portable-but-slow path after a driver or library upgrade. Did the regression appear only on one hardware vendor? If the same model degrades on AMD (ROCm/HIP) but not NVIDIA (CUDA), or on Intel (oneAPI/SYCL) but not either, you have an API-portability problem, not a model problem. The abstraction that let you run everywhere is leaking on one target. Step 3 and step 4 are the ones an accuracy dashboard can never reach, and they are increasingly common as teams spread inference across mixed fleets to control cost. The signature of a portability regression — throughput falls on one vendor’s stack while model quality is identical everywhere — is only legible if you are recording kernel throughput per hardware target. We go deeper on this in ML monitoring for ported inference paths and what it tracks in practice. What monitoring tells you about your compute-API choice over time A compute-API decision — CUDA versus a portable path like OpenCL or SYCL, or a serving runtime like TensorRT versus ONNX Runtime — is usually made once, at selection time, on the strength of a benchmark. That benchmark is a snapshot. The fleet it was measured on drifts: drivers update, models grow, batch profiles shift, new accelerator SKUs enter the pool. An API that was the right call in Q1 can quietly become the wrong one by Q4 without a single line of your code changing. Monitoring is what makes that decision auditable across its whole lifetime rather than only at the moment you signed off on it. If you record kernel throughput and utilisation per hardware target continuously, you can see the moment a runtime upgrade regressed the fused-attention path on your NVIDIA nodes, or the moment your OpenCL fallback stopped keeping pace with the CUDA path it was meant to shadow. Without those signals, the first evidence you get is the cost report, and by then the cheap fix window has closed. The trade-offs between a locked-in and a portable path are the subject of CUDA versus OpenCL and what each means in practice when porting AI workloads. The practical upshot is that monitoring data is not a passive record. It is the ongoing feed that keeps a one-time performance audit honest. An audit that ran at deployment tells you the fleet was well-utilised and the API was well-chosen then. Monitoring tells you whether that is still true, and flags the drift while it is still cheap to correct. Setting thresholds so degradation is caught before the bill Thresholds are where monitoring earns its keep or becomes noise. The temptation is to alert on absolute utilisation — “page me if the GPU drops below 70%”. That produces alert fatigue, because absolute utilisation varies legitimately with load. The more useful pattern is to alert on drift relative to a per-workload baseline, because a workload that has always run at 55% utilisation is not a problem, but the same workload dropping from 85% to 55% overnight is. A workable starting rubric, to be tuned against your own baselines: Utilisation drift: alert when the rolling median utilisation for a workload falls more than a set margin below its established baseline over a sustained window — not on any single dip. Kernel throughput regression: alert on any sustained drop in useful ops/sec or tokens/sec at fixed batch, since throughput at fixed input is the cleanest signal of a code-path regression. Latency tail drift: alert on p99, not mean, and tie the threshold to the SLA rather than to a percentage change — the tail is where SLAs actually break. Cost-per-request: derive it from accelerator-seconds per request and alert on trend, because a rising cost-per-request with flat accuracy is the earliest signal that infrastructure, not the model, has regressed. The point of anchoring alerts to per-workload baselines and to cost-per-request rather than raw utilisation is that it catches the slow, expensive regressions — the ones that never trip a hard failure and so never page anyone under a naive setup. Those are precisely the regressions that surface first in the finance report under an accuracy-only regime. FAQ What does working with ml model monitoring involve in practice? Model monitoring instruments the full inference path so you can tell, continuously, both whether a model is still producing correct predictions and whether it is still serving them efficiently. In practice it means recording two independent axes — model-quality signals like accuracy and input drift, and infrastructure signals like GPU utilisation, kernel throughput, memory pressure, and latency tails — and keeping them separate so a regression in one does not mask a healthy reading in the other. The value is diagnostic: monitoring earns its place by letting you rule causes in or out when something degrades. Which infrastructure-level signals should I monitor alongside model accuracy? Four families: GPU utilisation (how busy the compute units are), kernel-level throughput (useful work per second), memory pressure (HBM occupancy, allocation churn, transfer volume), and latency tails at p95/p99 rather than the mean. They matter together because they disambiguate each other — high utilisation with falling throughput means a regressed kernel, while low utilisation with healthy latency means a data-feed bottleneck. No single one of these tells the whole story. How can monitoring distinguish a data-drift problem from a GPU-utilisation or compute-API portability regression? By reading the axes together. If prediction-quality metrics moved, it is data or concept drift. If utilisation dropped while per-request throughput held, the serving path is starving the GPU. If kernel throughput dropped while utilisation stayed high, a compute-API or runtime path regressed. And if the regression appears on only one hardware vendor’s stack, it is an API-portability problem, not a model problem. What does monitoring reveal about whether my chosen compute API is still performing well across my current hardware fleet? A compute-API choice is usually validated once, at selection time, against a benchmark snapshot — but the fleet drifts as drivers update, models grow, and new SKUs arrive. Continuous per-target kernel-throughput and utilisation monitoring shows the moment a runtime upgrade or driver change regressed a code path on one vendor’s hardware. That turns a one-time API decision into something auditable over its whole lifetime rather than only at the moment it was made. How do I set thresholds and alerts for inference latency and cost so degradation is caught before it shows up in the bill? Alert on drift relative to a per-workload baseline rather than on absolute utilisation, which varies legitimately with load and produces alert fatigue. Watch p99 latency against the SLA, sustained kernel-throughput drops at fixed batch, and cost-per-request derived from accelerator-seconds. Anchoring alerts to baselines and to cost-per-request catches the slow, expensive regressions that never trip a hard failure and so surface first in the finance report. How does monitoring data feed back into a GPU Performance Audit to keep API and hardware decisions auditable over time? A performance audit run at deployment tells you the fleet was well-utilised and the API well-chosen at that moment; it is a snapshot. Monitoring data is the ongoing feed that keeps those findings honest — it flags when utilisation, throughput, or per-target performance drifts away from the audited baseline while the correction is still cheap. That makes API and hardware decisions auditable across their lifetime, not just at selection time. What monitoring differences matter when the same model runs across AMD, Intel, and NVIDIA accelerators? The critical addition is recording kernel throughput and utilisation per hardware target rather than in aggregate. A portability regression shows up as throughput falling on one vendor’s stack — CUDA on NVIDIA, ROCm/HIP on AMD, or oneAPI/SYCL on Intel — while model quality stays identical everywhere. If you only track fleet-wide averages, that single-vendor regression is invisible until it moves the cost report. When inference gets slower or more expensive, the useful question is never “is the model still accurate” alone — it is “which axis moved”. If you cannot answer that in hours because you never instrumented the compute layer, the answer arrives on the invoice instead. That gap between an accuracy-only view and a full inference-path view is exactly the failure class a GPU performance audit is built to close, with monitoring as its continuous feed.
