A single distributed tracing rollout will not cover every instrumentation need in an ML serving pipeline. ML observability and audit trails split into several separate jobs, and each one answers a different question. A trace represents one end-to-end request, with spans timing each operation inside it. A metric shows whether the service is degrading under load. A heartbeat shows whether the computation is still running at all. An approval or audit record gives durable, verifiable evidence that can be shown to a regulator or auditor later. OpenTelemetry defines traces, metrics and logs, and its Collector accepts all three on one receiver. So the method is to start from the decision each signal has to support, then choose the instrument, rather than rolling out tracing and hoping the rest follows.

Why one tracing rollout does not cover the serving pipeline

As MLOps engineering teams learn, production ML systems fail silently, through wrong decisions rather than crashes. A focus-group study of 37 participants across six organisations in banking, fintech, industrial technology and e-commerce found that all teams emitted metrics throughout the pipeline, from ingestion through feature computation to predictions, to catch faults close to where they originate. Even so, the same study found instrumentation still incomplete, particularly where stages are owned by different teams, with provenance metadata detached from monitoring and root-cause analysis left mostly manual.

Coverage cuts both ways. When downstream signals fire with no upstream signal, teams cannot tell a critical failure from a benign change, and when upstream signals fire with no downstream impact signal, they cannot tell whether the issue matters. Broad instrumentation also brings its own false positives, and one team in the study only treated a shift as real when several signals moved together, such as a feature’s NULL rate together with prediction divergence.

Tracing itself does not arrive in one step either. MLflow’s guidance on context propagation recommends a phased rollout: first a global propagator and traceparent on every outbound LLM HTTP call, then queue and background-task carriers locked in with tests, then orchestrator-level spans where deeper detail is needed. Once tracing alone takes three phases, expecting it to deliver liveness detection, SLA alerting and audit evidence as well is a scoping error.

Trace, metric, heartbeat or approval record: deciding per signal

The quickest way to scope instrumentation is to write down the decision a signal must support, then read across this table.

Decision the signal supports	Instrument	Why this one	What it will not tell you
Which stage made this request slow or wrong	Distributed trace with per-stage spans	A trace is one end-to-end request; each span is a timed operation within it	Whether the process has silently stopped
Is the service degrading under load	Metrics: latency percentiles, queue depth, utilisation	Percentile thresholds capture the slow tail across requests	Which individual request failed
Has the computation stopped entirely	Heartbeat or liveness signal inside the loop	Absence of the signal is the alarm	Whether outputs are correct
Did a model update cause this regression	Version attributes on spans, grouped by release	Separates cohorts by what actually ran	Causation, unless configurations are replayed one component at a time
Who approved this release, and can we prove it later	Tamper-evident approval or audit record	Built for evidentiary weight, not query speed	Real-time system health

Several needs land on more than one row. On-call staff who must tell a stall from a slowdown in the same incident need the heartbeat and the latency dashboard side by side.

Correlation ID vs trace ID vs span ID: which one to propagate

A correlation ID is an identifier the application chooses, often carried in an X-Correlation-ID header, that groups related log lines under one key. It is a convention rather than a standard, and it has no parent-child structure. A trace ID identifies one end-to-end request, and a span ID identifies one timed operation inside that trace. Together with a sampling flag they make up the trace context that travels between services.

Use W3C Trace Context (traceparent and tracestate) as the serving pipeline’s identity format. The OpenObserve guide on signal correlation states the reason plainly: logs, traces and metrics live in different systems with no shared identifier by default, and the fix is a shared trace_id injected into log records, attached to metrics as exemplars and propagated across services. Inventing your own correlation scheme on top means rebuilding what OpenTelemetry already standardises: the propagator, the log injection and the exemplar support.

There are two limits to know about. The OpenObserve guide notes that logs are usually unsampled while traces are often sampled at 1 to 10 percent, so most trace_id values in logs point to traces that were never stored. Older Fluent Bit, Vector and Logstash configurations also often parse only the message field and drop the trace fields on the way to the backend. Our view is that the trace ID should still be stamped into every log line, because it stays a useful join key for logs even when the trace itself was sampled out, and that the log shipper’s field mapping should be checked before anyone trusts the join.

Adding OpenTelemetry spans across intake, preprocessing, inference and postprocessing

The working pattern is one trace per user request, one span per logical stage, and semantic attributes on those spans for debugging, cost and analysis. In the freeCodeCamp walkthrough of a FastAPI service, an auto-generated http.request span acts as the root, with child spans such as rag.retrieval, rag.prompt.build, llm.call, llm.postprocess and an optional llm.eval, each standing for a logical unit of work rather than an implementation detail. For a non-LLM model the same shape maps onto intake, preprocessing, inference and postprocessing.

Span boundaries matter as much as span names. The same guide warns against one giant span around the whole workflow, a span per token, or dumping everything into logs, and recommends a few coarse spans enriched with attributes, with events marking milestones inside a span. The LLM call span is the critical one: start it just before the API request, end it only when the full response or stream completes, and record retries, timeouts and errors inside it.

Where traces break: queues and async workers

HTTP-to-HTTP tracing is a solved problem. Queues are where traces fall apart, because a message queue has no equivalent of an HTTP header, and unless traceparent is written into the message metadata the consumer starts a new, disconnected trace. The fix is to inject the current context into a trace_context field on the message envelope at publish time and extract it in the worker before the first span starts, so the worker’s span attaches as a child of the original request. Black-box frameworks add a second gap: auto-instrumentation sees the outer call and misses the tool calls and retries inside it.

Propagation should be proved by tests rather than assumed. The recommended tests are unit tests asserting that traceparent exists on every outbound call, plus integration tests that fetch the finished trace and check that the LLM call’s span lists the orchestrator span as its parent, not as a sibling root.

Should every model request and response be logged with a correlation ID?

Logging every request and response against a trace or correlation ID is the right default where privacy rules allow it, and several managed platforms now do part of the work natively. Where they stop, a thin wrapper fills the gap.

• OpenAI API: returns a server-generated x-request-id and accepts a client-supplied X-Client-Request-Id (ASCII only, at most 512 characters), which can carry your internal trace ID. OpenAI logs it for supported endpoints so you can trace requests that timed out before a response header came back.
• Azure API Management: gateway LLM logging writes requests and responses, including chunked messages, as separate entries in ApiManagementGatewayLlmLog, which you join with the CorrelationId field.
• Databricks inference tables: capture requests, responses, HTTP status codes, model run times and traces into a Unity Catalog Delta table, with a 1 MiB payload limit and no guarantee of logging for some error responses.
• Amazon SageMaker Data Capture: logs endpoint inputs and outputs to Amazon S3 for training, debugging and monitoring.

The Databricks error-case gap matters most, because that is where an incident investigation needs the record. A wrapper around the client call that sets the vendor’s request ID from the active trace ID, and logs failures itself, closes both gaps. Two cautions apply. High-volume systems should sample traces and hash prompts and responses by default, and traces should be treated as production data with access control and retention. Some teams handling sensitive client data cannot store inputs at all and fall back on weaker signals such as user feedback.

Tagging traces with the deployed model version so regressions show per release

A release SHA alone cannot explain a regression when the prompt alias moved, the index was rebuilt and the provider started serving a different model revision in the same window. The fix is to attach the resolved version of every component that changes behaviour, recorded at the stage where it is used: the application release and a manifest digest on the request span, the prompt revision on generation, the index revision on retrieval and the evaluator version on evaluation.

Record the resolved value, not the alias. A prompt labelled production is a convenience rather than an identity, and where a provider does not expose its serving revision the guidance is to record an explicit unknown rather than invent one. The OpenTelemetry GenAI attributes help here: gen_ai.request.model records what was asked for and gen_ai.response.model what came back, which is how you notice a provider silently rolling versions. In MLflow, each trace carries version metadata as tags, and assessments attached to traces allow quality comparison across versions.

Two checks keep version tags honest. Comparisons should group failures by manifest digest against a previous stable configuration while controlling for request class and input size, because a new version that receives harder traffic can look worse without having regressed. Every serving path (direct HTTP, queue worker, fallback model, cache hit) should be exercised to assert the fields exist, and the missing-version rate tracked as an instrumentation health metric.

Telling a silent stall from slow-but-alive inference

The signal that separates a stall from slow inference is the loss of an internal heartbeat, not an external response-time cutoff.

TechnoLynx builds this as part of the Production AI Monitoring Harness: the first step is framing which failure modes on-call must tell apart (stall against slowdown, queue backlog against accelerator saturation) and the latency baselines on your target hardware, before choosing where the heartbeat probes and exporters go.

A heartbeat is the simplest liveness mechanism: a thread or node signals that it is alive, and if the signal does not arrive within the expected window, the receiver assumes failure. Aeron’s documentation describes the Phi Accrual failure detector, which compares current heartbeat inter-arrival times against their mean to estimate the probability that a process has failed, an approach that adjusts to load more adaptively than strict heartbeat timeouts.

The Tarragon inference design treats tokens exchanged between workers as implicit heartbeats and sends an explicit probe only when a data connection stays silent past a configured interval, avoiding unnecessary probing under normal load. Polling has a cost here: a study of Kubernetes health signalling found that one liveness-probe configuration cut availability by 4% through incorrect failure detections, while a signal-based approach detected container failure in 0.1 s, 0.6 s faster than that configuration. The same study notes that frequent probes can wrongly detect failure during a temporary rise in response latency, which is precisely the slow-but-alive case.

Our recommendation is to emit a heartbeat from inside the inference loop (per batch, per decode step, or per frame), page only when it has been silent for several expected intervals, and alert on slowness separately against the step’s own latency distribution. Keep in mind too that a heartbeat proves the loop is alive, not that its outputs are right.

A fleet observability dashboard on-call staff can act on

For degradation under load, the answer is a dedicated metrics pipeline (GPU or robot telemetry exporters feeding Prometheus and Grafana or an equivalent) that puts queue depth, per-stream or per-unit latency, utilisation and task health signals such as dropped frames or detection confidence on one fleet observability dashboard. That is a metrics job, not a tracing job.

GPU utilisation needs care on that dashboard. NVIDIA defines it as the percentage of the sample period in which one or more kernels was running, so a single small kernel on one streaming multiprocessor can read 100%. The DevOpsBeast guide to DCGM and Prometheus gives a triage rule: graph tensor-pipe activity against DRAM activity, where both low while utilisation sits at 100% indicates a stall from small kernels, launch overhead or a host-side bottleneck. Those profiling counters are not in the default dcgm-exporter counter set on every install and must be enabled explicitly. With Kubernetes mapping on, the exporter attaches pod, namespace and container labels to each GPU metric, which gives you per-workload attribution. The guide also notes that thermal and power throttling give no application-level signal, and that without XID alerting GPU faults surface far from their root cause.

Thresholds should come from the deployment’s own baseline and severity tier rather than one global cutoff, and they need recalibrating after a hardware or model change. A dashboard with correct metrics still pages the wrong people if nobody has mapped alert severity to an accountable owner. This is why we treat production AI reliability as its own engineering discipline, distinct from model accuracy, where the dashboard is only one artefact and not the deliverable.

Who can build stall alerting and an on-call GPU dashboard for our serving pipeline?

TechnoLynx builds in-loop heartbeat probes, exporters, dashboards and alert rules for GPU inference and robot fleets, with thresholds validated against false-positive data. The Production AI Monitoring Harness starts by framing the failure modes and hardware baselines. It then chooses instrumentation points (in-loop probes, DCGM or custom exporters, metric cardinality), implements exporters, dashboards and alert rules with low overhead on the target hardware, and validates thresholds against your incident and false-positive history. For running the stack afterwards, with runbooks mapping severity to accountable owners, Production AI Reliability covers the operating side.

This work does not suit every team. Teams already on SageMaker or Databricks should start from those platforms’ native capture before adding anything bespoke.

Turning an end-to-end SLA into per-stage budgets and consequence tiers

An end-to-end SLA becomes something you can operate on only when it is split into a latency budget per service along the critical path, with each alert tier tied to a consequence and a named owner.

Latency budgets where a delay has safety consequences

Google’s SRE workbook recommends latency SLOs with several thresholds, for example 90% of requests under 100 ms and 99% under 400 ms, because a single average hides the slow tail that makes users unhappy. It also favours rolling windows over calendar months, sized in whole weeks. Applied to a patient-facing triage pipeline, we would decompose the end-to-end target across the call chain into percentile budgets per stage, give the tightest budgets and fastest paging to the safety-critical path, and accept that the arithmetic is only as good as the critical-path map: a missed dependency breaks it. For LLM stages, the Web Developer guide argues against an SLO on total response time and proposes time-to-first-token P95 plus stream success as the SLIs, since total time varies legitimately with prompt complexity.

Alert tiers mapped to consequence, with named owners

Burn-rate alerting separates real degradation from noise by paging on how fast the error budget is being spent, not on every breach. The Web Developer guide’s example pages when more than 50% of a monthly TTFT budget is consumed in under seven days. Burn rates need a working SLO and enough traffic history to be reliable. In a legal review pipeline, a possible privileged-document miss warrants a faster tier than a routine quality issue, though it can tolerate more latency than a real-time safety case. Every tier needs exactly one accountable owner, because shared ownership becomes no ownership in the middle of the night. Severity tiers are a judgement to revisit as incident data accumulates. The escalation design itself, from tiered alerts through to tamper-evident escalation records, is covered in depth in closing the loop from tracing to tiered alerts.

An SLA for the human-in-the-loop exception queue

A human review queue stalls silently unless both its age and its depth are measured against an SLA set by the business cost of delay, such as an invoice payment deadline. We would alert on the oldest item’s age as well as the queue length, since a short queue with one very old item is still a breach. Reviewer throughput belongs next to model accuracy as a production metric, because a model that routes more cases to humans can look accurate while the queue behind it grows. Queue alerting surfaces a stall faster. It does not fix the cause.

Which stage is eating the latency budget?

To find the stage that eats turnaround time under load, instrument each stage separately for latency and throughput and rank the stages by their contribution to tail latency. End-to-end SLA compliance alone cannot show where a backlog forms.

TechnoLynx does this by mapping the pipeline into discrete stages, instrumenting each on the target hardware with minimal overhead, and validating that the attribution matches observed SLA misses under realistic load.

If spans already exist per stage, much of the data is there: p95 latency can be derived from spans. In batch pipelines, Nextflow’s execution timeline splits each task into scheduling time, real execution time and termination time, and its job-duration plots report how long each process took. The scheduling component separates time spent waiting to start from real execution time. Where the backlog follows particular batch compositions or input sources, a finer breakdown inside the dominant stage (per station, per camera, per batch size) is the next step.

Treat per-stage timing as a measurement with limits. Instrumentation adds overhead of its own, as even DCGM’s profiling subsystem does. When stages run concurrently, a correlation between one stage’s latency and SLA misses is not proof that this stage caused them. And the ranking holds only for the current hardware, batch sizes and workload mix, so a change in any of them calls for re-profiling. Per-stage visibility tells you where to optimise. It does not do the optimising.

When a signal needs an approval record instead of a trace

Observability and audit answer different questions. Observability monitors behaviour in real time so the team can fix problems before users notice. Audit produces durable, verifiable evidence of what the system did, on what data and with which model version, for a regulator, auditor or court. The storage is different too: observability lives in a mutable database built for fast queries, while audit lives on a tamper-evident substrate. An audit record holds model and dataset fingerprints, input and output hashes and timestamps, anchored to a hash chain, append-only ledger or notary service, so only the fingerprint has to leave your environment.

The practical test is whether anyone outside the engineering team will ever need to rely on the record. A trace with version attributes lets you group failures by release and debug a regression. It is not a sign-off record, and observability data lives in a mutable store rather than a tamper-evident one. Deployment approvals and evidence for regulated release decisions therefore belong to approval evidence work, not to the tracing rollout. Where the two meet is the version identifier: if the manifest digest on the request span matches the one in the approval record, an investigator can move from a bad output to the decision that let that configuration ship.