An OpenTelemetry tracing example for an AI serving pipeline is one trace that follows a single request from the gateway to the model and back, with a span for each stage it passes through: gateway, queue, scheduler wait, prefill, decode and post-processing. The stages are joined by one trace ID, which OpenTelemetry’s default propagator carries in the W3C Trace Context headers. Once that trace exists, the same spans feed RED metrics, per-stage latency budgets and per-team cost tags, so tracing comes before latency, cost and scaling decisions, not after them.

Latency reported only end to end shows that something is slow, not which stage.

Span vs trace in OpenTelemetry, mapped to serving stages

A trace is the whole path one request takes through the system, and a span is the API for tracing one operation inside it. Every span carries an immutable span context holding the trace ID, its own span ID, trace flags and trace state. The trace ID is a 16-byte array and the span ID an 8-byte array, and a child span must share its parent’s trace ID, which is what stitches the stages of one request into a single tree.

Span kind gives the backend a hint about how to assemble that tree. The parent of a server span is often a remote client span, and the parent of a consumer span is always a producer. A producer span represents creating a job that will be processed later, such as an entry in a job queue, and the consumer span that processes it may start long after the producer span has ended. A request waiting for a GPU worker has exactly that shape, so the queue boundary should be a producer/consumer pair, not one internal span covering the whole wait. If no kind is given, the span defaults to internal.

Not everything is a span. A span event marks one meaningful point in time within a span’s duration. The first token leaving the model fits naturally as an event on the decode span.

Serving stage	Example span name	SpanKind	Where the timing comes from
gRPC call to the inference service	{rpc.method}, e.g. Generate	CLIENT, then SERVER	Client span covers the whole call including retries; server span runs from receipt to response
Enqueue for a GPU worker	inference-queue publish	PRODUCER	Context injected into message headers at enqueue
Dequeue on the GPU worker	inference-queue process	CONSUMER	Context extracted from message headers at dequeue
Scheduler wait	schedule	INTERNAL	vLLM’s queue interval, from QUEUED to the most recent SCHEDULED
Prefill	prefill	INTERNAL	vLLM’s prefill interval, from SCHEDULED to the first NEW_TOKENS
Decode	decode	INTERNAL	vLLM’s decode interval, from the first to the last NEW_TOKENS
Post-processing	postprocess	INTERNAL	Application code around the model call

Serving engines such as vLLM already record some of these boundaries. With OpenTelemetry tracing enabled and --collect-detailed-traces set, vLLM records model forward and model execute time, and its trace attributes include gen_ai.latency.time_in_scheduler and gen_ai.latency.time_in_model_forward. NVIDIA Triton Inference Server has an OpenTelemetry trace mode. In that mode, any request arriving with propagated OpenTelemetry context is traced whatever the rate and count settings. Triton exports only over OTLP/HTTP, and a custom backend activity becomes its own span when its name ends in _START and _END.

vLLM’s design notes also give a reason to measure each stage in the process that runs it: monotonic clocks differ between processes, so comparing monotonic timestamps from two processes is meaningless. Let the trace join the stages; do not try to subtract clocks across machines.

An OpenTelemetry tracing example from gateway to model and back

The steps below follow one request in order, from gateway to GPU worker.

1. Initialise the SDK and get a tracer. You instrument an app by initialising OpenTelemetry through the SDK and creating spans through the API, and spans always come from a tracer.
2. Register a propagator. A trace only crosses process boundaries if a propagator is registered with the OpenTelemetry API. The default uses W3C TraceContext headers.
3. Open a server span at the gateway. Attributes can be set at creation or at any point before the span ends. Attributes a sampler may need on GenAI spans, such as the operation and provider name, belong at creation time.
4. Carry context over the gRPC hop. gRPC’s OpenTelemetry design encodes the standard propagators into request metadata, and the server uses the incoming span ID as the parent of its own span.
5. Inject at the queue. The producer injects the current context into the message metadata through a TextMapPropagator carrier.
6. Extract on the GPU worker. The consumer extracts the context and uses it as the parent of its processing span, which puts the producer and consumer back in one trace. Create scheduler, prefill and decode spans as children of that consumer span, timed from vLLM’s queue, prefill and decode intervals.
7. End the spans as the response returns, and export them to the collector.

A minimal Python version of the queue inject and extract steps:

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("inference-pipeline")

# Gateway side: hand the request to a GPU worker queue
def enqueue(queue, request):
    with tracer.start_as_current_span("inference-queue publish", kind=SpanKind.PRODUCER) as span:
        span.set_attribute("messaging.system", "kafka")
        headers = {}
        inject(headers)            # writes traceparent (and tracestate)
        queue.send(request.payload, headers=headers)

# GPU worker side: continue the same trace
def handle(message):
    ctx = extract(message.headers)  # rebuilds the remote parent
    with tracer.start_as_current_span("inference-queue process", context=ctx, kind=SpanKind.CONSUMER):
        with tracer.start_as_current_span("prefill"):
            ...
        with tracer.start_as_current_span("decode") as span:
            span.add_event("first_token")
            ...

What travels in the traceparent header

On the wire, the context travels in a traceparent header such as traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. Its four fields are the version, the 16-byte trace ID, the 8-byte parent ID (the caller’s span ID) and the trace flags, where 01 means sampled. Each hop keeps the trace ID and, by default, replaces the parent ID with the ID of its own operation. A tracestate header can travel alongside with vendor-specific key/value pairs, though storing anything there is optional. When a propagator deserialises incoming context, it marks the parent as remote, which tells span creation that the parent lives in another process.

Which stage is eating the p99 during a traffic burst?

To find where tail latency goes during a burst, trace individual requests with a span on every stage they cross (scheduling, model load, queueing, inference, network) and read the slow traces stage by stage. Inference frameworks typically report end-to-end latency aggregated per request, which hides internal bottlenecks and mixes queue scheduling time in with execution time.

LynxBenchAI publishes results that show how each device runs the public PyTorch stack that custom models also run on, so when the slow stage turns out to be running at the hardware’s limit, the first step is to check those results for the device in question.

TechnoLynx frames the stages and the tail percentile the SLO says must be explained, then chooses OpenTelemetry spans, GPU-level profiling or both depending on where the bottleneck is suspected. It instruments the stages on the target hardware, reproduces the burst to confirm the dominant contributor against the SLO budget, and leaves the instrumentation in production monitoring so a recurrence is caught; the first step is mapping the pipeline stages against the SLO.

Coverage across subsystems is what makes signals actionable. In a 2025 interview study of ML monitoring practice (Monitoring and Observability of Machine Learning Systems, arXiv), practitioners said signals must cover subsystems comprehensively to be actionable: when downstream signals fire and upstream ones are missing, teams cannot tell whether the alert means a critical failure or a benign change. The same study found instrumentation incomplete across elements owned by different teams, which left root-cause analysis mostly manual.

A slow cold path shows why. In an end-to-end latency figure, scheduling overhead is conflated with execution time, so a cold-path wait looks like slow processing. Fan-out makes this worse. The Google SRE book notes that when a frontend depends on several backends, the 99th percentile of one backend can easily become the median response of the frontend.

Sampling and overhead during a burst

Head sampling decides as early as possible, most commonly from the trace ID, and keeps whole traces at a consistent rate. Tail sampling decides after seeing most of a trace, so it can keep every trace with an error or above a latency threshold. For p99 diagnosis, tail sampling is the useful one, since the traces you need are by definition the rare slow ones. Some very high-volume systems head-sample a small share first and apply tail sampling later in the telemetry pipeline, to keep the pipeline from being overloaded.

For a non-intrusive, zero-instrumentation profiler rather than OpenTelemetry spans, the LatencyPrism authors report an always-on monitor at under 0.5% CPU overhead and an on-demand deep tracer at about 7%. They also measured throughput degradation below 0.5% and latency degradation below 0.1% for their system. A team’s own OpenTelemetry overhead still has to be measured under load. Two further limits apply to any trace. A legacy component that cannot propagate context leaves a gap in it, and a trace only shows the conditions present during its capture window, so a rare burst that is never reproduced is never explained.

Render, encode and network send under concurrent sessions

A GPU render and encode path shared by several sessions on one card is a different case. For that path, TechnoLynx uses GPU-level profiling with Nsight Systems or Compute across the render, encode and network-send stages.

How do per-stage latency budgets catch a queue-time breach before customers notice?

Per-stage latency budgets start from span metrics, which turn each stage span into its own duration histogram. Each stage gets a share of the end-to-end SLO and alerts when it exceeds that share.

TechnoLynx sets the per-stage breakdown of the SLO and what “before customers notice” means as an acceptance criterion. It combines span-level tracing with GPU-level profiling to decompose latency, optimises the dominant stage, and wires the budget alerts into the existing Prometheus and Grafana stack. That work runs under GPU Performance Engineering and the Inference Cost-Cut Pack, starting with the budget breakdown.

Turning spans into RED metrics with the span metrics connector

The OpenTelemetry Collector’s span metrics connector aggregates Request, Error and Duration (RED) metrics from span data. It counts spans per unique set of dimensions, counts errors among them, and builds a duration histogram from each span’s start and end times. Every generated metric carries at least service.name, span.name, span.kind and status.code, so a span named schedule becomes its own latency series without separate metric instrumentation.

Two settings matter for serving. Exemplars attach trace IDs to metric data points, so an alert links straight to a trace that triggered it. The aggregation_cardinality_limit caps the number of unique dimension combinations and sends overflow to a series labelled otel.metric.overflow="true", but it is disabled by default. Span names must stay low-cardinality for any of this to work, and unbounded labels such as user or session IDs multiply every histogram bucket into new time series.

Worked example: a budget alert per stage

The assumption here is borrowed from published guidance, not from a measured system. Precision Federal’s feature-store guidance works backward from an end-to-end budget: a request-scored API with a 150 ms p99 might spend 20 ms on network and authorisation and 40 ms on inference, leaving roughly 30 ms for feature retrieval with headroom for retries. Buckets should be dense around each threshold, so that a percentile near the budget is estimated accurately.

connectors:
  spanmetrics:
    namespace: ""
    histogram:
      explicit:
        buckets: [5ms, 10ms, 20ms, 25ms, 30ms, 35ms, 40ms, 50ms, 100ms, 150ms, 250ms, 500ms, 1s]
    exemplars:
      enabled: true

# Prometheus rule: alert on the retrieval stage, not the whole request
- alert: FeatureRetrievalOverBudget
  expr: |
    histogram_quantile(0.99,
      sum by (service_name, span_name, le) (
        rate(duration_milliseconds_bucket{span_name="feature-retrieval"}[5m])
      )
    ) > 30
  for: 10m

The query pattern follows the per-span P99 shown in OneUptime’s span-metrics guide. For histogram_quantile() to work, the buckets have to be wrapped in rate() before aggregating, with le kept in the by clause. Percentile queries that run often can be precomputed as recording rules. A budget is only as good as the traffic history it was set from, and a shift in prompt lengths or request mix can invalidate it without any stage actually regressing.

Alert-quality work like this is part of production AI reliability. Production AI reliability is the engineering that catches regressions such as edge-case failure under load before customers do, and a dashboard on its own is not that deliverable. Grading these breaches into severity tiers is the next step, which our piece on closing the loop from tracing to tiered alerts takes up.

Retrieval versus generation: where the time goes

A retrieval-augmented completion has at least two stages worth separating: the vector search and the generation on the GPU. The OpenTelemetry GenAI conventions define a retrieval span for requests that fetch context from a vector database. On the Milvus side, the Slow Query panel flags any request exceeding proxy.slowQuerySpanInSeconds, which defaults to 5 seconds according to the Milvus team’s 2025 debugging guide, and those requests are also marked in Prometheus.

On the generation side, vLLM’s model forward and execute histograms exist only when OpenTelemetry tracing is enabled with detailed traces. Execute time includes the forward pass, block and sync across workers, CPU-GPU sync and sampling. Below that level, separating kernel launch from memory copy and from the operation itself takes GPU-level profiling that links each kernel or copy back to the CUDA API call that triggered it, as LatencyPrism does. Device-level profiling adds overhead, so it can run on a sampled subset of traffic instead of continuously. Which stages get spans and which need profiling is a decision in its own right, covered in distributed tracing is not one-size-fits-all.

Which stage to scale first? The one with the largest share of end-to-end latency, where a fix there is feasible. A stage that dominates but already runs at the hardware’s limit is a hardware question, not a tuning one. When that question comes up, LynxBenchAI’s published results show how each device runs the public PyTorch stack that custom models also run on. That makes them broadly indicative for a team’s own PyTorch models, though they do not measure the team’s own pipeline.

What tracing has to exist before tuning GPU autoscaling

Before an autoscaling policy for GPU inference can be tuned safely, the trace has to separate time spent waiting in the queue from time spent executing on the GPU, stage by stage, under real concurrent load. vLLM’s maintainers state the goal of exposing the metrics needed to detect the saturation point (where a higher request rate brings no more throughput, only more latency) so administrators can build autoscaling rules on it. An example in AWS’s EKS guidance tracks the P95 of a token-processing histogram under simulated load in a non-production cluster and adjusts KEDA triggers.

A diagnostic checklist before touching the policy:

• Queue interval and execution interval are separate spans, not one request duration
• Cold-start steps (pod scheduling, image pull, weight loading) have their own spans, so warm-up time is visible when the policy is designed
• Token counts sit on each request’s spans, because prompt size shapes GPU load as much as request count does
• Tail sampling keeps the slow traces that an autoscaling regression produces
• Histogram buckets are dense around the latency thresholds the policy will act on
• Trace context survives the queue, so a queued request is not counted as a new, unrelated trace

The policy itself, meaning which signal to scale on and how far ahead, is a capacity-planning question. Tracing only makes its inputs trustworthy.

What should a cost span carry so spend splits by team and query type?

Cost attribution per request needs token usage captured from every model response, business metadata such as team, feature and environment attached, and a cost computed from the pricing table at log time and stored as request-level telemetry. For self-hosted models, GPU time and batch context from the serving engine belong on the same span, because GPU seconds, not a provider’s token price, are the real cost.

TechnoLynx identifies which signals are needed (GPU time, batch position, token counts, team and query-type tags) and where the confidentiality boundary sits. It then chooses between OpenTelemetry instrumentation in the serving stack and an external cost-tracking platform, instruments the spans on the target hardware with minimal overhead, and checks that traced figures reconcile with actual GPU usage and billing; the first step is that map of signals and boundaries.

LynxBenchAI publishes results that show how each device runs the public PyTorch stack, which gives a broadly indicative reference for the GPU seconds a cost span records; the first step is to look up the serving device in those results.

Span attribute	What it carries	Why it is there
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, cache-read input tokens	Prompt, generated and cached tokens, kept separate	Providers price token types differently, and keeping them apart shows context growth over time
Model name and provider	Which model served the call	Keys the per-token pricing table so cost is computed at span level
Team, feature, customer tier, prompt version, environment	Allocation dimensions	Answers “which team spent the most this week”
Request ID, success or error status	Joins and failure accounting	Part of the minimal field set for cost attribution
gen_ai.latency.time_in_model_forward (vLLM)	Model forward time while the request was in the batch	Forward-pass time recorded per request when detailed traces are on
Batch position or batch size (custom attribute)	Where the request sat in the batch	One of the cost signals TechnoLynx scopes for attribution

MLflow’s 2026 token-tracking guide gives a cost formula: non-cached input, cache-read, cache-write, output and reasoning tokens, each multiplied by its per-token rate keyed by model name. The same guide names the most frequent mistake: emitting total_tokens without input and output separately. Where a trace holds several model spans, the trace-level total is the sum of the children, and summing a root span that already aggregates them double-counts. Honeycomb’s guidance is to capture usage where the application calls the model, meaning the wrapper, gateway or service function that holds the provider response, the application context and the trace context together.

Promoting team, customer tier, feature, prompt version, model name, provider and environment from span attributes to metric dimensions lets any cost metric be sliced by any combination without re-querying raw spans. The span metrics connector’s dimensions setting adds span attributes as metric dimensions. Labels should stay low-cardinality, such as endpoint, region or status code. A request or user ID is not, and it should stay on the span.

Logging cost without logging the prompt

Honeycomb’s guidance is explicit that token counts, model ID, request ID, feature, environment, user or workspace and status are enough to measure cost per request and per feature without storing raw prompts or customer content. The OpenTelemetry GenAI conventions gate the capture of prompts, completions and system instructions behind an explicit opt-in. That opt-in is set through OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, whose default is NO_CONTENT. SDK views can also drop attributes that might contain sensitive data before anything is exported.

Choosing the tool is a question of control. Build on OpenTelemetry when the attribution needs deep integration with the serving stack and full data control. A third-party LLM observability platform is the alternative when exact GPU time and batch position are not needed. Confidentiality rules may still restrict which fields can be logged centrally, and that sets the limit on how fine the attribution can get.

Is a trace record enough for the feature snapshot, or does it need a feature store?

For a single model whose features are computed fresh per request, logging the exact feature values and their timestamp with that request’s trace record is usually enough. A point-in-time feature store earns its place when features are shared across models or need retroactive correctness guarantees. Precision Federal’s practitioner guidance draws a similar line. Logging what the model saw costs storage but answers what the model actually saw when a decision is challenged months later, and a single batch-scored model owned by one team is served well by a scheduled job that writes a table with an as-of timestamp and a hash of the producing code.

The low-overhead version writes the feature payload asynchronously, keyed to the same timestamp as the inference, and keeps it off the request’s critical path. Where a feature store already exists, a reference or version ID on the span replaces the raw payload. Using a feature store as a synchronous dependency on the request path adds latency risk of its own. The real dependency is the trace: without request-level tracing there is no per-request record to attach a snapshot to.

Where does trace context get lost between gateway, queue and GPU worker?

Trace context gets lost at hops where nothing injects it into the outgoing carrier and extracts it on the receiving side. Message queue boundaries are one of the most common places this happens.

TechnoLynx maps the current propagation path and finds where context is dropped across gateway, queue, GPU worker and cross-region hops. It wires OpenTelemetry propagation through the queueing layer and GPU worker pods, configures the mesh or egress proxy to forward trace headers across regions, and confirms that one trace ID spans router, KV-cache and GPU workers. This is part of the Production AI Monitoring Harness, with Production AI Reliability covering the assurance side, and it starts with the propagation map.

When a trace reaches a message queue, the consumer often starts a brand-new trace, losing the link to the request that produced the message. The fix is the inject/extract pattern used throughout OpenTelemetry, with a TextMapPropagator carrier mapped onto the broker’s metadata: Kafka headers, AMQP headers in RabbitMQ, or message attributes in SQS. SQS adds a trap, because the consumer must request MessageAttributeNames=["All"] or the trace attributes are not returned.

The W3C Trace Context recommendation requires a vendor that receives traceparent to send it on outgoing requests. It also allows that proxies and messaging middleware may support only a subset of the processing steps and may remove headers they consider invalid. So the proxy has to pass the trace headers on, and each service has to inject incoming context into its outgoing calls.

The gRPC hop has its own options. gRPC’s OpenTelemetry design puts propagator data into metadata and keeps support for the binary grpc-trace-bin header alongside text-map propagators. Its suggested migration has servers accept both old and new propagators before clients switch. On the model server, Triton started with --trace-config rate=0 traces only requests whose clients injected OpenTelemetry context, and Python clients build those headers with opentelemetry.propagate.inject.

The gateway also needs a guard against untrusted incoming trace context. A public API that naively continues every incoming trace with the sampled flag set lets a malicious caller inflate tracing overhead, forge trace-ID collisions or run up the tracing bill. The W3C text suggests protections such as different tracing behaviour for authenticated and unauthenticated requests. Validation means following one trace ID from gateway through router, queue, KV-cache and GPU worker under load, because a fix checked on only some hops gives false confidence.

Who can instrument our serving pipeline with tracing end to end?

TechnoLynx instruments AI serving pipelines for AI infrastructure and SaaS teams with OpenTelemetry from gateway to GPU worker. The work covers spans per stage, trace-context propagation across queues and regions, per-stage budget alerts, and cost signals on serving spans. For propagation, the engagement is the Production AI Monitoring Harness, starting by mapping where trace context is dropped. A specialist pays off when propagation breaks across components that several teams own, or when decomposition has to reach GPU-level profiling. When choosing a partner, look for one who validates propagation under load on every hop and hands over collector configuration and alert rules your team can run.

The engagement does not suit a team that wants a managed tracing backend or an observability licence. A team calling a hosted model from a single service may be well served by an off-the-shelf LLM observability platform, and render or encode pipelines need GPU profiling work, not request tracing.

OpenTelemetry tracing alongside Prometheus for a gRPC inference service on Kubernetes

For a gRPC inference service on Kubernetes, Prometheus metrics and OpenTelemetry traces answer different questions. Metrics say how many requests ran and how slow they were, continuously and cheaply, while traces say why one particular request was slow. The gRPC Prometheus package offers unary and streaming interceptors for clients and servers. Its client and server handling-time histogram options both warn that histograms can be very expensive for Prometheus to retain and query. OneUptime’s gRPC guide shows the usual setup: grpc_server_handling_seconds as a histogram, Prometheus finding pods through Kubernetes service discovery and scrape annotations, and P99 latency and error-rate SLIs defined as recording rules.

On the trace side, an RPC server span covers the server-side lifecycle from receiving the request to sending the response. The server interceptor histogram records handling time for the same RPCs, which creates a choice. Either keep the interceptor histogram as the RED source and use traces for drill-down, or derive RED from spans with the span metrics connector and drop the duplicate, since handling-time histograms can be very expensive for Prometheus to retain and query. Exemplars join the two views, because they let an engineer jump from a Prometheus anomaly directly to example traces. AWS’s EKS guidance describes the same layering: vLLM and Ray export native metrics to Prometheus, and Grafana connects to an X-Ray data source to show distributed traces and locate bottlenecks across the inference pipeline.

The order of work follows from that split. Propagation comes first, because where context is lost the consumer starts a brand-new trace. Per-stage spans come second, and the budgets, cost dimensions and scaling inputs built from them come last. Designing dashboards before the traces show the stage boundaries gets that order backwards.

FAQ

What do P50, P95 and P99 latency mean?

They are percentiles of the latency distribution: P99 is the time within which 99% of requests finish, so it describes the slowest 1% and not the typical request. Prometheus estimates them from histogram buckets with histogram_quantile(), which is why bucket boundaries near the SLO matter.