Head-based sampling cannot preferentially keep error or slow traces, because it makes the keep-or-drop decision before either outcome exists. The OpenTelemetry Collector’s tail sampling processor can keep them, because it decides after the trace has finished. One worked example shows the size of the gap. Sample at 1% at the head, and a payment failure that hits a 50,000-request-a-day service about 50 times a day leaves a kept trace roughly once every two days. For extremely high-volume services, one answer is a hybrid. Head sampling cuts the obvious volume cheaply, then tail sampling chooses from what is left.
Through its production AI reliability work, TechnoLynx first quantifies current and projected request volume, trace size and tenant SLA tiers against target cost bounds. From there TechnoLynx chooses head, tail or hybrid sampling per tenant class and designs the retention tiers to match.
Head-based vs tail-based sampling: when a head rate alone is enough
Google Cloud Trace’s documentation draws the line clearly. In head-based sampling the decision is made when the component processing the span receives the request. In tail-based sampling the decision waits until the whole trace is available. The most common form of head sampling, consistent probability sampling, decides from the trace ID and a target percentage. That keeps whole traces intact at a steady rate such as 5%. It costs almost nothing, because dropped traces never create spans.
That cheapness has a price. A fixed rate keeps errors, rarely used routes and health checks in the same proportion, so the noisy traffic you care least about makes up most of what you store. In the Hindsight benchmark (Zhang et al., NSDI 2023), Jaeger’s 1% head sampling captured about 1% of edge-case traces, because it cannot tell them apart from normal requests.
Tail sampling looks at the assembled trace. Its documented uses include always sampling traces that contain an error, sampling on overall latency or span attributes, and applying different rates to low- and high-volume services. OpenTelemetry’s own guidance says that larger systems which have to sample almost always need tail sampling to balance data volume against usefulness.
Which sampling mode fits which requirement
| Requirement | Approach | Why |
|---|---|---|
| Control cost with representative visibility; healthy, uniform traffic | Head sampling at a fixed low rate | Cheapest option, no buffering, whole traces kept consistently |
| Never lose errors, slow requests or SLA-tenant traces; volume small enough to buffer every trace | Tail sampling only | Decisions see the full trace, at the cost of storing every span until the decision |
| The same guarantee at high volume | Hybrid: head to cut obvious volume, tail to select what is kept | Protects the telemetry pipeline from overload while keeping rule-based selection |
| Many tenants or label dimensions, in any mode | Add per-tenant limits and tiered retention | Without them one noisy tenant can exhaust collector memory and degrade observability for every other team |
What does 100% tracing cost, and how much does sampling save?
A published cost model gives a mid-size baseline. At 10,000 spans per second with 30-day retention, it estimates about $4,136 a month, of which self-hosted backend storage is $1,156. At 100,000 spans per second the same model scales infrastructure cost by roughly 8 to 10 times, which is not quite linear because batching gets more efficient. Infrastructure cost, in other words, follows span volume. SampleHST’s authors (2022) describe production servers that each produce several tens of gigabytes of trace data a day, adding up to terabytes across all servers. Most of those traces contain no anomaly.
OneUptime calls sampling the single most effective lever and estimates that a 10:1 ratio cuts storage and network cost by 90%. It adds that tail sampling in the Collector can keep 100% of error traces while dropping routine successful requests. OpenObserve gives a typical policy (keep every error, sample healthy traffic at 5 to 10%) and its working: with a 1% error baseline and 5% of healthy traffic sampled, roughly 1% + 5% × 99% ≈ 5.95% of traces are stored.
Tail sampling does not make the upstream bill disappear. The application still traces every request and pays to generate and ingest all of it. Tail sampling mainly lowers what gets committed to storage. The Hindsight paper itemises the cost of that upstream work:
- throughput can halve when tracing libraries lack optimisation;
- up to 200 MB/s of trace traffic per node;
- about 100× the collector capacity of 1% head sampling, even with perfect horizontal scaling.
The same paper measured 14% lower peak throughput for tail sampling than for head sampling. Coherent edge-case capture fell from about 100% at 1,000 requests per second to 71% at 2,000 and 28% at 3,000, as backpressure set in. OpenTelemetry’s documentation warns that tail samplers can need dozens or hundreds of stateful nodes, and may have to fall back to cheaper techniques when they cannot keep up.
The guarantee holds only if the rules are right. In our view, “slow” and “error” thresholds should be set per tenant and per SLA tier rather than globally. A single latency threshold will over-keep a tenant whose normal path is slow and miss breaches for a tenant whose SLA is tight.
Hybrid head and tail sampling with per-tenant caps
A hybrid head and tail sampling design keeps every anomalous or SLA-flagged trace that reaches the tail tier, samples routine traffic down hard, and puts a cap on each tenant before storage. That way growth in tenant count does not turn directly into growth in the storage bill.
TechnoLynx builds and tunes this layer on the target infrastructure: the sampling topology, buffer sizes, per-tenant cap thresholds and retention rules. TechnoLynx then runs burst and multi-tenant load tests to check that flagged traces survive and that storage grows more slowly than tenant count.
The arithmetic of the hybrid follows from the definitions. If the head keeps a fraction h of traces and the tail keeps a fraction t of what reaches it, the stored fraction is h × t. Error traces are different. A tail rule can keep every error it sees, but it only sees the errors the head let through, so error retention can be no higher than h.
How low can the SDK head rate go before tail sampling goes blind?
Head sampling in the SDK limits what the tail tier can find. Parseable’s guide says it plainly: if an SDK drops a trace through head sampling, the Collector never receives it and cannot keep it later when a child span reports an error. The OpenTelemetry probability-sampling specification enforces the same direction. Downstream samplers may lower the sampling probability, but raising it again would introduce a statistical error.
How low production rates go is a matter of record. Jaeger enables head sampling by default at 0.1%, and some production systems sample as few as 0.001% of requests. At those rates the tail tier receives only that fraction of traces, failures included.
Our answer is to treat the head rate as a per-flow setting rather than one global knob. Keep SLA-covered tenants and critical routes at 100% at the head so the tail rules see all of their traffic. Apply aggressive head rates only to high-volume, low-value services, which is the pipeline-protection case OpenTelemetry describes. With parent-based sampling the root span settles the decision for the whole trace. In practice that means the exemption has to be applied at the service that creates the root span.
The head rate also changes how metrics have to be computed. Elastic documents that when head and tail sampling are combined, its metrics are calculated from head-sampled events and weighted by the inverse of the head rate to estimate the full population.
How does the tail sampling processor decide what to keep, and why must every span reach one collector?
The tail sampling processor evaluates its policies as an OR: a trace is kept if any policy says keep, for example every error, every trace slower than two seconds, and 10% of everything else. Grafana’s example keeps every trace above 5,000 ms with a single latency policy. The OR composition has a further consequence. Decisions are plain keep or drop with no probability attached, so counts computed from the kept data no longer reflect real traffic. Parseable’s fix is to branch the full span stream into span metrics before tail sampling. A policy that keeps every error but only 5% of successes would otherwise report a heavily biased error rate.
Every span of a trace has to reach the same Collector instance. The processor buffers by trace ID in memory, so a span that lands on another replica is invisible to the decision. Uptrace calls this the most common way tail sampling fails in production, and it fails quietly, with no pipeline errors. Putting several sampling Collectors behind a round-robin Kubernetes Service produces exactly this split. OpenObserve recommends a gateway over a sidecar for tail sampling, which needs full traces.
Grafana’s documented fix is a two-layer collector topology:
- Stateless first layer. It enriches data, exports metrics and logs, and load-balances traces with the
loadbalancingexporter. A plain layer-4 load balancer is enough in front of it. - Trace-ID routing. The load-balancing exporter routes spans by
traceID, so the same trace always reaches the same backend, which is what stateful processing such as tail-based sampling needs. - Stateful second layer. It runs the tail sampling processor and metrics generation, then exports sampled traces.
Two details decide whether this holds up. With the DNS resolver, instances can briefly disagree on the ring after a refresh and send spans of one trace to several hosts. The Kubernetes resolver narrows that window. Routing by OpenTelemetry randomness value groups related traces (a session, a workflow) on one backend for group-level decisions. However, the traffic behind one key has no upper bound, so backends have to be sized for the largest group. Root spans matter too. Elastic’s APM Server decides when the root transaction ends, and silently drops every event of a trace whose root never arrives.
Why the tail sampling processor uses so much memory, and how to size decision_wait and num_traces
The processor buffers 100% of incoming traces before it applies policies, so memory grows in proportion to span volume. Two settings bound that memory: num_traces, the number of trace IDs held, and the per-trace span count. Neither is a byte limit. To estimate the footprint, multiply num_traces by average spans per trace and by span size.
When the buffer fills before a decision is due, the oldest pending trace is evicted and counted in otelcol_processor_tail_sampling_sampling_trace_dropped_too_early. Watch that metric. When it rises, the processor is removing pending traces before their sampling decision.
Worked example: sizing num_traces
Parseable’s starting formula is num_traces ≥ peak new traces/second × decision_wait (s) × safety factor. For 2,000 new traces per second, a 15-second wait and a safety factor of 1.5, that gives 2,000 × 15 × 1.5 = 45,000 traces, rounded up to 50,000 to absorb short bursts. The real footprint still depends on spans per trace, attributes, events, links and payload size. A load test with representative traces should include receivers, batches and exporter queues. A memory limiter protects the Collector but cannot stop incomplete traces when the tier is undersized.
decision_wait sets how long a trace may take to assemble. Late spans are the reason it cannot be very short. One adaptive tail-sampling processor waits an extra decision_delay (2 s by default) for stragglers after its 30-second timeout or a root span makes a trace ready. We would set the wait from the duration of the slowest trace you are obliged to keep, not from the median. A wait that is too long raises the required num_traces in direct proportion, through the formula above.
There are ways to buy more headroom. Elastic contributed two upstream changes that cut tail sampler memory by up to 65%. A span-ingest strategy releases traces before decision_wait elapses, and a Pebble-backed storage extension buffers on disk instead of in RAM. That lets operators raise decision_wait and num_traces without OOM kills, at roughly 2× the CPU. Overflow behaviour is worth checking too. Elastic’s APM Server silently drops trace events whose TTL expires before a decision. When its storage limit is reached, traces by default bypass sampling and are all indexed, which can overload Elasticsearch.
How do per-tenant limits and tiered storage keep the bill from tracking tenant count?
Per-tenant rate limiting keeps one team’s burst from degrading observability for every other team. OneUptime describes the failure: a team with a retry loop jumps to 500,000 spans per second, the shared collector’s memory spikes, and every team loses data at the worst possible moment. Their remedy combines routing connectors, the tail sampling processor’s rate_limiting policy per tenant pipeline, and a memory limiter in each routed pipeline. OneUptime’s rule of thumb is to measure a week of per-tenant span rates and set the steady limit at about 2× normal peak and the burst at about 2.5×.
Grafana Tempo enforces rate limits at the distributor, plus per-tenant live-trace counts and per-trace size limits downstream. Spans over a limit are discarded and counted in tempo_discarded_spans_total. Its runtime overrides set retention and ingestion per tenant, for example block_retention: 24h, rate_limit_bytes: 4000000 and max_traces_per_user: 10000 for one tenant.
We would evaluate the force-keep rule before a tenant’s rate limit, and tune each cap per tenant rather than set one global value. A cap that is too tight can still drop a real burst of distinct error signatures. Caps on label cardinality are the bluntest tool here, because merging dimensions can remove exactly the attribute you need to diagnose a rare fault.
Tiered retention covers the rest of the bill. OpenObserve recommends short hot windows at full fidelity, with sampled or aggregated data kept longer for trends. Groundcover’s log storage guide starts from 7 days hot, 30 warm and 365 cold. It puts the per-GB spread between hottest and coldest object classes at roughly 9 to 23× per cloud, and 50 to 100× once replicated SSD hot tiers are counted. The cost is query speed: Elastic’s benchmark shows 92 ms on hot storage against 6,257 ms for an uncached first frozen-tier run of the same 4 TB. Size hot and warm windows so incident work never depends on a cold read. Many platforms tier by age rather than content, so a longer window for error traces can mean a longer window for everything in the same index.
Worked multi-tenant case: three tiers at 16,384-GPU scale
The TempoTrace research design for AI infrastructure tracing is a useful reference, though not a Collector configuration. It runs always-on 64-byte metrics at 100% coverage, phase-aware sampling at 1% for compute and 10% for communication, and full tracing triggered at a 3σ anomaly, with overhead reported at ≤1.18%. Retention follows the tiers: metrics indefinitely (about 12 GB/day), sampled traces for 30 days (about 105 GB/day compressed) and triggered traces for 90 days, around 3.3 TB/day in total, with a 2 GB per-node ring buffer absorbing at least 15 s of triggered tracing. Sampling runs independently per tenant. A trigger on one tenant raises tracing only for that tenant, so a noisy tenant cannot push overhead onto its neighbours. That isolation is the property per-tenant limits in a Collector pipeline are trying to reproduce.
Proving the guarantee before you rely on it
The configuration alone proves nothing about the guarantee. Prove it under load. TechnoLynx injects tagged error, slow and SLA-tenant traces under burst and multi-tenant load and checks they are retained at the required rate. At the same time TechnoLynx measures storage growth and ingestion cost against budget. The Hindsight results show why to test at peak rather than average load. Tail capture that looks perfect at 1,000 requests per second degraded badly by 3,000.
The tail sampling processor can only keep spans that reach it. Which stages of a serving pipeline need their own spans is a separate design question, covered in instrumenting the serving pipeline with distributed tracing. Get the instrumentation right first, and the sampling rules have something worth selecting from.
