Why tail-based sampling is the only sane default for high-volume tracing

The sampling problem in distributed tracing

At any meaningful production scale, you cannot store every trace. A service handling 10,000 requests per second will generate tens of millions of spans per minute. Storing all of them is technically possible but economically absurd — the cost of full-fidelity tracing at high volume can easily exceed the cost of the infrastructure you're trying to monitor.

So you sample. The question is: when do you make the decision to keep or discard a trace, and what information do you base it on?

How head-based sampling fails you

Head-based sampling — the default in most OpenTelemetry SDK configurations — makes the keep/drop decision at the root span, before any child spans have been created. The logic is simple: flip a coin, keep 1% of traces, drop the rest.

The problem is catastrophic for production debugging: you sample before you know the outcome. The 0.3% of requests that hit your slow database query, the 0.01% that trigger an edge-case error — these are exactly the traces you need, and they get discarded at the same rate as the boring healthy traffic.

# ❌ Head-based sampling — do NOT use this as your only strategy
# in sampler config (e.g. OpenTelemetry SDK)
sampler:
  type: TraceIdRatioBased
  ratio: 0.01   # 1% — but decided BEFORE the trace completes

What tail-based sampling actually does

Tail-based sampling flips the model: every span is collected and buffered, and the keep/drop decision is made after the entire trace is complete (or after a configurable timeout). At decision time, you have full information:

• Did any span in this trace return an error status?
• Did any span exceed your latency SLA?
• Was this a rare code path (low traffic operation)?
• Did a downstream service return a 5xx?

You can now keep 100% of error traces, 100% of slow outliers, and sample down only the healthy, fast, boring majority. The result: your stored traces are signal-dense — every trace you look at was worth keeping.

The architecture: buffering, decision, flush

Tail-based sampling requires a component that sits between your services and your storage backend — typically the OpenTelemetry Collector running the tailsampling processor.

The Collector receives spans, groups them by trace ID, and holds them in memory for a configurable decision wait time (typically 10–30 seconds). Once the wait expires — or all expected spans have arrived — the sampler evaluates the configured policies against the complete trace and either forwards it to storage or discards it.

# ✅ Tail-based sampling via OTel Collector
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000      # traces held in memory simultaneously
    policies:
      # Always keep errors
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      # Always keep slow traces (>500ms)
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 500 }

      # Sample 2% of healthy traffic
      - name: healthy-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

How obseria.io implements it at scale

obseria.io's ingest layer runs a distributed tail sampler that avoids the single-collector bottleneck. Spans are routed by trace ID hash to a consistent shard — ensuring all spans for a given trace land on the same sampler instance — using a layer of consistent-hash load balancers in front of the Collector fleet.

This approach scales horizontally: adding Collector instances increases throughput linearly without any coordination overhead. Decisions are made locally on each shard; only the kept traces are forwarded to the storage tier.

# obseria.io recommended: load balancer -> tail sampler fleet
exporters:
  loadbalancing:
    protocol:
      otlp:
        tls: { insecure: false }
    resolver:
      dns:
        hostname: dataleak-collector-headless.monitoring.svc.cluster.local
        port: 4317

# Each collector instance runs the tail_sampling processor
# Consistent hashing ensures all spans for a trace go to the same instance

Configuration: what to keep, what to drop

A practical tail sampling policy for most production services follows this priority order:

1. 1

   Always keep errors

   Any trace with a span in ERROR status. Non-negotiable.

2. 2

   Always keep SLA breaches

   Traces where total duration exceeds your p99 SLA threshold (e.g. 500ms).

3. 3

   Keep rare operations

   Operations with very low request volume — you can afford to keep 100% of them.

4. 4

   Sample healthy traffic

   The boring fast successful majority. 1–5% is usually enough for baseline analysis.

Trade-offs and when head-based is fine

Tail-based sampling is not free. The memory requirement — buffering all in-flight traces for the decision window — is significant. At 10k rps with a 10s wait window and an average of 8 spans per trace, you're buffering roughly 800,000 spans at any given moment. Plan your Collector memory accordingly.

Head-based sampling is appropriate in two scenarios: when you are in early development and want maximum simplicity, or when you can afford to store everything and are only sampling for cost at very low rates. In both cases, layer tail-based sampling on top as soon as you hit production scale.