Tail-Based Sampling in the OpenTelemetry Collector: Keep Every Error, Save 80% of the Cost

Every team that introduces distributed tracing in production runs into the same question after a few weeks: do we really have to store every trace? The honest answer is no. Most traces are successful requests with normal latency, and nobody needs millions of those per hour. But the interesting traces, the ones with errors, timeouts and odd latencies, you want in full. Those are exactly the ones classic head-based sampling loses.

In this post, we describe how we run tail-based sampling with the OpenTelemetry Collector, which policies have proven themselves and which mistakes we made along the way. Everything described here works with the open-source collector; Obseria is just the backend the traces flow into at the end.

Head-based vs. tail-based: when is the decision made?

Head-based sampling decides at the start of a trace, in the SDK of the first service: "We keep this request with 10% probability." The decision is propagated via the trace context. It is cheap and simple, but at that point the SDK does not know whether the request will end with a 502 forty milliseconds later. 90% of all error traces are gone before they happen.

Tail-based sampling moves the decision to the end: the collector gathers all spans of a trace, waits until it is complete and then applies policies. Failed? Keep. Slower than p99? Keep. Rare endpoint? Keep. Everything else with 5% probability. The price is memory and an architectural requirement we will get to in a moment.

Five policies that have proven themselves

After two years of operation at customers with volumes from 10,000 to 2 million spans per second, five policies have become our standard. Here is the configuration of the tail_sampling processor:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 200000
    expected_new_traces_per_sec: 5000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1500 }
      - name: rare-endpoints
        type: string_attribute
        string_attribute:
          key: http.route
          values: [ "/api/v1/refund", "/api/v1/export/.*" ]
          enabled_regex_matching: true
      - name: debug-header
        type: string_attribute
        string_attribute: { key: debug.trace, values: ["true"] }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Three notes: decision_wait must be longer than your longest trace, otherwise late spans are evaluated against a decision that has already been made and get dropped. num_traces determines memory usage; expect roughly 1 to 2 KB per buffered trace. And policies are combined with OR: a trace is kept as soon as one policy matches.

The collector topology: all spans of a trace in one place

This is where the real difficulty lies. Tail-based sampling only works if all spans of a trace land on the same collector instance. With a round-robin load balancer in front, that is not the case: half of a trace's spans end up on instance A, the other half on B, and both make decisions with incomplete data.

The solution is a two-tier topology with the loadbalancing exporter:

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp: { tls: { insecure: true } }
    resolver:
      k8s:
        service: otel-sampler.observability
        ports: [4317]

The exporter hashes the trace ID and sends all spans of the same trace to the same sampler instance. The Kubernetes resolver watches the endpoints of the sampler service; when scaling, traces that are in flight are briefly misrouted. That is why we never scale tier 2 under load, only in quiet time windows.

Three mistakes we made

• decision_wait too short. Our first value was 2 seconds. Batch jobs with 30-second traces were systematically cut apart. Today, 10 to 30 seconds is standard, depending on the longest expected trace.
• Memory underestimated. At 50,000 new traces per second and a 10-second wait, you buffer 500,000 traces. At 2 KB per trace, that is 1 GB, more during load spikes. Plan limits and alerts on otelcol_processor_tail_sampling_sampling_trace_dropped_too_early.
• Metrics computed from spans after sampling. Anyone who derives RED metrics (rate, errors, duration) from sampled spans gets an error rate of 60%, because errors are overrepresented. The spanmetrics connector belongs before the sampler.

The results in numbers

Across all customers using this topology, we see a median stored trace volume of 18 to 22% of the original, with 100% coverage of all traces with error status or latency above the threshold. Since Obseria bills by stored spans rather than received spans, this translates one to one into the invoice.

What we contributed back

Two things from this work are now part of the OpenTelemetry project: a fix in the loadbalancing exporter that no longer drops spans when endpoints change, and additional metrics in the tail_sampling processor that show which policy made a decision. The latter are the foundation of the sampling dashboard Obseria creates automatically for every collector pipeline.

If you want to check your own collector configuration: otelbin visualizes pipelines and warns about the pitfalls described here, such as a spanmetrics connector behind the sampler.