Sampling

What is sampling?

Sampling reduces the amount of telemetry you transmit and store by keeping a representative or deliberately chosen subset. It matters most for distributed tracing: a service handling 5,000 requests per second with 20 spans per request produces 100,000 spans per second, roughly 8.6 billion per day. Keeping all of them is rarely necessary, because most successful requests look the same.

The goal is to discard redundancy without losing the signal: errors, slow requests, rare code paths and enough baseline traffic to understand normal behavior.

Sampling is not limited to traces. High-volume debug logs, RUM events and profiling data can be sampled too, and metrics are reduced in a different way, through aggregation. Whatever the signal, the decision should be deliberate and documented: a sampled dataset is only useful if everyone who queries it knows what was kept, what was dropped and at which rate, so that counts can be interpreted or extrapolated correctly.

Head-based sampling

With head-based sampling the decision is made when the root span starts, before anything is known about how the request will end. In OpenTelemetry SDKs the decision is made by a sampler:

• AlwaysOn / AlwaysOff: keep or drop everything.
• TraceIdRatioBased: keep a fixed fraction, derived deterministically from the trace ID, so every service that sees the same trace ID reaches the same decision.
• ParentBased: follow the decision of the parent span, carried in the sampled flag of the W3C traceparent header. The SDK default is ParentBased with AlwaysOn for root spans.

Head sampling is cheap and simple: dropped spans are never exported. Its weakness is that it is blind. At a 5 percent ratio, 95 percent of failed requests are discarded together with the successful ones. It also relies on correct context propagation; if a service ignores the sampled flag, traces break into fragments.

Tail-based sampling

Tail-based sampling waits until all spans of a trace have arrived and then decides based on the whole trace. In the OpenTelemetry Collector this is the job of the tail_sampling processor, which buffers spans per trace ID for a configurable decision_wait period and then evaluates policies. An illustrative configuration:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 1000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

A trace is kept if any policy matches, so this keeps every error, every trace slower than one second, and 5 percent of the rest. The trade-offs: the Collector needs memory to buffer traces, decisions are delayed, and all spans of a trace must reach the same Collector instance. In scaled deployments, a first Collector tier uses the loadbalancing exporter to route spans by trace ID to a second tier that does the sampling.

Head vs. tail-based sampling at a glance

Criterion	Head-based	Tail-based
Decision point	Start of the trace	After the trace completes
Can keep all errors and slow traces	No	Yes
Resource cost	Minimal, in the SDK	Memory and CPU in the Collector
Network savings	From the application onward	Only after the Collector
Operational complexity	Low	Higher: routing, buffering, scaling

Many teams combine both: a generous head ratio in high-volume services to protect the network, plus tail sampling in the Collector to choose what is stored.

Best practices

1. Derive metrics before sampling. Compute request rate, error rate and latency histograms from all spans, for example with the Collector's span metrics connector, so dashboards and SLOs stay accurate.
2. Record the sampling rate. Backends need it to extrapolate counts: at 10 percent, each kept trace represents ten.
3. Keep the interesting traffic explicitly: errors, high latency, specific customers or endpoints and new releases.
4. Sample logs and traces consistently when they are correlated, or keep logs for unsampled traces at a lower level of detail.
5. Watch the sampler itself: buffer usage, late-arriving spans and dropped traces are signs that decision_wait or capacity needs tuning.

Common mistakes

• Building error-rate alerts on sampled traces and then wondering why the numbers fluctuate.
• Different services using independent random sampling, which produces incomplete traces.
• Tail sampling across several Collector replicas without trace-ID-aware routing.
• Setting a single global ratio, so low-traffic but critical services end up with almost no traces.

How Obseria helps

Obseria supports tail-based sampling, so you can keep every error and slow trace while reducing routine traffic, and RED metrics are computed from the full stream before sampling. Filtered data costs only ingestion, which makes aggressive sampling of noisy services predictable on your bill. See our guide to tail-based sampling with the OpenTelemetry Collector and reducing observability costs pipeline-first.