OpenTelemetry Collector in Production: Best Practices for SREs

The OpenTelemetry Collector is the most flexible piece of an OpenTelemetry setup and, in many organizations, the least operated one. It starts as a sidecar someone copied from a tutorial, and a year later it carries every span, log and metric the company produces. When it falls over, observability goes dark exactly when you need it most.

This post collects the OpenTelemetry Collector best practices we apply in production, both in our own platform and when helping teams move existing OTel setups to Obseria. None of it is specific to one backend: the configuration examples use upstream components from the Collector core and contrib distributions.

Choose a two-tier topology: agent and gateway

A single Collector deployment that does everything is easy to start with and hard to operate. The pattern that holds up under load separates two roles:

• Agent tier: one Collector per node (a DaemonSet in Kubernetes, a system service on VMs). It receives OTLP from local applications, scrapes host metrics, tails log files and adds resource attributes such as host, pod and namespace. It stays small and does as little processing as possible.
• Gateway tier: a horizontally scaled Deployment behind a load balancer. It owns everything that needs a global view or more memory: tail-based sampling, PII masking, routing to multiple destinations, authentication and retries towards the backend.

This split has practical benefits. Agents can be updated with node maintenance without touching sampling logic. Gateways can be scaled on CPU without redeploying every node. And credentials for the backend live only in the gateway tier, not on every host.

If you use tail-based sampling, remember that all spans of a trace must reach the same gateway instance. Put a loadbalancing exporter in the agent tier that routes by trace ID, or add a dedicated sampling tier behind the gateway.

Put memory_limiter first, always

The most common Collector outage is not a bug but an out-of-memory kill. A traffic spike, a slow backend or a misbehaving client fills the internal buffers until the kernel or the kubelet terminates the process, and everything buffered is lost.

The memory_limiter processor prevents this by refusing data before the process runs out of memory. Receivers then return a retryable error, and well-behaved OTLP clients back off. It must be the first processor in every pipeline:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    send_batch_size: 8192
    send_batch_max_size: 10000
    timeout: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp/obseria]

Three details matter here. First, the percentages refer to the memory available to the process, so set a container memory limit; without one, the limiter has nothing meaningful to calculate against. Second, set the GOMEMLIMIT environment variable to roughly 80 to 90 percent of the container limit so the Go garbage collector works harder before the hard limit is reached. Third, keep batch near the end of the pipeline, after processors that drop or modify data, so batches contain only what will actually be exported.

Make exports survive backend outages

Every exporter built on the exporter helper has a sending queue and retry logic. The defaults keep the queue in memory, which means a Collector restart during a backend incident drops everything that was waiting. For the gateway tier, use a persistent queue backed by the file_storage extension:

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    timeout: 10s
    compaction:
      on_start: true
      on_rebound: true
      directory: /var/lib/otelcol/queue

exporters:
  otlp/obseria:
    endpoint: ingest.eu.obseria.example:4317
    headers:
      authorization: "Bearer ${env:OBSERIA_TOKEN}"
    compression: zstd
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 50000
      storage: file_storage/queue
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m

In Kubernetes, this requires a volume per gateway pod, typically via a StatefulSet with a PersistentVolumeClaim. Size the queue from your actual throughput: if a gateway receives 2,000 batches per minute and you want to ride out a ten-minute backend disruption, the queue needs room for at least 20,000 batches. Remember that queue_size counts batches, not individual spans or log records, unless you configure size-based queueing.

Also set max_elapsed_time deliberately. Retrying forever looks safe but keeps stale data in the queue and delays fresh data behind it. For most teams, ten to fifteen minutes is a reasonable upper bound.

Enrich at the edge, filter early

Resource attributes are what make telemetry navigable. Without k8s.namespace.name, k8s.deployment.name or cloud.region, every query starts with guesswork. Add them in the agent tier, where the Collector can see local metadata:

processors:
  resourcedetection:
    detectors: [env, system, ec2, eks]
    timeout: 2s
    override: false
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.pod.name
        - k8s.node.name
      labels:
        - tag_name: team
          key: app.kubernetes.io/team
          from: pod

Filtering works the other way round: drop data you will never query as early as possible. Health check spans, debug logs from noisy libraries and high-cardinality metric labels add cost at every hop. The filter processor with OTTL conditions handles most cases:

processors:
  filter/noise:
    error_mode: ignore
    traces:
      span:
        - attributes["http.route"] == "/healthz"
        - attributes["http.route"] == "/readyz"
    logs:
      log_record:
        - severity_number < SEVERITY_NUMBER_INFO

Set error_mode: ignore in production. With the default propagate, a single malformed record can cause the whole batch to be rejected.

Monitor the Collector itself

A Collector that silently drops data is worse than one that crashes, because nobody notices. The Collector exposes its own telemetry, and you should treat it with the same care as application metrics:

service:
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
  extensions: [health_check, file_storage/queue]

The metrics that matter most in day-to-day operation:

Metric	What it tells you	Alert when
otelcol_receiver_refused_*	Data rejected at the receiver, often by memory_limiter	Non-zero for more than 5 minutes
otelcol_exporter_send_failed_*	Exports that failed after retries	Any sustained increase
otelcol_exporter_queue_size	Current queue fill level	Above 70% of capacity
otelcol_processor_dropped_*	Data dropped inside a processor	Unexpected change versus baseline
otelcol_process_memory_rss	Resident memory of the process	Close to the container limit

Exact metric names carry suffixes such as _spans, _log_records or _metric_points, and Prometheus may append _total. Once these metrics land in Obseria, a simple OQL query shows where data is being lost across all gateways:

fetch metrics
| filter metric.name == "otelcol_exporter_send_failed_spans"
| summarize failed = sum(value), by:{k8s.pod.name, exporter}
| sort failed desc

Send the Collector's own telemetry through a separate, minimal pipeline. If the main pipeline is congested, you still want to see why.

Version configuration and roll out gradually

Collector configuration is production code. A typo in a processor name stops the whole Collector from starting; a wrong OTTL condition can drop every log line. Practices that prevent most incidents:

1. Keep configuration in Git and deploy it through the same pipeline as application code, including review.
2. Validate before deploying with otelcol validate --config=config.yaml in CI, using the exact distribution and version you run in production.
3. Build your own distribution with the OpenTelemetry Collector Builder (ocb). It contains only the components you use, which shrinks the attack surface and the binary.
4. Pin versions and read release notes. Contrib components move quickly, and configuration keys get renamed or deprecated.
5. Roll out in stages: one gateway replica or one node pool first, compare the self-monitoring metrics for 30 minutes, then continue.

Scaling and security essentials

Gateways scale well horizontally on CPU. Use a HorizontalPodAutoscaler on CPU utilization, set a PodDisruptionBudget so that node drains never remove all replicas at once, and spread replicas across zones. For OTLP over gRPC, remember that long-lived connections defeat simple L4 load balancing; use a gRPC-aware load balancer or set max_connection_age on the receiver so clients reconnect periodically.

On the security side, terminate TLS on receivers that accept traffic from outside the node, restrict which namespaces may send to the gateway with network policies, and keep backend tokens in secrets referenced through ${env:...}. Mask PII as close to the source as possible; the transform and redaction processors can hash or remove attributes such as e-mail addresses before data leaves your cluster.

How this fits with Obseria

Obseria accepts OTLP natively, so the Collector setup described here works unchanged: point the gateway exporter at your Obseria endpoint in the EU or UK region and existing SDKs deliver their first traces within minutes. Teams that prefer fewer moving parts can let the Obseria Agent handle host-level collection and use Obseria Pipeline for filtering, enrichment and PII masking at ingest. Both approaches can coexist, and because all data stays in OpenTelemetry format, you can export it via OTLP at any time.

Whatever you choose, the principles stay the same: protect memory, persist queues, enrich early, filter early and watch the Collector as closely as the services it observes.