Running the OpenTelemetry Collector at production scale: what we learned

Why we run the Collector ourselves

When obseria.io moved to a multi-tenant architecture in 2024, we evaluated every option: direct SDK-to-backend export, vendor-managed agents, and running the OpenTelemetry Collector ourselves. We chose the Collector for one reason: it gives us full control over the signal pipeline without locking customers into any SDK choice.

Today we run a fleet of Collectors across six availability zones, processing approximately 4 million spans per second at peak. This article documents what we got wrong, what we got right, and the specific configuration changes that made the difference.

The memory limiter: your most important processor

The first lesson we learned — expensively — is that the Collector will consume as much memory as you give it and then OOM-crash if you don't constrain it explicitly. The memory_limiter processor is not optional in production.

Put it first in every pipeline. It checks memory usage before each batch is processed and soft-refuses new data if you're above the limit — giving upstream components time to back off rather than crashing hard.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500        # hard limit — refuse at this point
    spike_limit_mib: 400   # headroom for spikes above limit_mib

  # memory_limiter MUST be first in the pipeline
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlp/dataleak]

Batching: the single biggest CPU win

The batch processor reduces the number of export calls dramatically. Without it, every span potentially triggers a separate gRPC call to your backend. With it, spans are accumulated into batches and exported together.

The right batch size depends on your volume and latency tolerance. For our fleet, 8,000 spans per batch with a 5-second maximum timeout gives the best trade-off between throughput and latency:

processors:
  batch:
    send_batch_size: 8000      # send when this many spans accumulate
    send_batch_max_size: 10000 # never exceed this even under memory pressure
    timeout: 5s                # send even if batch isn't full after 5s

# Result: ~90% reduction in outbound gRPC calls vs unbatched
# CPU impact: -35% on exporter goroutines

Queue sizing and back-pressure

The OTLP exporter has an internal retry queue. If your backend is temporarily unavailable, this queue absorbs the spike and retries — but it also consumes memory. Getting queue sizing right prevents both data loss (under-sized) and OOM crashes (over-sized).

exporters:
  otlp/dataleak:
    endpoint: ingest.obseria.io:4317
    sending_queue:
      enabled: true
      num_consumers: 10    # parallel export workers
      queue_size: 5000     # batches to buffer; each batch ~8k spans
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s  # give up after 5 minutes

With the above settings, a 5-minute backend outage can be absorbed without data loss — provided you have enough memory for 5000 × 8000 spans in the queue (~600MB at typical span sizes).

The 40% CPU reduction: compression

This was the biggest surprise. Enabling gzip compression on the OTLP exporter reduced our total Collector fleet CPU usage by 40% — not because compression is cheap, but because it dramatically reduced network I/O, which was the actual bottleneck causing goroutine stalls.

Span data is extremely compressible (JSON/protobuf with repetitive field names and values). Compression ratios of 8:1 are common, meaning 8× less data on the wire and 8× fewer bytes the kernel has to buffer and flush.

exporters:
  otlp/dataleak:
    endpoint: ingest.obseria.io:4317
    compression: gzip   # ← this single line cut our CPU by 40%
    headers:
      Authorization: "Bearer {DATALEAK_API_KEY{"}"}"

# Before: 24 vCPUs at 78% utilization across the fleet
# After:  24 vCPUs at 46% utilization — same throughput

Kubernetes deployment patterns

We run the Collector in two modes simultaneously: as a DaemonSet (one instance per node, receiving data from pods via localhost) and as a Deployment (centralized fleet for tail-based sampling and fan-out export).

Monitoring the Collector itself

The Collector exposes its own Prometheus metrics on port 8888 by default. The metrics you should alert on are:

# Critical Collector metrics to monitor
otelcol_processor_refused_spans_total        # memory limiter rejecting data
otelcol_exporter_send_failed_spans_total     # export failures
otelcol_exporter_queue_size                  # queue depth (alert at 80% of queue_size)
otelcol_processor_batch_batch_send_size_bytes # batch size distribution
otelcol_receiver_accepted_spans_total        # ingest rate

# Alert: queue_size > 0.8 * configured_queue_size for > 2 minutes
# Alert: refused_spans_total rate > 0 (memory limiter firing)
# Alert: send_failed_spans_total rate > 0 for > 30 seconds