A telemetry pipeline is the layer between telemetry sources and observability backends that collects logs, metrics, traces and other signals, processes them by filtering, sampling, enriching, masking and transforming, and routes the results to one or more destinations. It controls data quality, cost and compliance before data is stored.
What is a telemetry pipeline?
A telemetry pipeline, also called an observability pipeline, is the infrastructure that moves telemetry from where it is produced to where it is analyzed. Instead of every application and agent sending data straight to a backend, a pipeline sits in between and handles three jobs: collect, process and route.
Pipelines have become central because telemetry volume grows faster than budgets. Without a control point, every debug log, every high-cardinality metric and every health-check trace is stored at full price, and sensitive data can leak into systems that should never see it.
The three stages
1. Collect
Receivers accept data from SDKs, agents and infrastructure: OTLP from OpenTelemetry SDKs, Prometheus scrapes, syslog, file tails, cloud provider streams or Kafka topics. Good collection includes buffering and backpressure so that a slow backend does not cause data loss at the source.
2. Process
Processors change the data in flight. Typical operations:
- Filter and drop: remove debug logs, health-check spans or unused metrics.
- Sample: keep a representative share of traces, ideally with tail-based sampling that keeps errors and slow requests.
- Enrich: add resource attributes such as cluster, region, team or deployment version.
- Mask and redact: remove or hash personal data, tokens and secrets before storage.
- Transform: parse unstructured logs into fields, rename attributes to semantic conventions, convert logs to metrics.
- Aggregate: reduce cardinality or pre-compute metrics from spans.
3. Route
Exporters send each data stream where it belongs: security-relevant logs to a SIEM, full-fidelity data to cheap archive storage, curated data to the observability backend, and EU data only to EU regions.
Example: an OpenTelemetry Collector pipeline
The OpenTelemetry Collector is the most common open-source building block. An illustrative logs pipeline that drops debug logs, deletes a personal attribute and batches data before export:
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
filter/drop-debug:
error_mode: ignore
logs:
log_record:
- 'severity_number < SEVERITY_NUMBER_INFO'
attributes/pii:
actions:
- key: user.email
action: delete
batch: {}
service:
pipelines:
logs:
receivers: [otlp]
processors: [memory_limiter, filter/drop-debug, attributes/pii, batch]
exporters: [otlp]The order matters: the memory limiter comes first to protect the process, filtering happens early so later processors handle less data, and batching comes last before export.
Where telemetry pipelines run
Most production setups combine several tiers rather than a single central box:
- Agent tier: a collector or agent on each host or as a Kubernetes DaemonSet gathers local data, adds host and pod metadata and applies cheap filters close to the source.
- Gateway tier: a horizontally scaled pool of collectors receives data from agents and applications, applies central policies such as PII masking and tail-based sampling, and handles export. Tail-based sampling requires that all spans of a trace reach the same gateway instance, usually via trace-ID-aware load balancing.
- Buffer tier: for very high volumes, a message queue such as Kafka can decouple producers from processing and absorb backend outages.
- Managed tier: the observability platform's own ingest pipeline applies rules after data arrives but before it is stored.
Why a telemetry pipeline matters
- Cost control: data that is dropped or sampled before storage does not incur storage cost. Filtering at the source of the pipeline is usually the biggest single lever for observability spend.
- Compliance: masking personal data at ingest supports GDPR and internal data policies, because sensitive values never reach storage.
- Data quality: consistent attribute names and enrichment make queries and correlation work across teams.
- Flexibility: routing decouples sources from backends, so you can change or add destinations without re-instrumenting applications.
Best practices
- Define pipelines as code, review changes like application code and roll them out gradually.
- Monitor the pipeline itself: accepted, refused and dropped items, exporter queue size and latency.
- Filter and sample deliberately and document what you drop, so nobody searches for data that was never stored.
- Keep processing close to the source for volume reduction, and centralize policies such as PII rules so they apply everywhere.
Common mistakes
- Building a pipeline without buffering, so a backend outage loses data.
- Sampling traces head-based only and then missing the rare errors you needed.
- Masking PII in some pipelines but not all, leaving gaps in compliance.
How Obseria helps
Obseria Pipeline filters, enriches, masks PII and routes data at ingest, and filtered data costs only ingestion, not storage. It works with OpenTelemetry data and supports tail-based sampling, and you can export data via OTLP at any time. For a practical approach, read reducing observability costs pipeline-first.
