Shape your telemetry before it is stored

What is a telemetry pipeline?

A telemetry pipeline, often called an observability pipeline or log pipeline, is the processing layer between the sources that produce telemetry and the systems that store and analyze it. It receives logs, metrics, traces and events, transforms them according to rules and forwards the result to one or more destinations.

Without such a layer, every byte an application emits ends up in storage unchanged. That includes verbose debug output, repetitive health checks, personal data that should never have been logged and high-cardinality attributes nobody queries. A pipeline gives teams a single place to decide what is kept, how it is shaped and where it goes.

Why a telemetry pipeline matters

Telemetry volume grows with every new service, cluster and feature. Costs tend to grow with it, while the share of data that is actually used for troubleshooting stays small. At the same time, data protection rules such as the GDPR require that personal data is minimized and protected, and that is much easier to enforce before data is stored than afterwards.

A pipeline addresses both concerns. Platform teams can reduce noise centrally instead of asking every application team to change its logging. Security and compliance teams get one place to define masking rules. Engineers still get the signals they need, enriched with consistent context.

Typical processing steps

• Filter: drop records by severity, attribute, source or pattern, for example debug logs from production or load balancer health checks.
• Enrich: add attributes such as team, service, environment, region, Kubernetes metadata or the current deployment version.
• Mask PII: redact, hash or remove fields like email addresses, IP addresses, session tokens or customer IDs.
• Sample: use head-based sampling for predictable volume, or tail-based sampling to decide after a trace is complete, keeping traces with errors or unusual latency.
• Aggregate: convert high-volume log streams into metrics, for example request counts per status code and endpoint.
• Route: send each stream to the storage tier, retention period or destination that matches how it is used.

How Obseria Pipeline works

Obseria Pipeline is part of the ingest path of the Obseria platform. It accepts data from OpenTelemetry SDKs and the OpenTelemetry Collector over OTLP, from the optional Obseria Agent and from Prometheus remote write. You do not need to replace your existing instrumentation.

Rules are defined per source, service or attribute and apply to logs, metrics, traces and events alike. Processed data lands in Obseria Lake, the index-free store for all signals, where it can be queried with OQL, PromQL or SQL. Because the pipeline sits in front of storage, filtered data does not add storage cost, and aggregated metrics replace large volumes of raw logs where the detail is not needed.

A simple rule might look like this:

when source == "checkout" and level == "debug"
  drop
when attribute exists "user.email"
  mask "user.email"
when trace has error or duration > 2s
  keep trace

Rules can be tested against sample data before they are activated, and changes are versioned, so teams can review what changed and when.

Tail-based sampling for traces

Head-based sampling decides at the start of a request whether a trace is kept. It is cheap but random, so rare errors may be dropped. Tail-based sampling waits until all spans of a trace have arrived and then decides based on the result. This lets you keep every trace with an error, every trace above a latency threshold and a representative share of normal traffic. Obseria Pipeline supports both approaches, and you can combine them per service.

Telemetry pipeline best practices

1. Start by measuring. Look at which sources and attributes produce the most volume before writing rules.
2. Mask early. Apply PII rules as close to ingest as possible and treat them as a baseline for all sources.
3. Prefer aggregation to deletion. When a log stream is only used for counts, turn it into a metric instead of dropping it.
4. Sample by outcome. Keep errors and slow requests, sample routine success paths.
5. Review rules regularly. Services change, and rules that made sense last quarter may hide useful data today.