Cut Observability Costs by 50%+ with a Pipeline-First Approach

If your observability bill grows faster than your traffic, the problem is rarely the price per gigabyte. It is that everything gets stored by default: every debug log line, every successful health check span, every metric label someone added during an incident two years ago. The most effective way to reduce observability costs is to move the keep-or-drop decision to the front of the data flow, into a pipeline, instead of fighting retention settings and query limits afterwards.

This post describes the pipeline-first approach we recommend, walks through the techniques in order of impact and finishes with a worked cost example using Obseria's published prices.

Why observability costs explode

Telemetry volume scales with three things at once: traffic, the number of services and the verbosity of instrumentation. A move to microservices multiplies spans per request. Kubernetes multiplies log sources. Auto-instrumentation adds spans for every database call and HTTP client. None of this is wrong, but almost no team queries more than a fraction of what it collects.

Typical sources of waste we see when teams analyze their data:

• Health and readiness checks producing a span and a log line every few seconds per pod.
• Debug and trace level logs left on in production after an investigation.
• Successful, fast traces stored at 100% although only a small sample is ever opened.
• Duplicate events, for example the same error logged by the application, the sidecar and the ingress controller.
• High-cardinality metric labels such as user IDs or full URLs that multiply time series.

What pipeline-first means

Pipeline-first means that every signal passes through a processing layer before it reaches storage, and that this layer has explicit rules for each data class. The decision is no longer "store everything, delete later" but "store what has a known consumer". In practice, that pipeline can be the OpenTelemetry Collector, Obseria Pipeline at ingest or a combination of both.

The key insight is economic. In usage-based pricing, storing data is usually far more expensive than receiving it. With Obseria, logs, spans and RUM events cost $0.060 per million ingested but $0.540 per million stored, and data that Obseria Pipeline filters out costs only the ingestion price. Every record you decide not to store saves 90% of its cost, and every record you drop before it leaves your cluster saves all of it.

Step 1: filter noise at the source

Start with the cheapest win: data that has no consumer at all. Health checks, readiness probes and debug logs can usually be removed without discussion. In the OpenTelemetry Collector, the filter processor does this before data leaves the cluster:

processors:
  filter/drop-noise:
    error_mode: ignore
    traces:
      span:
        - attributes["http.route"] == "/healthz"
        - attributes["http.route"] == "/metrics"
    logs:
      log_record:
        - severity_number < SEVERITY_NUMBER_INFO
        - IsMatch(body, ".*GET /healthz.*")

Before you drop anything, measure it. A single OQL query shows which services and log levels generate the most volume, so you can target the biggest contributors first:

fetch logs
| summarize records = count(), by:{service.name, level}
| sort records desc
| limit 20

In most environments, the top five rows of this query account for a large part of the total log volume, and at least one of them is a level or source nobody looks at.

Step 2: sample traces by value, not by chance

Head-based sampling decides at the start of a request whether to keep a trace, before anyone knows whether it will fail. Tail-based sampling waits until the trace is complete and keeps what is valuable: errors, slow requests, rare endpoints and a small baseline of everything else.

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

With a 10% baseline plus all errors and slow requests, many teams store 10 to 20% of their spans while keeping every trace they would actually open during an incident. Rate-based metrics such as request count and latency percentiles should be computed before sampling, so dashboards remain accurate.

Step 3: aggregate and reduce metric cardinality

Metrics are cheaper per data point, but cardinality multiplies them. A latency histogram with a user_id label produces one series per user. Remove or bucket such labels in the pipeline:

processors:
  transform/cardinality:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "user_id")
          - delete_key(attributes, "request_id")
          - replace_pattern(attributes["http.url"], "/orders/[0-9]+", "/orders/{id}")

A second lever is turning repetitive logs into metrics. If a log line is only ever counted ("payment declined", "cache miss"), a counter metric answers the same question at a fraction of the cost. Keep a sample of the raw lines for context and drop the rest.

Step 4: mask and route by value

Not every record needs the same treatment. Audit and security logs may need long retention, application logs need fast search for 30 days, and verbose debug output is only useful for the hour after a deployment. Route each class explicitly instead of sending everything to one destination with one retention policy. Mask personal data such as e-mail addresses, IP addresses and payment references in the same step: data that never contains PII does not need special access controls, deletion workflows or separate storage, which removes cost and compliance work at the same time.

A worked example with Obseria prices

Consider a mid-sized platform with the following monthly volumes. The prices are Obseria's list prices per million records: logs and spans $0.060 ingested and $0.540 stored, metric data points $0.020 ingested and $0.180 stored.

Signal	Volume per month	Before: ingest + store	Pipeline rule	After
Logs	20 billion records	$1,200 + $10,800 = $12,000	60% filtered in Obseria Pipeline (ingest only)	$1,200 + $4,320 = $5,520
Spans	10 billion spans	$600 + $5,400 = $6,000	Tail-based sampling in the Collector keeps 15%	$90 + $810 = $900
Metrics	5 billion data points	$100 + $900 = $1,000	30% of series removed via cardinality rules	$70 + $630 = $700
Total		$19,000		$7,120

The calculation for logs: 20,000 million records times $0.060 is $1,200 for ingestion. Without filtering, storing all of them costs 20,000 times $0.540, or $10,800. If Obseria Pipeline filters 60% at ingest, all 20,000 million are still ingested ($1,200), but only 8,000 million are stored, which costs $4,320.

For spans, sampling happens in the Collector, so only the kept 15% (1,500 million spans) are sent at all: $90 for ingestion and $810 for storage. For metrics, the removed series never leave the cluster: 3,500 million data points cost $70 to ingest and $630 to store.

The total drops from $19,000 to $7,120 per month, a reduction of $11,880 or 62.5%. Errors, slow traces and every log line with a known consumer are still there. Your own numbers will differ, but the structure of the calculation is the same, and the largest lever is almost always storage.

Make it stick: ownership and guardrails

Cost reductions erode if nobody owns them. Three habits keep the savings in place:

1. Attribute volume to teams. Add a team resource attribute in the pipeline and review volume per team monthly. Teams that see their own numbers change their own logging.
2. Review rules like code. Pipeline rules live in Git, get reviewed and are deployed through CI. A rule that drops data needs the same scrutiny as a schema migration.
3. Keep an escape hatch. Allow a debug header or a feature flag that forces full sampling and verbose logs for a single request or tenant during an investigation.

Also be careful about what you never drop: audit logs, security events and anything required for compliance should bypass sampling rules entirely and be routed explicitly.

How Obseria supports a pipeline-first setup

Obseria Pipeline filters, enriches, masks PII and routes data at ingest, and it is included in every plan. Because filtered data costs only ingestion, you can start broad, observe what you actually query and tighten rules over time without re-instrumenting services. Tail-based sampling is supported natively, and existing OpenTelemetry Collector pipelines keep working unchanged. Pricing is per signal, with unlimited users and no host or seat fees, so the volume you control is the cost you control.

The fastest way to find your own savings is to run the volume query above against a week of real data. The 14-day trial requires no credit card, and the first traces from existing OTel SDKs typically arrive within 15 minutes.