Convert Raw Logs into Metrics with OpenObserve Pipelines

Why raw logs are the wrong unit of analysis

Raw logs answer the question "what happened?" — but the more operationally useful questions are almost always quantitative: "how often does this happen?", "what is the rate of change?", "is this getting worse over time?". These are metric questions, not log questions.

The problem is that most engineers instrument at the log level because it is the path of least resistance — a single logger.info() call with a message and some context. No counter increments, no histogram observations, no gauge updates. The log stream captures the raw event; extracting meaning from it at query time requires expensive full-scan aggregations over potentially billions of events.

OpenObserve Pipelines solve this by sitting between the ingest layer and the storage layer. Every log event passes through the pipeline, which extracts numeric signals, applies aggregations, and writes the results to the metrics store. The raw log can then be compressed, sampled, or expired on a shorter retention schedule — the derived metric captures the long-range signal at a fraction of the storage cost.

The OpenObserve Pipeline model: architecture and data flow

The pipeline model in OpenObserve is a directed acyclic graph (DAG) of functions applied to each inbound log event. Functions can drop events, modify fields, extract new fields, emit metrics, or fan out a single event into multiple derived records.

• Source

  Log stream from an OTel Collector, Fluentd, Vector, or direct API push. Partitioned by stream name (e.g. 'nginx-access', 'payment-service').

• Functions

  Ordered sequence of transforms: parse → filter → enrich → derive. Each function receives the current event document and returns a modified version (or null to drop).

• Metric sink

  Derived metrics are written to the time-series store with the labels extracted during the pipeline run. Stored as Prometheus-compatible counters, gauges, or histograms.

• Log sink

  The (optionally modified) log event continues to the log store. You can use the pipeline to redact PII, parse JSON, add fields, or drop events before storage.

Pipeline functions: parse, filter, transform, derive

There are four semantic categories of pipeline function. Most production pipelines use all four in sequence.

# 1. PARSE: extract structured fields from raw log body
# Input: {"message": "GET /api/payments 200 145ms user_id=4821"}

.method, .path, .status, .duration_ms, .user_id = parse_regex!(
  .message,
  r'^(?P<method>\w+) (?P<path>[^\s]+) (?P<status>\d+) (?P<duration_ms>\d+)ms user_id=(?P<user_id>\d+)'
)
.status = to_int!(.status)
.duration_ms = to_float!(.duration_ms)

# 2. FILTER: drop events we do not want in the metric
# Drop health check endpoints — they pollute request rate metrics
if .path == "/healthz" || .path == "/readyz" {
  abort
}

# 3. TRANSFORM: normalise and enrich
# Bucket status codes into classes
.status_class = if .status >= 500 { "5xx" } else if .status >= 400 { "4xx" } else { "2xx" }

# 4. DERIVE: emit a metric counter
metric = {
  "name": "http_requests_total",
  "type": "counter",
  "value": 1.0,
  "labels": {
    "method": .method,
    "path": .path,
    "status_class": .status_class,
    "service": .service.name
  }
}
emit_metric(metric)

Extracting metrics from structured logs

Structured logs — JSON-formatted events where fields are explicit key-value pairs — are the easiest input for pipeline-based metric extraction. The parsing step is trivial; most of the pipeline is transform and derive.

The common pattern for structured logs is: JSON parse (done automatically if the log body is valid JSON) → field validation → label selection → metric emission.

# OpenObserve pipeline definition for a structured log stream
# Stream: payment-service (JSON logs)

name: payment-service-metrics
stream: payment-service
functions:

  - name: validate_fields
    type: vrl
    source: |
      # Drop events missing required fields
      if !exists(.duration_ms) || !exists(.status) || !exists(.operation) {
        abort
      }
      # Coerce types
      .duration_ms = to_float!(.duration_ms)
      .status = to_int!(.status)

  - name: derive_request_counter
    type: metric
    metric_name: payment_requests_total
    metric_type: counter
    value: 1
    labels:
      operation: "{{ .operation }}"
      status: "{{ .status }}"
      environment: "{{ .environment }}"

  - name: derive_latency_histogram
    type: metric
    metric_name: payment_request_duration_ms
    metric_type: histogram
    value: "{{ .duration_ms }}"
    buckets: [10, 25, 50, 100, 250, 500, 1000, 2500]
    labels:
      operation: "{{ .operation }}"
      environment: "{{ .environment }}"

Extracting metrics from unstructured logs with regex and VRL

Unstructured logs — plain text lines from legacy applications, Nginx, Apache, or anything that predates structured logging — require a parse step before metric extraction. Regex is the workhorse here, but VRL's parse_grok and parse_apache_log built-ins handle common formats without hand-written patterns.

# Parse Nginx combined log format
# Input: 10.0.0.1 - - [08/Aug/2026:14:23:01 +0000] "GET /api/v2/users 200 1842" 0.045

parsed = parse_nginx_log!(.message, "combined")

.remote_addr    = parsed.client
.method         = parsed.method
.path           = parsed.path
.status         = parsed.status
.bytes_sent     = parsed.size
.request_time_s = parsed.request_time

# Normalise the path: strip IDs to avoid label explosion
# /api/v2/users/4821 → /api/v2/users/:id
.path_normalised = replace(.path, r'/[0-9a-f]{8,}', "/:id")
.path_normalised = replace(.path_normalised, r'/\d+', "/:id")

Aggregation functions: counters, histograms, gauges

OpenObserve Pipelines support three metric types that map directly to the Prometheus data model. Choosing the right type matters for storage efficiency, query capability, and alerting accuracy.

• TypeUse forVRL emitStorage cost
• CounterRequest counts, error counts, bytes transferredmetric_type: counterVery low — single float per label set per window
• GaugeActive connections, queue depth, current error ratemetric_type: gaugeLow — single float, overwritten each window
• HistogramLatency, request size, response sizemetric_type: histogram + bucketsMedium — one float per bucket per label set
• SummaryPrecomputed quantiles (avoid if possible)metric_type: summaryHigher — loses aggregatability across replicas

Pipeline performance: throughput, latency, and backpressure

Pipelines run synchronously in the ingest path — a slow or compute-intensive pipeline function adds latency to every log write. Understanding the performance characteristics is essential before deploying to a high-volume stream.

Throughput limits. A single pipeline instance in obseria.io processes approximately 80,000–120,000 events/second on standard hardware, depending on function complexity. Pipelines with only field extraction and counter emission are in the upper range. Pipelines with heavy regex parsing or multiple histogram derivations are in the lower range. For streams exceeding this, configure stream sharding to distribute pipeline execution.

Backpressure handling. If the pipeline function throws an unhandled error (e.g. a parse assertion failure with parse_regex!), the event is routed to the dead-letter queue, not dropped. This is the correct default — you can inspect failed events and fix the pipeline function without losing data.

# Performance tuning for high-volume pipelines

pipeline:
  name: nginx-access-high-volume
  stream: nginx-access

  performance:
    batch_size: 1000          # Process events in batches for efficiency
    parallelism: 4            # Number of parallel pipeline workers
    max_latency_ms: 50        # Emit metrics even if batch not full

  error_handling:
    on_parse_error: dlq       # Route to dead-letter queue (not drop)
    dlq_stream: nginx-access-dlq
    dlq_retention: 7d

  # Metric aggregation window
  # Counters and gauges are flushed every 10s
  # Histograms are flushed every 60s (higher storage cost)
  aggregation:
    counter_flush_interval: 10s
    histogram_flush_interval: 60s

Real-world example: HTTP access log → request rate + error rate

This is the most common pipeline configuration we see in production. The goal: convert a high-volume Nginx access log stream into two metrics — http_requests_total (counter) and http_errors_total (counter) — with labels for method, normalised path, and status class.

name: nginx-to-request-metrics
stream: nginx-access

functions:

  - name: parse_nginx
    type: vrl
    source: |
      parsed = parse_nginx_log!(.message, "combined")
      .method  = parsed.method
      .path    = parsed.path
      .status  = parsed.status
      .latency = parsed.request_time

      # Normalise path parameters
      .path_norm = replace(.path, r'/\d+', "/:id")
      .path_norm = replace(.path_norm, r'/[0-9a-f-]{36}', "/:uuid")

      # Status class
      .status_class = if .status >= 500 { "5xx" }
        else if .status >= 400 { "4xx" }
        else if .status >= 300 { "3xx" }
        else { "2xx" }

      # Drop /healthz and /metrics endpoints
      if includes(["/healthz", "/readyz", "/metrics"], .path) { abort }

  - name: emit_request_counter
    type: metric
    metric_name: http_requests_total
    metric_type: counter
    value: 1
    labels:
      method:       "{{ .method }}"
      path:         "{{ .path_norm }}"
      status_class: "{{ .status_class }}"

  - name: emit_error_counter
    type: metric_conditional
    condition: ".status >= 500"
    metric_name: http_errors_total
    metric_type: counter
    value: 1
    labels:
      method: "{{ .method }}"
      path:   "{{ .path_norm }}"
      status: "{{ .status }}"

  - name: emit_latency_histogram
    type: metric
    metric_name: http_request_duration_seconds
    metric_type: histogram
    value: "{{ .latency }}"
    buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5]
    labels:
      method: "{{ .method }}"
      path:   "{{ .path_norm }}"

With this pipeline in place, you can alert on error rate using a simple PromQL expression:

# 5xx error rate > 1% over last 5 minutes → fire alert
(
  sum(rate(http_errors_total[5m]))
  /
  sum(rate(http_requests_total[5m]))
) > 0.01

Real-world example: payment service logs → P99 latency histogram

Payment services are latency-sensitive and typically emit structured JSON logs. This pipeline extracts a latency histogram broken down by payment method and currency, which would be prohibitively expensive to store if derived from raw logs at query time.

name: payment-latency-histogram
stream: payment-service

functions:

  - name: validate_and_coerce
    type: vrl
    source: |
      # Only process completed payment events
      if .event_type != "payment.completed" && .event_type != "payment.failed" {
        abort
      }
      .duration_ms     = to_float!(.duration_ms)
      .payment_method  = string!(.payment_method)   # "card", "sepa", "paypal"
      .currency        = upcase(string!(.currency))  # "EUR", "USD", "GBP"
      .success         = .event_type == "payment.completed"

  - name: emit_duration_histogram
    type: metric
    metric_name: payment_duration_ms
    metric_type: histogram
    value: "{{ .duration_ms }}"
    buckets: [10, 25, 50, 100, 200, 500, 1000, 2000, 5000]
    labels:
      payment_method: "{{ .payment_method }}"
      currency:       "{{ .currency }}"
      success:        "{{ .success }}"

  - name: emit_payment_counter
    type: metric
    metric_name: payments_total
    metric_type: counter
    value: 1
    labels:
      payment_method: "{{ .payment_method }}"
      currency:       "{{ .currency }}"
      success:        "{{ .success }}"

# P99 payment latency by method (PromQL)
histogram_quantile(
  0.99,
  sum by (payment_method, le) (
    rate(payment_duration_ms_bucket[5m])
  )
)

# Payment success rate by currency
sum by (currency) (rate(payments_total{success="true"}[5m]))
/
sum by (currency) (rate(payments_total[5m]))

Alerting on pipeline-derived metrics

Once your logs are converted to metrics, alerting is straightforward — you use the same alerting infrastructure you use for any other metric. The key advantage over log-based alerting: metric alerts evaluate in milliseconds against pre-aggregated time-series data, whereas log-based alerts require a full-scan query on every evaluation cycle.

# obseria.io alert rule using pipeline-derived metrics

alerts:
  - name: HighPaymentErrorRate
    expr: |
      sum(rate(payments_total{success="false"}[5m]))
      /
      sum(rate(payments_total[5m]))
      > 0.02
    for: 2m
    severity: critical
    labels:
      team: payments
      runbook: https://runbooks.internal/payment-errors
    annotations:
      summary: "Payment error rate above 2% for {{ $labels.currency }}"
      description: |
        Payment failure rate is {{ $value | humanizePercentage }} over the last 5 minutes.
        Check the payment-service logs and upstream PSP status.

  - name: P99PaymentLatencyHigh
    expr: |
      histogram_quantile(0.99, sum by (payment_method, le) (
        rate(payment_duration_ms_bucket[5m])
      )) > 2000
    for: 3m
    severity: warning
    labels:
      team: payments

Common pitfalls and how to avoid them

• Label cardinality explosion

  Using raw path, user ID, request ID, or session ID as metric labels creates a unique time series per value. This can create millions of series overnight. Always normalise IDs out of paths and never use high-cardinality fields as labels. Use summary labels (status_class instead of status) wherever possible.

• Using parse_regex! in hot paths

  The ! suffix in VRL means 'abort on error'. In a high-volume stream, a single malformed log line will route the event to the dead-letter queue and emit a pipeline error counter. Use parse_regex (without !) and handle the error case explicitly to avoid DLQ buildup.

• Deriving metrics from sampled logs

  If your log stream is sampled (e.g. 10% of requests are logged), the derived metrics will undercount by the same factor. Either use the full log stream for metric derivation and a sampled stream for storage, or correct for the sampling rate in your PromQL expressions.

• Forgetting the flush interval in alerting math

  Pipeline-derived metrics are flushed on a configurable interval (typically 10–60s). Your alert 'for:' duration must be longer than the flush interval — otherwise the alert evaluates against stale data and fires spuriously. Use for: at least 2× the flush interval.

• No baseline period before switching alert sources

  Migrating from log-based alerts to metric-based alerts requires a baseline period where both run in parallel. Metric rates will differ from log aggregation counts due to timing windows, sampling, and aggregation semantics. Run both for at least 7 days before decommissioning log-based alerts.

OpenObserve Pipelines are available on all obseria.io plans with no additional cost. Pipeline function execution is counted against your ingest quota at a 1:1 ratio. Start building your first log-to-metric pipeline in the obseria.io sandbox — no setup required.