Metrics monitoring for Prometheus and OpenTelemetry

What is metrics monitoring?

Metrics monitoring is the continuous collection, storage and evaluation of numeric measurements over time. A metric is a named time series, such as request rate, error count, CPU utilization or queue depth, usually described by labels like service, region or pod. Because metrics are compact and fast to aggregate, they are the backbone of dashboards, alerts and SLOs.

In cloud-native environments, Prometheus metrics and OpenTelemetry metrics have become the common formats. Exporters expose infrastructure data, application libraries emit request and business metrics, and the Prometheus text format or OTLP carries them to a backend. The challenge is no longer generating metrics but storing, querying and paying for them at scale.

Why metrics monitoring matters

Metrics answer the question of whether something is wrong right now, and how it compares with normal behavior. They show saturation before an outage, reveal slow degradations that individual logs hide and provide the numbers for capacity planning and cost discussions. Every alert that fires within seconds of a problem usually starts with a metric.

Running this yourself gets harder as systems grow. Single Prometheus servers hit memory limits, long-term storage requires extra components, high-cardinality labels multiply series counts, and federated setups make global queries slow. Teams end up maintaining a monitoring system that needs its own monitoring.

Which metrics to monitor

A practical starting point is to combine the golden signals for services with the USE method for resources:

• Traffic: requests per second per service and endpoint.
• Errors: failed requests, exceptions and non-success status codes as a rate and ratio.
• Latency: request duration as histograms, so you can compute percentiles instead of averages.
• Saturation: CPU, memory, disk, connection pools, thread pools and queue depth.
• Business metrics: orders, sign-ups or payments that show the impact on the company.

How Obseria handles metrics monitoring

Obseria accepts metrics over Prometheus remote write and OTLP. Your existing exporters, scrape configurations and OpenTelemetry SDKs keep working, and the Obseria Agent is optional. Before data is stored, Obseria Pipeline can drop unused series, aggregate away labels you never query and add team or environment metadata.

Metrics land in Obseria Lake, the same index-free store that holds logs, traces and events. Retention is 13 months, so you can compare seasonal peaks without a second storage tier. You can keep your PromQL queries and alert rules:

sum by (service) (rate(http_server_request_duration_seconds_count{status=~"5.."}[5m]))
/
sum by (service) (rate(http_server_request_duration_seconds_count[5m]))

Because TopoGraph maps services, hosts and workloads automatically, a spike on a chart is one click away from the related traces, logs and recent deployments. Obseria Intelligence uses the same metrics for anomaly detection and causal root cause analysis and shows the evidence behind each conclusion.

Cardinality and cost control

Metric cost is driven by the number of series and data points. A single label containing user IDs or request paths can create millions of series. Obseria shows which metrics and labels contribute most to volume, so you can decide what to keep. Rules in Obseria Pipeline can rewrite, aggregate or drop data at ingest. Pricing is per ingested data point with unlimited users, so adding engineers to the platform never increases the bill.

Metrics monitoring best practices

1. Use histograms for latency. Averages hide tail latency; histograms allow percentiles and SLOs.
2. Keep labels bounded. Never put user IDs, session IDs or raw URLs into metric labels.
3. Follow naming conventions. OpenTelemetry semantic conventions make metrics comparable across services and teams.
4. Alert on rates and ratios. Error ratios and burn rates are more stable than raw counters.
5. Review unused metrics. Drop series nobody queries to cut noise and cost.
6. Correlate with other signals. Metrics show that something changed; traces and logs explain why.