Prometheus monitoring with remote write and PromQL
Keep your exporters, scrape configs and PromQL. Obseria receives Prometheus metrics via remote write or the OpenTelemetry Collector and adds long-term storage, topology and AI root cause.
Why monitor Prometheus with Obseria
Keep PromQL
Query Prometheus metrics in Obseria with PromQL, or switch to OQL and SQL when you want to join metrics with logs and traces.
Long-term storage without sharding
Metric data points are retained for 13 months in Obseria Lake, so you no longer operate Thanos, Cortex or federation layers just for history.
Context beyond metrics
TopoGraph links each series to the pod, host and service it came from, and Obseria Intelligence correlates metric anomalies with traces and logs.
Prometheus sources Obseria ingests
- Prometheus server. Existing servers forward samples through remote write with a bearer token and per-series relabeling.
- Prometheus Agent mode. Run Prometheus in agent mode as a lightweight scraper that only forwards data, without a local TSDB to manage.
- Exporters. node_exporter, blackbox_exporter, mysqld_exporter, kube-state-metrics and any other /metrics endpoint keep working unchanged.
- OpenTelemetry Collector. The prometheus receiver scrapes targets with standard scrape_configs and sends the data via OTLP.
- Kubernetes service discovery. Pods, services and endpoints discovered by Prometheus or the Obseria Agent DaemonSet are scraped automatically.
- Client libraries. Applications instrumented with Prometheus client libraries for Go, Java, Python or Node.js need no code changes.
The Prometheus signals that matter
Obseria collects these values automatically and shows them in ready-made dashboards and alerts.Data collection health
Shows whether all monitored systems are reachable and deliver their data on time, so no blind spots appear.
Data volume
Tells you how much monitoring data is collected and stored, which drives both storage needs and cost.
Delivery to Obseria
Reveals whether data is forwarded reliably and quickly, or whether it is piling up or getting lost on the way.
Alert reliability
Highlights alert rules that fail to run, meaning important warnings might never reach your team.
Show all 8 metrics in detail
| Metric | Unit | What it tells you |
|---|---|---|
up | 1 | Whether the last scrape of a target succeeded; the first signal of a broken exporter. |
scrape_duration_seconds | s | How long each scrape took; values near the scrape timeout lead to gaps. |
scrape_samples_scraped | {samples} | Samples returned per scrape; sudden jumps point to a cardinality explosion. |
prometheus_tsdb_head_series | {series} | Active series in the head block, the main driver of Prometheus memory use. |
prometheus_remote_storage_samples_pending | {samples} | Samples queued for remote write; a growing queue means the endpoint cannot keep up. |
prometheus_remote_storage_samples_failed_total | {samples} | Samples that failed to send with a non-recoverable error. |
prometheus_remote_storage_shards | {shards} | Parallel remote write shards currently in use; pinned at max_shards means throttling. |
prometheus_rule_evaluation_failures_total | {evaluations} | Recording and alerting rules that failed to evaluate. |
Set up Prometheus monitoring in minutes
- Step 1
Create an ingest token
Create a token with metric ingest scope in Obseria and store it as a file or Kubernetes secret next to Prometheus.
- Step 2
Add remote_write
Add a remote_write block with the Obseria endpoint and bearer token, then reload Prometheus. Optionally drop series you never query.
- Step 3
Query and alert
Open the Prometheus dashboards, run your PromQL queries and turn the important ones into alerts and SLOs.
# prometheus.yml: forward every scraped series to Obseria
remote_write:
- url: "https://<your-obseria-remote-write-endpoint>"
authorization:
type: Bearer
credentials_file: /etc/prometheus/obseria-token
queue_config:
capacity: 10000
max_shards: 50
max_samples_per_send: 2000
write_relabel_configs:
# drop high-churn series you never query
- source_labels: [__name__]
regex: "go_gc_duration_seconds.*"
action: drop
Out of the box
Dashboards, alerts and SLOs that are ready the moment data arrives.
- Prometheus remote write health and queue backlog
- Scrape target availability (up) by job
- Series cardinality by metric and label
- Alert: target down for more than 5 minutes
- Kubernetes cluster overview from kube-state-metrics
Prometheus monitoring: what matters
Why Prometheus monitoring needs more than one server
Prometheus is the default metrics system in cloud native environments. A single server with local storage is easy to run, but it was designed for a short retention window and for a single failure domain. As clusters multiply, teams end up running several servers per region, federation, and a long-term storage layer such as Thanos or Cortex. Prometheus monitoring then becomes two jobs: monitoring your workloads, and keeping the monitoring itself alive.
Obseria takes over the second job. Your Prometheus servers or agents keep scraping exporters as before and forward samples via Prometheus remote write. Obseria stores them for 13 months, answers PromQL queries and puts every series into the context of your topology, traces and logs.
The Prometheus metrics that matter
Every Prometheus setup should watch itself with a small set of meta metrics:
- Target health:
upis 1 for every successful scrape. Alert onup == 0per job, not per instance, to avoid alert storms during rollouts. - Scrape cost:
scrape_duration_secondsandscrape_samples_scrapedshow slow exporters and targets that suddenly expose far more series. - Cardinality:
prometheus_tsdb_head_seriesis the number of active series. It drives memory and is the first metric to check when Prometheus is OOM-killed. - Remote write health:
prometheus_remote_storage_samples_pending,prometheus_remote_storage_samples_failed_totalandprometheus_remote_storage_shardstell you whether data actually leaves the server. - Rule evaluation:
prometheus_rule_evaluation_failures_totalcatches recording and alerting rules that silently stopped working.
Common problems and how to detect them
Cardinality explosions
A label with user IDs, request paths or pod hashes can multiply series overnight. Watch the growth rate of head series and samples per scrape, and use write_relabel_configs to drop or aggregate labels before they leave the server. In Obseria, the Pipeline can additionally drop or rewrite labels at ingest.
Remote write falling behind
If the pending sample count grows and shards sit at max_shards, the network or the receiving endpoint is the bottleneck. A query such as rate(prometheus_remote_storage_samples_failed_total[5m]) > 0 is a good alert. Tune capacity, max_shards and max_samples_per_send in queue_config instead of raising scrape intervals.
Gaps after restarts
Local TSDB data is lost when a pod without a persistent volume restarts. With remote write, the history lives in Obseria, and Prometheus in agent mode only needs a small write-ahead log.
Best practices for scaling Prometheus
- Use Prometheus agent mode or the OpenTelemetry Collector at the edge and keep only one central store for queries.
- Add an external label for cluster and region so series from different servers never collide.
- Set
sample_limitper scrape job so a single misbehaving target cannot overload the server. - Follow naming conventions (
_total,_seconds,_bytes) so metric types are clear in every tool. - Review the top series by metric name regularly and delete what nobody queries.
How Obseria helps with Prometheus monitoring
Obseria accepts Prometheus data through remote write, through the prometheus receiver of the OpenTelemetry Collector, or by letting the Obseria Agent scrape /metrics endpoints directly. PromQL works for dashboards and alerts, and OQL lets you join the same metrics with logs and spans. TopoGraph maps series to Kubernetes workloads and hosts, so an alert on a latency histogram already shows which service and which deployment are affected. Obseria Intelligence detects anomalies on your Prometheus metrics and explains them with a chain of evidence. You pay per metric data point ingested and stored, not per host, and data points dropped in the Pipeline only cost ingestion.
A typical migration runs in parallel: add remote write to one cluster, compare dashboards and alerts side by side for a few days, then roll the same configuration out to the remaining servers. Because the scrape layer does not change, exporters, ServiceMonitors and recording rules stay exactly where they are, and you can stop the remote write at any time.
Related integrations
Related capabilities
Kubernetes Monitoring
See every cluster, node, pod and workload in context. Obseria connects Kubernetes metrics, events, logs and traces so you find the cause of a failing rollout in minutes.
Learn moreAnomaly Detection
Obseria Intelligence learns the normal rhythm of every service, flags real deviations instead of threshold noise and groups related anomalies into one problem with a probable root cause.
Learn moreLearn more
What is observability?
What is observability? Learn how metrics, logs, traces and profiles let you understand any system state, how it differs from monitoring, and how to start.
What is OpenTelemetry?
What is OpenTelemetry? Learn how OTel's APIs, SDKs, Collector, OTLP and semantic conventions produce vendor-neutral traces, metrics, logs and profiles.
Prometheus monitoring FAQ
Yes. Obseria supports PromQL for queries, dashboards and alerts. You can also use OQL or SQL on the same metrics when you want to correlate them with logs and traces.
No. Add a remote_write block and your servers keep scraping as before. Many teams later switch to Prometheus agent mode or the OpenTelemetry Collector to reduce the local footprint.
Metric data points are retained for 13 months, so you do not need a separate long-term storage layer such as Thanos or Cortex.
Drop series with write_relabel_configs before sending, or filter and aggregate them in Obseria Pipeline at ingest. Filtered data points are only charged for ingestion.
Yes. The Prometheus resource of the Operator supports a remoteWrite section, so you add the Obseria endpoint and a secret reference there. ServiceMonitors and PodMonitors stay as they are.
Start monitoring Prometheus today
Connect Prometheus in minutes with OpenTelemetry or the Obseria Agent. Free for 14 days.