PromQL

What is PromQL?

PromQL is the query language of Prometheus and the de facto standard for querying metrics in cloud-native environments. Many other systems, including Thanos, Cortex, Mimir, VictoriaMetrics and several commercial backends, accept PromQL or a close dialect. Every Grafana panel on Prometheus data, every Prometheus alerting rule and every recording rule is a PromQL expression.

PromQL is functional rather than SQL-like: you do not write SELECT ... FROM, you compose expressions. An expression evaluates to one of four types: an instant vector (one sample per series at a single timestamp), a range vector (a window of samples per series), a scalar or a string.

Selectors and label matchers

The simplest query is a metric name, which returns the latest value of every matching series. Curly braces filter by labels:

http_requests_total{job="checkout", status=~"5..", method!="OPTIONS"}

Four matchers exist: = (equal), != (not equal), =~ (regex match) and !~ (regex non-match). Regexes are fully anchored, so "5.." matches 500 to 599 but not 1500. Appending a duration in square brackets, for example [5m], turns an instant vector into a range vector, which is what functions like rate() expect.

rate(), aggregation and histogram_quantile()

Rates from counters

Counters only go up and reset to zero when a process restarts, so their raw value is rarely useful. rate() calculates the per-second average increase over a window and handles resets:

rate(http_requests_total{job="checkout"}[5m])

increase() returns the total increase over the window (rate multiplied by the window length), and irate() uses only the last two samples, which makes it more responsive but noisier. As a rule of thumb, the window should be at least four times the scrape interval.

Aggregation with sum by

Aggregation operators such as sum, avg, max, count and topk collapse series. by keeps the listed labels, without drops them:

sum by (service) (rate(http_requests_total[5m]))

This returns request throughput per service. The order matters: always apply rate() first, then sum. Summing raw counters before taking the rate hides counter resets and produces wrong results.

Percentiles from histograms

Classic Prometheus histograms are exposed as cumulative _bucket series with an le (less or equal) label. histogram_quantile() estimates a quantile from the bucket rates:

histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

Keep le in the by clause, otherwise the function has no buckets to work with. The result is an estimate that is only as precise as the bucket boundaries; native histograms in newer Prometheus versions improve this without manual bucket design.

Example: error ratio for an SLO

Binary operators combine expressions. Dividing two aggregated rates gives an error ratio per service, a typical golden signal and SLI:

sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))

Wrapped in an alerting rule with a threshold, for example > 0.01 and a for: 10m clause, this pages only when more than one percent of requests fail for ten minutes. For SLO-based alerting, the same ratio over several windows feeds multi-window burn-rate alerts. Other useful building blocks are the offset modifier for week-over-week comparisons and recording rules that precompute expensive expressions.

Recording and alerting rules

In Prometheus, rules live in YAML rule files that the server evaluates at a fixed interval. A recording rule stores the result of an expression as a new time series; an alerting rule fires when its expression returns any series for longer than the for duration:

groups:
  - name: checkout
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))
      - alert: CheckoutHighErrorRatio
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
          / sum by (service) (rate(http_requests_total[5m])) > 0.01
        for: 10m
        labels:
          severity: page

Recording rules make dashboards faster and keep alert expressions short and readable, because the expensive aggregation runs once per evaluation instead of on every page load.

Best practices

• Rate before you aggregate, and aggregate before you compute ratios or quantiles.
• Use recording rules for expressions that dashboards and alerts evaluate often, with the naming scheme level:metric:operations, for example service:http_requests:rate5m.
• Filter early. Narrow label matchers reduce the number of series loaded and keep queries fast.
• Watch cardinality. A query over a label with millions of values is slow no matter how well it is written.
• Be explicit about vector matching. When dividing series with different label sets, use on(), ignoring() and group_left deliberately.

Common mistakes

• Applying rate() to a gauge, or graphing a counter's raw value.
• Averaging precomputed quantiles from summaries across instances.
• Choosing a range shorter than two scrape intervals, which yields empty or jumpy results.
• Dropping le before histogram_quantile().

How Obseria helps

Obseria supports PromQL alongside OQL and SQL, so existing Prometheus dashboards and alert expressions keep working when you move to Obseria. Prometheus endpoints can be scraped or data sent via remote write and OTLP, see Prometheus monitoring. Metric data points are retained for 13 months, which makes PromQL comparisons with offset across quarters practical.