Label (Attribute)

What a label is

A label adds context to a measurement. The number 412 alone means nothing; http_requests_total{service="checkout", method="POST", status="500"} 412 tells you which service, which method and which outcome it counts. Labels let you slice data by these dimensions in queries, dashboards and alerts.

Different systems use different words for the same concept:

System	Term	Example
Prometheus	Label	job="api", instance="10.0.0.5:9100"
OpenTelemetry	Attribute (resource or signal level)	service.name, http.route
Kubernetes	Label (object metadata)	app.kubernetes.io/name: checkout
Many APM tools	Tag or dimension	env:prod

How labels work in Prometheus and OpenTelemetry

In Prometheus, a time series is identified by its metric name plus its full label set; the name itself is stored internally as the label __name__. Change one label value and you get a different series. Labels starting with a double underscore are reserved for internal use, for example the __meta_kubernetes_pod_label_* labels that service discovery provides during relabeling.

OpenTelemetry distinguishes two levels. Resource attributes describe the entity producing telemetry, such as service.name, service.version, k8s.pod.name or cloud.region, and apply to everything that entity emits. Signal attributes belong to an individual span, log record or metric data point, such as http.request.method or http.response.status_code. The semantic conventions standardize these names so that every language SDK and backend uses the same keys.

Labels and cardinality

Every distinct combination of label values on a metric creates a new time series. A metric with 20 services, 5 methods, 10 status codes and 50 pods can reach 50,000 series. Add a label with user IDs and the number explodes. This is why labels are the main driver of cardinality and, with it, of memory use, query speed and cost in metric systems.

The rule of thumb: metric labels should have bounded, low-cardinality values. Put unbounded identifiers such as user IDs, request IDs or full URLs on spans and logs, where they are expected, and use templated values like http.route="/users/{id}" instead of raw paths on metrics. Log systems that index labels, such as Loki, apply the same principle to stream labels.

Example: relabeling

Relabeling changes labels before data is stored. An illustrative Prometheus scrape configuration that copies a Kubernetes label and drops a noisy one:

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
    target_label: app
metric_relabel_configs:
  - action: labeldrop
    regex: pod_template_hash

relabel_configs act on targets before the scrape, metric_relabel_configs on samples after it. In OpenTelemetry the equivalent work happens in the Collector, for example with the attributes, resource or transform processors in a telemetry pipeline.

Labels in Kubernetes

In Kubernetes, labels are not just metadata for monitoring; they are how the platform itself works. Services, Deployments and NetworkPolicies select pods through label selectors such as app.kubernetes.io/name=checkout. The recommended label set includes app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/component, app.kubernetes.io/part-of and app.kubernetes.io/managed-by. Monitoring tools pick these up and turn them into Prometheus labels or OpenTelemetry resource attributes, for example through the Collector's Kubernetes attributes processor. That makes clean Kubernetes labeling the cheapest way to get consistent service names and ownership across all your telemetry. Agree on a small mandatory set, such as name, team and environment, and enforce it with admission policies so new workloads cannot be deployed without it.

Best practices

• Follow conventions. Use OpenTelemetry semantic conventions and the recommended Kubernetes labels instead of inventing names per team.
• Be consistent across signals. The same service.name on metrics, logs and traces is what makes correlation work.
• Keep metric labels bounded and review new labels for their value range before they reach production.
• Never put secrets or personal data in labels. They are copied into indexes, dashboards and alerts.
• Prefer enrichment at the source or pipeline over ad-hoc joins at query time.

Common mistakes

Common mistakes include using different keys for the same thing (env, environment, stage), putting timestamps or IDs into metric labels, encoding information in metric names instead of labels (requests_checkout_total instead of a service label), and relying on labels that change on every deploy, which breaks series continuity in PromQL queries.

How Obseria helps

Obseria accepts OpenTelemetry attributes and Prometheus labels and keeps them consistent across metrics, logs and traces, so a filter on service.name works everywhere. Obseria Lake stores logs and spans index-free with schema-on-read, so high-cardinality attributes there do not need to be declared in advance, and Obseria Pipeline can enrich, drop or mask attributes at ingest. In Kubernetes, pod and namespace metadata is attached automatically through Kubernetes monitoring.