What is observability?

Observability: a precise definition

Observability is a property of a system, not a product you buy. The term comes from control theory, where Rudolf Kálmán defined a system as observable if its internal state can be inferred from its external outputs. In software, the outputs are telemetry: the numbers, events and request paths a service emits while it runs. A system is highly observable when that telemetry is rich enough, and connected enough, that an engineer can explain any behavior, including behaviors nobody anticipated when the dashboards were built.

That last point is the important one. Traditional monitoring answers predefined questions: is CPU above 90 percent, is the error rate above 1 percent. Observability is about unknown unknowns: why did checkout latency rise only for customers on one mobile app version, only in one region, only when a specific feature flag was on? You cannot write an alert for every such combination in advance. You need data with enough dimensions and context to slice, compare and follow it to the cause.

Why observability matters now

A decade ago, many applications were a monolith on a few servers. When something broke, an engineer could log in, read one log file and usually find the problem. Modern systems look different:

• Distributed architectures. A single user request may cross an API gateway, a dozen microservices, message queues, managed databases and third-party APIs.
• Ephemeral infrastructure. Containers and serverless functions live for minutes or seconds. By the time you investigate, the pod that failed may no longer exist.
• Continuous delivery. Teams deploy many times a day, so the system you debug today is not the system you debugged last week.
• High cardinality. Problems often affect a narrow slice of traffic defined by user ID, tenant, region, version or flag, which aggregate metrics hide.

In this environment the cost of not knowing is high: longer incidents, higher mean time to resolution (MTTR), missed service level objectives (SLOs) and engineers who spend their time guessing. Observability shortens the path from symptom to cause and gives teams the confidence to ship faster.

The signals: metrics, logs, traces and profiles

Observability is often described with "three pillars". The pillars are useful vocabulary, but the real value comes from correlating them. Today most practitioners count four core signal types:

Signal	What it is	Best for	Typical limitation
Metrics	Numeric measurements aggregated over time (counters, gauges, histograms)	Trends, alerting, SLOs, capacity planning	Aggregation hides individual requests; high cardinality is expensive
Logs	Timestamped records of discrete events, ideally structured (JSON, key-value)	Detailed context, errors, audit trails	Volume and cost; hard to follow across services without IDs
Traces	The path of a single request through services, made of spans with timing and attributes	Latency breakdowns, dependencies, finding the slow or failing hop	Requires context propagation; usually sampled
Profiles	Continuous samples of where code spends CPU, memory or lock time	Explaining why a function is slow or costly, down to the line	Newer signal; tooling and standards are still maturing

Beyond these, real-user monitoring (RUM) events, synthetic checks, Kubernetes events, deployment markers and business events all add context. What turns this pile of data into observability is correlation: a trace ID in every log line, the same resource attributes (service name, version, host, pod) on every signal, and a topology model that knows which service depends on which. With that in place you can jump from a latency spike on a metric chart to the slow traces behind it, to the logs of the failing span, to the profile of the hot function.

How observability works in practice

An observability practice has four stages, and weakness in any one of them limits the whole:

1. Instrumentation. Code and infrastructure emit telemetry. This happens through auto-instrumentation agents, SDKs such as OpenTelemetry, eBPF-based collection, cloud provider APIs and log shippers. Good instrumentation adds rich, consistent attributes, not just raw numbers.
2. Collection and processing. Pipelines such as the OpenTelemetry Collector receive data, batch it, drop noise, enrich it with metadata (for example Kubernetes labels), mask personal data and route it to one or more backends.
3. Storage and query. A backend stores signals so they can be queried quickly and flexibly, including by high-cardinality fields. Storage design determines both what questions you can ask and what it costs.
4. Analysis and action. Dashboards, alerts, SLOs, trace views, log search and increasingly AI-assisted root cause analysis turn data into decisions: page someone, roll back a deployment, scale a service, fix a query.

A useful mental model is the debugging loop: notice a symptom (often via an SLO burn alert), form a hypothesis, slice the data to test it, and repeat until the cause is isolated. The faster each iteration of that loop, the more observable your system is in practice.

Key concepts you need to know

A handful of technical ideas determine whether an observability setup actually works under pressure.

Cardinality and dimensionality

Cardinality is the number of unique values a field can take. A status code has a handful; a user ID or container ID can have millions. Dimensionality is the number of fields attached to each event. Observability depends on both being high, because narrow failures are only visible when you can group by the field that defines them. Backends differ widely in how they handle this: many metric databases create one time series per unique label combination, which gets expensive quickly, while columnar and lakehouse-style stores can query wide, high-cardinality events more economically.

Context propagation

Distributed tracing works because each service passes a trace context to the next, typically via the W3C Trace Context traceparent HTTP header or equivalent message metadata. If one hop drops that header, the trace breaks into fragments. Propagating context through queues, async jobs and third-party calls is one of the most common gaps in real systems.

Sampling

Keeping every trace is rarely affordable at scale. Head-based sampling decides at the start of a request, which is cheap but blind to the outcome. Tail-based sampling waits until a trace is complete and keeps errors, slow requests and rare paths, which is more useful for debugging but needs a collector tier that can buffer spans.

Service level objectives

SLOs define what "good enough" means for users, for example 99.9 percent of checkout requests succeeding within 400 ms over 30 days. They connect observability data to business priorities and tell teams when reliability work should take precedence over new features.

Best practices for building observability

• Standardize on OpenTelemetry. Vendor-neutral instrumentation and semantic conventions make attributes consistent across teams and keep you free to change backends.
• Use structured, contextual data. Emit structured logs with trace and span IDs. Treat attributes such as customer tier, region and build version as first-class fields.
• Start from user-facing signals. Define SLOs on latency and error rate for critical user journeys, and alert on SLO burn rather than on every CPU spike.
• Apply proven frameworks. Use the golden signals (latency, traffic, errors, saturation), RED for services and USE for resources to decide what to measure first.
• Control data volume deliberately. Filter debug noise at ingest, use tail-based sampling to keep interesting traces, and tier retention by value rather than collecting everything forever.
• Make it part of delivery. Include instrumentation in the definition of done, mark deployments in telemetry and review observability gaps in every post-incident review.

Common observability pitfalls

• Tool sprawl. Separate tools for metrics, logs and traces mean manual correlation across browser tabs during an incident, exactly when time is scarce.
• Dashboards as a substitute for questions. Hundreds of dashboards nobody looks at do not make a system observable. Exploratory query matters more than static charts.
• Dropping cardinality to save money. Removing user, tenant or version dimensions cuts cost but also removes the context you need to find narrow failures.
• Alert fatigue. Threshold alerts on every resource metric train people to ignore pages. Fewer, symptom-based alerts work better.
• Ignoring privacy. Telemetry often contains personal data in URLs, headers and log messages. Masking must happen before data is stored, not after.

How Obseria helps

Obseria is an OpenTelemetry-native observability platform that keeps metrics, logs, traces, profiles and user experience data in one place. Obseria Lake stores all signals index-free with schema-on-read, so you can query high-cardinality fields without deciding up front what to index. Obseria Pipeline filters, enriches and masks PII at ingest, Obseria TopoGraph maps dependencies automatically, and Obseria Intelligence runs agentic root-cause investigations that show their chain of evidence. You can query with OQL, PromQL or SQL, and because data arrives via OTLP and can be exported via OTLP at any time, your instrumentation stays portable.