Instrumentation

What instrumentation is

Software does not explain itself. To know how long a request took, which database query it ran or why it failed, the code has to record that information and send it somewhere. Instrumentation is everything that makes this happen: API calls that create spans, counters and histograms that record measurements, loggers that write structured events, and agents that do some or all of this automatically.

Instrumentation is the first link in the telemetry chain. Collectors, pipelines, storage and dashboards can only work with what the instrumentation produced. If a critical code path is not instrumented, no backend can show you what happened in it.

Why instrumentation matters

The quality of an investigation is limited by the quality of the instrumentation behind it. Good instrumentation produces consistent service names, meaningful span boundaries, standard attribute names and correlated signals, so a latency spike on a dashboard can be followed to a trace, a log line and a profile in a few clicks. Poor instrumentation produces disconnected fragments: metrics without context, logs without trace IDs and traces that stop at the first message queue. Instrumentation is also where cost and privacy are decided first, because every attribute that is recorded must later be transported, stored and protected.

Automatic versus manual instrumentation

Aspect	Automatic (zero-code)	Manual (code-based)
How	Agent or library hooks into frameworks and clients at runtime	Developers call the OpenTelemetry API in application code
Coverage	HTTP servers and clients, gRPC, databases, messaging, popular frameworks	Business operations, custom protocols, domain attributes
Effort	Minutes, often only configuration	Ongoing development work
Context	Technical: routes, status codes, queries	Business: order value, tenant, feature flag

Most teams combine both. Automatic instrumentation gives broad coverage on day one; manual instrumentation adds the spans and attributes that make traces meaningful for the business. A third option, kernel-level instrumentation with eBPF, collects data from outside the process without touching code or runtimes.

Instrumentation with OpenTelemetry

OpenTelemetry separates the API, which libraries and applications call, from the SDK, which processes and exports the data. Library authors can instrument against the API without forcing a specific backend, and application owners choose the SDK configuration and exporter, typically OTLP. Semantic conventions define standard attribute names such as http.request.method, http.route and db.system, so telemetry from different languages looks consistent.

Zero-code options exist for many languages. Illustrative start commands:

# Java: attach the agent
OTEL_SERVICE_NAME=checkout java -javaagent:opentelemetry-javaagent.jar -jar app.jar

# Python: install instrumentations, then run through the wrapper
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=checkout opentelemetry-instrument python app.py

# Node.js: register auto-instrumentations at startup
OTEL_SERVICE_NAME=checkout node --require @opentelemetry/auto-instrumentations-node/register app.js

In Kubernetes, the OpenTelemetry Operator can inject these agents into pods based on annotations, so teams do not have to change container images.

Example: a manual span

Automatic instrumentation shows that a POST /checkout request took 900 ms. Manual instrumentation shows why:

with tracer.start_as_current_span("apply_discounts") as span:
    span.set_attribute("cart.items", len(cart.items))
    span.set_attribute("discount.rules_evaluated", rules_count)
    total = discount_engine.apply(cart)

The new span appears as a child of the automatically created server span, with attributes that let you filter slow traces by cart size.

Best practices

1. Start with automatic instrumentation to get service maps and request metrics quickly.
2. Set resource attributes such as service.name, service.version and deployment.environment.name everywhere; they are the backbone of filtering and correlation.
3. Add manual spans for business-critical paths, not for every function. Spans should represent meaningful units of work.
4. Follow semantic conventions instead of inventing attribute names per team.
5. Ensure context propagation across threads, queues and service boundaries, otherwise traces break apart.
6. Protect sensitive data. Do not record passwords, tokens or full request bodies; mask personal data at the source or in the pipeline.

Common mistakes

Over-instrumentation is as harmful as under-instrumentation: spans for every getter inflate data volume and hide the important work. Putting unbounded values such as user IDs into metric attributes creates cardinality problems. Proprietary agents with vendor-specific APIs make every later migration expensive. And instrumentation that is never verified in production, for example missing on a critical batch job, only shows its gaps during an incident.

How Obseria helps

Obseria is OpenTelemetry-native: existing OpenTelemetry SDKs and agents send data over OTLP, and the first trace typically arrives within 15 minutes. The Obseria Agent, installed once per host, adds infrastructure metrics, logs, profiles and security signals. Language guides cover Java, Python and Node.js, and What is OpenTelemetry? explains the underlying standard.