Agent (Observability Agent)

What an observability agent is

An observability agent is software that runs next to your workloads and turns local signals into telemetry a backend can use. On a virtual machine it is typically a system service; in Kubernetes it usually runs as a DaemonSet, so exactly one agent pod is scheduled on every node. The agent reads system counters, tails log files, receives traces and metrics from applications over local ports, and ships everything in batches.

The word is used for several kinds of software. Classic monitoring agents collect host metrics such as CPU, memory, disk and network. Log shippers like Fluent Bit or Vector focus on files and streams. The OpenTelemetry Collector can be deployed in "agent mode" on every node, as opposed to "gateway mode" where a central pool of collectors receives data from many agents. Some vendors also call in-process language libraries "agents", for example a Java agent attached with -javaagent, which instruments bytecode at load time.

How an agent works

Most agents follow the same pipeline, whatever their name:

1. Receive or scrape. Pull metrics from endpoints such as a Prometheus /metrics path, read /proc and cgroup statistics, tail container log files, or accept OTLP data pushed by instrumented applications on localhost:4317 (gRPC) and 4318 (HTTP).
2. Enrich. Attach resource attributes such as host.name, k8s.pod.name, k8s.namespace.name or cloud region, so every signal can be correlated later.
3. Process. Filter noisy data, drop or hash sensitive fields, batch and compress.
4. Buffer and export. Hold data in memory or on disk when the backend is slow or unreachable, retry with backoff, and send it onward.

Agents that use eBPF can additionally observe network flows, system calls or CPU stacks from the kernel without changing application code, which is how many continuous profilers and network monitors work.

Agent, gateway and agentless collection

Approach	Where it runs	Typical use
Agent	On every host or node	Host metrics, logs from local files, local OTLP endpoint, profiling
Gateway	Central pool of collectors	Tail-based sampling, routing, central policy, credentials
Agentless	Backend calls provider APIs	Managed cloud services, SaaS APIs, synthetic checks

The approaches complement each other. You cannot install an agent on a managed database or a load balancer, so those are monitored through cloud APIs. Conversely, nothing outside the host can see per-process CPU or local log files as precisely as an agent.

Example: OpenTelemetry Collector in agent mode

An illustrative minimal configuration for a node agent that receives OTLP, scrapes host metrics and forwards everything to a gateway:

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
  hostmetrics:
    collection_interval: 30s
    scrapers: { cpu: {}, memory: {}, filesystem: {}, network: {} }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80 }
  resourcedetection: { detectors: [env, system] }
  batch: {}
exporters:
  otlp:
    endpoint: gateway.observability.svc:4317
service:
  pipelines:
    metrics: { receivers: [otlp, hostmetrics], processors: [memory_limiter, resourcedetection, batch], exporters: [otlp] }
    traces:  { receivers: [otlp], processors: [memory_limiter, resourcedetection, batch], exporters: [otlp] }

Note the memory_limiter placed first: it protects the node from the agent itself when traffic spikes.

Best practices

• Set resource limits. Give the agent explicit CPU and memory requests and limits and an internal memory limiter, so it can never starve the workloads it observes.
• Monitor the agent. Track its own queue length, dropped data and export errors. A silent agent looks exactly like a healthy, quiet service.
• Prefer open protocols. Agents that speak OTLP let you change backends without redeploying instrumentation.
• Standardize rollout. Manage agent versions and configuration as code through Helm, an operator or configuration management, not by hand on individual hosts.
• Keep processing light at the edge. Do cheap enrichment and filtering in the agent; leave expensive, stateful work such as tail-based sampling to a gateway.

Common mistakes

Running several overlapping agents per host, one per tool, is a frequent source of overhead and inconsistent metadata. Another mistake is collecting everything at maximum resolution "just in case", which moves cost from compute to storage. Finally, many teams forget to pin agent versions, so a node upgrade silently changes metric names or labels.

How Obseria helps

The Obseria Agent is installed once per host or as a DaemonSet and collects metrics, logs, traces, profiles and security signals in one process, enriched with the topology that Obseria TopoGraph builds automatically. It accepts OTLP from your existing SDKs, so OpenTelemetry instrumentation keeps working, and supports Kubernetes monitoring and continuous profiling without extra sidecars.