Log aggregation is the process of collecting log records from many sources, such as servers, containers, applications and cloud services, and consolidating them in one central system. Along the way logs are parsed, enriched with metadata and normalized, so that they can be searched, correlated with metrics and traces, retained and analyzed in one place.
What log aggregation is
A single request in a modern system can produce log lines on a load balancer, an ingress controller, three microservices and a database, spread across dozens of nodes that may no longer exist an hour later. Log aggregation brings these lines together. Instead of connecting to individual machines with ssh and grep, engineers query one central store that holds logs from every source, with consistent timestamps and metadata.
Aggregation is the foundation for everything else teams do with logs: search during incidents, alerting on patterns, security investigations, audit requirements and correlation with metrics and traces.
The log aggregation pipeline
- Collect. An agent on each host reads log files, container output, journald or syslog. In Kubernetes, a node-level agent typically tails the files the container runtime writes under
/var/log/pods. Applications can also send logs directly via OTLP. - Parse. Turn raw lines into fields: JSON decoding, regular expressions or grok patterns for unstructured formats, and multiline handling so a Java stack trace becomes one record instead of forty.
- Enrich. Add context such as service name, Kubernetes namespace, pod, node, cloud region and, ideally, trace and span IDs.
- Filter and transform. Drop debug noise and health-check lines, mask personal data, sample very chatty sources.
- Buffer and ship. Batch, compress and send to the backend, with disk buffering and backpressure when the destination is slow.
- Store and index. Persist logs for search and analysis according to a retention policy.
Common collectors include Fluent Bit, Fluentd, Vector, Logstash and the OpenTelemetry Collector with its filelog receiver.
Storage models
| Model | How it works | Trade-off |
|---|---|---|
| Full-text index | Builds an inverted index over log content at ingest | Fast arbitrary search; indexing cost and storage overhead grow with volume |
| Label index only | Indexes a few labels, stores content in compressed chunks | Cheap ingest; content searches scan more data, labels must stay low in cardinality |
| Index-free lakehouse | Columnar storage on object storage, schema-on-read, massively parallel scans | No index to manage or pay for; query speed depends on the scan engine |
Why it matters
Without aggregation, logs from short-lived containers disappear when a pod is rescheduled, and incident responders lose the very evidence they need. Central logs also make it possible to answer cross-cutting questions, for example which services logged connection errors to a specific database in the last ten minutes, and to meet regulatory requirements for audit trails and retention.
Scaling the pipeline
Small setups ship logs straight from node agents to the backend. As volume grows, most teams add a gateway tier: a horizontally scaled pool of collectors that receives data from all agents and applies central processing such as PII masking, routing to different destinations and rate limiting. Some architectures also place a durable message queue such as Kafka between collection and storage to absorb bursts and allow replay. Whatever the layout, every stage needs bounded buffers and a clear policy for what happens under backpressure: block, spill to disk or drop the least valuable data first.
Example: enriching container logs
An illustrative OpenTelemetry Collector snippet that tails Kubernetes container logs, parses the container runtime format and adds pod metadata:
receivers:
filelog:
include: [ /var/log/pods/*/*/*.log ]
operators:
- type: container
processors:
k8sattributes: {}
batch: {}
service:
pipelines:
logs:
receivers: [filelog]
processors: [k8sattributes, batch]
exporters: [otlp]Best practices
- Log in a structured format. Structured logging as JSON or OpenTelemetry log records removes most fragile parsing.
- Include trace context. A
trace_idin each log line lets you jump from a slow trace to exactly the related logs. - Use UTC and precise timestamps, and keep the original event time rather than the ingestion time.
- Filter at the edge. Decide in the pipeline what is worth storing; dropping or summarizing noisy logs early is the most effective cost lever.
- Mask sensitive data before it leaves the host or pipeline, not after it has been stored.
- Monitor the pipeline itself: dropped records, buffer usage and end-to-end delay.
Common mistakes
Typical problems are unbounded debug logging in production, multiline stack traces split into separate records, missing service names that make logs impossible to attribute, and clocks out of sync across hosts. Another frequent issue is shipping everything to an expensive indexed tier and then reducing retention to a few days to control cost, which removes the history needed for slow-burning problems.
How Obseria helps
Obseria Log Analytics stores logs in Obseria Lake, an index-free lakehouse with schema-on-read, so new fields are searchable without reindexing. Obseria Pipeline filters, enriches and masks PII at ingest, and filtered data costs only ingestion. Logs are linked to traces and topology automatically and can be queried with OQL or SQL.
