Data retention in observability is the period for which telemetry such as logs, traces, metrics and RUM events is stored and queryable before it is deleted, downsampled or archived. Retention policies balance troubleshooting needs, trend analysis, compliance obligations, privacy rules and storage cost, and are often set differently per signal type.
What is data retention?
Data retention describes how long telemetry stays available after it has been ingested. For an observability platform this is a central design decision: retention determines whether you can still look at the logs of last month's incident, compare this quarter's latency with last year's, or answer an auditor's question about who accessed a system. At the same time every stored byte costs money, and personal data must not be kept longer than necessary.
Retention is rarely a single number. Different signals have different value over time, so most organizations define a policy per signal type, per environment and sometimes per data class.
Typical retention by signal type
| Signal | Main use over time | Common retention pattern |
|---|---|---|
| Traces and spans | Debugging individual requests, recent performance analysis | Days to weeks, often sampled |
| Logs | Incident investigation, audits, security forensics | Weeks for operational logs; longer, archived storage for audit and security logs |
| Metrics | Trends, capacity planning, SLO reporting, seasonality | Months to years, frequently downsampled |
| RUM and session data | User experience analysis, release comparisons | Weeks, with strict privacy controls |
The pattern follows the data volume: traces and logs are large and most valuable while fresh; metrics are compact and gain value when you can compare long periods. A self-managed Prometheus server, for example, keeps local data for 15 days by default unless you change --storage.tsdb.retention.time, which is why teams add long-term storage for metrics.
Why retention matters
- Troubleshooting: slow-burning problems, such as a memory leak or a gradual latency regression, are only visible if the data before the change still exists.
- Seasonality: year-over-year comparisons for Black Friday, quarter-end or annual batch jobs require at least 13 months of metrics, so the same period last year is still available.
- Compliance: security, financial and industry regulations can require that certain logs are kept for a defined minimum period.
- Privacy: the GDPR principle of storage limitation (Article 5(1)(e)) requires that personal data is not kept longer than needed for its purpose. Long retention of logs with user identifiers is a liability.
- Cost: storage grows linearly with retention. Doubling retention roughly doubles the stored volume you pay for.
Techniques to keep data longer for less
Downsampling and rollups
Metrics can be aggregated to coarser resolution as they age, for example from 10-second points to 5-minute and then 1-hour aggregates. Trends stay visible while storage shrinks dramatically. Keep minimum, maximum, sum and count when rolling up, so you do not lose spikes.
Filter and reduce at ingest
The cheapest data is data you never store. Drop debug logs in production, remove health-check noise, and use sampling for high-volume traces while keeping all errors and slow requests. Converting repetitive logs into metrics preserves the signal at a fraction of the volume.
Tiering and archiving
Hot storage serves fast interactive queries; older data can move to cheaper object storage that is slower to query but still available for audits. Make sure archived data stays in an open format so you can read it later without the original tool.
Derive long-lived signals from short-lived data
A practical pattern is to extract what you need long term before raw data expires. Span metrics, for example request counts, error counts and duration histograms per service and route, can be generated from traces at ingest and kept as metrics for months, while the individual spans expire after a few weeks. The same works for logs: count error events per service as a metric, and you keep the trend even after the raw lines are gone.
Best practices
- Define retention per signal and data class, documented and approved by engineering, security and the data protection officer.
- Mask or drop personal data before storage, so long retention does not turn into a privacy problem.
- Align retention with your incident process. If postmortems are written two weeks after an incident, logs must survive at least that long.
- Review cost and usage regularly. Check which data is actually queried and shorten retention or volume where nobody looks.
- Test deletion. A retention policy is only real if data is in fact deleted on schedule.
Common mistakes
- Keeping everything forever "just in case", then discovering that storage dominates the observability bill.
- Choosing 12 months for metrics, which makes exact year-over-year comparisons impossible on the last day of the window.
- Applying the same long retention to verbose debug logs and to compact metrics.
How Obseria helps
Obseria retains logs, spans and RUM events for 30 days and metrics and synthetic results for 13 months, enough for year-over-year comparisons. Storage is billed separately from ingestion, and data you filter out in Obseria Pipeline costs only ingestion, so you control stored volume at the source. Pipeline also masks PII at ingest, and data residency in the EU or UK supports GDPR requirements. Details are on the pricing page and in Log Analytics; for cost tactics see reducing observability costs pipeline-first.
