Security logging best practices: what to log, how to protect audit logs and how long to keep them

Security logging best practices start with a simple question that many teams never answer explicitly: which questions must your logs be able to answer after an incident? Who logged in from where, who granted themselves admin rights, which records an account read before it was disabled, which configuration change opened a firewall port. If your logs cannot answer these questions reliably, completely and with timestamps you can trust, they are operational noise rather than a security control.

This guide is a practical, vendor-neutral reference for engineering and security teams. It covers which events to log, what an audit log entry must contain, why structured logging matters, how to protect logs, how to handle personal data under the GDPR and how to derive a log retention period you can defend in an audit. It ends with a table you can adapt and a copyable security logging checklist.

Which events to log

Security logging is not "log everything at debug level". It is a deliberate selection of events that matter for detection, investigation and accountability. Six categories cover the vast majority of needs:

• Authentication: successful and failed logins, logouts, MFA challenges and failures, password resets, token issuance and refresh, session revocation, account lockouts.
• Authorization changes: role and group assignments, permission grants and revocations, new API keys and service accounts, changes to access policies.
• Administrative actions: user creation and deletion, impersonation or "login as" features, break-glass access, changes made through admin consoles or privileged CLIs.
• Data access: reads and exports of sensitive records, bulk queries, downloads, access to personal data or secrets; at least for high-value data sets.
• Configuration changes: infrastructure and application configuration, feature flags with security impact, firewall and network rules, changes to logging itself (disabling an audit log is an event in its own right).
• Security-relevant errors: input validation failures, rejected tokens, signature or certificate errors, rate-limit hits, access-denied responses, unexpected exceptions in authentication code.

Two rules help. First, log failures as carefully as successes; a burst of failed logins is often the earliest signal of an attack. Second, separate security events from application debug output. Use a dedicated event category or stream and a consistent log level policy, so that turning debug logging off in production never silently removes audit events.

What an audit log entry must contain

An audit log entry should let a person who was not involved reconstruct what happened. That requires the same core fields in every event, regardless of which service emits it:

• Who: the actor, as a stable user or service identifier (not only a display name), plus the authentication method and, where relevant, the impersonating or delegating identity.
• What: the action as a controlled vocabulary (for example user.role.granted), the target resource type and ID, and for changes the old and new value or a reference to them.
• When: a UTC timestamp with millisecond precision from a synchronized clock. Without time synchronization, correlating events across systems becomes guesswork.
• Where: source IP address or network zone, service name, host or pod, environment and region.
• Outcome: success, failure or denied, plus a reason code for failures.
• Correlation: a request ID or trace ID that links the audit event to the traces and application logs of the same request.

The correlation field is the one most often missing, and the one that saves the most time in an investigation. With OpenTelemetry, the active trace context can be attached to every log record automatically, so an analyst can jump from "admin disabled MFA for user X" to the exact request, the downstream calls and any errors around it.

Structured logging for security events

Free-text log lines are hard to query and easy to get wrong. Structured logging means emitting events as key-value records with a stable schema, so that "all failed admin logins from outside the EU in the last 30 days" is a query, not a regular-expression project. A minimal audit event could look like this (example values):

{
  "timestamp": "2026-10-21T08:14:03.512Z",
  "event.category": "audit",
  "event.action": "user.role.granted",
  "event.outcome": "success",
  "actor.id": "usr_4821",
  "actor.auth_method": "sso+mfa",
  "target.type": "user",
  "target.id": "usr_9907",
  "change.role.new": "billing-admin",
  "source.ip": "203.0.113.24",
  "service.name": "admin-api",
  "deployment.environment": "production",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Agree on the field names once and enforce them in a shared library or in the telemetry pipeline. Where OpenTelemetry semantic conventions define an attribute, use it; for audit-specific fields, document your own names. Version the schema.

Protecting logs: integrity, access and separation

Attackers who gain privileged access often try to delete or alter logs. An audit log that the compromised system itself can rewrite is weak evidence. Four measures make logs trustworthy:

• Ship logs off the host quickly. Forward events to a central store in near real time instead of keeping them only on the machine that produced them. Log aggregation is a security control, not only a convenience.
• Separate storage and permissions. Store audit logs in a separate account, project or tenant. People who administer production systems should not be able to delete their own audit trail.
• Make tampering evident. Use append-only or write-once storage, object locks, hash chains or signed batches, so that modifications and gaps can be detected.
• Control and log access to logs. Read access to audit logs is itself sensitive and should be role-based and logged. Encrypt logs in transit and at rest.

Finally, monitor the logging pipeline itself. A source that suddenly stops sending events is either broken or deliberately silenced; both deserve an alert.

Personal data in logs and the GDPR

Security logs almost always contain personal data: user IDs, IP addresses, email addresses, sometimes request payloads. The GDPR does not forbid this, but its principles apply in full. Article 5(1) requires among other things that data is collected for specified purposes (purpose limitation), limited to what is necessary (data minimisation) and kept in identifiable form no longer than necessary (storage limitation). Article 32 lists pseudonymisation and encryption among the possible security measures.

In practice this translates into a handful of engineering rules:

• Never log secrets or content you do not need: passwords, tokens, session cookies, full card numbers, health data or free-text message bodies do not belong in logs.
• Mask or pseudonymise at ingest: replace emails and names with stable pseudonymous IDs, truncate or hash IP addresses where the full value is not needed, and redact known sensitive fields before the data reaches long-term storage.
• Document the purpose: state in your records of processing why each log category exists (for example security incident detection and investigation) and do not reuse security logs for unrelated purposes such as performance reviews of employees.
• Delete on schedule: enforce retention automatically per category, and make sure backups and exports follow the same rules.

Involve your data protection officer early; in many organisations the works council also has a say in systems that can monitor employees.

Log retention: deriving the period instead of guessing it

The most common question about audit logs is "how long do we have to keep them?" The honest answer: there is no single legal rule that says "keep security logs for X days". You derive the retention period per log category from three inputs and document the reasoning.

1. Purpose

Ask how long the log is actually needed for its purpose. For incident investigation, a useful reference point is how long attacks typically stay undetected in your environment and how far back an investigation would need to look. For troubleshooting, a few days to weeks is often enough. Under the GDPR storage limitation principle, the purpose sets the upper bound for identifiable data.

2. Legal obligations

Some laws require specific records to be kept. In Germany, for example, § 257 HGB and § 147 AO set retention periods for business records: ten years for books and annual financial statements, eight years for accounting vouchers and six years for commercial and business letters. These obligations apply to business records, such as an invoice stored in your billing system, not to technical logs as such. A web server access log is generally not a business record and does not inherit a ten-year period just because the company is subject to the HGB. Sector rules (finance, health, telecommunications) can add specific requirements, so check what applies to you.

3. Frameworks and regulation

• ISO/IEC 27001:2022 contains the Annex A controls 8.15 Logging and 8.16 Monitoring activities. Control 8.15 covers producing, storing, protecting and analysing logs of activities, exceptions, faults and other relevant events; 8.16 covers monitoring for anomalous behaviour. The standard does not prescribe a fixed retention period; you define one in your ISMS based on risk.
• NIS2 (Directive (EU) 2022/2555) requires appropriate and proportionate risk-management measures in Article 21, including incident handling. For certain digital service providers, Implementing Regulation (EU) 2024/2690 adds that logs must be kept and backed up for a predefined period and protected against unauthorised access or changes, again without a fixed number. Our NIS2 monitoring guide covers the details.
• BSI IT-Grundschutz module OPS.1.1.5 Protokollierung requires a logging policy that defines what, how and where to log, time synchronisation of logging systems and, for larger environments, a central logging infrastructure. Check the current edition of the compendium for the exact requirements.

Combining these inputs typically leads to tiers: a short hot period with full detail for operations, a longer period for security-relevant events, and long-term retention only for the narrow set of records with a legal basis, ideally pseudonymised.

Reference table: event types, fields and retention rationale

Use this table as a starting point for your logging policy. The retention column shows the reasoning to document, not a legal requirement.

Event type	Fields to log	Sensitivity	Typical retention rationale
Authentication (login, MFA, logout, lockout)	actor ID, auth method, outcome, reason, source IP, user agent, trace ID	Medium to high (IP, user ID)	Long enough to investigate account takeover across your expected detection window
Authorization changes (roles, grants, API keys)	actor, target identity, old and new permission, approver, outcome	High	Life of the access plus investigation window; evidence for access reviews and audits
Admin actions and break-glass access	actor, impersonated identity, action, target, justification or ticket ID	High	Accountability for privileged use; aligned with audit cycles of your ISMS
Data access and exports	actor, data set, record count, query or filter reference, outcome	High (may reveal personal data)	Breach scoping: which data was accessed, needed to assess notification duties
Configuration changes	actor, component, diff or change reference, deployment ID, outcome	Medium	Root-cause analysis and change management evidence
Security-relevant errors (denied, invalid token, rate limit)	source, endpoint, error code, actor if known, trace ID	Medium	Detection of attack patterns; shorter retention often sufficient after aggregation
Application debug and access logs	request metadata, latency, status code, trace ID	Low to medium	Troubleshooting only; keep short and minimise personal data

Copyable security logging checklist

Copy this checklist into your wiki or ticket system and review it with engineering, security and your data protection officer:

SECURITY LOGGING CHECKLIST
[ ] Logging policy exists: what, how, where, owner, review date
[ ] Event catalogue covers authentication, authorization changes,
    admin actions, data access, configuration changes, security errors
[ ] Failures and denials are logged, not only successes
[ ] Every audit event has: who, what, when (UTC), where, outcome, trace ID
[ ] Structured format with a documented, versioned schema
[ ] All systems use synchronized clocks (NTP)
[ ] Events leave the source host in near real time
[ ] Audit logs stored separately; admins cannot delete their own trail
[ ] Append-only / write-once storage or hash chains for tamper evidence
[ ] Encryption in transit and at rest; role-based read access, logged
[ ] Alert when a log source stops sending events
[ ] No secrets, passwords, tokens or unnecessary content in logs
[ ] PII masked or pseudonymised before long-term storage
[ ] Purpose documented per log category (records of processing)
[ ] Retention defined per category with documented rationale
[ ] Automatic deletion, including backups and exports
[ ] Detection rules and dashboards tested against sample events
[ ] Policy reviewed at least annually and after major incidents

How Obseria helps with security logging

Obseria is an OpenTelemetry-native observability platform with EU data residency. It does not replace your ISMS or your legal assessment, but it provides the technical building blocks for the practices above: structured logs correlated with traces, processing before storage and retention you can configure per data stream. See Log Analytics and the security and reliability use case for details.

• Obseria Pipeline filters, masks PII and routes events before they reach storage, so data minimisation happens at ingest.
• Obseria Lake stores logs, metrics, traces and events without indexes, which makes longer retention for security-relevant streams affordable.
• Trace correlation links audit events to the requests, services and errors around them.
• Anomaly detection and alerting flag unusual patterns, such as spikes in failed logins or silent log sources.

Want to see how your audit events look as structured, correlated data? Start a free trial of Obseria and connect your first OpenTelemetry source in minutes.