Continuous profiling for code-level performance in production

What is continuous profiling?

Continuous profiling is the practice of collecting performance profiles from applications running in production, all the time, instead of attaching a profiler only when something goes wrong. A profile is a statistical sample of stack traces: at regular intervals the profiler records which function is executing, how much memory is allocated or which thread is waiting. Aggregated over time, these samples show precisely where an application spends its resources.

Traditional profilers are heavy and used locally or in staging. Continuous profilers are designed for production: they rely on low-frequency sampling, efficient stack unwinding and compact storage, so they can run on every instance with minimal impact.

Why continuous profiling matters

Metrics tell you that CPU is at 80 percent; traces tell you that a span took 900 milliseconds. Neither tells you which line of code is responsible. Continuous profiling closes that gap and adds a fourth signal next to metrics, logs and traces.

• Faster root cause: when latency regresses, you see the function that changed instead of bisecting commits.
• Lower infrastructure cost: small inefficiencies in hot paths are multiplied by every request and every replica. Profiles make them visible.
• No reproduction needed: intermittent issues, such as a slow code path that only triggers for some tenants, are captured when they happen.
• Better capacity planning: knowing what consumes CPU and memory helps you size workloads and set realistic limits in Kubernetes.

Profile types and what they reveal

Profile type	What it shows	Typical finding
CPU	Time spent executing on CPU per function	Expensive JSON serialization or regex in a hot loop
Wall clock	Total elapsed time including waiting	Blocking I/O or synchronous downstream calls
Allocations	Where memory is allocated and how much	Excessive short-lived objects causing GC pressure
Heap	Memory currently in use	Leaks and unbounded caches
Locks and contention	Time threads wait on mutexes	A shared lock that serializes concurrent requests

Which types are available depends on the runtime. The flame graph is the standard visualization: width represents the share of samples, and the call stack grows from the bottom up.

How Obseria does continuous profiling

The Obseria Agent collects profiles alongside metrics, logs, traces and security signals, so there is no extra profiler to roll out. Profiling runs with under 1% overhead. Profiles are stored in Obseria Lake and labeled with service, version, host, container and Kubernetes metadata, which makes it possible to filter and compare across any dimension.

The key difference is correlation. Because Obseria also holds your distributed traces, you can open a slow span and see the profile samples captured on the same instance at the same time. Obseria Intelligence uses profiles as evidence in root-cause investigations, for example to show that a latency regression after a release is caused by a new code path in a specific function. Differential flame graphs compare two versions side by side, so you can confirm that an optimization actually saved CPU before rolling it out everywhere.

Profiles also feed cost work. By aggregating CPU samples across all replicas of a service, you can estimate which functions account for the largest share of compute time and focus optimization effort where it reduces infrastructure spend the most, rather than on code that only looks slow in isolation.

Best practices for continuous profiling

1. Profile everywhere, not just in staging. Production traffic patterns and data volumes rarely match test environments.
2. Tag profiles with version information so every deployment can be compared with its predecessor.
3. Start with the widest frames. In a flame graph, optimize the functions that take the largest share of samples first.
4. Combine CPU and allocation profiles. High CPU in the garbage collector is usually an allocation problem.
5. Review profiles regularly, not only during incidents. A monthly look at top consumers often yields significant cost savings.
6. Link profiling to SLOs. Prioritize optimizations on endpoints that threaten latency objectives.