A service mesh is an infrastructure layer that manages communication between services through a network of proxies, typically sidecars or node-level proxies controlled by a central control plane. It provides mutual TLS, traffic management such as retries and traffic splitting, and uniform telemetry for every request, without changes to application code.
What is a service mesh?
A service mesh moves cross-cutting network concerns out of application code and into the platform. Instead of every team implementing retries, timeouts, encryption and metrics in their own language and framework, a mesh routes all service-to-service traffic through proxies that apply these policies consistently. Well-known implementations include Istio, Linkerd, Consul and Kuma, most of them running on Kubernetes.
Every mesh has two parts. The data plane is made of proxies that actually carry traffic; Istio uses Envoy, Linkerd uses its own lightweight Rust proxy. The control plane distributes configuration, certificates and routing rules to those proxies.
Sidecar and sidecarless architectures
| Model | How it works | Trade-offs |
|---|---|---|
| Sidecar | A proxy container is injected into every pod and intercepts its inbound and outbound traffic | Fine-grained Layer 7 control per workload; extra CPU and memory per pod; restarts needed to upgrade proxies |
| Ambient / node proxy | A per-node component handles Layer 4 and mTLS; optional Layer 7 proxies run only where needed | Lower resource overhead and simpler upgrades; Layer 7 features require extra configuration |
| eBPF-assisted | Kernel programs handle parts of networking and visibility, sometimes combined with proxies | Low overhead; Layer 7 visibility still depends on a proxy or protocol parsing |
Istio's ambient mode, generally available since Istio 1.24, splits the mesh into a node-level zero-trust tunnel and optional waypoint proxies. Cilium uses eBPF for its mesh features. The choice affects not only cost but also which telemetry you get by default.
What a mesh provides
- Security: mutual TLS between services with automatic certificate rotation, plus identity-based authorization policies.
- Traffic management: retries, timeouts, circuit breaking, fault injection and weighted routing, the basis of canary deployments and blue-green switches.
- Uniform telemetry: request counts, error codes and latency histograms for every hop, independent of the language a service is written in.
Service mesh observability: what you get and what you don't
Because every request passes a proxy, a mesh produces consistent golden signals for every service pair. Istio, for example, exposes metrics such as istio_requests_total and istio_request_duration_milliseconds with labels for source, destination and response code. That makes a mesh an excellent source for service dependency maps and for alerting on error ratios between specific services.
A mesh does not give you complete distributed traces on its own. The proxies can create spans for each hop, but they cannot know that an outbound call belongs to an inbound request inside the application. The application must forward trace headers such as W3C traceparent, which is called context propagation. Without it you get disconnected single-hop spans. A mesh also sees nothing inside the process: slow SQL, lock contention or a hot loop remain invisible without application instrumentation or profiling.
Client-side and server-side views
Mesh proxies report each request from both ends: the source proxy measures what the caller experienced, the destination proxy what the callee served. Comparing the two is useful. If the client side shows high latency but the server side does not, the time is spent in the network, in connection setup or in queueing before the request reaches the workload. If both rise together, look inside the destination service, where application traces and profiles take over.
Example: a weighted canary route
This illustrative Istio configuration sends 10 percent of traffic to version 2 of a service:
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: checkout
spec:
hosts: [checkout]
http:
- route:
- destination: {host: checkout, subset: v1}
weight: 90
- destination: {host: checkout, subset: v2}
weight: 10Combined with mesh metrics split by destination version, you can compare error ratio and p99 latency of v1 and v2 side by side and promote or roll back based on data rather than gut feeling. The subsets themselves are defined in a matching DestinationRule.
Best practices and common pitfalls
- Adopt a mesh for a concrete need, such as mTLS everywhere or traffic splitting, not only for metrics, since the operational cost is real.
- Combine mesh telemetry with application instrumentation via OpenTelemetry, so traces are end to end and include in-process work.
- Configure retries carefully. Retries at several layers multiply load during an outage and can turn a partial failure into a full one.
- Watch proxy overhead. Track CPU, memory and added latency of the proxies themselves, especially with sidecars at high request rates.
- Control metric cardinality. Per-pod and per-path labels on mesh metrics can create millions of series; aggregate where you can.
- Keep sampling consistent between proxies and applications so mesh spans and application spans of the same trace are kept together.
How Obseria helps
Obseria ingests mesh metrics via Prometheus scraping or OTLP and correlates them with application traces and logs from OpenTelemetry and the Obseria Agent. Obseria TopoGraph combines mesh traffic with infrastructure topology so you see which pods, nodes and services are involved in a failing call path. For Kubernetes specifics see Kubernetes monitoring, and for end-to-end request views distributed tracing.
