Why (and why not) a mesh · Prasanta Kumar Behera

What a service mesh actually provides (identity-based mTLS, traffic control, uniform telemetry), what it costs (resources, latency, operational complexity, new failure modes), the alternatives, and a checklist for deciding whether your platform needs one.

What a service mesh is

A service mesh puts a proxy in the path of service-to-service traffic, configured by a central control plane:

             control plane (config, certificates, service discovery)
               │            │            │
pod A [app ⇄ proxy] ═══mTLS═══ [proxy ⇄ app] pod B
                 (data plane: sidecars per pod, or per-node proxies)

The proxies handle mTLS, policy, routing and telemetry, so applications don't have to.

A mesh is like giving every house in town a smart doorbell and a mail sorter. Every letter between houses is sealed in a tamper-proof envelope with the sender's verified name (mTLS identity). The doorbell only opens for approved senders (policy), can send some letters to the new house next door as a trial (canary), and keeps a notebook of every delivery (telemetry). It's great, but every house now depends on the doorbell working.

Benefits

Capability	Without a mesh	With a mesh
Encryption in transit	Each app configures TLS	Automatic mTLS everywhere
Service identity & authz	IP-based NetworkPolicy	Identity-based L7 policies (SPIFFE IDs)
Traffic shifting	Deployment tricks, ingress only	Weighted routing between versions for internal calls
Resilience	Libraries per language	Timeouts, retries, circuit breaking in the proxy
Telemetry	Instrument each app	Golden signals for every service automatically

Costs

Resources: a sidecar in every pod (CPU/memory multiplied by pod count), or per-node proxies in ambient/sidecar-less designs.
Latency: each proxy hop adds a little; usually sub-millisecond to a few milliseconds, but measure it for your traffic.
Complexity: new CRDs, upgrades of the control plane and every proxy, certificate management, and a new layer to debug.
Failure modes: a bad policy or broken control plane can stop traffic cluster-wide; retries can amplify outages (lesson 05).

Alternatives

Need	Lighter option
Encryption only	CNI transparent encryption (WireGuard/IPsec in Cilium or Calico)
L3/L4 segmentation	NetworkPolicies (default deny)
North-south routing, canaries at the edge	Ingress / Gateway API, Argo Rollouts with the ingress
Telemetry	OpenTelemetry auto-instrumentation (see Observability with OpenTelemetry)
Retries/timeouts	Client libraries with sane defaults

Sidecar-less meshes (Istio ambient, Cilium service mesh with per-node proxies) reduce the per-pod cost and are changing this trade-off.

Decision checklist

Adopt a mesh when several of these are true:

[ ] Compliance or security requires mTLS with workload identity and identity-based authorisation between services.
[ ] Many services in several languages, where per-library resilience/telemetry is inconsistent.
[ ] You need fine-grained traffic control for internal calls (canaries, mirroring, fault injection).
[ ] The platform team has capacity to operate it (upgrades, certificates, debugging) as tier-0 infrastructure.

If only one is true, start with the lighter option for that need.

Try it: measure before you decide

Deploy a small two-service app in kind and load-test it (e.g. fortio or k6), recording p50/p99 latency and CPU/memory.
Install Linkerd or Istio (lessons 02–03), mesh the app, and repeat the load test. Record the difference.
Instead, enable CNI encryption in a Cilium lab (WireGuard) and repeat.
Fill in the decision checklist for a real or imagined platform.
Write a one-paragraph recommendation with your measurements.

Going deeper: organisational fit

A mesh is a platform product: onboarding docs, defaults, SLOs, and a team that owns it.
Adopt incrementally: one namespace, permissive mTLS, then strict; telemetry first, then policies (lessons 04 and 07).
Revisit the decision periodically; data-plane options change quickly.

Recap

A mesh gives identity-based mTLS, L7 policy, traffic control and uniform telemetry without app changes.
It costs resources, latency, operational complexity and new failure modes.
Lighter alternatives exist for single needs (CNI encryption, NetworkPolicy, Gateway API, OTel).
Adopt when several needs align and the team can run it as tier-0.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.