Kubernetes Service Mesh with Istio: A Practical Overview

What and Why

Kubernetes gives you Services, DNS, and rolling updates. It does not give you mTLS between every pod, fine-grained traffic shifting, automatic retries with jitter, or a single dashboard of golden signals across every microservice. A service mesh layers those features on top of your cluster without forcing application code changes.

Istio is the most feature-rich open-source mesh. It injects an Envoy sidecar next to every pod, and a control plane (istiod) configures those sidecars in real time. The app keeps making plain HTTP calls to its neighbors; Envoy transparently upgrades the connection to mTLS, applies routing rules, and emits telemetry.

Mental Model

Picture every pod gaining a tiny reverse proxy that owns its network interface. The proxy is identical everywhere and reads its config from istiod. The control plane translates high-level CRDs (VirtualService, DestinationRule, PeerAuthentication) into Envoy configuration. Your app is unaware any of this is happening.

That uniformity is the point. Every retry, every certificate, every metric comes from the same code path, so a platform team can ship cross-cutting changes without touching application repos.

Hands-on Example

Assume Istio is installed and the shop namespace is labeled istio-injection=enabled. Deploy two versions of a reviews service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  namespace: shop
spec:
  replicas: 2
  selector:
    matchLabels: { app: reviews, version: v1 }
  template:
    metadata:
      labels: { app: reviews, version: v1 }
    spec:
      containers:
        - name: reviews
          image: example/reviews:1.0
          ports: [{ containerPort: 9080 }]
---
apiVersion: v1
kind: Service
metadata:
  name: reviews
  namespace: shop
spec:
  selector: { app: reviews }
  ports: [{ port: 9080, name: http }]

Now shift 10 percent of traffic to v2 without changing the Service:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews
  namespace: shop
spec:
  host: reviews
  subsets:
    - name: v1
      labels: { version: v1 }
    - name: v2
      labels: { version: v2 }
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
  namespace: shop
spec:
  hosts: [reviews]
  http:
    - route:
        - destination: { host: reviews, subset: v1 }
          weight: 90
        - destination: { host: reviews, subset: v2 }
          weight: 10

[productpage pod]                 [istiod]
 |  app -> envoy (localhost)        ^
 |                                  | xDS config
 v                                  |
envoy --mTLS--> envoy --90%--> [reviews v1 pod]
                     --10%--> [reviews v2 pod]

Common Pitfalls

Sidecar injection only happens at pod creation. If you label a namespace after pods are running, you must restart the pods. Jobs and CronJobs are tricky because the sidecar never exits on its own; use holdApplicationUntilProxyStarts and EXIT_ON_ZERO_ACTIVE_CONNECTIONS or run jobs in a namespace without injection.

Strict mTLS turned on cluster-wide will break any client that is not in the mesh, including kube-apiserver health checks reaching pods directly and Prometheus scrapes from outside the mesh. Roll out PeerAuthentication in PERMISSIVE first, watch telemetry, then move to STRICT.

CRDs are not Kubernetes-native, so kubectl describe pod will not show you why a request was rejected. Get used to reading Envoy access logs and istioctl analyze.

Production Tips

Run istiod highly available with at least three replicas and a PodDisruptionBudget. Pin the Istio version in your install manifest and upgrade with the canary control-plane revision approach so you can roll back without reinjecting sidecars.

Budget for the resource overhead: each Envoy sidecar typically uses 50 to 200 MB of memory and adds about 1 ms of latency per hop. Multiply that by your pod count before committing.

Use Sidecar resources to scope which services each proxy knows about. Without it, every Envoy holds the full cluster service registry, and memory grows linearly with the number of services.

Wrap-up

Istio buys you mTLS, traffic shaping, and unified telemetry, but it adds a second control plane to operate. If you have fewer than ten services, a few well-placed libraries are cheaper. If you have hundreds of services across many teams, a mesh starts paying for itself the day you need to roll out a security policy everywhere at once.