23 - Service Mesh & Multi-Cluster
Why this matters
At scale, you need mTLS between every service, fine-grained traffic control (canary, retries, circuit breaking), and often more than one cluster (region, blast-radius, compliance). This is the "senior/staff SRE" layer.
Read this first — Definitions & Explanations
Service mesh
Infrastructure layer for service-to-service communication: mTLS, retries, timeouts, traffic splitting, observability (Istio, Linkerd, Consul connect, etc.).
Sidecar model
Classic meshes inject a proxy container next to each app container. Traffic is intercepted and policy is applied there. (Newer modes may avoid sidecars.)
Traffic splitting / canaries
Send 5% of traffic to v2, watch metrics, then ramp up — safer releases.
Multi-cluster
Running apps across more than one cluster for isolation, DR, or locality. Needs careful networking/identity design.
Tradeoff
Meshes add power and operational complexity. Adopt when the problems are real, not for fashion.
Official docs (read for detail)
Key Concepts
- Service mesh = sidecar proxy (Envoy, usually) injected into every pod, handles mTLS, retries, timeouts, circuit breaking, observability — without app code changes
- Istio: full-featured, complex, VirtualService/DestinationRule/Gateway CRDs
- Linkerd: simpler, lighter, less config surface — good contrast to learn against Istio
- mTLS everywhere: automatic service-to-service encryption + identity
- Traffic splitting for canary releases (weight-based routing)
- Multi-cluster patterns: cluster federation, active-active vs active-passive, Cluster API (CAPI) for declarative cluster lifecycle management across providers
YouTube search terms
- "Istio service mesh explained tutorial"
- "Linkerd vs Istio comparison"
- "Kubernetes mTLS service mesh explained"
- "Cluster API CAPI explained multi-cluster Kubernetes"
Hands-on lab (on prod-sim + a second local cluster)
# Install Istio
curl -L https://istio.io/downloadIstio | sh -
istioctl install --set profile=demo -y
kubectl label namespace default istio-injection=enabled
# Deploy the standard bookinfo sample app to see the mesh in action
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl get pods # note EVERY pod now has 2 containers (app + istio-proxy sidecar)
# Canary traffic split: 90% v1, 10% v3 of reviews service
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: reviews
spec:
hosts: ["reviews"]
http:
- route:
- destination: {host: reviews, subset: v1}
weight: 90
- destination: {host: reviews, subset: v3}
weight: 10
EOF
# Prove mTLS is active
istioctl proxy-config secret <any-pod> -o json | grep -i "trust-domain"
kubectl exec deploy/productpage-v1 -c istio-proxy -- openssl s_client -connect reviews:9080 2>&1 | grep -i "verify"
# Multi-cluster: spin up a second minikube cluster and try Cluster API basics
minikube start -p cluster2 --nodes 2 --driver docker
clusterctl init --infrastructure docker # CAPD - Cluster API Docker provider, good for local learning
clusterctl generate cluster my-capi-cluster --infrastructure docker --kubernetes-version v1.31.0 --control-plane-machine-count 1 --worker-machine-count 2 > capi-cluster.yaml
kubectl apply -f capi-cluster.yaml
kubectl get clusters,machines # watch CAPI provision an entirely new cluster declaratively
Common mistakes and how to avoid them
- Adding a mesh when you only need TLS or ingress. A mesh adds a proxy to every Pod and a control plane to run. Use it when you need its features, not by default.
- Ignoring sidecar resource use. Each sidecar consumes CPU and memory, which adds up across hundreds of Pods. Include it in capacity planning.
- Enabling strict mTLS before every workload is meshed. Unmeshed clients will fail to connect. Start in permissive mode and tighten once everything has a sidecar.
- Splitting canary traffic by replica count. Replica ratios are coarse and shift with scaling. Use weighted routing rules for exact percentages.
- Treating multi-cluster as just more nodes. You also need cross-cluster networking, shared identity and service discovery. Plan these before adding clusters.
Quick answers
What is a service mesh?
A service mesh is an infrastructure layer that handles service-to-service communication, such as mTLS, retries, timeouts and traffic splitting, through proxies placed next to your workloads. Istio and Linkerd are common examples. It moves networking logic out of application code.
How does a sidecar proxy work in a mesh?
A proxy container, such as Envoy, runs in every Pod and intercepts the Pod's inbound and outbound traffic. A control plane configures those proxies with routing, security and telemetry rules, so policy applies without changing the application.
How do canary releases work with a mesh?
The mesh routes a set percentage of requests to the new version, for example 5%, using weighted routing rules instead of relying on replica counts. You raise the weight as metrics stay healthy and roll back by setting it to zero.
Notes
(fill in your own words after watching + labbing)