23 - Service Mesh & Multi-Cluster

Why this matters

At scale, you need mTLS between every service, fine-grained traffic control (canary, retries, circuit breaking), and often more than one cluster (region, blast-radius, compliance). This is the "senior/staff SRE" layer.

Read this first — Definitions & Explanations

Service mesh

Infrastructure layer for service-to-service communication: mTLS, retries, timeouts, traffic splitting, observability (Istio, Linkerd, Consul connect, etc.).

Sidecar model

Classic meshes inject a proxy container next to each app container. Traffic is intercepted and policy is applied there. (Newer modes may avoid sidecars.)

Traffic splitting / canaries

Send 5% of traffic to v2, watch metrics, then ramp up — safer releases.

Multi-cluster

Running apps across more than one cluster for isolation, DR, or locality. Needs careful networking/identity design.

Tradeoff

Meshes add power and operational complexity. Adopt when the problems are real, not for fashion.

Official docs (read for detail)

• Service Mesh overview (concepts)
• Istio Documentation
• Cluster API
• Multi-cluster Services API

Key Concepts

• Service mesh = sidecar proxy (Envoy, usually) injected into every pod, handles mTLS, retries, timeouts, circuit breaking, observability — without app code changes
• Istio: full-featured, complex, VirtualService/DestinationRule/Gateway CRDs
• Linkerd: simpler, lighter, less config surface — good contrast to learn against Istio
• mTLS everywhere: automatic service-to-service encryption + identity
• Traffic splitting for canary releases (weight-based routing)
• Multi-cluster patterns: cluster federation, active-active vs active-passive, Cluster API (CAPI) for declarative cluster lifecycle management across providers

YouTube search terms

• "Istio service mesh explained tutorial"
• "Linkerd vs Istio comparison"
• "Kubernetes mTLS service mesh explained"
• "Cluster API CAPI explained multi-cluster Kubernetes"

Hands-on lab (on prod-sim + a second local cluster)

# Install Istio
curl -L https://istio.io/downloadIstio | sh -
istioctl install --set profile=demo -y
kubectl label namespace default istio-injection=enabled

# Deploy the standard bookinfo sample app to see the mesh in action
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl get pods   # note EVERY pod now has 2 containers (app + istio-proxy sidecar)

# Canary traffic split: 90% v1, 10% v3 of reviews service
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: ["reviews"]
  http:
  - route:
    - destination: {host: reviews, subset: v1}
      weight: 90
    - destination: {host: reviews, subset: v3}
      weight: 10
EOF

# Prove mTLS is active
istioctl proxy-config secret <any-pod> -o json | grep -i "trust-domain"
kubectl exec deploy/productpage-v1 -c istio-proxy -- openssl s_client -connect reviews:9080 2>&1 | grep -i "verify"

# Multi-cluster: spin up a second minikube cluster and try Cluster API basics
minikube start -p cluster2 --nodes 2 --driver docker
clusterctl init --infrastructure docker   # CAPD - Cluster API Docker provider, good for local learning
clusterctl generate cluster my-capi-cluster --infrastructure docker --kubernetes-version v1.31.0 --control-plane-machine-count 1 --worker-machine-count 2 > capi-cluster.yaml
kubectl apply -f capi-cluster.yaml
kubectl get clusters,machines   # watch CAPI provision an entirely new cluster declaratively

Notes

(fill in your own words after watching + labbing)