10 - Control Plane Deep Dive
Why this matters
Topic 01 gave you the map. Now you actually open each component up. This is where "I use Kubernetes" turns into "I understand Kubernetes."
Read this first — Definitions & Explanations
etcd deep dive
Uses Raft consensus. Production etcd clusters usually have 3 or 5 members for quorum. Losing quorum means the control plane can’t reliably write state.
API server internals (mental model)
Request path roughly: authn → authz → mutation admission → object validation → etcd → validation admission / webhooks → response. Watches let clients stream changes efficiently.
Admission controllers
Plugins/webhooks that can mutate or validate API requests before persistence (enforce labels, block privileged Pods, inject sidecars).
Scheduler phases
- Filtering: eliminate impossible nodes
- Scoring: rank remaining nodes
Then bind Pod to the winner.
Static Pods
Pod manifests read directly by kubelet from a local folder (often /etc/kubernetes/manifests). Control-plane components on kubeadm clusters commonly run this way. They appear in the API but aren’t managed by Deployments.
kube-proxy modes
- iptables: simple, widely used
- IPVS: better performance/scale for huge numbers of Services
Official docs (read for detail)
- Control Plane Components
- Operating etcd clusters for Kubernetes
- Admission Controllers
- kube-scheduler
- Static Pods
Key Concepts
- etcd: Raft consensus, quorum, why you need an odd number of members (3/5), watch mechanism
- kube-apiserver: admission controllers (mutating/validating), API aggregation layer, watch/list mechanics, resource versioning
- kube-controller-manager: built-in controllers (node, replication, endpoint, service account, etc.) — one binary, many loops
- kube-scheduler: filtering (predicates) then scoring (priorities) phases, custom schedulers, scheduler extenders
- kubelet: PLEG (Pod Lifecycle Event Generator), cgroup driver (systemd vs cgroupfs — must match containerd config), static pods
- kube-proxy: iptables vs IPVS mode tradeoffs
YouTube search terms
- "etcd Raft consensus explained"
- "Kubernetes admission controllers explained mutating validating"
- "Kubernetes scheduler filtering scoring explained"
- "kubelet static pods explained"
- "kube-proxy iptables vs IPVS"
Hands-on lab (on prod-sim)
# etcd: talk to it directly
minikube ssh -p prod-sim
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/minikube/certs/etcd/ca.crt \
--cert=/var/lib/minikube/certs/etcd/server.crt \
--key=/var/lib/minikube/certs/etcd/server.key \
member list -w table
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/minikube/certs/etcd/ca.crt \
--cert=/var/lib/minikube/certs/etcd/server.crt \
--key=/var/lib/minikube/certs/etcd/server.key \
get /registry/pods/default --prefix --keys-only
exit
# Static pods: how the control plane bootstraps itself
minikube ssh -p prod-sim
ls /etc/kubernetes/manifests/ # kubelet watches this dir directly, no API server needed
sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -A2 enable-admission-plugins
exit
# Admission controllers in action: try to create a pod violating a policy
kubectl run priv --image=nginx --overrides='{"spec":{"containers":[{"name":"priv","image":"nginx","securityContext":{"privileged":true}}]}}'
kubectl get pod priv # likely allowed here (no PSA restricted policy yet) — revisit in topic 17
# Scheduler: force it to explain its decision
kubectl get events --field-selector reason=Scheduled
kubectl get events --field-selector reason=FailedScheduling
Common mistakes and how to avoid them
- Ignoring etcd disk latency. Slow disks cause slow API responses and unstable leader elections. Run etcd on fast SSD storage.
- Running an even number of etcd members. Quorum needs a majority, so four members tolerate no more failures than three. Use three or five.
- Editing static Pod manifests carelessly. The kubelet restarts static Pods from
/etc/kubernetes/manifestswhen the file changes, so a typo can take the API server down. Back up the file before editing. - Assuming admission controllers only validate. Mutating webhooks can change objects, and a failing webhook can block creates. Set
failurePolicyand exclusions deliberately. - Forgetting certificate expiry. kubeadm certificates expire after one year by default. Renew with
kubeadm certs renewand track the dates.
Quick answers
What is etcd and why does it matter?
etcd is a consistent key-value store that holds all cluster state, including every object definition. The API server is its only client. If etcd is lost or corrupt, the cluster loses its state, which is why regular snapshots and an odd number of members matter.
What are admission controllers?
They intercept requests to the API server after authentication and authorization but before the object is stored. Mutating controllers can change the object, such as injecting a sidecar, and validating controllers can reject it, such as for a policy violation.
What are static Pods?
Static Pods are managed directly by the kubelet from manifest files on a node, usually in /etc/kubernetes/manifests, not through the API server. kubeadm runs the API server, scheduler, controller manager and etcd this way, and the kubelet restarts them if they fail.
Notes
(fill in your own words after watching + labbing)