06 - Scheduling: Affinity, Taints, Resources, PDBs
Why this matters
This is how you control where pods run and what happens when nodes are under pressure or being drained. Critical for prod stability and multi-tenant clusters.
Read this first — Definitions & Explanations
Scheduling
The process of choosing which node a Pod should run on. Done by kube-scheduler (usually).
Requests vs Limits
- requests: what the scheduler uses to place the Pod (“needs 250m CPU”)
- limits: hard cap at runtime (exceed CPU → throttled; exceed memory → often OOMKilled)
nodeSelector
Simple “must have this label” rule for nodes.
Affinity / anti-affinity
Richer rules: prefer/require certain nodes or relative placement to other Pods (spread replicas across zones/nodes).
Taints and Tolerations
- Taint on a node: “don’t schedule here unless you tolerate me”
- Toleration on a Pod: “I accept that taint”
Used to reserve nodes (GPU, control-plane, dedicated teams).
PodDisruptionBudget (PDB)
Limits how many Pods in a set can be voluntarily disrupted at once (drains, upgrades). Protects availability during maintenance.
Priority and preemption
Higher-priority Pods can preempt lower-priority ones when the cluster is full (advanced scheduling behavior).
Official docs (read for detail)
- Assigning Pods to Nodes
- Taints and Tolerations
- Resource Management for Pods and Containers
- Pod Disruption Budgets
- Scheduling Framework
Key Concepts
- Resource
requestsvslimits(CPU is compressible, memory is not — OOMKill risk) - QoS classes: Guaranteed, Burstable, BestEffort (drives eviction order)
- nodeSelector (simple) vs nodeAffinity (expressive, required/preferred)
- podAffinity / podAntiAffinity — co-locate or spread pods relative to other pods
- Taints & Tolerations — repel pods from nodes unless they tolerate the taint
- Topology spread constraints — spread pods evenly across zones/nodes
- PriorityClass — which pods get scheduled/evicted first under pressure
- PodDisruptionBudget (PDB) — protects availability during voluntary disruptions (drains, upgrades)
YouTube search terms
- "Kubernetes resource requests and limits explained"
- "Kubernetes taints and tolerations vs node affinity"
- "Kubernetes PodDisruptionBudget explained"
- "Kubernetes QoS classes explained"
Hands-on lab (on prod-sim)
# Taint a worker node, prove pods don't schedule there without toleration
kubectl taint nodes prod-sim-m03 dedicated=gpu:NoSchedule
kubectl create deployment plain --image=nginx --replicas=3
kubectl get pods -o wide # none should land on prod-sim-m03
# Add toleration + nodeAffinity to force it there
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values: ["prod-sim-m03"]
containers:
- name: gpu-pod
image: nginx
EOF
kubectl get pod gpu-pod -o wide # should be on prod-sim-m03
kubectl taint nodes prod-sim-m03 dedicated=gpu:NoSchedule- # cleanup
# Resource requests/limits + OOMKill demo
kubectl run oom --image=polinux/stress --restart=Never \
--requests='memory=50Mi' --limits='memory=100Mi' \
-- stress --vm 1 --vm-bytes 200M --vm-hang 1
kubectl get pod oom -w # watch it go OOMKilled
kubectl describe pod oom | grep -A3 "Last State"
# PodDisruptionBudget — protect a deployment during a drain
kubectl create deployment protected --image=nginx --replicas=3
cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: protected-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: protected
EOF
kubectl drain prod-sim-m02 --ignore-daemonsets --delete-emptydir-data
# watch it respect the PDB (won't evict below minAvailable)
kubectl uncordon prod-sim-m02
Common mistakes and how to avoid them
- Setting requests far above real usage. The scheduler reserves the full request, so nodes look full while sitting idle and you pay for unused capacity. Measure with
kubectl topand right-size. - Setting a very low CPU limit. A CPU limit throttles the container even when the node has spare CPU, which adds latency. Exceeding a memory limit gets the container OOMKilled, so set memory limits deliberately.
- Tainting nodes and forgetting system Pods. DaemonSets such as the CNI agent or log shipper will not run on a tainted node unless they tolerate the taint.
- Using required anti-affinity on a small cluster. Replicas beyond the number of nodes stay Pending. Use preferred anti-affinity or topology spread constraints for soft spreading.
- Having no PodDisruptionBudget. A node drain can evict every replica at once. Add a PDB with
minAvailableso voluntary disruptions keep enough Pods running.
Quick answers
How does the Kubernetes scheduler choose a node?
It filters out nodes that cannot run the Pod, for example for lack of resources, a taint or an affinity rule, then scores the remaining nodes and picks the best one. If no node passes the filter, the Pod stays Pending and an event explains why.
What is the difference between taints and affinity?
A taint on a node repels Pods unless they carry a matching toleration, so the node controls who may run on it. Affinity is set on the Pod and attracts it to, or away from, certain nodes or other Pods. Taints keep Pods out, affinity pulls them in.
What is the difference between resource requests and limits?
A request is the amount the scheduler reserves for the container and uses to pick a node. A limit is the maximum the container may use: exceeding the memory limit gets it killed, and exceeding the CPU limit throttles it.
Notes
(fill in your own words after watching + labbing)