13 - Troubleshooting & Debugging
Why this matters
This is the actual job. Anyone can kubectl apply a working manifest — mastery is
diagnosing a broken cluster/app fast, under pressure, with incomplete information.
Read this first — Definitions & Explanations
Debugging order that works
kubectl get/kubectl describe(Events!)kubectl logs/ previous logs- exec only if running
- node/runtime checks if it’s a node problem
CrashLoopBackOff
Container starts, exits, kubelet restarts it with backoff. Causes: bad command, missing config, failing dependency, failing liveness probe.
ImagePullBackOff
kubelet can’t pull the image (typo, private registry auth, network, rate limits).
Pending
Scheduler can’t place the Pod: insufficient CPU/memory, affinity/taints, PVC unbound, etc. Read Events.
describe Events
Often the fastest truth. Don’t skip them.
Useful tools
kubectl get events --sort-by=.lastTimestamp, kubectl top, kubectl get endpoints, node conditions (Ready, MemoryPressure).
Official docs (read for detail)
- Debug Running Pods
- Debug Services
- Determine the Reason for Pod Failure
- Debug Nodes / Troubleshoot Clusters
- Ephemeral Containers
Key Concepts
- The debugging funnel: Pod status → Events → Logs → Exec → Node → Network
kubectl describeEvents section is almost always the first real cluekubectl logs --previousfor crashed containers- Ephemeral debug containers (
kubectl debug) for distroless/minimal images with no shell - Common failure signatures:
CrashLoopBackOff,ImagePullBackOff,Pending(scheduling failure),Evicted,OOMKilled,CreateContainerConfigError - Node-level debugging: kubelet logs,
crictl, disk/memory pressure conditions kubectl get events --sort-by=.lastTimestamp -Aas a cluster-wide first look
YouTube search terms
- "Kubernetes troubleshooting CrashLoopBackOff step by step"
- "kubectl debug ephemeral containers explained"
- "Kubernetes Pending pod troubleshooting scheduling failures"
- "Kubernetes node NotReady troubleshooting"
Hands-on lab (on prod-sim)
# Manufacture and diagnose each classic failure
# 1. ImagePullBackOff
kubectl run bad-image --image=nginx:doesnotexist12345
kubectl describe pod bad-image | tail -15 # read the Events
# 2. CrashLoopBackOff
kubectl run crasher --image=busybox -- sh -c "exit 1"
kubectl logs crasher --previous
kubectl describe pod crasher | grep -A5 "Last State"
# 3. Pending (unschedulable — request more than any node has)
kubectl run huge --image=nginx --requests='cpu=100,memory=500Gi'
kubectl describe pod huge | grep -A5 Events # "Insufficient cpu/memory"
kubectl delete pod huge
# 4. CreateContainerConfigError (missing configmap/secret ref)
kubectl run missingcfg --image=nginx --env-from=configmap/doesnotexist
kubectl describe pod missingcfg | tail -10
# 5. Debug a "distroless" pod with no shell using ephemeral containers
kubectl run distroless --image=gcr.io/distroless/static-debian12 -- /nonexistent
kubectl debug -it distroless --image=busybox --target=distroless -- sh
# Cluster-wide triage habit
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get pods -A | grep -v Running | grep -v Completed
# Node-level: simulate disk pressure awareness
kubectl describe node prod-sim-m02 | grep -A10 Conditions
Common mistakes and how to avoid them
- Deleting the Pod before reading its state. You lose the Last State and the events attached to that Pod, which are exactly what explain the failure. Run
kubectl describe podandkubectl logs --previousfirst. - Reading only
kubectl logsfor a crashing container. It shows the current attempt, which is often empty. Add--previousto read the output of the attempt that crashed. - Debugging the Service when the Pods are not Ready. Run
kubectl get endpoints NAMEfirst. Empty endpoints mean a selector mismatch or Pods that fail readiness, so the Service itself is usually fine. - Treating Pending as a cluster bug. Pending almost always means the scheduler could not place the Pod. The Events in
kubectl describe podname the cause: insufficient CPU or memory, an untolerated taint, an unsatisfiable nodeSelector or an unbound PersistentVolumeClaim. - Forgetting the namespace.
kubectl get podsonly lists the current namespace. Use-n NAMEor-Abefore concluding that a workload does not exist.
Quick answers
What does CrashLoopBackOff mean?
The container starts, exits, and the kubelet restarts it with a delay that grows up to five minutes. It is a symptom, not a cause. Read kubectl logs --previous for the crash output and kubectl describe pod for the exit code and Last State.
Why is my Pod stuck in Pending?
The scheduler cannot place it. kubectl describe pod lists the reason in Events: insufficient CPU or memory, a taint no toleration matches, a nodeSelector or affinity rule no node satisfies, or a PersistentVolumeClaim that is not bound. Fix that condition and the Pod schedules.
How do I tell OOMKilled from an application crash?
Run kubectl describe pod and read Last State. Reason OOMKilled means the container exceeded its memory limit and was killed, usually with exit code 137. A plain exit code 1 is an application error, so read the logs instead and leave the memory limit alone.
Notes
(fill in your own words after watching + labbing)