14 - Backup & Restore (etcd, Velero)
Why this matters
Everything in Kubernetes lives in etcd. If etcd is gone with no backup, your entire cluster state (not workloads' data — the cluster's definition of itself) is gone. This is a CKA exam topic and a real disaster-recovery skill.
Read this first — Definitions & Explanations
Why etcd backup matters
etcd holds API objects. No good etcd backup ≈ you can lose cluster state (Deployments, RBAC, etc.). App data in disks/DBs is a separate concern.
etcd snapshot
Point-in-time backup of etcd. Restores must be done carefully with matching versions and procedures.
Velero
Popular tool for backing up Kubernetes resources and, with plugins, persistent volume data. Useful for namespace/cluster disaster recovery drills.
App backup vs cluster backup
- Cluster/API backup: objects in etcd (+ optionally PVs)
- Application backup: database dumps, object storage, etc.
Production usually needs both.
Test restores
A backup you’ve never restored is a hope, not a plan.
Official docs (read for detail)
Key Concepts
- etcd snapshot save/restore (
etcdctl snapshot save/restore) - What a restore actually does (restores to a new data dir, requires re-pointing etcd)
- Velero: backs up actual Kubernetes objects + can snapshot PV data (via cloud provider snapshot APIs)
- Backup scope decisions: whole cluster vs namespace vs label-selected resources
- Disaster recovery drill mentality: backups are worthless until you've tested a restore
YouTube search terms
- "etcd snapshot backup restore Kubernetes"
- "Velero Kubernetes backup restore tutorial"
- "Kubernetes disaster recovery etcd"
Hands-on lab (on prod-sim)
# etcd snapshot save + restore drill
minikube ssh -p prod-sim
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/minikube/certs/etcd/ca.crt \
--cert=/var/lib/minikube/certs/etcd/server.crt \
--key=/var/lib/minikube/certs/etcd/server.key \
snapshot save /tmp/etcd-backup.db
sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status /tmp/etcd-backup.db
exit
# Create something, "accidentally" delete it, understand what a restore would recover
kubectl create namespace important-stuff
kubectl -n important-stuff create configmap must-not-lose --from-literal=key=value
# (In a real DR drill: stop kube-apiserver, restore the snapshot to a new data-dir,
# update etcd manifest to point at it, restart. Full walkthrough:
# https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#restoring-an-etcd-cluster)
# Velero (application-level backup, easier to reason about for this exercise)
# Install Velero CLI, then with a local MinIO as the backup target (since no cloud provider here):
# https://velero.io/docs/main/contributions/minio/
# Once installed:
velero backup create important-stuff-backup --include-namespaces important-stuff
velero backup describe important-stuff-backup
kubectl delete namespace important-stuff
velero restore create --from-backup important-stuff-backup
kubectl get configmap -n important-stuff must-not-lose # should be back
Common mistakes and how to avoid them
- Never testing a restore. An untested backup is a guess. Practise restoring into a scratch cluster so you know the steps and the time it takes.
- Backing up etcd and nothing else. etcd holds object definitions, not the data on persistent volumes. Back up volumes separately with storage snapshots or Velero.
- Restoring a snapshot over live data. Restore the snapshot into a new data directory and point the etcd manifest at it, instead of overwriting the running data.
- Running snapshot commands without certificates.
etcdctlneeds the endpoint plus the CA, certificate and key flags. A bare command just fails to connect. - Keeping backups on the same node. A node failure then takes the backup too. Copy snapshots to separate storage, ideally in another location.
Quick answers
How do you back up etcd?
Run etcdctl snapshot save FILE with the endpoint and the CA, certificate and key flags, usually from a control plane node, then copy the file off the node. Verify it with etcdutl snapshot status and schedule the backup regularly.
What does Velero back up?
Velero backs up Kubernetes resources and, with a plugin, persistent volume data to object storage, and can restore them to the same or another cluster. It works by namespace, label or whole cluster, so it covers application backup that an etcd snapshot alone does not.
What is the difference between application backup and cluster backup?
Cluster backup captures control plane state, mainly the etcd snapshot, and recovers the whole cluster. Application backup captures chosen namespaces, resources and volumes, for example with Velero, so you can recover or migrate one app. Most teams need both.
Notes
(fill in your own words after watching + labbing)