20 - CRDs & Custom Controllers/Operators
Why this matters
This is the single best exercise for truly understanding the Kubernetes API machinery. Once you've written a controller, every built-in resource (Deployment, Service, etc.) stops being magic — they're all just controllers watching an API and reconciling state.
Read this first — Definitions & Explanations
CRD (CustomResourceDefinition)
Extends the Kubernetes API with your own resource types (CronTab, PostgresCluster, etc.). After installing a CRD, you can kubectl get <your-kind>.
Custom Resource (CR)
An instance of a CRD — an object stored in etcd via the API server like native resources.
Operator pattern
A controller that encodes operational knowledge: watch CRs, reconcile complex apps (create StatefulSets, backups, failovers). “Kubernetes-native automation” for a specific software system.
Why operators exist
YAML alone can’t capture day-2 operations. Operators continuously maintain desired complex state.
Official docs (read for detail)
- Custom Resources
- Extend the Kubernetes API with CustomResourceDefinitions
- Operator pattern
- Kubebuilder Book
Key Concepts
- CRD (CustomResourceDefinition): extends the API with your own resource types
- Custom Resource (CR): an instance of your CRD
- Controller: watches CRs (and related resources), reconciles actual state to match spec
- Operator pattern: CRD + controller that encodes operational knowledge (e.g. "how to backup/upgrade this database")
- Reconciliation loop: idempotent, level-triggered (not edge-triggered) — critical concept
kubebuildervsoperator-sdk— scaffolding toolsclient-goinformers/listers — how controllers efficiently watch the API without hammering it
YouTube search terms
- "Kubernetes CRD custom resource definition explained"
- "Kubernetes Operator pattern explained"
- "kubebuilder tutorial write your first controller"
- "Kubernetes controller reconciliation loop level-triggered explained"
Hands-on lab (on prod-sim)
# First, just use one to see the pattern from the outside
kubectl apply -f https://raw.githubusercontent.com/cert-manager/cert-manager/master/deploy/crds/crd-certificates.yaml
kubectl get crd | grep cert-manager
# Now build your own minimal operator with kubebuilder
brew install kubebuilder
mkdir ~/dev/website-operator && cd ~/dev/website-operator
kubebuilder init --domain example.com --repo example.com/website-operator
kubebuilder create api --group web --version v1 --kind Website --resource --controller
# Edit api/v1/website_types.go: add a Spec field like `Replicas int32` and `Image string`
# Edit controllers/website_controller.go Reconcile(): make it create/update a Deployment
# matching Spec.Replicas / Spec.Image whenever a Website CR changes.
# (Full walkthrough: https://book.kubebuilder.io/cronjob-tutorial/cronjob-tutorial.html
# — the CronJob tutorial in the kubebuilder book is the canonical hands-on guide, do that one first)
make manifests install run # runs your controller locally against prod-sim's API server
# In another terminal, create a CR and watch your controller react
cat <<EOF | kubectl apply -f -
apiVersion: web.example.com/v1
kind: Website
metadata:
name: my-site
spec:
replicas: 2
image: nginx:1.27
EOF
kubectl get deployments # should see one your controller created
kubectl get website my-site -o yaml
Common mistakes and how to avoid them
- Creating a CRD and expecting it to do something. A CRD only adds a new API type. The behaviour comes from a controller that watches those objects.
- Deleting a CRD casually. Deleting a CRD deletes every custom resource of that type. Back up the objects first.
- Writing a controller that is not idempotent. Reconcile can run many times for one change. It must converge on the desired state without creating duplicates.
- Skipping schema validation. Without an
openAPIV3Schema, bad input is accepted and fails later. Define the schema and required fields. - Ignoring the status subresource. Users need to see what the controller observed. Write progress and errors to
status, not tospec.
Quick answers
What is a CRD?
A CustomResourceDefinition adds a new resource type to the Kubernetes API, such as Database or Certificate. Once registered, you create objects of that type with kubectl like any built-in resource. A CRD only defines the data, and a controller must act on it.
What is the operator pattern?
An operator is a custom controller that encodes operational knowledge for an application, such as how to deploy, back up and upgrade a database. You declare the desired state in a custom resource, and the operator reconciles the cluster toward it continuously.
What is a reconcile loop in a controller?
It reads the desired state from the custom resource, compares it with the actual state of the cluster, and makes the changes needed to close the gap. It must be idempotent, because it can run many times for one change. Updating the resource's status records what happened.
Notes
(fill in your own words after watching + labbing)