20 - CRDs & Custom Controllers/Operators

Why this matters

This is the single best exercise for truly understanding the Kubernetes API machinery. Once you've written a controller, every built-in resource (Deployment, Service, etc.) stops being magic — they're all just controllers watching an API and reconciling state.

Read this first — Definitions & Explanations

CRD (CustomResourceDefinition)

Extends the Kubernetes API with your own resource types (CronTab, PostgresCluster, etc.). After installing a CRD, you can kubectl get <your-kind>.

Custom Resource (CR)

An instance of a CRD — an object stored in etcd via the API server like native resources.

Operator pattern

A controller that encodes operational knowledge: watch CRs, reconcile complex apps (create StatefulSets, backups, failovers). “Kubernetes-native automation” for a specific software system.

Why operators exist

YAML alone can’t capture day-2 operations. Operators continuously maintain desired complex state.

Official docs (read for detail)

• Custom Resources
• Extend the Kubernetes API with CustomResourceDefinitions
• Operator pattern
• Kubebuilder Book

Key Concepts

• CRD (CustomResourceDefinition): extends the API with your own resource types
• Custom Resource (CR): an instance of your CRD
• Controller: watches CRs (and related resources), reconciles actual state to match spec
• Operator pattern: CRD + controller that encodes operational knowledge (e.g. "how to backup/upgrade this database")
• Reconciliation loop: idempotent, level-triggered (not edge-triggered) — critical concept
• kubebuilder vs operator-sdk — scaffolding tools
• client-go informers/listers — how controllers efficiently watch the API without hammering it

YouTube search terms

• "Kubernetes CRD custom resource definition explained"
• "Kubernetes Operator pattern explained"
• "kubebuilder tutorial write your first controller"
• "Kubernetes controller reconciliation loop level-triggered explained"

Hands-on lab (on prod-sim)

# First, just use one to see the pattern from the outside
kubectl apply -f https://raw.githubusercontent.com/cert-manager/cert-manager/master/deploy/crds/crd-certificates.yaml
kubectl get crd | grep cert-manager

# Now build your own minimal operator with kubebuilder
brew install kubebuilder
mkdir ~/dev/website-operator && cd ~/dev/website-operator
kubebuilder init --domain example.com --repo example.com/website-operator
kubebuilder create api --group web --version v1 --kind Website --resource --controller

# Edit api/v1/website_types.go: add a Spec field like `Replicas int32` and `Image string`
# Edit controllers/website_controller.go Reconcile(): make it create/update a Deployment
# matching Spec.Replicas / Spec.Image whenever a Website CR changes.
# (Full walkthrough: https://book.kubebuilder.io/cronjob-tutorial/cronjob-tutorial.html
#  — the CronJob tutorial in the kubebuilder book is the canonical hands-on guide, do that one first)

make manifests install run   # runs your controller locally against prod-sim's API server

# In another terminal, create a CR and watch your controller react
cat <<EOF | kubectl apply -f -
apiVersion: web.example.com/v1
kind: Website
metadata:
  name: my-site
spec:
  replicas: 2
  image: nginx:1.27
EOF
kubectl get deployments   # should see one your controller created
kubectl get website my-site -o yaml

Notes

(fill in your own words after watching + labbing)