16 - Cluster Hardening & CIS Benchmarks
Why this matters
This is the "prevent the whole cluster from being compromised" layer — API server flags, kubelet config, RBAC hygiene, and automated auditing against a known standard.
Read this first — Definitions & Explanations
CIS Kubernetes Benchmark
A hardening checklist of secure configuration recommendations for control plane, workers, policies, and files.
API server hardening
Disable anonymous access where appropriate, restrict bind addresses, enable audit logs, use strong auth, limit dangerous admission settings.
Encryption at rest
Configure API server encryption providers so Secrets (and other resources) are encrypted in etcd — not just base64.
Keep control plane private
Don’t expose the Kubernetes API to the open internet without strong controls (VPN, private networks, locked security groups).
Patching
Nodes and components need regular updates. A “hardened once” cluster that never patches slowly becomes unsafe.
Official docs (read for detail)
- Securing a Cluster
- Controlling Access to the Kubernetes API
- Auditing
- PKI certificates and requirements
- CIS Kubernetes Benchmark (kube-bench)
Key Concepts
- CIS Kubernetes Benchmark: the industry-standard hardening checklist
- kube-bench: automated CIS benchmark scanner for your actual cluster
- API server hardening: disabling anonymous auth, restricting
--insecure-port(removed in modern k8s), audit logging - kubelet hardening:
--anonymous-auth=false,--authorization-mode=Webhook, read-only port disabled - Audit logging: who did what, when — audit policy levels (None/Metadata/Request/RequestResponse)
- Minimizing the attack surface: disable unused API versions/features, restrict
--enable-admission-plugins - RBAC least-privilege review (revisit topic 08 with a security lens)
YouTube search terms
- "CIS Kubernetes Benchmark explained"
- "kube-bench tutorial Kubernetes security scan"
- "Kubernetes audit logging explained"
- "Kubernetes API server hardening CKS"
Hands-on lab (on prod-sim)
# Run kube-bench against your real cluster
minikube ssh -p prod-sim
sudo docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro \
--rm aquasec/kube-bench:latest run --targets master,node,etcd
exit
# Read the [FAIL] items — these are real, actionable findings on YOUR cluster
# Enable audit logging on the API server (minikube-specific path)
minikube ssh -p prod-sim
sudo mkdir -p /etc/kubernetes/audit
cat <<EOF | sudo tee /etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
# Edit /etc/kubernetes/manifests/kube-apiserver.yaml to add:
# --audit-log-path=/var/log/audit.log
# --audit-policy-file=/etc/kubernetes/audit/policy.yaml
# plus matching volumeMounts/volumes for the audit dir and log path.
# Kubelet will auto-restart the static pod when the manifest changes.
exit
# After it restarts, generate an event and check the audit log
kubectl get pods
minikube ssh -p prod-sim -- sudo tail -20 /var/log/audit.log
# Check for anonymous-auth / insecure settings
minikube ssh -p prod-sim -- sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -E 'anonymous-auth|insecure'
Common mistakes and how to avoid them
- Leaving the API server reachable from the internet. Keep the endpoint private or restrict it to known networks with a firewall or authorized network list.
- Leaving the kubelet open. Disable anonymous authentication on the kubelet and set authorization mode to Webhook so requests are checked against the API server.
- Not encrypting Secrets at rest. Without an EncryptionConfiguration, Secrets sit unencrypted in etcd. Configure a provider and rewrite existing Secrets.
- Running kube-bench once and forgetting it. Settings drift and upgrades change defaults. Re-run the CIS checks after upgrades and on a schedule.
- Delaying upgrades. Kubernetes supports only the most recent minor releases with security fixes. An old cluster stops receiving patches.
Quick answers
What is the CIS Kubernetes Benchmark?
It is a published checklist of security settings for the control plane, nodes and policies, maintained by the Center for Internet Security. Tools such as kube-bench run those checks against a cluster and report passes and failures so you can fix the gaps.
How do you encrypt Secrets at rest in Kubernetes?
Create an EncryptionConfiguration with a provider such as aescbc or a KMS plugin and pass it to the API server with --encryption-provider-config. Then rewrite existing Secrets so they are encrypted. New and updated Secrets are stored encrypted in etcd.
What does hardening the API server involve?
Disable anonymous access where possible, require RBAC and authenticated clients, enable audit logging, keep the endpoint off the public internet, and turn on admission controls such as Pod Security Admission. Check the settings against the CIS benchmark after every upgrade.
Notes
(fill in your own words after watching + labbing)