15 - Monitoring & Logging
Why this matters
You can't operate what you can't observe. This is the standard stack you'll meet at almost every company running Kubernetes.
Read this first — Definitions & Explanations
metrics-server
Provides resource metrics for kubectl top and resource-based HPA. It is not long-term historical monitoring.
Prometheus
Pull-based metrics system that scrapes targets (/metrics). Stores time series for alerting and dashboards.
Grafana
Visualization layer — dashboards on top of Prometheus (and other data sources).
Logging patterns
Apps write logs → node agents collect → aggregator/store (ELK/EFK, Loki, cloud logging). kubectl logs is for live debugging, not retention.
Alerting
Alert on symptoms and SLOs (error rate, latency, saturation), not every debug line. Actionable alerts only.
Official docs (read for detail)
Key Concepts
- metrics-server: lightweight, powers
kubectl topand HPA — NOT for long-term storage - Prometheus: pull-based metrics, PromQL, ServiceMonitor/PodMonitor (via Prometheus Operator)
- Grafana: dashboards on top of Prometheus (and Loki)
- Loki: log aggregation, label-based (like Prometheus but for logs)
- kube-state-metrics: cluster object state as metrics (separate from metrics-server)
- Node-level logging: where container logs actually live on disk, log rotation
- Alerting: Alertmanager, routing, silence windows
YouTube search terms
- "Prometheus Grafana Kubernetes monitoring stack tutorial"
- "kube-state-metrics vs metrics-server explained"
- "Loki Kubernetes logging tutorial"
- "PromQL tutorial for beginners"
Hands-on lab (on prod-sim)
# You already have metrics-server. Add the full observability stack via Helm.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
kubectl get pods -n monitoring
# Access Grafana
kubectl -n monitoring port-forward svc/kps-grafana 3000:80
# open http://localhost:3000, default user admin, get password:
kubectl -n monitoring get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d
# Access Prometheus directly, run a PromQL query
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090
# open http://localhost:9090, try query: rate(container_cpu_usage_seconds_total[5m])
# Add Loki for logs
helm install loki grafana/loki-stack -n monitoring --set grafana.enabled=false
# Add Loki as a data source in Grafana (http://loki:3100), then explore logs by pod label
# Generate some load to see metrics move
kubectl run loadtest --image=busybox --restart=Never -- sh -c \
"while true; do wget -qO- http://kubernetes.default 2>/dev/null; done"
Common mistakes and how to avoid them
- Expecting
kubectl topto work without metrics-server. The command reads from the Metrics API, which metrics-server provides. Install it first, or the command reports that metrics are unavailable. - Alerting on raw CPU instead of symptoms. Page on what users feel, such as error rate and latency, and use resource alerts for capacity planning.
- Relying on container logs stored on the node. They disappear when the Pod or node is replaced. Ship logs to central storage with a node-level agent.
- Using high-cardinality labels in Prometheus. Labels such as user ID or request ID create a time series for every value and exhaust memory. Keep label values bounded.
- Having no retention plan. Prometheus keeps local data for a limited time, 15 days by default. Plan long-term storage if you need older history.
Quick answers
What does metrics-server provide?
It collects current CPU and memory usage from each kubelet and exposes it through the Metrics API. That powers kubectl top and lets the HPA scale on CPU. It stores no history, so use Prometheus for trends and alerting.
How does Prometheus collect metrics from Kubernetes?
Prometheus scrapes HTTP metrics endpoints on a schedule and discovers targets through the Kubernetes API, so new Pods are found automatically. The Prometheus Operator adds ServiceMonitor objects to declare what to scrape, and Grafana queries Prometheus for dashboards.
Why ship container logs off the node?
Logs live on the node's disk and are lost when a Pod is deleted or the node is replaced. A node-level agent, such as Fluent Bit running as a DaemonSet, forwards them to central storage so you can search them after the Pod is gone.
Notes
(fill in your own words after watching + labbing)