K8s by Example: Prometheus Monitoring

Prometheus is the standard for Kubernetes metrics. Applications expose metrics on an HTTP endpoint; Prometheus scrapes them periodically. The Prometheus Operator simplifies deployment with custom resources like ServiceMonitor and PodMonitor. Use for: alerting, dashboards, capacity planning, SLO tracking.

	terminal
Install kube-prometheus-stack via Helm. This deploys Prometheus, Alertmanager, Grafana, and the Prometheus Operator which provides ServiceMonitor, PodMonitor, and PrometheusRule CRDs.	`$ helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace NAME: prometheus STATUS: deployed NOTES: kube-prometheus-stack has been installed.`

metrics-app.yaml

Applications expose metrics on a /metrics endpoint in Prometheus format. Annotations tell Prometheus to scrape this Pod. Port 8080 serves the app; metrics may be on the same or different port.

apiVersion: v1
kind: Pod
metadata:
  name: web-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  containers:
    - name: app
      image: my-app:v1
      ports:
        - containerPort: 8080
          name: http

service-monitor.yaml

ServiceMonitor (from Prometheus Operator) defines how to scrape Services. It selects Services by label and specifies which port to scrape. More powerful than annotations for complex setups.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web-app-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: web-app
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
  namespaceSelector:
    matchNames:
      - default

pod-monitor.yaml

PodMonitor scrapes Pods directly without needing a Service. Useful for DaemonSets, Jobs, or Pods that don’t need Services. Selects Pods by label, specifies container port to scrape.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: batch-job-monitor
spec:
  selector:
    matchLabels:
      app: batch-processor
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 15s
  namespaceSelector:
    matchNames:
      - batch-jobs

service-with-monitor.yaml

Complete example: Deployment, Service, and ServiceMonitor together. The Service exposes the app; the ServiceMonitor tells Prometheus how to scrape it. Labels connect all three resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: api-server:v1
          ports:
            - containerPort: 8080
              name: http
            - containerPort: 9090
              name: metrics
---
apiVersion: v1
kind: Service
metadata:
  name: api-server
  labels:
    app: api-server
spec:
  selector:
    app: api-server
  ports:
    - port: 8080
      name: http
    - port: 9090
      name: metrics

prometheus-rules.yaml

PrometheusRule defines alerting and recording rules. Recording rules precompute expensive queries. Alerting rules fire when conditions are met for a duration. Labels route alerts to different receivers.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-server-rules
  labels:
    release: prometheus
spec:
  groups:
    - name: api-server
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate on {{ $labels.service }}"
        - record: api:request_duration:p99
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_bucket[5m]))
              by (le, service))

adapter-metrics.yaml

For apps that don’t expose Prometheus metrics, use an adapter sidecar (exporter). The exporter queries the app and translates its metrics to Prometheus format. Common exporters exist for Redis, MySQL, PostgreSQL, etc.

apiVersion: v1
kind: Pod
metadata:
  name: redis-with-exporter
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
spec:
  containers:
    - name: redis
      image: redis:7-alpine
      ports:
        - containerPort: 6379
    - name: exporter
      image: oliver006/redis_exporter:v1.55.0
      ports:
        - containerPort: 9121
          name: metrics
      env:
        - name: REDIS_ADDR
          value: "localhost:6379"

terminal

Verify Prometheus is discovering your targets. Check ServiceMonitor/PodMonitor resources, then query Prometheus targets endpoint or UI to confirm scraping is working.

$ kubectl get servicemonitors -A
NAMESPACE   NAME               AGE
default     api-server         5m
default     web-app-monitor    10m

$ kubectl get podmonitors -A
NAMESPACE    NAME                AGE
batch-jobs   batch-job-monitor   3m

$ kubectl port-forward svc/prometheus 9090:9090 -n monitoring &

$ curl localhost:9090/api/v1/targets | jq '.data.activeTargets[].labels'

$ curl localhost:9090/api/v1/query?query=up
{
  "data": {
    "result": [
      {"metric": {"job": "api-server"}, "value": [1234567890, "1"]}
    ]
  }
}

Index | GitHub | Use arrow keys to navigate |