K8s by Example: Admission Webhooks

Admission webhooks intercept API requests before resources are persisted. ValidatingWebhooks reject invalid resources. MutatingWebhooks modify resources (inject sidecars, add labels). Misconfigured webhooks can block all deployments and crash clusters.

validatingwebhook.yaml

ValidatingWebhookConfiguration defines when to call your webhook. rules specify which resources and operations trigger validation.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy
webhooks:
  - name: pod-policy.example.com
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]

Configure the webhook service endpoint and TLS. The API server calls this service for each matching request.

    clientConfig:
      service:
        name: pod-policy-webhook
        namespace: webhook-system
        path: /validate
        port: 443
      caBundle: LS0tLS1CRUd...  # Base64 CA cert
    admissionReviewVersions: ["v1"]
    sideEffects: None
    timeoutSeconds: 5

mutatingwebhook.yaml

MutatingWebhookConfiguration modifies resources. Common uses: inject sidecars (Istio), add default labels, set security context. Mutations happen before validation.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
  - name: sidecar.example.com
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
    namespaceSelector:
      matchLabels:
        sidecar-injection: enabled

Use namespaceSelector or objectSelector to limit scope. This prevents the webhook from affecting system namespaces.

    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - kube-system
            - kube-public
    objectSelector:
      matchLabels:
        inject-sidecar: "true"

webhook-failurepolicy.yaml

failurePolicy controls what happens when the webhook is unavailable. Fail: reject requests (safe but blocks deployments). Ignore: allow requests (risky but won’t block).

webhooks:
  - name: critical-validation.example.com
    failurePolicy: Fail       # Block if webhook down
    timeoutSeconds: 10

  - name: optional-mutation.example.com
    failurePolicy: Ignore     # Allow if webhook down
    timeoutSeconds: 5

Critical: Fail policy with broken webhooks blocks ALL matching requests, including kube-system pods. This can crash your cluster. Always exclude system namespaces.

# DANGEROUS: Fails on all pods, including system
rules:
  - resources: ["pods"]
failurePolicy: Fail

# SAFE: Exclude system namespaces
namespaceSelector:
  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: ["kube-system", "kube-public"]

terminal

List all webhooks in the cluster. When deployments fail mysteriously, check if a webhook is rejecting them.

$ kubectl get validatingwebhookconfigurations
NAME                        WEBHOOKS   AGE
pod-policy                  1          7d
gatekeeper-validating       1          30d

$ kubectl get mutatingwebhookconfigurations
NAME                        WEBHOOKS   AGE
istio-sidecar-injector      1          30d
cert-manager-webhook        1          30d

Check webhook details when debugging. Look at rules, failure policy, and service endpoint.

$ kubectl get validatingwebhookconfiguration pod-policy -o yaml
webhooks:
- name: pod-policy.example.com
  failurePolicy: Fail
  clientConfig:
    service:
      name: pod-policy-webhook
      namespace: webhook-system

terminal

Webhook rejections appear in pod events. Look for admission webhook denied messages with the rejection reason.

$ kubectl describe pod failing-pod | grep -A10 Events
Events:
  Type     Reason     Message
  ----     ------     -------
  Warning  Failed     Error creating: admission webhook
           "pod-policy.example.com" denied the request:
           containers must not run as root

$ kubectl get events --field-selector reason=FailedCreate

Check webhook pod logs to understand why requests are rejected. The webhook service must be running and healthy.

$ kubectl logs -n webhook-system -l app=pod-policy-webhook
Validating pod default/myapp
Denied: runAsRoot not allowed in production namespace

$ kubectl get pods -n webhook-system
NAME                         READY   STATUS
pod-policy-webhook-abc123    1/1     Running

terminal

Emergency fix: Delete the webhook configuration to unblock deployments. This removes the webhook entirely until you fix and redeploy it.

# If webhook is blocking all deployments:
$ kubectl delete validatingwebhookconfiguration broken-webhook
validatingwebhookconfiguration.admissionregistration.k8s.io "broken-webhook" deleted

# Deployments should now work
$ kubectl get pods -n kube-system
# Pods starting again

Alternative: patch the webhook to Ignore failure policy temporarily while you fix the underlying issue.

# Temporarily allow requests when webhook fails
$ kubectl patch validatingwebhookconfiguration broken-webhook \
    --type='json' \
    -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'

# Fix the webhook service, then restore Fail policy
$ kubectl patch validatingwebhookconfiguration broken-webhook \
    --type='json' \
    -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Fail"}]'

webhook-service.yaml

The webhook must be highly available. Use multiple replicas, proper resource limits, and PodDisruptionBudget. A down webhook with Fail policy blocks deployments.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-policy-webhook
  namespace: webhook-system
spec:
  replicas: 3                    # HA
  selector:
    matchLabels:
      app: pod-policy-webhook
  template:
    spec:
      affinity:
        podAntiAffinity:         # Spread across nodes
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
      containers:
        - name: webhook
          resources:
            requests:
              cpu: 100m
              memory: 128Mi

Index | Use arrow keys to navigate