K8s by Example: Taints & Tolerations

Taints mark nodes to repel Pods; tolerations allow Pods to schedule onto tainted nodes. Unlike affinity (attracting Pods to nodes), taints work by repulsion. Use for: dedicated nodes, GPU nodes, maintenance, preventing workloads from specific hardware.
taint-effects.yaml

Three taint effects: NoSchedule (new Pods won’t schedule), PreferNoSchedule (soft preference to avoid), NoExecute (existing Pods evicted, new Pods won’t schedule). Taints have key, value, and effect.

 
kubectl taint nodes worker-1 dedicated=gpu:NoSchedule

kubectl taint nodes worker-2 preferred=lowpriority:PreferNoSchedule

kubectl taint nodes worker-3 maintenance=true:NoExecute
toleration-exact.yaml

The Equal operator requires exact key and value match. This Pod can run on nodes with dedicated=gpu:NoSchedule taint. Without this toleration, it would be rejected.

 
apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  containers:
    - name: ml-training
      image: ml-trainer:v1
      resources:
        limits:
          nvidia.com/gpu: 1
toleration-exists.yaml

The Exists operator matches any taint with the specified key, regardless of value. Useful when you don’t care about the specific taint value. Omit value when using Exists.

 
apiVersion: v1
kind: Pod
metadata:
  name: tolerate-any-dedicated
spec:
  tolerations:
    - key: "dedicated"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: app
      image: my-app:v1
---
tolerations:
  - operator: "Exists"
    effect: "NoSchedule"
---
tolerations:
  - operator: "Exists"
toleration-seconds.yaml

tolerationSeconds specifies how long a Pod stays on a node after a NoExecute taint is applied. Useful for graceful migration during maintenance. Without it, Pods are evicted immediately.

 
apiVersion: v1
kind: Pod
metadata:
  name: graceful-eviction
spec:
  tolerations:
    - key: "node.kubernetes.io/not-ready"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 300
    - key: "node.kubernetes.io/unreachable"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 300
  containers:
    - name: app
      image: my-app:v1
dedicated-nodes.yaml

Dedicated nodes pattern: taint nodes for specific teams/workloads. Only Pods with matching tolerations (and affinity for targeting) can run there. Prevents noisy neighbors and enables cost allocation.

 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: team-alpha-app
spec:
  template:
    spec:
      tolerations:
        - key: "team"
          operator: "Equal"
          value: "alpha"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: team
                    operator: In
                    values: ["alpha"]
      containers:
        - name: app
          image: team-alpha-app:v1
maintenance-drain.yaml

Maintenance workflow: taint node with NoExecute to evict workloads. DaemonSets and critical Pods need tolerations to survive. After maintenance, remove the taint to allow scheduling again.

 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  template:
    spec:
      tolerations:
        - key: "maintenance"
          operator: "Exists"
          effect: "NoExecute"
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.7.0
built-in-taints.yaml

Kubernetes adds taints automatically for node conditions. Default tolerations allow Pods 5 minutes before eviction. Override with your own tolerations. Common built-in taints handle node failures gracefully.

 
tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300
terminal

Manage taints with kubectl. Add taints to repel Pods, remove taints (with minus suffix) to allow scheduling. View taints in node description or get nodes output.

 
$ kubectl taint nodes worker-1 dedicated=gpu:NoSchedule
node/worker-1 tainted

$ kubectl taint nodes worker-2 team=alpha:NoSchedule
node/worker-2 tainted

$ kubectl describe node worker-1 | grep -A3 Taints
Taints:             dedicated=gpu:NoSchedule

$ kubectl get nodes -o custom-columns=\
NAME:.metadata.name,TAINTS:.spec.taints
NAME       TAINTS
worker-1   [map[effect:NoSchedule key:dedicated value:gpu]]
worker-2   [map[effect:NoSchedule key:team value:alpha]]

$ kubectl taint nodes worker-1 dedicated=gpu:NoSchedule-
node/worker-1 untainted

Index | GitHub | Use arrow keys to navigate |