Kubernetes Taints and Tolerations: A Complete Guide

Taints and tolerations are Kubernetes’ mechanism for repelling pods from nodes. A taint on a node says “do not schedule here unless you explicitly tolerate me.” A toleration on a pod says “I can handle that taint.” Together, they let you dedicate nodes to specific workloads, protect control plane nodes, and drain nodes for maintenance.

How Taints and Tolerations Work

The interaction is simple:

1. An administrator adds a taint to a node
2. The scheduler checks each pod for matching tolerations
3. If the pod tolerates the taint, it can schedule on the node
4. If the pod does not tolerate the taint, the scheduler skips that node

This is the opposite of node affinity. Node affinity attracts pods to nodes. Taints repel pods from nodes. You often use both together for dedicated workloads.

Taint Effects

Every taint has an effect that determines what happens to pods that do not tolerate it:

NoSchedule

New pods without a matching toleration will not be scheduled on the node. Existing pods are unaffected:

kubectl taint nodes worker-3 gpu=true:NoSchedule

Only pods that tolerate the gpu=true taint can be scheduled on worker-3. Pods already running on the node stay.

PreferNoSchedule

A soft version of NoSchedule. The scheduler tries to avoid placing non-tolerating pods on the node but will do so if no other node is available:

kubectl taint nodes worker-4 env=staging:PreferNoSchedule

This is useful for expressing a preference without causing scheduling failures.

NoExecute

The strictest effect. Non-tolerating pods are evicted from the node, and new non-tolerating pods cannot be scheduled:

kubectl taint nodes worker-5 maintenance=true:NoExecute

All pods on worker-5 that do not tolerate this taint are evicted immediately. This is how kubectl drain works under the hood.

Adding and Removing Taints

# Add a taint
kubectl taint nodes worker-3 gpu=true:NoSchedule

# Verify taints on a node
kubectl describe node worker-3 | grep Taints

# Remove a specific taint (note the trailing minus)
kubectl taint nodes worker-3 gpu=true:NoSchedule-

# Remove all taints with a key (regardless of value and effect)
kubectl taint nodes worker-3 gpu-

The trailing - removes the taint. This is easy to miss and is one of the most common mistakes with taint commands.

Writing Tolerations

A pod tolerates a taint by including a matching toleration in its spec:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  tolerations:
    - key: gpu
      operator: Equal
      value: "true"
      effect: NoSchedule
  containers:
    - name: training
      image: ml-training:latest
      resources:
        limits:
          nvidia.com/gpu: 1

Toleration Operators

Equal (default) matches when the key, value, and effect all match:

tolerations:
  - key: gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

Exists matches when the key exists, regardless of value:

tolerations:
  - key: gpu
    operator: Exists
    effect: NoSchedule

To tolerate all taints with any key, use Exists without specifying a key:

tolerations:
  - operator: Exists

This is a blanket toleration. Use it sparingly, primarily for infrastructure DaemonSets that must run everywhere.

Built-in Taints

Kubernetes automatically applies taints to nodes in certain conditions:

Taint	When Applied
node.kubernetes.io/not-ready	Node is not ready
node.kubernetes.io/unreachable	Node is unreachable from the controller
node.kubernetes.io/memory-pressure	Node is low on memory
node.kubernetes.io/disk-pressure	Node is low on disk space
node.kubernetes.io/pid-pressure	Node has too many processes
node.kubernetes.io/unschedulable	Node is cordoned
node-role.kubernetes.io/control-plane	Node runs control plane components

Kubernetes adds default tolerations to all pods for not-ready and unreachable with a tolerationSeconds of 300 (5 minutes). This means pods stay on an unreachable node for 5 minutes before being evicted.

Eviction with tolerationSeconds

When using NoExecute taints, you can control how long a pod stays before eviction:

tolerations:
  - key: node.kubernetes.io/not-ready
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 60

This pod tolerates a not-ready node for 60 seconds before being evicted. Without tolerationSeconds, a tolerating pod stays indefinitely.

Pattern: Dedicated Nodes for a Team

Reserve nodes for a specific team by tainting them and adding tolerations to that team’s workloads:

# Taint the nodes
kubectl taint nodes worker-10 team=data:NoSchedule
kubectl taint nodes worker-11 team=data:NoSchedule
kubectl label nodes worker-10 worker-11 team=data

The team’s Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: data-pipeline
spec:
  replicas: 4
  selector:
    matchLabels:
      app: data-pipeline
  template:
    metadata:
      labels:
        app: data-pipeline
    spec:
      tolerations:
        - key: team
          operator: Equal
          value: data
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: team
                    operator: In
                    values:
                      - data
      containers:
        - name: pipeline
          image: data-pipeline:latest

The taint prevents other pods from landing on these nodes. The node affinity ensures the data team’s pods actually go to these nodes (tolerations alone do not attract pods).

Pattern: GPU Nodes

GPU nodes are expensive. Taint them so only workloads that need GPUs schedule there:

kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule

ML training pods tolerate the taint and request the GPU resource:

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
containers:
  - name: training
    resources:
      limits:
        nvidia.com/gpu: 1

Non-GPU workloads are automatically excluded from these expensive nodes.

Pattern: Graceful Node Maintenance

When you need to update or restart a node:

# Cordon the node (prevents new pods)
kubectl cordon worker-5

# Drain the node (evicts pods gracefully)
kubectl drain worker-5 --ignore-daemonsets --delete-emptydir-data

# Perform maintenance...

# Uncordon when done
kubectl uncordon worker-5

kubectl drain applies a NoExecute taint and waits for pods to terminate gracefully. The --ignore-daemonsets flag is necessary because DaemonSet pods have nowhere else to go.

Common Mistakes

Forgetting that tolerations do not attract. A toleration says “I can go here” but not “I should go here.” Without node affinity or a node selector, a tolerating pod might still be scheduled on non-tainted nodes.

Using blanket tolerations on application pods. This defeats the purpose of taints. Reserve operator: Exists (without a key) for infrastructure DaemonSets only.

Tainting nodes without a migration plan. If you taint a node with NoExecute while application pods are running and those pods have no toleration, they are evicted immediately. Use NoSchedule first to stop new pods, then migrate existing ones.

Practical Recommendations

Use taints and tolerations to isolate expensive resources (GPUs), dedicate nodes to teams or environments, and manage node lifecycle. Always pair taints with node affinity when you want pods on specific nodes. Use NoSchedule for soft dedication and NoExecute for hard eviction. Adjust tolerationSeconds on your pods’ default not-ready toleration based on how quickly you want failover to happen.