Kubernetes Taints and Tolerations Explained

bitcodematrix.com | Kubernetes Series

1. Introduction

Node affinity and nodeSelector attract Pods to nodes. Taints and tolerations work in the opposite direction: a taint is placed on a node and repels Pods, and a toleration is placed on a Pod to say that it may be scheduled onto nodes carrying a matching taint. Together they let you reserve nodes for special workloads, keep ordinary Pods away from expensive hardware, protect control plane nodes, and make Kubernetes react automatically when a node becomes unhealthy.

This article explains how taints and tolerations work, the three taint effects, how matching is done, how to add and remove taints with kubectl, which taints Kubernetes adds by itself, how to dedicate nodes by combining taints with nodeSelector or node affinity, how taint-based eviction and tolerationSeconds work, and how to troubleshoot Pods that stay Pending because of taints. It continues the scheduling series with the articles on nodeSelector, node affinity and Pod affinity.

After reading it you should be able to:

• Explain what a taint and a toleration are and how a Pod is matched against node taints.

• Choose between the NoSchedule, PreferNoSchedule and NoExecute effects.

• Add, view and remove taints, and write tolerations in Pod specs.

• Dedicate nodes for GPU, spot or team workloads using taints plus attraction rules.

• Understand node problem taints, eviction timing and common scheduling failures.

2. The Core Idea

Figure 1: Taints repel Pods; tolerations let Pods in

The most important thing to remember is the direction of the relationship. A toleration does not pull a Pod toward a tainted node. A Pod that tolerates a taint can still be scheduled on any other suitable node. If you want Pods to run only on certain nodes, you need both repulsion for others (the taint) and attraction for yours (nodeSelector or node affinity), as shown in section 7.

3. Taint Effects

The effect is part of the taint and part of the match. A toleration for NoSchedule does not protect a Pod from a NoExecute taint with the same key unless it also covers that effect.

4. Working with Taints

4.1 Adding a taint

# Syntax: kubectl taint nodes <node> <key>=<value>:<effect>

kubectl taint nodes node-1 dedicated=gpu:NoSchedule

# A taint without a value

kubectl taint nodes node-1 maintenance:NoSchedule

# Multiple taints on one node are allowed

kubectl taint nodes node-1 team=ml:NoSchedule

4.2 Viewing taints

kubectl describe node node-1 | grep -i taints

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

4.3 Removing a taint

# Repeat the taint with a trailing minus

kubectl taint nodes node-1 dedicated=gpu:NoSchedule-

# Remove all taints with a given key, any effect

kubectl taint nodes node-1 dedicated-

On managed Kubernetes services, nodes are usually created from node pools, and taints set manually with kubectl may disappear when nodes are replaced. Define taints in the node pool configuration so that every new node gets them, and so that autoscalers know about them.

5. Writing Tolerations

Tolerations are set in the Pod spec (in the Pod template for controllers such as Deployments). Each toleration has the following fields:

spec:

  tolerations:

    # Equal operator: key, value and effect must match

    - key: "dedicated"

      operator: "Equal"

      value: "gpu"

      effect: "NoSchedule"

    # Exists operator: any value for this key

    - key: "team"

      operator: "Exists"

      effect: "NoSchedule"

    # Tolerate a NoExecute taint for 10 minutes

    - key: "node.kubernetes.io/unreachable"

      operator: "Exists"

      effect: "NoExecute"

      tolerationSeconds: 600

Two special forms deserve care. A toleration with an empty key and operator Exists matches every taint, which is what some system components use but is rarely appropriate for applications. A toleration with an empty effect matches all effects for the given key.

5.1 How matching works

The scheduler considers all taints on a node and filters out those that the Pod tolerates. The Pod's fate then depends on the remaining taints:

• If at least one remaining taint has effect NoSchedule, the Pod is not scheduled on the node.

• If there is no NoSchedule taint but at least one PreferNoSchedule taint, the node is scored lower but remains possible.

• If at least one remaining taint has effect NoExecute, the Pod is not scheduled there and, if already running, is evicted.

A node with several taints therefore requires the Pod to tolerate every NoSchedule and NoExecute taint on it.

6. Taints Added by Kubernetes

Kubernetes itself uses taints, so it is normal to see some on your nodes. The node lifecycle controller adds taints automatically when a node has problems, and the control plane nodes are tainted to keep application Pods away. The list below shows common ones; the exact set depends on your version and platform.

6.1 Default tolerations added automatically

• The DefaultTolerationSeconds admission controller adds tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable (NoExecute) with tolerationSeconds of 300 to Pods that do not define their own. This means that Pods stay on a failed node for about five minutes before being evicted, which avoids reacting to short network blips.

• The DaemonSet controller automatically adds tolerations for several node condition taints (such as not-ready, unreachable, disk-pressure, memory-pressure and unschedulable) so that node-level agents keep running on troubled nodes.

• For extended resources such as GPUs, the ExtendedResourceToleration admission plugin, where enabled, adds a toleration for a taint named after the resource, so you can taint GPU nodes with the resource name and have GPU-requesting Pods tolerate it automatically.

These behaviours depend on the admission plugins enabled in your cluster, so confirm them in your distribution's documentation. Managed services typically enable the standard ones.

6.2 Cordon and drain

Commands you may use during maintenance rely on the same mechanism. kubectl cordon marks a node unschedulable, which results in the node.kubernetes.io/unschedulable taint, so no new Pods arrive. kubectl drain cordons the node and then evicts its Pods through the eviction API (respecting Pod Disruption Budgets), whereas a NoExecute taint evicts matching Pods directly and does not consult PDBs in the same way. Use drain for planned maintenance.

kubectl cordon node-1

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

kubectl uncordon node-1

7. Dedicating Nodes: Taints Plus Attraction

To dedicate a set of nodes to a workload, you need two things at once: other Pods must be kept off the nodes, and your Pods must be steered onto them.

# Dedicate gpu-node-1 to GPU workloads

kubectl label node gpu-node-1 workload=gpu

kubectl taint node gpu-node-1 dedicated=gpu:NoSchedule

# Pod template of the GPU workload

spec:

  nodeSelector:

    workload: gpu

  tolerations:

    - key: "dedicated"

      operator: "Equal"

      value: "gpu"

      effect: "NoSchedule"

  containers:

    - name: trainer

      image: registry.example.com/ml/trainer:1.0

      resources:

        limits:

          nvidia.com/gpu: 1

The image name is a placeholder. The nvidia.com/gpu resource is provided by the vendor's device plugin and applies only if it is installed. If you left out the nodeSelector, the GPU Pods would tolerate the taint but could still run on ordinary nodes; if you left out the taint, ordinary Pods could land on the GPU nodes.

8. Common Use Cases

9. Taint-Based Eviction and tolerationSeconds

A NoExecute taint has an active effect: the node lifecycle controller evicts Pods that do not tolerate it. With tolerationSeconds, a Pod can stay for a limited time. This is how Kubernetes deals with failing nodes.

1. A node stops reporting its status. After the node monitor grace period, the node controller marks it NotReady or Unknown and adds the not-ready or unreachable NoExecute taint.

2. Pods without their own tolerations carry the default 300-second toleration, so they remain assigned to the node for that time.

3. If the taint is still present when the toleration time expires, the Pods are evicted. Controllers such as Deployments create replacements on healthy nodes.

You can change this behaviour per workload. Applications that recover quickly on another node can use a short tolerationSeconds to fail over sooner. Applications that need to avoid unnecessary restarts during brief network interruptions can use a longer value. Remember that the node controller decides when a node is considered unreachable, and that Pods with local state or ReadWriteOnce volumes may also need additional time for the volume to be detached before the replacement can start.

spec:

  tolerations:

    - key: "node.kubernetes.io/unreachable"

      operator: "Exists"

      effect: "NoExecute"

      tolerationSeconds: 60

    - key: "node.kubernetes.io/not-ready"

      operator: "Exists"

      effect: "NoExecute"

      tolerationSeconds: 60

10. Taints and Tolerations vs Other Mechanisms

The mechanisms complement each other. Taints are controlled from the node side, so platform teams can protect nodes without relying on every application team to write the right affinity. Affinity and nodeSelector are controlled from the Pod side, so application teams can express their needs. Dedicated node pools usually need both.

11. Interaction with Other Features

12. Hands-On Walkthrough

This exercise taints a node, shows a Pod without a toleration being kept away, and then shows a tolerating Pod being allowed. Use a test cluster with at least two schedulable nodes and permission to taint nodes.

1. Pick a node and add a taint.

2. Create a Deployment without tolerations and check that none of its Pods land on the tainted node.

3. Create a Pod with a toleration and a nodeSelector for that node, and verify that it runs there.

4. Switch the taint to NoExecute and observe the eviction of Pods without tolerations.

5. Remove the taint and clean up.

kubectl get nodes

NODE=<chosen-node>

# 1. Taint the node and give it a label

kubectl taint nodes $NODE dedicated=demo:NoSchedule

kubectl label node $NODE workload=demo

# 2. Pods without tolerations avoid the node

kubectl create deployment plain --image=nginx:1.27 --replicas=6

kubectl get pods -l app=plain -o wide

# 3. A tolerating Pod pinned to the node

cat <<EOF | kubectl apply -f -

apiVersion: v1

kind: Pod

metadata:

  name: tolerant

spec:

  nodeSelector:

    workload: demo

  tolerations:

    - key: "dedicated"

      operator: "Equal"

      value: "demo"

      effect: "NoSchedule"

  containers:

    - name: app

      image: nginx:1.27

EOF

kubectl get pod tolerant -o wide

Expected result: none of the six plain Pods is placed on the tainted node, while the tolerant Pod runs on it. If the tolerant Pod has no nodeSelector, it could run anywhere, which illustrates that a toleration does not attract. Now see the effect of NoExecute:

# 4. Change to NoExecute: running Pods without tolerations are evicted

kubectl taint nodes $NODE dedicated=demo:NoSchedule-

kubectl taint nodes $NODE dedicated=demo:NoExecute

kubectl get pods -A -o wide --field-selector spec.nodeName=$NODE

# 5. Clean up

kubectl taint nodes $NODE dedicated=demo:NoExecute-

kubectl label node $NODE workload-

kubectl delete deployment plain

kubectl delete pod tolerant

Warning: a NoExecute taint evicts every Pod on the node that does not tolerate it, including application Pods that were already running there. Do this only on a disposable test node, since system DaemonSet Pods are usually protected by their own tolerations but other workloads are not. Expected result: Pods without matching tolerations are terminated and recreated elsewhere by their controllers.

13. Troubleshooting

# Diagnostics

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

kubectl describe pod <pending-pod> | sed -n '/Events:/,$p'

kubectl get pod <pod> -o jsonpath='{.spec.tolerations}'

kubectl get events -A --field-selector reason=TaintManagerEviction

14. Best Practices

• Combine taints with nodeSelector or node affinity when dedicating nodes; use a taint to repel and a label to attract.

• Define taints and labels in node pool or node group configuration so that replaced and autoscaled nodes keep them.

• Use descriptive keys (for example dedicated, workload-class or a company-prefixed key) and document them as part of your platform contract.

• Prefer NoSchedule for reservations, PreferNoSchedule for soft nudges, and use NoExecute deliberately because it evicts running Pods.

• Avoid broad tolerations such as an empty key with Exists in application Pods.

• Set tolerationSeconds for not-ready and unreachable taints where the default five minutes does not suit your failover needs.

• Enforce who may tolerate protected taints with admission policies, and restrict RBAC permissions to modify nodes.

• Add tolerations to DaemonSets for agents that must run everywhere, such as logging, monitoring and networking components.

• Use cordon and drain for planned maintenance rather than ad hoc NoExecute taints.

• Monitor Pending Pods and node taints so that unexpected taints are noticed quickly.

15. Conclusion

Taints and tolerations give platform teams a node-side way to control which Pods may run where. A taint marks a node as restricted, a toleration lets selected Pods ignore that restriction, and the effect (NoSchedule, PreferNoSchedule or NoExecute) decides how strictly it is enforced. They also power Kubernetes' own reaction to unhealthy nodes. Remember the direction of the relationship: tolerations permit, they do not attract, so dedicated nodes need a taint plus a nodeSelector or node affinity. Define taints in node pool configuration, keep tolerations narrow, tune tolerationSeconds for your failover needs, and check node taints first whenever Pods are Pending or disappearing. With these habits, taints and tolerations become a clear and reliable part of your scheduling toolkit.