Kubernetes Node Affinity and Pod Affinity Explained

bitcodematrix.com | Kubernetes Series

1. Introduction

The nodeSelector, covered in the previous article, lets you say "run this Pod only on nodes with this label". That is enough for simple needs, but real clusters ask for more. You may want a Pod to prefer SSD nodes but still run elsewhere if none is free. You may want replicas of a web service spread across different nodes and zones so that one failure does not take them all down. You may want an application to run on the same node, or in the same zone, as its cache to cut network latency. Kubernetes answers these needs with affinity and anti-affinity.

This article explains node affinity (rules based on node labels), Pod affinity and Pod anti-affinity (rules based on where other Pods run), the difference between required and preferred rules, how the topologyKey field works, how to write each with YAML, how they compare with topology spread constraints, and how to troubleshoot Pods that stay Pending because of them.

After reading it you should be able to:

Write node affinity rules with required and preferred terms and the available operators.
Use Pod affinity to co-locate Pods and Pod anti-affinity to spread them.
Explain topologyKey, labelSelector and namespaces in Pod affinity terms.
Choose between nodeSelector, node affinity, Pod affinity, anti-affinity and topology spread.
Diagnose Pending Pods caused by affinity rules and avoid common pitfalls.

2. Overview of Affinity

Node affinity, Pod affinity and Pod anti-affinity compared

Figure 1: Node affinity, Pod affinity and Pod anti-affinity

All affinity rules live under spec.affinity in the Pod specification (in the Pod template for Deployments and other controllers). There are three groups:

Type	Field	What it matches
Node affinity	spec.affinity.nodeAffinity	Labels on nodes
Pod affinity	spec.affinity.podAffinity	Labels on other Pods already running in a topology domain
Pod anti-affinity	spec.affinity.podAntiAffinity	Labels on other Pods, to avoid their topology domain

2.1 Required and preferred rules

Each type supports two kinds of rules, distinguished by long but descriptive field names.

Field	Behaviour
requiredDuringSchedulingIgnoredDuringExecution	Hard rule. A node that does not satisfy it is filtered out. If no node qualifies, the Pod stays Pending.
preferredDuringSchedulingIgnoredDuringExecution	Soft rule. Nodes that satisfy it receive extra score according to the weight (1 to 100), but the Pod can still run elsewhere.

The "IgnoredDuringExecution" part of the names means that the rules are checked only when the Pod is scheduled. If labels change later, or if the Pods that satisfied a Pod affinity rule leave, already running Pods are not evicted or moved. A stricter variant that would evict Pods is described in the Kubernetes design but is not available at the time of writing, so do not rely on it. Tools such as the descheduler can evict Pods that violate their rules so that they are placed again.

3. Node Affinity

Node affinity is a more expressive replacement for nodeSelector. It matches node labels with operators and supports preferences.

3.1 Structure

apiVersion: v1

kind: Pod

metadata:

spec:

affinity:

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: kubernetes.io/arch

operator: In

values: ["amd64", "arm64"]

- key: spot

operator: DoesNotExist

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 80

preference:

matchExpressions:

- key: disktype

operator: In

values: ["ssd"]

- weight: 20

preference:

matchExpressions:

- key: topology.kubernetes.io/zone

operator: In

values: ["zone-a"]

containers:

- name: app

image: nginx:1.27

In this example the node must be amd64 or arm64 and must not have a label named spot. Among those nodes, the scheduler adds 80 points for disktype=ssd and 20 points for zone-a. Zone names are placeholders that depend on your provider.

3.2 How terms combine

Several matchExpressions inside one nodeSelectorTerm are combined with AND.
Several nodeSelectorTerms in the list are combined with OR: it is enough that one term matches.
Several preferred entries each add their weight when matched; the node with the highest total score wins, together with other scoring plugins.
If a Pod has both a nodeSelector and node affinity, a node must satisfy both.

3.3 Operators

Operator	Meaning
In	The label value is one of the listed values
NotIn	The label value is not one of the listed values (nodes without the label also match)
Exists	The node has the label key, with any value
DoesNotExist	The node does not have the label key
Gt, Lt	The label value, read as an integer, is greater than or less than the single value given

The NotIn and DoesNotExist operators are the basis of node anti-affinity: you can keep Pods away from nodes with certain labels. There is also matchFields, which matches fields of the node object such as metadata.name, but labels are the usual and more portable choice.

4. Pod Affinity and Pod Anti-Affinity

Pod affinity rules are about relationships between Pods. Instead of asking what labels a node has, they ask which Pods are already running in the same topology domain as the node, and use the labels of those Pods.

4.1 The anatomy of a Pod affinity term

Field	Meaning
labelSelector	Selects the other Pods whose presence (or absence) matters, by their labels
topologyKey	A node label key that defines the "domain" in which Pods count as together; nodes with the same value for this key form one domain
namespaces / namespaceSelector	Which namespaces to look in for the selected Pods; by default only the namespace of the Pod being scheduled
weight (preferred rules only)	Score from 1 to 100 if the term is satisfied

4.2 Understanding topologyKey

The topologyKey decides how broadly "together" is defined. It must be a label key present on the nodes.

topologyKey	Domain
kubernetes.io/hostname	Each node is its own domain: "same node"
topology.kubernetes.io/zone	All nodes in the same availability zone form one domain: "same zone"
topology.kubernetes.io/region	All nodes in the same region
A custom label such as example.com/rack	All nodes with the same value of that label, for example a rack or a pool

Nodes that do not have the topologyKey label are not part of any domain and cannot satisfy the rule, which is a common source of unexpected Pending Pods.

4.3 Pod affinity: co-locate Pods

Use Pod affinity when two components benefit from being close, for example a web application and the cache it calls often.

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 3

selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

affinity:

podAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 100

podAffinityTerm:

topologyKey: topology.kubernetes.io/zone

labelSelector:

matchLabels:

app: cache

containers:

- name: app

image: nginx:1.27

This asks the scheduler to prefer nodes in a zone where Pods labelled app=cache already run. Using a preferred rule keeps the workload schedulable if the cache is not yet present or its zone is full. A required Pod affinity rule would leave the Pods Pending until a matching Pod exists in a suitable domain. The scheduler has a special case that lets the very first Pod of a group be placed when the rule refers to its own labels, so that self-affinity does not deadlock.

4.4 Pod anti-affinity: spread Pods

Anti-affinity is most often used to keep replicas of one service apart, so that a node or zone failure does not remove all of them.

spec:

affinity:

podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

- topologyKey: kubernetes.io/hostname

labelSelector:

matchLabels:

app: web

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 100

podAffinityTerm:

topologyKey: topology.kubernetes.io/zone

labelSelector:

matchLabels:

app: web

This example says: never place two app=web Pods on the same node (hard rule), and prefer to place them in different zones (soft rule). The hard rule has a consequence: if the Deployment has more replicas than there are eligible nodes, the extra replicas stay Pending. Use required anti-affinity only when it is truly necessary, and make sure that the cluster has enough nodes, or use the preferred form.

4.5 Namespaces

By default a Pod affinity term considers only Pods in the same namespace as the Pod being scheduled. To refer to Pods in other namespaces, list them in namespaces or select them with namespaceSelector. An empty namespaceSelector ({}) matches all namespaces.

podAffinityTerm:

topologyKey: kubernetes.io/hostname

labelSelector:

matchLabels:

app: cache

namespaceSelector:

matchLabels:

team: platform

4.6 Rolling updates and matchLabelKeys

During a rolling update, old and new Pods of a Deployment coexist and carry the same app label, so a required anti-affinity rule against app=web can make new Pods wait for old ones to disappear. Newer Kubernetes versions add matchLabelKeys and mismatchLabelKeys to Pod affinity terms, which let a rule consider the pod-template-hash so that each revision is handled separately. These fields are version dependent, so check the documentation for your release before using them.

5. Common Use Cases

Goal	Mechanism	Notes
Run on nodes with a certain capability, with a fallback	Node affinity (preferred)	Replaces a rigid nodeSelector
Run only on one of several allowed zones	Node affinity (required, In)	nodeSelector cannot express OR
Keep Pods off spot or special nodes	Node affinity (NotIn or DoesNotExist)	Or use taints for stronger repulsion
Place an app next to its cache or database	Pod affinity (preferred, zone or hostname)	Reduces latency and cross-zone traffic cost
Spread replicas across nodes	Pod anti-affinity (hostname)	Prefer soft unless you have enough nodes
Spread replicas across zones	Pod anti-affinity (zone) or topology spread constraints	Spread constraints scale better and give even distribution
Keep noisy or sensitive workloads apart	Pod anti-affinity between their labels	Alternatively use dedicated nodes with taints
Co-schedule Pods that must share a node-local resource	Pod affinity (required, hostname)	The matching Pod must exist first

6. Comparison with Other Placement Tools

Tool	Looks at	Main strength	Main limitation
nodeSelector	Node labels (equality)	Simplest	No preferences or OR
Node affinity	Node labels (operators)	Expressive, soft rules	More verbose
Pod affinity	Labels of other Pods	Co-location	Computationally heavier; depends on existing Pods
Pod anti-affinity	Labels of other Pods	Spreading	Hard rules can cause Pending; heavy on large clusters
Topology spread constraints	Distribution of matching Pods across domains	Even spread with maxSkew; scales better	Does not express co-location
Taints and tolerations	Node-side repulsion	Reserve nodes for specific Pods	Does not attract Pods by itself

Kubernetes documentation notes that inter-Pod affinity and anti-affinity require a significant amount of processing and are not recommended in clusters of several hundred nodes. For simply spreading replicas across zones and nodes, topology spread constraints are usually the better choice, as described in the article on resource management and scheduling. Use Pod anti-affinity when you need specific "never together" relationships between different workloads.

7. How the Scheduler Evaluates Affinity

Filtering: required node affinity, required Pod affinity and required Pod anti-affinity remove nodes that do not satisfy them, together with the checks for resources, taints and volumes.
Scoring: preferred rules add weighted scores to the nodes that satisfy them. The final choice combines these with other scoring plugins, such as resource balance and image locality.
Binding: the Pod is bound to the best node. From then on, the rules are not re-evaluated for this Pod.

Pod affinity rules are symmetric in one important way: when you schedule a new Pod, the scheduler also checks the required anti-affinity rules of Pods that are already running. If an existing Pod says "no app=web Pod in my zone", a new web Pod cannot enter that zone, even though the new Pod itself has no rule. This can surprise teams who look only at the manifest of the Pending Pod.

8. Interaction with Other Features

Feature	Interaction
Cluster Autoscaler or node provisioners	Autoscalers simulate new nodes using node group labels. Node affinity on labels that the node group provides can trigger scale-up. Pod anti-affinity is considered too, but it can make scale-up less efficient because each replica may require its own node.
Resource requests	Affinity only filters and scores; the Pod must still fit on the chosen node
Taints and tolerations	Evaluated in addition; a Pod needs a toleration for tainted nodes even if affinity attracts it
Persistent volumes	Zonal volumes restrict nodes; conflicting zone rules cause Pending
Pod Disruption Budgets	Spread rules improve the chances that a PDB can be honoured during drains
DaemonSets	DaemonSet Pods use node affinity internally to target specific nodes; you can add your own node affinity to limit which nodes run the agent
Rolling updates	Required anti-affinity can slow rollouts; consider surge settings and matchLabelKeys where available

9. Hands-On Walkthrough

This exercise demonstrates anti-affinity spreading and the effect of a hard rule when there are not enough nodes. Use a test cluster with at least two nodes and no taints that block ordinary Pods. Replace the replica numbers if your cluster has a different node count.

Note the number of nodes.
Deploy a Deployment with soft anti-affinity and observe spreading.
Switch to a hard rule with more replicas than nodes and observe Pending Pods.
Read the scheduler event to see how the rule is reported.

kubectl get nodes

# 1. Soft anti-affinity: replicas spread across nodes where possible

cat <<EOF | kubectl apply -f -

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 4

selector:

matchLabels:

app: spread-demo

template:

metadata:

labels:

app: spread-demo

spec:

affinity:

podAntiAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 100

podAffinityTerm:

topologyKey: kubernetes.io/hostname

labelSelector:

matchLabels:

app: spread-demo

containers:

- name: app

image: nginx:1.27

resources:

requests:

cpu: 50m

memory: 32Mi

EOF

kubectl get pods -l app=spread-demo -o wide

Expected result: the four Pods are distributed across the available nodes as evenly as the scores allow, and all of them run. Now make the rule hard:

# 2. Change to a required rule and scale beyond the node count

kubectl patch deployment spread-demo --type=json -p='[

{"op":"remove","path":"/spec/template/spec/affinity/podAntiAffinity/preferredDuringSchedulingIgnoredDuringExecution"},

{"op":"add","path":"/spec/template/spec/affinity/podAntiAffinity/requiredDuringSchedulingIgnoredDuringExecution","value":[{"topologyKey":"kubernetes.io/hostname","labelSelector":{"matchLabels":{"app":"spread-demo"}}}]}

kubectl scale deployment spread-demo --replicas=10

kubectl get pods -l app=spread-demo

kubectl describe pod <pending-pod> | sed -n '/Events:/,$p'

# Clean up

kubectl delete deployment spread-demo

Expected result (illustrative): at most one Pod per node is Running and the remaining Pods are Pending with an event similar to "0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules". The exact wording varies by version. This shows why required anti-affinity needs enough nodes, or a preferred rule or a topology spread constraint instead.

10. Troubleshooting

Symptom	Likely cause	What to check or do
Pending: node(s) didn't match Pod's node affinity/selector	No node satisfies the required node affinity or nodeSelector	kubectl get nodes --show-labels; fix labels or relax the rule
Pending: didn't match pod affinity rules	Required Pod affinity but no matching Pod in a suitable domain, or wrong namespace	Check labelSelector, namespaces and that the target Pods are running; consider a preferred rule
Pending: didn't match pod anti-affinity rules	Hard anti-affinity with more replicas than domains, or an existing Pod's rule excludes the node	Add nodes, use preferred rules, or use topology spread constraints
Rule seems ignored	Rule is preferred, weight too low, or placed in the wrong part of the manifest	Check required vs preferred; put affinity under spec.template.spec for controllers
No node qualifies although labels exist	topologyKey label missing on nodes, or typo in key or value	Compare topologyKey with actual node labels
Rolling update stalls	Required anti-affinity matches old and new Pods of the same Deployment	Use preferred rules, adjust maxSurge, or matchLabelKeys where available
Pods did not move after labels changed	Rules are IgnoredDuringExecution	Recreate Pods or use the descheduler
Scheduling is slow on a large cluster	Heavy Pod affinity or anti-affinity evaluation	Prefer topology spread constraints; reduce the number of rules
Autoscaler adds many nodes	Each replica requires its own node because of hard anti-affinity	Relax the rule, set realistic maximums, check cost

# Useful diagnostics

kubectl get nodes --show-labels

kubectl get nodes -L topology.kubernetes.io/zone,kubernetes.io/hostname

kubectl get pods -A -l app=cache -o wide

kubectl describe pod <pending-pod> | sed -n '/Events:/,$p'

11. Best Practices

Start with the simplest tool that works: nodeSelector for one hard label, node affinity for preferences and alternatives.
Prefer soft (preferred) rules unless a hard rule is a real requirement, to avoid unschedulable Pods.
Use topology spread constraints rather than Pod anti-affinity for generic spreading across zones and nodes, especially in large clusters.
Choose the topologyKey carefully and make sure all nodes carry that label.
Use meaningful, consistent labels on Pods and nodes, and document them.
For required anti-affinity, make sure the number of eligible domains is at least the number of replicas, including during rolling updates.
Combine affinity with taints and tolerations when nodes must be reserved, since affinity alone does not keep other Pods away.
Remember the symmetry: existing Pods' required anti-affinity rules can block new Pods.
Test rules in staging with a realistic node count, and watch for Pending Pods after changes.
Keep the number and complexity of rules small; complex rule sets are hard to reason about and expensive to evaluate.

12. Conclusion

Node affinity and Pod affinity give Kubernetes users fine control over placement. Node affinity extends the nodeSelector with operators, alternatives and weighted preferences, all based on node labels. Pod affinity and anti-affinity add relationships between workloads, using topologyKey to define whether "together" means the same node, zone or any other domain you label. The key decisions are simple: choose required rules only when a placement is mandatory, prefer soft rules for everything else, make sure there is enough capacity in the domains you require, and switch to topology spread constraints when your real goal is even distribution. With these guidelines, affinity rules improve performance and availability without turning into a source of Pending Pods.