Kubernetes Topology Spread Constraints Explained

bitcodematrix.com | Kubernetes Series

1. Introduction

A scheduler that only looks at free CPU and memory can place every replica of an application on the same node, or in the same availability zone. If that node or zone fails, the whole application goes down even though you ran several replicas. Topology spread constraints let you tell the scheduler to distribute matching Pods evenly across failure domains such as zones, nodes or regions, while still allowing a controlled amount of imbalance.

This article explains how topology spread constraints work, what skew means, each field of the constraint, how to spread across zones and nodes at the same time, the cluster-level defaults, how they differ from Pod anti-affinity, their limitations during scale-down and rolling updates, and how to troubleshoot Pods that stay Pending. It continues the scheduling series after nodeSelector, node affinity, Pod affinity and taints and tolerations.

After reading it you should be able to:

Explain skew and how maxSkew is evaluated by the scheduler.
Write a topologySpreadConstraints block for zones and nodes.
Choose between DoNotSchedule and ScheduleAnyway.
Decide when to use topology spread instead of Pod anti-affinity.
Understand the limitations and diagnose scheduling failures.

2. The Core Idea

A topology domain is a group of nodes that share the same value for a node label. The label key is called the topologyKey. For example, all nodes with the label topology.kubernetes.io/zone=zone-a form one domain, and all nodes in zone-b form another. When kubernetes.io/hostname is used as the key, every node is its own domain.

Four Pods in two zones; the empty third zone receives the next Pod

Figure 1: The scheduler keeps the Pod count per domain within maxSkew

Term	Meaning
Topology domain	A set of nodes sharing the same value for the topologyKey label, such as one zone.
topologyKey	The node label key that defines the domains, for example topology.kubernetes.io/zone.
Skew	The number of matching Pods in a domain minus the minimum number of matching Pods in any eligible domain.
maxSkew	The largest skew the constraint allows. It must be greater than zero.

3. Understanding Skew

Skew is calculated per domain. Suppose three zones hold 2, 2 and 0 matching Pods. The minimum is 0, so the skew of zone A is 2, the skew of zone B is 2 and the skew of zone C is 0. With maxSkew set to 1, placing the next Pod in zone A or B would make its skew 3, which breaks the limit, while placing it in zone C gives a skew of 1, which is allowed. The scheduler therefore selects zone C.

Zone A	Zone B	Zone C	Result with maxSkew 1
2	2	0	Next Pod must go to zone C
2	2	1	Next Pod must go to zone C (placing it in A or B would give a skew of 2)
3	3	3	Any zone is allowed, because the skew stays at most 1 after placement

Only Pods that match the labelSelector of the constraint are counted. A Pod that carries the labels itself counts as part of the spread, so the selector should normally match the labels of the workload that the constraint is attached to.

4. The Constraint Fields

The constraint is written in the Pod spec, usually inside the Pod template of a Deployment, StatefulSet or Job. You can define more than one constraint.

apiVersion: apps/v1

kind: Deployment

metadata:

name: web

spec:

replicas: 6

selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

topologySpreadConstraints:

- maxSkew: 1

topologyKey: topology.kubernetes.io/zone

whenUnsatisfiable: DoNotSchedule

labelSelector:

matchLabels:

app: web

containers:

- name: web

image: nginx:1.27

resources:

requests:

cpu: 100m

memory: 128Mi

Field	Purpose
maxSkew	Maximum allowed difference in matching Pod count between domains.
topologyKey	Node label key that defines the domains. Nodes without this label are not part of any domain.
whenUnsatisfiable	DoNotSchedule keeps the Pod Pending if the constraint cannot be met. ScheduleAnyway schedules it anyway but prefers placements that reduce skew.
labelSelector	Selects the Pods that are counted. Required for the constraint to be meaningful.
minDomains	Minimum number of eligible domains. If fewer exist, the global minimum is treated as 0. Applies only with DoNotSchedule.
matchLabelKeys	Label keys whose values are taken from the incoming Pod and added to the selector, useful for pod-template-hash during rolling updates.
nodeAffinityPolicy	Whether nodes excluded by the Pod nodeAffinity or nodeSelector are included in skew calculations (Honor or Ignore).
nodeTaintsPolicy	Whether tainted nodes are included in skew calculations (Honor or Ignore).

The fields minDomains, matchLabelKeys, nodeAffinityPolicy and nodeTaintsPolicy were added in later Kubernetes releases and have been promoted through alpha, beta and stable at different versions. Check the documentation for your cluster version before relying on them.

4.1 whenUnsatisfiable: hard and soft spreading

Value	Behavior
DoNotSchedule	A hard rule. The scheduler will not place the Pod if every option would exceed maxSkew. The Pod stays Pending. Use it when an even spread is required for availability.
ScheduleAnyway	A soft rule. The scheduler still places the Pod and gives priority to nodes that minimize the skew. Use it when availability of the Pod matters more than a perfect spread.

4.2 How minDomains works

Without minDomains, a cluster that currently has only one eligible zone can place every Pod in that zone, because the skew is always 0. If you set minDomains: 3 and only two zones exist, the scheduler treats the global minimum as 0, so with maxSkew 1 only one Pod per domain is placed and the rest stay Pending. This is useful together with the Cluster Autoscaler, because the Pending Pods signal that more zones or nodes are needed.

topologySpreadConstraints:

- maxSkew: 1

minDomains: 3

topologyKey: topology.kubernetes.io/zone

whenUnsatisfiable: DoNotSchedule

labelSelector:

matchLabels:

app: web

5. Spreading Across Zones and Nodes

Several constraints can be combined and the Pod must satisfy all of them. A common production pattern is a hard constraint on zones and a soft constraint on nodes: replicas must be balanced across zones, and within a zone they are spread across nodes as far as possible.

topologySpreadConstraints:

- maxSkew: 1

topologyKey: topology.kubernetes.io/zone

whenUnsatisfiable: DoNotSchedule

labelSelector:

matchLabels:

app: web

- maxSkew: 1

topologyKey: kubernetes.io/hostname

whenUnsatisfiable: ScheduleAnyway

labelSelector:

matchLabels:

app: web

Be careful when combining several hard constraints. The scheduler looks for a node that satisfies every one of them, and the combination can leave no valid node, which leaves the Pod Pending.

topologyKey	Typical use
topology.kubernetes.io/zone	Survive the loss of an availability zone.
kubernetes.io/hostname	Avoid placing many replicas on one node.
topology.kubernetes.io/region	Spread across regions in a multi-region cluster.
Custom label such as rack	Spread across racks or other failure domains you label yourself.

6. Cluster-Level Default Constraints

If a Pod defines no topologySpreadConstraints of its own, the scheduler applies default constraints when the Pod belongs to a Service, ReplicaSet, StatefulSet or ReplicationController. The built-in defaults are soft constraints, roughly a maxSkew of 3 on kubernetes.io/hostname and a maxSkew of 5 on topology.kubernetes.io/zone, both with ScheduleAnyway. Cluster administrators can change them in the scheduler configuration (PodTopologySpread plugin arguments, defaultConstraints). Once a Pod defines any constraint of its own, the defaults are not applied to it.

Managed services may configure the scheduler differently and often do not let you edit its configuration, so do not rely on cluster defaults for important workloads. Define the constraints in your manifests.

7. Topology Spread vs Pod Anti-Affinity

Pod anti-affinity can also keep replicas apart, but it is a blunter tool. Required anti-affinity on the hostname key allows at most one matching Pod per node, so a Deployment with more replicas than nodes leaves Pods Pending. Topology spread allows several Pods per domain as long as the skew stays within maxSkew.

Aspect	Topology spread constraints	Pod anti-affinity
Goal	Balance Pods evenly across domains	Keep Pods apart or away from specific Pods
Pods per domain	Several, within maxSkew	At most one for required anti-affinity
More replicas than domains	Works, with controlled imbalance	Required rule leaves extra Pods Pending
Reference to other workloads	Counts Pods that match the labelSelector	Can target other applications
Best for	Even spread of replicas for availability	Isolating one workload from another

They can be used together. For example, use topology spread to balance replicas across zones and a preferred anti-affinity rule to stay away from a noisy neighbor workload.

8. Interaction With Other Scheduling Features

Feature	Interaction
nodeSelector and node affinity	Nodes that are filtered out are excluded from the candidates. Whether they count in the skew calculation depends on nodeAffinityPolicy (default Honor in recent versions, which excludes them).
Taints and tolerations	Tainted nodes that the Pod does not tolerate cannot receive it. Whether they count as domains depends on nodeTaintsPolicy (default Ignore, which includes them in the calculation).
Resource requests	A node must still have enough allocatable resources. A zone with no free capacity can make DoNotSchedule Pods Pending.
Cluster Autoscaler	Pending Pods can trigger scale-up, and the autoscaler is aware of topology spread when simulating placement. Balancing node groups across zones may need the balance-similar-node-groups option.
Pod Disruption Budgets	Spread reduces the chance that one failure removes all replicas, while a PDB limits voluntary disruptions. They complement each other.

9. Limitations

Topology spread constraints are evaluated only when a Pod is scheduled. They do not continuously rebalance a running application.

Scale-down is not topology aware. Removing replicas can leave the spread uneven, because the controller chooses which Pods to delete without considering the constraint.
Rolling updates can skew the spread. The labelSelector matches both old and new Pods unless you add matchLabelKeys with pod-template-hash, so the calculation may include Pods that are about to be removed.
Node failures and drains move Pods to remaining nodes, and they do not move back automatically when the node returns.
Nodes without the topologyKey label are treated as not matching, so Pods with a DoNotSchedule rule cannot be placed on them.

To rebalance a running workload you can restart it (kubectl rollout restart) or use the Kubernetes Descheduler with the RemovePodsViolatingTopologySpreadConstraint strategy, which evicts Pods that break their constraints so that the scheduler can place them again.

10. Hands-On Walkthrough

This walkthrough assumes a cluster with at least three worker nodes labeled with different zones. On a managed service the zone labels are normally set for you. On a local test cluster you can set them manually.

Check the zone labels of your nodes.

kubectl get nodes -L topology.kubernetes.io/zone

If the labels are missing on a test cluster, add them.

kubectl label node worker-1 topology.kubernetes.io/zone=zone-a

kubectl label node worker-2 topology.kubernetes.io/zone=zone-b

kubectl label node worker-3 topology.kubernetes.io/zone=zone-c

Apply the Deployment from section 4 with 6 replicas and check where the Pods run.

kubectl apply -f web.yaml

kubectl get pods -l app=web -o wide

Count Pods per zone. With maxSkew 1 and 6 replicas on 3 zones you should see 2 Pods in each zone.

kubectl get pods -l app=web -o wide --no-headers | awk '{print $7}' | sort | uniq -c

Cordon the nodes of one zone, scale to 9 replicas and see which Pods stay Pending. This shows how DoNotSchedule behaves when a domain cannot accept Pods.

kubectl cordon worker-3

kubectl scale deployment web --replicas=9

kubectl get pods -l app=web

kubectl describe pod <pending-pod>

kubectl uncordon worker-3

Cordoned nodes are not eligible, so the remaining zones can only grow while the skew stays within the limit, and the extra Pods stay Pending until the zone is available again. Note that whether an unschedulable node still counts as a domain depends on the policies described in section 8, so compare what you see with your cluster version documentation.

11. Troubleshooting

Symptom	Likely cause and fix
Pod Pending with "didn't match pod topology spread constraints"	The constraint cannot be met. Check free capacity in each domain, relax maxSkew, use ScheduleAnyway, or add nodes in the missing domain.
All Pods land in one zone	The constraint is missing, the labelSelector does not match the Pod labels, or the cluster has only one zone label. Verify with kubectl get pods --show-labels.
Constraint ignored on some nodes	Those nodes lack the topologyKey label. Check with kubectl get nodes -L <key>.
Uneven spread after scale-down	Expected behavior. Restart the rollout or use the Descheduler.
Uneven spread after a rolling update	Add matchLabelKeys: [pod-template-hash] if your version supports it.
Pending Pods when combining constraints	Several DoNotSchedule rules leave no valid node. Make one of them ScheduleAnyway.
minDomains Pods Pending	Fewer eligible domains exist than minDomains. Add nodes or zones, or lower the value.

kubectl describe pod <pod-name> | grep -A5 Events

kubectl get nodes -L topology.kubernetes.io/zone

kubectl get pods -l app=web -o wide

12. Best Practices

Use a hard zone constraint with maxSkew 1 for workloads that must survive a zone failure, and a soft node constraint for even use of nodes.
Make the labelSelector match the workload labels exactly, and keep the Pod template labels stable.
Use ScheduleAnyway when staying available matters more than a perfect spread, for example during capacity shortages.
Set realistic resource requests, because a perfectly balanced but full zone still leaves Pods Pending.
Keep enough nodes in every zone, and use minDomains with the Cluster Autoscaler where it is supported.
Add matchLabelKeys with pod-template-hash for Deployments where your version supports it.
Plan how to rebalance: a rollout restart or the Descheduler after scale-down or node recovery.
Define constraints in your manifests instead of relying on cluster defaults, and test failure scenarios in a non-production cluster.

13. Conclusion

Topology spread constraints give you a precise way to distribute replicas across zones, nodes and other failure domains. The scheduler keeps the Pod count per domain within maxSkew, and whenUnsatisfiable decides whether the rule is strict or best effort. Compared with Pod anti-affinity they handle more replicas than domains gracefully, and they combine well with node affinity, taints and the Cluster Autoscaler. Remember that they apply only at scheduling time, so plan for rebalancing after scale-down, rolling updates and node failures.

Next in the series, consider reading about Pod Disruption Budgets and priority and preemption, which complete the picture of keeping applications available.