Kubernetes Scheduling Constraints in Production

bitcodematrix.com | Kubernetes Series

1. Introduction

Each Kubernetes scheduling feature is easy to learn on its own: nodeSelector, node affinity, Pod affinity, taints and tolerations, topology spread constraints and priority. Production is harder because these rules interact. A Pod that has a nodeSelector, a required anti-affinity rule and a hard zone constraint may have no valid node at all, and the only symptom is a Pending Pod at an inconvenient moment.

This article is the capstone of the scheduling series. It explains how the scheduler combines constraints, gives a decision guide for choosing the right mechanism, shows four production patterns with YAML, covers how scheduling interacts with volumes, autoscaling and governance, and ends with a FailedScheduling troubleshooting guide and a checklist.

After reading it you should be able to:

Explain how hard and soft constraints combine in the filter and score phases.
Choose the right constraint for a given requirement.
Apply proven patterns for web services, GPU pools, spot capacity and stateful workloads.
Prevent conflicting constraints and enforce them across teams.
Read a FailedScheduling message and find the cause quickly.

2. How the Scheduler Applies Constraints

The kube-scheduler handles each Pod in two main phases. In the filter phase it removes every node that cannot run the Pod. In the score phase it ranks the remaining nodes and picks the best. If the filter phase leaves no node, the Pod stays Pending and, when its priority allows, the scheduler tries preemption.

Hard rules filter nodes, soft rules score them, then the Pod is bound or preempts

Figure 1: Hard rules filter, soft rules score

The key idea for production is to separate requirements from preferences. A requirement must be true or the Pod must not run, and it belongs in a hard rule. A preference improves the outcome but must never block the Pod, and it belongs in a soft rule.

3. The Constraint Toolbox

Mechanism	Direction	Hard or soft	Typical purpose
Resource requests	Fit	Hard	Make sure the node has capacity.
nodeSelector	Pod to node	Hard	Simple match on a node label, such as disk type.
Node affinity	Pod to node	Both	Label matching with operators, plus weighted preferences.
Pod affinity	Pod to Pod	Both	Co-locate Pods, for example a cache next to an app.
Pod anti-affinity	Pod from Pod	Both	Keep replicas or noisy neighbors apart.
Taints and tolerations	Node repels Pod	Hard or soft, by effect	Reserve nodes and protect them from ordinary Pods.
Topology spread	Balance	Both	Spread replicas evenly across zones or nodes.
Priority and preemption	Order and eviction	Order, plus eviction	Decide who wins when capacity is short.

4. Choosing the Right Mechanism

If you need to	Use
Run on nodes with one specific label	nodeSelector
Use several labels, OR logic or soft preferences	Node affinity
Keep ordinary workloads off special nodes	Taint the nodes
Let a specific workload onto tainted nodes	Toleration (it does not attract the Pod, so add node affinity or nodeSelector as well)
Place a Pod near another service	Pod affinity
Spread replicas evenly over zones or nodes	Topology spread constraints
Never run two replicas on one node	Required Pod anti-affinity on kubernetes.io/hostname, or a spread constraint with maxSkew 1
Decide which Pods are evicted when the cluster is full	PriorityClass
Protect against voluntary disruption	PodDisruptionBudget (not a scheduling rule, but part of the design)

A good default for most replicated services is topology spread for balance, node affinity or nodeSelector for hardware needs, and taints only where nodes must be reserved.

5. Production Patterns

5.1 Highly available stateless service

Goal: survive the loss of a node or an availability zone and keep serving traffic. Use a hard zone spread, a soft node spread, a priority class, resource requests and a PodDisruptionBudget.

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 6

selector:

matchLabels:

app: api

template:

metadata:

labels:

app: api

spec:

priorityClassName: business-critical

topologySpreadConstraints:

- maxSkew: 1

topologyKey: topology.kubernetes.io/zone

whenUnsatisfiable: DoNotSchedule

labelSelector:

matchLabels:

app: api

- maxSkew: 1

topologyKey: kubernetes.io/hostname

whenUnsatisfiable: ScheduleAnyway

labelSelector:

matchLabels:

app: api

containers:

- name: api

image: registry.example.com/api:1.0

resources:

requests:

cpu: 250m

memory: 256Mi

limits:

memory: 256Mi

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

minAvailable: 4

selector:

matchLabels:

app: api

5.2 Dedicated GPU node pool

Goal: only GPU workloads run on expensive GPU nodes, and GPU workloads always land there. Taint the nodes to repel everyone else, add a toleration to GPU Pods, and use node affinity to attract them. Requesting the GPU resource also limits placement to nodes that advertise it.

# Node pool setup (done by your node pool settings or kubectl)

kubectl label node gpu-1 workload=gpu

kubectl taint node gpu-1 dedicated=gpu:NoSchedule

# Pod spec

spec:

tolerations:

- key: dedicated

operator: Equal

value: gpu

effect: NoSchedule

affinity:

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: workload

operator: In

values: ["gpu"]

containers:

- name: trainer

image: registry.example.com/ml/trainer:1.0

resources:

limits:

nvidia.com/gpu: 1

5.3 Spot or preemptible capacity for batch work

Goal: run interruptible jobs on cheaper capacity while keeping critical services on stable nodes. Taint the spot nodes, tolerate the taint in batch Pods, prefer spot nodes with a soft affinity, and use a low, non-preempting priority so batch jobs never evict running services.

apiVersion: scheduling.k8s.io/v1

kind: PriorityClass

metadata:

value: 1000

preemptionPolicy: Never

description: "Interruptible batch work"

---

# Pod spec for batch jobs

spec:

priorityClassName: batch

tolerations:

- key: capacity-type

operator: Equal

value: spot

effect: NoSchedule

affinity:

nodeAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 100

preference:

matchExpressions:

- key: capacity-type

operator: In

values: ["spot"]

Batch jobs on spot capacity must tolerate interruption: use retries, checkpoints and sensible termination grace periods.

5.4 Stateful workloads with zonal volumes

Block storage volumes are often tied to one zone. Once a volume exists in a zone, the Pod that uses it can run only in that zone, and the scheduler takes this into account through the volume node affinity. Use a StorageClass with volumeBindingMode: WaitForFirstConsumer so that the volume is created only after the scheduler has chosen a node, which avoids a volume being created in a zone where the Pod cannot run.

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

provisioner: ebs.csi.aws.com

volumeBindingMode: WaitForFirstConsumer

parameters:

type: gp3

For a StatefulSet that should spread across zones, add a zone spread constraint to the Pod template. Each replica gets its own volume in the zone where it is first scheduled. If a zone is later lost, the Pods that depend on its volumes cannot move, so replication at the application level is still required. The provisioner shown is an example for AWS, so use the one for your platform.

6. Combining Constraints Safely

Every hard rule you add shrinks the set of valid nodes. Review each Pod spec as a whole, not one rule at a time.

Combination	Risk and remedy
Required anti-affinity per node plus more replicas than nodes	Extra replicas stay Pending. Use topology spread with maxSkew instead, or make the rule preferred.
Hard zone spread plus a nodeSelector that exists in only some zones	Spread counts only eligible domains, so one zone can become an empty domain. Check nodeAffinityPolicy and label every zone consistently.
Toleration without node affinity	The Pod may run on tainted or untainted nodes. Add affinity to attract it to the dedicated nodes.
Taint without enough tolerating system Pods	DaemonSet Pods such as log or monitoring agents may not run on the nodes. Add tolerations to them.
Required Pod affinity to a workload that is not running	The Pod is Pending until the target exists. Prefer soft affinity unless the dependency is strict.
Several DoNotSchedule spread rules	No node satisfies all of them at once. Keep one hard rule and make the others ScheduleAnyway.
Large requests on small node pools	The Pod cannot fit on any node. Compare requests with node allocatable capacity.

7. Scheduling and Autoscaling

The Cluster Autoscaler reacts to Pending Pods, but only when adding a node of a configured node group would let the Pod schedule. Constraints therefore have to line up with how node groups are defined.

Labels and taints that Pods rely on must be set on the node group template, or the new node will not satisfy the Pod and the scale-up will not help.
For zone spread, make sure node groups exist in every zone and are balanced, using the autoscaler's balance-similar-node-groups option where supported.
Pods stuck for non-capacity reasons, such as a nodeSelector that no node group can satisfy, never trigger a scale-up.
Use negative-priority pause Pods to keep spare capacity so critical Pods start without waiting for a new node.
Managed autoscalers such as Karpenter read constraints differently and may choose instance types directly from them, so check the product documentation.

8. Governance: Enforcing Constraints Across Teams

Constraints written in a document are not enforced. In a shared cluster, use platform controls so that teams get safe defaults and cannot misuse powerful options.

Control	Use
PriorityClass with a global default	Ensures unlabeled Pods land in a sensible tier rather than priority 0.
ResourceQuota with scopeSelector on priority class	Limits how many resources each namespace may use at high priority.
LimitRange	Gives default requests and limits so that scheduling is based on realistic numbers.
Admission policy (ValidatingAdmissionPolicy, Kyverno or OPA Gatekeeper)	Require requests, topology spread on Deployments, or block use of system priority classes.
PodTolerationRestriction admission plugin or policy	Limit or default the tolerations a namespace may use, so that teams cannot tolerate dedicated node taints freely.
Mutation policy	Add default topology spread constraints or tolerations to Pods automatically.

Which of these are available depends on your Kubernetes version and distribution. Managed services may restrict admission plugin configuration, in which case policy engines are the usual choice.

9. Day-2 Operations

Rebalancing: constraints apply only at scheduling time. Use rollout restarts or the Descheduler to repair skew after scale-down, node recovery or upgrades.
Node maintenance: drain nodes one at a time, and make sure PodDisruptionBudgets allow progress while keeping enough replicas available.
Upgrades: surge capacity helps rolling updates meet hard spread and anti-affinity rules while new and old Pods coexist.
Monitoring: alert on Pods Pending for longer than a few minutes, on FailedScheduling events, and on the scheduler's pending Pod and scheduling attempt metrics.
Capacity reviews: compare requested resources against allocatable resources per node pool and zone.

kubectl get pods -A --field-selector=status.phase=Pending

kubectl get events -A --field-selector reason=FailedScheduling

kubectl describe node <node> | grep -A8 "Allocated resources"

10. Reading FailedScheduling Messages

When a Pod cannot be scheduled, its events contain one line that breaks down why each node was rejected, for example: 0/6 nodes are available: 2 node(s) had untolerated taint, 3 Insufficient cpu, 1 node(s) didn't match Pod's node affinity/selector. The numbers add up to the node count, and each phrase maps to one constraint.

Message fragment	Likely cause
Insufficient cpu / Insufficient memory	Requests larger than free capacity. Add nodes, lower requests, or use priority.
node(s) didn't match Pod's node affinity/selector	nodeSelector or node affinity matches no usable node. Check node labels.
node(s) had untolerated taint	Nodes are tainted and the Pod lacks a matching toleration.
node(s) didn't match pod affinity rules or pod anti-affinity rules	A Pod affinity or anti-affinity rule excludes the nodes.
node(s) didn't match pod topology spread constraints	A DoNotSchedule spread rule cannot be satisfied. Check capacity per zone and maxSkew.
node(s) had volume node affinity conflict	The Pod's volume is in a different zone than the allowed nodes.
node(s) were unschedulable	Nodes are cordoned. Uncordon them or add capacity.
Insufficient nvidia.com/gpu (or other extended resource)	No node advertises enough of that resource. Check the device plugin and node pool.

The exact wording of these messages changes between Kubernetes versions, so match on the meaning rather than the precise text.

kubectl describe pod <pod-name> | grep -A12 Events

kubectl get nodes --show-labels

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

kubectl get pod <pod-name> -o yaml | grep -A40 "^spec:"

11. Production Checklist

Check	Why it matters
Every Pod has CPU and memory requests	The scheduler places Pods by requests.
Hard rules are requirements, soft rules are preferences	Prevents unnecessary Pending Pods.
Replicated services spread across zones	Survives the loss of a zone.
Dedicated nodes use taint, toleration and affinity together	Keeps others out and the right Pods in.
Priority classes are few, documented and restricted	Avoids priority abuse and surprise evictions.
PodDisruptionBudgets exist for critical services	Limits voluntary disruption during drains.
Node group labels and taints match what Pods require	Lets the autoscaler help.
Policies enforce defaults across namespaces	Constraints are applied consistently.
Pending Pod and FailedScheduling alerts exist	Problems are seen before users notice.
A rebalancing process exists	Fixes drift that scheduling alone cannot.

12. Best Practices

Start simple: requests, a zone spread for replicated services and PDBs cover most needs.
Add taints and affinity only where nodes are genuinely special, and label node pools clearly.
Prefer soft rules where the workload can run without the preference.
Review the full Pod spec for hard-rule conflicts before rollout, and test failure cases such as losing a zone in a staging cluster.
Keep capacity headroom in every zone, because even perfect constraints cannot place Pods on a full cluster.
Enforce standards with policy rather than documentation.
Document the scheduling model for application teams, including which labels, taints and priority classes exist and what they mean.

13. Conclusion

Production scheduling is about combining a small number of well-understood tools. Requests make placement realistic, node affinity and taints control where workloads may run, topology spread keeps replicas apart, priority resolves shortages, and PodDisruptionBudgets, autoscaling and policy keep the system healthy over time. The most important habit is to treat hard rules as scarce: use them for real requirements and use soft rules for preferences. With that discipline, and with alerts on Pending Pods, most scheduling problems become short, predictable fixes instead of incidents.

This article completes the scheduling series. Together with the earlier articles on nodeSelector, node affinity, taints and tolerations, topology spread constraints and priority, it gives a full picture of how Kubernetes decides where Pods run.