Kubernetes Pod Priority and Preemption Explained (Guide)
Learn Kubernetes Pod priority and preemption: PriorityClass, preemptionPolicy, eviction order, system-critical classes, YAML examples and troubleshooting.
Kubernetes Pod Priority and Preemption Explained
bitcodematrix.com | Kubernetes Series
1. Introduction
When a cluster runs out of capacity, the scheduler has to decide which Pods matter most. By default every Pod is equal: a Pod that cannot fit simply stays Pending, even when less important workloads occupy the space it needs. Pod priority and preemption change this. You give Pods a priority, the scheduler places higher-priority Pods first, and when no node has room it can evict lower-priority Pods to make space.
This article explains PriorityClass objects, how priority orders the scheduling queue, how preemption chooses victims, how to create a non-preempting class, the built-in system classes, how priority differs from QoS classes and node-pressure eviction, how it works with the Cluster Autoscaler, and how to troubleshoot unexpected evictions. It continues the scheduling series after nodeSelector, node affinity, taints and tolerations and topology spread constraints.
After reading it you should be able to:
Create PriorityClass objects and assign them to Pods.
Explain how preemption selects a node and the victim Pods.
Use preemptionPolicy: Never for workloads that should not evict others.
Separate the roles of priority, QoS class and PodDisruptionBudgets.
Diagnose why a Pod was preempted or why preemption did not happen.
2. The Core Idea
Priority is an integer. A higher number means a more important Pod. It has two effects. First, the scheduler sorts the queue of pending Pods by priority, so important Pods are considered first. Second, if a pending Pod cannot be scheduled, the scheduler may try to free room by preempting (evicting) Pods with a lower priority.
Figure 1: Preemption frees room by evicting lower-priority Pods
3. Creating a PriorityClass
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 100000
globalDefault: false
description: "Business-critical services"
A Pod that names no class and has no global default gets a priority of 0. Negative values are allowed and are useful for Pods that should always be the first to give way.
3.1 Using the class in a Pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments
spec:
replicas: 3
selector:
matchLabels:
app: payments
template:
metadata:
labels:
app: payments
spec:
priorityClassName: high-priority
containers:
- name: payments
image: nginx:1.27
resources:
requests:
cpu: 250m
memory: 256Mi
kubectl get priorityclass
kubectl get pod <pod-name> -o jsonpath='{.spec.priority}{"\n"}'
Priority is resolved when the Pod is created. Changing a PriorityClass value later does not update the priority of existing Pods, and a PriorityClass that is in use cannot be assumed to be safely deletable, so plan a small set of classes and keep them stable.
4. How Preemption Works
Preemption is triggered by the scheduler when a Pod is unschedulable and has a preemptionPolicy of PreemptLowerPriority.
The Pod is pending because no node passes the filters, for example because of insufficient CPU or memory.
The scheduler looks for nodes where removing one or more lower-priority Pods would let the Pod fit, taking all other scheduling rules into account.
It selects the best node, preferring the one with the fewest victims, the least PodDisruptionBudget violations and the lowest-priority victims. Exact tie-breaking rules are internal and may change between versions.
The victims are deleted and shut down gracefully, honoring their terminationGracePeriodSeconds.
The Pod gets status.nominatedNodeName set to the chosen node and waits in the queue until the room is free, then it is scheduled.
The nominated node is a hint, not a reservation. While the victims are terminating, another Pod of higher priority may take the space, or the scheduler may find a better node, so the Pod can end up on a different node. The scheduler never preempts Pods of equal or higher priority.
kubectl get pod <pending-pod> -o jsonpath='{.status.nominatedNodeName}{"\n"}'
kubectl get events --field-selector reason=Preempted
4.1 What preemption cannot fix
Pods that are unschedulable for reasons unrelated to other Pods, such as a nodeSelector that matches no node or a taint they do not tolerate.
Cases where evicting lower-priority Pods would still not create enough room.
Required Pod affinity to lower-priority Pods: removing the Pods it needs would break the rule, so preemption is not used for that purpose.
4.2 PodDisruptionBudgets
The scheduler tries to respect PodDisruptionBudgets when it picks victims, but this is best effort. If no node can be freed without violating a budget, preemption may still go ahead and violate it. A PDB is therefore not a guarantee against preemption.
5. Non-Preempting Priority
Some workloads should be scheduled ahead of others in the queue but should not kill running Pods, for example data science batch jobs. Set preemptionPolicy: Never on the class.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority-nonpreempting
value: 100000
preemptionPolicy: Never
description: "Queue ahead of others without evicting running Pods"
A Pod of this class is placed ahead of lower-priority Pods in the scheduling queue and can be scheduled as soon as capacity appears, but it never triggers an eviction. It can still be preempted itself by a higher-priority Pod that does use PreemptLowerPriority.
6. Built-in System Priority Classes
Kubernetes ships two classes for essential components.
Do not use these for application Pods. By default, the system classes are not limited to the kube-system namespace in recent Kubernetes versions, so cluster administrators who want to restrict them should use a ResourceQuota with a scopeSelector on priority class names, or an admission policy.
7. Priority vs QoS Class vs Node-Pressure Eviction
Priority is easy to confuse with the QoS class (Guaranteed, Burstable, BestEffort) and with kubelet eviction. They are separate mechanisms.
The QoS class is calculated from the requests and limits of the containers and has no effect on scheduler preemption. A BestEffort Pod with a high priority is not protected from kubelet eviction in the same way that a Guaranteed Pod is, because the kubelet looks at usage versus requests first. See the earlier articles on QoS classes and OOMKilled for details.
8. Priority and the Cluster Autoscaler
Priority interacts with autoscaling in two useful ways. First, the Cluster Autoscaler can ignore Pods below a configured priority cutoff when deciding to scale up. In the default configuration, Pods with a priority below -10 are treated as expendable and do not trigger new nodes, and they do not stop nodes from being removed. Second, this enables the overprovisioning pattern.
Create a PriorityClass with a negative value, for example -10, or lower than any real workload.
Run a Deployment of "pause" Pods with that class and with resource requests that reserve the spare capacity you want.
When real workloads arrive, they preempt the pause Pods immediately and start without waiting for a new node.
The evicted pause Pods become Pending and trigger the autoscaler (if their priority is above its cutoff) or simply wait, so the buffer is rebuilt.
The cutoff value and the exact behavior depend on the autoscaler and its configuration, so choose the buffer priority carefully: it must be lower than every real workload, but above the cutoff if you want the buffer to be replaced automatically. Karpenter and managed autoscalers may handle this differently.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: overprovisioning
value: -1
globalDefault: false
description: "Placeholder Pods that real workloads can preempt"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: overprovisioning
spec:
replicas: 2
selector:
matchLabels:
app: overprovisioning
template:
metadata:
labels:
app: overprovisioning
spec:
priorityClassName: overprovisioning
containers:
- name: pause
image: registry.k8s.io/pause:3.9
resources:
requests:
cpu: "1"
memory: 1Gi
9. Designing a Priority Scheme
Keep the number of classes small, usually four to six, and document each one in the description field.
Set a global default of the standard tier so that unlabeled Pods do not default to 0 next to batch jobs.
Leave room between values so you can add classes later.
Control who may use high classes with ResourceQuota scopeSelector or an admission policy, otherwise any user can claim the top priority.
10. Risks and Limitations
Priority abuse: users who can pick any class will pick the highest. Restrict access.
Disruption of lower tiers: preemption evicts running Pods, which can lose in-flight work. Make batch jobs checkpoint or retry.
Preemption chains: evicted Pods may preempt even lower Pods or trigger scale-ups, which causes churn.
Starvation: a steady stream of high-priority Pods can keep low-priority Pods Pending indefinitely.
Graceful termination delay: the preemptor waits for victims to terminate, so long termination grace periods delay scheduling.
No guarantee on the nominated node: the Pod may be placed elsewhere.
11. Hands-On Walkthrough
This walkthrough uses a small test cluster. Use a non-production cluster because it deliberately evicts Pods.
Create two PriorityClasses, low (value 1000) and high (value 100000).
kubectl apply -f - <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: low
value: 1000
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high
value: 100000
EOF
Check how much allocatable CPU a worker node has, then run a low-priority Deployment whose total requests fill most of it.
kubectl describe node <node> | grep -A6 Allocatable
kubectl create deployment filler --image=nginx:1.27 --replicas=4
kubectl set resources deployment filler --requests=cpu=500m
kubectl patch deployment filler -p '{"spec":{"template":{"spec":{"priorityClassName":"low"}}}}'
Scale the filler until the node is nearly full, using the CPU numbers from your own cluster.
Create a high-priority Pod with a CPU request larger than the remaining space.
kubectl run important --image=nginx:1.27 --overrides='{"spec":{"priorityClassName":"high","containers":[{"name":"important","image":"nginx:1.27","resources":{"requests":{"cpu":"1500m"}}}]}}'
Watch what happens: some filler Pods are terminated and the important Pod is scheduled.
kubectl get pods -o wide -w
kubectl get pod important -o jsonpath='{.status.nominatedNodeName}{"\n"}'
kubectl get events --field-selector reason=Preempted
The Deployment controller recreates the evicted filler Pods, but they stay Pending because the node is now full. This shows the full cycle: preempt, schedule, replace and wait. Adjust the CPU values to match your node size, and delete the test objects afterwards.
12. Troubleshooting
kubectl get priorityclass
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIO:.spec.priority,CLASS:.spec.priorityClassName
kubectl describe pod <pod-name> | grep -A8 Events
13. Best Practices
Define a small, documented set of PriorityClasses and set a global default for standard workloads.
Reserve the system classes for cluster components, and restrict high application classes to trusted namespaces.
Use preemptionPolicy: Never for batch workloads that must wait politely but should not kill running Pods.
Always set resource requests, because the scheduler and preemption decisions are based on them.
Combine priority with PodDisruptionBudgets, but do not treat PDBs as a guarantee during preemption.
Make low-priority workloads tolerant to eviction: retries, checkpoints and sensible termination grace periods.
Use negative-priority pause Pods for fast scale-out headroom where the autoscaler supports it.
Monitor Preempted events and Pending Pods so that preemption does not hide a real capacity shortage.
14. Conclusion
Pod priority tells the scheduler what matters most when capacity is short, and preemption lets important Pods claim room from less important ones. PriorityClass keeps the rules simple, preemptionPolicy: Never provides a polite option for batch work, and the system classes protect core components. Remember that priority works at scheduling time and is separate from QoS classes and kubelet eviction, that PDBs are only best effort, and that the best defense against constant preemption is enough capacity. Used with requests and limits, taints, affinity, topology spread and autoscaling, priority completes a dependable scheduling strategy.