Kubernetes Pod Pending: Scheduler Troubleshooting Guide
Pod stuck in Pending? Learn how to find the cause in Kubernetes: FailedScheduling, resources, taints, PVCs, image pulls, with kubectl commands and fixes.
Kubernetes Scheduler Troubleshooting: Why Is My Pod Pending?
bitcodematrix.com | Kubernetes Series
1. Introduction
A Pod in the Pending state is one of the most common problems in Kubernetes, and one of the most confusing, because the word Pending covers several different situations. Sometimes the scheduler cannot find a node. Sometimes a node was chosen but the kubelet is still pulling an image or waiting for a volume. Sometimes nothing has looked at the Pod at all.
This article gives a repeatable method for finding the cause. It starts with the three commands that answer most cases, splits the problem into scheduler-side and kubelet-side causes, explains each common cause with its fix, covers Pods that are missing because a controller could not create them, and ends with a hands-on exercise and a command cheat sheet. It builds on the earlier articles in the scheduling series, and in particular on Scheduling Constraints in Production.
After reading it you should be able to:
Explain what the Pending phase means and what it does not mean.
Tell scheduler-side problems from kubelet-side problems in under a minute.
Read FailedScheduling events and fix resource, taint, selector, affinity, spread and volume causes.
Diagnose a scheduler that is not working, and Pods that never appear.
Follow a checklist and the right kubectl commands for every case.
2. What Pending Really Means
The Pod phase Pending means that the cluster has accepted the Pod but one or more of its containers is not running yet. This includes the time spent waiting to be scheduled and the time spent downloading images and setting up the Pod on the node. It does not by itself say that something is wrong: a Pod is Pending for a few seconds during every normal startup.
The most useful first question is whether the Pod has been assigned to a node, because that decides which component to investigate.
Figure 1: The NODE column splits the problem in two
3. The First Three Commands
See the Pod status and node assignment.
kubectl get pod <pod-name> -o wide
Read the events at the bottom of the description. This is where the reason almost always appears.
kubectl describe pod <pod-name>
If the events are long or have expired, list events for the Pod in time order.
kubectl get events --sort-by=.lastTimestamp --field-selector involvedObject.name=<pod-name>
Events are kept for a limited time, commonly about an hour, so check them early. If the Pod is old and has no events left, delete and recreate it, or look at the scheduler and kubelet logs.
4. Scheduler-Side Causes (No Node Assigned)
4.1 No events at all
If a Pod has no node and no FailedScheduling event, the scheduler is probably not looking at it.
kubectl get pod <pod-name> -o jsonpath='{.spec.schedulerName}{"\n"}'
kubectl get pod <pod-name> -o jsonpath='{.spec.schedulingGates}{"\n"}'
kubectl get pods -n kube-system | grep scheduler
kubectl logs -n kube-system <kube-scheduler-pod> --tail=50
4.2 FailedScheduling: reading the message
When the scheduler tries and fails, it adds a FailedScheduling event that explains why each node was rejected. The numbers add up to the number of nodes, and a node can appear under only one reason, the first one that rejected it.
Warning FailedScheduling default-scheduler
0/6 nodes are available: 1 node(s) were unschedulable,
2 node(s) had untolerated taint {dedicated: gpu},
3 Insufficient cpu.
The wording of these messages changes between Kubernetes versions, so match on the meaning rather than the exact text.
4.3 Checking resource capacity
Insufficient cpu or memory is based on requests, not on actual usage. A node can look idle in monitoring and still be full, because the scheduler adds up the requests of the Pods already placed there. Compare the allocatable amount with what is already requested.
kubectl describe node <node-name> | grep -A8 "Allocatable"
kubectl describe node <node-name> | grep -A12 "Allocated resources"
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources.requests}{"\n"}'
kubectl top nodes
A Pod's effective request is the sum of its app containers, or the largest init container if that is higher, plus any Pod overhead from its RuntimeClass.
A request larger than the allocatable capacity of the biggest node can never be scheduled, however much you scale down other workloads.
A LimitRange may add default requests that you did not write in the manifest. Check kubectl get limitrange -n <namespace>.
System Pods and DaemonSets also use requests on every node, which lowers what is left for applications.
4.4 PersistentVolumeClaims that are Pending
A Pod that uses a PersistentVolumeClaim cannot start until the claim is bound. If the claim is Pending, the Pod stays Pending as well, and the real cause is on the claim.
kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>
kubectl get storageclass
4.5 Priority, preemption and nominated nodes
A high-priority Pod that cannot be placed may trigger preemption. While victims shut down, the Pod stays Pending and shows a nominated node. This is normal for a short time, as long as it eventually starts.
kubectl get pod <pod-name> -o jsonpath='{.status.nominatedNodeName}{"\n"}'
kubectl get events --field-selector reason=Preempted
5. Kubelet-Side Causes (Node Assigned)
When the NODE column has a value, the scheduler has done its job. The Pod may still show Pending in the phase, but the STATUS column and the events usually name the real state.
kubectl describe pod <pod-name> | grep -A15 Events
kubectl describe node <node-name> | grep -A10 Conditions
kubectl get events -A --field-selector reason=FailedMount
CrashLoopBackOff, which starts after a container has run and failed, is a different problem from Pending.
6. The Pod Does Not Exist: Controller and Admission Failures
Sometimes a Deployment shows fewer Pods than requested and no Pending Pod exists. In this case the Pod was rejected before it was created, and the explanation is on the ReplicaSet or the controller, not on a Pod.
kubectl get deployment <name>
kubectl describe replicaset -l app=<label>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
7. Hands-On Walkthrough
This exercise creates a Pod that cannot be scheduled for two different reasons and fixes them one at a time. Use a test cluster.
Create a Pod with a very large memory request and a nodeSelector that no node has.
apiVersion: v1
kind: Pod
metadata:
name: pending-demo
spec:
nodeSelector:
disktype: nvme
containers:
- name: app
image: nginx:1.27
resources:
requests:
cpu: 100m
memory: 500Gi
Apply it and check the status.
kubectl apply -f pending-demo.yaml
kubectl get pod pending-demo -o wide
Read the events. You should see the Pod is not placed and the message lists the nodes that failed the selector.
kubectl describe pod pending-demo | grep -A8 Events
Fix the first problem by labeling a node, or by removing the selector, then recreate the Pod. Check the events again: now the reason should be insufficient memory.
kubectl label node <node-name> disktype=nvme
kubectl delete pod pending-demo
kubectl apply -f pending-demo.yaml
kubectl describe pod pending-demo | grep -A8 Events
Fix the second problem by lowering the memory request to something realistic, such as 128Mi, then recreate the Pod and confirm it is Running. Clean up afterwards.
kubectl delete pod pending-demo
kubectl label node <node-name> disktype-
The lesson is that the scheduler reports the first reason that rejected each node, so fixing one cause often reveals the next. Work through the events until the Pod runs.
8. Quick Reference
9. Prevention and Best Practices
Set realistic CPU and memory requests on every container, and use LimitRange defaults as a safety net.
Treat hard scheduling rules as requirements only, and use soft rules for preferences.
Keep headroom in every node pool and zone, and let the Cluster Autoscaler or an equivalent add nodes when Pods are Pending for capacity reasons.
Make sure labels and taints on node pools match what workloads require.
Alert on Pods Pending longer than a few minutes, and on FailedScheduling events, instead of waiting for users to report problems.
Use a StorageClass with WaitForFirstConsumer for zonal volumes.
Keep quotas, Pod Security levels and admission policies documented so that teams know why a Pod was rejected.
Record common causes and fixes in a team runbook that uses the commands from this article.
10. Conclusion
A Pending Pod always has a reason, and Kubernetes usually tells you what it is in the Pod events. The fastest path is to check whether a node has been assigned. If not, read the FailedScheduling message and compare it with requests, labels, taints, affinity rules, spread constraints, volumes and quotas. If a node is assigned, look at the kubelet side: images, volumes and the network. If no Pod exists, look at the ReplicaSet and admission. With this method, most Pending Pods can be diagnosed in a few minutes.