Pending PVC, Mount Failures and Volume Issues
Kubernetes Storage Troubleshooting: Pending PVC, Mount Failures and Volume Issues
Kubernetes Storage Troubleshooting: Pending PVC, Mount Failures and Volume Issues
bitcodematrix.com | Kubernetes Series
1. Introduction
Storage problems have a reputation for being slow and stressful to debug. A claim stays Pending, a Pod hangs in ContainerCreating, a volume mounts read-only, or a database loses its data directory permissions after a restart. The symptoms look similar from the outside, but the causes sit in very different places: the StorageClass, the CSI driver, the cloud provider, the scheduler, the node, or the application's own security settings.
This guide gives you a symptom-driven method for finding the cause quickly. It starts with a model of the storage chain, shows which commands and events matter, and then walks through the most common failures: Pending claims, Pods that cannot be scheduled or cannot attach, mount failures, runtime volume problems, resize problems and stuck deletions. It builds on the earlier articles in this series on storage architecture, PersistentVolumes, PersistentVolumeClaims, StorageClasses, CSI and StatefulSets.
2. The Storage Chain
Every volume passes through a sequence of stages. A failure at one stage blocks all later ones, so the key skill is to identify which stage has not completed.
Work from stage 1 upwards. A Pod stuck in ContainerCreating with a FailedMount event might really be a stage 2 problem, if the claim never got a valid volume, so always confirm the earlier stages before diving into node logs.
3. The Troubleshooting Toolbox
3.1 From the Cluster
kubectl get pod <pod> -n <ns> -o wide
kubectl describe pod <pod> -n <ns>
kubectl get pvc -n <ns>
kubectl describe pvc <claim> -n <ns>
kubectl get pv
kubectl describe pv <pv-name>
kubectl get storageclass
kubectl get events -n <ns> --sort-by=.lastTimestamp
kubectl get volumeattachments
kubectl get csidrivers
kubectl get csinodes
The describe output is the most valuable item here, because Kubernetes records storage failures as events on the Pod, the claim and the volume, and the text often includes the exact error returned by the driver or the cloud provider.
3.2 Driver and Controller Logs
kubectl get pods -A | grep -i csi
kubectl logs -n <driver-ns> <controller-pod> -c csi-provisioner
kubectl logs -n <driver-ns> <controller-pod> -c csi-attacher
kubectl logs -n <driver-ns> <node-pod> -c <driver-container>
Use the provisioner log for creation problems, the attacher log for attach problems and the node plugin log for mount problems. Container names vary by driver, so list them with kubectl describe pod.
3.3 From the Node
journalctl -u kubelet --since "30 min ago" | grep -i -E "volume|mount|attach"
findmnt | grep -i <pv-name>
lsblk
df -h
df -i
dmesg | tail -50
ls /var/lib/kubelet/pods/<pod-uid>/volumes/
Use these on the node that runs the failing Pod, which you can identify with kubectl get pod -o wide. Search the kubelet log for the phrases MountVolume.SetUp failed and AttachVolume.Attach failed, which introduce the underlying error text.
3.4 An Inspector Pod
To look at the data on a claim without the application, mount it in a throwaway Pod. A ReadWriteOnce volume can be used only from the node it is attached to, so either scale the application down first or pin the inspector to the same node:
apiVersion: v1
kind: Pod
metadata:
name: pvc-inspector
namespace: shop
spec:
nodeName: <node-of-the-app>
containers:
- name: inspector
image: busybox:1.36
command: ["sleep", "3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: app-data
Delete the inspector when you are done, because it holds the volume.
4. Reading Storage Events
The "timed out waiting for the condition" message is generic and appears after a long wait. It rarely identifies the root cause by itself, so always scroll up to earlier events and then read the logs of the component that owns the stage.
5. A Step-by-Step Workflow
Look at the Pod. Run kubectl describe pod and read the events at the bottom. Note whether the Pod is Pending (scheduling), ContainerCreating (attach or mount) or Running (runtime issue).
Check every claim the Pod uses. Is it Bound? If it is Pending, go to Section 6. If it does not exist, create it or fix the name in the Pod spec.
Check the PV behind the claim. Look at its phase, node affinity, access modes and source, and compare them with where and how the Pod runs.
If the Pod is Pending, read the scheduler message and go to Section 7.
If the Pod is ContainerCreating, decide between an attach failure (Section 8) and a mount failure (Section 9) by checking the VolumeAttachment and the events.
If the Pod is Running but misbehaves, go to Section 10 and test the volume from inside the container.
Resist the urge to restart things at random. Deleting a Pod, a claim or a driver Pod without understanding the stage can make matters worse, particularly with attached volumes.
6. Pending PersistentVolumeClaims
A Pending claim means that no volume has been bound yet. First decide whether the situation is expected, then find the blocking cause.
6.1 Expected: WaitForFirstConsumer
If the class uses WaitForFirstConsumer, the claim stays Pending until a Pod that uses it is scheduled. The event says it is waiting for a first consumer. No fix is needed; create the Pod. If the Pod exists and the claim is still Pending, the problem is usually that the Pod itself cannot be scheduled, so read the Pod's events.
6.2 The StorageClass Is Missing or Wrong
kubectl get pvc <claim> -n <ns> -o jsonpath='{.spec.storageClassName}'
kubectl get storageclass
The class name is misspelt or exists only in another cluster. Correct the claim or create the class. A claim's storageClassName cannot be edited, so you recreate the claim.
The claim omits the class and the cluster has no default class. Name a class explicitly or mark one as default.
The claim sets an empty string for the class, which disables dynamic provisioning, and no static PV matches.
6.3 The Provisioner Is Not Running or Does Not Match
If the claim shows ExternalProvisioning and nothing else, check that the CSI controller Pods are running and that the provisioner field of the class matches the driver name exactly:
kubectl get storageclass <class> -o jsonpath='{.provisioner}'
kubectl get csidrivers
kubectl get pods -A | grep -i csi
A mismatch means no sidecar picks up the claim. A crashing controller usually shows credential, permission or RBAC errors in its logs.
6.4 ProvisioningFailed
When the event says that provisioning failed, the text carries the cause. The common ones are:
Permissions. The driver's identity lacks rights to create volumes, tag them or use an encryption key. Fix the cloud role or the credentials Secret.
Quota or limits at the provider. The account, region or storage pool has reached its limit. Raise it or clean up unused volumes.
Invalid parameters. The StorageClass parameters are wrong for the driver, for example an unsupported disk type, a size below the minimum or an unknown key. Since parameters are immutable, create a corrected class.
Unsupported request. The access mode, volume mode or size is not supported by the back end, such as ReadWriteMany from block storage.
Topology. No zone satisfies the combination of allowedTopologies, Pod placement and available capacity.
6.5 Namespace Rules Reject or Block the Claim
A ResourceQuota or LimitRange violation normally rejects the claim immediately with an error, but quotas on storage classes can also be hit later by scaling. Check them with kubectl describe quota and kubectl describe limitrange in the namespace.
6.6 Static PVs Do Not Match
With static provisioning, a claim binds only when a PV matches all of these: the same class, a compatible access mode, a size that is at least the request, the same volume mode, a matching selector if one exists, and no reservation for another claim. Compare the claim and the candidate PV field by field:
kubectl get pvc <claim> -n <ns> -o yaml
kubectl get pv -o custom-columns=\
NAME:.metadata.name,SIZE:.spec.capacity.storage,\
MODES:.spec.accessModes,CLASS:.spec.storageClassName,\
STATUS:.status.phase,CLAIM:.spec.claimRef.name
A PV that is Released still carries the reference to its old claim and will not bind to a new one until the reference is cleared, as described in the PersistentVolume article.
6.7 Summary Table
7. Pods Pending Because of Storage
When the claim is fine but the Pod is Pending, the scheduler message tells you why. Read it in kubectl describe pod, in the Events section.
To see the per-node attach limits that a CSI driver reports, inspect the CSINode object and look for the allocatable count under the driver entry:
kubectl get csinode <node> -o yaml
Zonal problems deserve special attention in StatefulSets. Each replica's volume pins it to a zone, so a zone without free nodes leaves that replica Pending, even when other zones have capacity.
8. Attach Failures
Attach problems appear as FailedAttachVolume or Multi-Attach events and the Pod sits in ContainerCreating. Inspect the VolumeAttachment objects, which show whether Kubernetes thinks a volume is attached and any error:
kubectl get volumeattachment -o custom-columns=\
NAME:.metadata.name,ATTACHER:.spec.attacher,\
PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,\
ATTACHED:.status.attached
kubectl describe volumeattachment <name>
8.1 Multi-Attach Error
The message says that the volume is already exclusively attached to one node and cannot be attached to another. It occurs with single-attach volumes (ReadWriteOnce block storage) when a Pod moves to a different node while the old attachment still exists. Typical situations:
Rolling update of a Deployment with a ReadWriteOnce claim: the new Pod lands on a different node while the old Pod is still running. Use the Recreate strategy for single-replica workloads, or pin Pods to a node.
Node failure or drain: the old node never confirmed that the volume was unmounted, so the attach/detach controller waits before force-detaching, typically for several minutes. Be patient, or follow the safe procedure for non-graceful node shutdown if you have confirmed that the node is really down.
Replicas on different nodes sharing a claim: a Deployment with several replicas and one ReadWriteOnce claim can never work across nodes. Use a StatefulSet with per-replica claims, or a ReadWriteMany backend.
8.2 Other Attach Errors
Cloud API errors in the attacher log, such as permission denied, volume not found or throttling. Fix the identity or retry later.
Wrong zone: the volume and the node are in different zones, which should be impossible with proper topology but happens with manually created PVs. Fix the PV's node affinity or move the Pod.
Attach limit reached on the node. Spread workloads or use larger nodes with higher limits.
Stuck VolumeAttachment after a driver outage. Restart the attacher if it is unhealthy and wait for it to catch up. Do not delete VolumeAttachment objects or remove finalizers until you have verified in the cloud console that the volume is detached.
9. Mount Failures
Once the volume is attached, the kubelet and the CSI node plugin stage and mount it. Failures show as FailedMount events or MountVolume.SetUp failed lines. The message after the colon is the key.
For stubborn cases, correlate three timelines: the Pod events, the node plugin log and the kubelet log, all around the same timestamp. The first error in sequence, not the last, is usually the real cause.
9.1 Check the Mount on the Node
findmnt | grep <pv-name>
ls /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/
dmesg | grep -i -E "ext4|xfs|nfs|i/o error" | tail -20
Kernel messages about I/O errors or file system corruption point to failing storage, not Kubernetes configuration. In that case, involve the storage team or the cloud provider.
10. Problems Inside a Running Container
10.1 Permission Denied
The volume mounted, but the application cannot write. Compare the user the process runs as with the ownership of the mount:
kubectl exec <pod> -n <ns> -- id
kubectl exec <pod> -n <ns> -- ls -ld /data
kubectl exec <pod> -n <ns> -- ls -ln /data | head
The usual fix is to set the Pod's security context so that Kubernetes adjusts group ownership of the volume:
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
Whether fsGroup is applied depends on the volume type and the CSI driver's fsGroupPolicy. NFS exports with root squash can also map the container's root to an unprivileged user, so check the server settings. SELinux labels can cause denials on nodes where it is enforced; the audit log on the node shows them.
10.2 Read-Only File System
A volume that becomes read-only usually means one of three things: the mount was requested read-only (readOnly: true on the claim reference or the mount), the file system detected errors and remounted itself read-only, or the back end has gone away. Check the Pod spec, then look at dmesg on the node for errors. Recovery after file system errors may require stopping the Pod and running a file system check on the volume.
10.3 No Space Left on Device
kubectl exec <pod> -n <ns> -- df -h /data
kubectl exec <pod> -n <ns> -- df -i /data
Check both blocks and inodes. A volume with free space can still be full if it has run out of inodes, which happens with millions of small files. Expand the claim if the class allows it. Remember also that emptyDir volumes and container logs count against the node's ephemeral storage and can lead to Pod eviction when the node runs low, which looks like a storage failure but is not a PV problem.
10.4 Data Missing or Unexpected
Mount path shadows image content. Mounting a volume over a directory that the image populates hides those files. Mount at a different path, or copy the content in at start-up.
Wrong claim or an empty new volume. After recreating a claim, the Pod gets a fresh empty volume. Check which PV the claim is bound to and whether the old one was deleted by its reclaim policy.
emptyDir used by mistake. Data in an emptyDir disappears when the Pod is removed from the node.
lost+found in the root. Freshly formatted ext4 volumes contain this directory, and some databases refuse to start with a non-empty data directory. Use a subPath or a subdirectory for the data.
subPath mismatch. A changed subPath points to a different directory on the same volume.
11. Resize Problems
A resize that does not complete usually fails at one of three points:
Rejected request. The class lacks allowVolumeExpansion, or the claim was not dynamically provisioned. Enable expansion on the class if the driver supports it, then retry.
Back-end expansion failed. The condition or the events show VolumeResizeFailed, and the resizer sidecar log names the cause, such as a provider limit on how often a disk can be modified.
File system resize pending. The condition FileSystemResizePending means the back-end volume has grown but the file system has not. Many drivers resize online; others complete the resize when the Pod restarts. Restart the Pod if the driver requires it.
kubectl describe pvc <claim> -n <ns>
kubectl get pvc <claim> -n <ns> -o jsonpath='{.status.conditions}'
kubectl get pvc <claim> -n <ns> -o jsonpath='{.status.capacity.storage}'
Compare spec.resources.requests.storage (what you asked for) with status.capacity.storage (what exists). Shrinking is not supported, and resizing a StatefulSet's claims needs the manual procedure described in the StatefulSet article.
12. Stuck Deletions
12.1 PVC Stuck in Terminating
The protection finalizer keeps a claim alive while any Pod still uses it. Find the users and stop them:
kubectl describe pvc <claim> -n <ns> | grep -A5 "Used By"
kubectl get pods -n <ns> -o json | grep -i <claim>
Remove or scale down the Pods, and the claim finishes deleting. This is normal behaviour, not a fault.
12.2 PV Stuck in Terminating or Released
A PV in Terminating is still bound to a claim, or the driver has not finished deleting the back-end volume. A PV in Released with the Retain policy is waiting for an administrator, as described in the PersistentVolume article. Check the PV's events and the provisioner log before taking any action.
12.3 Namespace Stuck in Terminating
A namespace can hang when it contains claims or other objects whose finalizers cannot complete, for example because the CSI driver was removed first. Restore the driver so that the finalizers can run, and delete namespaces before uninstalling the storage drivers.
12.4 Last Resort: Removing Finalizers
Removing a finalizer by hand forces the API object to disappear without the cleanup that it was protecting. Do this only when no Pod uses the volume and you have confirmed the state of the back-end disk, because the wrong move can orphan a disk, leave it attached or lose data.
kubectl patch pvc <claim> -n <ns> -p '{"metadata":{"finalizers":null}}'
Afterwards, check the storage system itself for leftover volumes and attachments and clean them up.
13. Quick Reference by Symptom
14. Prevention and Monitoring
Alert on Pending claims that remain unbound for more than a few minutes, using kube-state-metrics, and on Pods stuck in ContainerCreating.
Alert on volume usage with the kubelet metrics kubelet_volume_stats_available_bytes and kubelet_volume_stats_capacity_bytes, and on inode exhaustion where available.
Watch storage operation errors. The controller and kubelet expose metrics such as storage_operation_errors_total and storage_operation_duration_seconds, and CSI sidecars expose their own operation metrics.
Use WaitForFirstConsumer for zonal storage, and keep node capacity in every zone that hosts stateful workloads.
Keep driver and cluster versions compatible, and test CSI driver upgrades before production.
Set quotas and limits per namespace and per class, so that capacity problems appear as clear errors early.
Standardise security contexts for stateful workloads, including fsGroup, so that permission problems do not appear at the first restart.
Practise recovery. Rehearse node failure, volume expansion and restore from snapshot in a test cluster, and keep a runbook that follows the workflow in Section 5.
Back up independently of the cluster, because some storage failures cannot be fixed, only restored from backup.
15. Conclusion
Storage troubleshooting becomes manageable when you stop treating it as one problem and start treating it as a chain of stages. Provisioning, binding, scheduling, attaching, mounting and using a volume each have their own owner, their own events and their own logs. The Pod and claim events tell you which stage stopped, and the logs of that stage's component tell you why.
Follow the workflow, read the events from the first error rather than the last, and be cautious with any step that deletes or forces state. With sound defaults such as delayed binding, enabled expansion, sensible security contexts and good monitoring, most storage incidents are either prevented or become short, well-understood fixes.