kubernetes

Pending PVC, Mount Failures and Volume Issues

Kubernetes Storage Troubleshooting: Pending PVC, Mount Failures and Volume Issues

By Shubhankar Tripathi • • 5 min read

Kubernetes Storage Troubleshooting: Pending PVC, Mount Failures and Volume Issues

bitcodematrix.com | Kubernetes Series

1. Introduction

Storage problems have a reputation for being slow and stressful to debug. A claim stays Pending, a Pod hangs in ContainerCreating, a volume mounts read-only, or a database loses its data directory permissions after a restart. The symptoms look similar from the outside, but the causes sit in very different places: the StorageClass, the CSI driver, the cloud provider, the scheduler, the node, or the application's own security settings.

This guide gives you a symptom-driven method for finding the cause quickly. It starts with a model of the storage chain, shows which commands and events matter, and then walks through the most common failures: Pending claims, Pods that cannot be scheduled or cannot attach, mount failures, runtime volume problems, resize problems and stuck deletions. It builds on the earlier articles in this series on storage architecture, PersistentVolumes, PersistentVolumeClaims, StorageClasses, CSI and StatefulSets.

2. The Storage Chain

Every volume passes through a sequence of stages. A failure at one stage blocks all later ones, so the key skill is to identify which stage has not completed.

Stage

What happens

Component

Typical failure signal

1

Claim is created and a class is chosen

API server, defaulting

PVC Pending, class not found

2

A volume is provisioned and a PV is created

CSI provisioner, storage back end

ProvisioningFailed events

3

Claim is bound to the PV

PV controller

PVC Pending, no matching PV

4

Pod is scheduled with volume constraints

Scheduler

Pod Pending with volume-related reasons

5

Volume is attached to the node

Attach/detach controller, external-attacher

FailedAttachVolume, Multi-Attach error

6

Volume is staged and mounted into the Pod

Kubelet, CSI node plugin

FailedMount, ContainerCreating

7

Container uses the volume

Application, file system

Permission denied, read-only, no space left

 

Work from stage 1 upwards. A Pod stuck in ContainerCreating with a FailedMount event might really be a stage 2 problem, if the claim never got a valid volume, so always confirm the earlier stages before diving into node logs.

3. The Troubleshooting Toolbox

3.1 From the Cluster

kubectl get pod <pod> -n <ns> -o wide

kubectl describe pod <pod> -n <ns>

kubectl get pvc -n <ns>

kubectl describe pvc <claim> -n <ns>

kubectl get pv

kubectl describe pv <pv-name>

kubectl get storageclass

kubectl get events -n <ns> --sort-by=.lastTimestamp

kubectl get volumeattachments

kubectl get csidrivers

kubectl get csinodes

The describe output is the most valuable item here, because Kubernetes records storage failures as events on the Pod, the claim and the volume, and the text often includes the exact error returned by the driver or the cloud provider.

3.2 Driver and Controller Logs

kubectl get pods -A | grep -i csi

kubectl logs -n <driver-ns> <controller-pod> -c csi-provisioner

kubectl logs -n <driver-ns> <controller-pod> -c csi-attacher

kubectl logs -n <driver-ns> <node-pod> -c <driver-container>

Use the provisioner log for creation problems, the attacher log for attach problems and the node plugin log for mount problems. Container names vary by driver, so list them with kubectl describe pod.

3.3 From the Node

journalctl -u kubelet --since "30 min ago" | grep -i -E "volume|mount|attach"

findmnt | grep -i <pv-name>

lsblk

df -h

df -i

dmesg | tail -50

ls /var/lib/kubelet/pods/<pod-uid>/volumes/

Use these on the node that runs the failing Pod, which you can identify with kubectl get pod -o wide. Search the kubelet log for the phrases MountVolume.SetUp failed and AttachVolume.Attach failed, which introduce the underlying error text.

3.4 An Inspector Pod

To look at the data on a claim without the application, mount it in a throwaway Pod. A ReadWriteOnce volume can be used only from the node it is attached to, so either scale the application down first or pin the inspector to the same node:

apiVersion: v1

kind: Pod

metadata:

  name: pvc-inspector

  namespace: shop

spec:

  nodeName: <node-of-the-app>

  containers:

  - name: inspector

    image: busybox:1.36

    command: ["sleep", "3600"]

    volumeMounts:

    - name: data

      mountPath: /data

  volumes:

  - name: data

    persistentVolumeClaim:

      claimName: app-data

Delete the inspector when you are done, because it holds the volume.

4. Reading Storage Events

Event or message

What it tells you

waiting for first consumer to be created before binding

Normal with WaitForFirstConsumer. No Pod uses the claim yet

ExternalProvisioning: waiting for a volume to be created, either by external provisioner or manually

A provisioner is expected to create the volume. If this lasts, the provisioner is missing, misnamed or failing

ProvisioningFailed

The driver's CreateVolume call failed. The event text states why

pod has unbound immediate PersistentVolumeClaims

The scheduler cannot place the Pod because a claim is not yet bound

node(s) had volume node affinity conflict

The volume can only be used in a place where the Pod cannot run

node(s) exceed max volume count

Nodes have reached their limit of attached volumes

Multi-Attach error ... Volume is already exclusively attached to one node

A single-attach volume is still attached elsewhere

FailedAttachVolume

The attach step failed in the attacher or the cloud API

FailedMount: Unable to attach or mount volumes ... timed out waiting for the condition

The kubelet gave up waiting for a volume; look at the earlier events for the real cause

MountVolume.SetUp failed

The node-side mount failed. The rest of the line carries the driver or mount error

VolumeResizeFailed or FileSystemResizeFailed

Expansion of the back end or the file system failed

 

The "timed out waiting for the condition" message is generic and appears after a long wait. It rarely identifies the root cause by itself, so always scroll up to earlier events and then read the logs of the component that owns the stage.

5. A Step-by-Step Workflow

  1. Look at the Pod. Run kubectl describe pod and read the events at the bottom. Note whether the Pod is Pending (scheduling), ContainerCreating (attach or mount) or Running (runtime issue).

  2. Check every claim the Pod uses. Is it Bound? If it is Pending, go to Section 6. If it does not exist, create it or fix the name in the Pod spec.

  3. Check the PV behind the claim. Look at its phase, node affinity, access modes and source, and compare them with where and how the Pod runs.

  4. If the Pod is Pending, read the scheduler message and go to Section 7.

  5. If the Pod is ContainerCreating, decide between an attach failure (Section 8) and a mount failure (Section 9) by checking the VolumeAttachment and the events.

  6. If the Pod is Running but misbehaves, go to Section 10 and test the volume from inside the container.

Resist the urge to restart things at random. Deleting a Pod, a claim or a driver Pod without understanding the stage can make matters worse, particularly with attached volumes.

6. Pending PersistentVolumeClaims

A Pending claim means that no volume has been bound yet. First decide whether the situation is expected, then find the blocking cause.

6.1 Expected: WaitForFirstConsumer

If the class uses WaitForFirstConsumer, the claim stays Pending until a Pod that uses it is scheduled. The event says it is waiting for a first consumer. No fix is needed; create the Pod. If the Pod exists and the claim is still Pending, the problem is usually that the Pod itself cannot be scheduled, so read the Pod's events.

6.2 The StorageClass Is Missing or Wrong

kubectl get pvc <claim> -n <ns> -o jsonpath='{.spec.storageClassName}'

kubectl get storageclass

  • The class name is misspelt or exists only in another cluster. Correct the claim or create the class. A claim's storageClassName cannot be edited, so you recreate the claim.

  • The claim omits the class and the cluster has no default class. Name a class explicitly or mark one as default.

  • The claim sets an empty string for the class, which disables dynamic provisioning, and no static PV matches.

6.3 The Provisioner Is Not Running or Does Not Match

If the claim shows ExternalProvisioning and nothing else, check that the CSI controller Pods are running and that the provisioner field of the class matches the driver name exactly:

kubectl get storageclass <class> -o jsonpath='{.provisioner}'

kubectl get csidrivers

kubectl get pods -A | grep -i csi

A mismatch means no sidecar picks up the claim. A crashing controller usually shows credential, permission or RBAC errors in its logs.

6.4 ProvisioningFailed

When the event says that provisioning failed, the text carries the cause. The common ones are:

  • Permissions. The driver's identity lacks rights to create volumes, tag them or use an encryption key. Fix the cloud role or the credentials Secret.

  • Quota or limits at the provider. The account, region or storage pool has reached its limit. Raise it or clean up unused volumes.

  • Invalid parameters. The StorageClass parameters are wrong for the driver, for example an unsupported disk type, a size below the minimum or an unknown key. Since parameters are immutable, create a corrected class.

  • Unsupported request. The access mode, volume mode or size is not supported by the back end, such as ReadWriteMany from block storage.

  • Topology. No zone satisfies the combination of allowedTopologies, Pod placement and available capacity.

6.5 Namespace Rules Reject or Block the Claim

A ResourceQuota or LimitRange violation normally rejects the claim immediately with an error, but quotas on storage classes can also be hit later by scaling. Check them with kubectl describe quota and kubectl describe limitrange in the namespace.

6.6 Static PVs Do Not Match

With static provisioning, a claim binds only when a PV matches all of these: the same class, a compatible access mode, a size that is at least the request, the same volume mode, a matching selector if one exists, and no reservation for another claim. Compare the claim and the candidate PV field by field:

kubectl get pvc <claim> -n <ns> -o yaml

kubectl get pv -o custom-columns=\

NAME:.metadata.name,SIZE:.spec.capacity.storage,\

MODES:.spec.accessModes,CLASS:.spec.storageClassName,\

STATUS:.status.phase,CLAIM:.spec.claimRef.name

A PV that is Released still carries the reference to its old claim and will not bind to a new one until the reference is cleared, as described in the PersistentVolume article.

6.7 Summary Table

Observation

Most likely cause

Event: waiting for first consumer

Normal; create or schedule the Pod

No events at all on the claim

Class missing, no default class, or empty class with no static PV

ExternalProvisioning forever

Provisioner not running or name mismatch

ProvisioningFailed with permission text

Driver identity lacks rights

ProvisioningFailed with a parameter error

Bad StorageClass parameters or request

Claim rejected on creation

ResourceQuota or LimitRange

PVs exist but claim stays Pending

Class, mode, size or selector mismatch, or PV reserved or Released

 

7. Pods Pending Because of Storage

When the claim is fine but the Pod is Pending, the scheduler message tells you why. Read it in kubectl describe pod, in the Events section.

Scheduler message

Cause and fix

persistentvolumeclaim "x" not found

The claim does not exist in the Pod's namespace. Create it or fix claimName

pod has unbound immediate PersistentVolumeClaims

The claim is not bound. Resolve the Pending claim first (Section 6)

node(s) had volume node affinity conflict

The PV is bound to a zone or node where the Pod cannot run. Check nodeSelector, affinity and zone capacity. The volume cannot move; the Pod must go to it

node(s) exceed max volume count

The nodes have reached their attach limit. Add nodes, spread Pods, or review the driver limit in the CSINode object

node(s) didn't find available persistent volumes to bind

With delayed binding and static or local PVs, no unbound PV matches on any candidate node. Add a matching PV or adjust the claim

Insufficient cpu or memory together with storage reasons

Several constraints combine. Fix the resource problem too, or the Pod will stay Pending

 

To see the per-node attach limits that a CSI driver reports, inspect the CSINode object and look for the allocatable count under the driver entry:

kubectl get csinode <node> -o yaml

Zonal problems deserve special attention in StatefulSets. Each replica's volume pins it to a zone, so a zone without free nodes leaves that replica Pending, even when other zones have capacity.

8. Attach Failures

Attach problems appear as FailedAttachVolume or Multi-Attach events and the Pod sits in ContainerCreating. Inspect the VolumeAttachment objects, which show whether Kubernetes thinks a volume is attached and any error:

kubectl get volumeattachment -o custom-columns=\

NAME:.metadata.name,ATTACHER:.spec.attacher,\

PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,\

ATTACHED:.status.attached

kubectl describe volumeattachment <name>

8.1 Multi-Attach Error

The message says that the volume is already exclusively attached to one node and cannot be attached to another. It occurs with single-attach volumes (ReadWriteOnce block storage) when a Pod moves to a different node while the old attachment still exists. Typical situations:

  • Rolling update of a Deployment with a ReadWriteOnce claim: the new Pod lands on a different node while the old Pod is still running. Use the Recreate strategy for single-replica workloads, or pin Pods to a node.

  • Node failure or drain: the old node never confirmed that the volume was unmounted, so the attach/detach controller waits before force-detaching, typically for several minutes. Be patient, or follow the safe procedure for non-graceful node shutdown if you have confirmed that the node is really down.

  • Replicas on different nodes sharing a claim: a Deployment with several replicas and one ReadWriteOnce claim can never work across nodes. Use a StatefulSet with per-replica claims, or a ReadWriteMany backend.

8.2 Other Attach Errors

  • Cloud API errors in the attacher log, such as permission denied, volume not found or throttling. Fix the identity or retry later.

  • Wrong zone: the volume and the node are in different zones, which should be impossible with proper topology but happens with manually created PVs. Fix the PV's node affinity or move the Pod.

  • Attach limit reached on the node. Spread workloads or use larger nodes with higher limits.

  • Stuck VolumeAttachment after a driver outage. Restart the attacher if it is unhealthy and wait for it to catch up. Do not delete VolumeAttachment objects or remove finalizers until you have verified in the cloud console that the volume is detached.

9. Mount Failures

Once the volume is attached, the kubelet and the CSI node plugin stage and mount it. Failures show as FailedMount events or MountVolume.SetUp failed lines. The message after the colon is the key.

Error text

Cause and fix

driver name ... not found in the list of registered CSI drivers

The node plugin is not running or not registered on that node. Check the node plugin Pod and the CSINode object, and restart the plugin Pod if needed

configmap "x" not found or secret "x" not found

The Pod references a ConfigMap or Secret volume that does not exist. Create it, or mark the volume optional if that is acceptable

wrong fs type, bad option, bad superblock

Unsupported or damaged file system, or invalid mountOptions. Verify the fstype, the options, and the file system with fsck on the node if safe

you might need a /sbin/mount.<type> helper program

The node lacks client tools such as NFS or CIFS utilities. Install them on the node image

mount.nfs: Connection timed out, access denied or No route to host

The NFS server is unreachable, blocked by a firewall, or the export does not allow this node. Test connectivity and export permissions from the node

stale file handle

The NFS export or path changed. Remount after correcting the server side

device or resource busy or already mounted

A previous mount was not cleaned up. Investigate the node's mounts and the previous Pod's termination

failed to create subPath directory or subPath errors

The subPath conflicts with an existing file, or ownership prevents creation. Check the path inside the volume

permission denied from the driver

Credentials for the storage system are wrong or expired, or node identity lacks access

rpc error: code = DeadlineExceeded

The driver call timed out, often because the back end is slow or overloaded. Check the storage system and driver logs

 

For stubborn cases, correlate three timelines: the Pod events, the node plugin log and the kubelet log, all around the same timestamp. The first error in sequence, not the last, is usually the real cause.

9.1 Check the Mount on the Node

findmnt | grep <pv-name>

ls /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/

dmesg | grep -i -E "ext4|xfs|nfs|i/o error" | tail -20

Kernel messages about I/O errors or file system corruption point to failing storage, not Kubernetes configuration. In that case, involve the storage team or the cloud provider.

10. Problems Inside a Running Container

10.1 Permission Denied

The volume mounted, but the application cannot write. Compare the user the process runs as with the ownership of the mount:

kubectl exec <pod> -n <ns> -- id

kubectl exec <pod> -n <ns> -- ls -ld /data

kubectl exec <pod> -n <ns> -- ls -ln /data | head

The usual fix is to set the Pod's security context so that Kubernetes adjusts group ownership of the volume:

spec:

  securityContext:

    runAsUser: 1000

    runAsGroup: 3000

    fsGroup: 2000

    fsGroupChangePolicy: OnRootMismatch

Whether fsGroup is applied depends on the volume type and the CSI driver's fsGroupPolicy. NFS exports with root squash can also map the container's root to an unprivileged user, so check the server settings. SELinux labels can cause denials on nodes where it is enforced; the audit log on the node shows them.

10.2 Read-Only File System

A volume that becomes read-only usually means one of three things: the mount was requested read-only (readOnly: true on the claim reference or the mount), the file system detected errors and remounted itself read-only, or the back end has gone away. Check the Pod spec, then look at dmesg on the node for errors. Recovery after file system errors may require stopping the Pod and running a file system check on the volume.

10.3 No Space Left on Device

kubectl exec <pod> -n <ns> -- df -h /data

kubectl exec <pod> -n <ns> -- df -i /data

Check both blocks and inodes. A volume with free space can still be full if it has run out of inodes, which happens with millions of small files. Expand the claim if the class allows it. Remember also that emptyDir volumes and container logs count against the node's ephemeral storage and can lead to Pod eviction when the node runs low, which looks like a storage failure but is not a PV problem.

10.4 Data Missing or Unexpected

  • Mount path shadows image content. Mounting a volume over a directory that the image populates hides those files. Mount at a different path, or copy the content in at start-up.

  • Wrong claim or an empty new volume. After recreating a claim, the Pod gets a fresh empty volume. Check which PV the claim is bound to and whether the old one was deleted by its reclaim policy.

  • emptyDir used by mistake. Data in an emptyDir disappears when the Pod is removed from the node.

  • lost+found in the root. Freshly formatted ext4 volumes contain this directory, and some databases refuse to start with a non-empty data directory. Use a subPath or a subdirectory for the data.

  • subPath mismatch. A changed subPath points to a different directory on the same volume.

11. Resize Problems

A resize that does not complete usually fails at one of three points:

  • Rejected request. The class lacks allowVolumeExpansion, or the claim was not dynamically provisioned. Enable expansion on the class if the driver supports it, then retry.

  • Back-end expansion failed. The condition or the events show VolumeResizeFailed, and the resizer sidecar log names the cause, such as a provider limit on how often a disk can be modified.

  • File system resize pending. The condition FileSystemResizePending means the back-end volume has grown but the file system has not. Many drivers resize online; others complete the resize when the Pod restarts. Restart the Pod if the driver requires it.

kubectl describe pvc <claim> -n <ns>

kubectl get pvc <claim> -n <ns> -o jsonpath='{.status.conditions}'

kubectl get pvc <claim> -n <ns> -o jsonpath='{.status.capacity.storage}'

Compare spec.resources.requests.storage (what you asked for) with status.capacity.storage (what exists). Shrinking is not supported, and resizing a StatefulSet's claims needs the manual procedure described in the StatefulSet article.

12. Stuck Deletions

12.1 PVC Stuck in Terminating

The protection finalizer keeps a claim alive while any Pod still uses it. Find the users and stop them:

kubectl describe pvc <claim> -n <ns> | grep -A5 "Used By"

kubectl get pods -n <ns> -o json | grep -i <claim>

Remove or scale down the Pods, and the claim finishes deleting. This is normal behaviour, not a fault.

12.2 PV Stuck in Terminating or Released

A PV in Terminating is still bound to a claim, or the driver has not finished deleting the back-end volume. A PV in Released with the Retain policy is waiting for an administrator, as described in the PersistentVolume article. Check the PV's events and the provisioner log before taking any action.

12.3 Namespace Stuck in Terminating

A namespace can hang when it contains claims or other objects whose finalizers cannot complete, for example because the CSI driver was removed first. Restore the driver so that the finalizers can run, and delete namespaces before uninstalling the storage drivers.

12.4 Last Resort: Removing Finalizers

Removing a finalizer by hand forces the API object to disappear without the cleanup that it was protecting. Do this only when no Pod uses the volume and you have confirmed the state of the back-end disk, because the wrong move can orphan a disk, leave it attached or lose data.

kubectl patch pvc <claim> -n <ns> -p '{"metadata":{"finalizers":null}}'

Afterwards, check the storage system itself for leftover volumes and attachments and clean them up.

13. Quick Reference by Symptom

Symptom

First check

Likely fix

PVC Pending

kubectl describe pvc

Create the Pod (delayed binding), fix class, provisioner or parameters

Pod Pending, volume node affinity conflict

PV node affinity and Pod placement

Align zone and node constraints, add capacity in the zone

Pod ContainerCreating

Pod events, VolumeAttachment

Resolve attach or mount error

Multi-Attach error

VolumeAttachment, node status

Recreate strategy, wait for detach, fix node failure

FailedMount with driver not found

CSINode, node plugin Pod

Restart or fix the node plugin

Permission denied

id and ls -ln in the container

Set fsGroup, runAsUser, check NFS squash

Read-only file system

Pod spec and dmesg

Remove readOnly, repair the file system

No space left on device

df -h and df -i

Expand the claim, clean up data

Resize stuck

PVC conditions

Enable expansion, restart the Pod, check the resizer log

PVC or PV Terminating

Used By, finalizers

Stop the Pods, wait for the driver, avoid forcing

 

14. Prevention and Monitoring

  • Alert on Pending claims that remain unbound for more than a few minutes, using kube-state-metrics, and on Pods stuck in ContainerCreating.

  • Alert on volume usage with the kubelet metrics kubelet_volume_stats_available_bytes and kubelet_volume_stats_capacity_bytes, and on inode exhaustion where available.

  • Watch storage operation errors. The controller and kubelet expose metrics such as storage_operation_errors_total and storage_operation_duration_seconds, and CSI sidecars expose their own operation metrics.

  • Use WaitForFirstConsumer for zonal storage, and keep node capacity in every zone that hosts stateful workloads.

  • Keep driver and cluster versions compatible, and test CSI driver upgrades before production.

  • Set quotas and limits per namespace and per class, so that capacity problems appear as clear errors early.

  • Standardise security contexts for stateful workloads, including fsGroup, so that permission problems do not appear at the first restart.

  • Practise recovery. Rehearse node failure, volume expansion and restore from snapshot in a test cluster, and keep a runbook that follows the workflow in Section 5.

  • Back up independently of the cluster, because some storage failures cannot be fixed, only restored from backup.

15. Conclusion

Storage troubleshooting becomes manageable when you stop treating it as one problem and start treating it as a chain of stages. Provisioning, binding, scheduling, attaching, mounting and using a volume each have their own owner, their own events and their own logs. The Pod and claim events tell you which stage stopped, and the logs of that stage's component tell you why.

Follow the workflow, read the events from the first error rather than the last, and be cautious with any step that deletes or forces state. With sound defaults such as delayed binding, enabled expansion, sensible security contexts and good monitoring, most storage incidents are either prevented or become short, well-understood fixes.