kubernetes

Kubernetes PersistentVolume Explained

Every stateful workload on Kubernetes eventually depends on one object: the PersistentVolume

By Shubhankar Tripathi • • 5 min read

Kubernetes PersistentVolume Explained

bitcodematrix.com | Kubernetes Series

1. Introduction

Every stateful workload on Kubernetes eventually depends on one object: the PersistentVolume, usually shortened to PV. It is the record that tells the cluster "this piece of real storage exists, this is how big it is, this is how it can be used, and this is what to do with it when nobody needs it any more". Developers rarely touch PVs directly, but when a database will not start, a claim stays Pending, or a disk keeps costing money after a project is deleted, the PV is where the answer lives.

This article is a close look at the PersistentVolume object itself. It complements the broader storage architecture article in this series by focusing on the PV specification, how binding works, the lifecycle phases, reclaim policies, backend types, node affinity and the practical procedures that administrators need: inspecting, reusing, moving and safely deleting volumes.

2. What Is a PersistentVolume?

A PersistentVolume is a cluster-scoped API object that represents a piece of storage which has been provisioned either by an administrator or automatically by a StorageClass. It has a lifecycle that is independent of any Pod that uses it. The PV object holds the details of the storage implementation, such as an NFS export, a cloud disk handled by a CSI driver, or a directory on a specific node.

Three objects work together, and it helps to be clear about who owns what:

Object

Role

Who normally manages it

PersistentVolume

The supply: a real volume and its properties

Administrator, or a provisioner on demand

PersistentVolumeClaim

The demand: a namespaced request for storage

Application team

StorageClass

The recipe: how to create PVs on demand

Platform team

 

A Pod never references a PV directly. It references a claim, and the claim is bound to a PV. This indirection is what makes Pod specifications portable between clusters whose storage differs completely.

3. Anatomy of a PersistentVolume

The manifest below describes a pre-existing cloud disk that an administrator is importing into the cluster through a CSI driver. The driver name and volume handle are examples only; take the real values and supported fields from your driver's documentation.

apiVersion: v1

kind: PersistentVolume

metadata:

  name: pv-data-01

  labels:

    tier: gold

spec:

  capacity:

    storage: 100Gi

  volumeMode: Filesystem

  accessModes:

  - ReadWriteOnce

  persistentVolumeReclaimPolicy: Retain

  storageClassName: fast-ssd

  mountOptions:

  - noatime

  csi:

    driver: ebs.csi.aws.com

    volumeHandle: vol-0123456789abcdef0

    fsType: ext4

Field

Meaning

capacity.storage

The size of the volume. Kubernetes uses it for matching claims; whether the backend actually enforces it depends on the storage system (an NFS export, for example, may not enforce any limit)

volumeMode

Filesystem (default) mounts the volume as a directory; Block presents a raw device

accessModes

How the volume can be mounted: ReadWriteOnce, ReadOnlyMany, ReadWriteMany or ReadWriteOncePod. Even if several are listed, a volume is mounted using only one mode at a time

persistentVolumeReclaimPolicy

What happens to the volume after its claim is deleted: Retain or Delete (Recycle is deprecated)

storageClassName

The class this PV belongs to. A claim can bind only to a PV of the same class. An empty value means the PV has no class

mountOptions

Extra options passed to the mount. They are not validated, so an invalid option makes the mount fail

nodeAffinity

Which nodes can access the volume. Required for local volumes and used for zonal storage

claimRef

The claim currently bound to this PV, or a reservation for a specific claim

csi, nfs, local and others

The backend-specific source of the volume. Exactly one source is set

 

Two pieces of metadata are also worth knowing. PVs created by a provisioner carry the annotation pv.kubernetes.io/provisioned-by, which names the driver that made them, and PVs bound automatically carry pv.kubernetes.io/bound-by-controller.

4. How Binding Works

A control loop in the Kubernetes controller manager, often called the PV controller, watches for unbound claims and looks for a PV that satisfies them. A PV is a candidate only if all of the following hold:

Criterion

Rule

storageClassName

Must be identical on the claim and the PV

accessModes

The PV must provide all access modes the claim requests

capacity

The PV must be at least as large as the request

volumeMode

Must be the same (Filesystem or Block)

selector (optional)

If the claim has a label selector, the PV must match its labels

claimRef

The PV must be unbound, or reserved for exactly this claim

 

When several PVs qualify, the controller picks the smallest one that is large enough. Binding is exclusive and one-to-one: once bound, a PV serves exactly one claim, even if the claim uses only part of the capacity. A claim for 10Gi can therefore end up with a 100Gi PV if that is the smallest match, and the rest of the space is simply not available to anyone else. If no PV matches, the claim stays Pending and is rechecked as new PVs appear.

4.1 Reserving a PV for a Specific Claim

Sometimes you want a particular PV to go to a particular claim, for example when importing an existing database disk. You can pre-bind in either direction. On the PV, set claimRef with the claim's namespace and name:

spec:

  claimRef:

    namespace: shop

    name: app-data

Or on the claim, name the PV with volumeName:

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: app-data

  namespace: shop

spec:

  storageClassName: fast-ssd

  volumeName: pv-data-01

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: 50Gi

The remaining criteria, such as class, access mode and size, must still match. Pre-binding reserves the PV so other claims cannot take it.

5. Static and Dynamic Provisioning

Aspect

Static provisioning

Dynamic provisioning

Who creates the PV

An administrator, in advance

A provisioner, when a claim appears

Backend storage

Must already exist or be created separately

Created automatically by the driver

Best for

Existing data, NFS exports, local disks, imports

Day-to-day workloads on cloud or software-defined storage

Effort

Manual and error-prone at scale

Self-service

Default reclaim policy

Whatever the administrator sets (Retain is common)

Taken from the StorageClass (Delete unless set otherwise)

 

With dynamic provisioning, you never write the PV manifest. After a claim binds, run kubectl get pv and you will find a generated PV with a name such as pvc- followed by a unique identifier, the class of the claim, the size requested, and a reclaim policy copied from the StorageClass. Treat these generated PVs as managed objects: change them only deliberately, for example to switch the reclaim policy to Retain.

6. PersistentVolume Phases

The status.phase field of a PV tells you where it is in its lifecycle:

Phase

Meaning and what to do

Available

Free and not bound. It can be claimed by a matching PVC

Bound

Bound to a claim and in normal use or ready for use

Released

The claim was deleted, but the PV has not been reclaimed. With Retain, it still holds data and its old claimRef, so it is not offered to new claims automatically

Failed

Automatic reclamation failed, for example because the backend volume could not be deleted. Read the PV events and fix the backend, then clean up manually

 

A typical path is Available, then Bound when a claim arrives, then Released when the claim is deleted, after which the reclaim policy decides the outcome: the PV and backend volume are deleted (Delete), or kept in the Released state for an administrator (Retain).

7. Reclaim Policies in Depth

7.1 Delete

When the claim is deleted, Kubernetes deletes the PV object and asks the plugin to delete the backend volume. This is the typical default for dynamically provisioned storage. It keeps costs under control but makes accidental deletion of a claim a data-loss event. Use it for caches, build workspaces and other data you can recreate.

7.2 Retain

The PV and its data survive claim deletion. This is the safer choice for databases and anything irreplaceable. The price is manual cleanup: Retained volumes and their backend disks continue to consume capacity and incur cost until an administrator removes them. Make sure that someone owns that clean-up task.

7.3 Changing the Policy on a Live PV

You can change the policy at any time, for example to protect a dynamically provisioned volume before a risky operation:

kubectl patch pv <pv-name> \

  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

7.4 Reusing a Released PV

A Released PV with Retain is still tied to its old claim. To make it available again, remove the claim reference:

kubectl patch pv <pv-name> -p '{"spec":{"claimRef":null}}'

The PV returns to Available, but the data is still on the volume. A new claim that binds to it will see the old contents. If that is not what you want, wipe or recreate the backend volume first. Never assume that a reused PV is empty.

8. Volume Sources and Backends

The source section of the PV says where the storage really is. The most important types today are:

Source

Description

Notes

csi

Any storage system with a CSI driver: cloud disks, file services, Ceph, SAN arrays

The standard, recommended way; most older in-tree cloud types have been removed in its favour

nfs

An existing NFS export

Supports ReadWriteMany; capacity is usually not enforced

local

A disk, partition or directory on one specific node

Needs nodeAffinity; data is lost to the Pod if the node fails

hostPath

A path on the node's filesystem

Meant only for single-node testing; not suitable for production

 

8.1 Local PersistentVolumes

Local volumes give excellent performance because the disk is attached to the node, and they suit workloads that replicate data themselves. Because the data cannot follow the Pod to another node, Kubernetes needs to know which node owns each volume. A local PV must declare nodeAffinity, and the StorageClass should delay binding until a Pod is scheduled:

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: local-storage

provisioner: kubernetes.io/no-provisioner

volumeBindingMode: WaitForFirstConsumer

apiVersion: v1

kind: PersistentVolume

metadata:

  name: pv-local-node1

spec:

  capacity:

    storage: 200Gi

  accessModes:

  - ReadWriteOnce

  persistentVolumeReclaimPolicy: Retain

  storageClassName: local-storage

  local:

    path: /mnt/disks/ssd1

  nodeAffinity:

    required:

      nodeSelectorTerms:

      - matchExpressions:

        - key: kubernetes.io/hostname

          operator: In

          values:

          - worker-node-1

The provisioner kubernetes.io/no-provisioner means that Kubernetes does not create these volumes itself. A separate local static provisioner project exists to discover disks on nodes and create PVs for them automatically. Remember that if the node is lost, the Pod using the volume cannot be rescheduled elsewhere with its data.

9. Node Affinity and Topology

The nodeAffinity field restricts which nodes can access the volume. For local volumes it pins the PV to a node. For zonal cloud disks, provisioners set it automatically so that the Pod is scheduled in the zone that holds the disk. The scheduler takes this affinity into account together with the Pod's own constraints.

If the constraints conflict, for example a Pod pinned to a zone that is different from the PV's zone, the Pod stays Pending with the event "volume node affinity conflict". For statically created zonal PVs, set nodeAffinity yourself using the topology label keys that your CSI driver and nodes use, which you can see on an existing PV created by the same driver. Mistakes here are a common source of Pending Pods.

10. Inspecting PersistentVolumes

kubectl get pv

kubectl get pv -o wide

kubectl describe pv <pv-name>

kubectl get pv <pv-name> -o yaml

The describe output shows the status, the bound claim, the reclaim policy, the access modes, the node affinity, the volume source and events. A custom view makes fleet-wide checks easy:

kubectl get pv -o custom-columns=\

NAME:.metadata.name,\

SIZE:.spec.capacity.storage,\

CLASS:.spec.storageClassName,\

RECLAIM:.spec.persistentVolumeReclaimPolicy,\

STATUS:.status.phase,\

CLAIM:.spec.claimRef.name,\

NAMESPACE:.spec.claimRef.namespace

Use it to find Released and Available volumes that may be orphaned and still costing money, and to confirm the reclaim policy of important data volumes. If you run kube-state-metrics, the kube_persistentvolume_status_phase metric allows you to alert when PVs are stuck in Released or Failed.

11. Moving a Volume to Another Namespace

Because a PV is cluster-scoped and a claim is namespaced, you can hand an existing volume from one namespace to another within the same cluster without copying data. Plan a maintenance window, because the workload must be stopped. The procedure is:

  1. Stop every Pod that uses the claim, for example by scaling the workload to zero, so that nothing writes to the volume.

  2. Set the PV's reclaim policy to Retain, as shown in Section 7.3, so that deleting the claim does not delete the data.

  3. Delete the old claim. The PV moves to Released.

  4. Clear the PV's claimRef so it becomes Available, as shown in Section 7.4.

  5. Create a new claim in the target namespace with volumeName set to the PV name, and the same class, a compatible access mode and a size no larger than the PV capacity.

  6. Confirm that the new claim is Bound, start the workload against it, and verify the data.

  7. Set the reclaim policy back to the value you want for the long term.

This works across namespaces but not across clusters. Moving data to another cluster requires a snapshot-and-restore or a backup tool.

12. Protection and Safe Deletion

Kubernetes applies the finalizer kubernetes.io/pv-protection to PVs, which prevents a PV that is bound to a claim from being removed. If you delete a bound PV object, it stays in the Terminating state until the claim releases it. CSI sidecars may add their own finalizers to ensure that the backend volume is cleaned up before the object disappears.

Follow these rules for safe deletion:

  • Delete the claim, not the PV, when you want to release a dynamically provisioned volume. The reclaim policy then does the right thing.

  • Before deleting, check the reclaim policy. With Delete, the backend data will be removed; with Retain, you will need to clean up the backend yourself.

  • Do not remove finalizers by hand to force-delete a PV unless you have confirmed that nothing uses the volume and you know the state of the backend, because this can leave orphaned disks or break the attachment bookkeeping.

13. Troubleshooting PersistentVolumes

Symptom

Likely cause and check

PVC Pending, and PVs exist

No PV matches. Compare class, access modes, size, volumeMode and any selector. A PV with a larger size but a different class will not match.

PV stays Available but claim does not bind

The PV is reserved by claimRef for a different claim, or the claim uses an empty class while the PV has one.

PV stuck in Released

Retain policy with the old claimRef still set. Clear it as in Section 7.4 after handling the data.

PV in Failed state

Reclamation could not delete the backend volume. Read the PV events and fix the backend permissions or state.

PV stuck in Terminating

Protection finalizers: the PV is still bound to a claim or the backend deletion is incomplete. Release the claim first.

Pod Pending with volume node affinity conflict

The PV's node affinity does not match where the Pod can run. Review nodeSelector, affinity rules and zones.

Mount fails with mount options error

Invalid or unsupported entries in mountOptions. Correct them on the PV.

Reused PV contains old data

Expected with Retain. Wipe or recreate the volume before reuse.

Disk still billed after the project is deleted

A Retained or Released PV and its backend disk remain. List PVs by phase and clean them up.

 

Always read the events first, with kubectl describe on the PV, the claim and the Pod, because they usually name the exact cause.

14. Best Practices

  • Prefer dynamic provisioning for normal workloads and keep manually created PVs for imports, NFS and local storage.

  • Set the reclaim policy deliberately: Retain for irreplaceable data, Delete for disposable data. Do not rely on the default.

  • Give PVs and StorageClasses clear names and labels, such as the owning team or tier, so that orphaned volumes can be traced.

  • Review Released, Available and Failed PVs regularly and clean them up, to avoid paying for unused disks.

  • Use WaitForFirstConsumer for zonal and local storage so that volumes land where Pods can actually run.

  • Restrict permissions to create and edit PVs with RBAC, because PVs are cluster-wide and can expose sensitive node paths or storage systems.

  • Avoid hostPath for anything but single-node testing, and design local PV workloads to replicate their own data.

  • Back up data independently of the PV, because the PV is only a pointer to storage and does not protect the contents.

15. Conclusion

A PersistentVolume is a small object with a large responsibility: it connects a Kubernetes cluster to real storage and defines how that storage may be matched, accessed, protected and eventually released. Once you understand the binding criteria, the four lifecycle phases and the two reclaim policies, most storage problems reduce to reading a few fields.

Use dynamic provisioning for everyday work, choose reclaim policies on purpose, and keep an eye on volumes that outlive their claims. With those habits, PersistentVolumes stop being a source of surprises and become the dependable foundation for stateful workloads.