Kubernetes PersistentVolume Explained
Every stateful workload on Kubernetes eventually depends on one object: the PersistentVolume
Kubernetes PersistentVolume Explained
bitcodematrix.com | Kubernetes Series
1. Introduction
Every stateful workload on Kubernetes eventually depends on one object: the PersistentVolume, usually shortened to PV. It is the record that tells the cluster "this piece of real storage exists, this is how big it is, this is how it can be used, and this is what to do with it when nobody needs it any more". Developers rarely touch PVs directly, but when a database will not start, a claim stays Pending, or a disk keeps costing money after a project is deleted, the PV is where the answer lives.
This article is a close look at the PersistentVolume object itself. It complements the broader storage architecture article in this series by focusing on the PV specification, how binding works, the lifecycle phases, reclaim policies, backend types, node affinity and the practical procedures that administrators need: inspecting, reusing, moving and safely deleting volumes.
2. What Is a PersistentVolume?
A PersistentVolume is a cluster-scoped API object that represents a piece of storage which has been provisioned either by an administrator or automatically by a StorageClass. It has a lifecycle that is independent of any Pod that uses it. The PV object holds the details of the storage implementation, such as an NFS export, a cloud disk handled by a CSI driver, or a directory on a specific node.
Three objects work together, and it helps to be clear about who owns what:
A Pod never references a PV directly. It references a claim, and the claim is bound to a PV. This indirection is what makes Pod specifications portable between clusters whose storage differs completely.
3. Anatomy of a PersistentVolume
The manifest below describes a pre-existing cloud disk that an administrator is importing into the cluster through a CSI driver. The driver name and volume handle are examples only; take the real values and supported fields from your driver's documentation.
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-data-01
labels:
tier: gold
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: fast-ssd
mountOptions:
- noatime
csi:
driver: ebs.csi.aws.com
volumeHandle: vol-0123456789abcdef0
fsType: ext4
Two pieces of metadata are also worth knowing. PVs created by a provisioner carry the annotation pv.kubernetes.io/provisioned-by, which names the driver that made them, and PVs bound automatically carry pv.kubernetes.io/bound-by-controller.
4. How Binding Works
A control loop in the Kubernetes controller manager, often called the PV controller, watches for unbound claims and looks for a PV that satisfies them. A PV is a candidate only if all of the following hold:
When several PVs qualify, the controller picks the smallest one that is large enough. Binding is exclusive and one-to-one: once bound, a PV serves exactly one claim, even if the claim uses only part of the capacity. A claim for 10Gi can therefore end up with a 100Gi PV if that is the smallest match, and the rest of the space is simply not available to anyone else. If no PV matches, the claim stays Pending and is rechecked as new PVs appear.
4.1 Reserving a PV for a Specific Claim
Sometimes you want a particular PV to go to a particular claim, for example when importing an existing database disk. You can pre-bind in either direction. On the PV, set claimRef with the claim's namespace and name:
spec:
claimRef:
namespace: shop
name: app-data
Or on the claim, name the PV with volumeName:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-data
namespace: shop
spec:
storageClassName: fast-ssd
volumeName: pv-data-01
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
The remaining criteria, such as class, access mode and size, must still match. Pre-binding reserves the PV so other claims cannot take it.
5. Static and Dynamic Provisioning
With dynamic provisioning, you never write the PV manifest. After a claim binds, run kubectl get pv and you will find a generated PV with a name such as pvc- followed by a unique identifier, the class of the claim, the size requested, and a reclaim policy copied from the StorageClass. Treat these generated PVs as managed objects: change them only deliberately, for example to switch the reclaim policy to Retain.
6. PersistentVolume Phases
The status.phase field of a PV tells you where it is in its lifecycle:
A typical path is Available, then Bound when a claim arrives, then Released when the claim is deleted, after which the reclaim policy decides the outcome: the PV and backend volume are deleted (Delete), or kept in the Released state for an administrator (Retain).
7. Reclaim Policies in Depth
7.1 Delete
When the claim is deleted, Kubernetes deletes the PV object and asks the plugin to delete the backend volume. This is the typical default for dynamically provisioned storage. It keeps costs under control but makes accidental deletion of a claim a data-loss event. Use it for caches, build workspaces and other data you can recreate.
7.2 Retain
The PV and its data survive claim deletion. This is the safer choice for databases and anything irreplaceable. The price is manual cleanup: Retained volumes and their backend disks continue to consume capacity and incur cost until an administrator removes them. Make sure that someone owns that clean-up task.
7.3 Changing the Policy on a Live PV
You can change the policy at any time, for example to protect a dynamically provisioned volume before a risky operation:
kubectl patch pv <pv-name> \
-p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
7.4 Reusing a Released PV
A Released PV with Retain is still tied to its old claim. To make it available again, remove the claim reference:
kubectl patch pv <pv-name> -p '{"spec":{"claimRef":null}}'
The PV returns to Available, but the data is still on the volume. A new claim that binds to it will see the old contents. If that is not what you want, wipe or recreate the backend volume first. Never assume that a reused PV is empty.
8. Volume Sources and Backends
The source section of the PV says where the storage really is. The most important types today are:
8.1 Local PersistentVolumes
Local volumes give excellent performance because the disk is attached to the node, and they suit workloads that replicate data themselves. Because the data cannot follow the Pod to another node, Kubernetes needs to know which node owns each volume. A local PV must declare nodeAffinity, and the StorageClass should delay binding until a Pod is scheduled:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-local-node1
spec:
capacity:
storage: 200Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /mnt/disks/ssd1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- worker-node-1
The provisioner kubernetes.io/no-provisioner means that Kubernetes does not create these volumes itself. A separate local static provisioner project exists to discover disks on nodes and create PVs for them automatically. Remember that if the node is lost, the Pod using the volume cannot be rescheduled elsewhere with its data.
9. Node Affinity and Topology
The nodeAffinity field restricts which nodes can access the volume. For local volumes it pins the PV to a node. For zonal cloud disks, provisioners set it automatically so that the Pod is scheduled in the zone that holds the disk. The scheduler takes this affinity into account together with the Pod's own constraints.
If the constraints conflict, for example a Pod pinned to a zone that is different from the PV's zone, the Pod stays Pending with the event "volume node affinity conflict". For statically created zonal PVs, set nodeAffinity yourself using the topology label keys that your CSI driver and nodes use, which you can see on an existing PV created by the same driver. Mistakes here are a common source of Pending Pods.
10. Inspecting PersistentVolumes
kubectl get pv
kubectl get pv -o wide
kubectl describe pv <pv-name>
kubectl get pv <pv-name> -o yaml
The describe output shows the status, the bound claim, the reclaim policy, the access modes, the node affinity, the volume source and events. A custom view makes fleet-wide checks easy:
kubectl get pv -o custom-columns=\
NAME:.metadata.name,\
SIZE:.spec.capacity.storage,\
CLASS:.spec.storageClassName,\
RECLAIM:.spec.persistentVolumeReclaimPolicy,\
STATUS:.status.phase,\
CLAIM:.spec.claimRef.name,\
NAMESPACE:.spec.claimRef.namespace
Use it to find Released and Available volumes that may be orphaned and still costing money, and to confirm the reclaim policy of important data volumes. If you run kube-state-metrics, the kube_persistentvolume_status_phase metric allows you to alert when PVs are stuck in Released or Failed.
11. Moving a Volume to Another Namespace
Because a PV is cluster-scoped and a claim is namespaced, you can hand an existing volume from one namespace to another within the same cluster without copying data. Plan a maintenance window, because the workload must be stopped. The procedure is:
Stop every Pod that uses the claim, for example by scaling the workload to zero, so that nothing writes to the volume.
Set the PV's reclaim policy to Retain, as shown in Section 7.3, so that deleting the claim does not delete the data.
Delete the old claim. The PV moves to Released.
Clear the PV's claimRef so it becomes Available, as shown in Section 7.4.
Create a new claim in the target namespace with volumeName set to the PV name, and the same class, a compatible access mode and a size no larger than the PV capacity.
Confirm that the new claim is Bound, start the workload against it, and verify the data.
Set the reclaim policy back to the value you want for the long term.
This works across namespaces but not across clusters. Moving data to another cluster requires a snapshot-and-restore or a backup tool.
12. Protection and Safe Deletion
Kubernetes applies the finalizer kubernetes.io/pv-protection to PVs, which prevents a PV that is bound to a claim from being removed. If you delete a bound PV object, it stays in the Terminating state until the claim releases it. CSI sidecars may add their own finalizers to ensure that the backend volume is cleaned up before the object disappears.
Follow these rules for safe deletion:
Delete the claim, not the PV, when you want to release a dynamically provisioned volume. The reclaim policy then does the right thing.
Before deleting, check the reclaim policy. With Delete, the backend data will be removed; with Retain, you will need to clean up the backend yourself.
Do not remove finalizers by hand to force-delete a PV unless you have confirmed that nothing uses the volume and you know the state of the backend, because this can leave orphaned disks or break the attachment bookkeeping.
13. Troubleshooting PersistentVolumes
Always read the events first, with kubectl describe on the PV, the claim and the Pod, because they usually name the exact cause.
14. Best Practices
Prefer dynamic provisioning for normal workloads and keep manually created PVs for imports, NFS and local storage.
Set the reclaim policy deliberately: Retain for irreplaceable data, Delete for disposable data. Do not rely on the default.
Give PVs and StorageClasses clear names and labels, such as the owning team or tier, so that orphaned volumes can be traced.
Review Released, Available and Failed PVs regularly and clean them up, to avoid paying for unused disks.
Use WaitForFirstConsumer for zonal and local storage so that volumes land where Pods can actually run.
Restrict permissions to create and edit PVs with RBAC, because PVs are cluster-wide and can expose sensitive node paths or storage systems.
Avoid hostPath for anything but single-node testing, and design local PV workloads to replicate their own data.
Back up data independently of the PV, because the PV is only a pointer to storage and does not protect the contents.
15. Conclusion
A PersistentVolume is a small object with a large responsibility: it connects a Kubernetes cluster to real storage and defines how that storage may be matched, accessed, protected and eventually released. Once you understand the binding criteria, the four lifecycle phases and the two reclaim policies, most storage problems reduce to reading a few fields.
Use dynamic provisioning for everyday work, choose reclaim policies on purpose, and keep an eye on volumes that outlive their claims. With those habits, PersistentVolumes stop being a source of surprises and become the dependable foundation for stateful workloads.