Kubernetes Storage Architecture: Volumes, Persistent Volumes and Claims
Volumes, Persistent Volumes and Claims
Kubernetes Storage Architecture: Volumes, Persistent Volumes and Claims
bitcodematrix.com | Kubernetes Series
1. Introduction
Containers are disposable by design. When a container restarts, its writable layer is gone, and when a Pod is rescheduled to another node, anything it wrote locally is lost. That model suits stateless web servers but not databases, message queues, file stores or any workload that must remember something. Kubernetes solves this with a storage architecture built from volumes, persistent volumes, persistent volume claims, storage classes and the Container Storage Interface (CSI).
This article explains each of those building blocks and how they fit together. You will learn the difference between a volume and a persistent volume, how a claim becomes a mounted disk, what happens inside a CSI driver, and how to operate storage in production, including expansion, snapshots, topology and troubleshooting.
2. Why Kubernetes Separates Storage Concerns
Kubernetes treats storage as a contract between two groups of people. Application developers want to say "I need 10 GiB of fast storage that one Pod can write to", without knowing whether that is an EBS volume, a Ceph image or an NFS export. Cluster administrators and platform teams decide which storage technologies exist, how they are provisioned and what they cost.
The API objects reflect this split. The claim (PersistentVolumeClaim) is the developer's request. The persistent volume (PersistentVolume) is the actual piece of storage. The StorageClass describes how to create storage on demand. Keeping them separate makes Pod specifications portable across clusters and clouds.
3. Volumes: The Pod-Level Building Block
A volume in Kubernetes is a directory that is made available to the containers of a Pod through the volumeMounts field. Unlike the container's own filesystem, a volume outlives individual container restarts, and all containers in a Pod can share it. What happens to the data after the Pod is gone depends on the volume type.
A simple example is an emptyDir shared between two containers, which is useful for sharing files between an application and a sidecar:
apiVersion: v1
kind: Pod
metadata:
name: shared-scratch
spec:
containers:
- name: writer
image: busybox:1.36
command: ["sh", "-c", "date > /work/out.txt; sleep 3600"]
volumeMounts:
- name: work
mountPath: /work
- name: reader
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: work
mountPath: /work
volumes:
- name: work
emptyDir:
sizeLimit: 1Gi
Be careful with hostPath. It exposes the node's filesystem to the container, ties the Pod to one node and is a common route to privilege escalation. Reserve it for system-level agents and avoid it for application data. Over the years, most in-tree cloud disk volume types were removed from Kubernetes in favour of CSI drivers, so durable storage should now be consumed through claims.
4. PersistentVolumes and PersistentVolumeClaims
A volume defined inside a Pod spec is tightly coupled to that Pod. Persistent storage needs a lifecycle of its own, which is what these two objects provide:
A PersistentVolume (PV) is a piece of storage in the cluster. It is a cluster-scoped resource, not tied to a namespace, and it records the capacity, access modes, reclaim policy and backend details.
A PersistentVolumeClaim (PVC) is a namespaced request for storage by a user. It states the size, access mode and optionally the StorageClass.
A useful analogy is that Pods consume node resources and PVCs consume PV resources. When a claim is created, Kubernetes finds or creates a PV that satisfies it, and binds the two together in a one-to-one relationship. A Pod then references the claim by name, never the PV directly.
4.1 Static Provisioning
In static provisioning, an administrator creates PVs in advance and users claim them. This suits existing storage that must be exposed to Kubernetes, such as a pre-existing NFS export:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-nfs-data
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: manual
nfs:
server: 10.0.0.5
path: /exports/data
A claim with a matching storageClassName, a compatible access mode and a request no larger than 50Gi can bind to this PV. For dynamic provisioning, which is the norm in modern clusters, see Section 6.
5. Access Modes and Volume Modes
Access modes describe how many nodes or Pods may use a volume at once. They are a statement about capability that the storage backend must support, not a restriction Kubernetes enforces on its own in every case.
Block storage such as a cloud disk is usually RWO. If a Deployment with several replicas needs shared writable storage, you need an RWX-capable backend, not simply a bigger disk.
The volumeMode field is separate. The default, Filesystem, formats the volume and mounts it as a directory. Block exposes the raw device to the container, which some databases use to manage the device directly. Block volumes are attached with volumeDevices rather than volumeMounts.
6. StorageClasses and Dynamic Provisioning
A StorageClass describes a class of storage and how to create it on demand. With dynamic provisioning, administrators do not hand-create PVs. A developer submits a PVC naming a class, and the matching provisioner creates the backend volume and the PV automatically.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
The provisioner and parameters shown here are for the AWS EBS CSI driver and are only an example. Every driver defines its own provisioner name and parameters, so take them from the driver's documentation. The fields that matter in every StorageClass are these:
One StorageClass can be marked as the cluster default with the annotation storageclass.kubernetes.io/is-default-class set to "true". A PVC that omits storageClassName receives the default class. Setting storageClassName to an empty string disables dynamic provisioning for that claim and requests a statically created PV instead.
7. The CSI Architecture
The Container Storage Interface is the standard through which Kubernetes talks to storage systems. Storage vendors implement a CSI driver once and it works with any orchestrator that supports CSI. The driver is deployed in the cluster as two cooperating parts:
The controller plugin, usually a Deployment, handles cluster-level operations such as creating, deleting, attaching, snapshotting and resizing volumes in the storage backend. Kubernetes sidecar containers (external-provisioner, external-attacher, external-resizer and external-snapshotter) watch the API and call the driver.
The node plugin, a DaemonSet, runs on every node and performs the local work of staging and mounting volumes into Pods. A node-driver-registrar sidecar registers it with the kubelet.
Two API objects describe drivers: CSIDriver declares a driver and its capabilities (for example whether it needs an attach step), and CSINode records which drivers are available on each node.
7.1 From Claim to Mounted Volume
The complete flow for a dynamically provisioned volume works as follows:
A user creates a PVC that references a StorageClass.
If the class uses WaitForFirstConsumer, Kubernetes waits until a Pod that uses the claim is scheduled. This lets the scheduler choose a node first, so the volume is created in a compatible zone.
The external-provisioner sidecar sees the pending claim and calls the driver's CreateVolume operation, which creates the volume in the storage backend.
Kubernetes creates a PV object that describes the new volume and binds it to the PVC. The claim moves to the Bound phase.
If the driver requires attachment, the volume is attached to the node that runs the Pod (the controller publish step, tracked by a VolumeAttachment object).
On the node, the kubelet asks the node plugin to stage the volume (format it if needed and mount it at a staging path) and then publish it, bind-mounting it into the Pod's directory.
The container starts with the volume mounted at the requested path.
Deletion runs in reverse: the volume is unmounted and detached, and then the backend volume is removed or kept according to the reclaim policy.
8. The Lifecycle of a PersistentVolume
A PV passes through a small set of phases:
8.1 Reclaim Policies
Delete: deleting the PVC also deletes the PV and the backend storage. This is the usual default for dynamically provisioned volumes and is convenient, but it destroys data when someone removes a claim by mistake.
Retain: the PV and its data are kept after the claim is deleted. The PV shows Released and is not reused automatically, because it still holds a reference to the old claim and old data. An administrator must clean it up or remove the reference manually.
A third value, Recycle, performed a basic scrub of the volume and has been deprecated in favour of dynamic provisioning. For important data, use Retain in the StorageClass or patch the policy on the PV.
8.2 Storage Object in Use Protection
Kubernetes places finalizers on PVCs and PVs (kubernetes.io/pvc-protection and kubernetes.io/pv-protection) so that a claim that is still mounted by a Pod cannot be removed out from under it. A PVC deleted while in use shows as Terminating until the last Pod using it is gone. This is expected behaviour, not a fault.
9. Hands-On: Create and Use a Claim
With the StorageClass from Section 6 in place, request storage with a claim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-data
namespace: shop
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 10Gi
Then mount it in a Pod by referencing the claim name:
apiVersion: v1
kind: Pod
metadata:
name: app
namespace: shop
spec:
containers:
- name: app
image: nginx:1.27
volumeMounts:
- name: data
mountPath: /var/lib/app
volumes:
- name: data
persistentVolumeClaim:
claimName: app-data
Verify the result:
kubectl get storageclass
kubectl get pvc -n shop
kubectl get pv
kubectl describe pvc app-data -n shop
kubectl exec -n shop app -- df -h /var/lib/app
With WaitForFirstConsumer, the claim legitimately stays in Pending until the Pod is created and scheduled. After that, it should show Bound with a PV name, and the df output should show the requested capacity (slightly less because of file system overhead).
10. StatefulSets and Per-Replica Storage
A Deployment shares one claim among all replicas, which is rarely what a database cluster needs. A StatefulSet uses volumeClaimTemplates to give each replica its own claim:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 3
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: db
image: postgres:16
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
This creates claims named data-db-0, data-db-1 and data-db-2. Each Pod keeps its own volume across restarts and rescheduling, and the Pod with a given ordinal always reattaches to the claim with the same ordinal. By default, the claims are not deleted when the StatefulSet is deleted or scaled down, which protects data. The persistentVolumeClaimRetentionPolicy field lets you change this behaviour deliberately. The example omits required settings such as database credentials for brevity.
11. Day-2 Operations
11.1 Expanding a Volume
If the StorageClass sets allowVolumeExpansion to true and the CSI driver supports it, you can grow a claim by increasing its requested size:
kubectl patch pvc app-data -n shop \
-p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
The controller expands the backend volume and the file system is then resized, often online while the Pod keeps running, depending on the driver. Watch the claim conditions for a status such as FileSystemResizePending. Shrinking a volume is not supported.
11.2 Snapshots
Volume snapshots capture the state of a volume at a point in time. They use three objects, VolumeSnapshotClass, VolumeSnapshot and VolumeSnapshotContent, which are Custom Resource Definitions that need to be installed together with the snapshot controller, and a CSI driver that supports snapshots.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: app-data-snap
namespace: shop
spec:
volumeSnapshotClassName: csi-snapclass
source:
persistentVolumeClaimName: app-data
To restore, create a new claim whose dataSource points at the snapshot:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-data-restored
namespace: shop
spec:
storageClassName: fast-ssd
dataSource:
name: app-data-snap
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
A dataSource that points at another PVC in the same namespace creates a clone in the same way, provided the driver supports it.
11.3 Snapshots Are Not Backups
A snapshot usually lives in the same storage system as the original volume, so it does not protect against losing that system or the account. A real backup strategy copies data off-cluster, using a tool such as Velero for Kubernetes-level backups or the database's own backup mechanism, and tests restores regularly.
12. Topology and Scheduling
Most block storage is zonal: a disk created in one availability zone can be attached only to nodes in that zone. If the volume is created before the scheduler picks a node, the Pod can later be forced to a zone where it cannot run. That is why WaitForFirstConsumer is recommended for zonal storage: the scheduler first selects a node, and the volume is then provisioned in that node's zone.
The resulting PV carries a nodeAffinity section that records where it can be used, and the scheduler respects it. When a Pod cannot be scheduled because of this, the event reads "volume node affinity conflict". Typical causes are a Pod that has been pinned to a different zone by its own affinity or nodeSelector rules, or a zone with no remaining node capacity. Design your node pools and spreading rules with storage zones in mind.
13. Security Considerations
File permissions. A new volume is often owned by root. Use the Pod securityContext fields fsGroup and runAsUser so that the application can write, and set fsGroupChangePolicy to OnRootMismatch to avoid slow recursive ownership changes on large volumes.
Avoid hostPath for application data, and restrict it with admission policies such as Pod Security Standards.
Namespace isolation. A PVC is namespaced, but PVs are not. Use RBAC to limit who can create PVs and StorageClasses, and use ResourceQuota to cap the storage a namespace can request.
Encryption. Enable encryption at rest through StorageClass parameters or the storage backend, and encrypt traffic to network storage where the backend supports it.
Secrets. Secret volumes are held in memory-backed storage on the node and not written to disk, but treat the Secret objects themselves with care.
spec:
securityContext:
runAsUser: 1000
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
14. Troubleshooting Storage
Start with the claim and the events, because Kubernetes records most storage problems there:
kubectl get pvc,pv -A
kubectl describe pvc <name> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get volumeattachments
kubectl get csidrivers
kubectl logs -n <driver-namespace> <csi-controller-pod> -c csi-provisioner
15. Best Practices
Use dynamic provisioning with well-named StorageClasses, such as fast-ssd and standard, and define a clear default.
Use volumeBindingMode WaitForFirstConsumer for zonal storage.
Use Retain for critical data, and always have a tested backup in addition to snapshots.
Enable allowVolumeExpansion, because growing a volume is much easier than migrating data.
Request the access mode you actually need. Prefer RWO or RWOP for single-writer databases and RWX only when sharing is required.
Use StatefulSets with volumeClaimTemplates for stateful applications that need per-replica storage.
Set ResourceQuota for storage per namespace, and monitor volume usage with metrics such as kubelet_volume_stats_used_bytes, alerting well before volumes fill up.
Keep CSI drivers, snapshot CRDs and sidecars updated according to the driver vendor's compatibility matrix, and test upgrades in non-production clusters first.
Avoid hostPath and local volumes for important data unless you fully accept the node-failure consequences.
16. Conclusion
Kubernetes storage is easier to reason about once you see its layers. Volumes give Pods directories. Persistent volumes and claims decouple durable storage from Pods and separate what applications ask for from what administrators provide. StorageClasses make provisioning on demand, and CSI drivers do the real work of creating, attaching and mounting volumes on the storage backend.
With that model, most problems become diagnosable by following the chain: claim, class, provisioner, PV, attachment, mount. Design with topology, reclaim policies and backups in mind, and your stateful workloads can be as reliable on Kubernetes as the storage beneath them.