kubernetes

Kubernetes Storage Architecture: Volumes, Persistent Volumes and Claims

Volumes, Persistent Volumes and Claims

By Shubhankar Tripathi • • 5 min read

Kubernetes Storage Architecture: Volumes, Persistent Volumes and Claims

bitcodematrix.com | Kubernetes Series

1. Introduction

Containers are disposable by design. When a container restarts, its writable layer is gone, and when a Pod is rescheduled to another node, anything it wrote locally is lost. That model suits stateless web servers but not databases, message queues, file stores or any workload that must remember something. Kubernetes solves this with a storage architecture built from volumes, persistent volumes, persistent volume claims, storage classes and the Container Storage Interface (CSI).

This article explains each of those building blocks and how they fit together. You will learn the difference between a volume and a persistent volume, how a claim becomes a mounted disk, what happens inside a CSI driver, and how to operate storage in production, including expansion, snapshots, topology and troubleshooting.

2. Why Kubernetes Separates Storage Concerns

Kubernetes treats storage as a contract between two groups of people. Application developers want to say "I need 10 GiB of fast storage that one Pod can write to", without knowing whether that is an EBS volume, a Ceph image or an NFS export. Cluster administrators and platform teams decide which storage technologies exist, how they are provisioned and what they cost.

The API objects reflect this split. The claim (PersistentVolumeClaim) is the developer's request. The persistent volume (PersistentVolume) is the actual piece of storage. The StorageClass describes how to create storage on demand. Keeping them separate makes Pod specifications portable across clusters and clouds.

3. Volumes: The Pod-Level Building Block

A volume in Kubernetes is a directory that is made available to the containers of a Pod through the volumeMounts field. Unlike the container's own filesystem, a volume outlives individual container restarts, and all containers in a Pod can share it. What happens to the data after the Pod is gone depends on the volume type.

Volume type

What it provides

Lifetime of the data

emptyDir

Empty scratch directory on the node, or in memory when medium is set to Memory

Deleted when the Pod is removed from the node; survives container restarts

configMap, secret, downwardAPI, projected

Injects configuration, credentials or Pod metadata as files

Follows the source object; not meant for application data

hostPath

Mounts a file or directory from the node's filesystem

Stays on the node; tied to that node and a security risk

persistentVolumeClaim

Mounts durable storage requested through a claim

Independent of the Pod, governed by the PV reclaim policy

ephemeral (generic ephemeral volume)

Creates a claim automatically for the Pod using a template

Deleted together with the Pod

CSI and network types (for example nfs)

Mounts storage provided by a driver or network file system

Depends on the backend

 

A simple example is an emptyDir shared between two containers, which is useful for sharing files between an application and a sidecar:

apiVersion: v1

kind: Pod

metadata:

  name: shared-scratch

spec:

  containers:

  - name: writer

    image: busybox:1.36

    command: ["sh", "-c", "date > /work/out.txt; sleep 3600"]

    volumeMounts:

    - name: work

      mountPath: /work

  - name: reader

    image: busybox:1.36

    command: ["sh", "-c", "sleep 3600"]

    volumeMounts:

    - name: work

      mountPath: /work

  volumes:

  - name: work

    emptyDir:

      sizeLimit: 1Gi

Be careful with hostPath. It exposes the node's filesystem to the container, ties the Pod to one node and is a common route to privilege escalation. Reserve it for system-level agents and avoid it for application data. Over the years, most in-tree cloud disk volume types were removed from Kubernetes in favour of CSI drivers, so durable storage should now be consumed through claims.

4. PersistentVolumes and PersistentVolumeClaims

A volume defined inside a Pod spec is tightly coupled to that Pod. Persistent storage needs a lifecycle of its own, which is what these two objects provide:

  • A PersistentVolume (PV) is a piece of storage in the cluster. It is a cluster-scoped resource, not tied to a namespace, and it records the capacity, access modes, reclaim policy and backend details.

  • A PersistentVolumeClaim (PVC) is a namespaced request for storage by a user. It states the size, access mode and optionally the StorageClass.

A useful analogy is that Pods consume node resources and PVCs consume PV resources. When a claim is created, Kubernetes finds or creates a PV that satisfies it, and binds the two together in a one-to-one relationship. A Pod then references the claim by name, never the PV directly.

Aspect

PersistentVolume

PersistentVolumeClaim

Scope

Cluster-wide

Namespaced

Created by

Administrator, or a provisioner automatically

Application developer or a StatefulSet template

Describes

The real storage and its properties

The request: size, access mode, class

Referenced by Pods

No

Yes, by claimName

 

4.1 Static Provisioning

In static provisioning, an administrator creates PVs in advance and users claim them. This suits existing storage that must be exposed to Kubernetes, such as a pre-existing NFS export:

apiVersion: v1

kind: PersistentVolume

metadata:

  name: pv-nfs-data

spec:

  capacity:

    storage: 50Gi

  accessModes:

  - ReadWriteMany

  persistentVolumeReclaimPolicy: Retain

  storageClassName: manual

  nfs:

    server: 10.0.0.5

    path: /exports/data

A claim with a matching storageClassName, a compatible access mode and a request no larger than 50Gi can bind to this PV. For dynamic provisioning, which is the norm in modern clusters, see Section 6.

5. Access Modes and Volume Modes

Access modes describe how many nodes or Pods may use a volume at once. They are a statement about capability that the storage backend must support, not a restriction Kubernetes enforces on its own in every case.

Access mode

Short form

Meaning

ReadWriteOnce

RWO

Mounted read-write by a single node; several Pods on that node can use it

ReadOnlyMany

ROX

Mounted read-only by many nodes

ReadWriteMany

RWX

Mounted read-write by many nodes; needs a shared file system such as NFS, CephFS or a cloud file service

ReadWriteOncePod

RWOP

Mounted read-write by a single Pod in the whole cluster (generally available since Kubernetes 1.29, CSI volumes only)

 

Block storage such as a cloud disk is usually RWO. If a Deployment with several replicas needs shared writable storage, you need an RWX-capable backend, not simply a bigger disk.

The volumeMode field is separate. The default, Filesystem, formats the volume and mounts it as a directory. Block exposes the raw device to the container, which some databases use to manage the device directly. Block volumes are attached with volumeDevices rather than volumeMounts.

6. StorageClasses and Dynamic Provisioning

A StorageClass describes a class of storage and how to create it on demand. With dynamic provisioning, administrators do not hand-create PVs. A developer submits a PVC naming a class, and the matching provisioner creates the backend volume and the PV automatically.

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: fast-ssd

provisioner: ebs.csi.aws.com

parameters:

  type: gp3

reclaimPolicy: Delete

allowVolumeExpansion: true

volumeBindingMode: WaitForFirstConsumer

The provisioner and parameters shown here are for the AWS EBS CSI driver and are only an example. Every driver defines its own provisioner name and parameters, so take them from the driver's documentation. The fields that matter in every StorageClass are these:

Field

Purpose

provisioner

The CSI driver (or other provisioner) that creates volumes for this class

parameters

Driver-specific settings such as disk type, IOPS, encryption or file system type

reclaimPolicy

What happens to the backend volume when the claim is deleted: Delete (the default for dynamically provisioned volumes) or Retain

allowVolumeExpansion

Whether claims of this class can be resized after creation

volumeBindingMode

Immediate provisions at once; WaitForFirstConsumer waits until a Pod using the claim is scheduled

mountOptions

Mount options passed to the file system mount

allowedTopologies

Restricts where volumes may be provisioned, such as specific zones

 

One StorageClass can be marked as the cluster default with the annotation storageclass.kubernetes.io/is-default-class set to "true". A PVC that omits storageClassName receives the default class. Setting storageClassName to an empty string disables dynamic provisioning for that claim and requests a statically created PV instead.

7. The CSI Architecture

The Container Storage Interface is the standard through which Kubernetes talks to storage systems. Storage vendors implement a CSI driver once and it works with any orchestrator that supports CSI. The driver is deployed in the cluster as two cooperating parts:

  • The controller plugin, usually a Deployment, handles cluster-level operations such as creating, deleting, attaching, snapshotting and resizing volumes in the storage backend. Kubernetes sidecar containers (external-provisioner, external-attacher, external-resizer and external-snapshotter) watch the API and call the driver.

  • The node plugin, a DaemonSet, runs on every node and performs the local work of staging and mounting volumes into Pods. A node-driver-registrar sidecar registers it with the kubelet.

Two API objects describe drivers: CSIDriver declares a driver and its capabilities (for example whether it needs an attach step), and CSINode records which drivers are available on each node.

7.1 From Claim to Mounted Volume

The complete flow for a dynamically provisioned volume works as follows:

  1. A user creates a PVC that references a StorageClass.

  2. If the class uses WaitForFirstConsumer, Kubernetes waits until a Pod that uses the claim is scheduled. This lets the scheduler choose a node first, so the volume is created in a compatible zone.

  3. The external-provisioner sidecar sees the pending claim and calls the driver's CreateVolume operation, which creates the volume in the storage backend.

  4. Kubernetes creates a PV object that describes the new volume and binds it to the PVC. The claim moves to the Bound phase.

  5. If the driver requires attachment, the volume is attached to the node that runs the Pod (the controller publish step, tracked by a VolumeAttachment object).

  6. On the node, the kubelet asks the node plugin to stage the volume (format it if needed and mount it at a staging path) and then publish it, bind-mounting it into the Pod's directory.

  7. The container starts with the volume mounted at the requested path.

Deletion runs in reverse: the volume is unmounted and detached, and then the backend volume is removed or kept according to the reclaim policy.

8. The Lifecycle of a PersistentVolume

A PV passes through a small set of phases:

Phase

Meaning

Available

The PV is free and not yet bound to a claim

Bound

The PV is bound to a PVC

Released

The claim was deleted but the PV has not yet been reclaimed; with Retain, the data is still there

Failed

Automatic reclamation failed

 

8.1 Reclaim Policies

  • Delete: deleting the PVC also deletes the PV and the backend storage. This is the usual default for dynamically provisioned volumes and is convenient, but it destroys data when someone removes a claim by mistake.

  • Retain: the PV and its data are kept after the claim is deleted. The PV shows Released and is not reused automatically, because it still holds a reference to the old claim and old data. An administrator must clean it up or remove the reference manually.

A third value, Recycle, performed a basic scrub of the volume and has been deprecated in favour of dynamic provisioning. For important data, use Retain in the StorageClass or patch the policy on the PV.

8.2 Storage Object in Use Protection

Kubernetes places finalizers on PVCs and PVs (kubernetes.io/pvc-protection and kubernetes.io/pv-protection) so that a claim that is still mounted by a Pod cannot be removed out from under it. A PVC deleted while in use shows as Terminating until the last Pod using it is gone. This is expected behaviour, not a fault.

9. Hands-On: Create and Use a Claim

With the StorageClass from Section 6 in place, request storage with a claim:

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: app-data

  namespace: shop

spec:

  accessModes:

  - ReadWriteOnce

  storageClassName: fast-ssd

  resources:

    requests:

      storage: 10Gi

Then mount it in a Pod by referencing the claim name:

apiVersion: v1

kind: Pod

metadata:

  name: app

  namespace: shop

spec:

  containers:

  - name: app

    image: nginx:1.27

    volumeMounts:

    - name: data

      mountPath: /var/lib/app

  volumes:

  - name: data

    persistentVolumeClaim:

      claimName: app-data

Verify the result:

kubectl get storageclass

kubectl get pvc -n shop

kubectl get pv

kubectl describe pvc app-data -n shop

kubectl exec -n shop app -- df -h /var/lib/app

With WaitForFirstConsumer, the claim legitimately stays in Pending until the Pod is created and scheduled. After that, it should show Bound with a PV name, and the df output should show the requested capacity (slightly less because of file system overhead).

10. StatefulSets and Per-Replica Storage

A Deployment shares one claim among all replicas, which is rarely what a database cluster needs. A StatefulSet uses volumeClaimTemplates to give each replica its own claim:

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: db

spec:

  serviceName: db

  replicas: 3

  selector:

    matchLabels:

      app: db

  template:

    metadata:

      labels:

        app: db

    spec:

      containers:

      - name: db

        image: postgres:16

        volumeMounts:

        - name: data

          mountPath: /var/lib/postgresql/data

  volumeClaimTemplates:

  - metadata:

      name: data

    spec:

      accessModes: ["ReadWriteOnce"]

      storageClassName: fast-ssd

      resources:

        requests:

          storage: 20Gi

This creates claims named data-db-0, data-db-1 and data-db-2. Each Pod keeps its own volume across restarts and rescheduling, and the Pod with a given ordinal always reattaches to the claim with the same ordinal. By default, the claims are not deleted when the StatefulSet is deleted or scaled down, which protects data. The persistentVolumeClaimRetentionPolicy field lets you change this behaviour deliberately. The example omits required settings such as database credentials for brevity.

11. Day-2 Operations

11.1 Expanding a Volume

If the StorageClass sets allowVolumeExpansion to true and the CSI driver supports it, you can grow a claim by increasing its requested size:

kubectl patch pvc app-data -n shop \

  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

The controller expands the backend volume and the file system is then resized, often online while the Pod keeps running, depending on the driver. Watch the claim conditions for a status such as FileSystemResizePending. Shrinking a volume is not supported.

11.2 Snapshots

Volume snapshots capture the state of a volume at a point in time. They use three objects, VolumeSnapshotClass, VolumeSnapshot and VolumeSnapshotContent, which are Custom Resource Definitions that need to be installed together with the snapshot controller, and a CSI driver that supports snapshots.

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshot

metadata:

  name: app-data-snap

  namespace: shop

spec:

  volumeSnapshotClassName: csi-snapclass

  source:

    persistentVolumeClaimName: app-data

To restore, create a new claim whose dataSource points at the snapshot:

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: app-data-restored

  namespace: shop

spec:

  storageClassName: fast-ssd

  dataSource:

    name: app-data-snap

    kind: VolumeSnapshot

    apiGroup: snapshot.storage.k8s.io

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: 10Gi

A dataSource that points at another PVC in the same namespace creates a clone in the same way, provided the driver supports it.

11.3 Snapshots Are Not Backups

A snapshot usually lives in the same storage system as the original volume, so it does not protect against losing that system or the account. A real backup strategy copies data off-cluster, using a tool such as Velero for Kubernetes-level backups or the database's own backup mechanism, and tests restores regularly.

12. Topology and Scheduling

Most block storage is zonal: a disk created in one availability zone can be attached only to nodes in that zone. If the volume is created before the scheduler picks a node, the Pod can later be forced to a zone where it cannot run. That is why WaitForFirstConsumer is recommended for zonal storage: the scheduler first selects a node, and the volume is then provisioned in that node's zone.

The resulting PV carries a nodeAffinity section that records where it can be used, and the scheduler respects it. When a Pod cannot be scheduled because of this, the event reads "volume node affinity conflict". Typical causes are a Pod that has been pinned to a different zone by its own affinity or nodeSelector rules, or a zone with no remaining node capacity. Design your node pools and spreading rules with storage zones in mind.

13. Security Considerations

  • File permissions. A new volume is often owned by root. Use the Pod securityContext fields fsGroup and runAsUser so that the application can write, and set fsGroupChangePolicy to OnRootMismatch to avoid slow recursive ownership changes on large volumes.

  • Avoid hostPath for application data, and restrict it with admission policies such as Pod Security Standards.

  • Namespace isolation. A PVC is namespaced, but PVs are not. Use RBAC to limit who can create PVs and StorageClasses, and use ResourceQuota to cap the storage a namespace can request.

  • Encryption. Enable encryption at rest through StorageClass parameters or the storage backend, and encrypt traffic to network storage where the backend supports it.

  • Secrets. Secret volumes are held in memory-backed storage on the node and not written to disk, but treat the Secret objects themselves with care.

spec:

  securityContext:

    runAsUser: 1000

    fsGroup: 2000

    fsGroupChangePolicy: OnRootMismatch

14. Troubleshooting Storage

Start with the claim and the events, because Kubernetes records most storage problems there:

kubectl get pvc,pv -A

kubectl describe pvc <name> -n <namespace>

kubectl describe pod <pod> -n <namespace>

kubectl get events -n <namespace> --sort-by=.lastTimestamp

kubectl get volumeattachments

kubectl get csidrivers

kubectl logs -n <driver-namespace> <csi-controller-pod> -c csi-provisioner

Symptom

Likely cause and check

PVC stuck in Pending

No default StorageClass or a wrong class name; the provisioner is not running; the class uses WaitForFirstConsumer and no Pod exists yet; quota exceeded. Read the events on the claim.

Pod stuck in ContainerCreating with FailedAttachVolume

The backend could not attach the volume: cloud permissions, a missing zone match or an attach limit on the node. Check the CSI controller logs.

Multi-Attach error

An RWO volume is still attached to another node, often after a node failure or while a Pod moves. Wait for detach, or investigate the old node and the VolumeAttachment.

FailedMount

The node plugin failed to mount: driver not running on that node, missing file system tools, or a wrong mount option. Check the node plugin logs.

Volume node affinity conflict

The PV is in a zone where the Pod cannot be scheduled. Review Pod affinity, nodeSelector and node pool zones.

PVC stuck in Terminating

A Pod still uses the claim, so the protection finalizer remains. Remove or stop the Pod.

PV stuck in Released

Retain policy: the PV still references the old claim. Clean the data and clear the claim reference, or create a new PV.

Expansion not completing

The class does not allow expansion, the driver lacks support, or a file system resize is pending until the Pod restarts.

Permission denied inside the container

Ownership mismatch. Set fsGroup or runAsUser appropriately.

No space left on device

The volume is full. Expand the claim and review application retention.

 

15. Best Practices

  • Use dynamic provisioning with well-named StorageClasses, such as fast-ssd and standard, and define a clear default.

  • Use volumeBindingMode WaitForFirstConsumer for zonal storage.

  • Use Retain for critical data, and always have a tested backup in addition to snapshots.

  • Enable allowVolumeExpansion, because growing a volume is much easier than migrating data.

  • Request the access mode you actually need. Prefer RWO or RWOP for single-writer databases and RWX only when sharing is required.

  • Use StatefulSets with volumeClaimTemplates for stateful applications that need per-replica storage.

  • Set ResourceQuota for storage per namespace, and monitor volume usage with metrics such as kubelet_volume_stats_used_bytes, alerting well before volumes fill up.

  • Keep CSI drivers, snapshot CRDs and sidecars updated according to the driver vendor's compatibility matrix, and test upgrades in non-production clusters first.

  • Avoid hostPath and local volumes for important data unless you fully accept the node-failure consequences.

16. Conclusion

Kubernetes storage is easier to reason about once you see its layers. Volumes give Pods directories. Persistent volumes and claims decouple durable storage from Pods and separate what applications ask for from what administrators provide. StorageClasses make provisioning on demand, and CSI drivers do the real work of creating, attaching and mounting volumes on the storage backend.

With that model, most problems become diagnosable by following the chain: claim, class, provisioner, PV, attachment, mount. Design with topology, reclaim policies and backups in mind, and your stateful workloads can be as reliable on Kubernetes as the storage beneath them.