kubernetes

StatefulSet and Persistent Storage

Kubernetes StatefulSet and Persistent Storage

By Shubhankar Tripathi • • 5 min read

Kubernetes StatefulSet and Persistent Storage

bitcodematrix.com | Kubernetes Series

1. Introduction

Deployments are perfect for applications whose Pods are interchangeable. Pods are created, destroyed and replaced freely, and nobody cares which replica handled a request. Databases, message brokers, distributed caches and coordination services are different: each member has an identity, its own data and often a specific role in a cluster. For these workloads Kubernetes provides the StatefulSet.

This article explains how StatefulSets work and how they combine with persistent storage. You will learn the guarantees they give, how to build one with a headless Service and volumeClaimTemplates, how identity, storage and ordering behave during scaling, updates and failures, and how to operate them safely. It builds on the earlier articles in this series on PersistentVolumes, PersistentVolumeClaims, StorageClasses and CSI.

2. Why Stateful Workloads Need More Than a Deployment

Consider a three-node database cluster. Each member must have a stable name that its peers can use to find it, its own disk that follows it across restarts, and a predictable start-up order so that a primary exists before replicas try to join. A Deployment cannot give these guarantees:

  • Pod names are random and change on every replacement.

  • All replicas share one Pod template and therefore one claim, so they cannot have separate disks.

  • Pods start and stop in parallel, with no ordering.

  • A replacement Pod is a new Pod with no relationship to the old one.

The StatefulSet controller manages a set of Pods like a Deployment does, but it adds the three missing properties: stable identity, stable storage and ordered operations.

3. The Guarantees of a StatefulSet

Property

Deployment

StatefulSet

Pod names

Random suffix, for example web-7d9f8c-xk2lp

Ordinal suffix: web-0, web-1, web-2

Network identity

Shared Service virtual IP only

Individual DNS name per Pod through a headless Service

Storage

One shared claim for all Pods, if any

A separate PVC for every Pod, created from a template

Replacement Pod

New identity

Same name, same DNS name and same PVC

Start and stop order

Parallel

Ordered by default: 0 first, highest ordinal removed first

Rolling update order

Any order

Reverse ordinal order, one Pod at a time by default

 

One thing a StatefulSet does not do is understand your application. It does not replicate data, elect a leader, or take backups. It provides the stable foundation, and the application, or an operator that manages it, builds the clustering behaviour on top.

4. Building a StatefulSet

A working StatefulSet always has two parts: a headless Service that provides the DNS identities, and the StatefulSet itself. The example below follows the pattern from the Kubernetes documentation, using a simple web server so that the behaviour is easy to observe.

apiVersion: v1

kind: Service

metadata:

  name: nginx

  labels:

    app: nginx

spec:

  clusterIP: None

  selector:

    app: nginx

  ports:

  - name: web

    port: 80

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: web

spec:

  serviceName: nginx

  replicas: 3

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      terminationGracePeriodSeconds: 30

      containers:

      - name: nginx

        image: registry.k8s.io/nginx-slim:0.24

        ports:

        - containerPort: 80

          name: web

        volumeMounts:

        - name: www

          mountPath: /usr/share/nginx/html

  volumeClaimTemplates:

  - metadata:

      name: www

    spec:

      accessModes: ["ReadWriteOnce"]

      storageClassName: fast-ssd

      resources:

        requests:

          storage: 1Gi

Four details deserve attention:

  • clusterIP: None makes the Service headless. It has no virtual IP, and DNS returns the individual Pod addresses instead.

  • serviceName must name that Service. The StatefulSet does not create it, so you must create it yourself.

  • volumeClaimTemplates defines the claim that each Pod receives. The name of the template, www here, is also the volume name used in volumeMounts.

  • storageClassName should be set explicitly. Remove it to use the cluster default class, and check that a class exists that can provision the volumes.

5. Stable Network Identity

Pods in a StatefulSet get the names web-0, web-1 and web-2, and each Pod's hostname equals its name. Together with the headless Service, every Pod also receives a stable DNS record:

web-0.nginx.<namespace>.svc.cluster.local

web-1.nginx.<namespace>.svc.cluster.local

web-2.nginx.<namespace>.svc.cluster.local

These names stay the same when a Pod is rescheduled, although its IP address may change. Applications should therefore use the DNS names, not the IP addresses, in their cluster configuration. You can verify them from inside the cluster:

kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm \

  -- nslookup web-0.nginx

The DNS name of the Service itself, nginx, returns the addresses of all ready Pods, which suits clients that want to discover every member.

A subtle point concerns readiness. By default, DNS records for the headless Service include only Pods that are ready. Clustered software often needs to find its peers before any member is ready, for example during the first bootstrap. Setting publishNotReadyAddresses to true on the headless Service makes DNS publish the records of unready Pods too, which solves this chicken-and-egg problem. Use it deliberately, because clients will then also be given unready addresses.

If clients need a single stable entry point to the cluster, for example to reach the current primary, create an additional ordinary Service with a selector that matches only the right Pods, which is a common pattern in database operators.

6. Persistent Storage for Every Replica

6.1 How the Claims Are Created

For each Pod, the StatefulSet controller creates a PersistentVolumeClaim from the template. The claim is named <template>-<statefulset>-<ordinal>, so the example creates www-web-0, www-web-1 and www-web-2. Each claim is provisioned by the StorageClass, bound to its own PV and mounted by exactly one Pod.

The relationship is stable. When Pod web-1 is deleted or its node fails, the replacement is again called web-1, and the controller mounts the existing claim www-web-1 into it. The Pod gets its old data back.

6.2 See It Work

  1. Apply the Service and the StatefulSet and wait until all Pods are ready: kubectl get pods -l app=nginx -w.

  2. List the claims with kubectl get pvc and confirm that three claims exist, one per Pod.

  3. Write a distinct file into each Pod's volume.

for i in 0 1 2; do

  kubectl exec web-$i -- sh -c 'echo "Hello from $(hostname)" > /usr/share/nginx/html/index.html'

done

  1. Delete a Pod, and let the controller recreate it.

kubectl delete pod web-0

kubectl get pods -l app=nginx -w

  1. Check that the new web-0 serves the same content as before.

kubectl exec web-0 -- cat /usr/share/nginx/html/index.html

The replacement Pod has the same name, and the file written earlier is still there, because the new Pod mounted the same claim.

6.3 What Happens to Claims on Scale-Down and Deletion

By default, claims outlive their Pods. Scaling a StatefulSet from three replicas to two removes web-2, but the claim www-web-2 stays. Scaling back up reattaches it to the new web-2, with whatever data it held, which may be stale. Deleting the StatefulSet also leaves the claims. This protects data but leaves storage that you pay for, which you must clean up deliberately.

The persistentVolumeClaimRetentionPolicy field lets you choose what happens in each situation:

spec:

  persistentVolumeClaimRetentionPolicy:

    whenDeleted: Retain

    whenScaled: Delete

Each setting can be Retain or Delete. Using Delete for whenScaled suits replicas that resync from a primary, because stale data is then never reused. Use Delete for whenDeleted only for disposable environments. Verify the status of this field in your Kubernetes version. Note too that the reclaim policy of the StorageClass still decides whether the underlying PV and disk are deleted when a claim is removed.

6.4 Topology and Zones

Zonal storage ties each replica to a location. With WaitForFirstConsumer, the volume for each Pod is created in the zone where that Pod is first scheduled, so combining it with a topology spread constraint spreads the replicas and their disks across zones. After that, the Pod follows its volume: web-1 can only run in the zone that holds www-web-1. If that zone has no free node capacity, the Pod stays Pending even if other zones have plenty. Keep spare capacity in every zone that hosts replicas, and make sure node pools exist in each zone.

6.5 Choosing Storage

  • Use ReadWriteOnce claims, one per replica, which suits block storage.

  • Pick a StorageClass with allowVolumeExpansion enabled, and consider Retain for irreplaceable data.

  • Local volumes give excellent performance but tie the Pod to a node, so use them only when the application replicates data itself and can rebuild a lost member.

  • Size claims with growth in mind, remembering that volumes can be expanded but not shrunk.

7. Ordering and Pod Management Policies

The podManagementPolicy field controls how Pods are started and stopped when the StatefulSet is created, scaled or deleted:

Policy

Behaviour

OrderedReady (default)

Pods are created one at a time in order 0, 1, 2. The controller waits for each Pod to be Running and Ready before creating the next. On scale-down, Pods are removed in reverse order, and each must be gone before the next is removed

Parallel

Pods are launched or terminated all at once, with no waiting for readiness or termination of other Pods

 

OrderedReady suits software with a strict start-up sequence, but it has a well-known consequence: if web-0 never becomes ready, web-1 and web-2 are never created, so one failing member blocks the whole set. Parallel speeds up start-up and recovery for software that tolerates members joining in any order, such as many distributed databases and caches.

Two behaviours of deletion are worth remembering. Deleting a StatefulSet does not guarantee any order of Pod termination, so if order matters, scale to zero first and then delete. And deleting with the orphan cascade option removes the StatefulSet object but leaves the Pods running, which is a useful technique for maintenance operations such as the volume expansion procedure in Section 11.

8. Updating a StatefulSet

The updateStrategy field decides how changes to the Pod template are rolled out:

  • RollingUpdate (default) replaces Pods one at a time in reverse ordinal order, from the highest to the lowest, and waits for each updated Pod to be Running and Ready before moving to the next. This means that, in a database with primary at ordinal 0, replicas are updated first and the primary last.

  • OnDelete does nothing automatically. A Pod is updated only when you delete it, which gives full manual control.

8.1 Staged Rollouts with partition

The partition setting limits the update to Pods whose ordinal is greater than or equal to the partition value. This allows canary rollouts:

spec:

  updateStrategy:

    type: RollingUpdate

    rollingUpdate:

      partition: 2

With three replicas and a partition of 2, only web-2 receives the new template. Lower the partition to 1, and web-1 is updated, then to 0 to finish. Pods below the partition keep the old template, even if they are deleted and recreated. Verify each stage before lowering the number.

8.2 Giving Pods Time to Become Stable

Setting minReadySeconds on the StatefulSet makes the controller wait for a Pod to stay ready for that many seconds before it counts as available and the update moves on. Combine it with good readiness probes so that a Pod that starts and then fails does not let the rollout continue to the next member.

8.3 Monitoring and Rolling Back

kubectl rollout status statefulset/web

kubectl rollout history statefulset/web

kubectl rollout undo statefulset/web

There is a known caveat with the default OrderedReady policy. If an update puts a Pod into a broken state that never becomes ready, the controller waits for it forever, and reverting the template does not automatically repair that Pod. You must delete the stuck Pod manually so that it is recreated from the corrected template. Because of this, test updates in a staging environment and use a partition for production rollouts.

9. Failure Scenarios

9.1 Pod or Container Crash

If a container crashes, the kubelet restarts it in the same Pod with the same volume. If the Pod is deleted or evicted, the controller creates a replacement with the same name and claim. In both cases the data on the volume survives.

9.2 Node Failure

StatefulSets provide at-most-one semantics: Kubernetes avoids running two Pods with the same identity, because that could corrupt data. When a node becomes unreachable, its Pods show Terminating or Unknown and the controller does not immediately start a replacement, because it cannot be sure that the old Pod has stopped. A single-attach volume may also still be attached to the failed node, which causes Multi-Attach errors for the replacement.

There are two ways to proceed:

  • Wait for the node to recover, or for the cloud provider's controller to remove the node object, which is the safest course.

  • Intervene deliberately once you have confirmed that the node is really down or powered off. Modern clusters support the non-graceful node shutdown feature, in which you apply the taint node.kubernetes.io/out-of-service with the NoExecute effect to the failed node, and Kubernetes then force-deletes its Pods and detaches their volumes. Check the status of this feature in your version.

Avoid routinely force-deleting Pods with a zero grace period. This removes the Pod from the API without any confirmation that it has stopped, so if the node was only partitioned and returns, two Pods may briefly run with the same identity and touch the same data.

9.3 Zone Failure

When a zone fails, the replicas and volumes in it become unavailable. A well-designed cluster keeps a quorum of members in the other zones, and the application remains available. Replicas in the failed zone return when the zone recovers, because their volumes cannot be attached elsewhere. Zonal storage therefore protects against node failure, but zone-level resilience must come from application-level replication across zones.

10. Running Real Databases

A StatefulSet gives each member a place to live. It does not make a database highly available. Production designs usually follow one of these patterns:

  • Application-level replication. The database replicates data between members, and a script or the application decides roles. A common technique is to derive the role from the ordinal in the Pod name, for example in an init container.

  • Operators. Mature operators exist for many databases and brokers. They create and manage StatefulSets, handle failover, backups and upgrades, and encode operational knowledge that is hard to reproduce in scripts. For critical databases, prefer a well-supported operator or a managed database service over hand-written manifests.

The small snippet below shows how a start-up script can read its ordinal from the hostname:

ORDINAL=${HOSTNAME##*-}

if [ "$ORDINAL" -eq 0 ]; then

  echo "starting as primary"

else

  echo "starting as replica of $(echo $HOSTNAME | sed 's/-[0-9]*$/-0/')"

fi

Treat this as an illustration of the idea only. Real clusters need proper bootstrap, replication and failover logic and should rely on the software's own tooling.

11. Day-2 Operations

11.1 Scaling

kubectl scale statefulset web --replicas=5

kubectl get pods,pvc -l app=nginx

Scaling up adds web-3 and web-4 with new claims; scaling down removes the highest ordinals first. Before removing members of a replicated database, make sure that the application has handed off their data.

11.2 Expanding Volumes

The volumeClaimTemplates field of an existing StatefulSet cannot normally be edited in place, so expansion is a two-part job. Patch the existing claims, then update the template for future replicas. A common method is:

  1. Confirm that the StorageClass has allowVolumeExpansion set to true and that the driver supports it.

  2. Patch each existing claim to the new size.

for i in 0 1 2; do

  kubectl patch pvc data-db-$i \

    -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

done

  1. Delete the StatefulSet object while leaving its Pods and claims in place.

kubectl delete statefulset db --cascade=orphan

  1. Re-apply the manifest with the larger size in volumeClaimTemplates, so that the controller adopts the running Pods.

  2. Verify that the claims show the new capacity and that the file systems have grown, using df inside the Pods.

Rehearse this procedure in a test environment first. Newer Kubernetes releases are adding support for updating the template directly, so check the documentation for your version.

11.3 Backups

Replication is not a backup, because a mistake or corruption is copied to every replica. Take regular backups using the application's own tools, or CSI snapshots of each claim combined with application-consistent quiescing, and store them outside the cluster. Test restores, including restoring a snapshot into a new claim and attaching it to a recreated Pod.

12. Designing for High Availability

Combine the following with the StatefulSet to get real resilience:

  • Spread across failure domains with topologySpreadConstraints or Pod anti-affinity, so that replicas do not share a node or zone.

  • A PodDisruptionBudget so that voluntary disruptions such as node drains never remove more members than the cluster can lose.

  • Readiness, liveness and startup probes that reflect real health, because ordered updates and OrderedReady start-up both depend on readiness.

  • Adequate termination grace with terminationGracePeriodSeconds and a preStop hook so that members can leave the cluster and flush data cleanly.

  • Resource requests and limits that guarantee the memory and CPU the database needs, to avoid evictions.

spec:

  template:

    spec:

      topologySpreadConstraints:

      - maxSkew: 1

        topologyKey: topology.kubernetes.io/zone

        whenUnsatisfiable: DoNotSchedule

        labelSelector:

          matchLabels:

            app: nginx

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  name: web-pdb

spec:

  maxUnavailable: 1

  selector:

    matchLabels:

      app: nginx

13. Troubleshooting

kubectl get statefulset,pods,pvc -l app=nginx

kubectl describe statefulset web

kubectl describe pod web-0

kubectl describe pvc www-web-0

kubectl get events --sort-by=.lastTimestamp

Symptom

Likely cause and check

Pod Pending, claim Pending

The StorageClass is missing or cannot provision, or WaitForFirstConsumer is waiting. Describe the claim and read the events.

Pod Pending, claim Bound

The volume's zone or node affinity conflicts with where the Pod can run, or the zone has no capacity. Check node pools and constraints.

web-0 not ready, others never created

OrderedReady blocks on the first Pod. Fix web-0, or use the Parallel policy if the application allows it.

Rollout stuck

An updated Pod never became ready. Fix the cause, then delete the stuck Pod so it is recreated from the corrected template.

Pod stuck Terminating after node failure

At-most-one semantics. Recover the node, or follow the out-of-service procedure if you have confirmed the node is down.

Multi-Attach error on replacement

The single-attach volume is still attached to the old node. Wait for detach or resolve the node state.

DNS name of a member does not resolve

The headless Service is missing, serviceName does not match it, or the Pod is not ready and publishNotReadyAddresses is not set.

Old claims remain after deleting the set

Expected by default. Delete the claims deliberately or use the retention policy.

Scaled-up replica has stale data

It reused an old claim. Delete the claim before scaling up, or use the whenScaled Delete policy for such replicas.

Permission denied on the data directory

Ownership mismatch. Set fsGroup or runAsUser in the Pod security context.

 

14. Best Practices

  • Always create the headless Service first and verify that serviceName matches it.

  • Set storageClassName explicitly, use a class with expansion enabled, and choose the reclaim policy deliberately.

  • Use well-designed readiness probes and, where the application allows it, the Parallel policy to avoid one member blocking the rest.

  • Roll out changes with a partition and verify each stage before continuing.

  • Spread replicas across zones, keep capacity in every zone, and protect them with a PodDisruptionBudget.

  • Decide the claim retention policy up front, and clean up unused claims to control cost.

  • Prefer a mature operator or a managed service for critical databases instead of building failover logic yourself.

  • Back up outside the cluster and test restores regularly, because replication is not a backup.

  • Practise the failure procedures, such as node loss and volume expansion, in a non-production cluster before you need them.

  • Avoid force-deleting Pods as a routine fix, and understand the at-most-one guarantee before you override it.

15. Conclusion

A StatefulSet gives each Pod a name that sticks, a DNS identity that peers can rely on, and a volume that follows it, while controlling the order in which members start, stop and update. Those three guarantees are the foundation on which clustered databases, brokers and caches are run on Kubernetes.

The guarantees come with responsibilities. Storage ties replicas to zones, ordered start-up can block on a single failing member, node failures must be handled deliberately, and the application still has to provide replication and backups. Understand these trade-offs, design for failure from the start, and use operators where the stakes are high, and StatefulSets will give your stateful workloads a dependable home.