StatefulSet and Persistent Storage
Kubernetes StatefulSet and Persistent Storage
Kubernetes StatefulSet and Persistent Storage
bitcodematrix.com | Kubernetes Series
1. Introduction
Deployments are perfect for applications whose Pods are interchangeable. Pods are created, destroyed and replaced freely, and nobody cares which replica handled a request. Databases, message brokers, distributed caches and coordination services are different: each member has an identity, its own data and often a specific role in a cluster. For these workloads Kubernetes provides the StatefulSet.
This article explains how StatefulSets work and how they combine with persistent storage. You will learn the guarantees they give, how to build one with a headless Service and volumeClaimTemplates, how identity, storage and ordering behave during scaling, updates and failures, and how to operate them safely. It builds on the earlier articles in this series on PersistentVolumes, PersistentVolumeClaims, StorageClasses and CSI.
2. Why Stateful Workloads Need More Than a Deployment
Consider a three-node database cluster. Each member must have a stable name that its peers can use to find it, its own disk that follows it across restarts, and a predictable start-up order so that a primary exists before replicas try to join. A Deployment cannot give these guarantees:
Pod names are random and change on every replacement.
All replicas share one Pod template and therefore one claim, so they cannot have separate disks.
Pods start and stop in parallel, with no ordering.
A replacement Pod is a new Pod with no relationship to the old one.
The StatefulSet controller manages a set of Pods like a Deployment does, but it adds the three missing properties: stable identity, stable storage and ordered operations.
3. The Guarantees of a StatefulSet
One thing a StatefulSet does not do is understand your application. It does not replicate data, elect a leader, or take backups. It provides the stable foundation, and the application, or an operator that manages it, builds the clustering behaviour on top.
4. Building a StatefulSet
A working StatefulSet always has two parts: a headless Service that provides the DNS identities, and the StatefulSet itself. The example below follows the pattern from the Kubernetes documentation, using a simple web server so that the behaviour is easy to observe.
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
clusterIP: None
selector:
app: nginx
ports:
- name: web
port: 80
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: nginx
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 30
containers:
- name: nginx
image: registry.k8s.io/nginx-slim:0.24
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 1Gi
Four details deserve attention:
clusterIP: None makes the Service headless. It has no virtual IP, and DNS returns the individual Pod addresses instead.
serviceName must name that Service. The StatefulSet does not create it, so you must create it yourself.
volumeClaimTemplates defines the claim that each Pod receives. The name of the template, www here, is also the volume name used in volumeMounts.
storageClassName should be set explicitly. Remove it to use the cluster default class, and check that a class exists that can provision the volumes.
5. Stable Network Identity
Pods in a StatefulSet get the names web-0, web-1 and web-2, and each Pod's hostname equals its name. Together with the headless Service, every Pod also receives a stable DNS record:
web-0.nginx.<namespace>.svc.cluster.local
web-1.nginx.<namespace>.svc.cluster.local
web-2.nginx.<namespace>.svc.cluster.local
These names stay the same when a Pod is rescheduled, although its IP address may change. Applications should therefore use the DNS names, not the IP addresses, in their cluster configuration. You can verify them from inside the cluster:
kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm \
-- nslookup web-0.nginx
The DNS name of the Service itself, nginx, returns the addresses of all ready Pods, which suits clients that want to discover every member.
A subtle point concerns readiness. By default, DNS records for the headless Service include only Pods that are ready. Clustered software often needs to find its peers before any member is ready, for example during the first bootstrap. Setting publishNotReadyAddresses to true on the headless Service makes DNS publish the records of unready Pods too, which solves this chicken-and-egg problem. Use it deliberately, because clients will then also be given unready addresses.
If clients need a single stable entry point to the cluster, for example to reach the current primary, create an additional ordinary Service with a selector that matches only the right Pods, which is a common pattern in database operators.
6. Persistent Storage for Every Replica
6.1 How the Claims Are Created
For each Pod, the StatefulSet controller creates a PersistentVolumeClaim from the template. The claim is named <template>-<statefulset>-<ordinal>, so the example creates www-web-0, www-web-1 and www-web-2. Each claim is provisioned by the StorageClass, bound to its own PV and mounted by exactly one Pod.
The relationship is stable. When Pod web-1 is deleted or its node fails, the replacement is again called web-1, and the controller mounts the existing claim www-web-1 into it. The Pod gets its old data back.
6.2 See It Work
Apply the Service and the StatefulSet and wait until all Pods are ready: kubectl get pods -l app=nginx -w.
List the claims with kubectl get pvc and confirm that three claims exist, one per Pod.
Write a distinct file into each Pod's volume.
for i in 0 1 2; do
kubectl exec web-$i -- sh -c 'echo "Hello from $(hostname)" > /usr/share/nginx/html/index.html'
done
Delete a Pod, and let the controller recreate it.
kubectl delete pod web-0
kubectl get pods -l app=nginx -w
Check that the new web-0 serves the same content as before.
kubectl exec web-0 -- cat /usr/share/nginx/html/index.html
The replacement Pod has the same name, and the file written earlier is still there, because the new Pod mounted the same claim.
6.3 What Happens to Claims on Scale-Down and Deletion
By default, claims outlive their Pods. Scaling a StatefulSet from three replicas to two removes web-2, but the claim www-web-2 stays. Scaling back up reattaches it to the new web-2, with whatever data it held, which may be stale. Deleting the StatefulSet also leaves the claims. This protects data but leaves storage that you pay for, which you must clean up deliberately.
The persistentVolumeClaimRetentionPolicy field lets you choose what happens in each situation:
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Delete
Each setting can be Retain or Delete. Using Delete for whenScaled suits replicas that resync from a primary, because stale data is then never reused. Use Delete for whenDeleted only for disposable environments. Verify the status of this field in your Kubernetes version. Note too that the reclaim policy of the StorageClass still decides whether the underlying PV and disk are deleted when a claim is removed.
6.4 Topology and Zones
Zonal storage ties each replica to a location. With WaitForFirstConsumer, the volume for each Pod is created in the zone where that Pod is first scheduled, so combining it with a topology spread constraint spreads the replicas and their disks across zones. After that, the Pod follows its volume: web-1 can only run in the zone that holds www-web-1. If that zone has no free node capacity, the Pod stays Pending even if other zones have plenty. Keep spare capacity in every zone that hosts replicas, and make sure node pools exist in each zone.
6.5 Choosing Storage
Use ReadWriteOnce claims, one per replica, which suits block storage.
Pick a StorageClass with allowVolumeExpansion enabled, and consider Retain for irreplaceable data.
Local volumes give excellent performance but tie the Pod to a node, so use them only when the application replicates data itself and can rebuild a lost member.
Size claims with growth in mind, remembering that volumes can be expanded but not shrunk.
7. Ordering and Pod Management Policies
The podManagementPolicy field controls how Pods are started and stopped when the StatefulSet is created, scaled or deleted:
OrderedReady suits software with a strict start-up sequence, but it has a well-known consequence: if web-0 never becomes ready, web-1 and web-2 are never created, so one failing member blocks the whole set. Parallel speeds up start-up and recovery for software that tolerates members joining in any order, such as many distributed databases and caches.
Two behaviours of deletion are worth remembering. Deleting a StatefulSet does not guarantee any order of Pod termination, so if order matters, scale to zero first and then delete. And deleting with the orphan cascade option removes the StatefulSet object but leaves the Pods running, which is a useful technique for maintenance operations such as the volume expansion procedure in Section 11.
8. Updating a StatefulSet
The updateStrategy field decides how changes to the Pod template are rolled out:
RollingUpdate (default) replaces Pods one at a time in reverse ordinal order, from the highest to the lowest, and waits for each updated Pod to be Running and Ready before moving to the next. This means that, in a database with primary at ordinal 0, replicas are updated first and the primary last.
OnDelete does nothing automatically. A Pod is updated only when you delete it, which gives full manual control.
8.1 Staged Rollouts with partition
The partition setting limits the update to Pods whose ordinal is greater than or equal to the partition value. This allows canary rollouts:
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 2
With three replicas and a partition of 2, only web-2 receives the new template. Lower the partition to 1, and web-1 is updated, then to 0 to finish. Pods below the partition keep the old template, even if they are deleted and recreated. Verify each stage before lowering the number.
8.2 Giving Pods Time to Become Stable
Setting minReadySeconds on the StatefulSet makes the controller wait for a Pod to stay ready for that many seconds before it counts as available and the update moves on. Combine it with good readiness probes so that a Pod that starts and then fails does not let the rollout continue to the next member.
8.3 Monitoring and Rolling Back
kubectl rollout status statefulset/web
kubectl rollout history statefulset/web
kubectl rollout undo statefulset/web
There is a known caveat with the default OrderedReady policy. If an update puts a Pod into a broken state that never becomes ready, the controller waits for it forever, and reverting the template does not automatically repair that Pod. You must delete the stuck Pod manually so that it is recreated from the corrected template. Because of this, test updates in a staging environment and use a partition for production rollouts.
9. Failure Scenarios
9.1 Pod or Container Crash
If a container crashes, the kubelet restarts it in the same Pod with the same volume. If the Pod is deleted or evicted, the controller creates a replacement with the same name and claim. In both cases the data on the volume survives.
9.2 Node Failure
StatefulSets provide at-most-one semantics: Kubernetes avoids running two Pods with the same identity, because that could corrupt data. When a node becomes unreachable, its Pods show Terminating or Unknown and the controller does not immediately start a replacement, because it cannot be sure that the old Pod has stopped. A single-attach volume may also still be attached to the failed node, which causes Multi-Attach errors for the replacement.
There are two ways to proceed:
Wait for the node to recover, or for the cloud provider's controller to remove the node object, which is the safest course.
Intervene deliberately once you have confirmed that the node is really down or powered off. Modern clusters support the non-graceful node shutdown feature, in which you apply the taint node.kubernetes.io/out-of-service with the NoExecute effect to the failed node, and Kubernetes then force-deletes its Pods and detaches their volumes. Check the status of this feature in your version.
Avoid routinely force-deleting Pods with a zero grace period. This removes the Pod from the API without any confirmation that it has stopped, so if the node was only partitioned and returns, two Pods may briefly run with the same identity and touch the same data.
9.3 Zone Failure
When a zone fails, the replicas and volumes in it become unavailable. A well-designed cluster keeps a quorum of members in the other zones, and the application remains available. Replicas in the failed zone return when the zone recovers, because their volumes cannot be attached elsewhere. Zonal storage therefore protects against node failure, but zone-level resilience must come from application-level replication across zones.
10. Running Real Databases
A StatefulSet gives each member a place to live. It does not make a database highly available. Production designs usually follow one of these patterns:
Application-level replication. The database replicates data between members, and a script or the application decides roles. A common technique is to derive the role from the ordinal in the Pod name, for example in an init container.
Operators. Mature operators exist for many databases and brokers. They create and manage StatefulSets, handle failover, backups and upgrades, and encode operational knowledge that is hard to reproduce in scripts. For critical databases, prefer a well-supported operator or a managed database service over hand-written manifests.
The small snippet below shows how a start-up script can read its ordinal from the hostname:
ORDINAL=${HOSTNAME##*-}
if [ "$ORDINAL" -eq 0 ]; then
echo "starting as primary"
else
echo "starting as replica of $(echo $HOSTNAME | sed 's/-[0-9]*$/-0/')"
fi
Treat this as an illustration of the idea only. Real clusters need proper bootstrap, replication and failover logic and should rely on the software's own tooling.
11. Day-2 Operations
11.1 Scaling
kubectl scale statefulset web --replicas=5
kubectl get pods,pvc -l app=nginx
Scaling up adds web-3 and web-4 with new claims; scaling down removes the highest ordinals first. Before removing members of a replicated database, make sure that the application has handed off their data.
11.2 Expanding Volumes
The volumeClaimTemplates field of an existing StatefulSet cannot normally be edited in place, so expansion is a two-part job. Patch the existing claims, then update the template for future replicas. A common method is:
Confirm that the StorageClass has allowVolumeExpansion set to true and that the driver supports it.
Patch each existing claim to the new size.
for i in 0 1 2; do
kubectl patch pvc data-db-$i \
-p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
done
Delete the StatefulSet object while leaving its Pods and claims in place.
kubectl delete statefulset db --cascade=orphan
Re-apply the manifest with the larger size in volumeClaimTemplates, so that the controller adopts the running Pods.
Verify that the claims show the new capacity and that the file systems have grown, using df inside the Pods.
Rehearse this procedure in a test environment first. Newer Kubernetes releases are adding support for updating the template directly, so check the documentation for your version.
11.3 Backups
Replication is not a backup, because a mistake or corruption is copied to every replica. Take regular backups using the application's own tools, or CSI snapshots of each claim combined with application-consistent quiescing, and store them outside the cluster. Test restores, including restoring a snapshot into a new claim and attaching it to a recreated Pod.
12. Designing for High Availability
Combine the following with the StatefulSet to get real resilience:
Spread across failure domains with topologySpreadConstraints or Pod anti-affinity, so that replicas do not share a node or zone.
A PodDisruptionBudget so that voluntary disruptions such as node drains never remove more members than the cluster can lose.
Readiness, liveness and startup probes that reflect real health, because ordered updates and OrderedReady start-up both depend on readiness.
Adequate termination grace with terminationGracePeriodSeconds and a preStop hook so that members can leave the cluster and flush data cleanly.
Resource requests and limits that guarantee the memory and CPU the database needs, to avoid evictions.
spec:
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: nginx
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: nginx
13. Troubleshooting
kubectl get statefulset,pods,pvc -l app=nginx
kubectl describe statefulset web
kubectl describe pod web-0
kubectl describe pvc www-web-0
kubectl get events --sort-by=.lastTimestamp
14. Best Practices
Always create the headless Service first and verify that serviceName matches it.
Set storageClassName explicitly, use a class with expansion enabled, and choose the reclaim policy deliberately.
Use well-designed readiness probes and, where the application allows it, the Parallel policy to avoid one member blocking the rest.
Roll out changes with a partition and verify each stage before continuing.
Spread replicas across zones, keep capacity in every zone, and protect them with a PodDisruptionBudget.
Decide the claim retention policy up front, and clean up unused claims to control cost.
Prefer a mature operator or a managed service for critical databases instead of building failover logic yourself.
Back up outside the cluster and test restores regularly, because replication is not a backup.
Practise the failure procedures, such as node loss and volume expansion, in a non-production cluster before you need them.
Avoid force-deleting Pods as a routine fix, and understand the at-most-one guarantee before you override it.
15. Conclusion
A StatefulSet gives each Pod a name that sticks, a DNS identity that peers can rely on, and a volume that follows it, while controlling the order in which members start, stop and update. Those three guarantees are the foundation on which clustered databases, brokers and caches are run on Kubernetes.
The guarantees come with responsibilities. Storage ties replicas to zones, ordered start-up can block on a single failing member, node failures must be handled deliberately, and the application still has to provide replication and backups. Understand these trade-offs, design for failure from the start, and use operators where the stakes are high, and StatefulSets will give your stateful workloads a dependable home.