Kubernetes Storage Architecture in Production
Kubernetes Storage Architecture in Production
Kubernetes Storage Architecture in Production
bitcodematrix.com | Kubernetes Series
1. Introduction
Stateless services are easy to run on Kubernetes. State is where production gets difficult. A single mis-sized volume, an untested backup, a zone that loses a disk or a StorageClass that silently deletes data on claim removal can turn a routine day into an outage. Knowing how PersistentVolumes, claims, StorageClasses and CSI drivers work is necessary, but it is not enough. Production needs an architecture: deliberate decisions about what is stored where, how it survives failure, how it is protected, how fast it must be, and what it costs.
This article pulls the earlier pieces of this series together into a design guide. It covers workload classification, choosing backends, building a StorageClass catalogue, designing for failure domains, backup and disaster recovery, performance, capacity and cost, security, observability, upgrades and three reference architectures. It ends with anti-patterns and a production readiness checklist.
2. Start With Requirements, Not Products
The most common mistake in storage design is to begin with a product ("let's use Ceph") instead of requirements. Write down, for each workload, the answers to a short list of questions:
The answers drive every later decision: the access mode, the class, the backend, the replication scheme and the backup policy. Record them in a short design document per workload, because they are also the evidence you will need when something fails.
3. Classify Your Workloads
Most clusters contain a mixture of very different storage consumers. Grouping them into a few classes keeps the architecture simple and avoids one-off solutions.
Notice the last row. Running a database on Kubernetes is possible and common, but it is not mandatory. For the most critical data, a managed database service can offer better backup, failover and support than you will build yourself. The right question is not "can we run it in the cluster?" but "should we?".
4. Choosing the Storage Backend
Three principles help here. First, prefer what your platform provides: on a managed cloud service, the provider's CSI driver is the best-tested option. Second, do not build a storage platform unless you need one, because running software-defined storage is a full-time job. Third, keep the number of backends small, because every additional backend multiplies the operational burden, from upgrades to monitoring and runbooks.
5. Designing the StorageClass Catalogue
StorageClasses are the interface between application teams and the platform. In production, treat them as a published product catalogue: few, well named, well documented and stable.
A production-grade class might look like the example below. As in earlier articles, the provisioner and parameters belong to one specific driver and serve as an illustration only; take the real values from your driver's documentation.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: critical-retain
provisioner: ebs.csi.aws.com
parameters:
type: gp3
encrypted: "true"
kmsKeyId: <kms-key-arn>
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
Follow these rules when you maintain the catalogue:
Always use WaitForFirstConsumer for zonal or local storage, so that volumes are created where Pods actually run.
Always allow expansion where the driver supports it. Growing a volume is a routine operation; migrating one is a project.
Decide the reclaim policy per class, and never leave it to chance. Delete suits disposable data. Retain suits irreplaceable data, together with a process to clean up retained volumes.
Turn on encryption in every class that holds real data, with keys that your organisation controls.
Keep exactly one default, and require production manifests to name their class explicitly.
Version classes instead of editing them. Several fields are immutable, so introduce a new class such as fast-v2 for breaking changes and migrate workloads on a schedule.
Publish documentation for each class: purpose, performance, cost, access modes, backup coverage and who to ask for help.
6. Designing for Failure Domains
A good storage architecture is defined by what it survives. Failure happens at several levels, and each storage choice protects against some of them but not others.
Several design consequences follow:
Spread replicas across zones with topology spread constraints, and replicate at the application layer, so that each zone holds its own copy of the data. This is the most reliable way to survive a zone failure.
Keep node capacity in every zone that hosts stateful replicas. A Pod follows its volume, so a zone with no free nodes leaves its replica Pending even if other zones have space.
Check provider-specific options such as volumes replicated across zones, which some platforms offer, and understand their performance and failover behaviour before relying on them.
Plan for slow reattachment after unclean node failures. The detach step of a single-attach volume may wait for several minutes, so your RTO must include it, or the application must tolerate the loss of a member without requiring instant replacement.
Treat local storage as disposable. Local volumes make sense only when the application can rebuild a lost member from its peers.
Avoid stretching one cluster across distant regions. Storage latency and consensus protocols make that fragile. Use separate clusters with replication or restore between them.
7. Backup, Snapshots and Disaster Recovery
7.1 Replication Is Not a Backup
Replication copies mistakes. A dropped table, a corrupted file or a ransomware attack is replicated to every member within seconds. A backup is an independent, point-in-time copy that is stored separately and can be restored after any such event. Snapshots taken within the same storage system are a useful first line, but they share the fate of that system and of the account that controls it.
7.2 A Layered Strategy
Snapshots of a running database may be only crash-consistent. For data that needs application consistency, quiesce or flush the application first, or use the database's own backup mechanism. A snapshot class can be defined with a retention-friendly deletion policy:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: critical-snapshots
driver: ebs.csi.aws.com
deletionPolicy: Retain
7.3 Disaster Recovery Drills
A backup you have never restored is only a hope. Schedule regular restore drills, and follow a standard procedure:
Choose a recent backup or snapshot of a representative workload, and declare the target RPO and RTO for the exercise.
Restore it into a separate namespace or cluster, never over the original.
Start the application against the restored volume and run integrity checks and a functional test.
Measure how long each step took, compare it with the RTO, and compare the age of the restored data with the RPO.
Record gaps, fix the runbook and automation, and repeat the drill on a schedule.
7.4 Protecting Against Accidental Deletion
Use Retain for critical classes, so deleting a claim does not delete the volume.
Restrict RBAC for deleting PVCs, PVs, snapshots and namespaces, and require review for destructive changes.
Be careful with GitOps pruning. If a manifest that defines a PVC is removed from the repository, an automated sync may delete the claim and, with a Delete policy, the data.
Remember that deleting a namespace deletes its claims.
Keep backup storage under separate credentials from the cluster, so that a compromised cluster cannot delete its own backups.
8. Performance Engineering
Storage performance is not a property of Kubernetes; it comes from the backend, the instance, the file system and the access pattern. In production, measure it before assuming it.
8.1 What to Understand
Latency, IOPS and throughput are different limits. A workload with many small random writes is limited by IOPS and latency, whereas a backup or analytics job is limited by throughput.
Provisioned versus burstable performance. Many cloud disks give performance related to their size or tier, and some burst above a baseline only for a time. A volume that is fast in a short test may slow down under sustained load.
Instance limits matter. A node has a cap on total disk bandwidth and on the number of attachable volumes, which can be lower than the sum of its disks' capabilities.
Network storage adds latency, particularly file protocols over shared networks. Do not put latency-sensitive databases on shared file storage.
Noisy neighbours exist on shared backends. Use dedicated tiers or local storage for demanding workloads.
File system and mount options such as ext4 or xfs and options like noatime can matter. Follow the database vendor's recommendations, and set them through the class.
Separate workloads such as write-ahead logs and data files onto different volumes if the database benefits from it.
8.2 Benchmark Before You Rely on It
Use a short-lived Pod or Job that mounts a test claim of the intended class and runs a benchmark tool such as fio with a profile that resembles the application:
fio --name=randwrite --directory=/data --rw=randwrite --bs=4k \
--iodepth=32 --direct=1 --size=2G --runtime=60 --time_based \
--ioengine=libaio --group_reporting
Run it on the same node types and in the same zones as production, record the results as a baseline, and repeat after changes such as driver upgrades. Delete the test claims afterwards to avoid cost.
9. Capacity, Cost and Quotas
Right-size deliberately. Start with realistic sizes and rely on expansion for growth, because volumes can grow but not shrink. Over-provisioning wastes money on every volume.
Apply quotas. Use ResourceQuota for total requested storage and the number of claims per namespace, and per-class limits for expensive tiers. Use LimitRange to set minimum and maximum claim sizes.
Hunt for orphans. Released and Failed PVs, retained volumes from deleted claims, unattached cloud disks and old snapshots all keep costing money. Review them on a schedule.
Label for showback. Put owner, team and cost-centre labels or tags on claims and, through the driver's parameters where supported, on the underlying disks.
Alert on growth. Track usage trends and alert before volumes fill up, giving enough time for expansion and for budget decisions.
Set retention. Apply retention periods to snapshots and backups, because unlimited retention grows without bound.
Watch ephemeral storage too. Node disks and logs are part of the bill and of the failure surface, as described in the ephemeral storage article.
10. Security and Compliance
Encryption at rest. Enable it in every class that holds real data, with customer-managed keys where required. Test what happens if key permissions are revoked, because volumes may become unusable.
Encryption in transit. Use TLS for network file and storage protocols where available, and for traffic between applications and databases.
Least-privilege access. Limit who can create StorageClasses, PVs, snapshots and claims, and who can mount which volumes. Remember that anyone who can create Pods in a namespace can mount that namespace's claims and Secrets.
Restrict dangerous volume types. Use Pod Security Standards or admission policies to prevent hostPath and other risky volumes in application namespaces.
Protect driver credentials. The CSI controller needs rights to create, attach and delete cloud volumes. Scope them tightly, prefer workload identity, and keep driver Pods in protected namespaces.
Control snapshots. Snapshots contain full copies of data and must carry the same access controls and encryption as the original.
Mind data remanence. Reusing a Retained volume exposes its old data to the new consumer. Wipe or recreate volumes when ownership changes.
Meet residency rules. Use allowedTopologies and class design to keep data in approved regions and zones.
Keep audit trails. Log and review creation, deletion, expansion and snapshot operations, both in Kubernetes and in the cloud account.
11. Observability and Alerting
Storage monitoring should answer three questions: is it available, is it fast enough, and is it about to run out? The signals below form a sensible baseline. Metric names depend on your monitoring stack, so adjust the examples.
Combine these with logs from the CSI controller and node plugins, Kubernetes events, and dashboards per storage class. Track the age of the last successful restore test as a metric, because it is the clearest measure of recovery readiness.
12. Upgrades and Change Management
Storage components are part of the cluster's critical path, and upgrades need the same care as networking.
CSI drivers and sidecars. Read the driver's compatibility matrix, upgrade drivers before or together with the Kubernetes upgrade as instructed, and pin versions. Roll out node plugins gradually.
Snapshot components. Keep the snapshot CRDs and the snapshot controller at compatible versions, and avoid removing CRDs, because that deletes the snapshot objects they define.
Node maintenance. Drain nodes one at a time, respect PodDisruptionBudgets, and expect stateful Pods to need time to detach and reattach volumes.
Test in staging with realistic data volumes and the same storage classes, including a failure test where a node is lost during the upgrade.
Have a rollback path. Take fresh snapshots or backups before upgrades, and know how to downgrade drivers.
Migrating between classes or backends. Claims cannot change their class. Migrate by restoring a snapshot into a new claim of the new class, or by copying data with a job, and rehearse the cut-over with the application's own tooling.
Moving off removed in-tree plugins. Make sure legacy volumes are served by CSI drivers before upgrading to versions that remove the old code.
Document changes. Treat StorageClass changes, driver upgrades and quota changes as reviewed changes with clear owners.
13. Three Reference Architectures
13.1 SaaS Web Application
A typical product has stateless web and API Pods, a relational database, user uploads and background jobs. A sound design keeps the database in a managed service with its own backups and failover, stores uploads in object storage accessed by API, and uses only ephemeral storage and a small read-write-many or object-backed cache in the cluster. The Pods are fully disposable, so cluster failure recovery is a redeployment plus a pointer to the managed data stores. The storage footprint inside the cluster is minimal, which is the point.
13.2 Self-Hosted Replicated Database
When data must stay in the cluster, use an operator that manages the database as a StatefulSet. Give each replica its own ReadWriteOnce volume from a class like critical-retain, with encryption and expansion enabled, and WaitForFirstConsumer. Spread the replicas across three zones using topology spread constraints, and use application-level replication so that each zone has a copy. Add a PodDisruptionBudget, continuous archiving or regular backups to object storage in another account, periodic snapshots and a quarterly restore drill. Monitor volume usage, replication lag and backup age.
13.3 Data and Machine Learning Platform
These platforms are throughput-heavy and large. Keep the datasets and artifacts in object storage, and use fast local NVMe or generic ephemeral volumes for scratch and caches during jobs. Use shared file storage only where a tool needs a POSIX file system, and size it for throughput. Persistent volumes are limited to notebooks and metadata stores, which are backed up. Because compute nodes are often ephemeral and costly, design jobs to checkpoint to object storage so that losing a node costs minutes, not days.
14. Multi-Cluster and Hybrid Considerations
Volumes belong to one cluster. A PersistentVolume cannot be shared by two clusters through Kubernetes. Moving data means backup and restore, snapshot copy, replication or application-level synchronisation.
Use consistent classes across clusters, with the same names and intent, so that manifests are portable even when drivers differ underneath.
Plan the recovery cluster. Decide whether the DR environment is warm (running, with replicated data) or cold (created on demand from backups), and test the process end to end, including DNS, secrets and storage classes.
Beware of cross-region latency for synchronous replication, and prefer asynchronous replication or backup shipping with an explicit RPO.
On-premises clusters need a storage platform of their own, so budget for the people and hardware that this requires.
15. Common Anti-Patterns
16. Production Readiness Checklist
17. Conclusion
Production storage on Kubernetes is less about any single object than about the decisions around them. Classify workloads honestly, keep the number of backends and classes small, design for the failure domains you must survive, layer your backups and rehearse restores, measure performance instead of assuming it, control cost and access, and watch the signals that precede incidents.
The Kubernetes primitives, PersistentVolumes, claims, StorageClasses, CSI drivers and StatefulSets, give you the building blocks. The architecture is what you build from them. Treat it as a product that has requirements, owners, documentation and a roadmap, and your stateful workloads will be as dependable as the rest of your platform.