kubernetes

Kubernetes Storage Architecture in Production

Kubernetes Storage Architecture in Production

By Shubhankar Tripathi • • 5 min read

Kubernetes Storage Architecture in Production

bitcodematrix.com | Kubernetes Series

1. Introduction

Stateless services are easy to run on Kubernetes. State is where production gets difficult. A single mis-sized volume, an untested backup, a zone that loses a disk or a StorageClass that silently deletes data on claim removal can turn a routine day into an outage. Knowing how PersistentVolumes, claims, StorageClasses and CSI drivers work is necessary, but it is not enough. Production needs an architecture: deliberate decisions about what is stored where, how it survives failure, how it is protected, how fast it must be, and what it costs.

This article pulls the earlier pieces of this series together into a design guide. It covers workload classification, choosing backends, building a StorageClass catalogue, designing for failure domains, backup and disaster recovery, performance, capacity and cost, security, observability, upgrades and three reference architectures. It ends with anti-patterns and a production readiness checklist.

2. Start With Requirements, Not Products

The most common mistake in storage design is to begin with a product ("let's use Ceph") instead of requirements. Write down, for each workload, the answers to a short list of questions:

Requirement

Question to answer

Durability

Can the data be recreated? If it is lost, what is the business impact?

Availability

How long can the service be unavailable if a node, a zone or the whole cluster fails?

RPO (recovery point objective)

How much recent data can you afford to lose, measured in time?

RTO (recovery time objective)

How quickly must service and data be restored?

Performance

What latency, IOPS and throughput does the workload need, and is the pattern read-heavy or write-heavy?

Access pattern

One writer, many readers or many writers? Block, file or object semantics?

Size and growth

How big is it now, how fast does it grow, and what is the retention requirement?

Security and compliance

Is encryption, key control, residency or audit required?

Cost

What is the budget per terabyte and per IOPS, and who pays?

 

The answers drive every later decision: the access mode, the class, the backend, the replication scheme and the backup policy. Record them in a short design document per workload, because they are also the evidence you will need when something fails.

3. Classify Your Workloads

Most clusters contain a mixture of very different storage consumers. Grouping them into a few classes keeps the architecture simple and avoids one-off solutions.

Workload type

Examples

Storage need

Typical choice

Stateless

Web and API services

None beyond scratch

Ephemeral storage only

Cache and scratch

Build caches, temporary processing

Fast, disposable

emptyDir or generic ephemeral volume

Single-instance database

Small database, internal tools

Durable block, one writer

ReadWriteOnce block volume with backups

Replicated database or broker

Clustered databases, message brokers

Per-replica durable block

StatefulSet with per-replica volumes, operator-managed

Shared files

Content uploads, shared assets

Many writers, file semantics

ReadWriteMany file storage

Large datasets

Analytics, machine learning

High throughput, mostly reads

Object storage plus local scratch

Critical relational data

Core business databases

Highest durability and support

Often a managed database service outside the cluster

 

Notice the last row. Running a database on Kubernetes is possible and common, but it is not mandatory. For the most critical data, a managed database service can offer better backup, failover and support than you will build yourself. The right question is not "can we run it in the cluster?" but "should we?".

4. Choosing the Storage Backend

Backend type

Strengths

Weaknesses

Fits

Cloud block storage (via CSI)

Simple, durable, snapshots, expansion, managed by the provider

Usually single-attach and zonal; performance tied to size and tier

Most databases and StatefulSets

Cloud or network file storage

ReadWriteMany, shared access

Higher latency, throughput limits, cost per GB

Shared content and legacy apps

NFS server

Simple, widely understood, ReadWriteMany

Single point of failure unless made highly available, performance limits

Small shared needs, on-premises

Software-defined storage (for example Ceph or similar systems)

Block, file and object from one platform, replication in the cluster, works on-premises

Operational complexity, needs expertise and dedicated hardware planning

On-premises or large platforms with a storage team

Local NVMe or local disks

Best latency and throughput, low cost

Data tied to the node, rebuilds needed after failure

Self-replicating databases, caches, scratch

Object storage (outside Kubernetes)

Virtually unlimited, cheap, durable

Not a file system, different API

Backups, datasets, artifacts, media

 

Three principles help here. First, prefer what your platform provides: on a managed cloud service, the provider's CSI driver is the best-tested option. Second, do not build a storage platform unless you need one, because running software-defined storage is a full-time job. Third, keep the number of backends small, because every additional backend multiplies the operational burden, from upgrades to monitoring and runbooks.

5. Designing the StorageClass Catalogue

StorageClasses are the interface between application teams and the platform. In production, treat them as a published product catalogue: few, well named, well documented and stable.

Class

Intent

Key settings

standard

General purpose, the default

Balanced performance tier, Delete, WaitForFirstConsumer, expansion on

fast

Latency-sensitive workloads

High-performance tier, encryption, WaitForFirstConsumer, expansion on

critical-retain

Data that must survive claim deletion

Like fast, with reclaimPolicy Retain

shared-files

ReadWriteMany needs

File-storage driver, documented throughput limits

local-nvme

Node-local, self-replicated data

no-provisioner or local provisioner, WaitForFirstConsumer

 

A production-grade class might look like the example below. As in earlier articles, the provisioner and parameters belong to one specific driver and serve as an illustration only; take the real values from your driver's documentation.

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: critical-retain

provisioner: ebs.csi.aws.com

parameters:

  type: gp3

  encrypted: "true"

  kmsKeyId: <kms-key-arn>

reclaimPolicy: Retain

allowVolumeExpansion: true

volumeBindingMode: WaitForFirstConsumer

Follow these rules when you maintain the catalogue:

  • Always use WaitForFirstConsumer for zonal or local storage, so that volumes are created where Pods actually run.

  • Always allow expansion where the driver supports it. Growing a volume is a routine operation; migrating one is a project.

  • Decide the reclaim policy per class, and never leave it to chance. Delete suits disposable data. Retain suits irreplaceable data, together with a process to clean up retained volumes.

  • Turn on encryption in every class that holds real data, with keys that your organisation controls.

  • Keep exactly one default, and require production manifests to name their class explicitly.

  • Version classes instead of editing them. Several fields are immutable, so introduce a new class such as fast-v2 for breaking changes and migrate workloads on a schedule.

  • Publish documentation for each class: purpose, performance, cost, access modes, backup coverage and who to ask for help.

6. Designing for Failure Domains

A good storage architecture is defined by what it survives. Failure happens at several levels, and each storage choice protects against some of them but not others.

Failure domain

What happens to a single-attach zonal block volume

What you need in addition

Container or Pod

Data survives; the Pod restarts and remounts the volume

Nothing special

Node

Data survives on the volume; the Pod must be rescheduled and the volume reattached, which can take minutes after an unclean failure

A plan for the RTO, and PodDisruptionBudgets for planned work

Zone

The volume is unavailable until the zone returns; it cannot attach to a node in another zone

Application-level replication across zones, or restore from backup or snapshot elsewhere

Region or cluster

Everything is unavailable

Off-site backups and a tested disaster recovery process

Storage system

Data may be corrupted or lost

Independent backups, preferably in another system or provider

 

Several design consequences follow:

  • Spread replicas across zones with topology spread constraints, and replicate at the application layer, so that each zone holds its own copy of the data. This is the most reliable way to survive a zone failure.

  • Keep node capacity in every zone that hosts stateful replicas. A Pod follows its volume, so a zone with no free nodes leaves its replica Pending even if other zones have space.

  • Check provider-specific options such as volumes replicated across zones, which some platforms offer, and understand their performance and failover behaviour before relying on them.

  • Plan for slow reattachment after unclean node failures. The detach step of a single-attach volume may wait for several minutes, so your RTO must include it, or the application must tolerate the loss of a member without requiring instant replacement.

  • Treat local storage as disposable. Local volumes make sense only when the application can rebuild a lost member from its peers.

  • Avoid stretching one cluster across distant regions. Storage latency and consensus protocols make that fragile. Use separate clusters with replication or restore between them.

7. Backup, Snapshots and Disaster Recovery

7.1 Replication Is Not a Backup

Replication copies mistakes. A dropped table, a corrupted file or a ransomware attack is replicated to every member within seconds. A backup is an independent, point-in-time copy that is stored separately and can be restored after any such event. Snapshots taken within the same storage system are a useful first line, but they share the fate of that system and of the account that controls it.

7.2 A Layered Strategy

Layer

Purpose

Notes

CSI volume snapshots

Fast local recovery from mistakes

Quick to take and restore; usually in the same system. Consider a snapshot class with a Retain deletion policy for important data

Application-native backups

Consistent, portable copies of databases

For example logical dumps or continuous archiving; often the most reliable restore path

Cluster-level backup tools

Back up Kubernetes objects and volume data together

Tools such as Velero are widely used; verify support for your drivers

Off-site copy

Protection from loss of the account, region or provider

Store in a different account or location, with immutability where available

 

Snapshots of a running database may be only crash-consistent. For data that needs application consistency, quiesce or flush the application first, or use the database's own backup mechanism. A snapshot class can be defined with a retention-friendly deletion policy:

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshotClass

metadata:

  name: critical-snapshots

driver: ebs.csi.aws.com

deletionPolicy: Retain

7.3 Disaster Recovery Drills

A backup you have never restored is only a hope. Schedule regular restore drills, and follow a standard procedure:

  1. Choose a recent backup or snapshot of a representative workload, and declare the target RPO and RTO for the exercise.

  2. Restore it into a separate namespace or cluster, never over the original.

  3. Start the application against the restored volume and run integrity checks and a functional test.

  4. Measure how long each step took, compare it with the RTO, and compare the age of the restored data with the RPO.

  5. Record gaps, fix the runbook and automation, and repeat the drill on a schedule.

7.4 Protecting Against Accidental Deletion

  • Use Retain for critical classes, so deleting a claim does not delete the volume.

  • Restrict RBAC for deleting PVCs, PVs, snapshots and namespaces, and require review for destructive changes.

  • Be careful with GitOps pruning. If a manifest that defines a PVC is removed from the repository, an automated sync may delete the claim and, with a Delete policy, the data.

  • Remember that deleting a namespace deletes its claims.

  • Keep backup storage under separate credentials from the cluster, so that a compromised cluster cannot delete its own backups.

8. Performance Engineering

Storage performance is not a property of Kubernetes; it comes from the backend, the instance, the file system and the access pattern. In production, measure it before assuming it.

8.1 What to Understand

  • Latency, IOPS and throughput are different limits. A workload with many small random writes is limited by IOPS and latency, whereas a backup or analytics job is limited by throughput.

  • Provisioned versus burstable performance. Many cloud disks give performance related to their size or tier, and some burst above a baseline only for a time. A volume that is fast in a short test may slow down under sustained load.

  • Instance limits matter. A node has a cap on total disk bandwidth and on the number of attachable volumes, which can be lower than the sum of its disks' capabilities.

  • Network storage adds latency, particularly file protocols over shared networks. Do not put latency-sensitive databases on shared file storage.

  • Noisy neighbours exist on shared backends. Use dedicated tiers or local storage for demanding workloads.

  • File system and mount options such as ext4 or xfs and options like noatime can matter. Follow the database vendor's recommendations, and set them through the class.

  • Separate workloads such as write-ahead logs and data files onto different volumes if the database benefits from it.

8.2 Benchmark Before You Rely on It

Use a short-lived Pod or Job that mounts a test claim of the intended class and runs a benchmark tool such as fio with a profile that resembles the application:

fio --name=randwrite --directory=/data --rw=randwrite --bs=4k \

    --iodepth=32 --direct=1 --size=2G --runtime=60 --time_based \

    --ioengine=libaio --group_reporting

Run it on the same node types and in the same zones as production, record the results as a baseline, and repeat after changes such as driver upgrades. Delete the test claims afterwards to avoid cost.

9. Capacity, Cost and Quotas

  • Right-size deliberately. Start with realistic sizes and rely on expansion for growth, because volumes can grow but not shrink. Over-provisioning wastes money on every volume.

  • Apply quotas. Use ResourceQuota for total requested storage and the number of claims per namespace, and per-class limits for expensive tiers. Use LimitRange to set minimum and maximum claim sizes.

  • Hunt for orphans. Released and Failed PVs, retained volumes from deleted claims, unattached cloud disks and old snapshots all keep costing money. Review them on a schedule.

  • Label for showback. Put owner, team and cost-centre labels or tags on claims and, through the driver's parameters where supported, on the underlying disks.

  • Alert on growth. Track usage trends and alert before volumes fill up, giving enough time for expansion and for budget decisions.

  • Set retention. Apply retention periods to snapshots and backups, because unlimited retention grows without bound.

  • Watch ephemeral storage too. Node disks and logs are part of the bill and of the failure surface, as described in the ephemeral storage article.

10. Security and Compliance

  • Encryption at rest. Enable it in every class that holds real data, with customer-managed keys where required. Test what happens if key permissions are revoked, because volumes may become unusable.

  • Encryption in transit. Use TLS for network file and storage protocols where available, and for traffic between applications and databases.

  • Least-privilege access. Limit who can create StorageClasses, PVs, snapshots and claims, and who can mount which volumes. Remember that anyone who can create Pods in a namespace can mount that namespace's claims and Secrets.

  • Restrict dangerous volume types. Use Pod Security Standards or admission policies to prevent hostPath and other risky volumes in application namespaces.

  • Protect driver credentials. The CSI controller needs rights to create, attach and delete cloud volumes. Scope them tightly, prefer workload identity, and keep driver Pods in protected namespaces.

  • Control snapshots. Snapshots contain full copies of data and must carry the same access controls and encryption as the original.

  • Mind data remanence. Reusing a Retained volume exposes its old data to the new consumer. Wipe or recreate volumes when ownership changes.

  • Meet residency rules. Use allowedTopologies and class design to keep data in approved regions and zones.

  • Keep audit trails. Log and review creation, deletion, expansion and snapshot operations, both in Kubernetes and in the cloud account.

11. Observability and Alerting

Storage monitoring should answer three questions: is it available, is it fast enough, and is it about to run out? The signals below form a sensible baseline. Metric names depend on your monitoring stack, so adjust the examples.

Signal

Example condition

Why it matters

Claims stuck Pending

kube_persistentvolumeclaim_status_phase with phase Pending for more than 10 minutes

Provisioning or binding problem

Volume nearly full

kubelet_volume_stats_available_bytes divided by kubelet_volume_stats_capacity_bytes below 0.15

Time to expand before an outage

Inodes nearly exhausted

kubelet_volume_stats_inodes_free divided by kubelet_volume_stats_inodes below 0.10

Full volume despite free space

PVs Failed or Released

kube_persistentvolume_status_phase for Failed or Released

Cleanup needed, possible cost leak

Node disk pressure

kube_node_status_condition for DiskPressure true

Evictions and scheduling problems

Storage operation errors

Increase in storage_operation_errors_total

Attach, mount or provisioning failures

Slow attach or mount

High percentiles of storage_operation_duration_seconds

Slow recovery after rescheduling

Backend health

Provider or storage system metrics for latency, IOPS, throttling and errors

Performance and capacity at the source

Backup freshness

Age of the latest successful backup and snapshot

RPO is being met

 

Combine these with logs from the CSI controller and node plugins, Kubernetes events, and dashboards per storage class. Track the age of the last successful restore test as a metric, because it is the clearest measure of recovery readiness.

12. Upgrades and Change Management

Storage components are part of the cluster's critical path, and upgrades need the same care as networking.

  • CSI drivers and sidecars. Read the driver's compatibility matrix, upgrade drivers before or together with the Kubernetes upgrade as instructed, and pin versions. Roll out node plugins gradually.

  • Snapshot components. Keep the snapshot CRDs and the snapshot controller at compatible versions, and avoid removing CRDs, because that deletes the snapshot objects they define.

  • Node maintenance. Drain nodes one at a time, respect PodDisruptionBudgets, and expect stateful Pods to need time to detach and reattach volumes.

  • Test in staging with realistic data volumes and the same storage classes, including a failure test where a node is lost during the upgrade.

  • Have a rollback path. Take fresh snapshots or backups before upgrades, and know how to downgrade drivers.

  • Migrating between classes or backends. Claims cannot change their class. Migrate by restoring a snapshot into a new claim of the new class, or by copying data with a job, and rehearse the cut-over with the application's own tooling.

  • Moving off removed in-tree plugins. Make sure legacy volumes are served by CSI drivers before upgrading to versions that remove the old code.

  • Document changes. Treat StorageClass changes, driver upgrades and quota changes as reviewed changes with clear owners.

13. Three Reference Architectures

13.1 SaaS Web Application

A typical product has stateless web and API Pods, a relational database, user uploads and background jobs. A sound design keeps the database in a managed service with its own backups and failover, stores uploads in object storage accessed by API, and uses only ephemeral storage and a small read-write-many or object-backed cache in the cluster. The Pods are fully disposable, so cluster failure recovery is a redeployment plus a pointer to the managed data stores. The storage footprint inside the cluster is minimal, which is the point.

13.2 Self-Hosted Replicated Database

When data must stay in the cluster, use an operator that manages the database as a StatefulSet. Give each replica its own ReadWriteOnce volume from a class like critical-retain, with encryption and expansion enabled, and WaitForFirstConsumer. Spread the replicas across three zones using topology spread constraints, and use application-level replication so that each zone has a copy. Add a PodDisruptionBudget, continuous archiving or regular backups to object storage in another account, periodic snapshots and a quarterly restore drill. Monitor volume usage, replication lag and backup age.

13.3 Data and Machine Learning Platform

These platforms are throughput-heavy and large. Keep the datasets and artifacts in object storage, and use fast local NVMe or generic ephemeral volumes for scratch and caches during jobs. Use shared file storage only where a tool needs a POSIX file system, and size it for throughput. Persistent volumes are limited to notebooks and metadata stores, which are backed up. Because compute nodes are often ephemeral and costly, design jobs to checkpoint to object storage so that losing a node costs minutes, not days.

14. Multi-Cluster and Hybrid Considerations

  • Volumes belong to one cluster. A PersistentVolume cannot be shared by two clusters through Kubernetes. Moving data means backup and restore, snapshot copy, replication or application-level synchronisation.

  • Use consistent classes across clusters, with the same names and intent, so that manifests are portable even when drivers differ underneath.

  • Plan the recovery cluster. Decide whether the DR environment is warm (running, with replicated data) or cold (created on demand from backups), and test the process end to end, including DNS, secrets and storage classes.

  • Beware of cross-region latency for synchronous replication, and prefer asynchronous replication or backup shipping with an explicit RPO.

  • On-premises clusters need a storage platform of their own, so budget for the people and hardware that this requires.

15. Common Anti-Patterns

Anti-pattern

Why it hurts and what to do instead

Relying on the default class for everything

Behaviour changes when the default changes. Name the class explicitly in production manifests

Delete reclaim policy on critical data

A deleted claim destroys the volume. Use a Retain class and controlled cleanup

Snapshots as the only backup

They share the failure domain of the volume. Add off-site, independent backups

Backups never restored

Unknown if they work. Run regular restore drills

Immediate binding with zonal disks

Volumes land in the wrong zone and Pods cannot start. Use WaitForFirstConsumer

One shared ReadWriteOnce claim for many Deployment replicas

Replicas on other nodes cannot attach. Use StatefulSet per-replica claims or ReadWriteMany

Databases on shared file storage

High latency and locking problems. Use block storage or local disks

hostPath for application data

Ties Pods to nodes and exposes the host. Use PVCs

No quotas or alerts

Surprise costs and full volumes. Apply quotas and usage alerts

Hand-rolled database failover scripts

Fragile and untested. Use mature operators or managed services

Upgrading CSI drivers casually

Can break attach and mount for every workload. Test and stage upgrades

 

16. Production Readiness Checklist

Area

Check

Classes

A small documented catalogue exists; one default; WaitForFirstConsumer for zonal storage; expansion enabled; reclaim policies chosen on purpose

Workloads

Each stateful workload has recorded requirements for durability, RPO, RTO and performance

Availability

Replicas spread across zones; capacity reserved in every zone; PodDisruptionBudgets in place; node-failure behaviour understood

Backup and DR

Layered backups including an off-site copy; restore drills run and measured against RPO and RTO; runbooks current

Security

Encryption at rest and in transit; RBAC and policy for volume types; driver credentials scoped; snapshots protected

Performance

Benchmarks recorded as a baseline for each class; instance and attach limits understood

Capacity and cost

Quotas and limits applied; orphaned volumes and snapshots reviewed; labels for showback; usage alerts

Observability

Alerts for Pending claims, full volumes and inodes, storage errors, DiskPressure and backup age; dashboards per class

Change management

Driver and sidecar versions pinned and tested; upgrade and rollback procedures rehearsed; class changes reviewed

 

17. Conclusion

Production storage on Kubernetes is less about any single object than about the decisions around them. Classify workloads honestly, keep the number of backends and classes small, design for the failure domains you must survive, layer your backups and rehearse restores, measure performance instead of assuming it, control cost and access, and watch the signals that precede incidents.

The Kubernetes primitives, PersistentVolumes, claims, StorageClasses, CSI drivers and StatefulSets, give you the building blocks. The architecture is what you build from them. Treat it as a product that has requirements, owners, documentation and a roadmap, and your stateful workloads will be as dependable as the rest of your platform.