Kubernetes CSI Driver
Kubernetes CSI Explained: How Container Storage Interfaces Work
Kubernetes CSI Explained: How Container Storage Interfaces Work
bitcodematrix.com | Kubernetes Series
1. Introduction
Every time a Pod gets a disk from a cloud provider, a Ceph cluster or a storage array, a CSI driver does the work. The Container Storage Interface is the standard that lets Kubernetes use almost any storage system without containing any storage-specific code. Earlier articles in this series showed what StorageClasses, PersistentVolumes and claims are. This article opens the box beneath them and explains how CSI works.
You will learn why CSI was created, what the specification defines, which components make up a driver and its Kubernetes helpers, what happens at each step of a volume's life, how optional features such as expansion and snapshots plug in, and how to install, observe, secure and troubleshoot drivers in a real cluster.
2. Why CSI Exists
In early Kubernetes, support for each storage system was compiled into the Kubernetes binaries. These in-tree volume plugins caused real problems:
Adding or fixing a storage plugin required a change to the core Kubernetes code and had to wait for a Kubernetes release.
Bugs in a plugin could affect core components such as the kubelet and the controller manager.
Vendors had to open their code to the Kubernetes project, and the project had to maintain code for dozens of systems it could not test.
CSI moves the storage code out of tree. A vendor ships a driver as ordinary containers, with its own release cycle, and Kubernetes talks to it through a standard interface. The spec is developed independently of Kubernetes and is shared by other container orchestrators. CSI reached general availability in Kubernetes 1.13. Over the following releases the project migrated the in-tree cloud volume plugins to CSI behind the scenes and then removed the old code, so on current clusters durable storage is delivered by CSI drivers that you install and manage.
3. CSI at a Glance
The CSI specification defines a set of gRPC services that a driver implements. Kubernetes components call these services over a local Unix domain socket. The specification splits the driver's responsibilities into three services:
Drivers advertise optional features through capabilities. A driver for simple network file storage might support only creating and mounting volumes, whereas an enterprise driver might also support attach, snapshots, cloning, expansion and volume statistics. Kubernetes asks the driver which capabilities it has and adapts its behaviour, so a single set of Kubernetes components works with very different back ends.
4. The Components Involved
CSI in Kubernetes is a cooperation between the vendor's driver and a set of generic helper containers, called sidecars, which are maintained by the Kubernetes storage community. The sidecars watch Kubernetes API objects and translate them into CSI calls, so that driver authors never have to write Kubernetes controller code.
Four API objects connect the pieces:
CSIDriver (cluster-scoped) declares a driver and its behaviour towards Kubernetes.
CSINode (one per node) records which drivers are registered on that node and their topology keys.
VolumeAttachment records that a volume should be attached to a node, and whether that has happened.
CSIStorageCapacity lets a driver publish the capacity available in each topology segment, which the scheduler uses when binding is delayed.
5. Deployment Architecture
A CSI driver is typically deployed as two workloads that run the same driver binary in different modes:
The controller plugin runs as a Deployment, normally with two or more replicas and leader election in the sidecars, so that only one active instance handles each operation. Its Pod contains the driver container and the provisioner, attacher, resizer and snapshotter sidecars, which talk to the driver over a shared Unix socket in an emptyDir volume.
The node plugin runs as a DaemonSet, with one Pod on every node that may use the storage. Its Pod contains the driver container, the node-driver-registrar sidecar and often a livenessprobe.
The node plugin is the more privileged half. It has to mount file systems into directories that the kubelet and container runtime can see, so its Pod normally runs privileged and uses bidirectional mount propagation on the kubelet directory. Without it, mounts made inside the driver's container would not appear on the host. The simplified fragment below shows the important parts of a node plugin Pod:
spec:
containers:
- name: csi-driver
image: example/csi-driver:1.0
securityContext:
privileged: true
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet
mountPropagation: Bidirectional
- name: socket-dir
mountPath: /csi
- name: node-driver-registrar
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:<version>
args:
- --csi-address=/csi/csi.sock
- --kubelet-registration-path=/var/lib/kubelet/plugins/example.csi.vendor.com/csi.sock
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
volumes:
- name: kubelet-dir
hostPath:
path: /var/lib/kubelet
type: Directory
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/example.csi.vendor.com
type: DirectoryOrCreate
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
Real drivers are normally installed from their vendor's Helm chart or manifests, which include all of this, so you rarely write it yourself. It is still valuable to recognise the structure when you debug. Note that the paths assume the kubelet's default root directory, and a cluster with a different kubelet root directory needs the driver configured to match.
6. The CSIDriver Object
The CSIDriver object tells Kubernetes how to treat a driver. Its name must match the name the driver reports in GetPluginInfo, and it is the same string that appears as the provisioner in StorageClasses.
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: example.csi.vendor.com
spec:
attachRequired: true
podInfoOnMount: false
volumeLifecycleModes:
- Persistent
fsGroupPolicy: File
storageCapacity: true
Many drivers also use additional fields, for example to request service account tokens for authentication to the storage back end. The driver's documentation defines what it needs.
7. A Volume's Life, Call by Call
The table below follows a dynamically provisioned volume through its life and shows which Kubernetes component makes each CSI call. The calls marked optional depend on the driver's capabilities and CSIDriver settings.
The staging step exists because several Pods on one node may use the same volume. The driver mounts the device once in a staging location, and each Pod then receives its own bind mount of it. On a typical node you can see the staging directory under the kubelet's plugins area and the per-Pod mounts under the Pod's directory in the kubelet root.
For the exact mount sequence when a Pod starts, remember who does what: the attach/detach controller in kube-controller-manager creates the VolumeAttachment object, the external-attacher turns that into a CSI call, and the kubelet drives the node-side calls through its volume manager, using the Unix socket that the driver registered.
8. Node Registration and Topology
The kubelet discovers a driver through a plugin registration mechanism. The node-driver-registrar sidecar places a registration socket in the kubelet's plugin registry directory. The kubelet connects to it, learns the driver's name and the location of its main CSI socket, and then calls the driver's NodeGetInfo operation.
NodeGetInfo returns the node's identifier in the storage system and its topology, such as the zone it sits in. The kubelet records these in the CSINode object and labels the node. This information is how Kubernetes later makes sure that a zonal volume and the Pod that uses it end up together: the topology is passed to CreateVolume, recorded as node affinity on the PV, and respected by the scheduler.
If registration fails, the symptom is a kubelet error on the node saying that the driver name was not found in the list of registered CSI drivers. Check the CSINode object and the node plugin Pod on that node first.
9. Optional Capabilities
9.1 Volume Expansion
When a PVC's size is increased, the external-resizer calls ControllerExpandVolume to grow the back-end volume. If the file system must also grow, the kubelet calls NodeExpandVolume on the node, often while the volume is in use. The StorageClass must have allowVolumeExpansion set to true.
9.2 Snapshots and Clones
Snapshot support is split between the snapshot CRDs and the cluster-wide snapshot-controller, which are installed once, and the external-snapshotter sidecar, which is part of each driver. A VolumeSnapshotClass names the driver and sets the deletion policy:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-snapclass
driver: example.csi.vendor.com
deletionPolicy: Delete
A snapshot can then be restored into a new claim through dataSource, as shown in earlier articles. Cloning uses CreateVolume with a source volume, and requires the driver to advertise that capability.
9.3 Raw Block Volumes
Drivers that support block mode can present the raw device to the container, as requested with volumeMode Block on the claim. In this case NodePublishVolume provides a device file rather than a mounted file system.
9.4 Ephemeral Inline Volumes
Some drivers do not manage durable data at all. They provide ephemeral inline volumes that are declared directly in the Pod and tied to the Pod's life, with no PV or PVC. A well-known example is the Secrets Store CSI driver, which mounts secrets from an external vault as files. The CSIDriver must list Ephemeral in its volumeLifecycleModes.
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: my-provider
The attributes are defined by the driver, so consult its documentation. Do not confuse these with generic ephemeral volumes, a separate feature that creates an ordinary PVC for the Pod and deletes it with the Pod.
9.5 Volume Statistics, Health and Modification
Drivers can report usage statistics through NodeGetVolumeStats, which is where the kubelet's per-claim usage metrics come from. An optional health monitor can surface abnormal volume conditions as events. Recent Kubernetes releases also allow changing attributes such as performance of an existing volume through the VolumeAttributesClass API, if the driver supports modification. Check the feature status in your version and in your driver.
10. Installing and Verifying a Driver
Installation is driver specific, so follow the vendor's guide, which usually uses a Helm chart. Before the driver, check that prerequisites exist, such as the snapshot CRDs and controller if you need snapshots, cloud permissions or credentials for the controller, and kernel modules or client tools needed on nodes (for example NFS or iSCSI clients). After installation, verify it layer by layer:
kubectl get csidrivers
kubectl get pods -A | grep -i csi
kubectl get csinodes
kubectl describe csinode <node-name>
kubectl get storageclass
A healthy installation shows the CSIDriver object, running controller and node Pods, a CSINode entry on each node that lists the driver, and a StorageClass that names it as the provisioner. Finish with a test claim and a test Pod to prove the whole chain, and delete them afterwards. Pin driver and sidecar versions, read the compatibility matrix that the driver publishes for Kubernetes versions, and update the driver when you upgrade the cluster, not afterwards.
11. Observability
Logs. The driver container and each sidecar log every CSI call and its result. For provisioning problems read the csi-provisioner logs; for attach problems read csi-attacher; for mount problems read the node plugin and the kubelet logs on that node.
Events. The claim, the Pod and the VolumeAttachment objects collect events such as ProvisioningFailed, FailedAttachVolume and FailedMount that quote the error returned by the driver.
Metrics. The kubelet exposes metrics about CSI operations, such as csi_operations_seconds, labelled by driver, method and gRPC status code. The sidecars expose their own operation metrics, such as csi_sidecar_operations_seconds. Alert on rising error rates and slow operations.
Capacity and usage. Monitor per-volume usage metrics from the kubelet and the storage back end's own quota and capacity.
12. Troubleshooting
A useful way to think about CSI problems is to ask which call failed, and which component made it. Start with the object that is stuck and follow it down:
kubectl describe pvc <claim> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get volumeattachments
kubectl describe volumeattachment <name>
kubectl logs -n <driver-namespace> <controller-pod> -c csi-provisioner
kubectl logs -n <driver-namespace> <controller-pod> -c csi-attacher
kubectl logs -n <driver-namespace> <node-pod> -c <driver-container>
Avoid editing finalizers or deleting VolumeAttachment objects to force progress, unless you have confirmed the state of the volume in the back end. A wrong move can leave a volume attached to two nodes or leave orphaned disks.
13. Security Considerations
Privileged node plugins. Node plugins run with high privileges on every node. Install drivers only from trusted sources, pin image versions or digests and review what the manifests request.
Least-privilege controllers. The controller plugin needs credentials for the storage back end. Give it only the permissions that it needs, use workload identity where it is available, and restrict who can read the Secrets involved.
RBAC for sidecars. Each sidecar needs specific rights on PVs, PVCs, VolumeAttachments and related objects. Use the roles from the driver's installation guide rather than broad cluster-admin bindings.
Namespace for driver Pods. Run drivers in a protected namespace, typically kube-system or a dedicated one, and prevent ordinary users from deploying into it.
File ownership. Choose fsGroupPolicy deliberately. Recursive ownership changes on large volumes are slow, and the Pod-level fsGroupChangePolicy can reduce that cost.
Encryption. Enable encryption at rest in the storage back end and use TLS where the driver talks to remote systems.
Socket exposure. CSI sockets are local Unix sockets. Do not mount the kubelet plugin directories into untrusted workloads.
14. Choosing a CSI Driver
On managed Kubernetes services, the provider's own CSI driver is usually the best starting point, as it is tested with the platform. Use additional drivers when you need features the default does not provide, such as shared file storage or software-defined storage.
15. Best Practices
Treat CSI drivers as platform components: version them, test upgrades in non-production clusters and roll them out with the same care as a CNI plugin.
Keep driver, sidecars and snapshot components compatible with the cluster version, and read release notes before upgrading.
Run the controller plugin with more than one replica and spread it across nodes or zones.
Use WaitForFirstConsumer and topology-aware classes for zonal storage.
Monitor driver logs, CSI operation error rates and latencies, and volume usage, and alert on failed attach and mount events.
Protect credentials and limit RBAC for the driver and its sidecars.
Document each driver's capabilities and limits for developers, including supported access modes and snapshot support.
Test the failure paths: node loss, driver restart and volume expansion, before you depend on them in production.
16. Conclusion
CSI turns storage into a pluggable part of Kubernetes. A small, well-defined gRPC contract divides the work: the controller side creates, attaches, resizes and snapshots volumes in the storage system, while the node side stages and mounts them for Pods. Generic sidecars translate Kubernetes objects into those calls, and a handful of API objects, CSIDriver, CSINode, VolumeAttachment and CSIStorageCapacity, tell the cluster what each driver can do and where.
With this model in mind, storage incidents stop being mysterious. You can ask which call failed, which component made it and which log holds the answer, and then fix it at the right layer. Choose drivers carefully, operate them as first-class platform components, and CSI gives your workloads storage that is portable, observable and ready for production.