kubernetes

Kubernetes CSI Driver

Kubernetes CSI Explained: How Container Storage Interfaces Work

By Shubhankar Tripathi • • 5 min read

Kubernetes CSI Explained: How Container Storage Interfaces Work

bitcodematrix.com | Kubernetes Series

1. Introduction

Every time a Pod gets a disk from a cloud provider, a Ceph cluster or a storage array, a CSI driver does the work. The Container Storage Interface is the standard that lets Kubernetes use almost any storage system without containing any storage-specific code. Earlier articles in this series showed what StorageClasses, PersistentVolumes and claims are. This article opens the box beneath them and explains how CSI works.

You will learn why CSI was created, what the specification defines, which components make up a driver and its Kubernetes helpers, what happens at each step of a volume's life, how optional features such as expansion and snapshots plug in, and how to install, observe, secure and troubleshoot drivers in a real cluster.

2. Why CSI Exists

In early Kubernetes, support for each storage system was compiled into the Kubernetes binaries. These in-tree volume plugins caused real problems:

  • Adding or fixing a storage plugin required a change to the core Kubernetes code and had to wait for a Kubernetes release.

  • Bugs in a plugin could affect core components such as the kubelet and the controller manager.

  • Vendors had to open their code to the Kubernetes project, and the project had to maintain code for dozens of systems it could not test.

CSI moves the storage code out of tree. A vendor ships a driver as ordinary containers, with its own release cycle, and Kubernetes talks to it through a standard interface. The spec is developed independently of Kubernetes and is shared by other container orchestrators. CSI reached general availability in Kubernetes 1.13. Over the following releases the project migrated the in-tree cloud volume plugins to CSI behind the scenes and then removed the old code, so on current clusters durable storage is delivered by CSI drivers that you install and manage.

3. CSI at a Glance

The CSI specification defines a set of gRPC services that a driver implements. Kubernetes components call these services over a local Unix domain socket. The specification splits the driver's responsibilities into three services:

Service

Responsibility

Example operations

Identity

Tells the caller who the driver is and what it can do

GetPluginInfo, GetPluginCapabilities, Probe

Controller

Cluster-level operations on volumes and snapshots, which need no access to a specific node

CreateVolume, DeleteVolume, ControllerPublishVolume, ControllerUnpublishVolume, ControllerExpandVolume, CreateSnapshot, DeleteSnapshot

Node

Node-level operations that must run on the node that uses the volume

NodeStageVolume, NodePublishVolume, NodeUnpublishVolume, NodeUnstageVolume, NodeExpandVolume, NodeGetInfo

 

Drivers advertise optional features through capabilities. A driver for simple network file storage might support only creating and mounting volumes, whereas an enterprise driver might also support attach, snapshots, cloning, expansion and volume statistics. Kubernetes asks the driver which capabilities it has and adapts its behaviour, so a single set of Kubernetes components works with very different back ends.

4. The Components Involved

CSI in Kubernetes is a cooperation between the vendor's driver and a set of generic helper containers, called sidecars, which are maintained by the Kubernetes storage community. The sidecars watch Kubernetes API objects and translate them into CSI calls, so that driver authors never have to write Kubernetes controller code.

Component

Role

external-provisioner

Watches PVCs, calls CreateVolume and creates the PV; handles DeleteVolume when a PV is released

external-attacher

Watches VolumeAttachment objects and calls ControllerPublishVolume and ControllerUnpublishVolume

external-resizer

Watches PVC size changes and calls ControllerExpandVolume

external-snapshotter

Watches snapshot content objects and calls CreateSnapshot and DeleteSnapshot

snapshot-controller

Cluster-wide controller that binds VolumeSnapshots to VolumeSnapshotContents; deployed once, not per driver

node-driver-registrar

Runs on each node and registers the driver with the kubelet through a registration socket

livenessprobe

Monitors the driver's health through the Probe call and exposes it to Kubernetes probes

external-health-monitor

Optional; reports volume health conditions as events

 

Four API objects connect the pieces:

  • CSIDriver (cluster-scoped) declares a driver and its behaviour towards Kubernetes.

  • CSINode (one per node) records which drivers are registered on that node and their topology keys.

  • VolumeAttachment records that a volume should be attached to a node, and whether that has happened.

  • CSIStorageCapacity lets a driver publish the capacity available in each topology segment, which the scheduler uses when binding is delayed.

5. Deployment Architecture

A CSI driver is typically deployed as two workloads that run the same driver binary in different modes:

  • The controller plugin runs as a Deployment, normally with two or more replicas and leader election in the sidecars, so that only one active instance handles each operation. Its Pod contains the driver container and the provisioner, attacher, resizer and snapshotter sidecars, which talk to the driver over a shared Unix socket in an emptyDir volume.

  • The node plugin runs as a DaemonSet, with one Pod on every node that may use the storage. Its Pod contains the driver container, the node-driver-registrar sidecar and often a livenessprobe.

The node plugin is the more privileged half. It has to mount file systems into directories that the kubelet and container runtime can see, so its Pod normally runs privileged and uses bidirectional mount propagation on the kubelet directory. Without it, mounts made inside the driver's container would not appear on the host. The simplified fragment below shows the important parts of a node plugin Pod:

spec:

  containers:

  - name: csi-driver

    image: example/csi-driver:1.0

    securityContext:

      privileged: true

    volumeMounts:

    - name: kubelet-dir

      mountPath: /var/lib/kubelet

      mountPropagation: Bidirectional

    - name: socket-dir

      mountPath: /csi

  - name: node-driver-registrar

    image: registry.k8s.io/sig-storage/csi-node-driver-registrar:<version>

    args:

    - --csi-address=/csi/csi.sock

    - --kubelet-registration-path=/var/lib/kubelet/plugins/example.csi.vendor.com/csi.sock

    volumeMounts:

    - name: socket-dir

      mountPath: /csi

    - name: registration-dir

      mountPath: /registration

  volumes:

  - name: kubelet-dir

    hostPath:

      path: /var/lib/kubelet

      type: Directory

  - name: socket-dir

    hostPath:

      path: /var/lib/kubelet/plugins/example.csi.vendor.com

      type: DirectoryOrCreate

  - name: registration-dir

    hostPath:

      path: /var/lib/kubelet/plugins_registry

      type: Directory

Real drivers are normally installed from their vendor's Helm chart or manifests, which include all of this, so you rarely write it yourself. It is still valuable to recognise the structure when you debug. Note that the paths assume the kubelet's default root directory, and a cluster with a different kubelet root directory needs the driver configured to match.

6. The CSIDriver Object

The CSIDriver object tells Kubernetes how to treat a driver. Its name must match the name the driver reports in GetPluginInfo, and it is the same string that appears as the provisioner in StorageClasses.

apiVersion: storage.k8s.io/v1

kind: CSIDriver

metadata:

  name: example.csi.vendor.com

spec:

  attachRequired: true

  podInfoOnMount: false

  volumeLifecycleModes:

  - Persistent

  fsGroupPolicy: File

  storageCapacity: true

Field

Meaning

attachRequired

Whether the driver needs a separate attach step. Network file systems such as NFS usually set false, which skips VolumeAttachment objects entirely

podInfoOnMount

Whether Kubernetes passes Pod information, such as name and namespace, to the driver at mount time

volumeLifecycleModes

Persistent for normal PV-backed volumes, Ephemeral for inline volumes declared directly in the Pod spec, or both

fsGroupPolicy

Whether Kubernetes may change file ownership using the Pod's fsGroup: None, File or ReadWriteOnceWithFSType

storageCapacity

Whether the driver publishes CSIStorageCapacity objects for capacity-aware scheduling

requiresRepublish

Whether the volume should be mounted again periodically, which is needed by drivers that refresh content such as secrets

 

Many drivers also use additional fields, for example to request service account tokens for authentication to the storage back end. The driver's documentation defines what it needs.

7. A Volume's Life, Call by Call

The table below follows a dynamically provisioned volume through its life and shows which Kubernetes component makes each CSI call. The calls marked optional depend on the driver's capabilities and CSIDriver settings.

Step

CSI call

Called by

What it does

1

CreateVolume

external-provisioner

Creates the volume in the storage back end with the requested size and parameters

2

ControllerPublishVolume (optional)

external-attacher

Attaches the volume to the chosen node, for example a cloud disk to a virtual machine

3

NodeStageVolume (optional)

kubelet

Prepares the volume once per node: finds the device, formats it if needed and mounts it at a staging path

4

NodePublishVolume

kubelet

Bind-mounts the staged volume into the Pod's own directory, where the container will see it

5

NodeUnpublishVolume

kubelet

Removes the mount from the Pod's directory when the Pod ends

6

NodeUnstageVolume (optional)

kubelet

Unmounts the staging path once no Pod on the node uses the volume

7

ControllerUnpublishVolume (optional)

external-attacher

Detaches the volume from the node

8

DeleteVolume

external-provisioner

Deletes the back-end volume if the reclaim policy is Delete

 

The staging step exists because several Pods on one node may use the same volume. The driver mounts the device once in a staging location, and each Pod then receives its own bind mount of it. On a typical node you can see the staging directory under the kubelet's plugins area and the per-Pod mounts under the Pod's directory in the kubelet root.

For the exact mount sequence when a Pod starts, remember who does what: the attach/detach controller in kube-controller-manager creates the VolumeAttachment object, the external-attacher turns that into a CSI call, and the kubelet drives the node-side calls through its volume manager, using the Unix socket that the driver registered.

8. Node Registration and Topology

The kubelet discovers a driver through a plugin registration mechanism. The node-driver-registrar sidecar places a registration socket in the kubelet's plugin registry directory. The kubelet connects to it, learns the driver's name and the location of its main CSI socket, and then calls the driver's NodeGetInfo operation.

NodeGetInfo returns the node's identifier in the storage system and its topology, such as the zone it sits in. The kubelet records these in the CSINode object and labels the node. This information is how Kubernetes later makes sure that a zonal volume and the Pod that uses it end up together: the topology is passed to CreateVolume, recorded as node affinity on the PV, and respected by the scheduler.

If registration fails, the symptom is a kubelet error on the node saying that the driver name was not found in the list of registered CSI drivers. Check the CSINode object and the node plugin Pod on that node first.

9. Optional Capabilities

9.1 Volume Expansion

When a PVC's size is increased, the external-resizer calls ControllerExpandVolume to grow the back-end volume. If the file system must also grow, the kubelet calls NodeExpandVolume on the node, often while the volume is in use. The StorageClass must have allowVolumeExpansion set to true.

9.2 Snapshots and Clones

Snapshot support is split between the snapshot CRDs and the cluster-wide snapshot-controller, which are installed once, and the external-snapshotter sidecar, which is part of each driver. A VolumeSnapshotClass names the driver and sets the deletion policy:

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshotClass

metadata:

  name: csi-snapclass

driver: example.csi.vendor.com

deletionPolicy: Delete

A snapshot can then be restored into a new claim through dataSource, as shown in earlier articles. Cloning uses CreateVolume with a source volume, and requires the driver to advertise that capability.

9.3 Raw Block Volumes

Drivers that support block mode can present the raw device to the container, as requested with volumeMode Block on the claim. In this case NodePublishVolume provides a device file rather than a mounted file system.

9.4 Ephemeral Inline Volumes

Some drivers do not manage durable data at all. They provide ephemeral inline volumes that are declared directly in the Pod and tied to the Pod's life, with no PV or PVC. A well-known example is the Secrets Store CSI driver, which mounts secrets from an external vault as files. The CSIDriver must list Ephemeral in its volumeLifecycleModes.

volumes:

- name: secrets

  csi:

    driver: secrets-store.csi.k8s.io

    readOnly: true

    volumeAttributes:

      secretProviderClass: my-provider

The attributes are defined by the driver, so consult its documentation. Do not confuse these with generic ephemeral volumes, a separate feature that creates an ordinary PVC for the Pod and deletes it with the Pod.

9.5 Volume Statistics, Health and Modification

Drivers can report usage statistics through NodeGetVolumeStats, which is where the kubelet's per-claim usage metrics come from. An optional health monitor can surface abnormal volume conditions as events. Recent Kubernetes releases also allow changing attributes such as performance of an existing volume through the VolumeAttributesClass API, if the driver supports modification. Check the feature status in your version and in your driver.

10. Installing and Verifying a Driver

Installation is driver specific, so follow the vendor's guide, which usually uses a Helm chart. Before the driver, check that prerequisites exist, such as the snapshot CRDs and controller if you need snapshots, cloud permissions or credentials for the controller, and kernel modules or client tools needed on nodes (for example NFS or iSCSI clients). After installation, verify it layer by layer:

kubectl get csidrivers

kubectl get pods -A | grep -i csi

kubectl get csinodes

kubectl describe csinode <node-name>

kubectl get storageclass

A healthy installation shows the CSIDriver object, running controller and node Pods, a CSINode entry on each node that lists the driver, and a StorageClass that names it as the provisioner. Finish with a test claim and a test Pod to prove the whole chain, and delete them afterwards. Pin driver and sidecar versions, read the compatibility matrix that the driver publishes for Kubernetes versions, and update the driver when you upgrade the cluster, not afterwards.

11. Observability

  • Logs. The driver container and each sidecar log every CSI call and its result. For provisioning problems read the csi-provisioner logs; for attach problems read csi-attacher; for mount problems read the node plugin and the kubelet logs on that node.

  • Events. The claim, the Pod and the VolumeAttachment objects collect events such as ProvisioningFailed, FailedAttachVolume and FailedMount that quote the error returned by the driver.

  • Metrics. The kubelet exposes metrics about CSI operations, such as csi_operations_seconds, labelled by driver, method and gRPC status code. The sidecars expose their own operation metrics, such as csi_sidecar_operations_seconds. Alert on rising error rates and slow operations.

  • Capacity and usage. Monitor per-volume usage metrics from the kubelet and the storage back end's own quota and capacity.

12. Troubleshooting

A useful way to think about CSI problems is to ask which call failed, and which component made it. Start with the object that is stuck and follow it down:

kubectl describe pvc <claim> -n <namespace>

kubectl describe pod <pod> -n <namespace>

kubectl get volumeattachments

kubectl describe volumeattachment <name>

kubectl logs -n <driver-namespace> <controller-pod> -c csi-provisioner

kubectl logs -n <driver-namespace> <controller-pod> -c csi-attacher

kubectl logs -n <driver-namespace> <node-pod> -c <driver-container>

Symptom

Likely cause and check

Claim Pending, no provisioning activity

The controller plugin is not running, or the provisioner name in the StorageClass does not match the CSIDriver name

ProvisioningFailed

CreateVolume failed: invalid parameters, back-end quota, or missing credentials or cloud permissions. Read the csi-provisioner logs

FailedAttachVolume

ControllerPublishVolume failed: permissions, attach limit on the node, or a volume still attached elsewhere. Inspect the VolumeAttachment and the csi-attacher logs

Multi-Attach error

A single-attach volume is still attached to another node. Wait for detach or investigate the old node

FailedMount: driver name not found in the list of registered CSI drivers

The node plugin is not running or not registered on that node. Check the CSINode object and the registrar logs

FailedMount with a mount error

Wrong file system or mount options, or missing client tools on the node. Read the node plugin logs

Mount works in the driver but Pod sees nothing

Mount propagation or kubelet directory mismatch. Confirm the node plugin uses the right kubelet path and Bidirectional propagation

Volume stuck Terminating or detach never completes

A VolumeAttachment is waiting for the attacher, or a finalizer remains. Check the attacher logs and the node state before removing anything by hand

Expansion pending

The driver lacks the capability, the file system resize is waiting for a Pod restart, or the resizer is not running

Snapshot not ready

The snapshot CRDs or the snapshot-controller are missing or the driver lacks the capability

 

Avoid editing finalizers or deleting VolumeAttachment objects to force progress, unless you have confirmed the state of the volume in the back end. A wrong move can leave a volume attached to two nodes or leave orphaned disks.

13. Security Considerations

  • Privileged node plugins. Node plugins run with high privileges on every node. Install drivers only from trusted sources, pin image versions or digests and review what the manifests request.

  • Least-privilege controllers. The controller plugin needs credentials for the storage back end. Give it only the permissions that it needs, use workload identity where it is available, and restrict who can read the Secrets involved.

  • RBAC for sidecars. Each sidecar needs specific rights on PVs, PVCs, VolumeAttachments and related objects. Use the roles from the driver's installation guide rather than broad cluster-admin bindings.

  • Namespace for driver Pods. Run drivers in a protected namespace, typically kube-system or a dedicated one, and prevent ordinary users from deploying into it.

  • File ownership. Choose fsGroupPolicy deliberately. Recursive ownership changes on large volumes are slow, and the Pod-level fsGroupChangePolicy can reduce that cost.

  • Encryption. Enable encryption at rest in the storage back end and use TLS where the driver talks to remote systems.

  • Socket exposure. CSI sockets are local Unix sockets. Do not mount the kubelet plugin directories into untrusted workloads.

14. Choosing a CSI Driver

Question

Why it matters

Which capabilities does it implement?

Check expansion, snapshots, cloning, block mode, topology and ReadWriteMany support against your needs

Which Kubernetes versions are supported?

The driver and sidecars must be compatible with your cluster version

Who maintains it?

Cloud and storage vendors publish drivers for their products; community drivers vary in maturity and support

How does it handle failure?

Look at detach behaviour on node loss, retry behaviour and documented failure modes

What are its limits?

Attach limits per node, volume count limits, size limits and performance tiers

How is it operated?

Installation method, upgrade path, metrics and logging quality

 

On managed Kubernetes services, the provider's own CSI driver is usually the best starting point, as it is tested with the platform. Use additional drivers when you need features the default does not provide, such as shared file storage or software-defined storage.

15. Best Practices

  • Treat CSI drivers as platform components: version them, test upgrades in non-production clusters and roll them out with the same care as a CNI plugin.

  • Keep driver, sidecars and snapshot components compatible with the cluster version, and read release notes before upgrading.

  • Run the controller plugin with more than one replica and spread it across nodes or zones.

  • Use WaitForFirstConsumer and topology-aware classes for zonal storage.

  • Monitor driver logs, CSI operation error rates and latencies, and volume usage, and alert on failed attach and mount events.

  • Protect credentials and limit RBAC for the driver and its sidecars.

  • Document each driver's capabilities and limits for developers, including supported access modes and snapshot support.

  • Test the failure paths: node loss, driver restart and volume expansion, before you depend on them in production.

16. Conclusion

CSI turns storage into a pluggable part of Kubernetes. A small, well-defined gRPC contract divides the work: the controller side creates, attaches, resizes and snapshots volumes in the storage system, while the node side stages and mounts them for Pods. Generic sidecars translate Kubernetes objects into those calls, and a handful of API objects, CSIDriver, CSINode, VolumeAttachment and CSIStorageCapacity, tell the cluster what each driver can do and where.

With this model in mind, storage incidents stop being mysterious. You can ask which call failed, which component made it and which log holds the answer, and then fix it at the right layer. Choose drivers carefully, operate them as first-class platform components, and CSI gives your workloads storage that is portable, observable and ready for production.