kubernetes

Kubernetes Ephemeral Storage Explained

Kubernetes Ephemeral Storage Explained

By Shubhankar Tripathi • • 5 min read

Kubernetes Ephemeral Storage Explained

bitcodematrix.com | Kubernetes Series

1. Introduction

Most Kubernetes storage discussions focus on persistent volumes, but a large share of the disk space that applications use is ephemeral: the writable layer of a container, temporary files in /tmp, scratch directories, caches and container logs. This storage is tied to the life of the Pod, and it lives on the node's own disks. It is also one of the most common causes of surprising Pod evictions and of nodes that suddenly report disk pressure.

This article explains what ephemeral storage is, where it lives on a node, which volume types provide it, how requests and limits work, how the kubelet enforces them and evicts Pods, and how to plan, monitor and troubleshoot it. It complements the earlier articles in this series on persistent storage.

2. What Is Ephemeral Storage?

Ephemeral storage is node-local storage whose data does not survive the Pod. When the Pod is deleted, evicted or rescheduled to another node, the data is gone. It is the right place for data that can be recreated, and the wrong place for anything you cannot afford to lose.

Source

Lifetime

Notes

Container writable layer

The container; lost when the container is recreated

Anything a process writes to the container's file system outside a volume

emptyDir volume (node disk)

The Pod; survives container restarts, lost when the Pod leaves the node

Shared by all containers in the Pod

emptyDir volume (memory)

The Pod

Backed by tmpfs and counted against memory, not disk

Container logs

Managed by the kubelet through log rotation

Written by the container runtime on the node, from the container's output

Generic ephemeral volume

The Pod; its claim is deleted with it

Provided by a StorageClass, not necessarily the node's own disk

Secret volumes

The Pod

Small, backed by tmpfs so that secrets are not written to disk

 

The contrast with persistent volumes is the lifecycle, not the technology. A persistent volume survives the Pod and can be reattached elsewhere. Ephemeral storage is created with the Pod and removed with it.

3. Where the Data Lives on a Node

The kubelet and the container runtime keep Pod data in a few well-known locations. The exact paths depend on the runtime and the kubelet configuration, but the defaults look like this:

Location

Content

/var/lib/kubelet

Kubelet state, including Pod directories with emptyDir volumes, projected volumes and mounts

/var/lib/containerd (or the runtime's equivalent)

Image layers and the writable layers of running containers

/var/log/pods and /var/log/containers

Container log files, with the latter holding links to the former

 

The kubelet distinguishes two logical file systems when it measures disk usage and makes eviction decisions:

  • nodefs is the main node file system that holds the kubelet directory and logs.

  • imagefs is the optional file system that the container runtime uses for images and container writable layers. If there is no dedicated image file system, both share one disk. Recent Kubernetes versions can also separate the container file system (containerfs) from the image storage for finer-grained decisions.

On many clusters everything sits on a single boot disk, which is often small by default. Sizing and separating these locations is one of the most important parts of node planning, as discussed in Section 11.

4. Volume Types for Ephemeral Data

4.1 emptyDir

The emptyDir volume is created empty when a Pod is assigned to a node and exists as long as the Pod runs there. It survives container crashes and restarts, which makes it more durable than the container's own writable layer, and all containers in the Pod can share it.

apiVersion: v1

kind: Pod

metadata:

  name: scratch-demo

spec:

  containers:

  - name: app

    image: busybox:1.36

    command: ["sh", "-c", "sleep 3600"]

    volumeMounts:

    - name: cache

      mountPath: /cache

    - name: fast

      mountPath: /fast

  volumes:

  - name: cache

    emptyDir:

      sizeLimit: 2Gi

  - name: fast

    emptyDir:

      medium: Memory

      sizeLimit: 256Mi

Two variants appear here. The default medium uses the node's disk and its usage counts as ephemeral storage. The Memory medium creates a tmpfs: it is fast and never touches the disk, but every byte written counts against the container's memory, so a runaway writer can cause an out-of-memory kill instead of a disk eviction. Always set a sizeLimit on both kinds.

4.2 Generic Ephemeral Volumes

When scratch space must be larger than the node disk allows, or must have specific performance, a generic ephemeral volume lets a Pod request a volume from any StorageClass. The volume is defined inline in the Pod and the system creates a PersistentVolumeClaim for it automatically. That claim is owned by the Pod and deleted with it.

volumes:

- name: scratch

  ephemeral:

    volumeClaimTemplate:

      metadata:

        labels:

          type: scratch

      spec:

        accessModes: ["ReadWriteOnce"]

        storageClassName: fast-ssd

        resources:

          requests:

            storage: 20Gi

The claim is named after the Pod and the volume, for example scratch-demo-scratch. Because it is a real claim, it gets normal capabilities of the class, such as capacity-aware scheduling and, if the driver supports them, snapshots and resizing. It also counts toward namespace quotas. One point of caution: users who can create Pods can indirectly create claims even without permission to create PVCs directly, so make sure your policies and quotas cover this path.

4.3 CSI Ephemeral Inline Volumes

Some CSI drivers support inline volumes that are declared directly in the Pod and live with it, such as drivers that mount secrets from an external vault. These were covered in the CSI article. They depend on driver support and are specific to the driver's purpose, whereas generic ephemeral volumes work with any dynamic provisioning class.

4.4 Comparison

Type

Backed by

Counted as

Typical use

Container layer

Node disk

Ephemeral storage

Incidental writes

emptyDir

Node disk

Ephemeral storage

Scratch, caches, sharing between containers

emptyDir (Memory)

RAM

Memory

Small fast temporary data

Generic ephemeral

StorageClass volume

PVC and quota for the class

Large or high-performance scratch space

CSI inline

The driver

Depends on the driver

Driver-specific content

 

5. Requests and Limits

Kubernetes lets you manage node-local disk like CPU and memory, with the resource name ephemeral-storage:

spec:

  containers:

  - name: app

    image: example/app:1.0

    resources:

      requests:

        ephemeral-storage: "2Gi"

      limits:

        ephemeral-storage: "4Gi"

Quantities use the usual suffixes: plain bytes, or E, P, T, G, M and k for decimal multiples, and Ei, Pi, Ti, Gi, Mi and Ki for binary multiples. Take care with Mi versus M, because they differ by about five percent.

5.1 What the Scheduler Does

The scheduler adds up the ephemeral-storage requests of all containers in the Pod and places the Pod only on a node whose allocatable ephemeral storage can cover it, after the requests of Pods already on that node. Allocatable is the node's capacity minus what is reserved for the system and the kubelet and the eviction thresholds. You can see it in the node description:

kubectl describe node <node-name> | grep -A8 -i "allocatable"

If no node fits, the Pod stays Pending with an event such as "Insufficient ephemeral-storage".

5.2 What Counts as Usage

For a Pod, the ephemeral storage in use is the sum of its containers' writable layers, its container logs, and its emptyDir volumes that use the node disk. Note that this sum is used for limit enforcement. Also note that ephemeral-storage requests and limits do not affect the Pod's Quality of Service class, which depends only on CPU and memory.

5.3 Defaults

A container that sets no ephemeral-storage request or limit is unbounded. It can consume node disk until the kubelet's node-level eviction takes action, which affects other Pods on the same node. Setting values on all containers, and enforcing them through LimitRange as shown in Section 9, makes that failure mode much less likely.

6. Enforcement and Eviction

6.1 Limits Per Pod

The kubelet monitors the ephemeral storage use of the Pods on its node, and enforces limits by evicting the Pod, not by throttling it, since disk cannot be throttled like CPU. A Pod is evicted if:

  • The total usage of the Pod exceeds the sum of its containers' ephemeral-storage limits.

  • A single container exceeds its own limit, counting its writable layer and logs.

  • An emptyDir volume exceeds its sizeLimit.

The evicted Pod ends in the Failed phase with the reason Evicted, and its status message names the cause, for example that the usage of an emptyDir volume exceeds its limit. If the Pod belongs to a Deployment or StatefulSet, the controller creates a replacement. If the new Pod behaves the same way, you get a loop of evictions.

6.2 How Usage Is Measured

By default the kubelet measures usage by periodically scanning the relevant directories. This is simple and works everywhere but is not instant, so a Pod can exceed its limit for a short time before the kubelet notices. Some environments can use file-system project quotas instead, which measure usage more efficiently and can enforce limits more precisely. It needs support from the underlying file system and a feature gate, so check the documentation for your Kubernetes version before relying on it.

6.3 Node-Pressure Eviction

Separately from Pod limits, the kubelet protects the node itself. It watches disk signals and, when they cross thresholds, it reclaims space and evicts Pods. The common signals are:

Signal

Meaning

nodefs.available

Free space on the node file system

nodefs.inodesFree

Free inodes on the node file system

imagefs.available

Free space on the image file system

imagefs.inodesFree

Free inodes on the image file system

 

Typical default hard thresholds on Linux are less than 10 percent available for nodefs, less than 15 percent available for imagefs and less than 5 percent of inodes free for nodefs, but both defaults and configuration vary, so check your kubelet settings. When a threshold is crossed, the following happens:

  1. The node reports the DiskPressure condition and receives the taint node.kubernetes.io/disk-pressure with the NoSchedule effect, so new Pods are not scheduled there.

  2. The kubelet first tries to reclaim space without evicting Pods, by deleting dead containers and unused images.

  3. If that is not enough, it evicts Pods. For ephemeral storage pressure, it ranks Pods roughly by whether their usage exceeds their requests, then by Pod priority, then by how much usage exceeds the request. Pods using more than they asked for are evicted first.

  4. Once the signal recovers, the condition clears and scheduling resumes after a short delay.

The practical lesson is that requests matter. A Pod with a realistic ephemeral-storage request is less likely to be a victim of node pressure, and Pods without requests that consume lots of disk are the first to go.

7. Container Logs

Container logs are a major and often forgotten part of ephemeral storage. The runtime writes everything a container prints to standard output and standard error into a log file on the node. The kubelet is responsible for rotating these files. Two kubelet settings control it, with typical defaults of 10Mi for the size of a log file and five files per container:

apiVersion: kubelet.config.k8s.io/v1beta1

kind: KubeletConfiguration

containerLogMaxSize: 10Mi

containerLogMaxFiles: 5

Chatty applications can still fill the disk between rotations or when many containers log heavily. Good practice is to:

  • Log to standard output rather than to files inside the container, so that rotation applies.

  • Ship logs off the node with a node-level agent, and keep local retention short.

  • Control log volume in the application, with sensible log levels in production.

  • Remember that log files count toward the Pod's ephemeral storage usage, so noisy containers can trigger their own eviction.

If an application writes log files to a directory in the container instead, those files grow in the writable layer or in a volume, with no rotation at all, unless the application rotates them itself.

8. Images and Garbage Collection

Image layers also live on the node's disk, and while they are not part of a Pod's limit, they compete for the same space and matter for node pressure. The kubelet runs image garbage collection when disk usage on the image file system passes a high threshold, with a default of 85 percent, and removes unused images until usage falls below a low threshold, with a default of 80 percent. It also removes dead containers.

A few consequences follow:

  • Large images consume a lot of space, and several versions of them can accumulate on busy nodes. Use slim base images and avoid leaving many unused versions around.

  • If a node keeps hitting the thresholds, garbage collection removes images that are then pulled again, which wastes time and bandwidth. That is a sign that the disk is too small or that Pods write too much.

  • Pre-pulled or cached images on node pools reduce start-up time, but they need space that you must plan for.

9. Governing Ephemeral Storage with LimitRange and ResourceQuota

A LimitRange sets defaults and bounds for containers in a namespace, so that Pods without explicit values still get sensible ones:

apiVersion: v1

kind: LimitRange

metadata:

  name: ephemeral-defaults

  namespace: shop

spec:

  limits:

  - type: Container

    defaultRequest:

      ephemeral-storage: 1Gi

    default:

      ephemeral-storage: 2Gi

    max:

      ephemeral-storage: 10Gi

A ResourceQuota caps the total for the namespace:

apiVersion: v1

kind: ResourceQuota

metadata:

  name: ephemeral-quota

  namespace: shop

spec:

  hard:

    requests.ephemeral-storage: 20Gi

    limits.ephemeral-storage: 40Gi

With a quota on ephemeral-storage in place, every new Pod in the namespace must specify a request and a limit, or have defaults applied by a LimitRange, otherwise it is rejected. Choose the values from real measurements, not guesses.

10. Practical Patterns

10.1 Read-Only Root File System with Writable Scratch Directories

A recommended security practice is to run containers with a read-only root file system, which stops attackers from modifying the image's files. Applications that need to write somewhere then get explicit emptyDir volumes for those paths:

spec:

  containers:

  - name: app

    image: example/app:1.0

    securityContext:

      readOnlyRootFilesystem: true

    volumeMounts:

    - name: tmp

      mountPath: /tmp

  volumes:

  - name: tmp

    emptyDir:

      sizeLimit: 500Mi

This also makes disk usage predictable, because writes go only to named, limited volumes.

10.2 Sharing Data Between Containers

An emptyDir is the standard way for containers in one Pod to exchange files. A sidecar can produce files that the main container reads, an init container can download or prepare data, and a log-shipping sidecar can read log files that the application writes into a shared directory.

10.3 Caches and Sort Space

Build tools, data-processing jobs and some databases use scratch space for temporary files. Give them an emptyDir with a sizeLimit that matches their need and a matching ephemeral-storage limit. For very large scratch space, use a generic ephemeral volume on a class sized for the purpose, so that the workload does not depend on the node's boot disk.

10.4 Checkpointing

Because an emptyDir survives container restarts, it can hold checkpoints that speed up recovery after a crash. It does not survive Pod deletion or node loss, so treat it only as an optimisation and never as the system of record.

10.5 What Not to Put in Ephemeral Storage

Never keep primary copies of data in ephemeral storage: databases, uploaded user content, queues with delivery guarantees, or anything for which losing a Pod would cause data loss. Use persistent volumes or managed services for those.

11. Node Planning and Capacity

  • Size the disks for the workload. Add up the typical image sizes, writable layers, logs, scratch space and headroom. Default boot disks on cloud node pools are often small, so set the size explicitly.

  • Separate the image and container storage onto their own disk where you can, so that heavy image churn does not starve the kubelet's directory, and the other way round.

  • Use fast local disks for scratch-intensive workloads by dedicating a node pool with local SSDs and scheduling those workloads there using labels and taints.

  • Reserve resources for the system. Configure the kubelet's system and kube reservations and eviction thresholds so that allocatable reflects reality.

  • Watch inodes as well as bytes. Workloads that create huge numbers of small files can exhaust inodes long before the disk is full.

  • Keep headroom. Aim to keep nodes well below the eviction thresholds in normal operation, so that bursts do not trigger evictions.

12. Monitoring and Inspecting

12.1 From the Cluster

kubectl describe node <node-name>

kubectl get pods -A --field-selector=status.phase=Failed

kubectl get events -A --field-selector reason=Evicted

kubectl describe pod <evicted-pod> -n <ns>

kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary"

The node description shows the allocatable ephemeral storage, the sum of requests and limits already allocated, and conditions such as DiskPressure. The kubelet's summary endpoint reports ephemeral storage use per Pod and per volume, which is the quickest way to find the heaviest consumers. Note that kubectl top reports CPU and memory only, not ephemeral storage.

12.2 From Inside a Pod

kubectl exec <pod> -n <ns> -- df -h

kubectl exec <pod> -n <ns> -- du -sh /tmp /cache

12.3 From the Node

df -h

df -i

du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* 2>/dev/null | sort -h | tail

du -sh /var/log/pods/* 2>/dev/null | sort -h | tail

crictl images

journalctl -u kubelet | grep -i -E "evict|disk|garbage"

12.4 Metrics and Alerts

Monitor node file system metrics, for example from node exporter (node_filesystem_avail_bytes and the inode equivalents), container file system usage from cAdvisor, and the resource requests and limits for ephemeral storage that kube-state-metrics exposes. Alert on low free space and low free inodes on nodes, on nodes reporting DiskPressure, and on Pods that are evicted. Useful insights come from comparing Pods' actual usage with their requests, so that you can right-size them.

Note that the kubelet volume statistics for PersistentVolumeClaims do not cover emptyDir volumes, so use the summary API and node metrics for those.

13. Troubleshooting

Symptom

Likely cause and check

Pod status Evicted, message about ephemeral storage

The Pod exceeded its limit or a volume exceeded its sizeLimit. Describe the Pod, find what grew (logs, writable layer, emptyDir), and raise the limit or reduce the writes.

Pod Pending, Insufficient ephemeral-storage

The requests exceed what any node has available. Reduce the request, add nodes with more disk, or free space.

Node reports DiskPressure and Pods are evicted

Disk or inodes are low on nodefs or imagefs. Find the heavy Pods and logs, clean unused images, and enlarge the disk.

Repeated evictions of replacement Pods

The replacement behaves the same way. Fix the cause or the limit; do not just restart.

Logs fill the disk

No external shipping, applications writing log files in the container, or rotation settings too generous. Fix log handling and review the kubelet log settings.

Container killed with OOMKilled and a memory-backed emptyDir

tmpfs usage counts toward container memory. Reduce the writes, set sizeLimit or use disk-backed storage.

No space left on device although the disk looks free

Inodes are exhausted or a tmpfs limit was reached. Check df -i and the volume's sizeLimit.

Many images pulled again and again

Image garbage collection is thrashing because the image disk is too small. Enlarge it or reduce image sizes.

Generic ephemeral volume stays Pending

The auto-created claim cannot be provisioned. Treat it like any other claim: check the class, quota and provisioner.

Data disappears after a restart

It was in the container layer or an emptyDir on a Pod that was rescheduled. Use a persistent volume for data that must survive.

Many Failed Pods remain in the cluster

Evicted Pods stay as objects until garbage collection. Delete them with kubectl delete pods --field-selector=status.phase=Failed in the affected namespace.

 

14. Best Practices

  • Set ephemeral-storage requests and limits on every container, based on measured usage, and enforce defaults with LimitRange.

  • Always set sizeLimit on emptyDir volumes, and use medium Memory only for small, fast, bounded data.

  • Run containers with a read-only root file system and give them explicit scratch volumes.

  • Log to standard output, ship logs off the node, and tune log rotation in the kubelet.

  • Use generic ephemeral volumes when scratch space is large or needs particular performance.

  • Size node disks deliberately, separate image storage where possible and keep headroom below eviction thresholds.

  • Monitor free space and inodes on nodes, DiskPressure conditions and Pod evictions, and review usage against requests regularly.

  • Keep valuable data out of ephemeral storage, and use persistent volumes or managed services for it.

  • Test how your workloads behave under disk pressure in a non-production cluster before an incident teaches you.

15. Conclusion

Ephemeral storage is the quiet part of Kubernetes storage: nobody provisions it deliberately, yet every container uses it. Because it shares the node's own disks, one misbehaving Pod can hurt its neighbours, and because the kubelet enforces it through eviction, a missing limit shows up as a surprise outage rather than a slow-down.

Treat it as a managed resource. Know where the data lives, declare requests and limits, bound every emptyDir, control logs and images, plan node disks and watch the signals that the kubelet watches. Then ephemeral storage becomes a predictable and efficient tool for scratch data and caches, while persistent volumes carry the data that must last.