Kubernetes Ephemeral Storage Explained
Kubernetes Ephemeral Storage Explained
Kubernetes Ephemeral Storage Explained
bitcodematrix.com | Kubernetes Series
1. Introduction
Most Kubernetes storage discussions focus on persistent volumes, but a large share of the disk space that applications use is ephemeral: the writable layer of a container, temporary files in /tmp, scratch directories, caches and container logs. This storage is tied to the life of the Pod, and it lives on the node's own disks. It is also one of the most common causes of surprising Pod evictions and of nodes that suddenly report disk pressure.
This article explains what ephemeral storage is, where it lives on a node, which volume types provide it, how requests and limits work, how the kubelet enforces them and evicts Pods, and how to plan, monitor and troubleshoot it. It complements the earlier articles in this series on persistent storage.
2. What Is Ephemeral Storage?
Ephemeral storage is node-local storage whose data does not survive the Pod. When the Pod is deleted, evicted or rescheduled to another node, the data is gone. It is the right place for data that can be recreated, and the wrong place for anything you cannot afford to lose.
The contrast with persistent volumes is the lifecycle, not the technology. A persistent volume survives the Pod and can be reattached elsewhere. Ephemeral storage is created with the Pod and removed with it.
3. Where the Data Lives on a Node
The kubelet and the container runtime keep Pod data in a few well-known locations. The exact paths depend on the runtime and the kubelet configuration, but the defaults look like this:
The kubelet distinguishes two logical file systems when it measures disk usage and makes eviction decisions:
nodefs is the main node file system that holds the kubelet directory and logs.
imagefs is the optional file system that the container runtime uses for images and container writable layers. If there is no dedicated image file system, both share one disk. Recent Kubernetes versions can also separate the container file system (containerfs) from the image storage for finer-grained decisions.
On many clusters everything sits on a single boot disk, which is often small by default. Sizing and separating these locations is one of the most important parts of node planning, as discussed in Section 11.
4. Volume Types for Ephemeral Data
4.1 emptyDir
The emptyDir volume is created empty when a Pod is assigned to a node and exists as long as the Pod runs there. It survives container crashes and restarts, which makes it more durable than the container's own writable layer, and all containers in the Pod can share it.
apiVersion: v1
kind: Pod
metadata:
name: scratch-demo
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: cache
mountPath: /cache
- name: fast
mountPath: /fast
volumes:
- name: cache
emptyDir:
sizeLimit: 2Gi
- name: fast
emptyDir:
medium: Memory
sizeLimit: 256Mi
Two variants appear here. The default medium uses the node's disk and its usage counts as ephemeral storage. The Memory medium creates a tmpfs: it is fast and never touches the disk, but every byte written counts against the container's memory, so a runaway writer can cause an out-of-memory kill instead of a disk eviction. Always set a sizeLimit on both kinds.
4.2 Generic Ephemeral Volumes
When scratch space must be larger than the node disk allows, or must have specific performance, a generic ephemeral volume lets a Pod request a volume from any StorageClass. The volume is defined inline in the Pod and the system creates a PersistentVolumeClaim for it automatically. That claim is owned by the Pod and deleted with it.
volumes:
- name: scratch
ephemeral:
volumeClaimTemplate:
metadata:
labels:
type: scratch
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
The claim is named after the Pod and the volume, for example scratch-demo-scratch. Because it is a real claim, it gets normal capabilities of the class, such as capacity-aware scheduling and, if the driver supports them, snapshots and resizing. It also counts toward namespace quotas. One point of caution: users who can create Pods can indirectly create claims even without permission to create PVCs directly, so make sure your policies and quotas cover this path.
4.3 CSI Ephemeral Inline Volumes
Some CSI drivers support inline volumes that are declared directly in the Pod and live with it, such as drivers that mount secrets from an external vault. These were covered in the CSI article. They depend on driver support and are specific to the driver's purpose, whereas generic ephemeral volumes work with any dynamic provisioning class.
4.4 Comparison
5. Requests and Limits
Kubernetes lets you manage node-local disk like CPU and memory, with the resource name ephemeral-storage:
spec:
containers:
- name: app
image: example/app:1.0
resources:
requests:
ephemeral-storage: "2Gi"
limits:
ephemeral-storage: "4Gi"
Quantities use the usual suffixes: plain bytes, or E, P, T, G, M and k for decimal multiples, and Ei, Pi, Ti, Gi, Mi and Ki for binary multiples. Take care with Mi versus M, because they differ by about five percent.
5.1 What the Scheduler Does
The scheduler adds up the ephemeral-storage requests of all containers in the Pod and places the Pod only on a node whose allocatable ephemeral storage can cover it, after the requests of Pods already on that node. Allocatable is the node's capacity minus what is reserved for the system and the kubelet and the eviction thresholds. You can see it in the node description:
kubectl describe node <node-name> | grep -A8 -i "allocatable"
If no node fits, the Pod stays Pending with an event such as "Insufficient ephemeral-storage".
5.2 What Counts as Usage
For a Pod, the ephemeral storage in use is the sum of its containers' writable layers, its container logs, and its emptyDir volumes that use the node disk. Note that this sum is used for limit enforcement. Also note that ephemeral-storage requests and limits do not affect the Pod's Quality of Service class, which depends only on CPU and memory.
5.3 Defaults
A container that sets no ephemeral-storage request or limit is unbounded. It can consume node disk until the kubelet's node-level eviction takes action, which affects other Pods on the same node. Setting values on all containers, and enforcing them through LimitRange as shown in Section 9, makes that failure mode much less likely.
6. Enforcement and Eviction
6.1 Limits Per Pod
The kubelet monitors the ephemeral storage use of the Pods on its node, and enforces limits by evicting the Pod, not by throttling it, since disk cannot be throttled like CPU. A Pod is evicted if:
The total usage of the Pod exceeds the sum of its containers' ephemeral-storage limits.
A single container exceeds its own limit, counting its writable layer and logs.
An emptyDir volume exceeds its sizeLimit.
The evicted Pod ends in the Failed phase with the reason Evicted, and its status message names the cause, for example that the usage of an emptyDir volume exceeds its limit. If the Pod belongs to a Deployment or StatefulSet, the controller creates a replacement. If the new Pod behaves the same way, you get a loop of evictions.
6.2 How Usage Is Measured
By default the kubelet measures usage by periodically scanning the relevant directories. This is simple and works everywhere but is not instant, so a Pod can exceed its limit for a short time before the kubelet notices. Some environments can use file-system project quotas instead, which measure usage more efficiently and can enforce limits more precisely. It needs support from the underlying file system and a feature gate, so check the documentation for your Kubernetes version before relying on it.
6.3 Node-Pressure Eviction
Separately from Pod limits, the kubelet protects the node itself. It watches disk signals and, when they cross thresholds, it reclaims space and evicts Pods. The common signals are:
Typical default hard thresholds on Linux are less than 10 percent available for nodefs, less than 15 percent available for imagefs and less than 5 percent of inodes free for nodefs, but both defaults and configuration vary, so check your kubelet settings. When a threshold is crossed, the following happens:
The node reports the DiskPressure condition and receives the taint node.kubernetes.io/disk-pressure with the NoSchedule effect, so new Pods are not scheduled there.
The kubelet first tries to reclaim space without evicting Pods, by deleting dead containers and unused images.
If that is not enough, it evicts Pods. For ephemeral storage pressure, it ranks Pods roughly by whether their usage exceeds their requests, then by Pod priority, then by how much usage exceeds the request. Pods using more than they asked for are evicted first.
Once the signal recovers, the condition clears and scheduling resumes after a short delay.
The practical lesson is that requests matter. A Pod with a realistic ephemeral-storage request is less likely to be a victim of node pressure, and Pods without requests that consume lots of disk are the first to go.
7. Container Logs
Container logs are a major and often forgotten part of ephemeral storage. The runtime writes everything a container prints to standard output and standard error into a log file on the node. The kubelet is responsible for rotating these files. Two kubelet settings control it, with typical defaults of 10Mi for the size of a log file and five files per container:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 10Mi
containerLogMaxFiles: 5
Chatty applications can still fill the disk between rotations or when many containers log heavily. Good practice is to:
Log to standard output rather than to files inside the container, so that rotation applies.
Ship logs off the node with a node-level agent, and keep local retention short.
Control log volume in the application, with sensible log levels in production.
Remember that log files count toward the Pod's ephemeral storage usage, so noisy containers can trigger their own eviction.
If an application writes log files to a directory in the container instead, those files grow in the writable layer or in a volume, with no rotation at all, unless the application rotates them itself.
8. Images and Garbage Collection
Image layers also live on the node's disk, and while they are not part of a Pod's limit, they compete for the same space and matter for node pressure. The kubelet runs image garbage collection when disk usage on the image file system passes a high threshold, with a default of 85 percent, and removes unused images until usage falls below a low threshold, with a default of 80 percent. It also removes dead containers.
A few consequences follow:
Large images consume a lot of space, and several versions of them can accumulate on busy nodes. Use slim base images and avoid leaving many unused versions around.
If a node keeps hitting the thresholds, garbage collection removes images that are then pulled again, which wastes time and bandwidth. That is a sign that the disk is too small or that Pods write too much.
Pre-pulled or cached images on node pools reduce start-up time, but they need space that you must plan for.
9. Governing Ephemeral Storage with LimitRange and ResourceQuota
A LimitRange sets defaults and bounds for containers in a namespace, so that Pods without explicit values still get sensible ones:
apiVersion: v1
kind: LimitRange
metadata:
name: ephemeral-defaults
namespace: shop
spec:
limits:
- type: Container
defaultRequest:
ephemeral-storage: 1Gi
default:
ephemeral-storage: 2Gi
max:
ephemeral-storage: 10Gi
A ResourceQuota caps the total for the namespace:
apiVersion: v1
kind: ResourceQuota
metadata:
name: ephemeral-quota
namespace: shop
spec:
hard:
requests.ephemeral-storage: 20Gi
limits.ephemeral-storage: 40Gi
With a quota on ephemeral-storage in place, every new Pod in the namespace must specify a request and a limit, or have defaults applied by a LimitRange, otherwise it is rejected. Choose the values from real measurements, not guesses.
10. Practical Patterns
10.1 Read-Only Root File System with Writable Scratch Directories
A recommended security practice is to run containers with a read-only root file system, which stops attackers from modifying the image's files. Applications that need to write somewhere then get explicit emptyDir volumes for those paths:
spec:
containers:
- name: app
image: example/app:1.0
securityContext:
readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir:
sizeLimit: 500Mi
This also makes disk usage predictable, because writes go only to named, limited volumes.
10.2 Sharing Data Between Containers
An emptyDir is the standard way for containers in one Pod to exchange files. A sidecar can produce files that the main container reads, an init container can download or prepare data, and a log-shipping sidecar can read log files that the application writes into a shared directory.
10.3 Caches and Sort Space
Build tools, data-processing jobs and some databases use scratch space for temporary files. Give them an emptyDir with a sizeLimit that matches their need and a matching ephemeral-storage limit. For very large scratch space, use a generic ephemeral volume on a class sized for the purpose, so that the workload does not depend on the node's boot disk.
10.4 Checkpointing
Because an emptyDir survives container restarts, it can hold checkpoints that speed up recovery after a crash. It does not survive Pod deletion or node loss, so treat it only as an optimisation and never as the system of record.
10.5 What Not to Put in Ephemeral Storage
Never keep primary copies of data in ephemeral storage: databases, uploaded user content, queues with delivery guarantees, or anything for which losing a Pod would cause data loss. Use persistent volumes or managed services for those.
11. Node Planning and Capacity
Size the disks for the workload. Add up the typical image sizes, writable layers, logs, scratch space and headroom. Default boot disks on cloud node pools are often small, so set the size explicitly.
Separate the image and container storage onto their own disk where you can, so that heavy image churn does not starve the kubelet's directory, and the other way round.
Use fast local disks for scratch-intensive workloads by dedicating a node pool with local SSDs and scheduling those workloads there using labels and taints.
Reserve resources for the system. Configure the kubelet's system and kube reservations and eviction thresholds so that allocatable reflects reality.
Watch inodes as well as bytes. Workloads that create huge numbers of small files can exhaust inodes long before the disk is full.
Keep headroom. Aim to keep nodes well below the eviction thresholds in normal operation, so that bursts do not trigger evictions.
12. Monitoring and Inspecting
12.1 From the Cluster
kubectl describe node <node-name>
kubectl get pods -A --field-selector=status.phase=Failed
kubectl get events -A --field-selector reason=Evicted
kubectl describe pod <evicted-pod> -n <ns>
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary"
The node description shows the allocatable ephemeral storage, the sum of requests and limits already allocated, and conditions such as DiskPressure. The kubelet's summary endpoint reports ephemeral storage use per Pod and per volume, which is the quickest way to find the heaviest consumers. Note that kubectl top reports CPU and memory only, not ephemeral storage.
12.2 From Inside a Pod
kubectl exec <pod> -n <ns> -- df -h
kubectl exec <pod> -n <ns> -- du -sh /tmp /cache
12.3 From the Node
df -h
df -i
du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* 2>/dev/null | sort -h | tail
du -sh /var/log/pods/* 2>/dev/null | sort -h | tail
crictl images
journalctl -u kubelet | grep -i -E "evict|disk|garbage"
12.4 Metrics and Alerts
Monitor node file system metrics, for example from node exporter (node_filesystem_avail_bytes and the inode equivalents), container file system usage from cAdvisor, and the resource requests and limits for ephemeral storage that kube-state-metrics exposes. Alert on low free space and low free inodes on nodes, on nodes reporting DiskPressure, and on Pods that are evicted. Useful insights come from comparing Pods' actual usage with their requests, so that you can right-size them.
Note that the kubelet volume statistics for PersistentVolumeClaims do not cover emptyDir volumes, so use the summary API and node metrics for those.
13. Troubleshooting
14. Best Practices
Set ephemeral-storage requests and limits on every container, based on measured usage, and enforce defaults with LimitRange.
Always set sizeLimit on emptyDir volumes, and use medium Memory only for small, fast, bounded data.
Run containers with a read-only root file system and give them explicit scratch volumes.
Log to standard output, ship logs off the node, and tune log rotation in the kubelet.
Use generic ephemeral volumes when scratch space is large or needs particular performance.
Size node disks deliberately, separate image storage where possible and keep headroom below eviction thresholds.
Monitor free space and inodes on nodes, DiskPressure conditions and Pod evictions, and review usage against requests regularly.
Keep valuable data out of ephemeral storage, and use persistent volumes or managed services for it.
Test how your workloads behave under disk pressure in a non-production cluster before an incident teaches you.
15. Conclusion
Ephemeral storage is the quiet part of Kubernetes storage: nobody provisions it deliberately, yet every container uses it. Because it shares the node's own disks, one misbehaving Pod can hurt its neighbours, and because the kubelet enforces it through eviction, a missing limit shows up as a surprise outage rather than a slow-down.
Treat it as a managed resource. Know where the data lives, declare requests and limits, bound every emptyDir, control logs and images, plan node disks and watch the signals that the kubelet watches. Then ephemeral storage becomes a predictable and efficient tool for scratch data and caches, while persistent volumes carry the data that must last.