Kubernetes Logging Architecture: How Container Logs Work

bitcodematrix.com | Kubernetes Series

1. Introduction

When a Pod misbehaves, the first thing most engineers do is run kubectl logs. That command works because Kubernetes has a simple but important logging design: containers write to standard output and standard error, the container runtime stores that output in files on the node, and the kubelet makes those files available. What Kubernetes does not provide on its own is long-term, cluster-wide log storage. For that you add a logging pipeline.

This article explains how container logs work from the application to the backend. It covers where log files live on a node, the log file format, how kubectl logs reads them, how rotation works, the three common logging patterns (node agent, sidecar and direct push), what other logs a cluster produces, a simple agent configuration, a hands-on exercise, and troubleshooting.

After reading it you should be able to:

Describe the path of a log line from a container to a central backend.
Find container log files on a node and understand their format.
Use kubectl logs effectively, including for crashed containers.
Explain log rotation and why logs can disappear.
Choose between a node-level agent, a sidecar and direct shipping.

2. The Core Idea

Kubernetes follows the principle that a containerized application should write its logs to standard output (stdout) and standard error (stderr) and should not manage log files itself. The container runtime captures those two streams, and everything else is built on top of the captured data.

A container writes to stdout, the runtime stores files, an agent ships them to a backend

Figure 1: The path of a container log line

Component	Role
Application	Writes log lines to stdout and stderr.
Container runtime	Captures the streams (for example containerd or CRI-O) and writes them to log files on the node.
kubelet	Manages the log files of the Pods on its node, rotates them, and serves them to kubectl logs.
Log agent	Reads the files, adds Kubernetes metadata and ships the records to a backend. This is not part of core Kubernetes.
Backend	Stores and indexes logs for search and alerting, for example Loki, Elasticsearch or a cloud logging service.

3. Where Container Logs Live on the Node

On a typical Linux node, the kubelet and the runtime store container logs under /var/log/pods, with one directory per Pod and one subdirectory per container. The files are named after the restart count of the container. A second set of symbolic links under /var/log/containers gives each container log a flat, descriptive name, which is the location most log agents watch.

/var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/0.log

/var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log

-> symbolic link to the file under /var/log/pods

Because the symbolic link name contains the Pod name, namespace, container name and container ID, a log agent can work out the Kubernetes metadata of a log file from its name alone, and then enrich the records through the Kubernetes API.

3.1 The log file format

With containerd and CRI-O, each line in the file follows the CRI logging format: a timestamp, the stream, a tag and the message.

2026-10-13T10:15:42.123456789Z stdout F Server started on port 8080

2026-10-13T10:15:43.001234567Z stderr F WARN cache miss for key user:42

2026-10-13T10:15:44.000000000Z stdout P A very long line that was split into

2026-10-13T10:15:44.000000001Z stdout F two parts by the runtime

Field	Meaning
Timestamp	RFC 3339 time at which the runtime received the line.
Stream	stdout or stderr.
Tag	F means a full line. P means a partial line, which the runtime split because it was very long. The agent must join the P parts with the following F line.
Message	The text written by the application.

Older clusters that used Docker as the runtime stored logs in a JSON format with the fields log, stream and time. Since dockershim was removed in Kubernetes 1.24, the CRI format is the common one, and good log agents handle both formats.

4. Reading Logs with kubectl logs

The kubectl logs command asks the API server, which contacts the kubelet on the node where the Pod runs, and the kubelet reads the log file and streams it back. This is why the command works only while the files exist on that node.

# Logs of a Pod (single container)

kubectl logs my-pod

# A specific container in a multi-container Pod

kubectl logs my-pod -c sidecar

# Follow the log, show timestamps, last 100 lines, last 10 minutes

kubectl logs my-pod -f --timestamps --tail=100 --since=10m

# Logs of the previous (crashed) container instance

kubectl logs my-pod --previous

# All Pods with a label, all containers, with a Pod name prefix

kubectl logs -l app=web --all-containers --prefix

# Logs of a Deployment (picks one Pod)

kubectl logs deployment/web

Option	Use
-f, --follow	Stream new lines as they arrive.
--previous	Show the logs of the last terminated instance of the container. Essential for CrashLoopBackOff.
-c <name>	Choose a container, including init containers.
--tail=N, --since=DURATION	Limit the amount of output.
--timestamps	Prefix each line with the runtime timestamp.
-l <selector>	Read from all Pods that match a label. For large selectors, kubectl limits the number of Pods it reads at once.

kubectl logs shows only what is still on the node. It does not search across Pods or time, it does not work after the Pod has been deleted, and it cannot show logs that rotation has already removed.

5. Log Rotation

Container logs can grow without limit, so the kubelet rotates them. When a log file reaches a maximum size, it is rotated, older files are kept up to a maximum count, and the oldest are removed. The defaults are commonly a size of 10 MiB and 5 files per container, and both can be changed in the kubelet configuration.

apiVersion: kubelet.config.k8s.io/v1beta1

kind: KubeletConfiguration

containerLogMaxSize: 50Mi

containerLogMaxFiles: 5

kubectl logs reads only the current file, so after a rotation you may not see older lines even though rotated files still exist on the node.
A chatty application can rotate through its entire allowance in minutes, and the oldest lines are gone before anyone looks.
Raising limits costs node disk space, so a log agent that ships logs quickly is a better solution than very large local files.
When a Pod is deleted or evicted, its log files are removed from the node. Logs are also lost if the node itself disappears before the agent has shipped them.

On managed Kubernetes services, the kubelet settings may be fixed or adjustable only through node pool configuration, so check your provider documentation.

6. Logging Patterns

6.1 Node-level logging agent (the standard approach)

A DaemonSet runs one logging agent per node. The agent mounts /var/log from the host, tails the container log files and sends the records to the backend. Applications need no changes, the cost per node is small and every container is covered, including Pods that crash and restart. Fluent Bit, Fluentd, Vector, Filebeat and Grafana Alloy or Promtail are common choices.

6.2 Sidecar container

Use a sidecar when an application cannot write to stdout, for example because it writes several log files inside the container. There are two variants.

Streaming sidecar: reads the application log files from a shared volume and prints them to its own stdout, so that the node agent and kubectl logs pick them up. It is simple and keeps the standard pipeline.
Agent sidecar: a full logging agent runs in the Pod and ships logs itself. It gives per-Pod configuration but uses more resources in every Pod.

apiVersion: v1

kind: Pod

metadata:

name: app-with-log-sidecar

spec:

containers:

- name: app

image: registry.example.com/legacy-app:1.0

volumeMounts:

- name: logs

mountPath: /var/log/app

- name: log-streamer

image: busybox:1.36

command: ["sh", "-c", "tail -n+1 -F /var/log/app/app.log"]

volumeMounts:

- name: logs

mountPath: /var/log/app

volumes:

- name: logs

emptyDir: {}

6.3 Direct push from the application

The application sends logs straight to a backend with a client library. It avoids files and agents, but it couples the code to the backend, loses logs if the backend is unreachable unless buffering is added, and bypasses kubectl logs. It is the least common choice.

Aspect	Node agent	Sidecar	Direct push
App changes	None	None or small	Yes
Resource cost	One agent per node	One container per Pod	In the app
Works with kubectl logs	Yes	Streaming sidecar: yes	No
Best for	Most workloads	Legacy apps with log files	Special cases

7. A Simple Node Agent Configuration

The following Fluent Bit snippet shows the main ideas of a node agent: tail the container log files, parse the CRI format, and enrich each record with Kubernetes metadata. Add an output section for your backend, and use the official Helm chart or manifests for the DaemonSet itself.

[INPUT]

Name tail

Path /var/log/containers/*.log

multiline.parser docker, cri

Tag kube.*

Mem_Buf_Limit 5MB

Skip_Long_Lines On

[FILTER]

Name kubernetes

Match kube.*

Merge_Log On

Keep_Log Off

# [OUTPUT] section: add the settings for your backend here

The multiline.parser setting handles both runtime formats and joins partial lines.
The kubernetes filter adds labels such as namespace, Pod name, container name and Pod labels to each record, usually by querying the API server, so give the agent a ServiceAccount with read permissions for Pods and Namespaces.
Memory limits and buffering protect the node when the backend is slow.
Agents need tolerations so that they also run on tainted nodes, otherwise logs from those nodes are not collected.

Option names differ between agent versions and products, so check the documentation of the agent you choose.

8. Other Logs in a Cluster

Source	Where to find it
Application containers	stdout and stderr, via /var/log/pods and kubectl logs.
kubelet and container runtime	On systemd nodes in the journal: journalctl -u kubelet and journalctl -u containerd. Otherwise under /var/log. Managed services often expose these through the provider.
Control plane components	kube-apiserver, kube-scheduler and kube-controller-manager usually run as static Pods and log to stdout, so kubectl logs -n kube-system works. Managed control planes are usually reached through the provider logging service.
Kubernetes events	Not log files. They are API objects, kept for a short time (commonly about an hour). View them with kubectl get events and export them if you need history.
Audit logs	A separate API server feature that records requests. It must be enabled with an audit policy and a backend.

kubectl logs -n kube-system kube-scheduler-<node-name>

kubectl get events -A --sort-by=.lastTimestamp

journalctl -u kubelet --since "1 hour ago"

9. Writing Logs That Work Well in Kubernetes

Log to stdout and stderr, not to files inside the container. Files in the writable layer are not rotated by the kubelet and are lost when the container restarts.
Write one event per line. Use structured JSON logs so that fields such as level, message, request ID and trace ID can be indexed.
Keep stack traces together: either log them as a single JSON field or configure multiline parsing in the agent.
Use log levels consistently, and make the level configurable through an environment variable.
Do not log secrets, tokens, passwords or personal data. Logs are copied to many places.
Avoid very high log volume in hot paths, since it costs CPU, disk, network and backend money.
Include correlation identifiers so that one request can be followed across services.

10. Hands-On Walkthrough

Run a Pod that writes a log line every second.

kubectl run logger --image=busybox:1.36 --restart=Never -- sh -c 'i=0; while true; do echo "line $i"; i=$((i+1)); sleep 1; done'

Read and follow the logs.

kubectl logs logger --tail=5

kubectl logs logger -f --timestamps

Find the Pod node and the log file on it. Use a node debug Pod, or SSH if you have access. The host file system is mounted at /host in the debug container.

kubectl get pod logger -o wide

kubectl debug node/<node-name> -it --image=busybox:1.36

# inside the debug container:

ls /host/var/log/pods/ | grep logger

tail -n 3 /host/var/log/pods/default_logger_*/logger/0.log

Notice the CRI format with timestamp, stream and F tag.
Create a Pod that crashes and read the previous container logs.

kubectl run crasher --image=busybox:1.36 -- sh -c 'echo "starting"; echo "fatal: config missing" >&2; exit 1'

kubectl get pod crasher

kubectl logs crasher --previous

Delete the Pods and confirm the logs are gone.

kubectl delete pod logger crasher

kubectl logs logger

The last command returns a not found error: without a logging agent, logs do not outlive the Pod. This is the main reason for a central logging pipeline.

11. Troubleshooting

Symptom	Likely cause and fix
kubectl logs shows nothing	The application writes to a file, not stdout. Change it or add a streaming sidecar. Also check that the container has actually started.
Error: container is waiting to start	The container has not started yet. Check Pod events for image or volume problems.
Cannot see why a container crashed	Use kubectl logs --previous and kubectl describe pod for the last state and exit code.
Older logs are missing	Rotation removed them, the Pod was recreated or the node was replaced. Ship logs to a backend and raise limits if needed.
Logs missing from the backend for one node	The agent is not running there (check DaemonSet tolerations and node selectors), or it has hit a buffer or permission problem.
Stack traces are split into many records	Add multiline parsing or log traces as one JSON field.
Partial lines appear in the backend	The agent does not join P-tagged lines. Use a CRI-aware parser.
Duplicate logs	Both a node agent and a sidecar agent ship the same logs, or a streaming sidecar duplicates stdout. Remove one path.
Node disk fills up with logs	Chatty applications, rotation set too high, or logs written to the writable layer. Lower limits, reduce volume or fix the app.
kubectl logs: forbidden	RBAC does not allow the pods/log resource. Grant get on pods/log.

12. Best Practices

Run a node-level agent as a DaemonSet for all workloads and send logs to a central backend, so that logs survive Pod and node loss.
Log to stdout and stderr in structured JSON, one event per line.
Set sensible kubelet rotation limits and watch node disk usage.
Give the agent resource requests and limits, tolerations for every node pool and the minimum RBAC permissions it needs.
Use sidecars only for applications that cannot log to stdout.
Define retention and access policies in the backend, and mask sensitive data.
Alert on the logging pipeline itself, such as agent restarts, dropped records and backend errors.
Export Kubernetes events if you need to investigate incidents after the short event retention period.

13. Conclusion

The Kubernetes logging model is simple: applications write to stdout and stderr, the runtime stores the output in files on the node, the kubelet rotates and serves those files, and kubectl logs reads them. Anything beyond that, including search, retention and logs that survive Pod and node loss, requires a cluster-level pipeline, most often a DaemonSet agent shipping to a central backend. If you keep logs structured, write them to stdout and watch rotation and agent health, you get reliable logging with very little complexity.