Kubernetes CNI Explained: How Pod Networking Actually Works
How Pod Networking Actually Works
Kubernetes CNI Explained: How Pod Networking Actually Works
bitcodematrix.com | Kubernetes Series
1. Introduction
When you create a Pod, it simply appears with an IP address and can talk to every other Pod in the cluster. Nothing in the Kubernetes control plane builds that network. The work is done by a small, pluggable component called a CNI plugin, and understanding it is the key to debugging most networking problems you will meet in production.
This article explains what the Container Network Interface (CNI) is, what happens step by step when a Pod gets its network, how traffic travels between Pods on the same node and on different nodes, how network policies are enforced, and how the popular plugins differ. It ends with hands-on commands and a troubleshooting guide.
2. The Kubernetes Networking Model
Kubernetes does not implement networking itself. Instead, it defines a set of requirements that every cluster network must satisfy:
Every Pod gets its own unique IP address, shared by all containers in that Pod.
Pods on any node can communicate with Pods on all other nodes without NAT.
Agents on a node, such as the kubelet, can communicate with all Pods on that node.
This is often called the "flat network" model: from a Pod's point of view, every other Pod is reachable at its real IP address. How that flat network is built, whether with tunnels, routing or cloud-provider constructs, is left entirely to the network plugin.
Two related topics are often confused with CNI. Services (ClusterIP, NodePort, LoadBalancer) give a stable virtual address to a set of Pods and are implemented by kube-proxy or by an eBPF-based replacement. Ingress and Gateway API handle traffic entering the cluster. CNI is lower in the stack: it is responsible only for giving each Pod a network interface, an IP address and connectivity.
3. What Is CNI?
The Container Network Interface is a specification, together with a set of libraries and reference plugins, for configuring network interfaces in Linux containers. It is a Cloud Native Computing Foundation project. The specification is deliberately small: a plugin is an executable that receives a command and a JSON configuration, does its work and prints a JSON result.
3.1 Who Calls the Plugin
The kubelet does not call CNI plugins directly. It asks the container runtime, such as containerd or CRI-O, to create a Pod sandbox through the Container Runtime Interface (CRI), and the runtime invokes the CNI plugins. Since the removal of dockershim in Kubernetes 1.24, this is the only model: the CNI configuration is read and executed by the container runtime.
3.2 The CNI Operations
The current version of the specification is 1.1.0. A plugin must understand the following operations, which the runtime selects through the CNI_COMMAND environment variable:
3.3 Inputs and Outputs
The runtime passes parameters through environment variables, mainly CNI_COMMAND, CNI_CONTAINERID, CNI_NETNS (the path to the Pod's network namespace), CNI_IFNAME (the interface name, usually eth0), CNI_ARGS and CNI_PATH (where to find other plugin binaries). The network configuration arrives on standard input as JSON. On success, the plugin prints a JSON result containing the interfaces, IP addresses and routes it configured.
4. What Happens When a Pod Is Created
The sequence below shows how a Pod gets its network. Knowing it makes failures much easier to locate.
The scheduler assigns the Pod to a node, and the kubelet on that node sees it.
The kubelet asks the container runtime to run a Pod sandbox through the CRI RunPodSandbox call.
The runtime creates a new network namespace for the Pod, together with the small sandbox container (often called the pause container) that holds it open.
The runtime reads the CNI configuration and calls the plugin with the ADD command, passing the path to the new namespace.
The plugin creates a veth pair, a virtual cable with two ends. One end is moved into the Pod's namespace and named eth0; the other stays in the host namespace.
The plugin asks its IPAM (IP address management) component for an address, assigns it to eth0 and configures routes inside the Pod, including the default route.
The plugin connects the host end of the veth pair to the node's network, either by attaching it to a Linux bridge or by adding routes and eBPF programs, depending on the design.
The plugin returns a JSON result. The runtime then starts the application containers, which join the same network namespace and therefore share the Pod's IP address.
When the Pod is deleted, the runtime calls the plugin with DEL, which removes the interface and returns the IP address to the pool. If DEL is missed, for example after a node crash, the GC operation introduced in spec 1.1.0 exists to clean up the leftovers.
5. Configuration and Binaries on a Node
By convention, two directories matter on every node:
/etc/cni/net.d/ holds the network configuration files. The runtime uses the file that sorts first alphabetically, so a stale or extra file can silently change which network is used.
/opt/cni/bin/ holds the plugin executables, such as bridge, host-local, portmap and the binary installed by your chosen network add-on.
Most clusters install these through a DaemonSet that copies the binary and configuration onto each node. A configuration list (a .conflist file) chains several plugins together. The runtime calls each plugin in order and passes the previous result to the next.
{
"cniVersion": "1.0.0",
"name": "examplenet",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.1.0/24"
}
},
{
"type": "portmap",
"capabilities": { "portMappings": true }
}
]
}
In this example, the bridge plugin builds the connectivity, host-local hands out addresses from the node's subnet, and portmap adds support for hostPort mappings.
6. Plugin Types and IPAM
The CNI project ships reference plugins in three groups, and production network add-ons reuse or replace these ideas.
How per-node ranges work. The cluster has a Pod CIDR, for example 10.244.0.0/16. When node CIDR allocation is enabled, the controller manager gives each node a slice of it, a /24 by default for IPv4, which is about 254 addresses. IPAM on that node allocates Pod addresses only from its own slice. Because every node owns a distinct range, the network can route to a Pod simply by knowing which node owns its subnet. Some plugins, such as Cilium and the cloud-native ones, manage addresses in their own way instead.
7. How Pod Traffic Travels
7.1 Pods on the Same Node
In a bridge-based design, each Pod's veth connects to a Linux bridge on the node, and the bridge forwards frames between Pods like a small switch. In routed or eBPF-based designs, there is no bridge: the host end of each veth is a routed interface, and the kernel or an eBPF program forwards the packet directly to the destination veth. Either way, same-node traffic never leaves the machine.
7.2 Pods on Different Nodes
Cross-node traffic is where plugins differ most. There are three broad approaches.
A common mistake is to treat the overlay as free. Encapsulation reduces the usable packet size, and a wrong MTU shows up as connections that work for small requests but hang on large transfers.
8. Network Policy and CNI
The NetworkPolicy resource lets you define which Pods may talk to which. It is an API only: Kubernetes stores the object, but enforcement is the job of the network plugin. If your plugin does not implement NetworkPolicy, you can create policies without error and they will have no effect. This is one of the most important facts to check when selecting a CNI.
A common first step is a default-deny policy for a namespace, followed by explicit allow rules:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: payments
spec:
podSelector: {}
policyTypes:
- Ingress
The empty podSelector matches every Pod in the namespace, and because no ingress rules are listed, all incoming traffic is denied until other policies allow it. Some plugins add their own, richer policy resources beyond the standard API, for example global rules or Layer 7 aware rules, but those are specific to that plugin.
9. Popular CNI Plugins
Managed Kubernetes services usually come with a default CNI, such as the Amazon VPC CNI on EKS, so check what your provider installs and which alternatives it supports before migrating. Capabilities change between releases, so confirm details in each project's documentation.
10. Hands-On: Inspecting Pod Networking
10.1 From the Cluster
kubectl get pods -A -o wide
kubectl get node <node-name> -o jsonpath='{.spec.podCIDR}'
kubectl exec <pod-name> -- ip addr
kubectl exec <pod-name> -- ip route
The first command shows each Pod's IP and node. The second shows the subnet assigned to a node. The last two show the interface and routes inside the Pod, where you should see eth0 with an address from that node's range. If the container image has no ip tool, use a debug container instead.
10.2 From the Node
ls /etc/cni/net.d/
ls /opt/cni/bin/
ip link show type veth
ip route
bridge link
These commands show the CNI configuration and binaries, the host ends of the Pods' veth pairs, the routes to local and remote Pod subnets, and, on bridge-based setups, the interfaces attached to the bridge. Output varies by plugin, which is itself a useful way to recognise what design your cluster uses.
11. Setting Up a Cluster Network
A new cluster has no Pod network until you install one. With kubeadm, you initialise the control plane with a Pod CIDR that matches your chosen plugin, then apply the plugin's manifest or Helm chart:
kubeadm init --pod-network-cidr=10.244.0.0/16
Until a CNI is installed, nodes report NotReady and CoreDNS Pods stay Pending, because the runtime reports that the network plugin is not initialised. This is expected behaviour, not a fault. Each plugin's documentation states which CIDR it expects, so always follow its installation guide for the exact flags and values.
12. Choosing a CNI
There is no single best plugin. Decide using these questions:
Do you need NetworkPolicy? If yes, choose a plugin that enforces it, or pair a simple data path with a policy engine.
What does your underlay allow? If you cannot advertise routes, an overlay is simpler. If you can, routed designs perform better.
Where does the cluster run? On a public cloud, the provider's native CNI often integrates best with load balancers, security groups and IP management.
What visibility do you need? eBPF-based plugins can provide flow-level observability without extra agents.
What scale do you expect? Check limits on nodes, Pods per node and IP addresses, especially for cloud-native designs.
What can your team operate? A simpler plugin that you understand is better than a powerful one you cannot debug.
13. Troubleshooting
A reliable method is to work outwards: confirm the Pod has an interface and IP, then test same-node Pod traffic, then cross-node traffic, then Service access. Each step isolates a different layer.
14. Best Practices
Plan address ranges before building the cluster. Pod, Service and node ranges must not overlap with each other or with networks you need to reach, such as VPCs or on-premises ranges.
Size the Pod CIDR for growth. Remember that each node reserves its own slice, and that the default limit is 110 Pods per node.
Set the MTU deliberately when using an overlay, and test with large payloads.
Run one primary CNI per cluster. Use a meta-plugin such as Multus only when Pods truly need extra interfaces.
Apply default-deny NetworkPolicies and open only the flows you need.
Test CNI upgrades in a non-production cluster first, and upgrade node by node, because the network plugin affects every Pod.
Monitor the CNI Pods and your plugin's metrics, and alert on IP exhaustion before it affects scheduling.
15. Conclusion
CNI is the small interface that turns the Kubernetes promise of a flat Pod network into real interfaces, addresses and routes. The container runtime calls a plugin whenever a Pod sandbox is created, the plugin wires the Pod's network namespace to the node, and the chosen data path, whether overlay, routed or cloud-native, carries the traffic between nodes. NetworkPolicy enforcement and observability depend on that same plugin, which is why the choice of CNI is one of the most consequential decisions in a cluster design.
With a clear picture of the ADD flow, the files under /etc/cni/net.d and /opt/cni/bin, and the layered troubleshooting approach in this article, you can move from guessing to diagnosing when Pod networking misbehaves.