kubernetes

Kubernetes CNI Explained: How Pod Networking Actually Works

How Pod Networking Actually Works

By Shubhankar Tripathi • • 5 min read

Kubernetes CNI Explained: How Pod Networking Actually Works

bitcodematrix.com | Kubernetes Series

1. Introduction

When you create a Pod, it simply appears with an IP address and can talk to every other Pod in the cluster. Nothing in the Kubernetes control plane builds that network. The work is done by a small, pluggable component called a CNI plugin, and understanding it is the key to debugging most networking problems you will meet in production.

This article explains what the Container Network Interface (CNI) is, what happens step by step when a Pod gets its network, how traffic travels between Pods on the same node and on different nodes, how network policies are enforced, and how the popular plugins differ. It ends with hands-on commands and a troubleshooting guide.

2. The Kubernetes Networking Model

Kubernetes does not implement networking itself. Instead, it defines a set of requirements that every cluster network must satisfy:

  • Every Pod gets its own unique IP address, shared by all containers in that Pod.

  • Pods on any node can communicate with Pods on all other nodes without NAT.

  • Agents on a node, such as the kubelet, can communicate with all Pods on that node.

This is often called the "flat network" model: from a Pod's point of view, every other Pod is reachable at its real IP address. How that flat network is built, whether with tunnels, routing or cloud-provider constructs, is left entirely to the network plugin.

Two related topics are often confused with CNI. Services (ClusterIP, NodePort, LoadBalancer) give a stable virtual address to a set of Pods and are implemented by kube-proxy or by an eBPF-based replacement. Ingress and Gateway API handle traffic entering the cluster. CNI is lower in the stack: it is responsible only for giving each Pod a network interface, an IP address and connectivity.

3. What Is CNI?

The Container Network Interface is a specification, together with a set of libraries and reference plugins, for configuring network interfaces in Linux containers. It is a Cloud Native Computing Foundation project. The specification is deliberately small: a plugin is an executable that receives a command and a JSON configuration, does its work and prints a JSON result.

3.1 Who Calls the Plugin

The kubelet does not call CNI plugins directly. It asks the container runtime, such as containerd or CRI-O, to create a Pod sandbox through the Container Runtime Interface (CRI), and the runtime invokes the CNI plugins. Since the removal of dockershim in Kubernetes 1.24, this is the only model: the CNI configuration is read and executed by the container runtime.

3.2 The CNI Operations

The current version of the specification is 1.1.0. A plugin must understand the following operations, which the runtime selects through the CNI_COMMAND environment variable:

Operation

Purpose

ADD

Attach a container to the network: create the interface, assign an IP address and set routes

DEL

Detach a container from the network and release its resources, including the IP address

CHECK

Verify that the container's network is still configured as expected

VERSION

Report which CNI specification versions the plugin supports

STATUS

Report whether the plugin is ready to accept ADD requests (added in spec 1.1.0)

GC

Let the runtime pass the list of valid attachments so the plugin can clean up stale resources such as leaked IP reservations (added in spec 1.1.0)

 

3.3 Inputs and Outputs

The runtime passes parameters through environment variables, mainly CNI_COMMAND, CNI_CONTAINERID, CNI_NETNS (the path to the Pod's network namespace), CNI_IFNAME (the interface name, usually eth0), CNI_ARGS and CNI_PATH (where to find other plugin binaries). The network configuration arrives on standard input as JSON. On success, the plugin prints a JSON result containing the interfaces, IP addresses and routes it configured.

4. What Happens When a Pod Is Created

The sequence below shows how a Pod gets its network. Knowing it makes failures much easier to locate.

  1. The scheduler assigns the Pod to a node, and the kubelet on that node sees it.

  2. The kubelet asks the container runtime to run a Pod sandbox through the CRI RunPodSandbox call.

  3. The runtime creates a new network namespace for the Pod, together with the small sandbox container (often called the pause container) that holds it open.

  4. The runtime reads the CNI configuration and calls the plugin with the ADD command, passing the path to the new namespace.

  5. The plugin creates a veth pair, a virtual cable with two ends. One end is moved into the Pod's namespace and named eth0; the other stays in the host namespace.

  6. The plugin asks its IPAM (IP address management) component for an address, assigns it to eth0 and configures routes inside the Pod, including the default route.

  7. The plugin connects the host end of the veth pair to the node's network, either by attaching it to a Linux bridge or by adding routes and eBPF programs, depending on the design.

  8. The plugin returns a JSON result. The runtime then starts the application containers, which join the same network namespace and therefore share the Pod's IP address.

When the Pod is deleted, the runtime calls the plugin with DEL, which removes the interface and returns the IP address to the pool. If DEL is missed, for example after a node crash, the GC operation introduced in spec 1.1.0 exists to clean up the leftovers.

5. Configuration and Binaries on a Node

By convention, two directories matter on every node:

  • /etc/cni/net.d/ holds the network configuration files. The runtime uses the file that sorts first alphabetically, so a stale or extra file can silently change which network is used.

  • /opt/cni/bin/ holds the plugin executables, such as bridge, host-local, portmap and the binary installed by your chosen network add-on.

Most clusters install these through a DaemonSet that copies the binary and configuration onto each node. A configuration list (a .conflist file) chains several plugins together. The runtime calls each plugin in order and passes the previous result to the next.

{

  "cniVersion": "1.0.0",

  "name": "examplenet",

  "plugins": [

    {

      "type": "bridge",

      "bridge": "cni0",

      "isGateway": true,

      "ipMasq": true,

      "ipam": {

        "type": "host-local",

        "subnet": "10.244.1.0/24"

      }

    },

    {

      "type": "portmap",

      "capabilities": { "portMappings": true }

    }

  ]

}

In this example, the bridge plugin builds the connectivity, host-local hands out addresses from the node's subnet, and portmap adds support for hostPort mappings.

6. Plugin Types and IPAM

The CNI project ships reference plugins in three groups, and production network add-ons reuse or replace these ideas.

Group

Examples and role

Main (interface creation)

bridge, ptp, macvlan, ipvlan and host-device create the network interface for the container

IPAM (address management)

host-local allocates from a per-node range, dhcp asks a DHCP server, static assigns fixed addresses

Meta (extra behaviour)

portmap handles hostPort, bandwidth limits traffic, tuning sets sysctls, firewall and vrf adjust filtering and routing

 

How per-node ranges work. The cluster has a Pod CIDR, for example 10.244.0.0/16. When node CIDR allocation is enabled, the controller manager gives each node a slice of it, a /24 by default for IPv4, which is about 254 addresses. IPAM on that node allocates Pod addresses only from its own slice. Because every node owns a distinct range, the network can route to a Pod simply by knowing which node owns its subnet. Some plugins, such as Cilium and the cloud-native ones, manage addresses in their own way instead.

7. How Pod Traffic Travels

7.1 Pods on the Same Node

In a bridge-based design, each Pod's veth connects to a Linux bridge on the node, and the bridge forwards frames between Pods like a small switch. In routed or eBPF-based designs, there is no bridge: the host end of each veth is a routed interface, and the kernel or an eBPF program forwards the packet directly to the destination veth. Either way, same-node traffic never leaves the machine.

7.2 Pods on Different Nodes

Cross-node traffic is where plugins differ most. There are three broad approaches.

Approach

How it works

Trade-offs

Overlay (VXLAN, Geneve, IP-in-IP)

Pod packets are encapsulated inside packets addressed between node IPs, then unwrapped on the destination node

Works on almost any underlay network; adds header overhead (VXLAN adds about 50 bytes) so MTU must be set correctly

Routed (BGP or static routes)

Nodes advertise their Pod subnets to the network, and packets travel unencapsulated

Better performance and easier tracing; requires an underlay that accepts the routes

Cloud-native (VPC routing or ENI IPs)

Pods receive addresses from the cloud network itself, for example the Amazon VPC CNI uses VPC IP addresses

Pods are directly routable inside the cloud network; Pod density can depend on cloud address limits

 

A common mistake is to treat the overlay as free. Encapsulation reduces the usable packet size, and a wrong MTU shows up as connections that work for small requests but hang on large transfers.

8. Network Policy and CNI

The NetworkPolicy resource lets you define which Pods may talk to which. It is an API only: Kubernetes stores the object, but enforcement is the job of the network plugin. If your plugin does not implement NetworkPolicy, you can create policies without error and they will have no effect. This is one of the most important facts to check when selecting a CNI.

A common first step is a default-deny policy for a namespace, followed by explicit allow rules:

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: default-deny-ingress

  namespace: payments

spec:

  podSelector: {}

  policyTypes:

  - Ingress

The empty podSelector matches every Pod in the namespace, and because no ingress rules are listed, all incoming traffic is denied until other policies allow it. Some plugins add their own, richer policy resources beyond the standard API, for example global rules or Layer 7 aware rules, but those are specific to that plugin.

9. Popular CNI Plugins

Plugin

Data path

NetworkPolicy

Typical fit

Flannel

VXLAN overlay by default, with a host-gw option

Not provided by Flannel itself

Small or simple clusters and learning environments

Calico

BGP routing, or IP-in-IP or VXLAN encapsulation; optional eBPF data plane

Yes, plus extended Calico policies

Clusters needing strong policy control or routed networking

Cilium

eBPF-based; tunnelling or native routing

Yes, plus identity-aware and Layer 7 policies

Observability, security focus and kube-proxy replacement

Amazon VPC CNI

Pods use VPC IP addresses from ENIs

Supported in recent versions

Amazon EKS clusters

Multus

Meta-plugin that attaches several interfaces to one Pod

Depends on the delegated plugins

Telecom and workloads needing secondary networks

 

Managed Kubernetes services usually come with a default CNI, such as the Amazon VPC CNI on EKS, so check what your provider installs and which alternatives it supports before migrating. Capabilities change between releases, so confirm details in each project's documentation.

10. Hands-On: Inspecting Pod Networking

10.1 From the Cluster

kubectl get pods -A -o wide

kubectl get node <node-name> -o jsonpath='{.spec.podCIDR}'

kubectl exec <pod-name> -- ip addr

kubectl exec <pod-name> -- ip route

The first command shows each Pod's IP and node. The second shows the subnet assigned to a node. The last two show the interface and routes inside the Pod, where you should see eth0 with an address from that node's range. If the container image has no ip tool, use a debug container instead.

10.2 From the Node

ls /etc/cni/net.d/

ls /opt/cni/bin/

ip link show type veth

ip route

bridge link

These commands show the CNI configuration and binaries, the host ends of the Pods' veth pairs, the routes to local and remote Pod subnets, and, on bridge-based setups, the interfaces attached to the bridge. Output varies by plugin, which is itself a useful way to recognise what design your cluster uses.

11. Setting Up a Cluster Network

A new cluster has no Pod network until you install one. With kubeadm, you initialise the control plane with a Pod CIDR that matches your chosen plugin, then apply the plugin's manifest or Helm chart:

kubeadm init --pod-network-cidr=10.244.0.0/16

Until a CNI is installed, nodes report NotReady and CoreDNS Pods stay Pending, because the runtime reports that the network plugin is not initialised. This is expected behaviour, not a fault. Each plugin's documentation states which CIDR it expects, so always follow its installation guide for the exact flags and values.

12. Choosing a CNI

There is no single best plugin. Decide using these questions:

  • Do you need NetworkPolicy? If yes, choose a plugin that enforces it, or pair a simple data path with a policy engine.

  • What does your underlay allow? If you cannot advertise routes, an overlay is simpler. If you can, routed designs perform better.

  • Where does the cluster run? On a public cloud, the provider's native CNI often integrates best with load balancers, security groups and IP management.

  • What visibility do you need? eBPF-based plugins can provide flow-level observability without extra agents.

  • What scale do you expect? Check limits on nodes, Pods per node and IP addresses, especially for cloud-native designs.

  • What can your team operate? A simpler plugin that you understand is better than a powerful one you cannot debug.

13. Troubleshooting

Symptom

Likely cause and check

Node stays NotReady

No CNI installed, or the configuration is missing or invalid in /etc/cni/net.d. Check the runtime and CNI DaemonSet logs.

Pod stuck in ContainerCreating

The CNI ADD call failed. Run kubectl describe pod and read the events, which usually include the plugin's error message.

Pods get no IP, or "no IP addresses available"

The node's Pod range or the cloud address pool is exhausted. Check CIDR size and per-node limits.

Pod-to-Pod works on one node but not across nodes

Overlay or routing is broken. Check that the underlay allows the tunnel protocol and ports, node firewalls and security groups.

Large transfers hang while small requests work

MTU mismatch caused by encapsulation overhead. Lower the Pod MTU or raise the underlay MTU.

NetworkPolicy has no effect

The CNI does not enforce NetworkPolicy. Verify plugin support and that its policy component is running.

Unexpected behaviour after adding a config file

Multiple files in /etc/cni/net.d; the runtime uses the first alphabetically. Remove stale files.

 

A reliable method is to work outwards: confirm the Pod has an interface and IP, then test same-node Pod traffic, then cross-node traffic, then Service access. Each step isolates a different layer.

14. Best Practices

  • Plan address ranges before building the cluster. Pod, Service and node ranges must not overlap with each other or with networks you need to reach, such as VPCs or on-premises ranges.

  • Size the Pod CIDR for growth. Remember that each node reserves its own slice, and that the default limit is 110 Pods per node.

  • Set the MTU deliberately when using an overlay, and test with large payloads.

  • Run one primary CNI per cluster. Use a meta-plugin such as Multus only when Pods truly need extra interfaces.

  • Apply default-deny NetworkPolicies and open only the flows you need.

  • Test CNI upgrades in a non-production cluster first, and upgrade node by node, because the network plugin affects every Pod.

  • Monitor the CNI Pods and your plugin's metrics, and alert on IP exhaustion before it affects scheduling.

15. Conclusion

CNI is the small interface that turns the Kubernetes promise of a flat Pod network into real interfaces, addresses and routes. The container runtime calls a plugin whenever a Pod sandbox is created, the plugin wires the Pod's network namespace to the node, and the chosen data path, whether overlay, routed or cloud-native, carries the traffic between nodes. NetworkPolicy enforcement and observability depend on that same plugin, which is why the choice of CNI is one of the most consequential decisions in a cluster design.

With a clear picture of the ADD flow, the files under /etc/cni/net.d and /opt/cni/bin, and the layered troubleshooting approach in this article, you can move from guessing to diagnosing when Pod networking misbehaves.