kubernetes

Kubernetes kube-proxy Explained: Service Traffic and Packet Routing

Service Traffic and Packet Routing

By Shubhankar Tripathi • • 5 min read

Kubernetes kube-proxy Explained: Service Traffic and Packet Routing

bitcodematrix.com | Kubernetes Series

1. Introduction

Pods are temporary. They are created, rescheduled and replaced, and each new Pod gets a new IP address. Applications cannot depend on those addresses, so Kubernetes gives them a stable virtual address through the Service object. But a Service is only a record in the API server. Something on every node must turn that record into real packet forwarding, and that something is kube-proxy.

This article explains what kube-proxy does, how it keeps node rules in sync with the cluster, how its proxy modes differ, and how a packet addressed to a Service IP ends up at a Pod. It closes with hands-on inspection commands, a troubleshooting guide and best practices.

2. The Problem kube-proxy Solves

Consider a frontend that needs to call a backend running as three replicas. The backend Pod IPs change whenever a Pod restarts, and the frontend should spread requests across all healthy replicas. A Service solves this by providing:

  • A stable virtual IP (the ClusterIP) and a DNS name that do not change for the life of the Service.

  • A selector that continuously tracks the set of ready backend Pods.

  • Load distribution across those Pods.

The ClusterIP is not assigned to any network interface, and no process listens on it. It exists only as a match condition in packet-filtering rules on each node. kube-proxy is the component that writes those rules.

3. What kube-proxy Is and Is Not

kube-proxy is a node-level agent that runs on every node, usually as a DaemonSet in the kube-system namespace. Its job is to watch the API server and program the node's kernel so that traffic to Service addresses is redirected to backend Pods.

Despite its name, in the modern modes it is not in the data path. kube-proxy configures the kernel and then steps aside; the kernel forwards the packets. Because of this, restarting kube-proxy does not interrupt existing connections, and the rules keep working while the agent is down, although they will not be updated until it returns.

kube-proxy is

kube-proxy is not

A controller that translates Services and EndpointSlices into kernel rules

A traffic-carrying proxy process in the iptables and nftables modes

Responsible for ClusterIP, NodePort and LoadBalancer traffic forwarding on a node

Responsible for giving Pods their IP addresses; that is the CNI plugin

Aware of Pod readiness and traffic policies

An ingress controller or Gateway API implementation, which handle HTTP-level routing

 

The division of labour is worth remembering: the CNI plugin makes Pods reachable by IP address, and kube-proxy makes Service addresses work on top of that connectivity.

4. How kube-proxy Works: The Sync Loop

kube-proxy follows a straightforward control loop:

  1. It connects to the API server and sets up watches on Service and EndpointSlice objects, and on Node information where needed.

  2. When a Service or its endpoints change, it updates an in-memory view of the desired state.

  3. It computes the rules needed for that state and writes them to the kernel, in batches, using the active backend (iptables, nftables or another).

  4. It repeats this periodically as well as on change, so that manual or accidental changes to its rules are corrected.

EndpointSlices are the objects that list the Pod IPs and ports behind a Service. They are split into small slices so that large Services do not produce huge objects, and the older Endpoints API is deprecated in their favour. Only endpoints that are ready are programmed as destinations, which is why a Pod failing its readiness probe stops receiving Service traffic without being deleted.

5. Which Service Types Involve kube-proxy

Service type

How it is reached

Role of kube-proxy

ClusterIP

Virtual IP, reachable only inside the cluster

Programs rules that redirect the IP and port to a backend Pod

NodePort

Every node IP on a port, by default in the range 30000 to 32767

Adds rules for the node port in addition to the ClusterIP rules

LoadBalancer

An external load balancer created by the cloud provider, which forwards to node ports

Handles the node-level forwarding behind the load balancer

Headless (clusterIP: None)

DNS returns the Pod IPs directly

Not involved; no virtual IP exists

ExternalName

DNS CNAME to an external name

Not involved; resolution happens in DNS only

 

6. Proxy Modes

The mode is chosen with the mode field in the kube-proxy configuration. Each mode uses a different kernel facility to implement the same Service behaviour.

6.1 iptables Mode

This is the long-standing default on Linux. kube-proxy writes rules into the nat table of iptables using a set of dedicated chains. Rules are evaluated one after another, so lookup time grows with the number of Services and endpoints, and every update rewrites a larger rule set. For most clusters this is perfectly adequate, but it is the main reason the alternatives exist.

6.2 nftables Mode

The nftables mode uses the newer Linux packet-filtering framework, which replaces iptables. kube-proxy creates its own table named kube-proxy and uses maps and sets, so Service lookup does not slow down as the number of Services grows, and rule updates are more efficient. The mode reached general availability in Kubernetes 1.33. It requires a sufficiently recent kernel (version 5.13 or later) and nft tooling, so verify node images before enabling it.

6.3 IPVS Mode (Deprecated)

IPVS mode used the kernel's IP Virtual Server load balancer, which offers hash-table lookups and several scheduling algorithms. It was introduced to address iptables scaling limits. It has been deprecated since Kubernetes 1.35, and the project recommends moving to nftables, so new clusters should not adopt it and existing IPVS clusters should plan a migration.

6.4 Windows and Removed Modes

On Windows nodes, kube-proxy uses its own kernelspace mode, built on the Windows networking stack. The original userspace mode, in which kube-proxy itself copied traffic, was the first implementation but is no longer available on Linux.

6.5 Comparison

Mode

Kernel facility

Lookup behaviour

Status

iptables

iptables nat chains

Sequential; grows with Service count

Long-standing default on Linux

nftables

nftables maps and sets

Constant-time lookup

Generally available since 1.33; recommended direction

ipvs

IPVS plus iptables helpers

Hash-table lookup

Deprecated since 1.35

 

Mode availability and defaults change between releases, so check the release notes for your Kubernetes version before changing modes.

7. The Journey of a ClusterIP Packet

The best way to understand kube-proxy is to follow one request. Assume a Pod at 10.244.1.10 calls a Service with ClusterIP 10.96.45.12 on port 80, which has three backend Pods listening on port 8080, one of them at 10.244.2.7.

  1. The client application resolves the Service name through cluster DNS and gets the ClusterIP 10.96.45.12.

  2. It sends a packet to 10.96.45.12:80. The packet leaves the Pod through its veth interface and enters the node's network stack.

  3. The packet reaches the netfilter hooks. In iptables mode, the entry chain KUBE-SERVICES matches the destination IP and port and jumps to the Service's own chain.

  4. The Service chain selects one backend. In iptables mode it does so with a random probability match; in nftables mode it uses a map lookup.

  5. The endpoint chain performs DNAT, rewriting the destination from 10.96.45.12:80 to 10.244.2.7:8080. The kernel's connection tracking (conntrack) records this translation.

  6. The node now routes the packet to 10.244.2.7 like any other Pod traffic. If the destination is on another node, the CNI plugin's overlay or routing carries it there.

  7. The backend replies to the client Pod's IP. When the reply passes back through the client's node, conntrack reverses the translation so the reply appears to come from 10.96.45.12, which is what the client expects.

Because the translation happens on the client's node, the decision is made locally and no central load balancer is involved. Each node load balances independently, which makes the design scale with the cluster.

7.1 A Simplified iptables Example

The rules below are simplified from the output of iptables-save. The chain name suffixes are hashes generated by kube-proxy, shown here as placeholders.

-A KUBE-SERVICES -d 10.96.45.12/32 -p tcp --dport 80 -j KUBE-SVC-EXAMPLE

 

-A KUBE-SVC-EXAMPLE -m statistic --mode random --probability 0.33333 -j KUBE-SEP-AAAA

-A KUBE-SVC-EXAMPLE -m statistic --mode random --probability 0.50000 -j KUBE-SEP-BBBB

-A KUBE-SVC-EXAMPLE -j KUBE-SEP-CCCC

 

-A KUBE-SEP-BBBB -p tcp -j DNAT --to-destination 10.244.2.7:8080

The three probabilities produce an even split: the first rule takes one third of connections, the second takes half of the remainder, and the last takes whatever is left. Note that load is balanced per connection, not per request, so long-lived connections such as HTTP keep-alive or gRPC streams stay on one backend.

Also note that a ClusterIP does not respond to ping. The rules match only the Service's configured protocol and port, so ICMP to the virtual IP has nothing to match. Test Services with the actual protocol, such as curl or a TCP connection.

8. NodePort and LoadBalancer Traffic

For NodePort Services, kube-proxy installs rules (in the KUBE-NODEPORTS chain in iptables mode) so that traffic arriving at any node on the node port is sent to the Service's backends. A LoadBalancer Service builds on this: the cloud load balancer forwards to node ports, and kube-proxy forwards onward to Pods.

Because the node that receives the packet may not host a backend Pod, the packet may need to travel to another node. To make the return path work, kube-proxy can also masquerade (source NAT) the packet so the reply comes back through the same node. The side effect is that the backend sees the node's IP instead of the real client IP.

8.1 externalTrafficPolicy

Value

Behaviour

Trade-off

Cluster (default)

Traffic may be forwarded to a Pod on any node, with source NAT where needed

Even spread across all Pods, but the client source IP is lost

Local

Traffic is sent only to Pods on the node that received it, with no extra hop or source NAT

Preserves the client IP, but a node with no local Pod drops the traffic and load can be uneven; load balancers rely on a health check to avoid such nodes

 

A matching field, internalTrafficPolicy, applies the same idea to traffic that originates inside the cluster. Setting it to Local keeps in-cluster traffic on the same node as the client when a local endpoint exists.

9. Session Affinity and Topology Preferences

By default, each new connection may land on a different backend. Setting sessionAffinity: ClientIP on a Service makes kube-proxy send connections from the same client IP to the same backend for a timeout period, which is 10800 seconds (three hours) by default.

apiVersion: v1

kind: Service

metadata:

  name: backend

spec:

  selector:

    app: backend

  ports:

  - port: 80

    targetPort: 8080

  sessionAffinity: ClientIP

  sessionAffinityConfig:

    clientIP:

      timeoutSeconds: 3600

The Service field trafficDistribution lets you express a preference for endpoints that are topologically close to the client, for example the same zone, which can reduce latency and cross-zone data charges. kube-proxy honours this preference when programming endpoints. Check your Kubernetes version's documentation for the values supported.

10. Configuring kube-proxy

kube-proxy reads a KubeProxyConfiguration file. In clusters built with kubeadm, it is stored in a ConfigMap named kube-proxy in the kube-system namespace and mounted into the DaemonSet Pods. A minimal example looks like this:

apiVersion: kubeproxy.config.k8s.io/v1alpha1

kind: KubeProxyConfiguration

mode: "nftables"

clusterCIDR: "10.244.0.0/16"

conntrack:

  maxPerCore: 32768

  min: 131072

The mode field selects the backend, clusterCIDR tells kube-proxy which addresses are Pod addresses so it can decide when to masquerade, and the conntrack block controls the connection-tracking table size. To change the mode, edit the ConfigMap and restart the DaemonSet:

kubectl edit configmap kube-proxy -n kube-system

kubectl rollout restart daemonset kube-proxy -n kube-system

Managed Kubernetes services often control this configuration themselves, so use the provider's documented method instead of editing it directly.

11. Hands-On: Inspecting kube-proxy

11.1 From the Cluster

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50

kubectl get svc,endpointslices -n <namespace>

kubectl describe svc <service-name> -n <namespace>

The logs report the proxy mode in use and any sync errors. The Service and EndpointSlice listing shows whether the Service has ready endpoints, which is the first thing to confirm when a Service does not respond.

11.2 From the Node

curl localhost:10249/proxyMode

sudo iptables-save -t nat | grep <service-name>

sudo nft list table ip kube-proxy

sudo conntrack -L | grep <cluster-ip>

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

The first command asks kube-proxy's local endpoint which mode it runs. Use the iptables-save command in iptables mode and the nft command in nftables mode. The conntrack commands show live translated connections and how full the tracking table is.

12. Replacing kube-proxy with eBPF

Some CNI plugins, notably Cilium and Calico in its eBPF mode, can implement Service load balancing directly in the kernel using eBPF programs. This removes the need for kube-proxy altogether, avoids long rule chains and can resolve Services earlier in the packet path. With kubeadm, kube-proxy installation is skipped with the option below, and the CNI plugin must be configured for kube-proxy replacement:

kubeadm init --skip-phases=addon/kube-proxy

This is a design decision that ties Service handling to your CNI. If you adopt it, follow the plugin's documentation closely, and remember that features such as NodePort handling and traffic policies are then provided by that plugin instead.

13. Troubleshooting

Symptom

Likely cause and check

Service times out or is refused

No ready endpoints. Check kubectl get endpointslices, the Service selector and Pod readiness.

Connects on port but wrong behaviour

Mismatch between port and targetPort, or the application listens on a different port than the Service targets.

Works from some nodes only

kube-proxy not running or failing on the affected nodes. Check the DaemonSet Pods and logs on those nodes.

Rules missing or stale

kube-proxy cannot sync. Look for errors in its logs, and check that the kernel modules and tools for the chosen mode are available.

Intermittent drops under load

Conntrack table full. Compare nf_conntrack_count with nf_conntrack_max and raise the limit if needed.

NodePort unreachable from outside

Node firewall or cloud security group blocks the port, or externalTrafficPolicy is Local and the node has no local endpoint.

Backend sees node IP, not client IP

Source NAT with externalTrafficPolicy Cluster. Use Local if you must preserve the client IP.

Pod cannot reach its own Service

Hairpin traffic issue with the CNI or bridge settings. Check the CNI plugin's hairpin configuration.

Pod IP works but Service IP does not

Pod networking is fine and the fault lies in the Service layer: check kube-proxy and the endpoints.

 

A dependable order is to test the Pod IP directly first, then the Service ClusterIP, then the NodePort. If the Pod IP works and the ClusterIP does not, kube-proxy and the Service definition are the place to look; if the Pod IP fails, the problem is in the CNI layer.

14. Best Practices

  • Know which mode each cluster runs, and plan a move from IPVS to nftables on any cluster still using IPVS.

  • Verify kernel and tooling requirements on node images before enabling nftables mode.

  • Always define readiness probes so that endpoints are removed from Services while Pods are not able to serve traffic.

  • Monitor the conntrack table, and size it for busy nodes with many short connections.

  • Use externalTrafficPolicy: Local only when you need the client IP, and make sure Pods run on the nodes that receive traffic.

  • Remember that load balancing is per connection, and use connection limits or client-side balancing for long-lived connections such as gRPC.

  • Keep the kube-proxy version in step with the cluster version, and review the configuration after each upgrade.

  • If you adopt an eBPF replacement, treat it as a CNI decision and test it in a non-production cluster first.

15. Conclusion

kube-proxy is the quiet component that makes Services work. It watches Services and EndpointSlices, converts them into kernel rules, and leaves the actual packet forwarding to netfilter. A request to a ClusterIP is simply rewritten with DNAT on the client's node, sent to a chosen Pod, and translated back on the return path.

Knowing the proxy modes, the packet journey and the traffic policies turns Service problems from guesswork into a short checklist: confirm the endpoints, confirm the kube-proxy rules, confirm connection tracking, and then look at the network underneath. As the ecosystem moves from iptables and IPVS toward nftables and eBPF, those same fundamentals continue to apply.