kubernetes

Kubernetes Network Troubleshooting: A Practical Step-by-Step Guide

Kubernetes Network Troubleshooting: A Practical Step-by-Step Guide

By Shubhankar Tripathi • • 5 min read

Kubernetes Network Troubleshooting: A Practical Step-by-Step Guide

bitcodematrix.com | Kubernetes Series

1. Introduction

Network problems in Kubernetes are hard because one request crosses many layers: the application, the Pod network built by the CNI plugin, Services programmed by kube-proxy, DNS, NetworkPolicies, an ingress or load balancer, and finally the nodes and the cloud or data-centre network underneath. A single "connection timed out" message can come from any of them.

This guide gives you a repeatable method. You will learn a layered mental model, how to prepare a debugging toolbox, and a nine-step workflow that moves from the application outwards, eliminating one layer at a time. It finishes with an error-message decoder, four worked case studies and prevention practices. It builds on the earlier articles in this series on CNI, kube-proxy and DNS, which go deeper into those individual layers.

2. A Layered Mental Model

Troubleshooting becomes much faster when you know which layer owns which symptom. Use this table as a map.

Layer

Component

Typical symptom

First check

1. Application

The process in the container

Connection refused on the port; app listens only on loopback

Logs and listening sockets inside the Pod

2. Pod network

CNI plugin

Pod has no IP; Pod-to-Pod fails, often only across nodes

kubectl describe pod; ip addr and ip route in the Pod

3. Service

EndpointSlices and kube-proxy

Pod IP works but ClusterIP does not

Service endpoints; kube-proxy logs

4. DNS

CoreDNS

Name fails while the IP works

nslookup from a debug Pod

5. Policy

NetworkPolicy (enforced by the CNI)

Timeouts for some flows only

kubectl get networkpolicy -A

6. External entry

NodePort, LoadBalancer, Ingress or Gateway

502, 503 and 504 errors; external timeouts

Resource status and controller logs

7. Node and underlay

Operating system, firewall, cloud network, MTU

Fails for some nodes only; large transfers hang

tcpdump, MTU test, security groups

 

The key idea is to test the lowest layer first that could explain the symptom, and move up only when it passes. If the Pod IP answers, layers 1 and 2 are healthy for that path, and there is no need to inspect the CNI.

3. Before You Start: Define the Problem

Most wasted debugging time comes from a vague problem statement. Before you run any command, write down the answers to these questions:

  1. Who talks to whom? Name the source Pod or client and the exact destination: a Pod IP, a Service name, a NodePort or an external URL.

  2. Which protocol and port? TCP, UDP or ICMP, and the port number. Ping working says nothing about TCP port 8080.

  3. What exactly is the error? Copy the exact message. Refused, timed out, no route and name not known mean very different things (see Section 6).

  4. Is it constant or intermittent, and for everyone or for some Pods and nodes? Intermittent or node-specific failures point to different causes than total failures.

  5. What changed? New deployment, NetworkPolicy, node upgrade, CNI or kube-proxy change, certificate or cloud security group edit.

  6. Did it ever work? A path that never worked is usually a configuration error; one that stopped working is usually a change or a failure.

4. Build the Toolbox

Application images are normally minimal and lack networking tools. Keep a debug image ready and run it in the same namespace as the failing workload, because namespace-specific policies and DNS search domains affect the result.

kubectl run nettest -n <namespace> --image=nicolaka/netshoot \

  --restart=Never --command -- sleep 3600

The netshoot image bundles curl, dig, nslookup, nc, ss, ip, traceroute, tcpdump and iperf3. To test a path from a specific node, pin the debug Pod to that node:

kubectl run nettest-n2 -n <namespace> --image=nicolaka/netshoot \

  --restart=Never \

  --overrides='{"spec":{"nodeName":"<node-name>"}}' \

  --command -- sleep 3600

To inspect a running Pod without changing it, attach an ephemeral debug container. It shares the Pod's network namespace, so you see exactly what the application sees:

kubectl debug -it <pod-name> -n <namespace> --image=nicolaka/netshoot

Remove debug Pods when you finish. Also keep SSH or node-debug access in mind for node-level tools such as ip, ss, tcpdump, conntrack, iptables and nft.

5. The Step-by-Step Workflow

5.1 Step 1: Check That the Pod Is Healthy

A Pod that is not Ready is removed from Service endpoints, which looks like a network fault but is not one.

kubectl get pods -n <namespace> -o wide

kubectl describe pod <pod-name> -n <namespace>

kubectl logs <pod-name> -n <namespace>

What to look for: the Pod must be Running and Ready, with an IP address and a node. In the events, look for failed probes, restarts and sandbox creation errors. Then confirm the application is actually listening on the expected port and on the right address:

kubectl exec -it <pod-name> -n <namespace> -- ss -lnt

A very common fault is an application bound to 127.0.0.1 instead of 0.0.0.0. It then works from inside the Pod and through kubectl port-forward, but is refused when reached through the Pod IP or a Service. If the image has no ss tool, use an ephemeral debug container.

5.2 Step 2: Check That the Pod Has a Network

kubectl exec -it nettest -n <namespace> -- ip addr

kubectl exec -it nettest -n <namespace> -- ip route

What to look for: an eth0 interface with an address from the node's Pod range and a default route. If a Pod is stuck in ContainerCreating, describe it and look for FailedCreatePodSandBox events that mention the CNI plugin. These indicate a CNI problem such as a missing configuration in /etc/cni/net.d, an exhausted IP pool, or a crashed CNI agent. Check the CNI Pods in kube-system on the affected node.

5.3 Step 3: Test Pod-to-Pod Connectivity by IP

Bypass Services and DNS completely and talk directly to the destination Pod IP, first from the same node and then from a different node.

kubectl get pod <target-pod> -n <namespace> -o wide

kubectl exec -it nettest -n <namespace> -- curl -sv --max-time 5 http://<pod-ip>:<port>

kubectl exec -it nettest -n <namespace> -- nc -zv -w 3 <pod-ip> <port>

Interpreting the results:

  • Works from the same node but fails from another node: the cross-node data path is broken. Suspect the overlay or routing, a blocked tunnel port on the underlay, an MTU problem or a CNI agent failure on one node.

  • Refused: the network works and nothing is listening on that port. Return to Step 1.

  • Timeout from everywhere: suspect a NetworkPolicy (Step 6) or a node firewall.

Do not rely on ping alone, because ICMP is often filtered and tells you nothing about TCP or UDP ports.

5.4 Step 4: Test DNS

kubectl exec -it nettest -n <namespace> -- nslookup kubernetes.default

kubectl exec -it nettest -n <namespace> -- nslookup <service>.<namespace>.svc.cluster.local

If the IP works but the name does not, the problem lies in DNS, not in the network path to the application. Follow the DNS troubleshooting workflow from the earlier article: check resolv.conf, CoreDNS health, the kube-dns endpoints and any NetworkPolicy blocking port 53 over UDP and TCP.

5.5 Step 5: Test the Service

If the Pod IP works but the Service does not, the fault is in the Service layer. Check the Service definition and its endpoints:

kubectl get svc <service> -n <namespace> -o wide

kubectl describe svc <service> -n <namespace>

kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service>

Common findings:

  • No endpoints: the selector does not match the Pod labels, or the Pods are not Ready. Compare the Service selector with the labels shown by kubectl get pods --show-labels.

  • Wrong targetPort: the Service forwards to a port the application does not use. The Pod IP works on the real port, but the Service fails.

  • Endpoints are present but traffic fails: test the ClusterIP and port from the debug Pod, and then check kube-proxy.

kubectl exec -it nettest -n <namespace> -- curl -sv --max-time 5 http://<cluster-ip>:<port>

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50

Remember that a ClusterIP does not answer ping, because the rules match only the Service's protocol and port. Use curl or nc. A useful trick for separating the application from the network is kubectl port-forward, which reaches the Pod through the API server and kubelet rather than the Pod network:

kubectl port-forward pod/<pod-name> 8080:<port> -n <namespace>

If port-forward works but access by Pod IP does not, suspect the loopback binding from Step 1, a NetworkPolicy or the cross-node path.

5.6 Step 6: Check NetworkPolicies

kubectl get networkpolicy -A

kubectl describe networkpolicy <name> -n <namespace>

Keep these rules in mind when reading policies:

  • A Pod is unrestricted for a direction (ingress or egress) until some policy selects it for that direction. After that, only traffic allowed by the union of all selecting policies is permitted.

  • A flow needs permission at both ends: egress allowed on the source Pod and ingress allowed on the destination Pod.

  • A default-deny egress policy also blocks DNS, so you must explicitly allow port 53 to CoreDNS over UDP and TCP.

  • Policies are enforced only if your CNI plugin supports them. Policies on an unsupported plugin are accepted by the API but silently ignored.

To confirm a policy is the cause, test in a non-production namespace by temporarily adding an allow rule or running the same test from a Pod the policy does not select. Never remove production policies as a first diagnostic step.

5.7 Step 7: Check External Entry Points

For traffic coming from outside the cluster, identify which entry point is used and check each hop.

Entry point

What to check

NodePort

The Service has endpoints; the node firewall and cloud security group allow the port (default range 30000 to 32767); externalTrafficPolicy: Local only works on nodes that run a backend Pod

LoadBalancer

EXTERNAL-IP is not stuck in pending (a pending address usually means no cloud controller or load balancer provisioner is available); health checks pass on the node ports; security groups allow load balancer to node traffic

Ingress

kubectl describe ingress shows the expected backend; the ingress controller Pods are healthy; read the controller logs for upstream errors

Gateway API

kubectl describe gateway and httproute; the Gateway shows Programmed True and an address; the route shows Accepted and ResolvedRefs True

 

Test from inside the cluster first by sending the request straight to the Service from a debug Pod. If that works, the fault is in the external layer; if it fails, the fault is in the cluster path and Steps 1 to 6 apply.

5.8 Step 8: Check Egress to the Outside World

kubectl exec -it nettest -n <namespace> -- nslookup example.com

kubectl exec -it nettest -n <namespace> -- curl -sv --max-time 5 https://example.com

If DNS resolves but the connection times out, check the path out of the cluster: egress NetworkPolicies, the node's route to the internet, the NAT or masquerade of Pod addresses (done by the CNI plugin or a cloud NAT gateway), firewall rules, and any required HTTP proxy settings in the application. Private clusters need an explicit NAT gateway or proxy for internet access.

5.9 Step 9: Check the Node and the Underlay

When failures depend on the node, or only large transfers fail, look below Kubernetes. Run these on the affected node:

ip route get <pod-ip>

ip link show type veth

sysctl net.ipv4.ip_forward

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

sudo iptables-save | grep -i drop

sudo nft list ruleset | grep -i drop

Underlay ports. Overlay networks need the underlay to allow the encapsulation traffic between nodes. Typical examples are VXLAN on UDP port 4789 or 8472 depending on the plugin, Geneve on UDP 6081, and IP-in-IP as IP protocol 4. Check your CNI plugin's documentation for the exact ports and protocols, and compare them with node firewalls and cloud security groups.

MTU. Encapsulation reduces the usable packet size. A mismatch lets small requests succeed while large responses or uploads stall. Test with the do-not-fragment option from a debug Pod:

kubectl exec -it nettest -n <namespace> -- ping -M do -s 1472 -c 3 <destination-ip>

kubectl exec -it nettest -n <namespace> -- ping -M do -s 1372 -c 3 <destination-ip>

For a path with a 1500-byte MTU, a payload of 1472 bytes fits exactly. If the larger test fails and the smaller succeeds, lower the Pod MTU or fix the underlay MTU. Adjust the sizes to match the MTU you expect.

Packet capture. When everything else is inconclusive, capture packets on both ends and see where they stop. Inside the Pod network namespace, use an ephemeral container; on the node, capture on the Pod's veth interface or the node's main interface:

kubectl debug -it <pod-name> -n <namespace> --image=nicolaka/netshoot

tcpdump -i any -nn host <peer-ip> and port <port>

If the packet leaves the source but never arrives at the destination, the loss is in between: policy, routing, firewall or the tunnel. If it arrives but no reply is sent, the destination application or its local policy is at fault. If a reply is sent but never arrives, check the return path, which is often asymmetric or blocked.

6. Decoding Error Messages

Message or symptom

What it usually means

Connection refused

The destination host was reached and actively rejected the connection: nothing listens on that port, or the app is bound to loopback. The network is working.

Connection timed out

Packets are being dropped silently: NetworkPolicy, firewall or security group, a broken route or tunnel, or a destination that is down.

No route to host

The sender has no valid route, or an ICMP unreachable came back: routing problem, missing endpoint, or a reject rule.

Name or service not known; temporary failure in name resolution

DNS failure. Test with nslookup and use the DNS workflow.

Connection reset by peer

The connection was established, then closed abruptly: application crash, timeout in a middlebox, or a protocol mismatch such as HTTP to a TLS port.

Works for small requests, hangs on large ones

MTU mismatch, usually caused by encapsulation overhead.

Delay of about 5 seconds, then works

A lost first packet and retry; commonly seen with DNS or conntrack problems.

HTTP 502

The proxy or gateway reached the backend but got an invalid or no response: crashed backend, wrong port or protocol mismatch.

HTTP 503

The proxy has no healthy backend available: no ready endpoints, failing health checks or an unresolved reference.

HTTP 504

The backend did not answer in time: slow application or a timeout setting that is too low.

 

7. Four Worked Case Studies

7.1 Case 1: Service Refuses Connections, Pod IP Works

A frontend gets "connection refused" calling the backend Service on port 80. From a debug Pod, curl to the backend Pod IP on port 3000 works, but the ClusterIP on port 80 is refused. Describing the Service shows targetPort 8080, while the application listens on 3000. Fix: change targetPort to 3000. The lesson: when the Pod IP works and the Service does not, compare the Service ports with what the application really listens on.

7.2 Case 2: Timeouts Only Between Nodes

Pods on the same node talk fine, but any Pod-to-Pod call across nodes times out. Pinned debug Pods confirm this pattern. The CNI Pods are Running, but a packet capture on the source node shows encapsulated packets leaving and none arriving at the destination node. The cause is a cloud security group that blocks the overlay's UDP port between nodes. Fix: allow the CNI plugin's tunnel port between all nodes. The lesson: same-node success with cross-node failure points at the underlay.

7.3 Case 3: Everything Times Out After a Default-Deny Policy

After a security team applies a default-deny egress policy to a namespace, applications log "temporary failure in name resolution" and calls to other services time out. The policy blocks DNS and the application flows. Fix: add an egress rule allowing UDP and TCP port 53 to CoreDNS, and add specific rules for each required service, with matching ingress rules on the destination namespaces. The lesson: deny policies need explicit DNS and application allowances at both ends.

7.4 Case 4: Large Uploads Hang

API calls work, but file uploads stall after a few kilobytes. Small pings succeed and a ping with a 1472-byte payload and the do-not-fragment option fails, while a 1372-byte one succeeds. The overlay adds headers and the Pod MTU is still 1500. Fix: set the Pod interface MTU in the CNI configuration to a value that fits inside the underlay MTU after encapsulation. The lesson: if only big transfers fail, test the MTU before anything else.

8. Collecting Evidence for Escalation

If you must escalate to a platform team or vendor, a complete report saves days. Include:

  • The exact source, destination, protocol, port and the exact error message.

  • The output of kubectl get pods -o wide for both ends, and the Service and EndpointSlice details.

  • Which steps of this workflow passed and which failed, with the commands used.

  • Logs from the CNI Pods, kube-proxy and CoreDNS on the affected nodes for the time of the failure.

  • The Kubernetes version, the CNI plugin and version, the kube-proxy mode, and any recent changes.

  • A packet capture from both ends, if you took one.

9. Prevention and Best Practices

  • Define readiness probes for every workload so that unready Pods never receive Service traffic.

  • Keep a runbook with the layered workflow and a ready-to-use debug image mirrored in your own registry.

  • Monitor the network layers: CoreDNS error rates and latency, kube-proxy sync errors, CNI agent health, conntrack utilisation and IP pool usage.

  • Adopt flow-visibility tooling if your CNI offers it, so you can see allowed and dropped flows instead of guessing.

  • Introduce NetworkPolicies gradually, test them in staging, and always include a DNS allowance in default-deny designs.

  • Plan Pod, Service and node CIDRs in advance so they never overlap with VPCs or on-premises networks, and document the plan.

  • Run synthetic checks that exercise key paths regularly, including cross-node, cross-namespace and external ingress paths.

  • Test CNI, kube-proxy and node upgrades in a non-production cluster first, and roll them out node by node.

10. Conclusion

Kubernetes network troubleshooting is not about memorising commands; it is about narrowing the search space. By defining the problem precisely, testing from the application outwards, and using each result to rule a layer in or out, you can turn an overwhelming "the network is broken" report into a specific, fixable fault. The Pod IP test separates the application and Pod network from the Service layer, the DNS test separates names from connectivity, the policy review separates intentional blocking from faults, and node-level captures settle what remains.

Keep the layered model close, build the habit of testing one layer at a time, and the next network incident will be a short, methodical exercise rather than a long night.