Kubernetes DNS Troubleshooting: Debugging Service Name Resolution
Debugging Service Name Resolution
Kubernetes DNS Troubleshooting: Debugging Service Name Resolution
bitcodematrix.com | Kubernetes Series
1. Introduction
"It works with the IP address but not with the name" is one of the most common sentences in Kubernetes support threads. Applications in a cluster find each other through DNS names such as payments.shop.svc.cluster.local, so when name resolution breaks, many unrelated applications appear to fail at once, with errors like "no such host", "temporary failure in name resolution" or plain timeouts.
This article gives you a mental model of how DNS works inside a cluster and a repeatable workflow for finding the fault. It covers the names Kubernetes creates, the Pod resolver configuration, CoreDNS, the common root causes with fixes, and a quick-reference table of commands.
2. How DNS Works in Kubernetes
Four pieces cooperate to resolve a name:
CoreDNS is the cluster DNS server. It runs as a Deployment in the kube-system namespace and answers queries for cluster names by reading Services and EndpointSlices from the API server. It forwards all other queries to an upstream resolver.
The kube-dns Service is the stable ClusterIP in front of the CoreDNS Pods. For historical reasons it is still named kube-dns, and CoreDNS Pods carry the label k8s-app=kube-dns.
The kubelet writes each Pod's /etc/resolv.conf, pointing it at the kube-dns ClusterIP and adding the search domains.
The network path (CNI plugin, kube-proxy and any NetworkPolicy) carries the query from the Pod to CoreDNS on port 53, over UDP and TCP.
A lookup therefore travels like this: the application calls its resolver, the resolver reads /etc/resolv.conf, sends the query to the kube-dns ClusterIP, kube-proxy translates that address to one of the CoreDNS Pods, CoreDNS answers from cluster data or forwards the query upstream, and the answer returns along the same path. A failure at any of these stages looks like a DNS failure, which is why a structured approach matters.
3. The DNS Names Kubernetes Creates
The suffix cluster.local is the default cluster domain, but it is configurable. If lookups for cluster.local names fail on an unusual cluster, check which domain the kubelet and CoreDNS were configured with. ExternalName Services return a CNAME record to the external name instead of an IP address.
4. The Pod Resolver: resolv.conf, Search Domains and ndots
Every Pod gets a generated resolver configuration. For a Pod in the namespace shop, it typically looks like this:
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
The nameserver is the ClusterIP of the kube-dns Service (the exact address depends on your Service CIDR). The search line and the ndots option explain why short names work.
4.1 Search Domains
From a Pod in the shop namespace, the name payments is tried as payments.shop.svc.cluster.local first, which finds the Service in the same namespace. To reach a Service in another namespace, use at least payments.billing, which expands to payments.billing.svc.cluster.local through the second search domain.
4.2 The ndots Option
The ndots value is the number of dots a name must contain before the resolver tries it as an absolute name first. With ndots:5, any name with fewer than five dots is tried with each search domain appended before it is tried as written. A lookup for api.example.com, which has two dots, can therefore generate several failing queries (api.example.com.shop.svc.cluster.local, then api.example.com.svc.cluster.local, then api.example.com.cluster.local) before the real one succeeds. This adds latency and load, and it is a frequent hidden cause of slow external calls.
4.3 dnsPolicy
The dnsPolicy field in the Pod spec controls how resolv.conf is built.
A Pod that uses hostNetwork: true with the default ClusterFirst policy falls back to the node's resolver, so it cannot resolve Service names. Setting ClusterFirstWithHostNet fixes this.
5. CoreDNS and the Corefile
CoreDNS is configured through a file called the Corefile, stored in the ConfigMap named coredns in the kube-system namespace. A typical default, as installed by kubeadm, looks like this:
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
Managed Kubernetes services may use a slightly different Corefile or manage it for you, so look at your own ConfigMap rather than assuming this layout.
6. Build a Debugging Toolbox
Production images often contain no DNS tools, so start a dedicated debug Pod in the same namespace as the failing application. The Kubernetes project's own DNS debugging guide uses a test image that includes nslookup:
kubectl run dnsutils -n <namespace> \
--image=registry.k8s.io/e2e-test-images/agnhost:2.39 \
--restart=Never --command -- sleep 3600
Check the Kubernetes documentation for the current image tag. Another popular choice is the netshoot image, which bundles dig, curl, tcpdump and many other tools. For a running Pod whose image has no tools, an ephemeral debug container lets you attach a toolbox without changing the Pod:
kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name>
Always test from the same namespace and, if possible, the same node as the failing Pod, because namespace-specific search domains, NetworkPolicies and node-specific problems can all change the result.
7. A Systematic Troubleshooting Workflow
Work from the application outwards. Each step eliminates one layer.
7.1 Step 1: Confirm the Pod Resolver Configuration
kubectl exec -it dnsutils -- cat /etc/resolv.conf
Check that the nameserver matches the kube-dns Service ClusterIP, that the search line lists your namespace, and that the dnsPolicy of the failing Pod is what you expect. Compare with the failing Pod itself if it has a shell.
7.2 Step 2: Test a Name That Must Work
kubectl exec -it dnsutils -- nslookup kubernetes.default
The kubernetes Service in the default namespace exists in every cluster. If this succeeds, cluster DNS works in general and the problem is specific to a name, a namespace or a Pod. If it fails, continue with the next steps.
7.3 Step 3: Test the Exact Name and Try the Fully Qualified Form
kubectl exec -it dnsutils -- nslookup payments
kubectl exec -it dnsutils -- nslookup payments.billing.svc.cluster.local
If the fully qualified name works but the short name does not, the cause is search-domain expansion: the Pod is in a different namespace or the short name lacks the namespace part. If neither works, verify that the Service exists and has endpoints:
kubectl get svc payments -n billing
kubectl get endpointslices -n billing -l kubernetes.io/service-name=payments
7.4 Step 4: Check CoreDNS Health
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
kubectl describe pods -n kube-system -l k8s-app=kube-dns
Look for Pods that are not Ready, restarts, OOMKilled terminations, and log lines about the API server being unreachable, upstream timeouts or loop detection.
7.5 Step 5: Check the kube-dns Service and Its Endpoints
kubectl get svc kube-dns -n kube-system
kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns
The Service must exist and must list the CoreDNS Pod IPs as ready endpoints. If it has no endpoints, CoreDNS is not ready. If it has endpoints but queries still time out, the problem lies in the network path.
7.6 Step 6: Query CoreDNS Directly, Bypassing the Service
Query a CoreDNS Pod IP directly and then the Service IP, to separate a Pod-network problem from a kube-proxy problem:
kubectl exec -it dnsutils -- nslookup kubernetes.default <coredns-pod-ip>
kubectl exec -it dnsutils -- nslookup kubernetes.default <kube-dns-cluster-ip>
If the direct Pod query works but the Service IP fails, suspect kube-proxy or its rules on that node. If both fail, suspect the CNI, a NetworkPolicy or a firewall between the Pod and CoreDNS.
7.7 Step 7: Test External Resolution
kubectl exec -it dnsutils -- nslookup example.com
If cluster names work but external names fail, the fault is in CoreDNS's upstream path: the forward plugin, the node's resolver or a firewall blocking egress DNS from the CoreDNS Pods.
8. Reading the Symptoms
9. Common Root Causes and Fixes
9.1 NetworkPolicy Blocks DNS
This is the most common cause in hardened clusters. A default-deny egress policy blocks DNS along with everything else, and applications time out on every lookup. Allow DNS traffic to CoreDNS explicitly, over both UDP and TCP:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: shop
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
TCP is needed because DNS falls back to TCP for large responses. If you use NodeLocal DNSCache, the policy must also allow traffic to its node-local address.
9.2 CoreDNS Pods Not Running or Not Ready
If CoreDNS is Pending, check that a CNI plugin is installed, because CoreDNS Pods cannot start without Pod networking. If it is CrashLoopBackOff, read the logs: the cause is usually a Corefile syntax error after an edit, a loop, or no connectivity to the API server. If it is OOMKilled, raise the memory limit and add replicas. Revert recent changes to the coredns ConfigMap first when the problem started right after an edit.
9.3 Forwarding Loop Detected
CoreDNS forwards external queries to the resolvers in the node's /etc/resolv.conf. On nodes where that file points at a local stub resolver such as 127.0.0.53 (systemd-resolved), CoreDNS may forward queries to itself and the loop plugin stops it with a "Loop detected" message. The fix is to make the kubelet pass the real upstream resolver file, usually /run/systemd/resolve/resolv.conf, through its resolvConf setting, or to set explicit upstream servers in the forward plugin of the Corefile. kubeadm normally configures this for you on systemd-resolved nodes.
9.4 Slow Lookups Caused by ndots
If external calls are slow but work, reduce the extra search-domain queries. Either use a fully qualified name with a trailing dot in application configuration (api.example.com.), or lower ndots for the workload:
spec:
dnsConfig:
options:
- name: ndots
value: "2"
Lowering ndots means short names with two or more dots are tried as absolute names first, so make sure internal names such as payments.billing still resolve as intended, or use their full form.
9.5 Wrong dnsPolicy
Symptoms are cluster names failing from one specific Pod while the debug Pod works. Check the Pod's dnsPolicy and hostNetwork settings, and use ClusterFirstWithHostNet for host-networked Pods that need cluster DNS.
9.6 Custom Domains and Private Upstreams
Names in a corporate domain often live on an internal DNS server that the default upstream cannot reach. Add a dedicated server block to the Corefile that forwards just that domain:
corp.example.com:53 {
errors
cache 30
forward . 10.0.0.2
}
The reload plugin applies the change after a short delay, so watch the CoreDNS logs to confirm that the new configuration was loaded without errors.
9.7 Load and Scale
Every Pod lookup, including the extra ndots queries, lands on CoreDNS. Large clusters need more replicas, which is often handled with the cluster-proportional DNS autoscaler. NodeLocal DNSCache runs a caching agent on each node, answers repeated queries locally, and reduces both latency and connection-tracking load. Consider it when you see intermittent delays or high query volume.
10. Turning On Logging and Metrics
To see every query CoreDNS handles, add the log plugin to the Corefile temporarily:
kubectl edit configmap coredns -n kube-system
Add the line log inside the server block next to errors, wait for the reload, then follow the logs. Each line shows the client, query name, type and response code. Remove the plugin when you finish, because logging every query is expensive on a busy cluster and may expose sensitive names.
For continuous visibility, scrape the Prometheus metrics on port 9153. Useful series include coredns_dns_requests_total, coredns_dns_responses_total (labelled by response code, so you can alert on rising SERVFAIL or NXDOMAIN rates) and coredns_dns_request_duration_seconds for latency.
11. Quick Reference
12. Best Practices
Always allow DNS egress in your default-deny NetworkPolicies, and test the policy with a lookup before rolling it out widely.
Run at least two CoreDNS replicas, spread across nodes, and give them appropriate memory limits and a PodDisruptionBudget.
Use fully qualified names (with a trailing dot) for external services in latency-sensitive applications, or tune ndots per workload.
Use namespace-qualified names such as service.namespace for cross-namespace calls, and avoid relying on search-domain accidents.
Consider NodeLocal DNSCache and the DNS autoscaler for large or busy clusters.
Monitor CoreDNS error rates and latency, and alert on rising SERVFAIL responses before users notice.
Keep a debug image available, and document the standard test commands so on-call engineers can follow the same workflow.
Version-control changes to the coredns ConfigMap, and test Corefile edits in a non-production cluster first.
13. Conclusion
DNS problems in Kubernetes feel mysterious only because several layers hide behind one lookup. Once you separate them, the diagnosis is mechanical: check the Pod's resolver file, test a name that must always work, confirm that CoreDNS and its Service endpoints are healthy, query CoreDNS directly to isolate network faults, and then test external resolution. The symptom, whether NXDOMAIN, SERVFAIL, a timeout or a five-second delay, usually points straight to the right layer.
Follow the workflow in this article, keep NetworkPolicies DNS-aware, and tune ndots and caching for your workloads, and most name-resolution incidents become short, routine fixes.