kubernetes

Kubernetes DNS Troubleshooting: Debugging Service Name Resolution

Debugging Service Name Resolution

By Shubhankar Tripathi • • 5 min read

Kubernetes DNS Troubleshooting: Debugging Service Name Resolution

bitcodematrix.com | Kubernetes Series

1. Introduction

"It works with the IP address but not with the name" is one of the most common sentences in Kubernetes support threads. Applications in a cluster find each other through DNS names such as payments.shop.svc.cluster.local, so when name resolution breaks, many unrelated applications appear to fail at once, with errors like "no such host", "temporary failure in name resolution" or plain timeouts.

This article gives you a mental model of how DNS works inside a cluster and a repeatable workflow for finding the fault. It covers the names Kubernetes creates, the Pod resolver configuration, CoreDNS, the common root causes with fixes, and a quick-reference table of commands.

2. How DNS Works in Kubernetes

Four pieces cooperate to resolve a name:

  • CoreDNS is the cluster DNS server. It runs as a Deployment in the kube-system namespace and answers queries for cluster names by reading Services and EndpointSlices from the API server. It forwards all other queries to an upstream resolver.

  • The kube-dns Service is the stable ClusterIP in front of the CoreDNS Pods. For historical reasons it is still named kube-dns, and CoreDNS Pods carry the label k8s-app=kube-dns.

  • The kubelet writes each Pod's /etc/resolv.conf, pointing it at the kube-dns ClusterIP and adding the search domains.

  • The network path (CNI plugin, kube-proxy and any NetworkPolicy) carries the query from the Pod to CoreDNS on port 53, over UDP and TCP.

A lookup therefore travels like this: the application calls its resolver, the resolver reads /etc/resolv.conf, sends the query to the kube-dns ClusterIP, kube-proxy translates that address to one of the CoreDNS Pods, CoreDNS answers from cluster data or forwards the query upstream, and the answer returns along the same path. A failure at any of these stages looks like a DNS failure, which is why a structured approach matters.

3. The DNS Names Kubernetes Creates

Object

DNS name

Answer

Service (ClusterIP)

<service>.<namespace>.svc.cluster.local

The ClusterIP

Headless Service

<service>.<namespace>.svc.cluster.local

The IPs of all ready Pods

StatefulSet Pod behind a headless Service

<pod>.<service>.<namespace>.svc.cluster.local

That Pod's IP

Named Service port (SRV record)

_<port-name>._<protocol>.<service>.<namespace>.svc.cluster.local

Port number and target host

Pod

<ip-with-dashes>.<namespace>.pod.cluster.local

The Pod IP

 

The suffix cluster.local is the default cluster domain, but it is configurable. If lookups for cluster.local names fail on an unusual cluster, check which domain the kubelet and CoreDNS were configured with. ExternalName Services return a CNAME record to the external name instead of an IP address.

4. The Pod Resolver: resolv.conf, Search Domains and ndots

Every Pod gets a generated resolver configuration. For a Pod in the namespace shop, it typically looks like this:

search shop.svc.cluster.local svc.cluster.local cluster.local

nameserver 10.96.0.10

options ndots:5

The nameserver is the ClusterIP of the kube-dns Service (the exact address depends on your Service CIDR). The search line and the ndots option explain why short names work.

4.1 Search Domains

From a Pod in the shop namespace, the name payments is tried as payments.shop.svc.cluster.local first, which finds the Service in the same namespace. To reach a Service in another namespace, use at least payments.billing, which expands to payments.billing.svc.cluster.local through the second search domain.

4.2 The ndots Option

The ndots value is the number of dots a name must contain before the resolver tries it as an absolute name first. With ndots:5, any name with fewer than five dots is tried with each search domain appended before it is tried as written. A lookup for api.example.com, which has two dots, can therefore generate several failing queries (api.example.com.shop.svc.cluster.local, then api.example.com.svc.cluster.local, then api.example.com.cluster.local) before the real one succeeds. This adds latency and load, and it is a frequent hidden cause of slow external calls.

4.3 dnsPolicy

The dnsPolicy field in the Pod spec controls how resolv.conf is built.

dnsPolicy

Behaviour

ClusterFirst (default)

Uses the cluster DNS service; names outside the cluster domain are forwarded to the upstream resolver

Default

Inherits the node's resolver configuration; cluster Service names will not resolve. Despite its name, this is not the default policy

ClusterFirstWithHostNet

Use for Pods with hostNetwork: true that still need cluster DNS

None

Ignores cluster defaults; everything must be supplied through dnsConfig

 

A Pod that uses hostNetwork: true with the default ClusterFirst policy falls back to the node's resolver, so it cannot resolve Service names. Setting ClusterFirstWithHostNet fixes this.

5. CoreDNS and the Corefile

CoreDNS is configured through a file called the Corefile, stored in the ConfigMap named coredns in the kube-system namespace. A typical default, as installed by kubeadm, looks like this:

.:53 {

    errors

    health {

        lameduck 5s

    }

    ready

    kubernetes cluster.local in-addr.arpa ip6.arpa {

        pods insecure

        fallthrough in-addr.arpa ip6.arpa

        ttl 30

    }

    prometheus :9153

    forward . /etc/resolv.conf {

        max_concurrent 1000

    }

    cache 30

    loop

    reload

    loadbalance

}

Plugin

Purpose

errors

Logs errors to standard output

health and ready

Provide the liveness and readiness endpoints used by Kubernetes probes

kubernetes

Answers cluster.local queries from Services and EndpointSlices read from the API server

prometheus

Exposes metrics on port 9153

forward

Sends everything else to upstream resolvers, here those listed in the node's /etc/resolv.conf

cache

Caches answers for up to 30 seconds

loop

Detects forwarding loops and stops CoreDNS if one is found

reload

Reloads the Corefile automatically after the ConfigMap changes

loadbalance

Randomises the order of address records in answers

 

Managed Kubernetes services may use a slightly different Corefile or manage it for you, so look at your own ConfigMap rather than assuming this layout.

6. Build a Debugging Toolbox

Production images often contain no DNS tools, so start a dedicated debug Pod in the same namespace as the failing application. The Kubernetes project's own DNS debugging guide uses a test image that includes nslookup:

kubectl run dnsutils -n <namespace> \

  --image=registry.k8s.io/e2e-test-images/agnhost:2.39 \

  --restart=Never --command -- sleep 3600

Check the Kubernetes documentation for the current image tag. Another popular choice is the netshoot image, which bundles dig, curl, tcpdump and many other tools. For a running Pod whose image has no tools, an ephemeral debug container lets you attach a toolbox without changing the Pod:

kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name>

Always test from the same namespace and, if possible, the same node as the failing Pod, because namespace-specific search domains, NetworkPolicies and node-specific problems can all change the result.

7. A Systematic Troubleshooting Workflow

Work from the application outwards. Each step eliminates one layer.

7.1 Step 1: Confirm the Pod Resolver Configuration

kubectl exec -it dnsutils -- cat /etc/resolv.conf

Check that the nameserver matches the kube-dns Service ClusterIP, that the search line lists your namespace, and that the dnsPolicy of the failing Pod is what you expect. Compare with the failing Pod itself if it has a shell.

7.2 Step 2: Test a Name That Must Work

kubectl exec -it dnsutils -- nslookup kubernetes.default

The kubernetes Service in the default namespace exists in every cluster. If this succeeds, cluster DNS works in general and the problem is specific to a name, a namespace or a Pod. If it fails, continue with the next steps.

7.3 Step 3: Test the Exact Name and Try the Fully Qualified Form

kubectl exec -it dnsutils -- nslookup payments

kubectl exec -it dnsutils -- nslookup payments.billing.svc.cluster.local

If the fully qualified name works but the short name does not, the cause is search-domain expansion: the Pod is in a different namespace or the short name lacks the namespace part. If neither works, verify that the Service exists and has endpoints:

kubectl get svc payments -n billing

kubectl get endpointslices -n billing -l kubernetes.io/service-name=payments

7.4 Step 4: Check CoreDNS Health

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

kubectl describe pods -n kube-system -l k8s-app=kube-dns

Look for Pods that are not Ready, restarts, OOMKilled terminations, and log lines about the API server being unreachable, upstream timeouts or loop detection.

7.5 Step 5: Check the kube-dns Service and Its Endpoints

kubectl get svc kube-dns -n kube-system

kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns

The Service must exist and must list the CoreDNS Pod IPs as ready endpoints. If it has no endpoints, CoreDNS is not ready. If it has endpoints but queries still time out, the problem lies in the network path.

7.6 Step 6: Query CoreDNS Directly, Bypassing the Service

Query a CoreDNS Pod IP directly and then the Service IP, to separate a Pod-network problem from a kube-proxy problem:

kubectl exec -it dnsutils -- nslookup kubernetes.default <coredns-pod-ip>

kubectl exec -it dnsutils -- nslookup kubernetes.default <kube-dns-cluster-ip>

If the direct Pod query works but the Service IP fails, suspect kube-proxy or its rules on that node. If both fail, suspect the CNI, a NetworkPolicy or a firewall between the Pod and CoreDNS.

7.7 Step 7: Test External Resolution

kubectl exec -it dnsutils -- nslookup example.com

If cluster names work but external names fail, the fault is in CoreDNS's upstream path: the forward plugin, the node's resolver or a firewall blocking egress DNS from the CoreDNS Pods.

8. Reading the Symptoms

What you see

What it usually means

NXDOMAIN (name does not exist)

CoreDNS answered and the name is not there: wrong Service name or namespace, Service not created, a typo, or a search-domain mismatch.

SERVFAIL

CoreDNS could not produce an answer: the upstream resolver failed, or CoreDNS cannot reach the API server.

"connection timed out; no servers could be reached"

The query never reached a working DNS server: CoreDNS down, no endpoints, NetworkPolicy, kube-proxy, CNI or firewall problem.

Lookups succeed, but only after about 5 seconds

A first query was lost and retried. Look for packet loss on UDP, conntrack problems, overloaded CoreDNS, or unanswered AAAA queries.

Names resolve but connections fail

DNS is healthy. The fault is in the Service, endpoints or the network after resolution; see the kube-proxy and CNI layers.

Resolves to an unexpected IP

Stale cache, a different Service with a similar name, or a headless Service returning Pod IPs.

Works from some Pods or nodes only

A node-specific problem: kube-proxy, CNI or firewall on those nodes, or a Pod with a different dnsPolicy.

 

9. Common Root Causes and Fixes

9.1 NetworkPolicy Blocks DNS

This is the most common cause in hardened clusters. A default-deny egress policy blocks DNS along with everything else, and applications time out on every lookup. Allow DNS traffic to CoreDNS explicitly, over both UDP and TCP:

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: allow-dns-egress

  namespace: shop

spec:

  podSelector: {}

  policyTypes:

  - Egress

  egress:

  - to:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: kube-system

      podSelector:

        matchLabels:

          k8s-app: kube-dns

    ports:

    - protocol: UDP

      port: 53

    - protocol: TCP

      port: 53

TCP is needed because DNS falls back to TCP for large responses. If you use NodeLocal DNSCache, the policy must also allow traffic to its node-local address.

9.2 CoreDNS Pods Not Running or Not Ready

If CoreDNS is Pending, check that a CNI plugin is installed, because CoreDNS Pods cannot start without Pod networking. If it is CrashLoopBackOff, read the logs: the cause is usually a Corefile syntax error after an edit, a loop, or no connectivity to the API server. If it is OOMKilled, raise the memory limit and add replicas. Revert recent changes to the coredns ConfigMap first when the problem started right after an edit.

9.3 Forwarding Loop Detected

CoreDNS forwards external queries to the resolvers in the node's /etc/resolv.conf. On nodes where that file points at a local stub resolver such as 127.0.0.53 (systemd-resolved), CoreDNS may forward queries to itself and the loop plugin stops it with a "Loop detected" message. The fix is to make the kubelet pass the real upstream resolver file, usually /run/systemd/resolve/resolv.conf, through its resolvConf setting, or to set explicit upstream servers in the forward plugin of the Corefile. kubeadm normally configures this for you on systemd-resolved nodes.

9.4 Slow Lookups Caused by ndots

If external calls are slow but work, reduce the extra search-domain queries. Either use a fully qualified name with a trailing dot in application configuration (api.example.com.), or lower ndots for the workload:

spec:

  dnsConfig:

    options:

    - name: ndots

      value: "2"

Lowering ndots means short names with two or more dots are tried as absolute names first, so make sure internal names such as payments.billing still resolve as intended, or use their full form.

9.5 Wrong dnsPolicy

Symptoms are cluster names failing from one specific Pod while the debug Pod works. Check the Pod's dnsPolicy and hostNetwork settings, and use ClusterFirstWithHostNet for host-networked Pods that need cluster DNS.

9.6 Custom Domains and Private Upstreams

Names in a corporate domain often live on an internal DNS server that the default upstream cannot reach. Add a dedicated server block to the Corefile that forwards just that domain:

corp.example.com:53 {

    errors

    cache 30

    forward . 10.0.0.2

}

The reload plugin applies the change after a short delay, so watch the CoreDNS logs to confirm that the new configuration was loaded without errors.

9.7 Load and Scale

Every Pod lookup, including the extra ndots queries, lands on CoreDNS. Large clusters need more replicas, which is often handled with the cluster-proportional DNS autoscaler. NodeLocal DNSCache runs a caching agent on each node, answers repeated queries locally, and reduces both latency and connection-tracking load. Consider it when you see intermittent delays or high query volume.

10. Turning On Logging and Metrics

To see every query CoreDNS handles, add the log plugin to the Corefile temporarily:

kubectl edit configmap coredns -n kube-system

Add the line log inside the server block next to errors, wait for the reload, then follow the logs. Each line shows the client, query name, type and response code. Remove the plugin when you finish, because logging every query is expensive on a busy cluster and may expose sensitive names.

For continuous visibility, scrape the Prometheus metrics on port 9153. Useful series include coredns_dns_requests_total, coredns_dns_responses_total (labelled by response code, so you can alert on rising SERVFAIL or NXDOMAIN rates) and coredns_dns_request_duration_seconds for latency.

11. Quick Reference

Command

Purpose

kubectl exec <pod> -- cat /etc/resolv.conf

Inspect the Pod resolver configuration

nslookup kubernetes.default

Baseline test that cluster DNS works

kubectl get pods -n kube-system -l k8s-app=kube-dns

Check CoreDNS Pod health

kubectl logs -n kube-system -l k8s-app=kube-dns

Read CoreDNS errors

kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns

Confirm the kube-dns Service has ready endpoints

kubectl get configmap coredns -n kube-system -o yaml

View the Corefile

nslookup <name> <server-ip>

Query a specific DNS server, bypassing the Service

kubectl get networkpolicy -A

Find policies that may block DNS

 

12. Best Practices

  • Always allow DNS egress in your default-deny NetworkPolicies, and test the policy with a lookup before rolling it out widely.

  • Run at least two CoreDNS replicas, spread across nodes, and give them appropriate memory limits and a PodDisruptionBudget.

  • Use fully qualified names (with a trailing dot) for external services in latency-sensitive applications, or tune ndots per workload.

  • Use namespace-qualified names such as service.namespace for cross-namespace calls, and avoid relying on search-domain accidents.

  • Consider NodeLocal DNSCache and the DNS autoscaler for large or busy clusters.

  • Monitor CoreDNS error rates and latency, and alert on rising SERVFAIL responses before users notice.

  • Keep a debug image available, and document the standard test commands so on-call engineers can follow the same workflow.

  • Version-control changes to the coredns ConfigMap, and test Corefile edits in a non-production cluster first.

13. Conclusion

DNS problems in Kubernetes feel mysterious only because several layers hide behind one lookup. Once you separate them, the diagnosis is mechanical: check the Pod's resolver file, test a name that must always work, confirm that CoreDNS and its Service endpoints are healthy, query CoreDNS directly to isolate network faults, and then test external resolution. The symptom, whether NXDOMAIN, SERVFAIL, a timeout or a five-second delay, usually points straight to the right layer.

Follow the workflow in this article, keep NetworkPolicies DNS-aware, and tune ndots and caching for your workloads, and most name-resolution incidents become short, routine fixes.