Kubernetes Service Discovery: How Pods Find Other Pods
Learn how Kubernetes Pods discover Services using DNS, selectors and EndpointSlices, including namespaces, headless Services, troubleshooting and practical labs.
Kubernetes Service Discovery: How Pods Find Other Pods
A practical guide to Kubernetes DNS, Services, EndpointSlices, namespaces, headless Services, troubleshooting and real-world application communication.
Introduction
In a Kubernetes application, Pods frequently need to communicate with other Pods. A frontend may call an API, an API may call a database, and one microservice may call another microservice. The challenge is that Pod IP addresses are not stable application identities. Pods can be recreated, replaced during deployments, or scheduled onto different Nodes.
Kubernetes solves this problem by providing Service discovery. Instead of requiring an application to track individual Pod IP addresses, applications can normally communicate with a Service by name. Kubernetes DNS provides the name-resolution mechanism, while the Service and its backend EndpointSlices provide the relationship between the stable Service identity and the current backend endpoints.
This article explains the complete path: application → DNS name → Service → EndpointSlice → backend Pod, and shows how to troubleshoot each layer.
1. What Is Kubernetes Service Discovery?
Service discovery is the mechanism by which an application finds a network endpoint for another application or service. In Kubernetes, the most common pattern is to create a Service in front of a set of Pods and allow client Pods to discover that Service through cluster DNS.
• The Service provides a stable logical endpoint.
• The Service selector identifies matching Pods when a selector is configured.
• The EndpointSlice API tracks the network endpoints backing the Service.
• Kubernetes DNS publishes DNS records for Services and Pods.
• A service-proxy implementation routes Service traffic toward backend endpoints.
Kubernetes documentation describes the Service API as providing a stable, long-lived IP address or hostname for a service implemented by one or more backend Pods, even though the individual Pods can change over time.
2. The Core Problem: Pod IPs Can Change
Suppose an API application has three Pods:
A frontend application could technically connect directly to these IP addresses, but this is not a good service-discovery model. If a Pod is deleted and recreated, its replacement may receive a different IP address.
The application should therefore depend on a stable logical identity rather than individual Pod IPs.
3. The Kubernetes Service Discovery Architecture
The important distinction is that DNS discovery and traffic routing are related but are not the same operation. DNS answers the question, “What network identity should I use?” The Service dataplane then handles traffic sent to the Service endpoint.
4. How a Pod Finds a Service
Consider a Service named `orders` in the `production` namespace. A Pod in the same namespace can normally use the short name:
A client can also use the namespace-qualified name:
The fully qualified Service DNS form is:
The cluster domain is configurable; `cluster.local` is the common default rather than a value that every cluster must use.
Kubernetes documentation states that a normal Service receives an A and/or AAAA DNS record that resolves to the Service's cluster IP, while a headless Service resolves to the set of selected Pod IPs.
5. DNS Search Domains Make Short Names Practical
Kubernetes configures Pod DNS settings so applications can commonly resolve Services using short names. A client Pod's DNS search list normally includes its own namespace and the cluster's default domain.
You may see search entries similar to:
The exact values depend on the cluster configuration. The important operational point is that a query such as `orders` is interpreted using the Pod's DNS search configuration.
6. Namespace Matters
A Service name without a namespace is resolved relative to the namespace of the client Pod. This is one of the most common causes of DNS confusion.
For example, assume:
From the `frontend` namespace, this query looks for `orders` in the `frontend` namespace:
To reach the Service in `backend`, specify the namespace:
Or use the fully qualified name:
7. What Happens When You Call a Service Name?
Suppose an application executes:
1. The application asks the Pod's configured DNS resolver to resolve `orders`.
2. Kubernetes cluster DNS processes the Service DNS query.
3. For a normal Service, the DNS response points to the Service's cluster IP.
4. The client sends traffic toward that Service IP and port.
5. The Service dataplane selects an appropriate backend endpoint.
6. Traffic is delivered to one of the Pods represented by the Service's EndpointSlices.
The exact packet-processing path depends on the cluster's networking implementation. Kubernetes provides kube-proxy as a default Service proxy implementation, while some Pod network implementations provide their own Service proxy integrated with their networking stack.
8. Services Use Selectors to Find Backend Pods
A typical Service selects Pods using labels.
The backend Pods might have:
The Service selector `app: orders` matches the Pod label. The control plane then manages EndpointSlices representing the matching network endpoints.
9. EndpointSlices: The Backend Map
EndpointSlices are a critical part of modern Kubernetes Service networking. They represent sets of network endpoints associated with Services and allow Kubernetes to scale Service backend information efficiently.
For a Service with a selector, the control plane automatically creates and manages EndpointSlices containing references to Pods matching the selector. EndpointSlices can include endpoint conditions such as `ready`, `serving`, and `terminating`.
Kubernetes documentation identifies EndpointSlices as the source of truth for kube-proxy when it determines how to route internal Service traffic.
10. Service Discovery Is Not the Same as Pod Discovery
A common misconception is that every application discovers another application by directly discovering Pod IP addresses. The normal Kubernetes microservice pattern is different.
For most application-to-application communication, use a Service name rather than relying on individual Pod IPs.
11. Normal Service vs Headless Service
A normal Service has a cluster IP. Its DNS record resolves to the Service's virtual IP.
A headless Service sets `clusterIP: None`.
For a headless Service, DNS returns the IP addresses of the selected Pods instead of a single cluster IP. This is useful when the client needs to know individual backend endpoints, including common patterns used with StatefulSets.
Headless Service discovery therefore changes the discovery model: the client receives a set of endpoint addresses instead of one virtual Service IP.
12. Why StatefulSets Commonly Use Headless Services
StatefulSets provide stable Pod identities. A headless Service can provide DNS records associated with those stable identities, making it possible for applications to address individual StatefulSet Pods.
The exact DNS names depend on the StatefulSet Pod hostname, the headless Service name, namespace and cluster domain. This pattern is useful for distributed systems where individual members need stable identities rather than anonymous load-balanced access.
13. Service Ports vs Container Ports
Service discovery does not remove the need to configure ports correctly.
A client can therefore call the Service on port 80 while the application listens on port 8080 in the backend Pod.
14. Readiness and Service Discovery
Service endpoint availability is closely related to Pod readiness. EndpointSlice conditions contain readiness and serving information for endpoints. A Pod that is not ready should generally not receive normal Service traffic.
This is why readiness probes are important in production applications. A Pod may be running while its application is still initializing or unable to serve requests.
A readiness failure is different from a liveness failure: readiness controls whether the workload should receive traffic, while liveness can trigger a container restart when the liveness probe repeatedly fails.
15. DNS Resolution and Traffic Routing Are Different Layers
This separation is extremely useful during troubleshooting. If DNS resolution fails, investigate DNS. If DNS resolves but the Service has no endpoints, investigate selectors and endpoint conditions. If endpoints exist but traffic fails, investigate the application, network policy, service dataplane and surrounding network path.
16. Debugging Service Discovery
Step 1: Verify the Service
Step 2: Verify the Service selector
Step 3: Check EndpointSlices
If there are no expected endpoints, compare the Service selector with the Pod labels. Kubernetes' Service debugging documentation specifically identifies selector mismatch as a common cause of an empty endpoint list.
Step 4: Test DNS from inside the cluster
Step 5: Test the Service
Step 6: Test the Pod directly
If appropriate for troubleshooting, test one backend Pod directly. This helps determine whether the problem is the application itself or the Service path.
17. Troubleshooting DNS Failures
If a Service name cannot be resolved, check the DNS layer before changing the Service.
• Confirm the client Pod is using the expected DNS configuration.
• Confirm the Service exists in the namespace you are querying.
• Use a namespace-qualified name when the Service is in another namespace.
• Check the cluster DNS deployment and its logs.
• Verify that the DNS service can access the Service and EndpointSlice resources.
In clusters using CoreDNS, the DNS deployment must have sufficient permissions to list and watch relevant resources such as Services and EndpointSlices. Kubernetes provides a dedicated DNS troubleshooting guide for this scenario.
18. Troubleshooting an Empty EndpointSlice
Suppose:
shows no expected endpoints.
1. Inspect the Service selector.
2. Inspect the labels on the Pods.
3. Confirm the Pods are in the same namespace as the Service.
4. Check Pod readiness and termination state.
5. Confirm the Service port and targetPort are correct.
19. Cross-Namespace Service Discovery
Namespaces provide logical boundaries for Kubernetes resources. Service DNS names include the namespace so that applications can intentionally communicate across namespaces.
This is particularly useful in environments where teams or environments are separated into namespaces, for example `frontend`, `backend`, `staging` and `production`.
20. Service Discovery in a Microservices Architecture
Consider a typical application:
The frontend does not need to know the IP addresses of API Pods. It can call `api`. The API does not need to know the IP addresses of order Pods. It can call `orders`. Kubernetes DNS and Services provide stable discovery while the backend Pods can change.
21. What Happens During a Deployment?
During a Deployment rollout, old Pods can be replaced by new Pods. Their IP addresses can change. A Service remains the stable logical endpoint, while EndpointSlices are updated as backend endpoints change.
1. Deployment creates or updates Pods.
2. New Pods become ready according to their readiness behavior.
3. EndpointSlice information changes as endpoint state changes.
4. Service traffic is routed using the current backend endpoint set.
5. Old Pods can be removed without requiring clients to change the Service name.
This is one of the main reasons Service-based discovery is more resilient than hard-coding Pod IP addresses.
22. Headless Service and Client-Side Load Balancing
With a headless Service, DNS can return multiple backend Pod IPs. The application becomes responsible for consuming those addresses appropriately. Some distributed applications use this pattern when they need awareness of individual members rather than a single virtual Service endpoint.
Do not choose a headless Service merely because it sounds faster. It changes the discovery and routing model and should be selected because the application requires endpoint-level awareness.
23. Common Mistakes
Mistake 1: Using Pod IPs in application configuration
Pod IPs are not intended to be stable application identities. Use a Service name for normal application communication.
Mistake 2: Forgetting the namespace
A short Service name resolves within the client's namespace. Use `service.namespace` or the full DNS name for cross-namespace communication.
Mistake 3: Selector mismatch
A Service can exist even when its selector matches no Pods. Always inspect labels and EndpointSlices.
Mistake 4: Confusing DNS with routing
Successful DNS resolution does not guarantee that the backend application is reachable or healthy.
Mistake 5: Assuming every Service returns Pod IPs
Normal Services resolve to the Service cluster IP. Headless Services resolve to the selected Pod IPs.
Mistake 6: Testing from the wrong namespace
A DNS query such as `orders` can fail simply because the client is in a different namespace.
24. Security Considerations
Service discovery makes communication easier, but it is not a security authorization mechanism. A Service name being resolvable does not by itself mean that an application should be allowed to communicate with another application.
• Use NetworkPolicy where traffic restrictions are required and supported by the cluster network implementation.
• Avoid exposing internal Services externally without a business or operational requirement.
• Use TLS or application-level authentication where sensitive traffic requires it.
• Treat namespace separation as organizational and resource isolation, not as a complete network security boundary.
• Review DNS and Service access as part of the application's overall security model.
25. Hands-On Lab: Discover a Service from Another Pod
Create a Deployment and a ClusterIP Service:
Create a temporary client Pod:
From the client Pod, resolve the Service:
Then test the Service:
Finally, inspect the backend endpoints:
Now delete one backend Pod and observe the endpoint set again. The Service name remains unchanged while the backend Pod set changes.
26. Practical Architecture Example
Imagine a production platform with a public frontend, internal API services and a database.
The important design principle is that applications communicate using stable Service identities. The underlying Pod IPs can change without forcing every client configuration to change.
27. Interview Questions
What problem does Kubernetes Service discovery solve?
Answer: It allows workloads to find stable application endpoints even though backend Pods can be replaced and their IP addresses can change.
How does a Pod normally find a Service?
Answer: Through Kubernetes DNS using the Service name.
What does a normal Service DNS record resolve to?
Answer: The Service's cluster IP, using an A and/or AAAA record depending on the Service IP families.
What does a headless Service DNS record resolve to?
Answer: The IP addresses of the selected backend Pods rather than a Service cluster IP.
Why is namespace important in Service discovery?
Answer: A short Service name is resolved in the client's namespace; a different namespace must be specified for cross-namespace access.
What is EndpointSlice?
Answer: A Kubernetes API resource representing sets of network endpoints associated with a Service.
What is a common cause of a Service with no backend endpoints?
Answer: A Service selector does not match the labels on the intended Pods.
Does DNS perform the actual application load balancing?
Answer: DNS discovery and Service traffic routing are separate layers. Normal Service DNS returns the Service cluster IP, after which the Service dataplane routes traffic toward backend endpoints.
Why are readiness probes important for Service traffic?
Answer: Readiness state influences whether an endpoint should receive normal Service traffic.
When would you use a headless Service?
Answer: When clients need endpoint-level discovery, such as certain StatefulSet or distributed-system patterns.
28. Production Checklist
• Use Service names instead of hard-coded Pod IP addresses for normal application communication.
• Confirm the client and Service namespaces when debugging DNS.
• Verify Service selectors match intended Pod labels.
• Inspect EndpointSlices when a Service does not receive traffic.
• Check Pod readiness before assuming the backend is available.
• Verify Service port and targetPort values.
• Test DNS resolution from a real client Pod.
• Separate DNS troubleshooting from Service routing and application troubleshooting.
• Use headless Services only when endpoint-level discovery is actually required.
• Use NetworkPolicy and application authentication/TLS for security requirements.
29. Summary
Kubernetes Service discovery allows applications to communicate without depending on unstable Pod IP addresses. The normal pattern is simple: create a Service, select the appropriate backend Pods, and allow client workloads to discover the Service through Kubernetes DNS.
A normal Service gives clients a stable Service identity and cluster IP. EndpointSlices represent the current backend endpoints. The Service dataplane routes traffic toward those endpoints. A headless Service changes the model by allowing DNS to return the selected Pod addresses directly.
For production troubleshooting, remember the sequence: check the DNS name and namespace → inspect the Service → verify selectors → inspect EndpointSlices → check Pod readiness → test the application and network path. Understanding these layers turns Service discovery from a black box into a predictable Kubernetes networking workflow.
30. Related Kubernetes Topics
• Kubernetes Service Explained: ClusterIP, NodePort and LoadBalancer
• Kubernetes Pod Networking Explained
• Kubernetes DNS and CoreDNS Explained
• Kubernetes Ingress Explained
• Kubernetes Gateway API Explained
• Kubernetes NetworkPolicy Explained
• Kubernetes StatefulSet Explained
• Kubernetes Container Probes: Liveness, Readiness and Startup Probes
Official Kubernetes References
DNS for Services and Pods — https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
EndpointSlices — https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
Services, Load Balancing, and Networking — https://kubernetes.io/docs/concepts/services-networking/
Debugging DNS Resolution — https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
Debug Services — https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/