Kubernetes Control Plane vs Worker Node: Architecture, Components and Communication
Kubernetes Control Plane vs Worker Node: Architecture, Components and Communication
Kubernetes Control Plane vs Worker Node: Architecture, Components and Communication
bitcodematrix.com | Kubernetes Series
1. Introduction
Every Kubernetes cluster is made of two kinds of machines with two very different jobs. The control plane decides what should happen: it stores the desired state, schedules work and reacts to change. The worker nodes do the work: they run your containers, wire up their networking and storage, and report back. Almost every operational question about Kubernetes, from "why is my Pod Pending?" to "what happens if a master node dies?", becomes easier once you can place each component on one side of this line and know how the two sides talk to each other.
This article explains the architecture of both sides, describes every major component and its role, shows how the components communicate and secure that communication, follows a request from kubectl to a running container, and examines failure scenarios, high availability, upgrades and troubleshooting. It builds on the earlier articles in this series on CNI, kube-proxy, DNS and storage.
2. The Big Picture
The diagram below shows the main components and the direction of the main connections. Keep one design rule in mind while reading it: the API server is the hub. Every other component, whether on the control plane or on a worker node, talks to the API server, and the API server is the only component that reads and writes etcd.
Figure 1: Control plane and worker node components, and how they connect.
3. Control Plane Components
3.1 kube-apiserver
The API server is the front door of the cluster. It exposes the Kubernetes REST API, usually on port 6443, and every kubectl command, controller, scheduler decision and kubelet report passes through it. For each request it performs a fixed sequence:
Authentication establishes who is calling, using client certificates, tokens or an external identity provider.
Authorization decides whether that identity may perform the action, normally with RBAC.
Admission control runs mutating and validating checks, which can modify the object (for example inject defaults) or reject it (for example violate a policy).
Validation and persistence check the object against its schema and write it to etcd.
The API server also provides the watch mechanism. Clients open a long-lived connection and receive a stream of changes, so that components react to events without polling. The API server is stateless, since all data lives in etcd, which is why you can run several instances behind a load balancer for availability and throughput.
3.2 etcd
etcd is a distributed, consistent key-value store and the single source of truth for the cluster: every object that you create, from Deployments to Secrets, is stored there. It uses the Raft consensus algorithm, so a majority of its members, the quorum, must agree on each write. This has practical consequences:
Use an odd number of members. An even number tolerates no more failures than the odd number below it.
etcd is very sensitive to disk latency and network latency, so give it fast storage and keep its members close together.
It has a storage quota, with a default backend size limit of 2 GiB, after which writes are refused until it is compacted and defragmented.
Back it up regularly and test restores. Losing etcd without a backup means losing the cluster's state.
It listens for clients on port 2379 and for peer traffic on port 2380. Only the API server should reach it.
3.3 kube-scheduler
The scheduler watches for Pods that have no node assigned, and picks the best node for each one. It works in two phases: filtering removes nodes that cannot run the Pod (not enough resources, taints without tolerations, unsatisfied affinity, volume constraints), and scoring ranks the remaining nodes. It then writes the result back to the API server as a binding. The scheduler does not start the Pod; it only decides where it will run.
3.4 kube-controller-manager
Kubernetes is driven by control loops: a controller observes the current state, compares it with the desired state, and acts to reduce the difference. The controller manager bundles many of these loops into one process. Examples include:
The Deployment and ReplicaSet controllers create and delete Pods to match the replica count and manage rollouts.
The Node controller notices when nodes stop reporting and marks them unhealthy.
The EndpointSlice controller keeps the list of Pod addresses behind each Service up to date.
The Job, namespace, service account and PersistentVolume controllers manage their own object types.
Controllers do not talk to each other directly. They communicate only by changing objects in the API server, which other controllers then observe. This loose coupling makes the system resilient and extensible.
3.5 cloud-controller-manager
When a cluster runs on a cloud provider, this optional component contains the provider-specific control loops: it creates cloud load balancers for Services of type LoadBalancer, configures routes, and removes Node objects when the cloud reports that a virtual machine has gone. Separating it from the core allows cloud providers to release changes independently. On-premises clusters usually have none.
3.6 Add-Ons
Some essential services are not control plane components in the strict sense, yet every cluster needs them. They run as ordinary workloads, usually in the kube-system namespace. The most important is cluster DNS (CoreDNS), together with the network plugin, a metrics provider and optionally an ingress controller or Gateway implementation.
4. Worker Node Components
4.1 kubelet
The kubelet is the agent that runs on every node, and it is the bridge between the cluster and the machine. Its responsibilities include:
Registering the node with the API server and keeping its status and capacity up to date.
Watching for Pods that the scheduler has assigned to its node, and making the local state match each Pod specification.
Driving the container runtime through the Container Runtime Interface (CRI) to pull images and start, stop and monitor containers.
Setting up volumes and networking for each Pod by calling the CSI node plugin and, through the runtime, the CNI plugin.
Running health checks (liveness, readiness and startup probes) and restarting containers according to the Pod's restart policy.
Reporting Pod and node status, and enforcing resource limits and eviction when the node is under pressure.
The kubelet exposes its own API on port 10250, which the API server uses for logs, exec and port-forwarding. It also runs static Pods, which are defined by manifest files in a local directory instead of in the API server. This is how tools such as kubeadm run the control plane components themselves.
4.2 Container Runtime
The container runtime does the real work of running containers. It implements the CRI so that the kubelet can use it, and common choices are containerd and CRI-O. Underneath, a low-level OCI runtime such as runc creates the isolated Linux processes with their namespaces and control groups. Since the removal of dockershim in Kubernetes 1.24, the kubelet no longer talks to Docker Engine directly, although images built with Docker run unchanged on CRI runtimes.
4.3 kube-proxy
kube-proxy runs on each node and turns Service objects into kernel rules, so that traffic to a Service's virtual IP is redirected to one of its backend Pods. It watches Services and EndpointSlices through the API server and programs iptables or nftables rules. Some network plugins replace it with their own eBPF implementation. A previous article in this series covers it in detail.
4.4 Network and Storage Plugins
A CNI plugin gives every Pod its network interface and IP address and connects nodes together, and a CSI node plugin attaches and mounts volumes for Pods on that node. Both run on every node, usually as DaemonSets, and both are driven by the kubelet and the container runtime.
5. How the Components Communicate
5.1 A Hub-and-Spoke Design
All communication revolves around the API server:
Controllers, the scheduler, the kubelets and kube-proxy are clients of the API server. They read objects and watch for changes, and they write back status or decisions.
The API server is the only component that talks to etcd.
Components never call each other directly to coordinate, with a few exceptions such as the API server calling kubelets and extension API servers.
This pattern is why Kubernetes behaves so predictably: the state of the cluster is just a set of objects, and every component is a client that reacts to them.
5.2 Node to Control Plane
The kubelet and kube-proxy connect outwards to the API server over HTTPS, normally on port 6443, usually through a load balancer or a stable endpoint address. They authenticate with client certificates or tokens. A new node typically uses a short-lived bootstrap credential to request its own certificate, which is then signed and rotated automatically, a process known as TLS bootstrapping.
Every node also maintains a heartbeat. Besides updating its status periodically, each kubelet renews a small Lease object in the kube-node-lease namespace, which is cheap for the API server and etcd to handle even in large clusters. The node controller uses the freshness of this lease to decide whether a node is alive.
5.3 Control Plane to Node
The API server connects to the kubelet's HTTPS endpoint, on port 10250, for three kinds of requests: fetching container logs, running commands in containers (kubectl exec and attach), and port-forwarding. It can also proxy requests to nodes, Pods and Services. Because these connections are initiated from the control plane, the kubelet must authenticate and authorize them, and production clusters should disable anonymous access and use webhook authorization on the kubelet.
In environments where the control plane cannot open connections into the nodes, for example when it is hosted in a separate network, Kubernetes supports a konnectivity proxy: agents on the nodes open outbound tunnels to a server on the control plane, and the API server sends its node-bound traffic through these tunnels.
5.4 Security of the Channels
Mutual TLS protects the connections between components, with certificates issued by the cluster's certificate authority.
etcd has its own certificates for client and peer connections, and should accept only the API servers.
RBAC and the Node authorizer restrict each kubelet to the objects relevant to its own node, so that a compromised node cannot read every Secret in the cluster.
Certificates expire. Clusters created with kubeadm issue most certificates for one year by default, so renewal must be automated or scheduled.
5.5 Ports at a Glance
Use these as a firewall and security group checklist, and restrict them: only administrators and nodes should reach the API server, only the API servers should reach etcd, and only the control plane should reach the kubelet port.
6. Following a Request from kubectl to a Running Container
The best way to see how the pieces fit together is to follow what happens when you apply a Deployment. Each step involves exactly the components described above.
You run kubectl apply. kubectl sends the Deployment to the API server, which authenticates you, authorizes the request, runs admission checks, validates the object and stores it in etcd.
The Deployment controller, watching the API server, sees the new Deployment and creates a ReplicaSet. The ReplicaSet controller then creates the requested number of Pod objects. At this point the Pods exist only as records, without a node.
The scheduler sees the unscheduled Pods, filters and scores the nodes, and writes a binding to the API server that assigns each Pod to a node.
The kubelet on each chosen node, watching the API server, notices a Pod assigned to it.
The kubelet asks the container runtime, through the CRI, to create the Pod sandbox. The runtime calls the CNI plugin to give the Pod its network namespace and IP address.
If the Pod uses persistent volumes, the kubelet works with the CSI node plugin to mount them. The runtime pulls the images and starts the containers.
The kubelet runs the probes and reports the Pod's status back to the API server. When the Pod is ready, the EndpointSlice controller adds its address to the matching Service.
kube-proxy on every node sees the updated EndpointSlice and programs its rules, so traffic to the Service now reaches the new Pod.
Notice that the control plane never starts a container, and the worker nodes never decide what to run. The two sides are connected only through the objects stored in the API server.
kubectl get pods -o wide
kubectl describe pod <pod-name>
kubectl get events --sort-by=.lastTimestamp
The events listed here are the visible trace of this sequence: Scheduled (the scheduler), Pulling, Pulled, Created and Started (the kubelet and runtime).
7. Where the Components Run
In a conventional setup, control plane nodes carry the taint node-role.kubernetes.io/control-plane with the NoSchedule effect, so ordinary Pods are not scheduled on them. This protects the control plane from noisy workloads. Small test clusters sometimes remove the taint to use all the capacity, which is acceptable for learning but not for production.
8. Failure Scenarios
Knowing what stops working, and what keeps working, when each component fails is the foundation of operating a cluster. A key property is that the data plane keeps running without the control plane: containers already started by the kubelets keep running, and the rules that kube-proxy and the CNI have programmed keep forwarding traffic.
8.1 What Happens When a Node Stops Reporting
This sequence is worth understanding in detail:
The kubelet stops renewing its lease and updating its status.
After a grace period, which is around 40 seconds by default, the node controller marks the node Unknown, so its Ready condition is no longer true, and adds the taints node.kubernetes.io/unreachable and node.kubernetes.io/not-ready.
Pods normally tolerate those taints for a period, which by default is 300 seconds, a value added by an admission plugin. After that, the Pods are evicted.
Controllers such as ReplicaSets see that replicas are missing and create replacements, which the scheduler places on healthy nodes.
For stateful workloads with single-attach volumes, replacement can take longer because the volume must be detached from the failed node first, as discussed in the storage articles.
9. High Availability
A single control plane node is a single point of failure for management, even though workloads survive its loss. A highly available design includes:
Three or five control plane nodes, each running an API server, a scheduler and a controller manager.
A load balancer or virtual IP in front of the API servers, and nodes and clients configured to use that stable endpoint, not an individual server.
Leader election for the scheduler and the controller manager. All instances run, but only one is active at a time, and another takes over if it fails. The API servers, in contrast, are all active at the same time.
A resilient etcd cluster of three or five members. kubeadm supports two topologies: stacked, where etcd runs on the control plane nodes, which is simpler, and external, where etcd has its own dedicated nodes, which separates failure domains at the price of more machines to manage.
Spread across failure domains, for example three zones within one region. etcd needs low latency between its members, so avoid stretching a control plane across distant regions.
Regular etcd backups and a rehearsed restore procedure, because HA protects against machine failure but not against deletion or corruption of data.
On the worker side, availability comes from running several replicas of each workload spread across nodes and zones, using topology spread constraints and PodDisruptionBudgets, and from having enough spare capacity to absorb the loss of a node or a zone.
10. The Node Lifecycle
10.1 Joining a Cluster
A new worker node installs a container runtime and the kubelet, obtains a bootstrap token and the cluster's endpoint and CA information, and registers. With kubeadm, this is done with the join command. The kubelet then requests a client certificate, the request is approved and signed, and the Node object appears with its labels, capacity and conditions. The node reports Ready when the kubelet, the runtime and the network plugin are working.
10.2 Conditions and Pressure
The kubelet continuously reports conditions such as Ready, MemoryPressure, DiskPressure and PIDPressure. When a pressure condition is true, the node receives a taint that stops new Pods from being scheduled there, and the kubelet may evict Pods to recover resources.
10.3 Maintenance
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node-name>
Cordon marks a node unschedulable without touching running Pods. Drain also evicts the Pods gracefully, respecting PodDisruptionBudgets, so that controllers recreate them elsewhere. After maintenance, uncordon makes the node schedulable again. Note that drain with delete-emptydir-data discards emptyDir contents, as explained in the ephemeral storage article.
11. Inspecting a Cluster
The older componentstatuses command is deprecated and gives unreliable results, so use the health endpoints of the components instead.
12. Scaling and Version Management
12.1 Scale Considerations
The Kubernetes project documents tested limits for large clusters, including up to 5,000 nodes, 150,000 Pods in total and a default of at most 110 Pods per node. In practice, the control plane limits are reached through the number of objects and the rate of requests: etcd size and latency, API server load from many watchers and controllers, and heavy list calls. The API server protects itself with API Priority and Fairness, which queues and prioritises requests. For large or busy clusters, monitor API server latency and request rates, etcd database size and disk latency, and scale the control plane machines before the limits bite.
12.2 Upgrades and Version Skew
Kubernetes defines supported version differences between components, so upgrades follow a fixed order:
Upgrade the control plane first, then the worker nodes. In an HA control plane, upgrade the API servers one at a time.
The kubelet must not be newer than the API server, and recent releases allow it to be up to three minor versions older. kube-proxy follows the same rule as the kubelet.
The scheduler and controller manager may be one minor version older than the API server, but not newer.
kubectl may differ from the API server by one minor version in either direction.
Check the official version skew policy for the exact rules of your release, and upgrade one minor version at a time. Before every upgrade, back up etcd, read the release notes and the deprecation list, and test in a non-production cluster. Drain and upgrade worker nodes gradually, respecting PodDisruptionBudgets.
13. Common Misconceptions
14. Security Best Practices
Expose the API server only to the networks that need it, and prefer private endpoints with strong authentication and RBAC.
Protect etcd: mutual TLS, access only from the API servers, encryption at rest for Secrets, and secured backups.
Harden the kubelet: disable anonymous authentication, use webhook authorization, and restrict access to port 10250.
Keep control plane nodes dedicated, tainted and access-controlled, with minimal software installed.
Enable audit logging on the API server, and send it to a system that attackers with cluster access cannot alter.
Automate certificate renewal and rotation, and track expiry dates.
Use admission controls and policy engines to enforce standards before workloads reach the nodes.
Keep every component up to date within the supported versions, and subscribe to security announcements.
15. Best Practices
Design the control plane for high availability from the start, with three or more nodes, a stable API endpoint and tested etcd backups.
Keep control plane and worker concerns separate: dedicated control plane nodes, and workloads only on workers.
Size and monitor the control plane: API server latency, etcd disk latency and database size, controller work queues and scheduler latency.
Run enough worker nodes and spare capacity to survive node and zone failures, and spread replicas with topology constraints.
Use PodDisruptionBudgets, readiness probes and graceful termination so that drains and upgrades are safe.
Rehearse failure scenarios in a test cluster: lose a control plane node, lose etcd quorum, lose a worker, and restore from backup.
Follow the upgrade order and version skew policy, one minor version at a time.
Prefer a managed control plane where it suits your needs, and spend your effort on the workloads and the worker fleet.
16. Conclusion
The split between the control plane and the worker nodes is the central idea of Kubernetes architecture. The control plane holds the desired state in etcd behind the API server, and its controllers and scheduler continuously work to turn that desired state into reality. The worker nodes, through the kubelet, the container runtime, kube-proxy and the network and storage plugins, carry out the actual work and report back. Everything meets at the API server, which is why the whole system is so consistent and so extensible.
With this model in mind, many operational questions become simple: Pods are Pending because the scheduler could not place them, a Deployment is not creating Pods because the controller manager is unhealthy, a node is NotReady because its kubelet stopped reporting, and running workloads survive a control plane outage because the data plane does not need it to keep running. Design for high availability, protect the channels between the components, upgrade in the right order, and practise failure, and your clusters will be as dependable as the architecture behind them.