Kubernetes Metrics Explained: Metrics Server and Resource Metrics

bitcodematrix.com | Kubernetes Series

1. Introduction

Many Kubernetes features need to know how much CPU and memory Pods and nodes are using right now. The Horizontal Pod Autoscaler needs it to scale Deployments, kubectl top needs it to show usage, and people need it to see which workloads are hungry. Kubernetes does not collect this data on its own. A small add-on called Metrics Server does it, and it exposes the numbers through the Kubernetes API.

This article explains what Metrics Server is and what it is not, how the resource metrics pipeline works, the different kinds of metrics in a cluster, how to install and verify it, how to use kubectl top and the Metrics API, how to interpret CPU and memory values correctly, how it relates to the autoscalers and to Prometheus, and how to troubleshoot the most common errors. It supports the earlier articles on requests and limits, HPA, VPA and resource troubleshooting.

After reading it you should be able to:

Explain the role of the kubelet, cAdvisor, Metrics Server and the Metrics API.
Install Metrics Server and check that it works.
Use kubectl top and read its units and limits correctly.
Distinguish resource metrics, custom metrics, external metrics and full monitoring.
Fix errors such as "Metrics API not available" and unknown HPA targets.

2. The Core Idea

Every node runs a kubelet, and the kubelet includes cAdvisor, a component that measures the CPU and memory usage of the containers on the node. Metrics Server periodically collects these measurements from all kubelets, keeps the latest values in memory, and publishes them as an aggregated API called metrics.k8s.io. Clients such as kubectl top and the HPA ask the Kubernetes API server, which forwards the request to Metrics Server.

Kubelet and cAdvisor feed Metrics Server, which serves the Metrics API to kubectl top and autoscalers

Figure 1: How resource metrics reach kubectl top and the autoscalers

Component	Role
cAdvisor (inside the kubelet)	Measures container CPU and memory usage on the node.
kubelet resource endpoint	Exposes the usage of the node and its Pods to Metrics Server, typically on port 10250.
Metrics Server	Collects the data from every kubelet and serves the latest values. It is a cluster add-on, not part of the core control plane.
API aggregation layer	Lets the API server route requests for metrics.k8s.io to Metrics Server.
Metrics API	The API that clients use: nodes and pods resources with CPU and memory values.

3. The Four Kinds of Metrics

The word metrics is used for several different things in Kubernetes. Keeping them apart avoids a lot of confusion.

Kind	Provided by	Used for
Resource metrics (CPU and memory)	Metrics Server, via metrics.k8s.io	kubectl top, HPA on CPU and memory, VPA recommendations.
Custom metrics	An adapter such as Prometheus Adapter, via custom.metrics.k8s.io	Scaling on application metrics associated with Kubernetes objects, such as requests per second per Pod.
External metrics	An adapter, via external.metrics.k8s.io	Scaling on data from outside the cluster, such as a queue length. KEDA is a popular alternative for event-driven scaling.
Monitoring metrics	Prometheus or another monitoring system scraping kubelet, cAdvisor, kube-state-metrics and applications	History, dashboards and alerts.

Two related monitoring sources are worth knowing. The kubelet exposes detailed container metrics in the Prometheus format at /metrics/cadvisor, which monitoring systems scrape. kube-state-metrics is a separate component that turns the state of Kubernetes objects into metrics, for example the desired and available replicas of a Deployment or the phase of a Pod. It reports object state, not resource usage.

4. Installing Metrics Server

Many managed services install it for you or offer it as an add-on, and local tools often have an add-on switch. Check first whether it is already there.

kubectl get deployment metrics-server -n kube-system

kubectl get apiservice v1beta1.metrics.k8s.io

If it is missing, install it from the official manifest or Helm chart.

# Official manifest

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Or with Helm

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/

helm upgrade --install metrics-server metrics-server/metrics-server -n kube-system

# Local clusters

minikube addons enable metrics-server

Check the deployment and the API registration. The APIService must show Available as True.

kubectl rollout status deployment/metrics-server -n kube-system

kubectl get apiservice v1beta1.metrics.k8s.io

kubectl top nodes

4.1 Requirements

The API aggregation layer must be enabled on the API server (it is by default on standard and managed clusters).
Metrics Server must reach every kubelet, normally on port 10250, so firewalls and security groups must allow traffic from the Metrics Server Pods to the nodes.
Kubelet certificates must be valid for the addresses that Metrics Server uses, or the connection fails. See the troubleshooting section.
The kubelet must allow authentication and authorization by tokens (the webhook mode), which is the normal setting.
The API server must be able to reach the Metrics Server Pod network. On some managed setups this needs extra configuration, so follow the provider guide.

4.2 The insecure TLS flag

On local clusters such as kind or minikube, kubelets often use self-signed certificates, and the common fix is the argument --kubelet-insecure-tls on the Metrics Server container. This disables verification of the kubelet certificate. It is acceptable for a throwaway development cluster, but it should not be used in production. In production, fix the kubelet serving certificates instead.

5. Using kubectl top

# Nodes

kubectl top nodes

# Pods in a namespace, and in all namespaces

kubectl top pods -n production

kubectl top pods -A

# Per container

kubectl top pods -n production --containers

# Sort by usage

kubectl top pods -A --sort-by=memory

kubectl top pods -A --sort-by=cpu

# By label

kubectl top pods -l app=web

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%

worker-1 312m 7% 2840Mi 36%

worker-2 1204m 30% 5102Mi 65%

NAME CPU(cores) MEMORY(bytes)

web-6d8f9-abc 45m 212Mi

Value	How to read it
CPU(cores)	CPU used, averaged over a short window. 1000m (millicores) equals one CPU core. Metrics Server may show nanocores (n) in the raw API.
CPU%	Node CPU usage as a percentage of the node allocatable CPU (not of its total).
MEMORY(bytes)	The working set: memory in use that cannot be easily reclaimed. It is the figure the kubelet and the OOM killer care about, and it is not the same as the total memory including cache.
MEMORY%	Node memory working set as a percentage of node allocatable memory.

6. The Metrics API Directly

kubectl top is a client of the Metrics API. You can query the API directly, which is useful for scripts and for debugging.

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes

kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/<pod-name>"

{

"kind": "PodMetrics",

"metadata": {"name": "web-6d8f9-abc", "namespace": "default"},

"timestamp": "2026-10-13T09:15:30Z",

"window": "15.0s",

"containers": [

{"name": "web", "usage": {"cpu": "45123456n", "memory": "217088Ki"}}

]

}

The timestamp and the window tell you when and over how long the usage was measured. The values are only the most recent sample, and no history is kept.

7. Who Uses Resource Metrics

Consumer	How it uses the data
kubectl top	Shows the current usage of nodes and Pods.
Horizontal Pod Autoscaler	Compares usage with the Pod requests to compute utilization and decide the replica count.
Vertical Pod Autoscaler	Uses current usage together with its own history (from the metrics it stores or from Prometheus) to recommend requests.
Kubernetes Dashboard and other tools	Display usage for nodes and Pods.

The scheduler does not use these metrics. It places Pods by requests, not by actual usage, so a node can look idle in kubectl top and still be full for scheduling.

For the HPA, every container in the Pod must define a CPU request when scaling on CPU utilization. Without it the HPA reports that it cannot compute utilization and shows unknown for the target. If Metrics Server is missing or unhealthy, the HPA cannot read its metrics and stops scaling on resources.

8. Metrics Server vs Prometheus

Aspect	Metrics Server	Prometheus-style monitoring
Purpose	Current usage for autoscaling and kubectl top	Observability: history, alerts, dashboards
History	None, latest value only	Time series with retention
Metrics	CPU and memory of nodes and Pods	Thousands: kubelet, cAdvisor, kube-state-metrics, applications
Storage	In memory	On disk or remote storage
Needed for HPA on CPU and memory	Yes	No, unless used through an adapter for custom metrics
Replaces the other	No	No

Production clusters normally run both: Metrics Server for the autoscalers and kubectl top, and a monitoring system for everything else.

9. Reading the Numbers Correctly

Usage is not requests. kubectl top shows what is used. Requests and limits are what you declared. Compare them to see whether workloads are over- or under-provisioned. See the article on requests and limits.
Memory is the working set. A container is killed for exceeding its memory limit based on usage the kernel counts against the cgroup, which is close to the working set. A value near the limit means OOMKilled risk.
CPU is an average over a short window. Short bursts and throttling are not visible. A container can be throttled at its CPU limit while kubectl top shows a modest number. Use container CPU throttling metrics in your monitoring system for that.
Data is slightly delayed. The sample is only as fresh as the scrape interval, which is commonly 15 seconds in recent versions and was longer in older ones. A new Pod shows no metrics for the first moments.
Node percentages use allocatable, not capacity. System reservations are already subtracted.
Only running containers are reported. Completed or crashed containers show no current usage.

10. Hands-On Walkthrough

Confirm that Metrics Server works.

kubectl get apiservice v1beta1.metrics.k8s.io

kubectl top nodes

Deploy a Pod that burns some CPU, limited so that it does not hurt the node.

apiVersion: v1

kind: Pod

metadata:

name: cpu-burner

spec:

containers:

- name: burner

image: busybox:1.36

command: ["sh", "-c", "while true; do :; done"]

resources:

requests:

cpu: 100m

memory: 32Mi

limits:

cpu: 200m

memory: 64Mi

Wait about a minute and check usage. CPU should be close to the 200m limit.

kubectl apply -f cpu-burner.yaml

sleep 60

kubectl top pod cpu-burner --containers

Query the raw API and compare it with kubectl top.

kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/cpu-burner

Sort all Pods by CPU to find the biggest consumers, then clean up.

kubectl top pods -A --sort-by=cpu | head

kubectl delete pod cpu-burner

Notice that the usage sits at the limit because the container is throttled there. This is a good example of why usage alone does not tell you whether a container needs more CPU.

11. Troubleshooting

Symptom	Likely cause and fix
error: Metrics API not available	Metrics Server is not installed, not ready, or the APIService is not Available. Check the Deployment and kubectl get apiservice v1beta1.metrics.k8s.io.
APIService shows False (MissingEndpoints or FailedDiscoveryCheck)	The API server cannot reach the Metrics Server Pod. Check Pod readiness, the Service and network policy, and on some setups hostNetwork or security group rules.
Logs show x509: cannot validate certificate ... doesn't contain any IP SANs	Kubelet certificates do not match the address used. Fix kubelet serving certificates, or use --kubelet-preferred-address-types, or in development only --kubelet-insecure-tls.
Logs show connection refused or timeouts to port 10250	Network, firewall or security group blocks Metrics Server from reaching nodes.
kubectl top works for nodes but not for a new Pod	The Pod is new. Wait one or two scrape intervals.
HPA shows <unknown> for CPU	Metrics Server unavailable, or the Pod has no CPU request. Add requests and check kubectl top pods.
HPA error: failed to get cpu utilization: missing request for cpu	A container in the Pod has no CPU request. Add it, including for sidecars.
Metrics are missing for some nodes	Kubelet unreachable or unhealthy on those nodes. Check node status and Metrics Server logs.
Metrics Server Pod restarts or OOMKilled	It needs more resources in very large clusters. Increase its requests and limits.
kubectl top shows more memory than expected	It shows the working set, which includes active file cache. Compare with application-level memory.

kubectl get pods -n kube-system -l k8s-app=metrics-server

kubectl logs -n kube-system deployment/metrics-server --tail=50

kubectl describe apiservice v1beta1.metrics.k8s.io

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes

12. Production Configuration

Run at least two replicas for high availability, use Pod anti-affinity or topology spread so they land on different nodes, and add a PodDisruptionBudget. The official release provides a high availability manifest, and the Helm chart has a replicas setting.
Give Metrics Server resource requests, because it is a critical component for autoscaling. The needs grow with the number of nodes and Pods, so follow the sizing guidance in its documentation.
Use properly signed kubelet serving certificates instead of --kubelet-insecure-tls.
Set a high enough priority class (for example system-cluster-critical) so it is not evicted first.
Pin the version, and check compatibility with your Kubernetes version before upgrades.
Monitor its health, and alert when the APIService is not Available, since HPA stops working without it.

helm upgrade --install metrics-server metrics-server/metrics-server -n kube-system \

--set replicas=2 \

--set priorityClassName=system-cluster-critical

13. Best Practices

Treat Metrics Server as part of the autoscaling stack and monitor it like any production component.
Use kubectl top for quick checks, and a monitoring system for trends, alerts and capacity planning.
Set requests on every container so that HPA utilization and capacity planning are meaningful.
Compare usage with requests and limits regularly, and right-size workloads.
Remember the limitations: no history, short window, no throttling data.
Avoid insecure kubelet TLS settings outside local development.
Document which metrics adapters and monitoring systems exist, so teams know where each kind of metric comes from.

14. Conclusion

Metrics Server is a small component with a big role. It turns the CPU and memory measurements of the kubelets into the Metrics API, which powers kubectl top and the autoscalers. It deliberately keeps only the latest values, so it complements, rather than replaces, a monitoring system such as Prometheus. If you install it correctly, secure its kubelet connection, run it with high availability, and read its numbers with the working set and throttling caveats in mind, you get a reliable base for autoscaling and everyday resource checks.