Kubernetes Monitoring Architecture

bitcodematrix.com | Kubernetes Series

1. Introduction

A Kubernetes cluster has many moving parts: nodes, the control plane, Pods that come and go, and applications of many kinds. Monitoring means collecting data about all of them, keeping it long enough to see trends, showing it in a useful way and alerting people when something needs attention. Kubernetes itself provides only the first small piece of this, the resource metrics used by kubectl top and the autoscalers. A complete monitoring setup is something you design and add.

This article explains a standard monitoring architecture built around Prometheus. It covers the layers of a cluster that need monitoring, the components and how they fit together, how Prometheus discovers targets, the Prometheus Operator and its custom resources, the key metrics and alerts, scaling and long-term storage, security, a hands-on installation, and troubleshooting. It builds on the earlier articles on Metrics Server and logging architecture.

After reading it you should be able to:

Describe each layer of a Kubernetes monitoring system and its data sources.
Explain how Prometheus, Alertmanager and Grafana work together.
Use ServiceMonitors and PrometheusRules to monitor applications and alert on problems.
Choose the right metrics and alerting method for nodes, workloads and the control plane.
Plan for scale, retention, security and troubleshooting.

2. What Needs to Be Monitored

Layer	What it tells you	Typical source
Nodes	CPU, memory, disk, network and load of the machines	node_exporter
Containers and Pods	Resource usage and throttling per container	kubelet with cAdvisor
Kubernetes objects	Desired and actual state: replicas, Pod phase, restarts, node conditions	kube-state-metrics
Control plane	API server latency, scheduler queue, controller queues, etcd health	Component /metrics endpoints
Applications	Requests, errors, latency and business metrics	Application /metrics endpoints or exporters
Cluster add-ons	Ingress controller, DNS, service mesh, storage	Their own metrics endpoints

Metrics are only one of the three signals of observability. Logs are covered in the logging architecture article, and traces, which show a request across services, are usually collected with OpenTelemetry and a tracing backend. This article focuses on metrics, which are the basis of dashboards and alerts.

3. The Architecture at a Glance

Prometheus scrapes metric sources, sends alerts to Alertmanager and serves Grafana

Figure 1: Prometheus at the center of the monitoring pipeline

Component	Role
Prometheus	Discovers targets, scrapes their metrics over HTTP (a pull model), stores them in a local time series database, evaluates alerting and recording rules.
node_exporter	Runs on every node (usually as a DaemonSet) and exposes operating system metrics.
kubelet and cAdvisor	Expose container resource metrics at /metrics/cadvisor and kubelet metrics at /metrics.
kube-state-metrics	Watches the Kubernetes API and exposes the state of objects as metrics. It reports state, not usage.
Alertmanager	Receives alerts from Prometheus, groups, deduplicates, routes and silences them, and sends notifications.
Grafana	Visualizes metrics with dashboards, and queries Prometheus (and other data sources).
Long-term storage	Optional systems such as Thanos, Mimir, Cortex or VictoriaMetrics that keep data longer and across clusters.

4. How Prometheus Finds Targets

Pods change constantly, so Prometheus cannot rely on a fixed list of addresses. It uses Kubernetes service discovery: it watches the Kubernetes API for nodes, Pods, Services, Endpoints or EndpointSlices and builds its list of scrape targets from them. Relabeling rules decide which discovered objects are scraped and which labels are attached.

scrape_configs:

- job_name: kubernetes-pods

kubernetes_sd_configs:

- role: pod

relabel_configs:

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

action: keep

regex: "true"

Hand-written scrape configuration works but becomes hard to manage. Most clusters use the Prometheus Operator, which turns scrape and alert configuration into Kubernetes objects.

5. The Prometheus Operator and Its Resources

The Prometheus Operator watches custom resources and generates the Prometheus configuration for you. The most important resources are listed below.

Resource	Purpose
Prometheus	Defines a Prometheus server deployment: replicas, retention, storage and which monitors it selects.
Alertmanager	Defines an Alertmanager deployment.
ServiceMonitor	Tells Prometheus to scrape the Pods behind a Service, selected by labels, on a named port.
PodMonitor	Scrapes Pods directly, without a Service.
PrometheusRule	Defines alerting rules and recording rules.
Probe and ScrapeConfig	Scrape black-box probes or other external targets.

The kube-prometheus-stack Helm chart installs the Operator, Prometheus, Alertmanager, Grafana, node_exporter, kube-state-metrics and a large set of ready-made dashboards and alert rules. It is the most common starting point.

5.1 Monitoring an application with a ServiceMonitor

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

namespace: production

labels:

release: monitoring

spec:

selector:

matchLabels:

app: web

endpoints:

- port: metrics

interval: 30s

path: /metrics

The selector matches labels on the Service, not on the Pods.
The port value is the name of a port in the Service, so the Service port must be named (here, metrics).
The Prometheus resource decides which ServiceMonitors it picks up, using label and namespace selectors. In kube-prometheus-stack the default expects the label release equal to the Helm release name, which is why the label appears above.

5.2 Alerting with a PrometheusRule

apiVersion: monitoring.coreos.com/v1

kind: PrometheusRule

metadata:

namespace: production

labels:

release: monitoring

spec:

groups:

- name: web

rules:

- alert: WebHighErrorRate

expr: |

sum(rate(http_requests_total{app="web",status=~"5.."}[5m]))

/ sum(rate(http_requests_total{app="web"}[5m])) > 0.05

for: 10m

labels:

severity: critical

annotations:

summary: "More than 5% of web requests fail"

runbook_url: "https://wiki.example.com/runbooks/web-errors"

The metric name http_requests_total is an example. Use the metric names that your application exposes.

6. Metrics That Matter

Metric (source)	What it shows
kube_pod_status_phase (kube-state-metrics)	Number of Pods per phase, for example Pending or Failed.
kube_pod_container_status_restarts_total	Container restarts. Rapid growth means crash loops.
kube_pod_container_status_last_terminated_reason	Why containers last terminated, for example OOMKilled.
kube_deployment_status_replicas_available	Available replicas compared with the desired number.
kube_node_status_condition	Node Ready and pressure conditions.
kube_pod_container_resource_requests and limits	Declared requests and limits, to compare with usage.
container_cpu_usage_seconds_total (cAdvisor)	CPU used per container, used with rate().
container_memory_working_set_bytes	Memory working set per container, close to what the OOM killer sees.
container_cpu_cfs_throttled_periods_total	How often a container was throttled at its CPU limit.
node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes	Node CPU, memory and disk (node_exporter).
apiserver_request_duration_seconds, apiserver_request_total	API server latency and request counts.
scheduler_pending_pods	Pods waiting in the scheduler queues.
etcd_server_has_leader, etcd_disk_wal_fsync_duration_seconds	etcd health and disk latency.
workqueue_depth	Backlog of controller work queues.

Metric names and labels change between component versions, so confirm them in the /metrics output or in the documentation of the version you run.

7. Useful Alerts

Good alerts describe symptoms that need a human, not every internal change. These PromQL examples are common starting points. Tune the thresholds for your environment.

Alert	PromQL expression
Pods Pending too long	sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0, with for: 15m
Container restarting often	increase(kube_pod_container_status_restarts_total[1h]) > 3
Node not Ready	kube_node_status_condition{condition="Ready",status="true"} == 0, with for: 5m
Deployment missing replicas	kube_deployment_status_replicas_available < kube_deployment_spec_replicas, with for: 10m
CPU throttling	sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m])) / sum by (pod) (rate(container_cpu_cfs_periods_total[5m])) > 0.25
Memory near limit	container_memory_working_set_bytes{container!=""} / on (namespace,pod,container) kube_pod_container_resource_limits{resource="memory"} > 0.9
Node disk almost full	node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1
Prometheus down	up{job="prometheus"} == 0

7.1 Alerting practices

Alert on symptoms that affect users, such as errors and latency, and use cause-based alerts like CPU or restarts as supporting signals.
Use the for clause to avoid alerts caused by brief spikes.
Add a severity label, a summary and a runbook link to every alert, and route by severity in Alertmanager.
Group related alerts and use inhibition so that one node failure does not page you for every Pod on it.
Review alerts regularly, and remove those that nobody acts on.
Use a dead-man switch alert that always fires, to prove the whole alerting path works.

8. Dashboards and Methods

Grafana dashboards turn metrics into pictures. kube-prometheus-stack ships dashboards for nodes, Pods, namespaces, workloads and the control plane. Two methods help to decide what to show.

Method	Meaning
USE (for resources such as nodes)	Utilization, Saturation and Errors for each resource.
RED (for services)	Rate of requests, Errors and Duration (latency).
Four golden signals	Latency, traffic, errors and saturation.

Keep dashboards focused: an overview of the cluster, one per service with RED metrics, and detailed views for troubleshooting.

9. Scale, Retention and High Availability

Local storage: Prometheus keeps data on local disk for a limited time (the default retention is 15 days). Use persistent volumes so that data survives restarts.
High availability: run two identical Prometheus replicas that scrape the same targets, with Alertmanager running in a cluster. Deduplication is handled by Alertmanager and by Thanos or similar systems.
Long-term storage: send data with remote_write to Thanos, Mimir, Cortex, VictoriaMetrics or a managed service, for retention of months or years and a global view of many clusters.
Cardinality: memory use grows with the number of unique time series. Avoid labels with unbounded values such as user IDs or full URLs, and drop metrics you do not use.
Scrape interval: 15 to 30 seconds is typical. Shorter intervals cost more storage and CPU.
Sharding: very large environments split targets across several Prometheus servers, or use agent mode with a central store.

Managed offerings such as Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus, Azure Monitor managed service for Prometheus, and commercial platforms like Datadog remove much of this operational work. They often keep the same Prometheus-style collection, so the concepts in this article still apply.

10. Security

Grant Prometheus only the Kubernetes API permissions it needs for discovery, using a dedicated ServiceAccount and read-only RBAC.
Protect the Prometheus, Alertmanager and Grafana endpoints with authentication and TLS, and do not expose them directly to the internet.
Use NetworkPolicies to control who can reach metrics endpoints, and expose application metrics only on an internal port.
Remember that metrics can leak sensitive information in labels. Review what you expose.
Restrict who may create ServiceMonitors and PrometheusRules, since they can add scrape targets and change alerting.

11. Hands-On Walkthrough

Install kube-prometheus-stack with Helm into a monitoring namespace.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \

-n monitoring --create-namespace

Wait until the Pods are running.

kubectl get pods -n monitoring

Open Grafana locally and read the generated admin password.

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

kubectl get secret -n monitoring monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 -d; echo

Open the Prometheus UI and check the targets (Status, Targets) and run a query.

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

# Example queries in the UI

sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))

sum by (phase) (kube_pod_status_phase)

Create a PrometheusRule with the label release: monitoring, for example a test alert that always fires, and watch it appear in the Alerts page and in Alertmanager.

apiVersion: monitoring.coreos.com/v1

kind: PrometheusRule

metadata:

namespace: monitoring

labels:

release: monitoring

spec:

groups:

- name: test

rules:

- alert: AlwaysFiring

expr: vector(1)

labels:

severity: info

annotations:

summary: "Test alert to verify the alerting path"

Remove the test alert and, if this was a test cluster, uninstall the release.

kubectl delete prometheusrule test-alert -n monitoring

helm uninstall monitoring -n monitoring

Service names and labels depend on the Helm release name and chart version. If a name differs, list the Services with kubectl get svc -n monitoring.

12. Troubleshooting

Symptom	Likely cause and fix
A target is DOWN in Prometheus	Wrong port or path, the Pod is not ready, NetworkPolicy blocks Prometheus, or the endpoint needs TLS or authentication. Check the error shown on the Targets page.
ServiceMonitor is not discovered	Its labels do not match the Prometheus serviceMonitorSelector (often the release label), or its namespace is not selected. Check the Prometheus resource.
ServiceMonitor found but no targets	The selector does not match the Service labels, or the port name does not match a named Service port.
PrometheusRule is ignored	Rule labels do not match the ruleSelector, or the expression has errors. Check the Rules page.
Alerts fire but no notification arrives	Alertmanager routing or receiver configuration is wrong, or the alert is silenced or inhibited. Check the Alertmanager UI.
Control plane targets are down or absent	On managed clusters the control plane is not scrapeable. Disable those rules and use the provider metrics.
Gaps in graphs after a restart	Prometheus has no persistent volume, or is OOMKilled. Add storage and memory.
Prometheus uses too much memory	High cardinality. Find the largest metrics, drop unused ones, and remove unbounded labels.
Dashboards show No data	Wrong data source or variable, metric names changed, or the exporter is not running.
kubectl top works but Prometheus lacks container data	The kubelet cAdvisor target is down or the scrape is blocked. Check the kubelet job.

kubectl get servicemonitor,podmonitor,prometheusrule -A

kubectl get prometheus -n monitoring -o yaml | grep -A5 -i selector

kubectl logs -n monitoring prometheus-monitoring-kube-prometheus-prometheus-0 -c prometheus --tail=50

13. Best Practices

Use kube-prometheus-stack or a managed service as the baseline, instead of building everything by hand.
Keep Metrics Server for autoscaling and Prometheus for monitoring. They have different jobs.
Instrument applications with RED metrics and expose them in the Prometheus format.
Define monitoring as code: ServiceMonitors, rules and dashboards in Git, deployed with your applications.
Alert on symptoms, link every alert to a runbook, and review alert noise regularly.
Plan retention and long-term storage early, and control cardinality.
Run Prometheus and Alertmanager with high availability and persistent storage in production.
Monitor the monitoring system itself, and test the alert path with a dead-man switch.
Secure access to all monitoring components.

14. Conclusion

A Kubernetes monitoring architecture is a pipeline: exporters and endpoints produce metrics, Prometheus discovers and scrapes them, rules turn them into alerts, Alertmanager delivers the alerts, and Grafana shows the data. The Prometheus Operator makes this manageable through ServiceMonitors and PrometheusRules, and the kube-prometheus-stack chart gives a working baseline in minutes. The real work is in choosing meaningful metrics and alerts, controlling cardinality, planning storage and keeping the system itself reliable. With these in place, you can see problems early and understand them quickly.