Kubernetes Admission Controllers
Kubernetes Admission Controllers Explained
bitcodematrix.com | Kubernetes Series
1. Introduction
Authentication proves who you are, and authorization decides whether you may perform an action. Neither of them looks at what you are actually asking for. A user who is allowed to create Pods can submit a Pod that runs as root, uses an unapproved image, has no resource limits or mounts the host's file system, and RBAC will approve it without comment. Something else has to inspect the object itself. That something is admission control.
Admission controllers are plugins in the Kubernetes API server that intercept requests after authentication and authorization and before the object is stored. They can change the object (set defaults, inject sidecars), reject it (policy violations, exhausted quotas), or both. Many features that people think of as core Kubernetes behaviour, such as default storage classes, namespace lifecycle rules, resource quotas and Pod Security enforcement, are implemented as admission controllers.
This article explains where admission fits, the difference between mutating and validating controllers, the built-in controllers, how dynamic admission webhooks work and how to write them safely, the newer CEL-based policies that often replace them, how to choose between the mechanisms, and how to operate, secure and troubleshoot admission in production. It builds on the earlier articles in this series on authentication and authorization, Pod Security Standards and SecurityContext.
2. Where Admission Fits
Every write request to the API server passes through the same pipeline. Admission is the stage that sees the complete object, together with the identity of the caller:
Figure 1: Admission control inside the API server request pipeline.
Two ordering details matter. Mutation always happens before validation, so validating controllers see the final form of the object, including any defaults and injected containers. And the checks run in a fixed sequence, so a single rejection anywhere stops the entire request.
2.1 What Admission Applies To
Write operations only. Admission runs for create, update and delete requests, and for connect requests such as exec and port-forward. Reads (get, list and watch) never pass through it.
Objects at the moment of the request. Admission does not retroactively evaluate objects that already exist. A new policy does not evict or fix running Pods, which is the same behaviour as the Pod Security labels.
Objects created by controllers too. When a Deployment's ReplicaSet creates a Pod, that Pod goes through admission like any other. This is why a policy can accept a Deployment and still block its Pods, and why the failure shows up as events on the ReplicaSet.
Dry runs. A request made with server-side dry run (kubectl apply --dry-run=server) runs the whole admission chain and then stops before persisting, which makes it ideal for testing.
3. Mutating and Validating Controllers
Some built-in plugins are both. Mutating controllers can interact with each other, because a later one may alter what an earlier one saw. For this reason, Kubernetes supports reinvocation: a mutating webhook can ask to be called again if a later mutation changed the object, by setting reinvocationPolicy to IfNeeded. Even so, you should not depend on the order in which different mutating webhooks run, and you should write mutations so that they are idempotent, producing the same result if applied twice.
4. The Built-In Admission Controllers
The API server ships with many admission plugins compiled in. A default set is enabled, and you can turn others on or off with the API server flags --enable-admission-plugins and --disable-admission-plugins. The following table lists the ones that you meet most often. The default set changes between versions, so check the documentation for your release.
Two practical consequences follow. First, you have probably already benefited from admission without writing anything: the default ServiceAccount on your Pods, the default storage class on your claims and the rejection of an over-quota request all come from this layer. Second, on managed Kubernetes services you usually cannot change the list of enabled plugins, so dynamic mechanisms such as webhooks and policies are your way to extend admission.
5. Dynamic Admission: Webhooks
5.1 How a Webhook Call Works
Webhooks let you run your own logic as a service, without changing or restarting the API server. The sequence is:
You deploy a service, typically a Pod behind a Kubernetes Service, that serves HTTPS and implements the admission API.
You register it with a ValidatingWebhookConfiguration or a MutatingWebhookConfiguration, which defines which requests are sent to it and how to reach it.
When a matching request arrives, the API server sends an AdmissionReview object to the webhook over HTTPS. It contains a unique request identifier, the operation, the resource, the user information, the new object and, for updates, the old object.
The webhook replies with an AdmissionReview response that carries the same identifier and a decision: allowed true or false, an optional message and status code for rejections, optional warnings, and, for mutating webhooks, a JSON Patch describing the changes, encoded in base64.
The API server applies the patch (mutating) or rejects the request with the webhook's message (validating).
5.2 A Validating Webhook Configuration
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: require-team-label.example.com
webhooks:
- name: require-team-label.example.com
admissionReviewVersions: ["v1"]
sideEffects: None
failurePolicy: Fail
timeoutSeconds: 5
rules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
scope: "Namespaced"
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["kube-system", "policy-system"]
clientConfig:
service:
name: policy-webhook
namespace: policy-system
path: /validate
port: 443
caBundle: <base64-encoded CA certificate>
5.3 Failure Policy: The Most Important Decision
Every webhook is a new dependency of the API server. If the webhook is slow or down, its failure policy decides what happens to the cluster:
Fail keeps the policy strict. Requests are rejected when the webhook cannot answer, which is correct for security-critical checks, but a broken webhook can block deployments, scaling and even recovery.
Ignore keeps the cluster available. Requests pass through when the webhook cannot answer, which is safer for convenience features and non-critical mutations, but it means that policy can be silently skipped during an outage.
A well-known trap is the self-deadlock. If a Fail webhook matches Pods, and the webhook itself runs as a Pod, then when its Pods are deleted the cluster cannot create replacements, because the API server asks the missing webhook for permission to create them. Prevent it by excluding the webhook's own namespace and the system namespaces with a namespaceSelector, by running several replicas with a PodDisruptionBudget and anti-affinity, and by keeping the webhook's dependencies minimal.
Note that webhooks are not called for requests that modify the webhook configuration objects themselves. This is deliberate, so that an administrator can always remove a misbehaving webhook, and it is the basis of the emergency recovery steps in Section 12.
6. Writing and Running Webhooks Safely
Writing and operating a webhook is real engineering work. Before you build one, check whether a CEL-based policy (Section 7) or an existing policy engine (Section 9) can meet the need.
7. ValidatingAdmissionPolicy: Policies in CEL
A ValidatingAdmissionPolicy is a declarative alternative to validating webhooks. The rules are written in the Common Expression Language (CEL) and evaluated inside the API server, so there is nothing to deploy, no network call, no certificate and no webhook outage to plan for. The feature reached general availability in Kubernetes 1.30.
Two objects work together. The policy defines what to check, and the binding says where to apply it and what to do on a violation:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: no-latest-tag
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.spec.template.spec.containers.all(c, !c.image.endsWith(':latest'))"
message: "Images must not use the latest tag."
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: no-latest-tag-binding
spec:
policyName: no-latest-tag
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchLabels:
environment: production
The expression has access to the incoming object, the old object for updates, request information and optional parameters. The binding's validationActions can be Deny (reject), Warn (accept but send a warning to the client) or Audit (accept and record in the audit log), alone or combined. That makes a safe rollout natural: start with Warn and Audit, review the results, and then switch to Deny.
7.1 More Capabilities
Parameters. A policy can refer to a parameter resource, such as a ConfigMap or a custom resource, through paramKind and the binding's paramRef. One policy can then serve many environments with different values, for example allowed registries.
Variables and match conditions. Reusable variables keep expressions readable, and matchConditions skip requests early.
Audit annotations. Policies can add structured information to audit events.
Custom messages and reasons. You can choose the message that users see and the HTTP reason that is returned.
7.2 Strengths and Limits
8. MutatingAdmissionPolicy
The same idea has been extended to mutation. A MutatingAdmissionPolicy defines changes to objects with CEL, without a webhook. It supports two ways to express a change: an apply configuration, which describes the fields to set in a declarative form, and a JSON Patch built from CEL expressions, which gives a migration path from existing mutating webhooks. Policies are bound to resources with a binding object, in the same way as validating ones.
The feature went through alpha and reached beta in Kubernetes 1.34, where it needed a feature gate. According to the Kubernetes project documentation, it is generally available from version 1.36. Because maturity and the API version depend on your cluster version, check the documentation of the release that you run before using it, and confirm that your managed provider has enabled it.
Typical uses are modest, well-defined defaults: adding a label or an annotation, setting a security context default, or injecting a standard environment variable. For complex, stateful or externally driven mutation, such as sidecar injection that depends on other systems, a webhook or a policy engine is still appropriate.
Related work in the project aims to let administrators configure admission policies and webhooks from files at API server start-up, so that they are in force before the first request and cannot be removed through the API. Treat this as an emerging capability, and check its status for your version.
9. Policy Engines
Many organisations use a general-purpose policy engine instead of writing individual webhooks. The most widely used ones, Kyverno and OPA Gatekeeper, are themselves admission webhooks that evaluate a library of policies.
Policy engines add features that plain admission lacks, such as background scanning of existing resources and reports of violations, which helps when you introduce a rule to a cluster that already has workloads. They also bring the operational cost of a webhook: they must be highly available, correctly scoped and monitored. Because both of them use the same underlying admission mechanism, everything in Sections 5 and 6 about failure policy and availability applies to them.
10. Choosing the Right Mechanism
A good rule of thumb is to climb the ladder only as far as needed: built-in controllers first, then in-process CEL policies, then a policy engine, and a custom webhook last. Each step up adds capability, but also operational risk.
11. Common Use Cases
12. Operating Admission Safely
12.1 Roll Out New Rules Gradually
Start with Warn and Audit in the binding, or with the audit mode of your policy engine, and review who would be affected.
Test with server-side dry runs against real manifests, for example in CI.
Enable Deny first in non-production namespaces, then in production, one namespace group at a time.
Announce changes, document each rule with its reason and the fix, and put clear text in the message, because the message is what developers will read.
12.2 Monitor
The API server exposes metrics for admission, including the duration of each admission controller and webhook and counts of webhook rejections and errors. Watch them for slow webhooks and for rising rejection or error rates, and alert when a webhook's latency approaches its timeout. Use audit logs to see which requests were denied and by which policy, and audit annotations from policies for extra detail.
12.3 Inspect What Is Installed
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get validatingadmissionpolicies
kubectl get validatingadmissionpolicybindings
kubectl describe validatingwebhookconfiguration <name>
Review these regularly. A forgotten webhook from a removed tool is a classic source of mysterious failures: the configuration remains, the service behind it is gone, and every matching request fails.
12.4 Emergency Recovery
If a webhook with failurePolicy Fail is broken and blocks important operations, you can still fix it, because the API server does not call webhooks for changes to the webhook configurations themselves. With sufficient permissions, either delete the configuration, or change its failure policy:
kubectl delete validatingwebhookconfiguration <name>
kubectl patch validatingwebhookconfiguration <name> --type=json \
-p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'
Keep this procedure in your runbook, make sure that the people on call have the right to run it, and remember to restore enforcement and fix the root cause afterwards.
13. Security Considerations
Webhook configuration rights are powerful. Anyone who can create or edit webhook configurations or admission policies can block or alter every matching request in the cluster, including security-sensitive ones. Treat these permissions as nearly equivalent to cluster-admin, and restrict them in RBAC.
Webhooks see full objects. Data sent to a webhook can include Secrets and credentials if the rules match them. Avoid matching Secrets unless needed, secure the webhook's service and logs, and treat third-party webhooks as part of your trusted computing base.
Protect the policy itself. Run policy engines and webhooks in protected namespaces, with limited access and hardened Pods, because a compromise there gives control over admission.
Do not rely on admission alone. It is one layer, together with authentication, RBAC, network policies and runtime controls. It also does not retroactively check existing objects, so scan for those separately.
Beware of bypass paths. Exemptions, excluded namespaces, and Ignore failure policies are deliberate holes. Keep them few and review them often.
Audit. Enable audit logging, and monitor changes to webhook configurations, policies, bindings and exemptions.
14. Hands-On Walkthrough
This exercise uses only built-in features, so it works on any recent cluster (the CEL policy step needs Kubernetes 1.30 or later). Use a test cluster.
Create a namespace with a LimitRange that defines default resources. This is a built-in mutating and validating controller (LimitRanger).
kubectl create namespace adm-demo
kubectl label namespace adm-demo environment=production
kubectl apply -n adm-demo -f - <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 128Mi
default:
cpu: 200m
memory: 256Mi
EOF
Create a Pod with no resources, and see that admission added them (mutation).
kubectl run plain -n adm-demo --image=busybox:1.36 \
--restart=Never --command -- sleep 3600
kubectl get pod plain -n adm-demo \
-o jsonpath='{.spec.containers[0].resources}'
The output shows requests and limits that you never wrote. They were injected by the mutating part of the LimitRanger plugin before the object was stored.
Add a CEL validating policy and binding in Warn and Audit mode, using the manifests from Section 7. Replace the binding's validationActions with Warn and Audit, and apply both objects.
kubectl apply -f no-latest-tag.yaml
kubectl apply -f no-latest-tag-binding.yaml
Create a Deployment that uses the latest tag, and read the warning.
kubectl create deployment latest-app -n adm-demo --image=nginx:latest
The Deployment is created, and kubectl prints a warning with the policy's message. Nothing was blocked, which is exactly the point of Warn mode.
Switch the binding to Deny, and repeat.
kubectl patch validatingadmissionpolicybinding no-latest-tag-binding \
--type=merge -p '{"spec":{"validationActions":["Deny"]}}'
kubectl create deployment latest-two -n adm-demo --image=nginx:latest
The request is now rejected with an error that names the policy and the binding and shows your message. Creating a Deployment with a pinned tag, such as nginx:1.27, succeeds.
Check the effect without changing anything, using a server-side dry run.
kubectl create deployment latest-three -n adm-demo --image=nginx:latest \
--dry-run=server
The dry run runs through the same admission chain, and returns the same rejection, without creating the object.
Clean up.
kubectl delete validatingadmissionpolicybinding no-latest-tag-binding
kubectl delete validatingadmissionpolicy no-latest-tag
kubectl delete namespace adm-demo
15. Troubleshooting
16. Best Practices
Use built-in controllers and Pod Security first, then CEL policies, then policy engines, and write a custom webhook only when nothing simpler fits.
Roll out every new rule with Warn and Audit before Deny, and test with server-side dry runs in CI.
Scope rules tightly by resource, operation and namespace, and exclude system namespaces and the webhook's own namespace.
Choose the failure policy deliberately: Fail for security-critical checks backed by a highly available service, and Ignore for convenience features.
Run webhooks and policy engines with several replicas, anti-affinity, a PodDisruptionBudget and automated certificate rotation, and monitor their latency and errors.
Write idempotent mutations, and never rely on the order of different mutating webhooks.
Write clear rejection messages that explain what is wrong and how to fix it.
Keep policies in version control, with review, tests and documented owners.
Restrict who can create or modify webhook configurations, policies and bindings, and audit changes to them.
Regularly inventory the installed webhooks and policies, and remove those that are unused.
Keep an emergency procedure to disable a broken webhook, and rehearse it.
Combine admission with RBAC, network policies, runtime security and scanning of existing resources, because no single layer is enough.
17. Conclusion
Admission controllers are the point at which Kubernetes looks at the object itself, after it has decided that the caller is who they claim to be and may act. Built-in plugins supply the defaults and guardrails that make the platform behave sensibly. Webhooks open the pipeline to custom logic, at the price of running and securing a critical service. CEL-based policies bring the most common rules into the API server itself, and policy engines package them with libraries, reports and automation.
Use admission as the place to encode your organisation's standards, but do it with the discipline you would apply to any production dependency: start with the simplest mechanism, roll rules out gradually, design for failure, watch the metrics and keep an emergency exit. Done well, admission control lets teams move quickly, because the platform itself prevents the mistakes that would otherwise have to be caught in review or discovered in production.