Kubernetes Secrets Explained: Managing Sensitive Configuration
Kubernetes Secrets
Kubernetes Secrets Explained: Managing Sensitive Configuration
bitcodematrix.com | Kubernetes Series
1. Introduction
Every real application needs credentials: database passwords, API tokens, TLS private keys, registry logins. Putting them in container images or in ConfigMaps is dangerous, because both are easy to copy and read. Kubernetes offers a dedicated object for this purpose, the Secret, together with mechanisms to deliver sensitive values to Pods and to control who can see them.
There is, however, a widespread misunderstanding about Secrets: many people assume that a Secret is encrypted by default. It is not. A default Secret is only base64-encoded, and the security of the whole design depends on additional measures such as encryption at rest, tight RBAC and careful delivery. This article explains how Secrets work, what protects them and what does not, how to create and consume them, how to rotate them, and how to integrate external secret managers. It complements the earlier articles on ConfigMaps and on emptyDir, ConfigMap and Secret volumes.
2. What Is a Secret?
A Secret is a namespaced API object that holds a small amount of sensitive data, as key-value pairs. Structurally it resembles a ConfigMap, but Kubernetes treats it differently in several ways:
Values in the API are stored and returned base64-encoded. This is a transport encoding for arbitrary bytes, not a protection.
When mounted as a volume, a Secret is kept in memory (tmpfs) on the node, so it is not written to the node's disk.
The kubelet receives only the Secrets that are used by Pods scheduled to its own node, because of the Node authorizer.
Secrets can be encrypted in etcd when the API server is configured to do so.
Tooling treats them with more care: kubectl describe shows only the sizes of values, and audit policies can avoid logging their content.
Like ConfigMaps, a Secret is limited to about 1 MiB and can be referenced only from Pods in the same namespace.
2.1 The Honest Security Model
It helps to be precise about what a Secret does and does not give you:
In short, a Secret is a building block. Security comes from the combination of encryption, access control, delivery method, rotation and good habits around it.
3. Secret Types
The type field of a Secret declares its purpose, and Kubernetes validates that the expected keys are present for built-in types.
Using the specific type, for example kubernetes.io/tls for certificates, gives early validation and lets other tools, such as ingress controllers and Gateway implementations, find the data in the format that they expect.
4. Creating Secrets
4.1 With kubectl
kubectl create secret generic db-creds -n shop \
--from-literal=username=app \
--from-literal=password='S3cr3t-value'
kubectl create secret generic api-key -n shop \
--from-file=api-key=./api-key.txt
kubectl create secret tls web-tls -n shop \
--cert=tls.crt --key=tls.key
kubectl create secret docker-registry regcred -n shop \
--docker-server=registry.example.com \
--docker-username=ci-bot \
--docker-password='<token>'
Be aware that values typed on the command line can end up in your shell history and in process listings. For real credentials, prefer reading from a file with restricted permissions, or from your secret manager, rather than typing them as literals.
4.2 With a Manifest
apiVersion: v1
kind: Secret
metadata:
name: db-creds
namespace: shop
type: Opaque
stringData:
username: app
password: S3cr3t-value
A manifest can use either of two fields. With data, the values must already be base64-encoded. With stringData, you write plain text, and the API server encodes it when storing the object. The stringData field is write-only: when you read the Secret back, you see only data. If both fields contain the same key, stringData wins.
4.3 Common Creation Mistakes
Trailing newline. Encoding with echo instead of echo -n adds a newline to the value, which breaks logins in subtle ways. Use echo -n, or stringData.
Double encoding. Putting a base64 value in stringData, or encoding twice, results in the application receiving encoded text.
Plain manifests in Git. A YAML file with a Secret is only base64-encoded, and committing it exposes the value. See Section 11.
Wrong namespace. The Secret must exist in the Pod's namespace.
To read a value back for troubleshooting, decode it, preferably in a protected terminal:
kubectl get secret db-creds -n shop -o jsonpath='{.data.password}' | base64 -d
5. Consuming Secrets
A Pod can use a Secret as environment variables, as files in a volume, or as image pull credentials. The Deployment fragment below shows all three:
spec:
template:
spec:
imagePullSecrets:
- name: regcred
automountServiceAccountToken: false
containers:
- name: app
image: registry.example.com/shop/app:1.0
env:
- name: DB_USER
valueFrom:
secretKeyRef:
name: db-creds
key: username
volumeMounts:
- name: db-creds
mountPath: /etc/secrets
readOnly: true
volumes:
- name: db-creds
secret:
secretName: db-creds
defaultMode: 0400
5.1 Environment Variables
The secretKeyRef field injects a single key, and envFrom with secretRef injects all keys. They are convenient and need no application changes, but they carry real risks. Environment variables are inherited by child processes, can appear in crash dumps and diagnostic endpoints, and are visible to anyone who can inspect the container on the node. They are also fixed at start-up and never update.
5.2 Files in a Volume
A secret volume turns each key into a read-only file, held in memory on the node. You control permissions with defaultMode or a per-file mode, and the files are refreshed when the Secret changes, except in subPath mounts. For most sensitive data, files are the better choice, because file permissions provide a basic control, and rotation without a restart becomes possible if the application re-reads the file.
5.3 Image Pull Secrets
The imagePullSecrets field in a Pod, or in a ServiceAccount used by Pods, tells the kubelet which Secret of type kubernetes.io/dockerconfigjson to use when pulling images. Attaching it to the ServiceAccount avoids repeating it in every manifest.
5.4 Reducing Exposure
Mount only the Secrets and keys each container needs.
Use readOnly mounts and restrictive file modes, and ensure that the process user can read the files. If you set fsGroup, check the resulting permissions.
Never place secret values in command-line arguments, because they appear in process listings and in the Pod specification.
Set automountServiceAccountToken to false for Pods that do not need to call the Kubernetes API, so that no credential is mounted needlessly.
6. Where Secrets Live and Who Can See Them
Follow a Secret from creation to use, and you see several places where it exists, each with its own protection:
The takeaway is that encryption at rest and RBAC are the two controls that you must configure yourself, while the rest is provided or encouraged by default.
7. Encryption at Rest
By default, the API server stores Secrets in etcd without encryption. Anyone who can read etcd, or a backup of it, can read every Secret. To prevent this, configure encryption at rest on the API server with an EncryptionConfiguration file, passed through the --encryption-provider-config flag:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded 32-byte key>
- identity: {}
The order of providers matters. The first provider is used to encrypt new writes, and all listed providers can be used to decrypt. The identity provider means "no encryption", and placing it last lets the API server read objects that were stored before encryption was enabled.
7.1 Local Keys or a KMS
Providers such as aescbc and secretbox use keys that sit in the configuration file on the control plane nodes, so anyone who obtains that file can decrypt the data. The stronger approach is a KMS provider, which uses envelope encryption: each object is encrypted with a data key, and that key is encrypted by a key that never leaves an external key management service. Version 2 of the KMS provider API is the recommended one.
providers:
- kms:
apiVersion: v2
name: my-kms
endpoint: unix:///var/run/kmsplugin/socket.sock
timeout: 3s
- identity: {}
On managed Kubernetes services, you typically enable envelope encryption through a cluster setting that references a key in the provider's key service, and the provider handles the API server configuration.
7.2 Existing Secrets and Verification
Enabling encryption affects only objects written afterwards. To encrypt Secrets that already exist, rewrite them:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
On clusters where you control etcd, verify the result by reading a Secret directly from etcd and checking that the stored value starts with a provider prefix, such as k8s:enc:aescbc:v1:key1, and not with readable text. Plan for key rotation as well: add the new key, make it the first entry, restart the API servers, and re-encrypt all Secrets before removing the old key. Always back up your keys and your configuration, because losing them makes the data unrecoverable.
8. Access Control
8.1 Understand What Grants Access
Several permissions expose Secrets, and some of them are not obvious:
get, list and watch on secrets. The list and watch verbs return the full data of every Secret in scope. Treat them as equivalent to read access to all values.
Creating Pods in a namespace. Anyone who can create a Pod can mount any Secret in that namespace, or use it as an environment variable, and read it from inside the container.
exec into Pods. Anyone who can run commands in a container can read its mounted Secrets and environment.
Access to etcd, backups or node file systems. These bypass Kubernetes RBAC entirely.
8.2 Grant the Minimum
Give workloads and people only the access that they need. A Role that allows reading one named Secret looks like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: read-db-creds
namespace: shop
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["db-creds"]
verbs: ["get"]
Note that resourceNames restricts the get verb to the named object, but granting list or watch on secrets, even with resource names, can still expose more than intended, so avoid those verbs for Secrets. Avoid cluster-wide roles that include secrets, and review bindings regularly, including the permissions of controllers and operators that you install.
8.3 Other Controls
Namespaces as boundaries. Separate teams and environments into namespaces, and keep their Secrets apart.
ServiceAccount hygiene. Use a dedicated ServiceAccount per workload, and disable token auto-mounting where it is not needed.
Audit logging. Configure the audit policy to record Secret access at the Metadata level, so that you know who touched which Secret without writing the values into the audit log.
Policy enforcement. Use admission policies to block risky patterns, such as Secrets passed as plain environment variables where files are required, or manifests that contain obvious credentials.
Immutability. Setting immutable to true on a Secret prevents in-place changes and reduces watch load, with the same versioning pattern as for ConfigMaps.
9. Rotation and Updates
Credentials should change over time, whether on a schedule or after a suspected leak. Kubernetes makes rotation possible, but you must design for it.
A safe rotation usually follows an overlap pattern, because systems rarely switch instantly:
Create the new credential at the source (the database, the API provider), while the old one still works.
Update the Secret, or create a new versioned Secret, and roll it out to the workloads.
Confirm that all consumers use the new credential.
Revoke the old credential at the source.
Where possible, move away from long-lived static credentials altogether. Short-lived, dynamically issued credentials, from a secrets manager or from workload identity, shrink the window of exposure, and make rotation automatic rather than an event.
10. Service Account Tokens and Workload Identity
Pods that call the Kubernetes API, or external services that trust the cluster, need an identity. Older versions of Kubernetes created a long-lived token Secret for every ServiceAccount. Current versions no longer create these automatically. Instead, Pods receive projected, short-lived, audience-bound tokens that the kubelet refreshes, and that are invalidated when the Pod is deleted. You can also request a token explicitly:
kubectl create token my-serviceaccount -n shop --duration=1h
Create a long-lived token Secret only when an external system cannot use the short-lived mechanisms, and treat it as a high-value credential. For access to cloud services, use the platform's workload identity features, which let a ServiceAccount assume a cloud role without static keys stored in a Secret. This removes an entire category of credentials from your cluster.
11. Secrets in GitOps and CI/CD
A declarative workflow wants every object in Git, but a plain Secret manifest in Git is a leaked secret. There are three common solutions:
Whichever you choose, keep Secrets out of CI logs by using masking, avoid printing environment variables during builds, and scan repositories and images for accidentally committed credentials.
12. External Secret Managers
For production workloads, many organisations keep the source of truth for secrets in a dedicated manager, such as a cloud secrets service or a vault product, and integrate it with Kubernetes. Three integration patterns are common:
These integrations bring auditing, central rotation, fine-grained policies and often dynamic credentials. They also add dependencies, so plan for what happens when the manager is unreachable: decide whether Pods may start with cached values and how long, and test it.
13. Hands-On Walkthrough
Create a Secret from literals.
kubectl create secret generic demo-creds -n shop \
--from-literal=username=demo \
--from-literal=password='first-password'
Look at it, and notice that describe hides the values while get shows them encoded.
kubectl describe secret demo-creds -n shop
kubectl get secret demo-creds -n shop -o yaml
kubectl get secret demo-creds -n shop -o jsonpath='{.data.password}' | base64 -d
Mount it in a Pod as a volume and read it.
apiVersion: v1
kind: Pod
metadata:
name: secret-demo
namespace: shop
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: creds
mountPath: /etc/creds
readOnly: true
volumes:
- name: creds
secret:
secretName: demo-creds
defaultMode: 0400
kubectl exec secret-demo -n shop -- cat /etc/creds/password
Rotate the value and wait a minute or two, then read the file again.
kubectl patch secret demo-creds -n shop \
-p '{"stringData":{"password":"second-password"}}'
kubectl exec secret-demo -n shop -- cat /etc/creds/password
The mounted file shows the new value after the refresh, which shows that rotation without a restart is possible for files. If the value had been injected as an environment variable, it would still show the old password.
Clean up.
kubectl delete pod secret-demo -n shop
kubectl delete secret demo-creds -n shop
14. When a Secret Leaks
Assume that sooner or later a credential will be exposed: committed to a repository, printed in a log, or visible to someone who should not have seen it. A prepared response matters more than perfect prevention.
Rotate at the source first. Revoke or change the credential in the system that it protects, such as the database or the API provider. Changing the Kubernetes Secret alone does not invalidate the leaked value.
Deploy the new credential to the workloads using the overlap pattern, and verify that they work.
Find the exposure. Check Git history, CI logs, container images, audit logs and backups, and determine who and what could have used the value.
Remove copies. Purge the value from repositories and logs where possible, recognising that history rewriting does not undo exposure that has already happened.
Check for misuse. Review access logs at the target system for the period of exposure.
Fix the cause. Adjust RBAC, pipelines or habits, add scanning and policy controls, and record the lesson.
15. Troubleshooting
16. Best Practices
Enable encryption at rest for Secrets, preferably with a KMS provider, and protect etcd and its backups.
Apply least-privilege RBAC, avoid list and watch on Secrets, and remember that creating Pods or exec access also exposes them.
Deliver Secrets as read-only files with restrictive permissions, and avoid environment variables and command-line arguments for sensitive values.
Disable ServiceAccount token auto-mounting where the Pod does not call the API, and use projected short-lived tokens and workload identity instead of static keys.
Never commit plain Secret manifests, and choose encrypted-in-Git, external references or pipeline injection for GitOps.
Prefer an external secrets manager for production, with central auditing and dynamic, short-lived credentials where possible.
Design for rotation: use overlap, files that can be re-read, and rollouts tied to Secret changes, and rotate on a schedule.
Use separate Secrets per application and environment, and use the specific Secret types for TLS and registry credentials.
Turn on audit logging for Secret access at the Metadata level, and alert on unusual reads.
Keep a leak-response runbook, and practise it, including rotation at the source.
Scan repositories, images and CI logs for credentials, and enforce policies through admission control.
17. Conclusion
Kubernetes Secrets are a necessary part of running real applications, but they are not magic. A default Secret is base64-encoded text with a few helpful handling rules; it becomes truly protected only when you add encryption at rest, narrow RBAC, careful delivery to Pods and a plan for rotation. The biggest gains often come from reducing the number and lifetime of secrets: workload identity instead of static keys, short-lived credentials instead of permanent ones, and an external manager as the source of truth.
Treat secrets management as a lifecycle, from creation through delivery and use to rotation and revocation. If you build each stage deliberately, with the controls in this article, sensitive configuration stays out of the wrong hands, and an incident becomes a routine rotation rather than a crisis.