kubernetes

Kubernetes Secrets Explained: Managing Sensitive Configuration

Kubernetes Secrets

By Shubhankar Tripathi • • 5 min read

Kubernetes Secrets Explained: Managing Sensitive Configuration

bitcodematrix.com | Kubernetes Series

1. Introduction

Every real application needs credentials: database passwords, API tokens, TLS private keys, registry logins. Putting them in container images or in ConfigMaps is dangerous, because both are easy to copy and read. Kubernetes offers a dedicated object for this purpose, the Secret, together with mechanisms to deliver sensitive values to Pods and to control who can see them.

There is, however, a widespread misunderstanding about Secrets: many people assume that a Secret is encrypted by default. It is not. A default Secret is only base64-encoded, and the security of the whole design depends on additional measures such as encryption at rest, tight RBAC and careful delivery. This article explains how Secrets work, what protects them and what does not, how to create and consume them, how to rotate them, and how to integrate external secret managers. It complements the earlier articles on ConfigMaps and on emptyDir, ConfigMap and Secret volumes.

2. What Is a Secret?

A Secret is a namespaced API object that holds a small amount of sensitive data, as key-value pairs. Structurally it resembles a ConfigMap, but Kubernetes treats it differently in several ways:

  • Values in the API are stored and returned base64-encoded. This is a transport encoding for arbitrary bytes, not a protection.

  • When mounted as a volume, a Secret is kept in memory (tmpfs) on the node, so it is not written to the node's disk.

  • The kubelet receives only the Secrets that are used by Pods scheduled to its own node, because of the Node authorizer.

  • Secrets can be encrypted in etcd when the API server is configured to do so.

  • Tooling treats them with more care: kubectl describe shows only the sizes of values, and audit policies can avoid logging their content.

Like ConfigMaps, a Secret is limited to about 1 MiB and can be referenced only from Pods in the same namespace.

2.1 The Honest Security Model

It helps to be precise about what a Secret does and does not give you:

What Secrets help with

What Secrets do not do by themselves

Keeping credentials out of images and manifests of Pods

Encrypt data at rest. This needs explicit configuration

Separate access control from ConfigMaps

Stop people with read access from seeing the values

Avoiding persistence on the node's disk when mounted

Protect values placed in environment variables from being exposed in diagnostics

Limiting distribution to the nodes that need them

Rotate credentials or manage their lifecycle

 

In short, a Secret is a building block. Security comes from the combination of encryption, access control, delivery method, rotation and good habits around it.

3. Secret Types

The type field of a Secret declares its purpose, and Kubernetes validates that the expected keys are present for built-in types.

Type

Purpose and expected keys

Opaque (default)

Arbitrary user-defined data; no validation of keys

kubernetes.io/tls

A TLS certificate and key, with the keys tls.crt and tls.key

kubernetes.io/dockerconfigjson

Credentials for pulling images from a private registry, stored under the key .dockerconfigjson

kubernetes.io/basic-auth

Basic authentication credentials, with the keys username and password

kubernetes.io/ssh-auth

An SSH private key, with the key ssh-privatekey

kubernetes.io/service-account-token

A legacy long-lived service account token, populated by the control plane

bootstrap.kubernetes.io/token

Bootstrap tokens used when nodes join a cluster

 

Using the specific type, for example kubernetes.io/tls for certificates, gives early validation and lets other tools, such as ingress controllers and Gateway implementations, find the data in the format that they expect.

4. Creating Secrets

4.1 With kubectl

kubectl create secret generic db-creds -n shop \

  --from-literal=username=app \

  --from-literal=password='S3cr3t-value'

 

kubectl create secret generic api-key -n shop \

  --from-file=api-key=./api-key.txt

 

kubectl create secret tls web-tls -n shop \

  --cert=tls.crt --key=tls.key

 

kubectl create secret docker-registry regcred -n shop \

  --docker-server=registry.example.com \

  --docker-username=ci-bot \

  --docker-password='<token>'

Be aware that values typed on the command line can end up in your shell history and in process listings. For real credentials, prefer reading from a file with restricted permissions, or from your secret manager, rather than typing them as literals.

4.2 With a Manifest

apiVersion: v1

kind: Secret

metadata:

  name: db-creds

  namespace: shop

type: Opaque

stringData:

  username: app

  password: S3cr3t-value

A manifest can use either of two fields. With data, the values must already be base64-encoded. With stringData, you write plain text, and the API server encodes it when storing the object. The stringData field is write-only: when you read the Secret back, you see only data. If both fields contain the same key, stringData wins.

4.3 Common Creation Mistakes

  • Trailing newline. Encoding with echo instead of echo -n adds a newline to the value, which breaks logins in subtle ways. Use echo -n, or stringData.

  • Double encoding. Putting a base64 value in stringData, or encoding twice, results in the application receiving encoded text.

  • Plain manifests in Git. A YAML file with a Secret is only base64-encoded, and committing it exposes the value. See Section 11.

  • Wrong namespace. The Secret must exist in the Pod's namespace.

To read a value back for troubleshooting, decode it, preferably in a protected terminal:

kubectl get secret db-creds -n shop -o jsonpath='{.data.password}' | base64 -d

5. Consuming Secrets

A Pod can use a Secret as environment variables, as files in a volume, or as image pull credentials. The Deployment fragment below shows all three:

spec:

  template:

    spec:

      imagePullSecrets:

      - name: regcred

      automountServiceAccountToken: false

      containers:

      - name: app

        image: registry.example.com/shop/app:1.0

        env:

        - name: DB_USER

          valueFrom:

            secretKeyRef:

              name: db-creds

              key: username

        volumeMounts:

        - name: db-creds

          mountPath: /etc/secrets

          readOnly: true

      volumes:

      - name: db-creds

        secret:

          secretName: db-creds

          defaultMode: 0400

5.1 Environment Variables

The secretKeyRef field injects a single key, and envFrom with secretRef injects all keys. They are convenient and need no application changes, but they carry real risks. Environment variables are inherited by child processes, can appear in crash dumps and diagnostic endpoints, and are visible to anyone who can inspect the container on the node. They are also fixed at start-up and never update.

5.2 Files in a Volume

A secret volume turns each key into a read-only file, held in memory on the node. You control permissions with defaultMode or a per-file mode, and the files are refreshed when the Secret changes, except in subPath mounts. For most sensitive data, files are the better choice, because file permissions provide a basic control, and rotation without a restart becomes possible if the application re-reads the file.

5.3 Image Pull Secrets

The imagePullSecrets field in a Pod, or in a ServiceAccount used by Pods, tells the kubelet which Secret of type kubernetes.io/dockerconfigjson to use when pulling images. Attaching it to the ServiceAccount avoids repeating it in every manifest.

5.4 Reducing Exposure

  • Mount only the Secrets and keys each container needs.

  • Use readOnly mounts and restrictive file modes, and ensure that the process user can read the files. If you set fsGroup, check the resulting permissions.

  • Never place secret values in command-line arguments, because they appear in process listings and in the Pod specification.

  • Set automountServiceAccountToken to false for Pods that do not need to call the Kubernetes API, so that no credential is mounted needlessly.

Aspect

Environment variable

Mounted file

Exposure in diagnostics

Higher

Lower, controlled by file permissions

Updates in running Pods

No

Yes, with a delay, except subPath

Application changes

None

Application reads from a path

Stored on node disk

Not as a file, but present in the container configuration

No; held in tmpfs

Recommended for sensitive data

Only when unavoidable

Yes

 

6. Where Secrets Live and Who Can See Them

Follow a Secret from creation to use, and you see several places where it exists, each with its own protection:

Location

Exposure

Protection

Your workstation and pipeline

Command history, files, CI logs, Git repositories

Do not commit; mask in CI; use secret managers

The API server and etcd

Stored base64-encoded unless encryption at rest is enabled; also in etcd backups

Encryption at rest, access control for etcd and backups

The network

Between clients, API server, etcd and kubelets

TLS, which Kubernetes components use by default

The kubelet and the node

In memory as a tmpfs when mounted; in container configuration when used as environment variables

Node authorizer, node hardening, restricted node access

The running container

Visible to the application and to anyone who can exec into it

RBAC for exec, least privilege, short-lived credentials

 

The takeaway is that encryption at rest and RBAC are the two controls that you must configure yourself, while the rest is provided or encouraged by default.

7. Encryption at Rest

By default, the API server stores Secrets in etcd without encryption. Anyone who can read etcd, or a backup of it, can read every Secret. To prevent this, configure encryption at rest on the API server with an EncryptionConfiguration file, passed through the --encryption-provider-config flag:

apiVersion: apiserver.config.k8s.io/v1

kind: EncryptionConfiguration

resources:

- resources:

  - secrets

  providers:

  - aescbc:

      keys:

      - name: key1

        secret: <base64-encoded 32-byte key>

  - identity: {}

The order of providers matters. The first provider is used to encrypt new writes, and all listed providers can be used to decrypt. The identity provider means "no encryption", and placing it last lets the API server read objects that were stored before encryption was enabled.

7.1 Local Keys or a KMS

Providers such as aescbc and secretbox use keys that sit in the configuration file on the control plane nodes, so anyone who obtains that file can decrypt the data. The stronger approach is a KMS provider, which uses envelope encryption: each object is encrypted with a data key, and that key is encrypted by a key that never leaves an external key management service. Version 2 of the KMS provider API is the recommended one.

providers:

- kms:

    apiVersion: v2

    name: my-kms

    endpoint: unix:///var/run/kmsplugin/socket.sock

    timeout: 3s

- identity: {}

On managed Kubernetes services, you typically enable envelope encryption through a cluster setting that references a key in the provider's key service, and the provider handles the API server configuration.

7.2 Existing Secrets and Verification

Enabling encryption affects only objects written afterwards. To encrypt Secrets that already exist, rewrite them:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

On clusters where you control etcd, verify the result by reading a Secret directly from etcd and checking that the stored value starts with a provider prefix, such as k8s:enc:aescbc:v1:key1, and not with readable text. Plan for key rotation as well: add the new key, make it the first entry, restart the API servers, and re-encrypt all Secrets before removing the old key. Always back up your keys and your configuration, because losing them makes the data unrecoverable.

8. Access Control

8.1 Understand What Grants Access

Several permissions expose Secrets, and some of them are not obvious:

  • get, list and watch on secrets. The list and watch verbs return the full data of every Secret in scope. Treat them as equivalent to read access to all values.

  • Creating Pods in a namespace. Anyone who can create a Pod can mount any Secret in that namespace, or use it as an environment variable, and read it from inside the container.

  • exec into Pods. Anyone who can run commands in a container can read its mounted Secrets and environment.

  • Access to etcd, backups or node file systems. These bypass Kubernetes RBAC entirely.

8.2 Grant the Minimum

Give workloads and people only the access that they need. A Role that allows reading one named Secret looks like this:

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

  name: read-db-creds

  namespace: shop

rules:

- apiGroups: [""]

  resources: ["secrets"]

  resourceNames: ["db-creds"]

  verbs: ["get"]

Note that resourceNames restricts the get verb to the named object, but granting list or watch on secrets, even with resource names, can still expose more than intended, so avoid those verbs for Secrets. Avoid cluster-wide roles that include secrets, and review bindings regularly, including the permissions of controllers and operators that you install.

8.3 Other Controls

  • Namespaces as boundaries. Separate teams and environments into namespaces, and keep their Secrets apart.

  • ServiceAccount hygiene. Use a dedicated ServiceAccount per workload, and disable token auto-mounting where it is not needed.

  • Audit logging. Configure the audit policy to record Secret access at the Metadata level, so that you know who touched which Secret without writing the values into the audit log.

  • Policy enforcement. Use admission policies to block risky patterns, such as Secrets passed as plain environment variables where files are required, or manifests that contain obvious credentials.

  • Immutability. Setting immutable to true on a Secret prevents in-place changes and reduces watch load, with the same versioning pattern as for ConfigMaps.

9. Rotation and Updates

Credentials should change over time, whether on a schedule or after a suspected leak. Kubernetes makes rotation possible, but you must design for it.

Consumption method

Effect of changing the Secret

Mounted volume

Files are refreshed with a delay of up to a minute or two. The application must re-read them

Mounted volume with subPath

Never refreshed; needs a restart

Environment variable

Never refreshed; needs a Pod restart

Immutable Secret

Cannot change; create a new version and roll out

 

A safe rotation usually follows an overlap pattern, because systems rarely switch instantly:

  • Create the new credential at the source (the database, the API provider), while the old one still works.

  • Update the Secret, or create a new versioned Secret, and roll it out to the workloads.

  • Confirm that all consumers use the new credential.

  • Revoke the old credential at the source.

Where possible, move away from long-lived static credentials altogether. Short-lived, dynamically issued credentials, from a secrets manager or from workload identity, shrink the window of exposure, and make rotation automatic rather than an event.

10. Service Account Tokens and Workload Identity

Pods that call the Kubernetes API, or external services that trust the cluster, need an identity. Older versions of Kubernetes created a long-lived token Secret for every ServiceAccount. Current versions no longer create these automatically. Instead, Pods receive projected, short-lived, audience-bound tokens that the kubelet refreshes, and that are invalidated when the Pod is deleted. You can also request a token explicitly:

kubectl create token my-serviceaccount -n shop --duration=1h

Create a long-lived token Secret only when an external system cannot use the short-lived mechanisms, and treat it as a high-value credential. For access to cloud services, use the platform's workload identity features, which let a ServiceAccount assume a cloud role without static keys stored in a Secret. This removes an entire category of credentials from your cluster.

11. Secrets in GitOps and CI/CD

A declarative workflow wants every object in Git, but a plain Secret manifest in Git is a leaked secret. There are three common solutions:

Approach

How it works

Considerations

Encrypted secrets in Git

Tools such as Sealed Secrets or SOPS store encrypted values in the repository; a controller or the pipeline decrypts them for the cluster

Key management for the decryption key is critical; rotation requires re-encrypting

External references in Git

Git holds only a reference to a secret in a manager, and an operator in the cluster fetches the value

Values never appear in Git; depends on the operator and the manager

Secrets created by the pipeline

The CI/CD system injects the value into the cluster at deploy time from its own secret store

Simple, but the pipeline becomes a high-value target, and the cluster state is not fully in Git

 

Whichever you choose, keep Secrets out of CI logs by using masking, avoid printing environment variables during builds, and scan repositories and images for accidentally committed credentials.

12. External Secret Managers

For production workloads, many organisations keep the source of truth for secrets in a dedicated manager, such as a cloud secrets service or a vault product, and integrate it with Kubernetes. Three integration patterns are common:

Pattern

How it works

Trade-offs

Sync into Kubernetes Secrets

An operator reads values from the manager and creates ordinary Kubernetes Secrets

Works with every application, but copies of the values live in etcd and need the usual protection

Mount through a CSI driver

The Secrets Store CSI driver mounts values from the manager directly as files in the Pod, with an optional sync to a Secret

Values need not live in etcd when you avoid the sync, but the Pod depends on the driver and the manager at start-up

Agent or sidecar injection

A sidecar or init container authenticates to the manager and writes secrets into a shared in-memory volume

Supports dynamic, short-lived secrets and leases, with more moving parts

 

These integrations bring auditing, central rotation, fine-grained policies and often dynamic credentials. They also add dependencies, so plan for what happens when the manager is unreachable: decide whether Pods may start with cached values and how long, and test it.

13. Hands-On Walkthrough

  1. Create a Secret from literals.

kubectl create secret generic demo-creds -n shop \

  --from-literal=username=demo \

  --from-literal=password='first-password'

  1. Look at it, and notice that describe hides the values while get shows them encoded.

kubectl describe secret demo-creds -n shop

kubectl get secret demo-creds -n shop -o yaml

kubectl get secret demo-creds -n shop -o jsonpath='{.data.password}' | base64 -d

  1. Mount it in a Pod as a volume and read it.

apiVersion: v1

kind: Pod

metadata:

  name: secret-demo

  namespace: shop

spec:

  containers:

  - name: app

    image: busybox:1.36

    command: ["sh", "-c", "sleep 3600"]

    volumeMounts:

    - name: creds

      mountPath: /etc/creds

      readOnly: true

  volumes:

  - name: creds

    secret:

      secretName: demo-creds

      defaultMode: 0400

kubectl exec secret-demo -n shop -- cat /etc/creds/password

  1. Rotate the value and wait a minute or two, then read the file again.

kubectl patch secret demo-creds -n shop \

  -p '{"stringData":{"password":"second-password"}}'

kubectl exec secret-demo -n shop -- cat /etc/creds/password

The mounted file shows the new value after the refresh, which shows that rotation without a restart is possible for files. If the value had been injected as an environment variable, it would still show the old password.

  1. Clean up.

kubectl delete pod secret-demo -n shop

kubectl delete secret demo-creds -n shop

14. When a Secret Leaks

Assume that sooner or later a credential will be exposed: committed to a repository, printed in a log, or visible to someone who should not have seen it. A prepared response matters more than perfect prevention.

  1. Rotate at the source first. Revoke or change the credential in the system that it protects, such as the database or the API provider. Changing the Kubernetes Secret alone does not invalidate the leaked value.

  2. Deploy the new credential to the workloads using the overlap pattern, and verify that they work.

  3. Find the exposure. Check Git history, CI logs, container images, audit logs and backups, and determine who and what could have used the value.

  4. Remove copies. Purge the value from repositories and logs where possible, recognising that history rewriting does not undo exposure that has already happened.

  5. Check for misuse. Review access logs at the target system for the period of exposure.

  6. Fix the cause. Adjust RBAC, pipelines or habits, add scanning and policy controls, and record the lesson.

15. Troubleshooting

Symptom

Likely cause and check

Pod in CreateContainerConfigError

A secretKeyRef points to a Secret or key that does not exist. Check the name, key and namespace, or mark the reference optional

Pod in ContainerCreating with a secret not found mount error

A secret volume refers to a missing Secret. Create it in the Pod's namespace

ImagePullBackOff with authentication errors

The image pull Secret is missing, in the wrong namespace, not referenced, or contains wrong or expired credentials. Verify with a manual pull using the same credentials

Login fails with a correct-looking value

A trailing newline or double encoding. Decode the value and inspect it, then recreate it with echo -n or stringData

Application cannot read the mounted file

defaultMode too strict for the process user, or fsGroup changed permissions. Review runAsUser, fsGroup and file mode

New value not visible in the Pod

The Secret is used as an environment variable or via subPath, is immutable, or the refresh delay has not passed. Restart the Pods or wait

TLS errors after updating a certificate Secret

Certificate and key do not match, a chain is incomplete, or the application did not reload. Verify the pair and restart or reload

Forbidden error reading a Secret

RBAC does not allow get for that Secret, or the wrong ServiceAccount is used. Check the Role, RoleBinding and the Pod's ServiceAccount

Secrets unreadable after an API server change

The encryption configuration or key was lost or changed incorrectly. Restore the previous configuration and keys from backup immediately

Object rejected as too large

The Secret exceeds about 1 MiB. Store large files elsewhere and keep only references

Secret value visible somewhere unexpected

It was used in arguments, logs or environment diagnostics. Move it to a file mount, fix the logging, and rotate the credential

 

16. Best Practices

  • Enable encryption at rest for Secrets, preferably with a KMS provider, and protect etcd and its backups.

  • Apply least-privilege RBAC, avoid list and watch on Secrets, and remember that creating Pods or exec access also exposes them.

  • Deliver Secrets as read-only files with restrictive permissions, and avoid environment variables and command-line arguments for sensitive values.

  • Disable ServiceAccount token auto-mounting where the Pod does not call the API, and use projected short-lived tokens and workload identity instead of static keys.

  • Never commit plain Secret manifests, and choose encrypted-in-Git, external references or pipeline injection for GitOps.

  • Prefer an external secrets manager for production, with central auditing and dynamic, short-lived credentials where possible.

  • Design for rotation: use overlap, files that can be re-read, and rollouts tied to Secret changes, and rotate on a schedule.

  • Use separate Secrets per application and environment, and use the specific Secret types for TLS and registry credentials.

  • Turn on audit logging for Secret access at the Metadata level, and alert on unusual reads.

  • Keep a leak-response runbook, and practise it, including rotation at the source.

  • Scan repositories, images and CI logs for credentials, and enforce policies through admission control.

17. Conclusion

Kubernetes Secrets are a necessary part of running real applications, but they are not magic. A default Secret is base64-encoded text with a few helpful handling rules; it becomes truly protected only when you add encryption at rest, narrow RBAC, careful delivery to Pods and a plan for rotation. The biggest gains often come from reducing the number and lifetime of secrets: workload identity instead of static keys, short-lived credentials instead of permanent ones, and an external manager as the source of truth.

Treat secrets management as a lifecycle, from creation through delivery and use to rotation and revocation. If you build each stage deliberately, with the controls in this article, sensitive configuration stays out of the wrong hands, and an incident becomes a routine rotation rather than a crisis.