Kubernetes Secrets: Best Practices for Production

What and Why

A Kubernetes Secret is an API object meant for small pieces of sensitive data: passwords, tokens, TLS keys. Pods consume them as environment variables or mounted files. Secrets are first-class objects, which makes them easy to manage with kubectl, but the default storage is just base64-encoded data inside etcd. That is encoding, not encryption, and it is a frequent source of confusion.

Treating Secrets as truly secret takes a few extra steps: encrypting etcd, scoping RBAC, choosing the right consumption mode, and integrating with an external secret store for rotation. Get these right and your platform earns the right to host production credentials.

Mental Model

Think of a Secret as a “labeled envelope” in etcd. Anyone with read access to that envelope can open it. The envelope itself is not locked unless you turn on encryption at rest. Anyone who can exec into a Pod that mounts the Secret can also read it from inside the container. So security really comes down to two questions: who can read the etcd object, and who can read the running Pod.

External secret managers like AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager sit one layer up. A controller syncs values into Kubernetes Secrets on demand or projects them directly into Pods, giving you central rotation and audit while keeping the Pod-side experience simple.

Hands-on Example

A minimal Secret and a Pod that consumes it as a file rather than an environment variable.

apiVersion: v1
kind: Secret
metadata:
  name: stripe-keys
  namespace: payments
type: Opaque
stringData:
  api-key: "sk_live_REPLACE_ME"
  webhook-secret: "whsec_REPLACE_ME"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      automountServiceAccountToken: false
      containers:
        - name: app
          image: registry.example.com/checkout:3.1.0
          volumeMounts:
            - name: stripe
              mountPath: /var/run/secrets/stripe
              readOnly: true
      volumes:
        - name: stripe
          secret:
            secretName: stripe-keys
            defaultMode: 0400

Mounted Secrets update automatically when the source changes, usually within a minute. Environment variables do not; they require a Pod restart. Prefer file mounts when you want hot rotation.

 +-----------------+        sync         +-------------------+
 |  Vault / AWS    | ------------------> | ExternalSecret    |
 |  Secret Store   |                     | Controller        |
 +-----------------+                     +-------------------+
                                                 |
                                                 v
                                        +-------------------+
                                        | K8s Secret object |
                                        | (etcd, encrypted) |
                                        +-------------------+
                                                 |
                                                 v
                                        +-------------------+
                                        | Pod volumeMount   |
                                        | /var/run/secrets  |
                                        +-------------------+

Common Pitfalls

Checking Secrets into Git as plain YAML is the classic mistake. Even sealed-secrets or SOPS-encrypted files belong in a separate, restricted repo with strict review.

Logging Secret values is the second. A single console.log(process.env) in a Node app dumps every secret to your log aggregator, where retention policies almost never match your rotation policies.

Using Secrets as environment variables on a busy node is risky because any process that can read /proc/<pid>/environ can read them. File mounts with defaultMode: 0400 and a non-root user are safer.

Finally, forgetting RBAC. A ServiceAccount in default with secrets/get across the cluster is functionally an admin credential. Scope Secret access per namespace and prefer get over list.

Production Tips

Enable etcd encryption at rest with an EncryptionConfiguration resource on the API server, using a KMS provider rather than a static key. This is the single highest-impact change for most clusters.

Adopt an external secrets operator (External Secrets Operator or Vault Agent Injector) so the source of truth lives outside the cluster. Rotations become a one-line update in Vault, not a multi-cluster rollout.

Use short-lived credentials wherever possible: IRSA on AWS, Workload Identity on GCP, and projected service account tokens for in-cluster auth. They make Secret leakage far less catastrophic because the blast radius is measured in minutes.

Audit kubectl get secret -A access regularly. Most clusters have ten times more readers than they need.

Wrap-up

Kubernetes Secrets are convenient but not magic. Encrypt etcd, scope RBAC tightly, prefer file mounts with restrictive modes, and integrate an external secret store for rotation. With those four habits, your secrets are genuinely secret.