Secrets management beyond Kubernetes Secrets

The most dangerous misconception in a new Kubernetes setup is that a Secret is secret. It is not. Kubernetes Secrets are base64-encoded strings stored in etcd — encoding, not encryption. Anyone with kubectl get secret -o yaml access, a raw etcd backup, or read permission on the API server's underlying storage can recover every credential in seconds. That gap between "it's in a Secret" and "it's actually protected" is where breaches happen — and the breach statistics are stark. GitGuardian's State of Secrets Sprawl 2024 found 23.77 million new hardcoded secrets in public GitHub repositories in a single year, up 25% year-over-year. Verizon's 2025 DBIR identified credential abuse as the single most common initial access vector, responsible for 22% of all breaches. And 64% of secrets older than four years remain valid — meaning organisations detected a leak, maybe filed a ticket, and never rotated. The median remediation time for a GitHub-leaked secret is 94 days; by that point the secret has been indexed, discovered, and almost certainly used.

Production secrets management is not one tool. It is a set of layered controls: encrypt etcd, remove credential values from git and images, centralise sensitive secrets behind an audited manager, choose the right injection pattern for each workload, and build rotation so it propagates without a redeploy. This post walks all of it, with the concrete configuration that makes each layer actually work in a real cluster.

The etcd problem: encryption at rest is table stakes

By default, Kubernetes writes Secrets to etcd in plaintext — the base64 encoding is a client-side presentation detail, not a storage transformation. An attacker with a cold etcd backup recovers every credential in the cluster without needing a live API server. Enabling encryption at rest fixes this, and it is the first control every cluster should have regardless of everything else.

The mechanism is an EncryptionConfiguration manifest mounted to the kube-apiserver process:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

The identity: {} provider at the end lets the API server read existing unencrypted Secrets during migration. Once you have re-written all Secrets through the API (forcing re-encryption), remove the identity provider. Do the rewrite with:

kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

The managed cloud option is the better default. On EKS, GKE, and AKS you hand encryption key management to your cloud KMS (AWS KMS, Google Cloud KMS, Azure Key Vault) using envelope encryption: the data encryption key (DEK) is itself encrypted by a master key in the cloud KMS. Losing the cluster does not expose data unless the attacker also holds KMS access. On EKS this is a single flag at cluster creation:

aws eks create-cluster \
  --name prod-cluster \
  --region us-east-1 \
  --encryption-config '[{
    "resources": ["secrets"],
    "provider": {
      "keyArn": "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123"
    }
  }]'

On GKE the equivalent is Customer-Managed Encryption Keys (CMEK) via --database-encryption-key at cluster creation. On AKS it is the --enable-encryption-at-host flag combined with a Key Vault key URI on the node pool.

Getting secrets out of git and images

Credentials committed to a repository are effectively permanent, even if you rewrite history. Every developer who cloned the repo has a copy. Every CI runner that fetched the branch has a copy. GitHub's own search index has a copy within minutes of a push. GitGuardian found that 43% of cloud-related secrets found in public repos were Google Cloud API keys — an attacker with that key can enumerate cloud resources before the team even notices the exposure. Baking secrets into container image layers is equally dangerous: every pull of the image exposes every layer's metadata, and image registries often have broader read access than clusters do.

Two mature patterns get credential values entirely out of git:

Sealed Secrets

Bitnami's Sealed Secrets operator runs a controller in your cluster that holds a private key. You encrypt a Kubernetes Secret with the matching public key using the kubeseal CLI. The resulting SealedSecret CRD is safe to commit; only the in-cluster controller can decrypt it.

# Generate a dry-run Secret and pipe directly through kubeseal
kubectl create secret generic db-password \
  --from-literal=password='s3cr3t' \
  --dry-run=client -o yaml | \
  kubeseal \
    --controller-namespace kube-system \
    --controller-name sealed-secrets \
    --format yaml > sealed-db-password.yaml

The output YAML is safe in git. When applied to the cluster, the controller decrypts and creates the real Secret in the target namespace. The approach integrates cleanly with Argo CD and Flux because the SealedSecret manifest is just another git object.

Back up the controller's private key before you need it:

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-controller-key-backup.yaml

Store this backup in a vault or a separate secure S3 bucket — not in the same git repo you just sealed.

External Secrets Operator

External Secrets Operator (ESO) is a CNCF project that reads from an external secret store and synchronises values into Kubernetes Secrets at runtime. Your git repo stores an ExternalSecret resource — a reference, not a value. The actual credential lives in AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, or one of 40+ other supported backends.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: payments
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password
    - secretKey: username
      remoteRef:
        key: prod/payments/db
        property: username

ESO re-syncs on refreshInterval, so a rotation in the backing store propagates automatically. One critical nuance: that propagation only reaches the pod if the credential is mounted as a file volume — environment variable bindings capture the Secret's value at container start and do not update until the pod restarts. More on this in the injection patterns section.

A real secrets manager for the sensitive stuff

For database credentials, signing keys, OAuth client secrets, and anything that touches a compliance boundary (PCI-DSS, SOC 2, HIPAA), a dedicated secrets manager is not optional — it is how you produce the audit trail that shows an assessor exactly who accessed what and when. AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and HashiCorp Vault are all mature options for static secret storage. The differentiator between Vault and the cloud-native managers is dynamic secrets, which Vault does best and nothing else matches.

Dynamic secrets: the only architectural shift that actually reduces breach impact

A dynamic secret is generated on demand for a specific requestor, given a short time-to-live (TTL), and automatically revoked when the lease expires. Compare the risk profile:

| Credential type | Exposure window | Blast radius on compromise | |---|---|---| | Long-lived static password | Until manually rotated (often years) | Every service that ever held it | | Dynamic secret, 1-hour TTL | 60 minutes | One lease for one service instance | | Dynamic secret, pod identity, 15-minute TTL | 15 minutes | One pod's lifetime |

Vault's database secrets engine generates PostgreSQL, MySQL, MongoDB, and other credentials via a privileged admin connection. When a pod requests credentials, Vault creates a real database user, sets a TTL on the lease, and returns the ephemeral username and password. When the lease expires or is revoked, that user is dropped from the database. There is no credential to rotate because none persists.

Enable the database secrets engine and configure a role for a PostgreSQL backend:

vault secrets enable database
 
vault write database/config/payments-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="payments-app" \
  connection_url="postgresql://{{username}}:{{password}}@db.payments.svc:5432/payments" \
  username="vault-admin" \
  password="$(vault kv get -field=password kv/db-admin)"
 
vault write database/roles/payments-app \
  db_name=payments-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN \
    PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public \
    TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="4h"

Any service authenticating to Vault with the payments-app role gets a unique database credential valid for one hour. The admin password never leaves Vault.

Injection patterns: how secrets reach the pod

How you get a secret value into a container is as consequential as where you store it. Three patterns exist, with meaningfully different security and operational properties.

Environment variables — avoid for sensitive secrets

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Convenient, but environment variables are visible in the process environment file under /proc/PID/environ, in application crash dumps, in kubectl describe pod output when manifests are logged by audit systems, and to every library and agent running in the same process. For non-sensitive config (feature flags, service URLs) they are fine. For database passwords and signing keys, they are an unnecessary surface. They also do not update on Secret rotation — the value is captured at container startup and frozen there until the pod restarts.

Volume mounts — the right default for rotatable secrets

volumes:
  - name: db-secret
    secret:
      secretName: db-credentials
      defaultMode: 0400
containers:
  - name: app
    volumeMounts:
      - name: db-secret
        mountPath: /run/secrets/db
        readOnly: true

The credential is a file, readable only by a process that explicitly opens it. The kubelet re-syncs Secret-backed volumes within its syncFrequency period (default 60 seconds), so when an ESO refresh updates the underlying Secret object, the volume-mounted file updates automatically — no pod restart required. This is the only injection mechanism that gives you zero-downtime rotation propagation.

CSI Secret Store Driver — cleanest for compliance workloads

The Secrets Store CSI Driver fetches secrets directly from the external store at pod mount time and writes them to an in-memory tmpfs volume. The values never touch the Kubernetes Secret API and never sit in etcd. Combined with the Vault CSI provider:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: payments
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.infra.svc.cluster.local:8200"
    roleName: "payments-app"
    objects: |
      - objectName: "db-password"
        secretPath: "database/creds/payments-app"
        secretKey: "password"

The secret exists in memory on the node for the pod's lifetime and is gone when the pod terminates. For regulated workloads, removing etcd as a persistence layer for credential values is often worth the additional operational overhead. The trade-off: there is no persisted Kubernetes Secret to query for debugging, so emergency access patterns need to account for this.

Rotation: making it actually work

The hard part of rotation is not generating a new credential — it is swapping the old one for the new one without returning errors to users. Every rotation procedure must explicitly answer: how does a running service transition from the old value to the new one while both need to be valid simultaneously?

The pattern that works without downtime for file-mounted secrets:

1. Generate the new credential alongside the old one. Do not revoke first. On AWS Secrets Manager this is the AWSPENDING staging label. On GCP Secret Manager this is a new version left in DISABLED state until tested. In Vault, the old lease stays active while you test the new one.
2. Publish both versions to the store using the versioning mechanism the store provides.
3. Wait for propagation: ESO refreshInterval + kubelet syncFrequency (up to ~2 minutes total for volume mounts). For a refreshInterval: 15m and a syncFrequency: 60s, budget 16 minutes for the new value to reach every pod.
4. Verify the new credential is working in production (database session monitoring, application health checks, audit log confirmation).
5. Revoke the old credential only after confirming no active session is using it.
6. Remove the old version from the store.

AWS Secrets Manager automates this five-step process for RDS and other managed services via Lambda rotation functions. The configuration is declarative:

{
  "SecretId": "prod/payments/db",
  "RotationLambdaARN": "arn:aws:lambda:us-east-1:123456789012:function:payments-db-rotator",
  "RotationRules": {
    "AutomaticallyAfterDays": 30,
    "Duration": "2h"
  }
}

The Duration field is the overlap window — both credentials are valid during that two-hour period. AWS's reference Lambda implementations handle the four required steps (createSecret, setSecret, testSecret, finishSecret) for RDS, Redshift, and DocumentDB. For other databases you implement the same four functions against the Secrets Manager SDK.

Rotation without a restart — the migration path

If your services bind database passwords as environment variables today, rotation forces a rolling restart. The migration path:

1. Move the Secret binding from env.valueFrom.secretKeyRef to a volumeMount at /run/secrets/db/password.
2. Update your connection pool configuration to read the password from the file path rather than an environment variable. In Go, that is a direct os.ReadFile on reconnect. In Java with HikariCP, implement a custom DataSourceProvider that re-reads the file at reconnect time. Most modern connection pools support this pattern.
3. Set ESO refreshInterval to match your rotation schedule (e.g., refreshInterval: 12h for a 30-day rotation schedule with comfortable propagation margin).

After the migration, a credential rotates in the backing store, ESO syncs it to the Secret object, the kubelet updates the volume-mounted file within the sync window, and the connection pool picks up the new password on its next reconnect cycle — no pod restart, no traffic disruption.

Real-world pitfalls and fixes

These are failure modes that appear repeatedly when teams move from "no secrets management" to "we have Vault."

1. Vault lease accumulation hitting the ceiling

A Vault Agent sidecar is configured with lease renewal. Each deployment of the service creates new pods that obtain new Vault leases. Old leases from terminated pods are not explicitly revoked. Over weeks, the number of active leases grows toward Vault's max_lease_count (default 300,000 on HCP Vault). When the limit is hit, Vault begins rejecting new lease requests.

Fix: add a preStop lifecycle hook that calls vault lease revoke -prefix auth/kubernetes/cluster/login/ before the pod terminates. Separately, tune default_lease_ttl to be short enough that stale leases expire quickly: vault write sys/mounts/database/tune default_lease_ttl=1h max_lease_ttl=4h.

2. ESO refresh does not update env-var-bound secrets

A team sets ESO refreshInterval: 5m expecting that credential rotation propagates within five minutes. It does — to the Kubernetes Secret object. But the pods bind the credential as an environment variable. The env is set at container start and does not update until the pod restarts. The secret rotates in the store, ESO syncs it, the Secret object is current, but every running pod still uses the old value.

Fix: switch to volumeMount, or accept that env-var-bound secrets require an explicit rolling restart and schedule it as part of the rotation procedure.

3. Sealed Secrets controller key lost on cluster rebuild

A cluster is rebuilt after an incident. The team has all SealedSecret manifests in git but did not back up the controller's private key. Every sealed secret is permanently unreadable.

Fix: run kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > controller-key-backup.yaml and store this in a vault or secure object storage bucket, not in the same repository. Verify the backup exists before any cluster maintenance window.

4. Kubernetes RBAC with namespace-wide secret read permissions

A service account has a Role granting get on secrets in a namespace — a common default applied carelessly. A compromised pod with that service account can read every credential in the namespace, not just its own.

Fix: scope RBAC grants with resourceNames to the specific Secret the pod needs:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-secret-reader
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials", "signing-key"]
    verbs: ["get"]

If you use the CSI Secret Store Driver, you can remove this RBAC grant entirely — the driver fetches directly from Vault and creates no Kubernetes Secret object for the pod to read via the API.

5. Secrets committed during a debug session

A .env.local file or a kubeconfig with a CI service account token gets committed during a late-night debugging session. GitGuardian flags it within minutes of the push — but the git history is already distributed to every developer and CI runner with a clone. The token is searchable on GitHub within the hour.

Fix: a pre-commit hook using detect-secrets or trufflehog blocks the commit before it happens:

pip install detect-secrets
detect-secrets scan > .secrets.baseline

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

Rotate any exposed token regardless of whether the pre-commit hook would have caught it — assume the value was compromised the moment it touched the filesystem.

Decision table: matching the architecture to the constraint

| Constraint | Recommended approach | |---|---| | Single cluster, GitOps-first, no compliance audit | Sealed Secrets + etcd KMS encryption | | Multi-cluster or multi-cloud | External Secrets Operator + cloud-native store (AWS SM, GCP SM, Azure KV) | | Regulated workload (PCI-DSS, SOC 2, HIPAA) | HashiCorp Vault with audit logging + ESO for Kubernetes sync | | Database credentials and signing keys | Vault dynamic secrets engine, TTL of 1 hour or less | | Static config — feature flags, non-sensitive service URLs | Kubernetes ConfigMap — no secret management overhead needed | | Zero-downtime rotation | Volume mounts + ESO refresh; migrate away from env-var bindings for any rotatable credential | | Air-gapped or no cloud dependency | Self-hosted Vault (Community Edition) or Bitnami Sealed Secrets | | Compliance audit trail required | Vault audit backend (file or socket) or AWS CloudTrail on Secrets Manager |