What Are Kubernetes Secrets Best Practices?

Detailed Answer

How Kubernetes Secrets Work

A Secret is a namespaced Kubernetes object that holds key-value pairs of sensitive data. Secrets can be mounted as files in a volume or exposed as environment variables.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: default
type: Opaque
data:
  username: cG9zdGdyZXM=     # base64 of "postgres"
  password: czNjcjN0UEBzcw== # base64 of "s3cr3tP@ss"

# Create a Secret from literal values
kubectl create secret generic db-credentials \
  --from-literal=username=postgres \
  --from-literal=password='s3cr3tP@ss'

# Create from files
kubectl create secret generic tls-cert \
  --from-file=cert.pem \
  --from-file=key.pem

Consuming Secrets in Pods

As Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:latest
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

As Volume Mounts

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: db-credentials
        defaultMode: 0400  # Read-only for owner

Volume-mounted Secrets are automatically updated when the Secret changes (with a delay of up to the kubelet sync period). Environment variable Secrets are not updated until the Pod restarts.

Best Practice 1: Enable Encryption at Rest

By default, Secrets are stored in plaintext in etcd. Anyone with etcd access can read all Secrets. Enable encryption:

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # Fallback for reading unencrypted secrets

# Add to kube-apiserver flags:
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Re-encrypt all existing Secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

In managed Kubernetes (EKS, GKE, AKS), encryption at rest is typically enabled by default with the provider's KMS.

Best Practice 2: Use External Secret Managers

For production, store secrets in a dedicated secret manager and sync them into Kubernetes:

External Secrets Operator

# Install ESO
# helm install external-secrets external-secrets/external-secrets -n external-secrets

# Configure the secret store
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

---
# Sync a secret from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/database
        property: username
    - secretKey: password
      remoteRef:
        key: prod/database
        property: password

Secrets Store CSI Driver

# Mount secrets directly from Vault as files
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.internal:8200"
    roleName: "app-role"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/database"
        secretKey: "password"

Best Practice 3: Restrict RBAC Access

Limit who can read Secrets:

# Role that allows reading only specific Secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials", "api-keys"]
    verbs: ["get"]

# Audit who can access Secrets
kubectl auth can-i get secrets --as=system:serviceaccount:default:my-sa -n production

Best Practice 4: Disable Auto-Mount of Service Account Tokens

Every Pod gets a service account token mounted by default. Disable it when not needed:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  automountServiceAccountToken: false
  containers:
    - name: app
      image: myapp:latest

Best Practice 5: Never Store Secrets in Git

Use tools like Sealed Secrets or SOPS to encrypt Secrets before committing:

# Sealed Secrets: encrypt a Secret for GitOps
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml
# Only the cluster's Sealed Secrets controller can decrypt it

Best Practice 6: Use Immutable Secrets

For Secrets that should never change, mark them as immutable to prevent accidental modification and improve performance (kubelet skips watching them):

apiVersion: v1
kind: Secret
metadata:
  name: static-api-key
type: Opaque
immutable: true
data:
  key: bXktYXBpLWtleQ==

Summary Checklist

• Enable encryption at rest in etcd
• Use an external secret manager for production
• Restrict Secret access via RBAC with resourceNames
• Disable service account token auto-mount when not needed
• Never commit plaintext Secrets to Git
• Mount Secrets as volumes rather than environment variables (more secure, auto-updated)
• Use immutable Secrets where appropriate
• Rotate secrets regularly and audit access