How Does Secret Encryption at Rest Work in Kubernetes?

Detailed Answer

Kubernetes Secrets are not encrypted by default. They are stored as base64-encoded values in etcd, which means anyone with direct etcd access can read every Secret in the cluster. Encryption at rest protects Secrets by encrypting them before they are written to etcd.

The Default (Insecure) State

Without encryption at rest, Secrets are stored like this in etcd:

# Reading a Secret directly from etcd (base64 encoded, NOT encrypted)
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret | hexdump -C
# You can see the Secret values in plaintext

Enabling Encryption at Rest

1. Create an EncryptionConfiguration

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      # The first provider is used for encryption
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      # Fallback: read unencrypted Secrets written before encryption was enabled
      - identity: {}

Generate a random encryption key:

head -c 32 /dev/urandom | base64
# Outputs a 32-byte key suitable for AES-256

2. Configure the API Server

# Add to kube-apiserver flags
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

3. Restart the API Server

After restart, all new Secrets are encrypted. Existing Secrets remain unencrypted until you re-encrypt them.

4. Re-encrypt Existing Secrets

# This reads every Secret and writes it back, triggering encryption
kubectl get secrets -A -o json | kubectl replace -f -

Encryption Providers

| Provider | Algorithm | Key Management | Recommended For | |----------|-----------|----------------|-----------------| | identity | None (plaintext) | N/A | Never in production | | aescbc | AES-CBC (padding) | Local file | Development/testing | | aesgcm | AES-GCM (authenticated) | Local file | Better than CBC but still local keys | | secretbox | XSalsa20+Poly1305 | Local file | Strong encryption, local keys | | kms v1/v2 | External KMS | External service | Production |

KMS Provider (Production Recommended)

KMS providers delegate encryption to an external key management service. The API server sends the data encryption key (DEK) to the KMS for wrapping/unwrapping, so the master key never leaves the KMS.

AWS KMS Example

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: aws-encryption-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}

The KMS plugin runs as a sidecar or DaemonSet and communicates with AWS KMS:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-encryption-provider
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: aws-encryption-provider
  template:
    metadata:
      labels:
        app: aws-encryption-provider
    spec:
      containers:
        - name: provider
          image: aws-encryption-provider:latest
          args:
            - --key=arn:aws:kms:us-east-1:123456789:key/abcd-1234
            - --listen=/var/run/kmsplugin/socket.sock
            - --region=us-east-1
          volumeMounts:
            - name: socket
              mountPath: /var/run/kmsplugin
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
      volumes:
        - name: socket
          hostPath:
            path: /var/run/kmsplugin

Key Rotation

Local Key Rotation

# Step 1: Add new key as the first entry
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key2              # New key (used for encryption)
              secret: <new-key>
            - name: key1              # Old key (used for decryption)
              secret: <old-key>
      - identity: {}

# Step 2: Restart API server to pick up new config

# Step 3: Re-encrypt all Secrets with the new key
kubectl get secrets -A -o json | kubectl replace -f -

# Step 4: Remove old key from config (after verifying all Secrets are re-encrypted)

KMS Key Rotation

With KMS, you rotate the master key in the KMS service (e.g., AWS KMS automatic rotation). The DEK cache in the API server is refreshed automatically.

Verifying Encryption

# Check if a Secret is encrypted in etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# Encrypted Secrets start with "k8s:enc:aescbc:" or "k8s:enc:kms:"
# Unencrypted Secrets show the JSON plaintext

Managed Kubernetes

| Provider | Encryption at Rest | Default | |----------|-------------------|---------| | EKS | AWS KMS envelope encryption | Optional (must enable) | | GKE | Google-managed encryption | Enabled by default | | AKS | Azure Key Vault | Optional |

# Enable on EKS
aws eks associate-encryption-config \
  --cluster-name my-cluster \
  --encryption-config '[{
    "resources": ["secrets"],
    "provider": {
      "keyArn": "arn:aws:kms:us-east-1:123456789:key/abcd"
    }
  }]'

Common Mistakes

1. Assuming Secrets are encrypted by default — they are not
2. Storing the encryption key in the same place as etcd — if an attacker gets etcd, they get the key too
3. Not re-encrypting existing Secrets — old Secrets remain unencrypted after enabling encryption
4. Using identity provider first — the first provider is used for encryption, so identity first means no encryption
5. Not backing up the encryption key — if the key is lost, all encrypted Secrets become unrecoverable