Kubernetes ServiceAccount Token Issues

Step-by-Step Troubleshooting

Service account tokens are the primary authentication mechanism for pods communicating with the Kubernetes API. When they fail, any in-cluster application that needs API access stops working. This guide covers all common token failure modes.

1. Check if the Token Is Mounted

Verify the token file exists inside the pod.

# Check if the token is mounted
kubectl exec <pod-name> -- ls -la /var/run/secrets/kubernetes.io/serviceaccount/

# Expected files:
# ca.crt  - cluster CA certificate
# namespace - the pod's namespace
# token - the service account JWT token

If the directory does not exist or the token file is missing, check if automounting is disabled.

# Check the pod spec
kubectl get pod <pod-name> -o jsonpath='{.spec.automountServiceAccountToken}'

# Check the service account
kubectl get serviceaccount <sa-name> -n <namespace> -o jsonpath='{.automountServiceAccountToken}'

If automounting is disabled (false), either enable it or manually mount the token via a projected volume.

2. Verify the Service Account Exists

Check if the referenced service account is present in the namespace.

# Check which SA the pod uses
kubectl get pod <pod-name> -o jsonpath='{.spec.serviceAccountName}'

# Verify the SA exists
kubectl get serviceaccount <sa-name> -n <namespace>

If the service account was deleted, recreate it and restart the pod.

kubectl create serviceaccount <sa-name> -n <namespace>
kubectl delete pod <pod-name>

3. Check Token Validity

Examine the mounted token to verify it is valid.

# Get the token from inside the pod
kubectl exec <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

# Decode the JWT token (without verifying signature)
kubectl exec <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d'.' -f2 | base64 -d 2>/dev/null | jq .

Check the decoded token for:

• exp: Expiration timestamp — is it in the past?
• iss: Issuer — does it match the API server?
• sub: Subject — does it reference the correct service account?
• aud: Audience — does it match what the API server expects?

4. Test Token Authentication

Try using the token to authenticate to the API server.

# From inside the pod
kubectl exec <pod-name> -- sh -c '
  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  APISERVER=https://kubernetes.default.svc
  CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  curl -s --cacert $CACERT -H "Authorization: Bearer $TOKEN" $APISERVER/api/v1/namespaces/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/pods
'

If this returns a 401, the token is not being accepted. If it returns a 403, the token is valid but lacks permissions (which is an RBAC issue, not a token issue).

5. Handle Projected Token Expiration (Kubernetes 1.22+)

Projected service account tokens expire and are rotated by the kubelet. If rotation fails, the token becomes stale.

# Check the projected volume configuration
kubectl get pod <pod-name> -o yaml | grep -A20 "projected:"

# Check the token expiration
kubectl exec <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.exp | todate'

If the token is expired:

1. Check if the kubelet on the node is healthy (it handles rotation)
2. Restart the pod to get a fresh token

# Check kubelet health on the node
kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

# Restart the pod
kubectl delete pod <pod-name>

6. Handle Legacy Token Migration (Kubernetes 1.24+)

Starting in Kubernetes 1.24, service accounts no longer automatically get a Secret with a long-lived token. Applications that depended on reading the token from a Secret must be updated.

# Check if there is a legacy token secret
kubectl get secrets -n <namespace> | grep <sa-name>

# If no secret exists and the application needs one, create a long-lived token manually
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: <sa-name>-token
  namespace: <namespace>
  annotations:
    kubernetes.io/service-account.name: <sa-name>
type: kubernetes.io/service-account-token
EOF

# Or use the TokenRequest API for a time-limited token
kubectl create token <sa-name> -n <namespace> --duration=24h

Note: Long-lived tokens are less secure than projected tokens. Prefer using projected tokens and ensuring your application reads the token from the mounted file on each request.

7. Fix Audience Mismatch

If the token audience does not match what the verifier expects, authentication fails.

# Check the token's audience
kubectl exec <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.aud'

# The default audience is the API server's --service-account-issuer value
# Check what the API server expects
kubectl get pod -n kube-system kube-apiserver-<node> -o yaml | grep service-account-issuer

If you need a token with a specific audience (for example, for a webhook or external service), use the TokenRequest API.

kubectl create token <sa-name> -n <namespace> --audience=<target-audience>

Or configure a projected volume with a specific audience:

volumes:
- name: custom-token
  projected:
    sources:
    - serviceAccountToken:
        audience: "my-service"
        expirationSeconds: 3600
        path: token

8. Check Application Token Handling

Many applications cache the token at startup and never re-read it. With projected tokens that expire, this causes failures after the token's lifetime.

# Check if the application re-reads the token
# Look for the token file access time
kubectl exec <pod-name> -- stat /var/run/secrets/kubernetes.io/serviceaccount/token

If the application caches the token, update it to re-read the token file before each API call, or use an official Kubernetes client library that handles token rotation automatically. All official client libraries (Go, Python, Java) support automatic token refresh when using in-cluster configuration.

9. Check Kubelet Token Rotation

The kubelet is responsible for refreshing projected tokens before they expire.

# Check kubelet logs for token rotation issues
# On the node:
journalctl -u kubelet --no-pager --since "1 hour ago" | grep -i "token\|serviceaccount"

If the kubelet is not rotating tokens, verify the kubelet has connectivity to the API server and the node is in Ready state.

10. Verify Token Authentication Works

After resolving the issue, confirm the service account token is working.

# Verify from inside the pod
kubectl exec <pod-name> -- sh -c '
  curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    https://kubernetes.default.svc/api/v1/namespaces/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/pods | head -5
'

# Check that the application logs no longer show 401 errors
kubectl logs <pod-name> --tail=20

# Verify the token is being rotated (check modification time)
kubectl exec <pod-name> -- stat /var/run/secrets/kubernetes.io/serviceaccount/token

The token is working when API calls succeed, the application no longer logs authentication errors, and the token file shows recent modification times indicating rotation is active.