How Does the Kubernetes API Server Authentication Chain Work?

Detailed Answer

The Authentication Pipeline

Every API request passes through authentication before anything else happens:

Request arrives at API server
  -> Authentication (multiple authenticators tried in order)
  -> Authorization (RBAC / ABAC / Webhook)
  -> Admission Control
  -> Persistence to etcd

The API server runs multiple authenticators simultaneously. Each request is passed to every configured authenticator until one accepts it. If none accept, the request is rejected with a 401 Unauthorized response.

Authentication Methods

1. X.509 Client Certificates

The most common method for cluster administrators. The API server validates the client certificate against its configured CA (Certificate Authority).

The certificate's Common Name (CN) becomes the username, and the Organization (O) fields become group memberships:

# Generate a client certificate for user "jane" in group "developers"
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr \
  -subj "/CN=jane/O=developers"

# Sign with the cluster CA
openssl x509 -req -in jane.csr \
  -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key \
  -CAcreateserial \
  -out jane.crt -days 365

# Configure kubectl
kubectl config set-credentials jane \
  --client-certificate=jane.crt \
  --client-key=jane.key

kubectl config set-context jane-ctx \
  --cluster=my-cluster \
  --user=jane

kubectl config use-context jane-ctx

The API server flag --client-ca-file specifies which CA to trust:

kube-apiserver --client-ca-file=/etc/kubernetes/pki/ca.crt

2. ServiceAccount Tokens

Every Pod in Kubernetes gets a ServiceAccount token automatically (unless disabled). These are JWTs signed by the API server.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: default

# View the projected token in a Pod
kubectl exec my-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

# The token is a JWT with claims like:
# {
#   "iss": "https://kubernetes.default.svc",
#   "sub": "system:serviceaccount:default:app-sa",
#   "aud": ["https://kubernetes.default.svc"],
#   "exp": 1719500000
# }

Since Kubernetes 1.24, ServiceAccount tokens are bound tokens that are:

• Time-limited (default 1 hour, auto-refreshed by kubelet)
• Audience-scoped
• Bound to a specific Pod (invalidated when the Pod is deleted)

3. OIDC (OpenID Connect)

OIDC integrates Kubernetes with external identity providers like Okta, Azure AD, Google, or Keycloak. Users authenticate through the IdP and receive a JWT that the API server validates.

# API server OIDC flags
kube-apiserver \
  --oidc-issuer-url=https://accounts.google.com \
  --oidc-client-id=my-k8s-cluster \
  --oidc-username-claim=email \
  --oidc-groups-claim=groups \
  --oidc-ca-file=/etc/kubernetes/pki/oidc-ca.crt

Users authenticate with kubectl using an OIDC plugin:

# kubeconfig with OIDC
users:
  - name: jane
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        command: kubectl
        args:
          - oidc-login
          - get-token
          - --oidc-issuer-url=https://accounts.google.com
          - --oidc-client-id=my-k8s-cluster
          - --oidc-client-secret=<secret>

4. Webhook Token Authentication

The API server forwards bearer tokens to an external service for validation:

# Webhook config file
apiVersion: v1
kind: Config
clusters:
  - name: auth-service
    cluster:
      server: https://auth.internal/authenticate
      certificate-authority: /etc/kubernetes/pki/auth-ca.crt
users:
  - name: apiserver
    user:
      client-certificate: /etc/kubernetes/pki/apiserver.crt
      client-key: /etc/kubernetes/pki/apiserver.key
contexts:
  - context:
      cluster: auth-service
      user: apiserver
    name: webhook
current-context: webhook

kube-apiserver --authentication-token-webhook-config-file=/etc/kubernetes/webhook-config.yaml

5. Anonymous Access

If no authenticator accepts the request and anonymous access is enabled, the request is assigned the username system:anonymous and group system:unauthenticated. This is typically disabled in production.

Authorization with RBAC

After authentication, RBAC determines what the identity can do:

# ClusterRole: define permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
# ClusterRoleBinding: bind to an identity
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jane-pod-reader
subjects:
  - kind: User
    name: jane
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Debugging Authentication Issues

# Check who you are authenticated as
kubectl auth whoami  # Kubernetes 1.27+

# Test if a specific identity can perform an action
kubectl auth can-i create pods --as=jane
kubectl auth can-i get secrets --as=system:serviceaccount:default:app-sa

# View API server logs for authentication failures
# Look for "Unable to authenticate the request" messages

# Decode a ServiceAccount token
kubectl get secret <sa-token> -o jsonpath='{.data.token}' | base64 -d | jwt decode -

Best Practices

1. Use OIDC for human users to leverage existing identity infrastructure and MFA.
2. Use short-lived ServiceAccount tokens (bound tokens) for workloads.
3. Disable anonymous authentication in production.
4. Use distinct ServiceAccounts per workload rather than sharing the default account.
5. Audit authentication events via the API server audit log.
6. Rotate client certificates before expiry and automate renewal.