What does kube-apiserver do and why is it central to the cluster?

Detailed Answer

The kube-apiserver is the front-end for the Kubernetes control plane. It implements the Kubernetes API as a RESTful service over HTTPS and acts as the gateway through which all cluster communication flows. No component, whether internal (scheduler, controller manager, kubelet) or external (kubectl, CI/CD pipelines), modifies cluster state without going through the API server.

Request Lifecycle

Every API request follows a well-defined pipeline:

1. Authentication -- The API server determines who is making the request using one or more authentication modules (client certificates, bearer tokens, OIDC, service account tokens, or webhook token authentication).

2. Authorization -- Once the identity is established, the API server checks whether the user is allowed to perform the requested action. Kubernetes supports multiple authorization modes including RBAC (Role-Based Access Control), ABAC, Node authorization, and webhook authorization. RBAC is the most widely used.

3. Admission Control -- After authorization, the request passes through a chain of admission controllers. These are plugins that can mutate or validate the request. There are two phases:

   • Mutating admission webhooks can modify the object (e.g., injecting a sidecar container, setting default resource limits).
   • Validating admission webhooks can accept or reject the request but cannot modify it.

4. Validation and Persistence -- The API server validates the object schema, writes it to etcd, and returns a response.

Key Responsibilities

The API server also provides watch functionality. Components like the scheduler, controllers, and kubelets open long-lived watch connections to the API server to receive real-time notifications when objects change. This event-driven model is the foundation of Kubernetes's declarative reconciliation pattern.

# You can observe watch events directly:
kubectl get pods --watch -o json

# Check API server configuration flags on a kubeadm cluster:
kubectl -n kube-system get pod kube-apiserver-controlplane -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n'

API Server Flags and Configuration

The API server is configured through command-line flags in its static pod manifest. Some of the most important flags include:

# Key configuration flags
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.0.0.10
    - --secure-port=6443
    - --etcd-servers=https://127.0.0.1:2379
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --authorization-mode=Node,RBAC
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml

Audit Logging

The API server supports audit logging that records who did what, when, and to which resources. This is critical for security and compliance:

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods"]
  verbs: ["create", "delete"]
- level: None
  resources:
  - group: ""
    resources: ["endpoints", "services"]
    verbs: ["get", "list", "watch"]

Aggregation Layer

The API server supports an aggregation layer that allows you to extend the Kubernetes API with custom API servers. This is how the metrics-server exposes metrics.k8s.io and how custom resources served by aggregated API servers work alongside native resources:

# List API services including aggregated ones
kubectl get apiservices

# Check a specific aggregated API
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

Health and Debugging

# Check API server health endpoints
kubectl get --raw='/healthz?verbose'
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'

# View API server metrics (Prometheus format)
kubectl get --raw='/metrics' | head -50

# List all API resources available
kubectl api-resources
kubectl api-versions

Performance Considerations

The API server uses an in-memory cache (the watch cache) to serve read requests without hitting etcd for every call. The --watch-cache-sizes flag lets you tune cache sizes for high-churn resources. For large clusters (5,000+ nodes), tuning --max-requests-inflight and --max-mutating-requests-inflight prevents request overload.