What Are Static Pods in Kubernetes?

Detailed Answer

Static Pods are Pods that are directly managed by the kubelet daemon on a node, rather than being created through the Kubernetes API server. The kubelet watches a designated directory on the local filesystem for Pod manifest files and automatically creates, monitors, and restarts these Pods.

How Static Pods Work

1. The kubelet is configured with a staticPodPath (default: /etc/kubernetes/manifests).
2. The kubelet watches this directory for YAML or JSON Pod manifests.
3. When a manifest file is added, the kubelet creates the Pod on the node.
4. The kubelet monitors the Pod and restarts it if it crashes.
5. When a manifest file is removed, the kubelet deletes the corresponding Pod.
6. When a manifest file is modified, the kubelet recreates the Pod with the new spec.

Control Plane Bootstrapping

The most common use of static Pods is in clusters bootstrapped with kubeadm. The control plane components run as static Pods:

# On a control plane node
ls /etc/kubernetes/manifests/
# etcd.yaml
# kube-apiserver.yaml
# kube-controller-manager.yaml
# kube-scheduler.yaml

This creates a bootstrapping solution to a chicken-and-egg problem: you need the API server running to create Pods, but you need Pods to run the API server. Static Pods solve this because the kubelet can start them directly without an API server.

Example Static Pod Manifest

# /etc/kubernetes/manifests/my-static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-static-pod
  namespace: default
  labels:
    role: static
spec:
  containers:
    - name: web
      image: nginx:1.27
      ports:
        - containerPort: 80
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
        limits:
          cpu: "250m"
          memory: "256Mi"
      livenessProbe:
        httpGet:
          path: /
          port: 80
        periodSeconds: 10
  hostNetwork: true

To create this static Pod, simply place the file in the kubelet's static Pod directory:

# Copy manifest to the static pod path
sudo cp my-static-pod.yaml /etc/kubernetes/manifests/

# The kubelet will detect it and start the Pod within seconds
# Check using crictl on the node
sudo crictl pods | grep my-static-pod

Mirror Pods

When the kubelet starts a static Pod, it also creates a mirror Pod in the API server. This mirror is a read-only representation that lets you see the static Pod via kubectl:

# The mirror Pod has the node name appended
kubectl get pods
# NAME                        READY   STATUS    RESTARTS   AGE
# my-static-pod-node01        1/1     Running   0          5m

# You can view it but cannot modify it
kubectl delete pod my-static-pod-node01
# The kubelet will immediately recreate the mirror Pod

Mirror Pods have an ownerReference pointing to the Node object, which distinguishes them from regular Pods.

Configuring the Static Pod Path

The static Pod path is set in the kubelet configuration:

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
staticPodPath: /etc/kubernetes/manifests

You can also configure it via the kubelet flag --pod-manifest-path, though the configuration file approach is preferred.

Static Pods vs. DaemonSets

Both static Pods and DaemonSets run Pods on specific nodes, but they serve different purposes:

| Feature | Static Pods | DaemonSets | |---------|-------------|------------| | Managed by | Kubelet | DaemonSet controller (API server) | | Requires API server | No | Yes | | Can use kubectl to manage | No (mirror only) | Yes | | Rolling updates | Manual (edit manifest file) | Automated | | Scheduling | Always on the configured node | Controlled by node selectors and tolerations | | Use case | Control plane components | Node-level agents (log collectors, monitoring) |

Updating Static Pods

To update a static Pod, you modify the manifest file on the node. The kubelet detects the change, terminates the old Pod, and starts a new one with the updated spec.

# Edit the manifest directly on the node
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

# The kubelet will automatically recreate the Pod
# Monitor the process
sudo crictl pods --name kube-apiserver

For control plane upgrades, kubeadm upgrade automates this process by modifying the manifest files in /etc/kubernetes/manifests/.

Troubleshooting Static Pods

# Check kubelet logs for static Pod issues
sudo journalctl -u kubelet -f

# Verify the static Pod path is configured
sudo cat /var/lib/kubelet/config.yaml | grep staticPodPath

# Check if the manifest is valid YAML
python3 -c "import yaml; yaml.safe_load(open('/etc/kubernetes/manifests/my-pod.yaml'))"

# List running containers on the node
sudo crictl ps | grep my-static-pod

Common issues:

• Manifest syntax errors: The kubelet silently ignores invalid manifests. Check kubelet logs.
• Image pull failures: The node must be able to pull the container image. Static Pods respect the same image pull policies as regular Pods.
• Resource conflicts: If a static Pod requests a host port already in use, it will fail to start.

Best Practices

1. Use static Pods only for control plane components or infrastructure bootstrapping -- not for application workloads.
2. Use DaemonSets instead for node-level agents that benefit from API-server management and rolling updates.
3. Back up static Pod manifests -- they are critical infrastructure configuration stored on individual nodes.
4. Monitor mirror Pods through normal monitoring tools to detect static Pod failures.
5. Avoid manual edits to kubeadm-managed manifests unless you understand the implications for cluster upgrades.