How Do You Run a DaemonSet on Control Plane Nodes?

Detailed Answer

By default, DaemonSets do not run on control plane nodes because these nodes carry a taint that repels workloads. To ensure node-level agents run everywhere — including the control plane — you must add the appropriate tolerations.

The Control Plane Taint

Control plane nodes are tainted during cluster initialization:

kubectl describe node control-plane-1 | grep Taints
# Taints: node-role.kubernetes.io/control-plane:NoSchedule

This NoSchedule taint prevents any Pod without a matching toleration from being placed on the node.

Adding a Toleration for the Control Plane

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      tolerations:
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.7.0
          ports:
            - containerPort: 9100
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "200m"
              memory: "128Mi"

The operator: Exists means "match any value for this key," which works regardless of whether the taint has a value.

Tolerating All Taints

For infrastructure-critical DaemonSets that must run on every node regardless of taints, use a blanket toleration:

spec:
  template:
    spec:
      tolerations:
        - operator: "Exists"

This tolerates all taints. Use this sparingly and only for truly critical infrastructure like CNI plugins or security agents.

Common Taints to Consider

A comprehensive DaemonSet should tolerate multiple taints:

tolerations:
  # Control plane nodes
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"
  # Nodes with disk pressure
  - key: "node.kubernetes.io/disk-pressure"
    operator: "Exists"
    effect: "NoSchedule"
  # Nodes with memory pressure
  - key: "node.kubernetes.io/memory-pressure"
    operator: "Exists"
    effect: "NoSchedule"
  # Unschedulable nodes (being drained)
  - key: "node.kubernetes.io/unschedulable"
    operator: "Exists"
    effect: "NoSchedule"
  # Not-ready nodes
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300
  # Unreachable nodes
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300

tolerationSeconds for NoExecute

The NoExecute effect is different from NoSchedule. When a node becomes unreachable, Kubernetes adds a NoExecute taint. Pods without a toleration are evicted immediately. The tolerationSeconds field specifies how long the Pod should stay before eviction:

- key: "node.kubernetes.io/not-ready"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 300  # Stay for 5 minutes before eviction

For DaemonSets, you may want to omit tolerationSeconds to keep the Pod running indefinitely, since it is expected to be on every node.

Priority Classes for DaemonSets

In addition to tolerations, set a high priority class to prevent DaemonSet Pods from being preempted:

spec:
  template:
    spec:
      priorityClassName: system-node-critical
      tolerations:
        - operator: "Exists"
      containers:
        - name: cni-plugin
          image: calico/node:v3.27

The system-node-critical priority class (priority value 2000001000) ensures the DaemonSet Pod is among the last to be evicted during resource pressure.

Verifying Coverage

After deploying a DaemonSet with tolerations, verify it runs on all nodes:

# Check DaemonSet status
kubectl get daemonset node-exporter -n kube-system

# Expected: DESIRED = CURRENT = READY = total node count
# NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
# node-exporter    5         5         5       5            5

# Verify Pods are on control plane nodes
kubectl get pods -n kube-system -l app=node-exporter -o wide

If DESIRED does not match your total node count, check for missing tolerations or nodeSelector restrictions.