kubectl drain

When to Use kubectl drain

kubectl drain is the standard command for safely removing all workloads from a node before maintenance. It cordons the node first, then gracefully evicts pods while respecting PodDisruptionBudgets.

How Drain Works

The drain process follows these steps:

1. The node is cordoned (marked unschedulable).
2. All pods on the node are identified.
3. DaemonSet pods are skipped (with --ignore-daemonsets).
4. Pods managed by controllers (Deployments, StatefulSets, Jobs) are evicted using the Eviction API.
5. PodDisruptionBudgets are checked before each eviction.
6. The command waits for all pods to terminate.

kubectl drain worker-node-1 --ignore-daemonsets --delete-emptydir-data
# node/worker-node-1 cordoned
# evicting pod default/my-app-7d4b8c5f9-x2k4j
# evicting pod monitoring/prometheus-node-exporter-abc12
# pod/my-app-7d4b8c5f9-x2k4j evicted
# node/worker-node-1 drained

Required Flags Explained

In practice, kubectl drain almost always needs additional flags:

# Minimum viable drain command
kubectl drain worker-node-1 --ignore-daemonsets --delete-emptydir-data

• --ignore-daemonsets: DaemonSet pods cannot be rescheduled (they must run on every node). Without this flag, drain refuses to proceed.
• --delete-emptydir-data: Pods using emptyDir volumes will lose that data. This flag acknowledges that risk.
• --force: Required for standalone pods not managed by a controller. These pods are permanently deleted, not rescheduled.

PodDisruptionBudgets (PDBs)

PDBs are the safety mechanism that prevents drain from causing outages:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2        # or maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

When draining, if evicting a pod would leave fewer than minAvailable healthy replicas, the eviction blocks. This ensures your application maintains availability during maintenance.

# Drain blocks if PDB would be violated
kubectl drain worker-node-1 --ignore-daemonsets
# evicting pod default/my-app-7d4b8c5f9-x2k4j
# error when evicting pods/"my-app-7d4b8c5f9-x2k4j" -n "default":
# Cannot evict pod as it would violate the pod's disruption budget.

# Add a timeout to avoid indefinite blocking
kubectl drain worker-node-1 --ignore-daemonsets --timeout=300s

Handling Stubborn Pods

Some pods resist eviction:

# Standalone pods (not managed by a controller)
kubectl drain node-1 --ignore-daemonsets --force
# WARNING: deleting pods not managed by ReplicationController, Job, or DaemonSet

# Pods with local storage
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# Pods with PDB violations — wait or override
kubectl drain node-1 --ignore-daemonsets --timeout=600s --disable-eviction

The --disable-eviction flag (Kubernetes 1.18+) bypasses the Eviction API and directly deletes pods, ignoring PDBs. Use this only as a last resort.

Dry-Run Before Draining

Always preview what will happen:

kubectl drain worker-node-1 --ignore-daemonsets --delete-emptydir-data --dry-run=client
# node/worker-node-1 cordoned (dry run)
# evicting pod default/my-app-7d4b8c5f9-x2k4j (dry run)
# evicting pod kube-system/coredns-5d78c9869d-l8n4f (dry run)

Rolling Drain for Zero Downtime

When draining multiple nodes, do it sequentially to maintain capacity:

#!/bin/bash
NODES="worker-1 worker-2 worker-3"
for node in $NODES; do
  echo "Draining $node..."
  kubectl drain "$node" --ignore-daemonsets --delete-emptydir-data --timeout=300s

  echo "Performing maintenance on $node..."
  ssh "$node" "sudo apt-get update && sudo apt-get upgrade -y && sudo reboot"

  echo "Waiting for $node to rejoin..."
  kubectl wait --for=condition=Ready "node/$node" --timeout=600s

  echo "Uncordoning $node..."
  kubectl uncordon "$node"

  echo "$node complete. Proceeding to next node."
done

StatefulSet Considerations

StatefulSets require extra care during drains:

• Pods have stable identities and persistent storage.
• PVCs remain bound even after pod eviction.
• Pods are rescheduled to other nodes but reattach to the same PVCs (if the storage supports it).
• Zone-specific storage may prevent rescheduling if no other node is in the same zone.

Always verify StatefulSet pods are healthy after drain:

kubectl get pods -l app=my-statefulset -o wide