kubectl cordon

When to Use kubectl cordon

kubectl cordon marks a node as unschedulable. The Kubernetes scheduler will stop placing new pods on that node, but any pods already running will continue uninterrupted. This is the first step in a graceful node maintenance workflow.

How It Works

When you cordon a node, Kubernetes sets the spec.unschedulable: true field on the node object:

kubectl cordon worker-node-1
# node/worker-node-1 cordoned

kubectl get node worker-node-1
# NAME             STATUS                     ROLES    AGE   VERSION
# worker-node-1   Ready,SchedulingDisabled    <none>   45d   v1.28.5

The node's status shows SchedulingDisabled alongside its readiness state. The node is still Ready — it just won't accept new pods.

Node Maintenance Workflow

The standard pattern for node maintenance is cordon, drain, maintain, then uncordon:

# Step 1: Prevent new pods
kubectl cordon worker-node-1

# Step 2: Evict existing pods (this step is separate)
kubectl drain worker-node-1 --ignore-daemonsets --delete-emptydir-data

# Step 3: Perform maintenance (OS update, hardware repair, etc.)

# Step 4: Return node to service
kubectl uncordon worker-node-1

Sometimes you only need to cordon without draining — for example, when debugging a node issue and you want existing pods to stay while you investigate.

Cordoning by Label

When performing rolling maintenance across a node pool, label selectors help:

# Cordon all GPU nodes
kubectl cordon -l node-type=gpu

# Cordon all nodes in a specific zone
kubectl cordon -l topology.kubernetes.io/zone=us-east-1a

# Verify
kubectl get nodes -l node-type=gpu

Impact on Existing Pods

Cordoning does NOT affect pods already running on the node:

• Running pods continue to run.
• DaemonSet pods continue to run.
• Jobs in progress continue to completion.
• Pod health checks and readiness probes are unaffected.

Only the scheduler's decision for new pods is affected. If a pod on a cordoned node crashes and gets restarted by the kubelet, it restarts on the same node (the kubelet manages restarts, not the scheduler).

However, if a pod is deleted and needs to be rescheduled (e.g., it is managed by a Deployment), the new replacement pod will be scheduled on a different uncordoned node.

Monitoring Cordoned Nodes

Track which nodes are cordoned:

# List all unschedulable nodes
kubectl get nodes | grep SchedulingDisabled

# JSON query for automation
kubectl get nodes -o json | jq -r '.items[] | select(.spec.unschedulable==true) | .metadata.name'

Capacity Planning Considerations

Before cordoning, ensure remaining nodes have sufficient capacity:

# Check current resource allocation
kubectl describe node worker-node-1 | grep -A 5 "Allocated resources"

# List pods on the node
kubectl get pods --all-namespaces --field-selector spec.nodeName=worker-node-1

# Check capacity of remaining nodes
kubectl top nodes

If cordoning a node leaves the cluster with insufficient capacity, pending pods will appear with Unschedulable events.

Difference Between Cordon and Taint

Both cordon and taints can prevent scheduling, but they work differently:

| Feature | Cordon | Taint | |---------|--------|-------| | Mechanism | Sets spec.unschedulable | Adds taint to node | | Effect on existing pods | None | Can evict (with NoExecute) | | Pod exceptions | None — no new pods at all | Pods with matching tolerations can still schedule | | Granularity | Binary on/off | Key/value pairs with effects |

Use cordon for simple maintenance scenarios. Use taints when you need selective scheduling (e.g., only GPU workloads on GPU nodes).