Kubernetes Pod Pending

Step-by-Step Troubleshooting

1. Check Pod Events

The scheduler writes events explaining why a pod cannot be scheduled.

kubectl describe pod <pod-name> -n <namespace>

Look at the Events section for messages like:

0/5 nodes are available: 2 Insufficient cpu, 3 Insufficient memory.
0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector.
0/5 nodes are available: 5 node(s) had taint {key: NoSchedule}, that the pod didn't tolerate.
persistentvolumeclaim "my-pvc" not found

If there are no events at all, the scheduler may not be running.

2. Check Cluster Resource Availability

See how much capacity is available across nodes.

# Check resource usage per node
kubectl top nodes

# Check allocatable vs allocated resources
kubectl describe nodes | grep -A5 "Allocated resources"

# Detailed view for a specific node
kubectl describe node <node-name>

Compare the pod's resource requests against available allocatable resources:

# Check the pending pod's resource requests
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources.requests}'

3. Check Node Selectors and Affinity Rules

If the pod has scheduling constraints, verify nodes match them.

# Check pod's node selector
kubectl get pod <pod-name> -o jsonpath='{.spec.nodeSelector}'

# Check pod's node affinity
kubectl get pod <pod-name> -o jsonpath='{.spec.affinity}'

# List nodes with their labels
kubectl get nodes --show-labels

# Check if any nodes match a specific label
kubectl get nodes -l <label-key>=<label-value>

If no nodes have the required labels, add them:

kubectl label node <node-name> <label-key>=<label-value>

4. Check Taints and Tolerations

Nodes may have taints that prevent scheduling.

# List all node taints
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name): \(.spec.taints // [] | map(.key + "=" + (.value // "") + ":" + .effect) | join(", "))"'

# Check the pod's tolerations
kubectl get pod <pod-name> -o jsonpath='{.spec.tolerations}' | jq .

If the pod needs to run on a tainted node, add a toleration:

spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "special"
      effect: "NoSchedule"

5. Check PersistentVolumeClaims

If the pod mounts a PVC, verify it is bound.

# Check PVC status
kubectl get pvc -n <namespace>

# Describe the PVC for details
kubectl describe pvc <pvc-name> -n <namespace>

Common PVC issues:

• Pending PVC: The storage class provisioner may be broken or no matching PV exists
• Storage class does not exist: Check kubectl get storageclass
• Zone mismatch: The PV is in a different availability zone than the node

# Check storage class provisioner
kubectl get storageclass
kubectl describe storageclass <class-name>

# Check if the provisioner pods are running
kubectl get pods -n kube-system | grep -i provisioner

6. Check ResourceQuotas

Namespace quotas may prevent the pod from being created.

# Check quota in the namespace
kubectl describe resourcequota -n <namespace>

# Check LimitRanges
kubectl describe limitrange -n <namespace>

If the quota is exhausted:

Name:       compute-quota
Resource    Used    Hard
--------    ----    ----
cpu         4       4
memory      8Gi     8Gi

Options: increase the quota, reduce requests on existing pods, or deploy to a different namespace.

7. Verify the Scheduler is Running

If the pod has no scheduling events at all, check the scheduler.

# Check scheduler pods
kubectl get pods -n kube-system -l component=kube-scheduler

# Check scheduler logs
kubectl logs -n kube-system -l component=kube-scheduler --tail=50

8. Check for Pod Topology Spread Constraints

If the pod uses topology spread constraints, ensure they can be satisfied.

kubectl get pod <pod-name> -o jsonpath='{.spec.topologySpreadConstraints}' | jq .

If whenUnsatisfiable: DoNotSchedule is set and the constraint cannot be met, the pod stays Pending. Consider using ScheduleAnyway during scaling events.

9. Apply the Fix

Based on your findings:

# Scale up the cluster (cloud-specific)
# Add more nodes to satisfy resource requests

# Remove a taint
kubectl taint node <node-name> <key>:<effect>-

# Fix node selector
kubectl patch deployment <deploy-name> --type=json \
  -p='[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value": {"zone": "us-east-1a"}}]'

# Increase resource quota
kubectl edit resourcequota <quota-name> -n <namespace>

10. Verify Resolution

# Watch the pod start
kubectl get pod <pod-name> -w

# Verify it is running on the expected node
kubectl get pod <pod-name> -o wide

The pod should transition from Pending to ContainerCreating to Running.