Kubernetes Pod Evicted

Step-by-Step Troubleshooting

1. Identify Evicted Pods

List all evicted pods across the cluster.

# Find evicted pods
kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted

# Count evictions per node
kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase=="Failed" and .status.reason=="Evicted") | .spec.nodeName' | sort | uniq -c | sort -rn

2. Check the Eviction Reason

Describe the evicted pod to see what resource was under pressure.

kubectl describe pod <evicted-pod-name> -n <namespace>

Look for messages like:

Status:     Failed
Reason:     Evicted
Message:    The node was low on resource: memory. Threshold quantity: 100Mi, available: 48Mi.

This tells you which resource triggered the eviction and how far past the threshold the node was.

3. Check Node Conditions

Examine the node that evicted the pods.

# Check node conditions
kubectl describe node <node-name> | grep -A5 Conditions

# Check node resource usage
kubectl top node <node-name>

# Check allocatable vs capacity
kubectl get node <node-name> -o jsonpath='{.status.allocatable}' | jq .
kubectl get node <node-name> -o jsonpath='{.status.capacity}' | jq .

Key conditions to look for:

• MemoryPressure: Available memory is below the eviction threshold
• DiskPressure: Available disk space is below the eviction threshold
• PIDPressure: Available process IDs are below the threshold

4. Identify Resource-Hungry Pods on the Node

Find which pods were consuming the most resources on the affected node.

# List pods on the node sorted by memory usage
kubectl top pods -A --sort-by=memory | head -20

# List pods on a specific node
kubectl get pods -A --field-selector spec.nodeName=<node-name> -o wide

# Check resource requests and limits for pods on the node
kubectl get pods -A --field-selector spec.nodeName=<node-name> -o json | \
  jq -r '.items[] | "\(.metadata.name) req:\(.spec.containers[0].resources.requests.memory // "none") lim:\(.spec.containers[0].resources.limits.memory // "none")"'

5. Check QoS Classes

Pods are evicted in order of their QoS class. Check which class the evicted pod was in.

kubectl get pod <pod-name> -o jsonpath='{.status.qosClass}'

| QoS Class | Criteria | Eviction Priority | |-----------|----------|-------------------| | BestEffort | No requests or limits set | First to be evicted | | Burstable | Requests set but not equal to limits | Second | | Guaranteed | Requests equal to limits for all containers | Last to be evicted |

6. Clean Up Evicted Pods

Evicted pods remain in the Failed state and consume etcd storage. Clean them up.

# Delete all evicted pods across all namespaces
kubectl get pods -A --field-selector=status.phase=Failed -o json | \
  jq -r '.items[] | select(.status.reason=="Evicted") | "\(.metadata.namespace) \(.metadata.name)"' | \
  while read ns name; do kubectl delete pod "$name" -n "$ns"; done

7. Fix Resource Configuration

Prevent future evictions by setting proper resource requests and limits.

# Promote pods to Guaranteed QoS
resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "250m"

Set up LimitRanges to enforce defaults:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: <namespace>
spec:
  limits:
    - default:
        memory: "512Mi"
        cpu: "500m"
      defaultRequest:
        memory: "256Mi"
        cpu: "250m"
      type: Container

8. Adjust Eviction Thresholds (If Needed)

If your workloads are being evicted too aggressively, review the kubelet eviction thresholds.

# Check current kubelet configuration
kubectl get --raw /api/v1/nodes/<node-name>/proxy/configz | jq '.kubeletconfig.evictionHard'

Default hard eviction thresholds:

• memory.available < 100Mi
• nodefs.available < 10%
• imagefs.available < 15%
• nodefs.inodesFree < 5%

9. Use Pod Priority for Critical Workloads

Assign higher priority to critical pods so they are evicted last.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "For critical application pods"
---
apiVersion: v1
kind: Pod
metadata:
  name: critical-app
spec:
  priorityClassName: high-priority
  containers:
    - name: app
      image: myapp:v1

10. Scale the Cluster

If evictions are happening because nodes are genuinely overloaded, add more capacity.

# If using Cluster Autoscaler, check its status
kubectl get pods -n kube-system -l app=cluster-autoscaler
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=50

# Manually scale a node group (cloud-specific)
# AWS EKS
eksctl scale nodegroup --cluster=<cluster> --name=<nodegroup> --nodes=5