Kubernetes PIDPressure

Step-by-Step Troubleshooting

When a Kubernetes node enters the PIDPressure condition, it signals that process ID resources are critically low. This guide walks through identifying the source of excessive PID usage and resolving the issue before pods are evicted or workloads are disrupted.

1. Identify Nodes with PID Pressure

Start by checking which nodes have the PIDPressure condition set to true.

kubectl get nodes -o custom-columns=NAME:.metadata.name,PID_PRESSURE:.status.conditions[?(@.type=="PIDPressure")].status

For detailed node conditions and recent events, describe the affected node.

kubectl describe node <node-name>

Look at the Conditions section for PIDPressure and the Events section for any eviction-related messages. You will see entries like NodeHasPIDPressure when the threshold has been crossed.

2. Check the Node's PID Usage

SSH into the affected node or use a debug pod to inspect kernel PID utilization.

# Using a debug pod
kubectl debug node/<node-name> -it --image=ubuntu -- bash

# Once inside, check the maximum PID limit
cat /proc/sys/kernel/pid_max

# Check current number of running processes
ls -1 /proc | grep -c '^[0-9]'

# Or use ps to get a count
ps aux | wc -l

Compare the current process count against the pid_max value. If the count is approaching the limit, you have confirmed PID exhaustion.

3. Identify Which Pods Are Consuming the Most PIDs

List all pods on the node and check their PID usage.

# Get all pods on the affected node
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name> -o wide

# For each suspicious pod, check its process count
kubectl exec <pod-name> -- ps aux | wc -l

If you have access to cAdvisor metrics or a monitoring stack, query for container PID counts.

# If using Prometheus, query for top PID consumers
# container_processes metric shows process count per container
kubectl top pods --sort-by=cpu --all-namespaces | head -20

From the debug pod on the node, you can also inspect PID usage per cgroup.

# On the node, check PID counts per cgroup
find /sys/fs/cgroup -name "pids.current" -exec sh -c 'echo "$1: $(cat $1)"' _ {} \; | sort -t: -k2 -n -r | head -20

4. Look for Zombie Processes

Zombie processes hold PID slots but consume no CPU or memory, making them easy to miss.

# On the node, count zombie processes
ps aux | awk '{if ($8 == "Z") print}' | wc -l

# List zombie processes to identify their parent
ps aux | awk '{if ($8 == "Z") print $0}'

If you find many zombies, the parent process is not properly calling wait() to reap its children. The fix is to use an init system inside the container. Docker and Kubernetes support the shareProcessNamespace setting and tools like tini as the container entrypoint to handle child reaping.

# Example: enable shared process namespace for zombie reaping
spec:
  shareProcessNamespace: true
  containers:
  - name: app
    image: myapp:latest

5. Check Kubelet PID Limits and Eviction Thresholds

The kubelet has configurable settings for PID management.

# On the node, check kubelet configuration
# Look for --pod-max-pids and eviction thresholds
ps aux | grep kubelet | grep -oP '(pod-max-pids|eviction)[^ ]*'

# Or check the kubelet config file
cat /var/lib/kubelet/config.yaml | grep -A5 -i pid

The key settings are:

• --pod-max-pids: Maximum PIDs allowed per pod (default is -1, meaning unlimited)
• evictionHard and evictionSoft: Thresholds for node-level PID pressure

6. Mitigate Immediately: Evict or Delete Offending Pods

If you have identified a pod consuming excessive PIDs, you can delete it to relieve pressure immediately.

# Delete the offending pod
kubectl delete pod <pod-name> --grace-period=30

# If the pod is part of a deployment, scale down temporarily
kubectl scale deployment <deployment-name> --replicas=0

If multiple pods on the node need to be relocated, you can cordon the node and drain it.

# Prevent new pods from scheduling on this node
kubectl cordon <node-name>

# Gracefully drain existing pods
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

7. Set Pod-Level PID Limits

Configure the kubelet to enforce per-pod PID limits to prevent any single pod from monopolizing PIDs.

# In kubelet config (config.yaml)
podPidsLimit: 1024

After updating the kubelet configuration, restart the kubelet service.

sudo systemctl restart kubelet

You can also use LimitRange resources at the namespace level combined with admission controllers to enforce PID-related policies.

8. Monitor PID Usage Going Forward

Set up monitoring to catch PID pressure before it causes evictions.

# Check kubelet metrics for PID usage
curl -s http://localhost:10255/metrics | grep pid

# Key metrics to watch:
# node_processes_running - current number of processes on the node
# container_processes - process count per container

If using Prometheus and Grafana, create alerts for when PID usage crosses 80 percent of the node's pid_max value. This gives you time to investigate before the kubelet triggers evictions.

9. Increase the Kernel PID Limit (If Appropriate)

If the node's workload legitimately requires many processes, consider raising the PID limit.

# Check current limit
cat /proc/sys/kernel/pid_max

# Increase temporarily
echo 65536 | sudo tee /proc/sys/kernel/pid_max

# Make permanent via sysctl
echo "kernel.pid_max = 65536" | sudo tee -a /etc/sysctl.d/99-pid-max.conf
sudo sysctl --system

Be cautious with this approach — raising the limit without understanding why PIDs are being consumed can mask a deeper problem like a process leak.

10. Verify Resolution

After taking corrective action, confirm that the PIDPressure condition has cleared.

# Check the node condition
kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="PIDPressure")]}'

# Verify pods are running normally
kubectl get pods --field-selector spec.nodeName=<node-name> --all-namespaces

# Uncordon the node if it was previously cordoned
kubectl uncordon <node-name>

The kubelet updates node conditions on a regular interval (default every 10 seconds for status updates), so allow a short period before checking. If the PIDPressure condition reverts to False and no pods are being evicted, the issue is resolved.