kubectl debug

When to Use kubectl debug

kubectl debug is a modern debugging tool that addresses the limitations of kubectl exec. When containers are built from minimal or distroless images with no shell or diagnostic tools, exec is useless. Debug solves this by injecting a debug container with the tools you need, either as an ephemeral container in the running pod or in a copy of the pod.

Ephemeral Containers

Ephemeral containers are added to a running pod without restarting it. They share the pod's network namespace and can optionally share the process namespace with a target container:

# Add a debug container targeting the app container
kubectl debug my-pod -it --image=busybox --target=app

# Use a full debugging toolkit
kubectl debug my-pod -it --image=nicolaka/netshoot --target=app

# Use a custom debug image with your tools
kubectl debug my-pod -it --image=myregistry/debug-tools:latest --target=app

With --target=app, the debug container shares the process namespace with the app container. This means you can:

• See the target container's processes with ps aux
• Inspect files in the target container's filesystem via /proc/<PID>/root/
• Use debugging tools like strace on the target container's processes

# Inside the ephemeral container:
# List target container processes
ps aux

# Access target container filesystem
ls /proc/1/root/app/

# Trace system calls on the main process
strace -p 1

Debugging Pod Copies

When you cannot modify the running pod, or need to change the container image for debugging, create a copy:

# Create a copy with a debug container added
kubectl debug my-pod -it --copy-to=my-pod-debug --image=busybox --share-processes

# Create a copy with a different container image
kubectl debug my-pod -it --copy-to=my-pod-debug --set-image=app=ubuntu

# Create a copy with all containers replaced
kubectl debug my-pod -it --copy-to=my-pod-debug --set-image=*=ubuntu

Pod copies are independent pods. They run alongside the original and must be deleted manually when debugging is complete:

# Clean up debug copies
kubectl delete pod my-pod-debug

Node Debugging

Debug node-level issues by creating a privileged pod on the target node:

# Debug a node
kubectl debug node/worker-1 -it --image=ubuntu

# Inside the debug pod, the host filesystem is at /host
chroot /host
# Now you have a shell on the node itself

# Or access host files directly
ls /host/var/log/
cat /host/etc/kubernetes/kubelet.conf
journalctl --directory=/host/var/log/journal

The node debug pod runs with hostPID, hostNetwork, and the host filesystem mounted at /host. This gives you full access to diagnose kubelet issues, network configuration, storage problems, and kernel parameters.

Debugging Distroless Containers

Distroless images are becoming standard for security-hardened deployments. They have no shell, no package manager, and no debugging tools:

# The app container has no shell
kubectl exec my-pod -- /bin/sh
# error: exec failed: unable to start container process

# Use debug instead
kubectl debug my-pod -it --image=nicolaka/netshoot --target=app

# Now you have full networking tools
# tcpdump, curl, dig, nslookup, ip, ss, netstat, etc.

Popular debug images include:

• busybox: Minimal Unix utilities (sh, ls, cat, wget)
• nicolaka/netshoot: Comprehensive network debugging toolkit
• ubuntu/debian: Full Linux distribution for general debugging
• alpine: Lightweight Linux with apk package manager

Debugging CrashLoopBackOff

When a pod is crashing too quickly to exec into, create a debug copy with a different entrypoint:

# Copy the pod with a sleep command to prevent crashing
kubectl debug my-pod -it --copy-to=my-pod-debug \
  --set-image=app=busybox -- sleep infinity

# Now exec into the stable copy and inspect
kubectl exec -it my-pod-debug -- /bin/sh

# Or create a copy that shares the original image but overrides the command
kubectl debug my-pod -it --copy-to=my-pod-debug \
  --container=app -- /bin/sh

Practical Debugging Workflows

# Network debugging: check DNS and connectivity
kubectl debug my-pod -it --image=nicolaka/netshoot --target=app
# dig my-service.default.svc.cluster.local
# curl -v http://api-service:8080/health
# tcpdump -i eth0 port 8080

# File system debugging: inspect application files
kubectl debug my-pod -it --image=busybox --target=app
# ls /proc/1/root/app/config/
# cat /proc/1/root/app/logs/error.log

# Performance debugging: profile a Java application
kubectl debug my-pod -it --image=openjdk:17 --target=app
# jcmd 1 Thread.print
# jmap -heap 1

Best Practices

Use ephemeral containers when possible, as they do not disturb the running pod. Always use --target to share the process namespace for deep debugging. Clean up pod copies after debugging. Use purpose-built debug images with the tools you need rather than large general-purpose images. Restrict debug access in production with RBAC policies on the pods/ephemeralcontainers subresource.