How Do You Use Ephemeral Containers for Debugging?

Detailed Answer

Ephemeral containers are a special type of container that can be added to an already-running Pod for troubleshooting purposes. Unlike regular containers, they are never automatically restarted and are not part of the original Pod spec. They were introduced as stable in Kubernetes 1.25.

Why Ephemeral Containers Exist

Modern container best practices recommend using minimal or distroless images that contain only the application binary and its runtime dependencies. This means no shell, no curl, no netstat, no ps -- nothing useful for debugging.

When a Pod running such an image has a problem, you cannot kubectl exec into it because there is no shell to execute. Ephemeral containers solve this by letting you inject a fully-equipped debugging container into the Pod's namespaces.

Using kubectl debug

The kubectl debug command is the primary way to work with ephemeral containers.

Debug a Running Pod

# Add an ephemeral container with debugging tools
kubectl debug -it my-pod --image=busybox:1.37 --target=app

# The --target flag shares the process namespace with the specified container
# This lets you see processes running in the 'app' container

This command:

1. Adds a new ephemeral container to the running Pod.
2. Attaches an interactive terminal to it.
3. Shares the process namespace of the app container (via --target).

Debug with a Full Toolkit

# Use a more complete debugging image
kubectl debug -it my-pod --image=nicolaka/netshoot --target=app

# Inside the ephemeral container, you now have access to:
# - curl, wget (HTTP debugging)
# - tcpdump, nmap (network debugging)
# - strace, ltrace (system call tracing)
# - dig, nslookup (DNS debugging)
# - ss, netstat (socket inspection)

Debug by Creating a Copy of the Pod

If you need to modify the Pod spec (for example, change the command or add environment variables), you can create a debug copy:

# Create a copy of the Pod with a different container image
kubectl debug my-pod -it --copy-to=my-pod-debug --container=app --image=myapp/server:debug

# Create a copy that changes the command to a shell
kubectl debug my-pod -it --copy-to=my-pod-debug --set-image=app=busybox:1.37

# Create a copy that shares process namespaces
kubectl debug my-pod -it --copy-to=my-pod-debug --share-processes

Debug a Node

Ephemeral containers can also debug nodes by creating a privileged Pod:

# Start a debugging session on a node
kubectl debug node/my-node -it --image=ubuntu:24.04

# This creates a Pod with hostPID, hostNetwork, and mounts the node's root filesystem
# The node filesystem is available at /host
chroot /host
journalctl -u kubelet --no-pager -l

Ephemeral Container Spec

When using the API directly, an ephemeral container looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  ephemeralContainers:
    - name: debugger
      image: busybox:1.37
      command: ["sh"]
      stdin: true
      tty: true
      targetContainerName: app
      securityContext:
        capabilities:
          add: ["SYS_PTRACE"]

The targetContainerName field enables process namespace sharing with the specified container, allowing you to see its processes with ps aux.

Practical Debugging Scenarios

Scenario 1: Debugging Network Issues

# Attach a network debugging container
kubectl debug -it my-pod --image=nicolaka/netshoot --target=app

# Inside the ephemeral container:
# Check DNS resolution
dig kubernetes.default.svc.cluster.local

# Check connectivity to another service
curl -v http://my-service:8080/healthz

# Capture network traffic
tcpdump -i eth0 -n port 8080

# Check listening ports
ss -tlnp

Scenario 2: Debugging a Distroless Container

# The main container uses gcr.io/distroless/static -- no shell available
kubectl exec my-pod -- sh
# error: unable to start container process: exec: "sh": executable file not found

# Use ephemeral container instead
kubectl debug -it my-pod --image=busybox:1.37 --target=app

# Now you can inspect the process
ps aux
cat /proc/1/environ | tr '\0' '\n'
ls -la /proc/1/root/app/

Scenario 3: Inspecting Shared Volumes

# Ephemeral containers can mount the same volumes as the target container
kubectl debug -it my-pod --image=busybox:1.37 --target=app

# Check files written by the application
ls /app/data/
cat /app/config/settings.yaml

Limitations of Ephemeral Containers

1. Cannot be removed: Once added, an ephemeral container remains in the Pod spec (though it stops running when you exit). The Pod must be deleted to fully clean up.
2. No ports or probes: Ephemeral containers cannot define ports, readiness probes, or liveness probes.
3. No guaranteed resources: While you can set resource requests/limits, ephemeral containers are not intended for long-running workloads.
4. RBAC required: Users need the pods/ephemeralcontainers permission to add ephemeral containers.

RBAC for Ephemeral Containers

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-debugger
rules:
  - apiGroups: [""]
    resources: ["pods/ephemeralcontainers"]
    verbs: ["patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]

Best Practices

1. Keep a standard debug image in your container registry that includes your team's preferred debugging tools.
2. Use --target to share the process namespace when you need to inspect the application's processes or filesystem.
3. Grant ephemeral container RBAC only to operators who need production debugging access.
4. Prefer kubectl debug --copy-to when you need to modify the Pod spec, so you do not affect the running Pod.
5. Clean up debug Pods after your session -- ephemeral containers persist until the Pod is deleted.