Kubernetes Exit Code 143

Step-by-Step Troubleshooting

1. Determine if Exit Code 143 is Expected

Exit code 143 is normal during:

• Rolling updates (kubectl rollout)
• Scale-down events
• Pod deletions (kubectl delete pod)
• Node drains (kubectl drain)

Exit code 143 is unexpected if:

• It appears without any user action or scheduled event
• The pod restarts with CrashLoopBackOff showing exit code 143
• Multiple pods are terminated simultaneously without explanation

kubectl describe pod <pod-name>

2. Check What Triggered the Termination

# Check pod events
kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'

# Check for recent deployments
kubectl rollout history deployment/<deploy-name>

# Check for HPA activity
kubectl describe hpa <hpa-name>

# Check for node drain events
kubectl get events --field-selector reason=Drain

# Check Kubernetes audit logs (if enabled)

3. Verify Graceful Shutdown is Working

When SIGTERM is sent, the application should:

1. Stop accepting new requests
2. Complete in-flight requests
3. Close database connections and release resources
4. Exit cleanly

# Check how long the container took to stop
kubectl describe pod <pod-name> | grep -A5 "Last State"

# If the container took the full grace period, it may not be handling SIGTERM

If the container exits immediately with 143, SIGTERM handling is working. If it takes exactly terminationGracePeriodSeconds and exits with 137, it is being SIGKILL-ed because it did not shut down in time.

4. Implement SIGTERM Handling

Ensure your application handles SIGTERM properly.

Node.js:

process.on('SIGTERM', () => {
  console.log('SIGTERM received, shutting down gracefully...');
  server.close(() => {
    console.log('Server closed');
    process.exit(0);  // Exit cleanly after cleanup
  });
});

Python:

import signal
import sys

def handle_sigterm(signum, frame):
    print("SIGTERM received, shutting down...")
    # Close connections, flush buffers
    server.shutdown()
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

Go:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()

// Start server
go server.ListenAndServe()

// Wait for signal
<-ctx.Done()
log.Println("Shutting down gracefully...")
server.Shutdown(context.Background())

Java (Spring Boot):

Spring Boot handles SIGTERM automatically with server.shutdown=graceful:

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

5. Adjust Termination Grace Period

If your application needs more time to shut down:

spec:
  terminationGracePeriodSeconds: 60  # Default is 30
  containers:
    - name: app
      image: myapp:v1

The timeline:

1. SIGTERM sent at T=0
2. Application has 60 seconds to shut down
3. If still running at T=60, SIGKILL is sent (exit code 137)

6. Use preStop Hooks

For additional cleanup or to give load balancers time to deregister the pod:

spec:
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]

The preStop hook runs before SIGTERM is sent. This is useful for:

• Giving the service mesh/load balancer time to remove the pod from endpoints
• Running cleanup scripts
• Draining connections from external load balancers

The preStop sleep + shutdown time must be less than terminationGracePeriodSeconds.

7. Configure PodDisruptionBudgets

Prevent too many pods from being terminated simultaneously during voluntary disruptions.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2  # or maxUnavailable: 1
  selector:
    matchLabels:
      app: myapp

This ensures at least 2 replicas are always running during node drains and rolling updates.

8. Investigate Unexpected Terminations

If exit code 143 appears without any user action:

# Check for autoscaler activity
kubectl get events -A | grep -i "scale\|autoscal"

# Check for cluster autoscaler scale-down
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100 | grep -i "scale down\|terminate"

# Check for preemption
kubectl get events -A | grep -i preempt

# Check node conditions
kubectl get nodes -o custom-columns='NAME:.metadata.name,STATUS:.status.conditions[?(@.type=="Ready")].status'

9. Verify Graceful Shutdown

Test that your application shuts down properly:

# Deploy and wait for the pod to be running
kubectl apply -f deployment.yaml
kubectl wait --for=condition=Ready pod -l app=myapp

# Delete the pod and observe
kubectl delete pod <pod-name> &
kubectl logs <pod-name> -f

# You should see graceful shutdown logs
# The pod should exit with code 143 (or 0 if it catches SIGTERM and exits 0)

# Check the exit code
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

A well-behaved application will either exit with code 0 (caught SIGTERM and exited cleanly) or 143 (default SIGTERM behavior). Both are acceptable.