Kubernetes Multi-Attach Error

Step-by-Step Troubleshooting

Multi-Attach errors create a deadlock: the new pod cannot start because the volume is attached elsewhere, and the old attachment is not releasing. This guide walks through identifying the stale attachment and safely resolving it.

1. Get the Error Details

Start with the pod events to identify the exact volume and node involved.

kubectl describe pod <pod-name>

The event will show something like:

Warning  FailedAttachVolume  Multi-Attach error for volume "pvc-abc12345"
Volume is already exclusively attached to one node and can't be attached to another

Note the volume name (PV name) from the error.

2. Identify the Volume and Current Attachment

Find out where the volume is currently attached.

# Get the PV details
kubectl get pv <pv-name> -o yaml

# Check VolumeAttachments for this volume
kubectl get volumeattachment -o json | jq '.items[] | select(.spec.source.persistentVolumeName=="<pv-name>") | {name: .metadata.name, node: .spec.nodeName, attached: .status.attached}'

This tells you which node currently holds the volume attachment. There may be two VolumeAttachment objects — one old (attached) and one new (pending).

3. Check the Status of the Old Node

Determine if the node holding the volume is still functional.

kubectl get node <old-node-name>
kubectl describe node <old-node-name> | grep -A5 "Conditions"

If the node is NotReady, it may have crashed and cannot properly detach the volume. The attach/detach controller will eventually force-detach, but this can take 6+ minutes by default.

4. Check if the Old Pod Still Exists

See if there is a pod still using the volume on the old node.

# Find pods using this PVC
kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="<pvc-name>") | "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName) - \(.status.phase)"'

If an old pod is still in Terminating state on the failed node, it is holding the volume attachment open.

# Check if there is a stuck terminating pod
kubectl get pods --all-namespaces --field-selector spec.nodeName=<old-node> | grep Terminating

5. Wait for Automatic Recovery (Short Timeout)

Kubernetes will eventually force-detach volumes from failed nodes. The timeline is approximately:

• node-monitor-grace-period: 40 seconds (default) — node marked NotReady
• pod-eviction-timeout: 5 minutes (default) — pods are evicted
• After eviction, the attach/detach controller processes the detach

# Watch the VolumeAttachment status
kubectl get volumeattachment -w | grep <pv-name>

If you can wait 6-10 minutes, the system will often resolve itself. However, if the application requires faster recovery, proceed to manual intervention.

6. Force Delete the Stuck Pod

If a terminating pod on the old node is blocking the detach, force delete it.

# Force delete the old pod
kubectl delete pod <old-pod-name> --grace-period=0 --force

This removes the pod from the API server, allowing the attach/detach controller to proceed with detaching the volume from the old node.

7. Delete the Stale VolumeAttachment

If the VolumeAttachment is stuck, delete it to unblock the new attachment.

# Delete the old VolumeAttachment
kubectl delete volumeattachment <old-attachment-name>

# If it is stuck in finalizer, remove the finalizer
kubectl patch volumeattachment <old-attachment-name> -p '{"metadata":{"finalizers":null}}' --type=merge
kubectl delete volumeattachment <old-attachment-name>

After deleting the old VolumeAttachment, the controller should create a new one for the correct node.

8. Force Detach at the Cloud Provider Level

If Kubernetes-level operations do not release the volume, force detach directly at the cloud provider.

# Get the cloud volume ID from the PV
kubectl get pv <pv-name> -o jsonpath='{.spec.csi.volumeHandle}'

# AWS: Force detach the EBS volume
aws ec2 detach-volume --volume-id <vol-id> --force

# GCP: Detach the persistent disk
gcloud compute instances detach-disk <old-instance-name> --disk=<disk-name> --zone=<zone>

# Azure: Detach the managed disk
az vm disk detach --resource-group <rg> --vm-name <old-vm> --name <disk-name>

Warning: Only force detach when you are certain the old pod is not actively writing to the volume. Force detach while a process is writing can corrupt the filesystem.

9. Delete and Recreate the Pod

After the volume is free, the pod may need to be recreated.

# Delete the stuck pod if it has not automatically recovered
kubectl delete pod <pod-name>

# Watch for the new pod to start
kubectl get pods -w | grep <workload-name>

For StatefulSet pods, the controller will automatically recreate the pod with the same name and PVC binding.

10. Verify Volume Is Attached to the Correct Node

Confirm the volume is now attached to the new node and the pod is running.

# Check VolumeAttachment
kubectl get volumeattachment -o custom-columns=NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached | grep <pv-name>

# Check pod status
kubectl get pod <pod-name> -o wide

# Verify data integrity
kubectl exec <pod-name> -- ls -la <mount-path>
kubectl exec <pod-name> -- df -h <mount-path>

After confirming the pod is running and the volume data is intact, consider implementing measures to prevent recurrence: using node health monitoring, configuring faster eviction timeouts if data safety allows, or switching to ReadWriteMany volumes if the workload can benefit from shared access.