How Does Ordered Deployment Work in StatefulSets?

Detailed Answer

Ordered deployment is one of the defining features of StatefulSets. It ensures that Pods are created, updated, and deleted in a predictable sequence, which is essential for applications that have initialization dependencies.

The Default Behavior: OrderedReady

When you create a StatefulSet with replicas: 3, Kubernetes does not launch all three Pods at once. Instead:

1. Pod-0 is created first
2. Kubernetes waits until Pod-0 is Running and Ready (all readiness probes pass)
3. Pod-1 is created
4. Kubernetes waits until Pod-1 is Running and Ready
5. Pod-2 is created

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
spec:
  serviceName: "etcd-headless"
  replicas: 3
  podManagementPolicy: OrderedReady  # This is the default
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
        - name: etcd
          image: quay.io/coreos/etcd:v3.5.12
          ports:
            - containerPort: 2379
              name: client
            - containerPort: 2380
              name: peer
          readinessProbe:
            httpGet:
              path: /health
              port: 2379
            initialDelaySeconds: 10
            periodSeconds: 5
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

Why Ordering Matters

Consider an etcd cluster:

1. etcd-0 starts and initializes a new cluster as the sole member
2. etcd-1 starts and joins the cluster by contacting etcd-0
3. etcd-2 starts and joins by contacting the existing members

If all three started simultaneously, they might each try to form their own cluster, leading to a split-brain scenario.

Termination Order

Deletion happens in reverse ordinal order:

1. etcd-2 is terminated first
2. Once fully stopped, etcd-1 is terminated
3. Finally, etcd-0 is terminated

This protects the primary/leader (typically ordinal 0) by ensuring it is the last to go, giving the cluster time to handle member removal gracefully.

Scaling Behavior

The same ordering applies to scaling:

Scale up (3 → 5):

• Pod-3 is created and must be Ready before Pod-4 is created

Scale down (5 → 3):

• Pod-4 is terminated and must fully stop before Pod-3 is terminated

# Scale up — Pods are added in order
kubectl scale statefulset etcd --replicas=5

# Watch the ordered creation
kubectl get pods -l app=etcd -w
# etcd-3   0/1   Pending   0   0s
# etcd-3   1/1   Running   0   15s
# etcd-4   0/1   Pending   0   0s
# etcd-4   1/1   Running   0   12s

When a Pod Gets Stuck

If Pod-1 fails to become Ready during initial creation or scaling:

• Pod-2 is never created
• The StatefulSet reports a condition indicating the blocked state
• Manual intervention is required (fix the Pod, check probes, examine logs)

# Check why a Pod is not Ready
kubectl describe pod etcd-1
kubectl logs etcd-1

# Check StatefulSet status
kubectl get statefulset etcd -o yaml | grep -A 5 conditions

Overriding the Order

If your application does not need ordered deployment, you can use Parallel pod management policy (covered in the related question on pod management policies). This is useful for workloads that need stable identity and storage but not ordered startup — like some cache clusters.