What Are Admission Webhooks in Kubernetes?

Detailed Answer

Admission webhooks are a powerful extensibility mechanism in Kubernetes. They let you intercept and modify (or reject) any API request before the object is persisted to etcd. This is the foundation for tools like Istio sidecar injection, OPA Gatekeeper, and Kyverno.

Where Webhooks Fit in the API Request Flow

Client → API Server
           ├── Authentication (who are you?)
           ├── Authorization (are you allowed?)
           ├── Mutating Admission Webhooks (modify the object)
           ├── Object Schema Validation
           ├── Validating Admission Webhooks (accept or reject)
           └── Persist to etcd

Types of Admission Webhooks

| Type | When It Runs | Can Modify Object | Can Reject Request | |------|-------------|-------------------|-------------------| | MutatingAdmissionWebhook | Before validation | Yes | Yes | | ValidatingAdmissionWebhook | After mutation | No | Yes |

Mutating webhooks always run before validating webhooks. Multiple webhooks of the same type run in alphabetical order by name.

MutatingWebhookConfiguration Example

This webhook injects a sidecar container into every Pod created in namespaces labeled sidecar-injection: enabled:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
  - name: sidecar.example.com
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Fail
    timeoutSeconds: 10
    clientConfig:
      service:
        name: sidecar-injector
        namespace: webhook-system
        path: /mutate
      caBundle: <base64-encoded-ca-cert>
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    namespaceSelector:
      matchLabels:
        sidecar-injection: enabled

ValidatingWebhookConfiguration Example

This webhook rejects any Pod that runs as root:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: deny-root-pods
webhooks:
  - name: deny-root.example.com
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Fail
    timeoutSeconds: 5
    clientConfig:
      service:
        name: policy-webhook
        namespace: webhook-system
        path: /validate
      caBundle: <base64-encoded-ca-cert>
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: ["kube-system"]

Webhook Server Implementation

A minimal webhook server receives an AdmissionReview request and returns an AdmissionReview response:

// Simplified Go webhook handler
func handleValidate(w http.ResponseWriter, r *http.Request) {
    var review admissionv1.AdmissionReview
    json.NewDecoder(r.Body).Decode(&review)

    var pod corev1.Pod
    json.Unmarshal(review.Request.Object.Raw, &pod)

    allowed := true
    reason := ""

    if pod.Spec.SecurityContext != nil &&
       pod.Spec.SecurityContext.RunAsUser != nil &&
       *pod.Spec.SecurityContext.RunAsUser == 0 {
        allowed = false
        reason = "Pods must not run as root (UID 0)"
    }

    review.Response = &admissionv1.AdmissionResponse{
        UID:     review.Request.UID,
        Allowed: allowed,
        Result:  &metav1.Status{Message: reason},
    }
    json.NewEncoder(w).Encode(review)
}

Critical Configuration Fields

| Field | Purpose | Recommendation | |-------|---------|----------------| | failurePolicy | Behavior when webhook is unreachable | Fail for security policies, Ignore for non-critical mutations | | timeoutSeconds | Max wait time (1-30s) | Keep under 10s to avoid API latency | | sideEffects | Whether webhook has side effects | Set to None unless it truly has external effects | | namespaceSelector | Which namespaces to intercept | Always exclude kube-system to prevent lockouts | | reinvocationPolicy | Re-run after other mutating webhooks | Set to IfNeeded for webhooks that depend on other mutations |

Avoiding Cluster Lockouts

The most dangerous mistake with admission webhooks is creating a webhook that intercepts its own resources. If the webhook Pod goes down and failurePolicy: Fail is set, the API server cannot create a new Pod for the webhook service — a deadlock.

Prevent this by:

1. Excluding the webhook's own namespace via namespaceSelector
2. Using objectSelector to skip system-critical Pods
3. Setting failurePolicy: Ignore for non-security-critical webhooks

namespaceSelector:
  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
        - kube-system
        - webhook-system

Debugging Admission Webhooks

# Check webhook configurations
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations

# See if a webhook rejected a request
kubectl describe pod <failing-pod> | grep -A 5 Events

# Check webhook server logs
kubectl logs -n webhook-system deploy/sidecar-injector

# Test with dry-run
kubectl apply -f pod.yaml --dry-run=server -v=6

Performance Considerations

Each webhook adds latency to API requests. In a cluster with many webhooks, this can significantly slow down deployments. Monitor webhook latency with:

# API server metrics
# apiserver_admission_webhook_admission_duration_seconds_bucket