How Does RBAC Audit Logging Work in Kubernetes?

Detailed Answer

Kubernetes audit logging records every request to the API server, including the identity making the request, the resource accessed, and whether it was allowed or denied. For RBAC, audit logs are essential for detecting unauthorized access attempts, tracking permission changes, and meeting compliance requirements.

Audit Policy

An audit policy defines which events are logged and at what level of detail:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all RBAC changes at RequestResponse level
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources:
          - roles
          - rolebindings
          - clusterroles
          - clusterrolebindings

  # Log secret access at Metadata level (don't log secret values)
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]

  # Log all forbidden responses
  - level: Request
    omitStages:
      - RequestReceived
    verbs: ["get", "list", "create", "update", "delete"]

  # Skip noisy health checks and system requests
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]

  - level: None
    resources:
      - group: ""
        resources: ["endpoints", "services"]
    verbs: ["watch", "list"]

  # Default: log metadata for everything else
  - level: Metadata
    omitStages:
      - RequestReceived

Audit Levels

| Level | What Is Logged | Use For | |-------|---------------|---------| | None | Nothing | High-volume, low-risk events | | Metadata | Timestamp, user, verb, resource, response code | Most resources | | Request | Metadata + request body | Create/update operations | | RequestResponse | Metadata + request + response body | Sensitive operations (RBAC, secrets access) |

Enabling Audit Logging

Configure the API server with audit policy and backend:

# kube-apiserver flags
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100

For webhook backends (send to external systems):

--audit-webhook-config-file=/etc/kubernetes/audit-webhook.yaml
--audit-webhook-batch-max-size=100
--audit-webhook-batch-max-wait=5s

RBAC-Specific Audit Events

Detecting Unauthorized Access (403 Forbidden)

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/secrets",
  "verb": "list",
  "user": {
    "username": "developer@example.com",
    "groups": ["dev-team", "system:authenticated"]
  },
  "responseStatus": {
    "metadata": {},
    "code": 403,
    "reason": "Forbidden"
  },
  "requestReceivedTimestamp": "2026-03-19T10:15:30Z"
}

Detecting RBAC Changes

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "verb": "create",
  "objectRef": {
    "resource": "clusterrolebindings",
    "name": "new-admin-binding",
    "apiGroup": "rbac.authorization.k8s.io"
  },
  "user": {
    "username": "admin@example.com"
  },
  "responseStatus": {
    "code": 201
  }
}

Useful Audit Log Queries

# Find all 403 Forbidden events
cat /var/log/kubernetes/audit.log | \
  jq 'select(.responseStatus.code == 403)'

# Find RBAC changes
cat /var/log/kubernetes/audit.log | \
  jq 'select(.objectRef.apiGroup == "rbac.authorization.k8s.io")'

# Find who accessed secrets
cat /var/log/kubernetes/audit.log | \
  jq 'select(.objectRef.resource == "secrets") | {user: .user.username, verb: .verb, name: .objectRef.name}'

# Find cluster-admin binding creations
cat /var/log/kubernetes/audit.log | \
  jq 'select(.objectRef.resource == "clusterrolebindings" and .verb == "create")'

Alerting on RBAC Events

Set up alerts for these high-priority RBAC events:

| Event | Priority | Alert | |-------|----------|-------| | ClusterRoleBinding to cluster-admin created | Critical | Immediate notification | | Repeated 403 from same user | High | Potential unauthorized access attempt | | RBAC resources modified outside change window | High | Unauthorized change | | ServiceAccount created in kube-system | Medium | Potential privilege escalation | | Impersonation used | Medium | Audit trail for privileged actions |

Example Prometheus Alert

If you ship audit events as metrics:

groups:
  - name: rbac-alerts
    rules:
      - alert: ClusterAdminBindingCreated
        expr: |
          increase(apiserver_audit_event_total{
            verb="create",
            resource="clusterrolebindings"
          }[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "A new ClusterRoleBinding was created"

      - alert: ExcessiveForbiddenRequests
        expr: |
          increase(apiserver_audit_event_total{
            code="403"
          }[5m]) > 50
        labels:
          severity: warning
        annotations:
          summary: "High rate of forbidden API requests"

Best Practices

1. Always log RBAC changes at RequestResponse level
2. Never log secret values — use Metadata level for secrets
3. Ship logs externally — audit logs on the API server node can be lost if the node fails
4. Set retention policies matching your compliance requirements (typically 90-365 days)
5. Alert on anomalies — unusual 403 patterns, RBAC changes outside business hours, or new cluster-admin bindings
6. Test your audit policy — apply it in a staging cluster first to verify it captures what you need without excessive volume