Kubernetes 403 Forbidden

Step-by-Step Troubleshooting

403 Forbidden errors mean the identity is known but lacks authorization. This guide walks through identifying exactly what permission is missing and how to grant it properly.

1. Read the Error Message Carefully

The 403 error message contains exactly what permission is needed.

# Example error:
# Error from server (Forbidden): deployments.apps is forbidden:
# User "john" cannot create resource "deployments" in API group "apps" in the namespace "production"

This tells you:

• Who: The user or service account (User "john")
• What action: The verb (cannot create)
• What resource: The resource and API group (deployments in apps)
• Where: The namespace (production)

2. Check Current Permissions

Use kubectl auth can-i to test what the user or service account is allowed to do.

# Check if you can perform the specific action
kubectl auth can-i create deployments -n production

# Check as a specific user
kubectl auth can-i create deployments -n production --as=john

# Check as a service account
kubectl auth can-i create deployments -n production --as=system:serviceaccount:<namespace>:<sa-name>

# List all permissions for a user
kubectl auth can-i --list --as=john -n production

# List all permissions for a service account
kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<sa-name> -n <namespace>

3. Find Existing RBAC Bindings

Check what Roles and Bindings already exist for the subject.

# Check RoleBindings in the namespace
kubectl get rolebinding -n <namespace> -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects

# Check ClusterRoleBindings
kubectl get clusterrolebinding -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects | grep -E "<username>|<sa-name>"

# Describe a specific binding to see details
kubectl describe rolebinding <binding-name> -n <namespace>

4. Check the Role or ClusterRole

If a binding exists, verify the Role grants the needed permissions.

# Check a Role
kubectl describe role <role-name> -n <namespace>

# Check a ClusterRole
kubectl describe clusterrole <role-name>

# Get the role in YAML to see exact rules
kubectl get role <role-name> -n <namespace> -o yaml

Look at the rules section. Each rule specifies apiGroups, resources, and verbs. Verify the rule covers:

• The correct apiGroup (e.g., "" for core, "apps" for deployments, "batch" for jobs)
• The correct resource (e.g., deployments, pods, services)
• The correct verb (e.g., get, list, create, update, delete, patch, watch)

5. Create or Update the Required RBAC

If no appropriate Role or binding exists, create them.

# Create a Role with the needed permissions
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-manager
  namespace: production
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch"]
EOF

# Bind the Role to the user
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: john-deployment-manager
  namespace: production
subjects:
- kind: User
  name: john
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: deployment-manager
  apiGroup: rbac.authorization.k8s.io
EOF

For service accounts:

# Create a service account
kubectl create serviceaccount <sa-name> -n <namespace>

# Bind a role to it
kubectl create rolebinding <binding-name> --role=<role-name> --serviceaccount=<namespace>:<sa-name> -n <namespace>

# For cluster-wide access
kubectl create clusterrolebinding <binding-name> --clusterrole=<role-name> --serviceaccount=<namespace>:<sa-name>

6. Handle Cluster-Scoped Resources

Resources like nodes, namespaces, PersistentVolumes, and ClusterRoles are cluster-scoped and require ClusterRole + ClusterRoleBinding.

# Check if the resource is cluster-scoped
kubectl api-resources | grep <resource-name>
# Look at the NAMESPACED column — false means cluster-scoped

# Create a ClusterRoleBinding for cluster-scoped resources
kubectl create clusterrolebinding <binding-name> --clusterrole=<role-name> --user=<username>

A common mistake is creating a RoleBinding for a cluster-scoped resource — this will not work because RoleBindings are namespace-scoped.

7. Fix Service Account in Pod Spec

If a pod's service account lacks permissions, ensure the pod uses the correct service account.

# Check what service account the pod uses
kubectl get pod <pod-name> -o jsonpath='{.spec.serviceAccountName}'

# Update the Deployment to use the correct service account
kubectl patch deployment <deployment-name> -p '{"spec":{"template":{"spec":{"serviceAccountName":"<correct-sa>"}}}}'

8. Use Built-In ClusterRoles

Kubernetes provides built-in ClusterRoles for common use cases.

# List built-in ClusterRoles
kubectl get clusterrole | grep -E '^admin|^edit|^view|^cluster-admin'

# view: read-only access to most resources
# edit: read-write access to most resources (no RBAC changes)
# admin: full access within a namespace (including RBAC)
# cluster-admin: full access to everything (use sparingly)

# Bind a built-in role
kubectl create rolebinding <name> --clusterrole=edit --user=<user> -n <namespace>

9. Debug with Impersonation

If you are a cluster admin, test permissions by impersonating the affected user.

# Impersonate a user
kubectl get deployments -n production --as=john

# Impersonate a service account
kubectl get pods -n default --as=system:serviceaccount:default:my-sa

# Impersonate with a group
kubectl get nodes --as=john --as-group=developers

10. Verify the Fix

Confirm the 403 error is resolved.

# Test the specific permission
kubectl auth can-i <verb> <resource> -n <namespace> --as=<subject>

# Retry the original command
kubectl <original-command>

# Verify the binding exists
kubectl get rolebinding -n <namespace> | grep <binding-name>
kubectl get clusterrolebinding | grep <binding-name>

The 403 error is resolved when kubectl auth can-i returns "yes" for the required operation and the original command succeeds. Apply the principle of least privilege — only grant the specific permissions needed, not broader access than required.