What does the kubelet do on each Kubernetes node?

Detailed Answer

The kubelet is the primary agent that runs on every node in a Kubernetes cluster, both control plane nodes and worker nodes. Unlike other control plane components that can run as pods, the kubelet runs as a native system service (typically managed by systemd) because it is responsible for running pods in the first place.

Core Responsibilities

Node Registration -- When a kubelet starts, it registers the node with the API server, advertising the node's capacity (CPU, memory, ephemeral storage, extended resources like GPUs), its labels, and its network information.

Pod Lifecycle Management -- The kubelet watches the API server for pods assigned to its node. When it receives a pod spec, it instructs the container runtime to pull images, create containers, set up networking, and mount volumes. When a pod is deleted, it ensures graceful termination.

Health Monitoring -- The kubelet executes three types of health probes defined in pod specs:

• Liveness probes -- Determine if a container is running. Failure triggers a container restart.
• Readiness probes -- Determine if a container is ready to serve traffic. Failure removes the pod from Service endpoints.
• Startup probes -- Protect slow-starting containers by delaying liveness checks.

Status Reporting -- The kubelet periodically reports node status (conditions, capacity, allocatable resources) and pod status back to the API server. This is the heartbeat mechanism the control plane uses to detect node failures.

kubelet Configuration

The kubelet is typically configured via a KubeletConfiguration file:

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
staticPodPath: /etc/kubernetes/manifests
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
maxPods: 110
nodeStatusUpdateFrequency: 10s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80

Inspecting the kubelet

# Check kubelet service status
systemctl status kubelet
journalctl -u kubelet -f --no-pager

# View kubelet configuration
kubectl proxy &
curl -s http://localhost:8001/api/v1/nodes/$(hostname)/proxy/configz | python3 -m json.tool

# Check node status as reported by kubelet
kubectl describe node worker-1

# Key conditions to look for:
# Ready        - kubelet is healthy and ready to accept pods
# MemoryPressure - node is running low on memory
# DiskPressure   - node is running low on disk space
# PIDPressure    - too many processes on the node

Static Pods

Static pods are a special category managed directly by the kubelet without involving the API server for creation. The kubelet monitors a directory (default /etc/kubernetes/manifests) for pod manifests:

# On a kubeadm control plane node:
ls /etc/kubernetes/manifests/
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# The kubelet creates these pods and also creates "mirror pods" in the API server
# so they are visible via kubectl:
kubectl get pods -n kube-system | grep controlplane

Pod Admission and Resource Management

Before starting a pod, the kubelet performs admission checks:

1. Resource validation -- Ensures the node has enough resources for the pod's requests
2. Pod admission plugins -- Checks eviction thresholds, PID limits, etc.
3. Security context -- Validates the pod's security settings against node policies

The kubelet also manages resource enforcement through cgroups:

# View cgroup configuration for a pod
# The kubelet creates cgroup hierarchies for each pod and container
ls /sys/fs/cgroup/kubepods/

# Check resource usage at the pod level
kubectl top pod my-pod

# View detailed pod resource allocation
kubectl get pod my-pod -o jsonpath='{.spec.containers[*].resources}'

Eviction

When node resources become scarce, the kubelet performs eviction to protect node stability. It monitors resources against configured thresholds and evicts pods in priority order:

# Eviction signals and their default thresholds:
# memory.available < 100Mi    -> evict pods
# nodefs.available < 10%      -> evict pods
# imagefs.available < 15%     -> evict pods
# nodefs.inodesFree < 5%      -> evict pods

# Check current eviction configuration
kubectl get node worker-1 -o jsonpath='{.status.conditions}' | python3 -m json.tool

Certificate Management

The kubelet uses TLS certificates to authenticate with the API server. It supports automatic certificate rotation:

# kubelet certificates location
ls /var/lib/kubelet/pki/
# kubelet-client-current.pem  kubelet.crt  kubelet.key

# The kubelet.conf kubeconfig file references these certificates
cat /etc/kubernetes/kubelet.conf