K8s by Example: Node Components

Worker nodes run your workloads. Each node has: kubelet (Pod lifecycle), kube-proxy (networking), and container runtime (runs containers). Nodes register with control plane and receive work via the API Server.

node-architecture.txt

The kubelet manages Pods, kube-proxy handles networking rules, and the container runtime executes containers.

┌─────────────────────────────────────────────────────┐
│                    Worker Node                      │
│  ┌──────────────────────────────────────────────┐   │
│  │                   kubelet                    │   │
│  └──────────────────────────────────────────────┘   │
│  ┌─────────────────┐  ┌─────────────────────────┐   │
│  │   kube-proxy    │  │   Container Runtime     │   │
│  └─────────────────┘  └─────────────────────────┘   │
│  ┌──────────────────────────────────────────────┐   │
│  │              Pods (containers)               │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

terminal

The kubelet is the node agent. It watches for Pods assigned to its node, starts containers, runs probes, and reports status. It doesn’t manage non-Kubernetes containers.

$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled)
   Active: active (running) since Mon 2024-01-15 10:00:00 UTC

	terminal
View kubelet logs for troubleshooting. The config file at `/var/lib/kubelet/config.yaml` contains runtime settings.	`$ journalctl -u kubelet -f Jan 15 10:30:00 node-1 kubelet: I0115 10:30:00 node.go:123] Node node-1 ready Jan 15 10:30:05 node-1 kubelet: I0115 10:30:05 kubelet.go:456] SyncLoop ADD`

	terminal
Kubelet communicates with container runtime via CRI (Container Runtime Interface). containerd and CRI-O are the main production runtimes.	`$ kubectl get nodes -o wide NAME STATUS VERSION CONTAINER-RUNTIME node-1 Ready v1.29.0 containerd://1.7.0 node-2 Ready v1.29.0 containerd://1.7.0`

	terminal
Use `crictl` to interact directly with the container runtime. Useful for debugging container issues.	`$ crictl ps CONTAINER IMAGE STATE NAME abc123 nginx:latest Running nginx def456 redis:7 Running redis $ crictl images IMAGE TAG SIZE docker.io/nginx latest 142MB docker.io/redis 7 130MB`

terminal

kube-proxy implements Services networking. It watches Services and Endpoints, then configures iptables or IPVS rules to route traffic to Pods.

$ kubectl logs -n kube-system -l k8s-app=kube-proxy | head -10
I0115 10:00:00 server.go:225] Using iptables Proxier
I0115 10:00:00 config.go:315] Starting service config controller
I0115 10:00:00 config.go:224] Starting endpoint slice config controller

terminal

Default mode is iptables - creates NAT rules for each Service. IPVS mode is better for large clusters with many Services.

$ iptables -t nat -L KUBE-SERVICES -n | head -10
Chain KUBE-SERVICES (2 references)
target     prot opt source    destination
KUBE-SVC-XYZ  tcp  --  0.0.0.0/0  10.96.0.1    /* default/kubernetes:https */
KUBE-SVC-ABC  tcp  --  0.0.0.0/0  10.96.0.10   /* kube-system/kube-dns:dns */

terminal

Node status is reported by kubelet. Conditions show health: Ready, MemoryPressure, DiskPressure, PIDPressure. All except Ready should be False.

$ kubectl describe node node-1 | grep -A10 Conditions
Conditions:
  Type             Status  Reason
  ----             ------  ------
  MemoryPressure   False   KubeletHasSufficientMemory
  DiskPressure     False   KubeletHasNoDiskPressure
  PIDPressure      False   KubeletHasSufficientPID
  Ready            True    KubeletReady

	terminal
Allocatable shows resources available for Pods after system overhead. Some capacity is reserved for system processes.	`$ kubectl describe node node-1 \| grep -A6 Capacity Capacity: cpu: 4 memory: 16Gi pods: 110 Allocatable: cpu: 3800m memory: 15Gi`

	terminal
Node registration adds automatic labels: hostname, OS, arch, zone. Cloud controllers add instance-type and region labels.	`$ kubectl get node node-1 --show-labels NAME STATUS LABELS node-1 Ready kubernetes.io/hostname=node-1, kubernetes.io/os=linux, kubernetes.io/arch=amd64, topology.kubernetes.io/zone=us-east-1a`

	terminal
Cordon marks a node as unschedulable. New Pods won’t be placed on it, but existing Pods keep running.	`$ kubectl cordon node-1 node/node-1 cordoned $ kubectl get nodes NAME STATUS ROLES AGE node-1 Ready,SchedulingDisabled worker 30d node-2 Ready worker 30d`

terminal

Drain evicts all Pods from a node, respecting PodDisruptionBudgets. Use for maintenance, upgrades, or decommissioning.

$ kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
node/node-1 already cordoned
evicting pod default/nginx-abc123
evicting pod default/redis-xyz456
pod/nginx-abc123 evicted
pod/redis-xyz456 evicted
node/node-1 drained

	terminal
Uncordon returns a node to service, allowing new Pods to be scheduled on it.	`$ kubectl uncordon node-1 node/node-1 uncordoned`

	terminal
Debug node issues by checking resource usage, kubelet logs, and container runtime status.	`$ kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% node-1 250m 6% 4096Mi 25% node-2 180m 4% 3584Mi 22%`

	terminal
Use `kubectl debug` to get a shell on a node for advanced troubleshooting. This creates a privileged Pod with host access.	`$ kubectl debug node/problem-node -it --image=ubuntu Creating debugging pod node-debugger-problem-node-abc123 root@problem-node:/# cat /host/var/log/syslog \| tail root@problem-node:/# exit`

Index | GitHub | Use arrow keys to navigate |