K8s by Example: Leader Election

Leader election ensures only one instance of a distributed application performs certain tasks at a time. Kubernetes provides leader election via Lease objects. The leader holds the lease; if it fails, another instance acquires it. Use for: singleton workers, scheduled tasks, controller patterns, avoiding duplicate processing.
leader-election-deployment.yaml

Deploy multiple replicas where only one is the active leader. The sidecar container handles leader election using the Kubernetes API. The main app checks a local file or endpoint to determine if it’s the leader.

 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
spec:
  replicas: 3
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
    spec:
      serviceAccountName: leader-election
      containers:
        - name: worker
          image: my-worker:v1
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

The leader-election sidecar uses the leader-elector image. It creates a Lease object and continuously renews it. When this Pod is the leader, it exposes that status via a local endpoint or file.

 
        - name: leader-elector
          image: registry.k8s.io/leader-elector:0.5
          args:
            - --election=worker-election
            - --http=0.0.0.0:4040
          ports:
            - containerPort: 4040
leader-election-rbac.yaml

Leader election requires permissions to create and update Lease objects. The ServiceAccount needs get, create, and update on leases in the coordination.k8s.io API group.

 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: leader-election
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: leader-election
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: leader-election
subjects:
  - kind: ServiceAccount
    name: leader-election
roleRef:
  kind: Role
  name: leader-election
  apiGroup: rbac.authorization.k8s.io
lease.yaml

A Lease object stores leader election state. holderIdentity is the current leader. leaseDurationSeconds is how long the lease is valid. renewTime is when the leader last renewed. If the leader fails to renew, another Pod can acquire it.

 
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: worker-election
spec:
  holderIdentity: worker-7d8f9b6c4d-abc12
  leaseDurationSeconds: 15
  acquireTime: "2024-01-15T10:30:00Z"
  renewTime: "2024-01-15T10:31:45Z"
  leaseTransitions: 3
leader-election-native.yaml

Many applications implement leader election natively using client-go or similar libraries. Configure lease name, namespace, duration, and callbacks for when leadership is acquired or lost.

 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller
spec:
  replicas: 2
  template:
    spec:
      serviceAccountName: leader-election
      containers:
        - name: controller
          image: my-controller:v1
          args:
            - --leader-elect=true
            - --leader-election-id=controller-leader
            - --leader-election-namespace=$(POD_NAMESPACE)
            - --leader-election-lease-duration=15s
            - --leader-election-renew-deadline=10s
            - --leader-election-retry-period=2s
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
terminal

Inspect leases to see current leaders. The HOLDER column shows which Pod holds the lease. Watch for leadership transitions during failures or deployments.

 
$ kubectl get leases
NAME               HOLDER                      AGE
worker-election    worker-7d8f9b6c4d-abc12     1h
controller-leader  controller-5f6g7h8i9j-xyz   30m

$ kubectl describe lease worker-election
Name:         worker-election
Holder:       worker-7d8f9b6c4d-abc12
Acquire Time: 2024-01-15T10:30:00Z
Renew Time:   2024-01-15T11:31:45Z
Transitions:  3

$ kubectl get pods -l app=worker
NAME                      READY   STATUS    ROLE
worker-7d8f9b6c4d-abc12   2/2     Running   leader
worker-7d8f9b6c4d-def34   2/2     Running   standby
worker-7d8f9b6c4d-ghi56   2/2     Running   standby

Index | GitHub | Use arrow keys to navigate |