High Availability with Leader Election¶

Run multiple controller replicas so a leader crash, node drain, or upgrade never stalls HAProxy configuration delivery.

Overview¶

The controller supports running multiple replicas for high availability using leader election based on Kubernetes Leases. Only the elected leader performs the render-validate-deploy work; all replicas keep their Kubernetes-resource caches warm and serve admission webhook requests so failover is instant.

Benefits of HA deployment:

Zero-downtime during controller upgrades (rolling updates)
Automatic failover if leader pod crashes (~30-35 seconds; voluntary handoffs like rolling updates release the lease immediately)
All replicas ready to take over immediately (hot caches, ready webhooks)
Balanced leader distribution across nodes

How it works:

All replicas watch Kubernetes resources, run the admission webhook, and discover HAProxy pods. Only the elected leader runs the render-validate-deploy pipeline. See Leader Election for the full all-replica vs leader-only component split.
Leader election determines which replica drives the pipeline and pushes configs via the Dataplane API.
When the leader fails, followers automatically elect a new one. Cached state from all-replica components (validated config, discovered HAProxy pods) is replayed on BecameLeaderEvent so the new leader starts with current state; the reconciler also fires immediately so the new leader produces a fresh render.
Leadership transitions are logged and tracked via Prometheus metrics.

Configuration¶

Enable Leader Election¶

Leader election is enabled by default when deploying with 2+ replicas via Helm:

# values.yaml (chart defaults)
replicaCount: 2  # Run 2 replicas for HA

controller:
  config:
    controller:
      leaderElection:
        enabled: true
        leaseName: ""         # Defaults to the Helm release fullname
        leaseDuration: 30s    # Max time followers wait before taking over
        renewDeadline: 20s    # Leader retries renewal for this long
        retryPeriod: 5s       # Interval between renewal attempts

Note

These match the controller's built-in defaults (pkg/core/config/defaults.go: 30s / 20s / 5s) — deliberately 2x the values kube-controller-manager and kube-scheduler ship with (15s / 10s / 2s). The extra renewDeadline headroom lets the leader ride out multi-second API-server or CPU starvation stalls without losing the lease; losing it costs a full controller reinitialization before the replica can lead again. Tune down toward the client-go convention if you prefer faster crash-failover over starvation headroom.

Disable Leader Election¶

For development or single-replica deployments:

# values.yaml
replicaCount: 1

controller:
  config:
    controller:
      leaderElection:
        enabled: false  # Disabled in single-replica mode

Timing Parameters¶

The timing parameters control failover speed and tolerance:

Parameter	Chart default	Purpose	Recommendations
`leaseDuration`	30s	Max time followers wait before taking over	Increase for flaky networks (60s+)
`renewDeadline`	20s	How long leader retries before giving up	Must be < `leaseDuration`
`retryPeriod`	5s	Interval between leader renewal attempts	Should be < `renewDeadline`

Failover time calculation:

Worst-case failover = leaseDuration + retryPeriod
Chart default       = 30s + 5s = ~35s (typically faster)

When the leader crashes without releasing, followers must wait for the lease to expire (leaseDuration from the last renewal) plus at most one acquire retry (retryPeriod) before one of them takes over; renewDeadline only bounds when the failed leader itself gives up, it does not gate the standby. During this window, HAProxy continues serving traffic with its last known configuration — no traffic is dropped, but new resource changes are not processed until a new leader is elected.

Clock skew tolerance:

Skew tolerance = leaseDuration - renewDeadline
Chart default  = 30s - 20s = 10s

If clock skew exceeds this tolerance, brief split-brain may occur where two replicas both believe they are leader. NTP-synchronized nodes typically have sub-second skew, well within the default tolerance. In environments with looser time sync, raise leaseDuration (and renewDeadline proportionally).

Deployment¶

Standard HA Deployment¶

Deploy with 2-3 replicas (default Helm configuration):

helm install haptic oci://registry.gitlab.com/haproxy-haptic/haptic/charts/haptic \
  --version 0.2.0-alpha.1 --set replicaCount=2

Scaling¶

Scale the deployment dynamically:

# Scale to 3 replicas
kubectl scale deployment haptic-controller -n haptic --replicas=3

# Scale back to 2
kubectl scale deployment haptic-controller -n haptic --replicas=2

Autoscaling¶

The chart ships an optional HorizontalPodAutoscaler:

autoscaling:
  enabled: true
  minReplicas: 2      # keep at least 2 for failover
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

Because the controller is leader-elected, autoscaling adds webhook-serving and warm-cache capacity (and faster failover) — not render/validate/deploy throughput. The reconciliation pipeline always runs on the single elected leader regardless of replica count, so CPU-based scaling driven by the leader's reconcile load adds standbys rather than parallel workers. To scale HAProxy data-plane throughput, scale HAProxy instead (haproxy.keda autoscaling or more HAProxy replicas).

RBAC Requirements¶

The controller requires these additional permissions for leader election:

apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update"]

These are automatically configured in the Helm chart's ClusterRole.

Monitoring Leadership¶

Check Current Leader¶

The Lease resource is named after the chart fullname — haptic for helm install haptic …, but <release>-haptic when the release name doesn't contain "haptic". Override by setting controller.config.controller.leaderElection.leaseName.

# List leases in the release namespace
kubectl get lease -n haptic

# View Lease resource (replace <release> with your Helm release name)
kubectl get lease -n haptic <release> -o yaml

# Output shows current leader:
# spec:
#   holderIdentity: <release>-controller-7d9f8b4c6d-abc12

View Leadership Status in Logs¶

# Leader logs show:
kubectl logs -n haptic deployment/haptic-controller | grep -E "leader|election"

# Example output:
# level=INFO msg="Leader election started" identity=pod-abc12 lease=<release>
# level=INFO msg="Became leader: pod-abc12" identity=pod-abc12

Prometheus Metrics¶

Monitor leader election via metrics endpoint:

kubectl port-forward -n haptic deployment/haptic-controller 9090:9090
curl http://localhost:9090/metrics | grep leader_election

Key metrics:

# Current leader (should be 1 across all replicas)
sum(haptic_leader_election_is_leader)

# Identify which pod is leader
haptic_leader_election_is_leader{pod=~".*"} == 1

# Leadership transition rate (should be low)
rate(haptic_leader_election_transitions_total[1h])

Troubleshooting¶

Check these areas in order of likelihood:

RBAC permissions (most common) -- service account missing lease permissions
Environment variables -- POD_NAME / POD_NAMESPACE not injected
API server connectivity -- network policies or firewall blocking access
Clock skew -- NTP not configured or excessive drift between nodes

No Leader Elected¶

Symptoms:

No deployments happening
All replicas show is_leader=0
Logs show constant election failures

Common causes:

Missing RBAC permissions:

The controller's ServiceAccount name is the Helm release fullname (unless you overrode serviceAccount.name):

SA=$(kubectl get deployment haptic-controller -n haptic -o jsonpath='{.spec.template.spec.serviceAccountName}')
kubectl auth can-i get leases    --as=system:serviceaccount:haptic:$SA
kubectl auth can-i create leases --as=system:serviceaccount:haptic:$SA
kubectl auth can-i update leases --as=system:serviceaccount:haptic:$SA

Missing environment variables:

kubectl get pod <pod-name> -o yaml | grep -A2 "POD_NAME\|POD_NAMESPACE"

# Should show:
# - name: POD_NAME
#   valueFrom:
#     fieldRef:
#       fieldPath: metadata.name

API server connectivity:

kubectl logs <pod-name> | grep "connection refused\|timeout"

Multiple Leaders (Split-Brain)¶

Symptoms:

sum(haptic_leader_election_is_leader) > 1
Multiple pods deploying configs simultaneously
Conflicting deployments in HAProxy

This should never happen with proper Kubernetes Lease implementation. If it does:

Check for severe clock skew between nodes:
```
# On each node
timedatectl status
```
Verify Kubernetes API server health:
```
kubectl get --raw /healthz
```

Restart all controller pods:

kubectl rollout restart deployment haptic-controller -n haptic

Frequent Leadership Changes¶

Symptoms:

rate(haptic_leader_election_transitions_total[1h]) > 5
Logs show frequent "Lost leadership" / "Became leader" messages
Deployments failing intermittently

Common causes:

Resource contention - Leader pod can't renew lease in time:

kubectl top pods -n haptic
kubectl describe pod <leader-pod> | grep -A10 "Limits\|Requests"

Solution: Increase CPU/memory limits

Network issues - API server communication delays:
```
kubectl logs -n haptic <pod-name> | grep "lease renew\|deadline"
```
Solution: Increase leaseDuration and renewDeadline
Node issues - Leader pod node experiencing problems:
```
kubectl describe node <node-name>
```
Solution: Drain and investigate node

Leader Not Deploying¶

Symptoms:

One replica shows is_leader=1
No deployment errors in logs
HAProxy configs not updating

Diagnosis:

# Check leader logs for deployment activity
kubectl logs -n haptic <leader-pod> | grep -i "deploy"

# Verify leader-only components started (requires debug log level)
kubectl logs -n haptic <leader-pod> | grep -i "deployer starting\|deployment scheduler starting\|starting deployment components"

Common causes:

Deployment components failed to start (check logs for errors)
Rate limiting preventing deployment (check drift prevention interval)
HAProxy instances unreachable (check network connectivity)

Best Practices¶

Replica Count¶

Development:

1 replica with leaderElection.enabled: false

Staging:

2 replicas with leader election enabled

Production:

2-3 replicas across multiple availability zones
A PodDisruptionBudget (minAvailable: 1) is created automatically once replicaCount > 1 — no action needed. Tune or disable it via:
```
podDisruptionBudget:
  enabled: true
  minAvailable: 1
```

Resource Allocation¶

The leader does the heavy lifting (render + validate + deploy + status writes), but every replica still has the full Kubernetes-resource cache loaded in memory. CPU usage is markedly higher on the leader; memory is similar across leader and followers. Size them all the same so a freshly-elected follower handles peak load without resizing — the chart defaults already do this:

# chart default — sized for the typical 50–200 Ingress range
resources:
  requests:
    cpu: 100m
    memory: 512Mi      # memory request = limit (pod stays Burstable — no CPU limit)
  limits:
    memory: 512Mi      # CPU limit deliberately omitted to avoid GOMAXPROCS throttling

For larger or smaller workloads see the sizing table in Performance — Controller Resource Sizing. Don't shrink memory below what the watch-set needs (rule of thumb: ~1KB per Ingress + EndpointSlice churn) or the leader will OOMKill mid-deploy and the lease will flap.

Anti-Affinity¶

Distribute replicas across nodes for better availability:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: haptic
          topologyKey: kubernetes.io/hostname

Monitoring and Alerts¶

Set up Prometheus alerts for leader election health:

groups:
  - name: haproxy-ic-leader-election
    rules:
      # No leader
      - alert: NoLeaderElected
        expr: sum(haptic_leader_election_is_leader) < 1
        for: 1m
        annotations:
          summary: "No HAProxy controller leader elected"

      # Multiple leaders (split-brain)
      - alert: MultipleLeaders
        expr: sum(haptic_leader_election_is_leader) > 1
        annotations:
          summary: "Multiple HAProxy controller leaders detected (split-brain)"

      # Frequent transitions
      - alert: FrequentLeadershipChanges
        expr: rate(haptic_leader_election_transitions_total[1h]) > 5
        for: 15m
        annotations:
          summary: "HAProxy controller experiencing frequent leadership changes"

Migration from Single-Replica¶

To migrate an existing single-replica deployment to HA:

Verify RBAC permissions (Helm chart updates this automatically)

Update values.yaml:

replicaCount: 2
controller:
  config:
    controller:
      leaderElection:
        enabled: true

Upgrade with Helm:

helm upgrade haptic oci://registry.gitlab.com/haproxy-haptic/haptic/charts/haptic \
  -n haptic --reuse-values \
  -f values.yaml

Verify leadership:

kubectl logs -f -n haptic deployment/haptic-controller | grep leader

Confirm one leader:

# Query each pod's metrics; exactly one should report is_leader 1
for pod in $(kubectl get pods -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller -o name); do
  echo "$pod:"
  kubectl exec -n haptic $pod -- wget -qO- localhost:9090/metrics | grep is_leader
done

High Availability with Leader Election¶

Overview¶

Configuration¶

Enable Leader Election¶

Disable Leader Election¶

Timing Parameters¶

Deployment¶

Standard HA Deployment¶

Scaling¶

Autoscaling¶

RBAC Requirements¶

Monitoring Leadership¶

Check Current Leader¶

View Leadership Status in Logs¶

Prometheus Metrics¶

Troubleshooting¶

No Leader Elected¶

Multiple Leaders (Split-Brain)¶

Frequent Leadership Changes¶

Leader Not Deploying¶

Best Practices¶

Replica Count¶

Resource Allocation¶

Anti-Affinity¶

Monitoring and Alerts¶

Migration from Single-Replica¶

See Also¶