Manage agent certificates¶
The default chart creates and automatically renews the certificate authority (CA) and both controller-to-agent identities. It doesn't require cert-manager. You can select cert-manager explicitly or supply externally managed identity Secrets. Both peers reload mounted certificates without restarting.
The commands below use Bash, kubectl, jq, Python 3, and OpenSSL, with release
haptic in namespace haptic. You need permission to inspect Secrets and Jobs;
manual certificate replacement also requires updating Secrets and exec access.
For the authentication model, see Security.
Default renewal¶
The bootstrap Job creates certificates after chart preflight validation. An
hourly CronJob checks them and renews the CA, its key, and both identity keys when
30 days remain. With haproxy.agent.tls.certValidityDays below 90 days, renewal
starts with one third of the lifetime remaining. The default lifetime is 365 days.
Changing the value affects the next issuance; it doesn't immediately replace
certificates that aren't due for renewal.
The issuer Secret stores the private CA key and the current certificate generation. Back up this Secret securely. Neither controller nor agent mounts it. Helm upgrades and offline rendering preserve certificates because jobs reconcile the durable state through the Kubernetes API.
During renewal, the new identities can authenticate against the old CA for one hour, and both peers temporarily trust the previous CA. This covers staggered Secret projections. Previous trust expires even if a later cleanup Job fails. An interrupted Job resumes the stored generation instead of creating another CA.
Monitor renewal failures
Authentication stops if an identity expires or a partially completed CA rotation outlasts the one-hour trust overlap. The controller can't deliver routing, endpoint, or certificate updates, while running HAProxy workers retain their last configuration. Fix the failed Job and rerun it; renewal uses Kubernetes credentials and can recover expired agent identities. Local socket liveness probes keep the agent running during certificate failure so it can load the repaired identity.
Inspect recent runs:
kubectl get cronjob haptic-agent-renewal -n haptic
kubectl get jobs -n haptic --sort-by=.metadata.creationTimestamp
Run a check immediately:
job="haptic-agent-renewal-$(date +%s)"
kubectl create job "$job" --from=cronjob/haptic-agent-renewal -n haptic
kubectl wait --for=condition=complete "job/$job" -n haptic --timeout=5m
kubectl logs "job/$job" -n haptic
Don't patch internally managed identity data: the next Job restores its saved generation. A missing identity Secret is recreated. A missing or corrupt issuer Secret requires restoring its backup; the Job refuses to replace existing identities with an unrelated CA. Runtime-created Secrets survive Helm uninstall.
Use cert-manager¶
Install cert-manager before selecting this mode. The chart doesn't auto-detect it or change an existing installation's issuer when cert-manager appears.
For a new release:
helm install haptic \
oci://registry.gitlab.com/haproxy-haptic/haptic/charts/haptic \
--namespace haptic --create-namespace \
--set haproxy.agent.tls.certManager.enabled=true
The chart creates a self-signed root Certificate, a CA Issuer, and separate
server and client Certificate resources. Cert-manager renews the identities
30 days before expiry and replaces their private keys. For lifetimes below
90 days, renewal starts with one third of the lifetime remaining.
The root certificate lasts at least ten years and renews with the same CA key, at least two identity lifetimes before expiry. Subsequent identity renewals distribute the renewed root certificate. This renews the root certificate; it doesn't rotate the CA key. Cert-manager's CA issuer alone doesn't renew its CA or immediately replace dependent certificates.
To select your existing issuer, use these values:
haproxy:
agent:
tls:
certManager:
enabled: true
createIssuer: false
issuerRef:
name: organisation
kind: ClusterIssuer
group: cert-manager.io
The issuer must issue both TLS roles and populate ca.crt in each identity
Secret. You own that issuer's CA lifetime and trust transitions.
To change providers on an existing release, use new issuer and identity Secret names and expect a controller and agent rollout. Keep the old Secrets until both workloads are ready. A provider must not take over another provider's Secrets.
To switch from the default manager to the chart-created cert-manager issuer, install cert-manager, then run:
umask 077
helm get values haptic --namespace haptic --output yaml > haptic-current-values.yaml
helm upgrade haptic \
oci://registry.gitlab.com/haproxy-haptic/haptic/charts/haptic \
--namespace haptic --reset-values --values haptic-current-values.yaml \
--set haproxy.agent.tls.certManager.enabled=true \
--set haproxy.agent.tls.issuerSecretName=haptic-cm-issuer \
--set haproxy.agent.tls.serverSecretName=haptic-cm-agent \
--set haproxy.agent.tls.clientSecretName=haptic-cm-controller
kubectl rollout status deployment/haptic-controller -n haptic
kubectl rollout status deployment/haptic-haproxy -n haptic
Check expiry¶
Check both identities and their CA. This command prints expiry dates and exits with an error if any certificate expires within 30 days or can't be read. It reads only public certificates from the Secrets.
set -euo pipefail
needs_rotation=0
for deployment in haptic-controller haptic-haproxy; do
secret=$(kubectl get deployment "$deployment" -n haptic -o json |
jq -er '.spec.template.spec.volumes[] | select(.name == "agent-tls").secret.secretName')
for key in tls.crt ca.crt; do
printf '%s %s: ' "$secret" "$key"
if ! kubectl get secret "$secret" -n haptic -o json |
jq -er --arg key "$key" '.data[$key]' | base64 -d |
openssl x509 -noout -enddate -checkend 2592000; then
needs_rotation=1
fi
done
done
test "$needs_rotation" -eq 0
Run the check regularly through your monitoring system. Certificates that stay inside the renewal window indicate a failed renewal process. With an external issuer, monitor its CA lifetime as well as the identity lifetimes.
Supply external identities¶
Prepare agent.crt, agent.key, controller.crt, controller.key, and ca.crt
from your certificate issuer. The agent certificate needs the serverAuth
extended key usage and DNS SAN agent.example.internal. The controller
certificate needs clientAuth and DNS SAN controller.example.internal.
ca.crt must trust the peer's issuer. Each certificate file includes any
intermediate certificates after its leaf certificate.
-
Create the namespace if it doesn't exist.
-
Create the two identity Secrets.
kubectl create secret generic haptic-agent-identity -n haptic \ --type=kubernetes.io/tls \ --from-file=tls.crt=agent.crt --from-file=tls.key=agent.key \ --from-file=ca.crt=ca.crt kubectl create secret generic haptic-controller-identity -n haptic \ --type=kubernetes.io/tls \ --from-file=tls.crt=controller.crt --from-file=tls.key=controller.key \ --from-file=ca.crt=ca.crtIf your certificate manager already supplies these Secrets with these keys, proceed to the next step. Use your deployment's Secret names and certificate SANs in the values file when they differ from this example.
-
Write the TLS values.
-
Include the file when installing or upgrading the release.
For a new release:
helm install haptic \ oci://registry.gitlab.com/haproxy-haptic/haptic/charts/haptic \ --namespace haptic --values agent-tls-values.yamlFor an existing release, save its custom values and include them in the upgrade:
umask 077 helm get values haptic --namespace haptic --output yaml > haptic-current-values.yaml helm upgrade haptic \ oci://registry.gitlab.com/haproxy-haptic/haptic/charts/haptic \ --namespace haptic --reset-values \ --values haptic-current-values.yaml --values agent-tls-values.yamlThe upgrade applies new chart defaults, your saved custom values, and then the TLS settings. Keep these externally supplied identities under your Secret manager for subsequent renewal.
Rotate a certificate authority¶
This procedure applies to external Secrets you control directly with
haproxy.agent.tls.managed: false. Complete each stage before starting the next.
If another certificate manager owns the Secrets, coordinate these stages through
that manager so it doesn't restore the old data.
Existing identities must remain valid throughout the overlap. The example grants
one hour of overlap; HAPTIC rejects deadlines more than 24 hours away. Renewing
only leaf certificates under an unchanged CA doesn't require a trust change.
Update each tls.crt/tls.key pair atomically and wait for its projected files.
Prepare replacement identities¶
-
Identify the installed Secrets and required peer names.
set -euo pipefail umask 077 rotation_dir=$(mktemp -d) server_secret=$(kubectl get deployment haptic-haproxy -n haptic -o json | jq -r '.spec.template.spec.volumes[] | select(.name == "agent-tls").secret.secretName') client_secret=$(kubectl get deployment haptic-controller -n haptic -o json | jq -r '.spec.template.spec.volumes[] | select(.name == "agent-tls").secret.secretName') agent_name=$(kubectl get deployment haptic-controller -n haptic -o json | jq -r '.spec.template.spec.containers[] | select(.name == "controller").env[] | select(.name == "AGENT_TLS_SERVER_NAME").value') controller_name=$(kubectl get deployment haptic-haproxy -n haptic -o json | jq -r '.spec.template.spec.initContainers[] | select(.name == "agent").args[] | select(startswith("--tls-client-name=")) | ltrimstr("--tls-client-name=")') test -n "$server_secret" && test -n "$client_secret" test -n "$agent_name" && test -n "$controller_name" -
Generate a new CA and role-specific identities.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \ -keyout "$rotation_dir/ca.key" -out "$rotation_dir/ca.crt" -days 365 \ -subj /CN=haptic-agent-ca \ -addext 'basicConstraints=critical,CA:TRUE' \ -addext 'keyUsage=critical,keyCertSign,cRLSign' for role in agent controller; do name=$agent_name usage=serverAuth if [ "$role" = controller ]; then name=$controller_name; usage=clientAuth; fi openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \ -keyout "$rotation_dir/$role.key" -out "$rotation_dir/$role.csr" \ -subj "/CN=$name" printf 'subjectAltName=DNS:%s\nextendedKeyUsage=%s\nkeyUsage=critical,digitalSignature\nbasicConstraints=critical,CA:FALSE\n' \ "$name" "$usage" > "$rotation_dir/$role.ext" openssl x509 -req -in "$rotation_dir/$role.csr" \ -CA "$rotation_dir/ca.crt" -CAkey "$rotation_dir/ca.key" \ -set_serial "0x$(openssl rand -hex 16)" -days 365 \ -extfile "$rotation_dir/$role.ext" -out "$rotation_dir/$role.crt" done -
Define a check for the projected files on every replica.
wait_projected() { local component=$1 container=$2 file=$3 expected=$4 local deadline=$((SECONDS + 180)) pods pod ready while [ "$SECONDS" -lt "$deadline" ]; do pods=$(kubectl get pods -n haptic \ -l "app.kubernetes.io/instance=haptic,app.kubernetes.io/component=$component" \ -o json | jq -r '.items[] | select(.metadata.deletionTimestamp == null).metadata.name') ready=true [ -n "$pods" ] || ready=false for pod in $pods; do if [ "$expected" = absent ]; then kubectl exec -n haptic "$pod" -c "$container" -- \ test ! -e "/etc/haptic/agent-tls/..data/$file" || ready=false elif ! kubectl exec -n haptic "$pod" -c "$container" -- \ cat "/etc/haptic/agent-tls/..data/$file" | cmp -s "$expected" -; then ready=false fi done [ "$ready" = true ] && return 0 sleep 2 done echo "TLS files haven't reached every $component replica; repair projection before continuing" >&2 return 1 }Secret projection is asynchronous. A successful Secret update alone doesn't establish that every peer trusts the new CA.
Distribute overlapping trust¶
-
Add the new CA while retaining each peer's previous trust with a deadline.
deadline=$(python3 -c 'from datetime import datetime,timedelta,timezone; print((datetime.now(timezone.utc)+timedelta(hours=1)).strftime("%Y-%m-%dT%H:%M:%SZ"))') for secret in "$server_secret" "$client_secret"; do kubectl get secret "$secret" -n haptic -o json | jq -e '.data["previous-ca.crt"] == null' >/dev/null kubectl get secret "$secret" -n haptic -o jsonpath='{.data.ca\.crt}' | base64 -d > "$rotation_dir/$secret-old-ca.crt" jq -n --rawfile ca "$rotation_dir/ca.crt" \ --rawfile previous "$rotation_dir/$secret-old-ca.crt" --arg until "$deadline" \ '{data:{"ca.crt":($ca|@base64),"previous-ca.crt":($previous|@base64),"previous-ca-until":($until|@base64)}}' | kubectl patch secret "$secret" -n haptic --type=merge --patch-file=/dev/stdin doneComplete any existing rotation before starting another one. For an interrupted rotation, follow the recovery steps below.
-
Wait for both sides to receive the trust bundle.
Replace identities¶
-
Update each certificate and key together.
for role in agent controller; do secret=$server_secret [ "$role" != controller ] || secret=$client_secret jq -n --rawfile cert "$rotation_dir/$role.crt" --rawfile key "$rotation_dir/$role.key" \ '{data:{"tls.crt":($cert|@base64),"tls.key":($key|@base64)}}' | kubectl patch secret "$secret" -n haptic --type=merge --patch-file=/dev/stdin done -
Wait for every replica to receive its new identity.
-
Verify the authenticated controller-to-agent connection.
Revoke the previous CA¶
-
Remove the previous CA and deadline from both Secrets.
for secret in "$server_secret" "$client_secret"; do kubectl patch secret "$secret" -n haptic --type=merge \ -p '{"data":{"previous-ca.crt":null,"previous-ca-until":null}}' done wait_projected loadbalancer agent previous-ca.crt absent wait_projected controller controller previous-ca.crt absentOld clients now fail authentication, including on reused connections. Requests authenticated before revocation can finish within their operation deadlines. A retained
previous-ca.crtalso stops granting trust when its deadline passes. -
Remove the temporary private keys after storing any material your certificate issuer needs.
Recover an interrupted rotation¶
Before the overlap expires, resume at the first incomplete stage. Check every replica's projected files before revoking trust. Invalid certificate or trust files reject new management operations; they don't cause an HTTP fallback.
If the deadline expires before identities are replaced, finish projecting the new identities through the Kubernetes API. Existing HAProxy workers keep their last valid configuration. Restore any missing external identity Secret from your backup, or have its issuer recreate it under the intended CA.