Troubleshooting

Common issues and solutions for HAPTIC.

Namespace

All kubectl commands below assume the default installation namespace haptic. Replace -n haptic with your namespace if you installed elsewhere.

Quick Symptom Reference

Symptom	Section
Pod in CrashLoopBackOff	Controller Not Starting
Pods running, no reconciliation activity	Controller Running But Not Processing
"template rendering failed" in logs	Invalid Template Syntax
"validation failed" / HAProxy errors	Configuration Validation Failures
"connection refused" to Dataplane API	Cannot Connect to Dataplane API
Controller reports success but HAProxy unchanged	Configuration Not Updating
503 errors / no servers in HAProxy stats	Requests Not Reaching Backend
SSL handshake failures	SSL/TLS Issues
High CPU or slow reconciliation	Slow Reconciliation
OOMKilled / gradual memory growth	High Memory Usage
"shm-stats-file-max-objects" / reload failures	Shared Memory Stats Limit

Controller Issues

Controller Not Starting

Symptoms: CrashLoopBackOff, repeated restarts, initialization errors

Diagnosis:

kubectl get pods -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller
kubectl logs -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller --tail=100
kubectl describe pod -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller

Common Causes:

Cause	Check	Solution
Missing HAProxyTemplateConfig	kubectl get haproxytemplateconfig	Reinstall Helm chart
Invalid credentials Secret	kubectl get secret -n haptic haptic-credentials -o jsonpath='{.data}' (Helm names it <release>-credentials)	Recreate secret with correct keys
RBAC permissions	kubectl auth can-i list ingresses --all-namespaces --as=system:serviceaccount:<ns>:<sa>	Verify ClusterRole/ClusterRoleBinding

Controller Running But Not Processing

Symptoms: Pods running, no reconciliation activity

Diagnosis:

kubectl logs -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller | grep -i "watch\|sync complete"

Common Causes:

Cause	Check	Solution
Informers not syncing	Logs show "timeout waiting for cache sync"	Check API server connectivity, network policies
No matching resources	kubectl get ingresses -A	Verify resources exist in watched namespaces
Ingress class mismatch	kubectl get ingress <name> -o jsonpath='{.spec.ingressClassName}'	The Ingress must reference the class the chart created; also check any watchedResources.*.fieldSelector namespace filter
Leader election (HA)	kubectl get lease -n haptic (the Lease is named after the Helm release)	Ensure one pod shows is_leader=1

Configuration Issues

Invalid Template Syntax

Symptoms: "template rendering failed" errors

Diagnosis:

kubectl logs -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller | grep -i "template\|render"

Solution:

1. Check template syntax in HAProxyTemplateConfig

2. Inspect the last rendered output via the debug server — port-forward first (see Debugging Guide):

   kubectl port-forward -n haptic deployment/haptic-controller 8080:8080
   curl http://localhost:8080/debug/vars/rendered
   

3. See Templating Guide

Configuration Validation Failures

Symptoms: "validation failed", HAProxy errors

Common Errors:

Error	Cause	Solution
backend expects <name>	Invalid HAProxy syntax	Fix template, test with haproxy -c -f config.cfg
unable to load file	Missing map/cert file	Define in maps section, use pathResolver.GetPath()
invalid address	Bad server address	Verify EndpointSlices exist, check service names

Validation Test Failures

Symptoms: haptic-controller validate fails

Quick Debugging:

# Step 1: Run with verbose output
haptic-controller validate -f config.yaml --verbose

# Step 2: See full rendered content
haptic-controller validate -f config.yaml --dump-rendered

# Step 3: Check template execution
haptic-controller validate -f config.yaml --trace-templates

See Validation Tests for detailed debugging.

HAProxy Pod Issues

Cannot Connect to Dataplane API

Symptoms: "connection refused", "timeout", deployment failures

Diagnosis:

HAPROXY_POD=$(kubectl get pods -n haptic -l app.kubernetes.io/component=loadbalancer -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n haptic $HAPROXY_POD 5555:5555
# Substitute your actual dataplane password; see spec.credentialsSecretRef
curl -u admin:<password> http://localhost:5555/v3/info

Common Causes:

Cause	Check	Solution
Dataplane not running	kubectl logs $HAPROXY_POD -c dataplane	Verify container started, check port conflicts
Wrong credentials	Compare secret vs dataplaneapi.yaml	Update the credentials Secret — credentialsloader picks it up live; also rotate the matching dataplaneapi.yaml on the HAProxy sidecar
Network policy	kubectl get networkpolicy	Update egress rules for controller → HAProxy

Configuration Not Updating

Symptoms: Controller shows success but HAProxy has old config

Diagnosis:

HAPROXY_POD=$(kubectl get pods -n haptic -l app.kubernetes.io/component=loadbalancer -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n haptic $HAPROXY_POD -c haproxy -- ls -lh /etc/haproxy/haproxy.cfg
kubectl logs -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller | grep -i "deployment.*succeeded"

Common Causes:

Cause	Check	Solution
Volume mount issue	kubectl get pod $HAPROXY_POD -o yaml \| grep -A5 volumeMounts	Ensure both containers share config volume
HAProxy not reloading	kubectl logs $HAPROXY_POD -c dataplane	Check reload command, master socket access

Shared Memory Stats Limit

Opt-in feature

This only applies when haproxy.shmStats.enabled: true is set in Helm values (the default is false) and HAProxy is 3.3+ — the shm-stats file is gated by semver_gte in the chart templates. If you're on the default config you will not see these errors; this section is for operators who turned shm-stats on for performance.

Symptoms: 100% deployment error rate, HAProxy reload failures with shm-stats-file-max-objects errors

Diagnosis:

kubectl logs -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller | grep "shm-stats"

Look for:

[ALERT] memory error while setting up shared counters for .../SRV_N server:
Cannot add additional object to '/dev/shm/haproxy-stats' file,
maximum number already reached (50000).

Common Causes:

Cause	Check	Solution
Too many HAProxy objects for the configured limit	Count ingresses/services: kubectl get ingresses -A --no-headers \| wc -l	Increase haproxy.shmStats.maxObjects in Helm values
Cluster grew beyond initial sizing	Compare object count to maxObjects value	Recalculate using the formula below

Solution:

Each HAProxy frontend, backend, and server directive counts as one shm-stats object. The file is fixed-size and cannot be resized on reload. Increase haproxy.shmStats.maxObjects in your Helm values:

haproxy:
  shmStats:
    enabled: true
    maxObjects: 100000  # default: 50000

Sizing formula: (number of backends + number of servers) × 1.2 safety margin. Each object uses ~4KiB of shared memory. For example, 100,000 objects require ~390Mi in /dev/shm, which counts against the pod's memory limit.

Warning

After changing maxObjects, verify that haproxy.resources.limits.memory is large enough to accommodate the increased /dev/shm usage. The shm volume is memory-backed and counts against the pod's memory limit.

Routing Issues

Requests Not Reaching Backend

Symptoms: 503 errors, timeouts, no servers in HAProxy stats

Diagnosis:

HAPROXY_POD=$(kubectl get pods -n haptic -l app.kubernetes.io/component=loadbalancer -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n haptic $HAPROXY_POD -c haproxy -- cat /etc/haproxy/haproxy.cfg | grep -A10 "backend"
kubectl get endpointslices -l kubernetes.io/service-name=<service>

Common Causes:

Cause	Check	Solution
No endpoints	kubectl get endpointslices	Verify backend pods running and ready
Backend not created	Controller logs for backend errors	Review template logic, check Ingress references
Routing not matching	Test with curl -H "Host: ..."	Verify Host header, check ACLs and map files

SSL/TLS Issues

Symptoms: SSL handshake failures, certificate errors

Diagnosis:

HAPROXY_POD=$(kubectl get pods -n haptic -l app.kubernetes.io/component=loadbalancer -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n haptic $HAPROXY_POD -c haproxy -- ls -lh /etc/haproxy/ssl/

# Port-forward HAProxy's HTTPS port, then probe the handshake.
# Stop the forward with `kill %1` (or Ctrl+C) when done.
kubectl port-forward -n haptic $HAPROXY_POD 443:443 &
openssl s_client -connect localhost:443 -servername your-host.example.com < /dev/null

Common Causes:

Cause	Check	Solution
Certificate not deployed	Check sslCertificates section	Define template, watch secret, use b64decode
Wrong cert path	grep "bind.*ssl.*crt" haproxy.cfg	Use pathResolver.GetPath("cert.pem", "cert")

Performance Issues

Slow Reconciliation

Symptoms: Changes take minutes, high CPU

Diagnosis:

kubectl port-forward -n haptic deployment/haptic-controller 9090:9090
curl http://localhost:9090/metrics | grep reconciliation_duration_seconds

Solutions:

• Use namespace restrictions in watchedResources
• Add label selectors to filter resources
• Use cached store for large resources
• Optimize templates: cache values with {% var %}, reduce nested loops

High Memory Usage

Symptoms: OOMKilled events, gradual memory growth

Solutions:

# Filter large fields
watchedResourcesIgnoreFields:
  - metadata.managedFields
  - metadata.annotations['kubectl.kubernetes.io/last-applied-configuration']

# Use cached store for secrets (fetches on-demand; TTL is auto-derived
# from driftPreventionInterval, not user-configurable)
watchedResources:
  secrets:
    store: on-demand

# Limit watch scope
watchedResources:
  ingresses:
    namespace: production
    labelSelector: "app=myapp"

Getting Help

Collect Diagnostic Information

# Controller version
kubectl get deployment -n haptic haptic-controller -o jsonpath='{.spec.template.spec.containers[0].image}'

# Controller logs
kubectl logs -n haptic -l app.kubernetes.io/name=haptic,app.kubernetes.io/component=controller --tail=500 > controller-logs.txt

# Configuration
kubectl get haproxytemplateconfig -n haptic haptic-config -o yaml > config.yaml

# HAProxy config (sanitize sensitive data!)
kubectl exec -n haptic $HAPROXY_POD -c haproxy -- cat /etc/haproxy/haproxy.cfg > haproxy.cfg

Enable Debug Logging

The controller supports multiple log levels via the LOG_LEVEL environment variable (case-insensitive):

Level	Description
ERROR	Errors only
WARN (or WARNING)	Warnings and errors
INFO	Important state changes (default)
DEBUG	Detailed debugging information
TRACE	Very verbose, per-item iteration logs

# Enable debug logging
kubectl set env -n haptic deployment/haptic-controller LOG_LEVEL=DEBUG

# Enable trace logging (very verbose)
kubectl set env -n haptic deployment/haptic-controller LOG_LEVEL=TRACE

The log level can also be configured via the HAProxyTemplateConfig CRD's spec.logging.level field. When set, the CRD value takes precedence over the LOG_LEVEL environment variable, and changes take effect without a pod restart:

# In values.yaml
controller:
  logLevel: INFO  # Initial LOG_LEVEL env var (used until the CRD is loaded)
  config:
    logging:
      level: DEBUG  # Written to spec.logging.level — overrides env var at runtime

Note

TRACE level produces extremely verbose output, including per-resource iteration logs, HTTP fetch retries, and test runner details. Use only when debugging specific issues.

Access the Debug Server

The Helm chart enables the debug server on port 8080 by default (same port as /healthz). Port-forward to reach it:

kubectl port-forward -n haptic deployment/haptic-controller 8080:8080

/healthz and /debug/* share the same listener, so setting controller.debugPort: 0 disables both and breaks the Kubernetes liveness/readiness probes — restrict access via NetworkPolicy instead. To move both endpoints to a different port, set controller.debugPort: <port> (and update the forward accordingly).

Available endpoints:

• /debug/vars — internal state (config, credentials metadata, rendered output, resources, events, uptime)
• /debug/vars/<name> — a single variable; supports ?field={.jsonpath} for subselection
• /debug/pprof/ — Go profiling

See the Debugging Guide for the full endpoint catalogue.

See Also

• Getting Started
• CRD Reference
• Validation Tests
• Templating Guide