Debugging Guide¶

This guide explains how to debug and troubleshoot the HAProxy Template Ingress Controller using built-in introspection and debugging tools.

Overview¶

The controller provides a debug HTTP server that exposes internal state, event history, and Go profiling endpoints. These tools help you understand controller behavior without requiring log analysis.

Key features:

Real-time controller state inspection
Event history for tracking controller activity
JSONPath field selection for targeted queries
Go profiling for performance analysis

Enabling Debug Endpoints¶

The debug HTTP server is configured via Helm values:

# values.yaml
controller:
  debugPort: 6060  # Set to 0 to disable

By default, the debug port is 6060. The server binds to all interfaces (0.0.0.0) for kubectl port-forward compatibility.

Accessing Debug Endpoints¶

From Outside the Cluster¶

Use kubectl port-forward to access the debug server:

# Forward debug port from controller pod
kubectl port-forward -n <namespace> deployment/haptic-controller 6060:6060

# Access endpoints
curl http://localhost:6060/debug/vars

From Inside the Cluster¶

Access directly via pod IP or service:

# Get pod IP
POD_IP=$(kubectl get pod -n <namespace> <pod-name> -o jsonpath='{.status.podIP}')
curl http://${POD_IP}:6060/debug/vars

Debug Variables¶

List All Variables¶

curl http://localhost:6060/debug/vars

Returns a list of all available debug variable paths:

{
  "vars": [
    "config",
    "credentials",
    "rendered",
    "auxfiles",
    "resources",
    "events",
    "state",
    "uptime"
  ]
}

Configuration State¶

Get the current controller configuration:

# Full configuration
curl http://localhost:6060/debug/vars/config

# Just the version
curl 'http://localhost:6060/debug/vars/config?field={.version}'

# Template names
curl 'http://localhost:6060/debug/vars/config?field={.config.templates}'

Response:

{
  "config": {
    "templates": {
      "main": "global\n  maxconn {{ maxconn }}\n..."
    },
    "watched_resources": [...]
  },
  "version": "12345",
  "updated": "2025-01-15T10:30:45Z"
}

Rendered HAProxy Config¶

Get the most recently rendered HAProxy configuration:

# Full rendered config
curl http://localhost:6060/debug/vars/rendered

# Just the config text (useful for saving to file)
curl 'http://localhost:6060/debug/vars/rendered?field={.config}' | jq -r '.'

# Config size and timestamp
curl 'http://localhost:6060/debug/vars/rendered?field={.size}'
curl 'http://localhost:6060/debug/vars/rendered?field={.timestamp}'

Response:

{
  "config": "global\n  maxconn 2000\n  log stdout local0\n\ndefaults\n...",
  "timestamp": "2025-01-15T10:30:45Z",
  "size": 4567
}

Resource Counts¶

Get counts of watched Kubernetes resources:

# All resource counts
curl http://localhost:6060/debug/vars/resources

# Specific resource type
curl 'http://localhost:6060/debug/vars/resources?field={.ingresses}'

Response:

{
  "ingresses": 5,
  "services": 12,
  "haproxy-pods": 2
}

Auxiliary Files¶

Get SSL certificates, map files, and general files used in the last deployment:

curl http://localhost:6060/debug/vars/auxfiles

Response:

{
  "files": {
    "ssl_certificates": [
      {
        "name": "tls-cert",
        "path": "/etc/haproxy/ssl/tls-cert.pem"
      }
    ],
    "map_files": [...],
    "general_files": [...]
  },
  "timestamp": "2025-01-15T10:30:45Z",
  "summary": {
    "ssl_count": 2,
    "map_count": 1,
    "general_count": 3
  }
}

Event History¶

Get recent controller events:

# Last 100 events (default)
curl http://localhost:6060/debug/vars/events

Response:

[
  {
    "timestamp": "2025-01-15T10:30:45Z",
    "type": "config.validated",
    "summary": "config.validated"
  },
  {
    "timestamp": "2025-01-15T10:30:46Z",
    "type": "reconciliation.triggered",
    "summary": "reconciliation.triggered"
  }
]

Full State Dump¶

Get all controller state in a single response:

curl http://localhost:6060/debug/vars/state

Warning

The full state dump can return very large responses. Prefer specific variables or JSONPath field selection for production debugging.

Controller Uptime¶

curl http://localhost:6060/debug/vars/uptime

JSONPath Field Selection¶

All debug endpoints support JSONPath field selection using kubectl-style syntax:

# Basic field access
curl 'http://localhost:6060/debug/vars/config?field={.version}'

# Nested field access
curl 'http://localhost:6060/debug/vars/config?field={.config.templates.main}'

# Array indexing
curl 'http://localhost:6060/debug/vars/events?field={[0]}'

See Kubernetes JSONPath documentation for full syntax.

Go Profiling¶

The debug server includes Go pprof endpoints for performance analysis:

# CPU profile (30 second sample)
curl http://localhost:6060/debug/pprof/profile?seconds=30 > cpu.pprof

# Heap profile
curl http://localhost:6060/debug/pprof/heap > heap.pprof

# Goroutine dump
curl http://localhost:6060/debug/pprof/goroutine?debug=1

# Block profile
curl http://localhost:6060/debug/pprof/block > block.pprof

# Mutex profile
curl http://localhost:6060/debug/pprof/mutex > mutex.pprof

# All profiles summary
curl http://localhost:6060/debug/pprof/

Analyze profiles with go tool pprof:

go tool pprof -http=:8080 cpu.pprof

Common Debugging Workflows¶

Configuration Not Loading¶

Check if config is loaded:

curl http://localhost:6060/debug/vars/config

If error "config not loaded yet", check controller logs for parsing errors:

kubectl logs -n <namespace> deployment/haptic-controller | grep -i error

Verify HAProxyTemplateConfig exists:

kubectl get haproxytemplateconfig -n <namespace>

HAProxy Config Not Updating¶

Check rendered config timestamp:

curl 'http://localhost:6060/debug/vars/rendered?field={.timestamp}'

Check recent events for reconciliation activity:

curl http://localhost:6060/debug/vars/events | jq '.[] | select(.type | contains("reconciliation"))'

Verify resource counts match expected:

curl http://localhost:6060/debug/vars/resources

Template Rendering Errors¶

Check if config is valid:

curl 'http://localhost:6060/debug/vars/config?field={.version}'

Look for template-related events:

curl http://localhost:6060/debug/vars/events | jq '.[] | select(.type | contains("template"))'

Try rendering with debug output (see Templating Guide)

Memory Issues¶

Get current heap usage:

curl http://localhost:6060/debug/pprof/heap?debug=1 | head -20

Generate heap profile for analysis:

curl http://localhost:6060/debug/pprof/heap > heap.pprof
go tool pprof -top heap.pprof

Check resource counts (large counts may indicate memory pressure):

curl http://localhost:6060/debug/vars/resources

High CPU Usage¶

Generate CPU profile:

curl http://localhost:6060/debug/pprof/profile?seconds=30 > cpu.pprof

Analyze hot spots:

go tool pprof -top cpu.pprof

Check reconciliation frequency in events:

curl http://localhost:6060/debug/vars/events | jq '[.[] | select(.type == "reconciliation.triggered")] | length'

Deployment Failures¶

Check recent events for deployment status:

curl http://localhost:6060/debug/vars/events | jq '.[] | select(.type | contains("deployment"))'

Verify HAProxy pods are discovered:

curl 'http://localhost:6060/debug/vars/resources?field={."haproxy-pods"}'

Check rendered config for syntax errors:

curl 'http://localhost:6060/debug/vars/rendered?field={.config}' > haproxy.cfg
haproxy -c -f haproxy.cfg

Security Considerations¶

Credentials: The /debug/vars/credentials endpoint exposes only metadata, NOT actual passwords
Access Control: Debug endpoints have no built-in authentication - use kubectl port-forward or network policies to restrict access
Large Responses: Full state dump can expose sensitive configuration - use specific variables instead
Production Usage: Consider disabling debug port in high-security environments:

controller:
  debugPort: 0  # Disabled

Scripting Examples¶

Monitor Reconciliation Activity¶

#!/bin/bash
# Watch reconciliation events
while true; do
    count=$(curl -s http://localhost:6060/debug/vars/events | \
            jq '[.[] | select(.type == "reconciliation.completed")] | length')
    echo "$(date): $count reconciliations completed"
    sleep 10
done

Export Rendered Config¶

#!/bin/bash
# Export current HAProxy config with timestamp
TIMESTAMP=$(curl -s 'http://localhost:6060/debug/vars/rendered?field={.timestamp}' | jq -r '.')
curl -s 'http://localhost:6060/debug/vars/rendered?field={.config}' | jq -r '.' > "haproxy-${TIMESTAMP}.cfg"
echo "Exported to haproxy-${TIMESTAMP}.cfg"

Health Check Script¶

#!/bin/bash
# Check controller health via debug endpoints
set -e

CONFIG_VERSION=$(curl -s 'http://localhost:6060/debug/vars/config?field={.version}' 2>/dev/null || echo "error")
UPTIME=$(curl -s http://localhost:6060/debug/vars/uptime 2>/dev/null || echo "error")
RESOURCES=$(curl -s http://localhost:6060/debug/vars/resources 2>/dev/null || echo "{}")

echo "Config Version: $CONFIG_VERSION"
echo "Uptime: $UPTIME"
echo "Resources: $RESOURCES"

if [ "$CONFIG_VERSION" = "error" ]; then
    echo "ERROR: Cannot access debug endpoints"
    exit 1
fi

Debugging Guide¶

Overview¶

Enabling Debug Endpoints¶

Accessing Debug Endpoints¶

From Outside the Cluster¶

From Inside the Cluster¶

Debug Variables¶

List All Variables¶

Configuration State¶

Rendered HAProxy Config¶

Resource Counts¶

Auxiliary Files¶

Event History¶

Full State Dump¶

Controller Uptime¶

JSONPath Field Selection¶

Go Profiling¶

Common Debugging Workflows¶

Configuration Not Loading¶

HAProxy Config Not Updating¶

Template Rendering Errors¶

Memory Issues¶

High CPU Usage¶

Deployment Failures¶

Security Considerations¶

Scripting Examples¶

Monitor Reconciliation Activity¶

Export Rendered Config¶

Health Check Script¶

See Also¶