Monitoring¶
Monitor HAPTIC with Prometheus metrics: setup, the metrics reference, alerting rules, and dashboards.
Overview¶
The controller exposes Prometheus metrics via an HTTP endpoint, providing visibility into reconciliation performance, deployment status, resource counts, and leader election state.
Key monitoring areas:
- Reconciliation cycle performance and errors
- HAProxy deployment latency and success rates
- Configuration validation status
- Kubernetes resource counts
- Leader election for HA deployments
Two metric sources
Most of this guide is about the controller's metrics — the haptic_* family on port 9090, which describe reconciliation, deployment, and leader-election health. HAProxy itself exposes a separate Prometheus endpoint on port 8404 carrying live traffic, backend health, and response-code data — see HAProxy Data-Plane Metrics. The chart's bundled ServiceMonitor/PodMonitor scrape the controller only.
Enabling Metrics¶
Metrics are enabled by default. The controller serves Prometheus metrics at /metrics on the metrics port (default :9090), which is separate from the debug port. No additional configuration is needed beyond pointing Prometheus at this endpoint.
The controller reads its metrics port from the METRICS_PORT env var (default 9090). To disable the metrics server, set METRICS_PORT=0 on the controller container. Via Helm, use extraEnv:
The controller.config.controller.metricsPort Helm value is display-only: it isn't part of the CRD schema, and the chart strips it before serializing (the apiserver would reject it otherwise). Setting controller.config.controller.metricsPort: 0 in Helm values does not disable the metrics server.
Accessing Metrics¶
Prometheus Scrape Configuration¶
Add a scrape config for the controller:
scrape_configs:
- job_name: 'haptic'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
regex: haptic
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_number]
regex: "9090"
action: keep
ServiceMonitor (Prometheus Operator)¶
If using Prometheus Operator, enable the ServiceMonitor in Helm:
# values.yaml
monitoring:
serviceMonitor:
enabled: true
interval: 30s
labels:
release: prometheus # Match your Prometheus selector
The chart also ships a PodMonitor (monitoring.podMonitor.enabled) for setups that scrape pods directly instead of via the Service — enable whichever your Prometheus setup uses.
Add custom labels, a scrape timeout, relabelings, or metricRelabelings for larger setups:
# values.yaml
monitoring:
serviceMonitor:
enabled: true
interval: 15s
scrapeTimeout: 10s
labels:
release: prometheus
team: platform
# Stamp a cluster label onto every scraped series
relabelings:
- sourceLabels: [__address__]
targetLabel: cluster
replacement: production
# Drop a metric you don't want to store
metricRelabelings:
- sourceLabels: [__name__]
regex: 'haptic_event_subscribers'
action: drop
If a NetworkPolicy is in effect, also allow Prometheus to reach the metrics port — see Networking.
Manual Access¶
# Port-forward to metrics endpoint
kubectl port-forward -n haptic deployment/haptic-controller 9090:9090
# Fetch metrics
curl http://localhost:9090/metrics
Metrics Reference¶
Reconciliation Metrics¶
| Metric | Type | Description |
|---|---|---|
haptic_reconciliation_total |
Counter | Total reconciliation cycles triggered |
haptic_reconciliation_duration_seconds |
Histogram | Time spent in reconciliation cycles |
haptic_reconciliation_errors_total |
Counter | Failed reconciliation cycles |
Key queries:
# Reconciliation rate per second
rate(haptic_reconciliation_total[5m])
# Average reconciliation duration
rate(haptic_reconciliation_duration_seconds_sum[5m]) /
rate(haptic_reconciliation_duration_seconds_count[5m])
# Success rate percentage
100 * (1 - (
rate(haptic_reconciliation_errors_total[5m]) /
rate(haptic_reconciliation_total[5m])
))
Deployment Metrics¶
| Metric | Type | Description |
|---|---|---|
haptic_deployment_total |
Counter | Total deployment attempts |
haptic_deployment_duration_seconds |
Histogram | Time spent deploying to HAProxy |
haptic_deployment_errors_total |
Counter | Failed deployments |
haptic_haproxy_reloads_total |
Counter | HAProxy reloads triggered by deployments. A reload forks the HAProxy process; reload rate (vs runtime-API updates) is the canonical capacity/SLO signal |
haptic_dataplane_api_operations_total |
Counter | DataPlane API operations issued across deployments (structural changes applied to HAProxy) |
Key queries:
# Deployment rate
rate(haptic_deployment_total[5m])
# HAProxy reload rate — the capacity/SLO signal (a reload forks the process)
rate(haptic_haproxy_reloads_total[5m])
# Share of deployments that needed a reload vs runtime-only updates
rate(haptic_haproxy_reloads_total[5m]) / rate(haptic_deployment_total[5m])
# 95th percentile deployment latency
histogram_quantile(0.95, rate(haptic_deployment_duration_seconds_bucket[5m]))
# Deployment success rate
100 * (1 - (
rate(haptic_deployment_errors_total[5m]) /
rate(haptic_deployment_total[5m])
))
Fleet Convergence & Config Staleness¶
These gauges answer "did my change reach every HAProxy pod, and if not, for how long?" They are the noise-free replacement for alerting on rate(haptic_deployment_errors_total): the deploy scheduler self-heals transient failures, so a nonzero error rate no longer means the fleet is actually broken. These gauges report the converged state, not the attempt outcome.
They are populated leader-only — only the leader deploys. Followers reset them when they lose leadership, so haptic_haproxy_fleet_converged < haptic_haproxy_fleet_size is 0 < 0 (false) on followers and never false-alerts.
| Metric | Type | Description |
|---|---|---|
haptic_haproxy_fleet_size |
Gauge | HAProxy pods the last deployment targeted |
haptic_haproxy_fleet_converged |
Gauge | HAProxy pods now at the desired config. Alert on haptic_haproxy_fleet_converged < haptic_haproxy_fleet_size |
haptic_last_full_sync_timestamp_seconds |
Gauge | Unix timestamp (seconds) of the last time the whole fleet converged. Seeded to controller start time, so before the first full sync staleness reads as uptime rather than the whole epoch. In steady state (no config or pod changes) it advances with the periodic drift-prevention deploy, so any staleness threshold you alert on must exceed spec.dataplane.driftPreventionInterval (default 60s) |
haptic_deployment_consecutive_failures |
Gauge | Consecutive deployments that did not fully converge the fleet; resets to 0 on the first full sync |
Key queries:
# Pods not yet at the desired config right now (0 = fully converged)
haptic_haproxy_fleet_size - haptic_haproxy_fleet_converged
# How long since the whole fleet last converged (config staleness, seconds)
time() - haptic_last_full_sync_timestamp_seconds
# Deploys that failed to fully converge, back to back — alert on this instead of
# the error counter, now that transient deploy failures self-heal
haptic_deployment_consecutive_failures
The bundled HAProxyFleetDiverged alert (see Alerting Rules) fires when pods stay behind the desired config — the robust, cadence-independent signal. A staleness alert on time() - haptic_last_full_sync_timestamp_seconds is left to you: in steady state that value tracks the drift-prevention cadence, so a safe threshold depends on your configured driftPreventionInterval (a fixed default would false-fire for operators who raise it).
Runtime Fast-Path Metrics¶
The runtime fast path applies runtime-eligible server changes (weight, address, port, admin state) directly to the running HAProxy worker via the Dataplane API, bypassing a config reload. applies stuck at 0 while fires climbs means the fast path runs but the render diff never carries a runtime-eligible change.
| Metric | Type | Description |
|---|---|---|
haptic_runtime_fast_path_fires_total |
Counter | Runtime-eligible fast-path apply attempts (one per pod per reconcile) |
haptic_runtime_fast_path_applies_total |
Counter | Fast-path attempts that applied at least one runtime-eligible server update |
haptic_runtime_fast_path_failures_total |
Counter | Fast-path attempts that errored (best-effort; the scheduled deploy converges) |
haptic_runtime_fast_path_server_updates_total |
Counter | Total runtime-eligible server updates applied via the fast path |
Key queries:
# Fraction of fast-path attempts that carried a runtime-eligible change
rate(haptic_runtime_fast_path_applies_total[5m]) /
rate(haptic_runtime_fast_path_fires_total[5m])
# Runtime server updates applied without a reload
rate(haptic_runtime_fast_path_server_updates_total[5m])
Validation Metrics¶
| Metric | Type | Description |
|---|---|---|
haptic_validation_total |
Counter | Total validation attempts |
haptic_validation_errors_total |
Counter | Failed validations |
Key queries:
# Validation rate
rate(haptic_validation_total[5m])
# Validation success rate
100 * (1 - (
rate(haptic_validation_errors_total[5m]) /
rate(haptic_validation_total[5m])
))
Resource Metrics¶
| Metric | Type | Labels | Description |
|---|---|---|---|
haptic_resource_count |
Gauge | type |
Current count of watched resources |
haptic_haproxy_pods_rejected_total |
Counter | reason |
HAProxy pods refused admission by the discovery component. Persistent non-zero growth typically means the controller cannot talk to the deployed HAProxy pods (for example, the bundled HAProxy major.minor differs from the chart's haproxyVersion). |
haptic_config_rejected_total |
Counter | validator |
HAProxyTemplateConfig loads refused by the config-validation gate. The validator label names which check rejected it (basic, template, jsonpath, validationtests, or coordinator when a validator timed out). Non-zero growth means the leader is refusing new config and continuing on the last-good one — alert on it: the operator's latest change is not live. |
Key queries:
# All resource counts
haptic_resource_count
# Specific resource types
haptic_resource_count{type="ingresses"}
haptic_resource_count{type="services"}
haptic_resource_count{type="haproxy-pods"}
# Resource count changes
delta(haptic_resource_count[1h])
# Rejected HAProxy pods, broken down by reason
sum by (reason) (rate(haptic_haproxy_pods_rejected_total[5m]))
# Config rejected (leader refusing new config) — alert if > 0
sum by (validator) (rate(haptic_config_rejected_total[5m]))
Event Metrics¶
| Metric | Type | Description |
|---|---|---|
haptic_event_subscribers |
Gauge | Active event subscribers |
haptic_events_published_total |
Counter | Total events published |
Key queries:
# Event publishing rate
rate(haptic_events_published_total[5m])
# Subscriber count (should be constant)
haptic_event_subscribers
# Subscriber changes (indicates component restarts)
delta(haptic_event_subscribers[5m])
Leader Election Metrics¶
| Metric | Type | Description |
|---|---|---|
haptic_leader_election_is_leader |
Gauge | 1 if this replica is leader, 0 otherwise |
haptic_leader_election_transitions_total |
Counter | Leadership transitions (gain/loss) |
haptic_leader_election_time_as_leader_seconds_total |
Counter | Cumulative time as leader |
Key queries:
# Current leader count (should be exactly 1)
sum(haptic_leader_election_is_leader)
# Identify leader pod
haptic_leader_election_is_leader == 1
# Leadership transition rate
rate(haptic_leader_election_transitions_total[1h])
# Average time as leader per transition
haptic_leader_election_time_as_leader_seconds_total /
haptic_leader_election_transitions_total
Webhook Metrics¶
Exposed when the validating admission webhook is enabled (webhook.enabled=true).
| Metric | Type | Labels | Description |
|---|---|---|---|
haptic_webhook_requests_total |
Counter | gvk, result |
Total admission requests by GroupVersionKind and result |
haptic_webhook_request_duration_seconds |
Histogram | — | Time spent processing webhook requests |
haptic_webhook_validation_total |
Counter | gvk, result |
Validation outcomes (allowed / denied) per GVK |
Key queries:
# Denial rate per resource kind
sum by (gvk) (rate(haptic_webhook_validation_total{result="denied"}[5m]))
# 95th percentile webhook latency (must stay well under the 10s admission timeout)
histogram_quantile(0.95, rate(haptic_webhook_request_duration_seconds_bucket[5m]))
Reconciliation Queue & Parser Cache¶
| Metric | Type | Description |
|---|---|---|
haptic_reconciliation_queue_wait_seconds |
Histogram | Time a triggered reconciliation waits in the coordinator queue before processing starts; rising values indicate the controller can't keep up with change volume |
haptic_parser_cache_hits_total |
Counter | Parser cache hits — the controller caches parsed HAProxy configs so repeated reconciliations don't re-parse unchanged input |
haptic_parser_cache_misses_total |
Counter | Parser cache misses |
# Parser cache hit ratio
rate(haptic_parser_cache_hits_total[5m]) /
(rate(haptic_parser_cache_hits_total[5m]) + rate(haptic_parser_cache_misses_total[5m]))
Event Bus Backpressure¶
These complement haptic_events_published_total / haptic_event_subscribers from above.
| Metric | Type | Labels | Description |
|---|---|---|---|
haptic_events_dropped_total |
Counter | — | Total events dropped because a subscriber's buffer was full |
haptic_events_dropped_critical_total |
Counter | — | Drops from critical subscribers (alert if > 0 — indicates lost reconciliation work) |
haptic_events_dropped_observability_total |
Gauge | — | Drops from observability-only subscribers (expected under load, non-alerting) |
haptic_events_dropped_by_subscriber_total |
Counter | subscriber, event_type |
Per-subscriber drop counts for diagnosing which component is falling behind |
Build Info¶
| Metric | Type | Labels | Description |
|---|---|---|---|
haptic_build_info |
Gauge | version, haproxy_version, go_version |
Always 1; useful for joining build metadata into other queries |
# Pin a query to controller version 0.1.0
haptic_reconciliation_total * on() group_left(version) haptic_build_info{version="0.1.0"}
HAProxy Data-Plane Metrics¶
Every metric above comes from the controller (haptic_*, port 9090) — they describe reconciliation, deployment, and leader-election health, not live traffic. HAProxy itself exposes a separate Prometheus endpoint carrying the data-plane signals operators usually watch most closely: per-frontend request rates, per-backend response-code breakdowns, and session counts.
The bundled config enables HAProxy's built-in Prometheus exporter on the status frontend (port 8404, path /metrics) by default — it is served from the always-on status-extra-100-prometheus-exporter snippet, so no extra flag is required. The HAProxy Service exposes that port under the name stats.
The chart's ServiceMonitor / PodMonitor do not scrape HAProxy
Both bundled monitors collect the controller's haptic_* metrics only: the PodMonitor selects app.kubernetes.io/component: controller, and the ServiceMonitor scrapes the metrics port (9090) that only the controller Service exposes. Neither targets HAProxy's 8404. To collect HAProxy's haproxy_* metrics, add a separate scrape (or ServiceMonitor) pointed at the HAProxy pods.
Scrape Configuration¶
Plain Prometheus — keep HAProxy pods (component: loadbalancer) and scrape their 8404 container port:
scrape_configs:
- job_name: 'haproxy'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
regex: loadbalancer
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_number]
regex: "8404"
action: keep
Prometheus Operator — a ServiceMonitor against the HAProxy Service's stats port:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: haproxy
spec:
selector:
matchLabels:
app.kubernetes.io/name: haptic
app.kubernetes.io/component: loadbalancer
endpoints:
- port: stats # 8404
path: /metrics
Key Queries¶
HAProxy labels frontend/backend metrics with proxy (the section name) and server metrics with server:
# Request rate per frontend
sum by (proxy) (rate(haproxy_frontend_http_requests_total[5m]))
# 5xx response rate per backend
sum by (proxy) (rate(haproxy_backend_http_responses_total{code="5xx"}[5m]))
# Active sessions per backend
sum by (proxy) (haproxy_backend_current_sessions)
The full metric set is HAProxy's own, not HAPTIC's — see the HAProxy Prometheus exporter reference for every exposed series and its labels.
Alerting Rules¶
If you deploy via the Helm chart, it ships a built-in PrometheusRule (enable with monitoring.prometheusRule.enabled) covering a core set of nine controller alerts, each individually togglable via monitoring.prometheusRule.defaultRules.*. The rules below are a broader recommended set you can copy and adapt for any Prometheus setup.
Recommended Alerts¶
groups:
- name: haptic
rules:
# Reconciliation failures
- alert: HAProxyICHighReconciliationErrorRate
expr: |
rate(haptic_reconciliation_errors_total[5m]) /
rate(haptic_reconciliation_total[5m]) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "High reconciliation error rate (>10%)"
description: "Controller is failing to reconcile configurations"
# Deployment latency
- alert: HAProxyICHighDeploymentLatency
expr: |
histogram_quantile(0.95,
rate(haptic_deployment_duration_seconds_bucket[5m])
) > 5
for: 5m
labels:
severity: warning
annotations:
summary: "95th percentile deployment latency >5s"
description: "Deploying configs to HAProxy is taking too long"
# Fleet diverged — some HAProxy pods are not at the desired config.
# Prefer this over the deploy error counter: transient failures self-heal.
- alert: HAProxyFleetDiverged
expr: haptic_haproxy_fleet_converged < haptic_haproxy_fleet_size
for: 5m
labels:
severity: warning
annotations:
summary: "HAProxy fleet is diverged"
description: "Some HAProxy pods have not converged on the desired config for 5m"
# Validation failures
- alert: HAProxyICValidationFailures
expr: |
rate(haptic_validation_errors_total[5m]) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Configuration validation failing"
description: "HAProxy configuration has syntax or validation errors"
# Component crash
- alert: HAProxyICComponentStopped
expr: |
delta(haptic_event_subscribers[5m]) < 0
labels:
severity: critical
annotations:
summary: "Event subscriber count decreased"
description: "A controller component may have crashed"
# No leader elected (HA)
- alert: HAProxyICNoLeader
expr: sum(haptic_leader_election_is_leader) < 1
for: 1m
labels:
severity: critical
annotations:
summary: "No HAProxy controller leader elected"
description: "No controller replica is elected as leader"
# Multiple leaders (split-brain)
- alert: HAProxyICMultipleLeaders
expr: sum(haptic_leader_election_is_leader) > 1
labels:
severity: critical
annotations:
summary: "Multiple HAProxy controller leaders detected"
description: "Split-brain condition - multiple replicas think they are leader"
# Frequent leadership changes
- alert: HAProxyICFrequentLeadershipChanges
expr: rate(haptic_leader_election_transitions_total[1h]) > 5
for: 15m
labels:
severity: warning
annotations:
summary: "Frequent leadership transitions"
description: "Controller leadership changing too often, may indicate cluster instability"
# No HAProxy pods discovered
- alert: HAProxyICNoHAProxyPods
expr: haptic_resource_count{type="haproxy-pods"} < 1
for: 5m
labels:
severity: critical
annotations:
summary: "No HAProxy pods discovered"
description: "Controller cannot find any HAProxy pods to manage"
# Critical events dropped (lost reconciliation work)
- alert: HAProxyICCriticalEventsDropped
expr: increase(haptic_events_dropped_critical_total[5m]) > 0
labels:
severity: critical
annotations:
summary: "Critical events dropped from event bus"
description: "A critical subscriber's buffer overflowed; reconciliation work has been lost"
Tuning alert thresholds
The thresholds above suit typical production environments. For high-churn environments (frequent deployments, many short-lived resources), increase the for duration on reconciliation and deployment alerts to avoid noise. For development clusters, consider relaxing error rate thresholds or disabling non-critical alerts entirely.
Dashboard Examples¶
The chart ships a complete built-in Grafana dashboard (29 panels) — enable it with monitoring.grafanaDashboard.enabled: true (the default useBuiltIn: true renders dashboards/haptic.json into a <release>-grafana-dashboard ConfigMap that the Grafana sidecar auto-discovers; set a custom one via grafanaDashboard.customDashboard). The queries and JSON template below are for building your own dashboard or extending the bundled one.
Grafana Dashboard Queries¶
Reconciliation Overview Panel:
# Success rate (stat panel)
100 * (1 - (
rate(haptic_reconciliation_errors_total[5m]) /
rate(haptic_reconciliation_total[5m])
))
# Rate over time (graph)
rate(haptic_reconciliation_total[5m])
rate(haptic_reconciliation_errors_total[5m])
Deployment Latency Panel:
# P50, P95, P99 latencies
histogram_quantile(0.50, rate(haptic_deployment_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(haptic_deployment_duration_seconds_bucket[5m]))
histogram_quantile(0.99, rate(haptic_deployment_duration_seconds_bucket[5m]))
Resource Count Panel:
# All resource types
haptic_resource_count
# Stacked area chart by type
haptic_resource_count{type=~"ingresses|services|endpoints"}
Leader Election Panel:
# Current leader indicator
haptic_leader_election_is_leader == 1
# Transition count over time
increase(haptic_leader_election_transitions_total[1h])
Dashboard JSON Template¶
Example Grafana dashboard structure (use as a starting point):
{
"title": "HAPTIC",
"panels": [
{
"title": "Reconciliation Rate",
"targets": [
{"expr": "rate(haptic_reconciliation_total[5m])"}
]
},
{
"title": "Reconciliation Success Rate",
"targets": [
{"expr": "100 * (1 - rate(haptic_reconciliation_errors_total[5m]) / rate(haptic_reconciliation_total[5m]))"}
]
},
{
"title": "Deployment Latency",
"targets": [
{"expr": "histogram_quantile(0.95, rate(haptic_deployment_duration_seconds_bucket[5m]))"}
]
},
{
"title": "Resource Counts",
"targets": [
{"expr": "haptic_resource_count"}
]
},
{
"title": "Leader Status",
"targets": [
{"expr": "haptic_leader_election_is_leader"}
]
}
]
}
This is a starting point — add panels using the PromQL queries above for more detailed views of deployment latency distribution, resource counts over time, or per-pod leader status.
Operational Insights¶
Key Health Indicators¶
| Indicator | Healthy Range | Action if Unhealthy |
|---|---|---|
| Reconciliation success rate | >99% | Check logs for template/validation errors |
| Deployment success rate | >99% | Check HAProxy pod connectivity |
| P95 deployment latency | <2s | Check HAProxy DataPlane API performance |
| Leader count | Exactly 1 | Check HA configuration and network |
| Event subscribers | Should not decrease during normal operation | Restart controller if dropping |
Capacity Planning¶
Monitor these metrics for capacity planning:
# Reconciliation frequency (how often config changes)
rate(haptic_reconciliation_total[1h]) * 3600
# Ingress growth rate
deriv(haptic_resource_count{type="ingresses"}[1d])
# Average reconciliation overhead
avg_over_time(haptic_reconciliation_duration_seconds_sum[1d]) /
avg_over_time(haptic_reconciliation_duration_seconds_count[1d])
Troubleshooting with Metrics¶
High reconciliation error rate:
- Check
haptic_validation_errors_total- template/config issues - Check
haptic_deployment_errors_total- HAProxy connectivity issues - Review controller logs for specific error messages
Missing metrics:
- Verify the metrics server is enabled —
METRICS_PORTenv var on the controller container is non-zero (default9090; thecontroller.config.controller.metricsPortHelm value is display-only and never reaches the controller) - Check ServiceMonitor selector matches Prometheus configuration
- Verify network policies allow scraping
Leader election issues:
- Check if
sum(haptic_leader_election_is_leader) != 1 - Review
rate(haptic_leader_election_transitions_total[1h])for instability - See High Availability Guide for troubleshooting
Integration with Existing Monitoring¶
Prometheus Operator¶
Enable the bundled ServiceMonitor as shown in ServiceMonitor (Prometheus Operator) above. The chart renders it to target the controller Service in the release namespace — no explicit namespace selector is needed.
Victoria Metrics¶
Use the same Prometheus scrape configuration — Victoria Metrics is compatible.
Datadog¶
Configure Datadog Agent to scrape Prometheus metrics:
See Also¶
- Debugging Guide - Runtime introspection and troubleshooting
- High Availability - Leader election configuration
- Troubleshooting Guide - General troubleshooting