Unreleased documentation. Choose your installed release in the version menu. Features described here may be absent from that release.
Access logging¶
Read request logs from the vector container in a default installation. If you
disable Vector, read them from the haproxy container instead. These commands
use the haptic namespace:
# Default install (vector.enabled=true)
kubectl logs -n haptic -l app.kubernetes.io/instance=haptic,app.kubernetes.io/component=loadbalancer -c vector
# With vector.enabled=false
kubectl logs -n haptic -l app.kubernetes.io/instance=haptic,app.kubernetes.io/component=loadbalancer -c haproxy
HTTP frontends emit JSON records like this one. TCP frontends log connections and use a smaller set of fields:
{"ts":"2026-07-25T19:05:19.615000Z","req_id":"019f9ae9-3a61-7814-8601-774735249ecd","trace_id":"","client_ip":"10.244.0.1","frontend":"https","backend":"default_echo_echo_80","server":"echo-7c9d8b6f5-2xk9p","method":"GET","host":"echo.example.com","listener_port":"443","path":"/api/v1","http_version":"HTTP/1.1","status":200,"bytes":73,"request_time_ms":0,"queue_time_ms":0,"connect_time_ms":1,"response_time_ms":3,"total_time_ms":4,"retries":0,"term":"----","resource":"default/echo","denied_by":"","tls_version":"TLSv1.3","tls_sni":"echo.example.com"}
Raw access records have no syslog prefix; ts carries the timestamp. If you
collect HAProxy's stdout directly, allow for non-JSON startup, health-check, and
process messages alongside the access records.
Add the settings below to your Helm values file
and apply them with helm upgrade.
Core fields¶
| Field | Meaning |
|---|---|
ts |
Request accept time in Coordinated Universal Time (UTC), with microsecond precision |
req_id |
HAProxy-generated request ID. Forwarding it to your application is opt-in; see Request IDs |
server_pod |
Backend pod name, also recorded as server. No backend pod is identified when HAProxy answers the request itself |
namespace, service |
Namespace and name of the Kubernetes Service for the selected backend. The Service can be in a different namespace from the routing resource |
destination_ip |
Destination address as HAProxy sees it, after any address translation by your load balancer or cluster network |
instance_pod, instance_node |
HAProxy pod and node that handled the request. Empty when the pod identity environment variables aren't supplied |
trace_id |
Distributed trace ID from a valid inbound traceparent. With tracing enabled, HAPTIC creates one if the client supplies none; otherwise the field is empty |
client_ip |
Client address, after any src-ip-header rewrite |
frontend, backend, server |
HAProxy frontend, backend, and server names |
method, host, path, http_version |
Request identity. path excludes the query string |
listener_port |
Port used in the routing lookup, as a string. For a Gateway, this is the allocated pod port, which can differ from the public listener port. Empty on frontends without routing logic |
status, bytes |
Response status and bytes sent to the client (JSON numbers) |
request_time_ms, queue_time_ms, connect_time_ms, response_time_ms, total_time_ms |
Milliseconds spent receiving the request, waiting for a connection slot, connecting to the backend, waiting for response headers, and completing the request. -1 means that phase didn't complete. The HTTP total excludes idle keep-alive time; the TCP total covers the whole session |
retries |
Backend connection retries |
term |
HAProxy's 4-character termination state — separates a client abort from a server abort, a timeout, and a response HAProxy generated itself |
resource |
<namespace>/<name> of the Ingress, HTTPRoute or custom resource that owns the matched route — the join key back to Kubernetes |
denied_by |
Which gate blocked the request; empty when the backend answered |
cache_degraded, rate_limit_degraded, waf_degraded, schema_degraded |
These mark a dependency-degraded cache, limiter, WAF, or schema-validation path. A strict policy can set both its degraded field and denied_by. Empty when the corresponding feature didn't encounter a degraded dependency |
route |
Matched host and path rule, with prefix matches marked *, such as echo.example.com/api/*. Present when tracing or the request metrics path label needs it |
bytes_in |
Request body bytes from the client (%U) — no request line or headers, which HAProxy doesn't count. Present only when the request_size request metric is enabled, since nothing else reads it |
Template libraries add fields for the features you configure, each only when
that feature is in use: waf_action, waf_rule_id and waf_score;
rate_limit_allowed and rate_limit_remaining; cache (HIT/MISS/STALE) and
app_backend; auth_status and consumer; schema_outcome; tls_version,
tls_sni and tls_resumed; mtls_verify and mtls_cn; gw_route;
captured_headers; client_ip_peer.
Use denied_by to identify why HAPTIC rejected a request. Values include
rate_limit_local, rate_limit_shared, rate_limit_shared_unavailable, waf,
jwt_signature, jwt_expired, api_key, hmac, basic_auth,
consumer_groups, body_too_large, schema_invalid and the *_unavailable
fail-closed variants.
Add your own fields¶
controller:
config:
templatingSettings:
extraContext:
accessLog:
fields:
tenant: req.hdr(X-Tenant)
region: str(prod-eu)
Each value is one HAProxy sample expression, captured into a transaction variable
at request time and emitted as a JSON string. Use str(<value>) for a constant
label. Field names must match ^[A-Za-z_][A-Za-z0-9_]{0,39}$ and must not
collide with a built-in field; expressions must not contain whitespace, #, "
or a backslash. A violation fails the render with a message naming the field.
Because the capture happens at request time, a value that doesn't exist yet reads
empty — a WAF verdict, a cache status, an auth outcome, or anything else a
SPOA hub message produces later in the transaction. For
those, contribute a log-fields-* snippet
instead: its items are evaluated when the line is written, after every filter has
run.
To log the query string, opt in with query: query — it's excluded by default
because query strings are a common accidental carrier of tokens and session ids.
Raise accessLog.maxLineBytes (default 16384, accepted range 1024–65535) if
custom fields or captured request headers push records past it: HAProxy truncates
a longer line mid-byte, which makes the record unparseable. A value outside the
range fails the render rather than silently truncating every record.
Where the logs go¶
Set accessLog.targets to send records to your own collector. The following
example replaces the default destination with a TCP collector running in the
same pod:
controller:
config:
templatingSettings:
extraContext:
accessLog:
targets:
shipper: # a name you choose; it keys the target
ring:
name: accesslog
address: 127.0.0.1:6514 # a log-shipper sidecar on loopback
HAProxy's process and alert messages keep going to stdout. If you replace the Vector destination, its log-derived request metrics and traces stop receiving records too. Choose access controls and retention for the destination: access records include client addresses and request paths.
Each target gets a copy of every eligible access record. Configure multiple targets to send records to more than one collector:
| Field | Meaning |
|---|---|
address |
stdout, stderr, fd@<n>, <host>:<port> (UDP), [<ipv6>]:<port>, an absolute socket path, or ring@<name> |
format |
raw, rfc3164, rfc5424, local, priority, short, timed, iso. Defaults to raw for stdout/stderr and rfc5424 otherwise |
facility, level |
Syslog facility (default local0) and level, either info (default) or debug |
ring |
Send through a buffered TCP ring instead of a bare address |
level is a maximum severity filter, and HAProxy emits access records at
info. Anything stricter — notice, warning, err — therefore drops every
record while haproxy -c still reports the config as valid, so the chart accepts
only the two levels that deliver.
An address of ring@<name> must name a ring some target in this list declares.
HAProxy accepts a dangling reference at config check and then refuses to start
with unknown ring named, so the render rejects it instead.
Why a ring for a sidecar¶
A ring is a buffered TCP client: records queue in memory when the collector is
unavailable and flush when it reconnects, up to the configured buffer capacity.
A plain <host>:<port> target uses UDP and provides no replay buffer.
Configure each ring with these fields:
nameandaddress: a host and port, such ascollector:514or[::1]:514. HAProxy 3.4 doesn't accept a Unix socket as a ring server; use a plain-path logging target for a Unix-socket collector.size: buffer bytes, default65536. Keep it at least 256 bytes larger thanmaxLineBytesto avoid truncating records into invalid JSON.logProto:legacyfor newline-delimited RFC 6587, oroctet-count.connectTimeoutandserverTimeout: connection and server timeouts.serverOptions: additional HAProxy server keywords, inserted verbatim, such as TLS settings.
A collector reads this as ordinary syslog carrying a JSON payload. In Vector, a
syslog source parses the envelope and one remap recovers the record:
sources:
haproxy_access:
type: syslog
mode: tcp
address: 0.0.0.0:6514
transforms:
parsed:
type: remap
inputs: [haproxy_access]
source: |
. = parse_json!(string!(.message))
Keep these constraints in mind:
- A ring server's address is resolved when the config is parsed. A Service DNS
name that doesn't resolve at that moment fails the render. Use a loopback
sidecar address or a literal IP, or pass
resolvers/init-addrthroughserverOptions. - Any file referenced from
serverOptions(aca-file, a clientcrt) must exist wherever the config is validated — the controller pod — not only in the HAProxy pod. Deliver such material through the chart's file mechanism so both see it. - A plain-path (Unix socket) target does no buffering. It's the way to reach a collector on a socket, since HAProxy 3.4 rejects a Unix socket as a ring server. If the socket is absent, HAProxy continues serving and reports log-delivery errors. Those records are lost; use a ring when you need bounded buffering.
Dropping records you don't need¶
To reduce log volume, suppress successful HTTP requests:
Suppression drops 2xx/3xx records only when no gate denied the request. Denials, 4xx, and 5xx remain eligible for logging, subject to the transport's loss behavior.
The default retains successful requests because they help diagnose retries and intermittent failures. Suppression also removes those requests from log-derived metrics and traces. Choose retention and access controls for the data you log; this setting doesn't remove sensitive fields from the records you keep.
The rule runs after HTTP responses, including responses HAProxy generates
itself. TCP-mode frontends are unaffected. The internal TCP frontend already
uses option dontlog-normal; TLS passthrough logs each connection.
The access log is lossy under back-pressure¶
The default connection to Vector uses a Unix datagram socket. If its receive queue fills, HAProxy discards records and continues serving requests. Access logs, and the metrics and traces derived from them, can therefore miss traffic.
Monitor haproxy_process_dropped_logs_total on HAProxy's Prometheus endpoint,
or DroppedLogs in the stats socket's show info output. These counters track
records HAProxy discards; they don't detect records lost or rejected later in
your collection pipeline.
The chart's HAProxyAccessLogRecordsDropped alert is available through
controller.monitoring.prometheusRule. If it fires, check Vector's health and
CPU budget. You can also reduce log volume,
which reduces the requests available for log-derived metrics and traces.
Vector sidecar¶
The default Vector sidecar receives access logs, writes them to stdout, and builds request metrics and optional traces from them. It also exports the SPOA hub's metrics. See monitoring setup for scrape endpoints and settings.
To remove the sidecar:
HAProxy then logs to its own stdout. Log-derived metrics and traces are no longer available; Prometheus can still scrape HAProxy and the SPOA hub directly.
HAPTIC updates Vector's configuration as your settings change, and Vector reloads the file automatically. If Vector exits or repeatedly fails its health check, its supervisor restarts the process while HAProxy keeps serving. Records can be lost during startup or recovery. A container-level failure, such as an out-of-memory termination, can still affect pod readiness.
Request IDs¶
HAPTIC generates a UUIDv7 for each request and records it in req_id. The ID
contains a timestamp and random bits, with no client or load-balancer address.
To send it to your application as X-Request-ID, add this Ingress annotation:
The request ID annotations also let you
choose a header name or accept an inbound ID. If you accept a client-supplied
ID, the header can differ from the HAProxy-generated req_id in the access log.
For UUIDv4 instead, override the directive through a defaults-settings-*
snippet with a band above 150:
controller:
config:
templateSnippets:
defaults-settings-160-request-id:
template: |
unique-id-format %[uuid()]
With tracing disabled, trace_id records a valid inbound traceparent for log
correlation; HAPTIC creates no trace context. With
extraContext.tracing.enabled, HAPTIC
adopts valid inbound context or creates a new trace, then propagates it to the
backend. Trace-context propagation is separate from the
haproxy-haptic.org/request-id annotation,
which forwards the request ID in a header.
Contribute a field from your own library¶
log-fields-* is the extension point. Use it instead of
accessLog.fields when the value only exists at log time,
or when a template library should contribute the field for every install that
enables the feature. A snippet emits named log-format items and nothing else:
controller:
config:
templateSnippets:
log-fields-900-my-feature:
template: |
%(my_field)[var(txn.my_var)]
Only items available at log time are legal. HAProxy rejects path, pathq,
req.hdr(), res.hdr() and req.ssl_sni inside a log-format, so materialise
anything request- or response-scoped into a transaction variable first
(http-request set-var(txn.my_var) req.hdr(X-Thing)). Type an item (:sint,
:bool) only when its fetch always resolves — an unresolved typed item renders
"" into a numeric slot.
Change the destination or the whole format¶
Use accessLog.targets to change access log destinations
or facilities. global-settings-100-logging controls HAProxy's process logs.
To replace the access record format, override util-log-format-http (HTTP-mode
frontends) or util-log-format-tcp (TCP-mode frontends). Note that a
defaults-section log-format can't reference HTTP-scoped fetches at all,
which is why the format is emitted per frontend.