Changelog¶
All notable changes to HAPTIC — the controller and its Helm chart — are documented in this file. Controller changes are listed first; chart changes (values, templates, chart defaults) follow under each release's "Helm chart" subsection.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[Unreleased]¶
Added¶
-
On HAProxy 3.4, adding or removing a route — an Ingress, an HTTPRoute, or a custom-CRD route — whose backend is dynamic-shaped (an empty body, a dynamic-capable balance) and whose per-route logic is map-driven now propagates with no HAProxy reload: the backend is created, populated and published, or drained and deleted, over the runtime API. On 3.0–3.3 the route still reloads (no runtime
add backend), but its servers are always added and removed at runtime. A route whose backend carries afilter(for example response compression, on by default for Ingress viahaproxy-haptic.org/compress-enable) stays structural and reloads on add or remove. -
New
spec.maps.<name>.ordered(defaulttrue) declares whether the position of an entry inside a map file changes what HAProxy does with it. Set it tofalsefor a map read withmap_str,map_beg,map_ipormap_str_int— those find a key by its own value, so a new entry can be appended over the runtime API instead of rewriting the file and reloading. Keep the default formap_reg,map_sub,map_dom,map_dirandmap_end, which HAProxy evaluates as a first-match-wins list. HAProxyCfgpod status reports the applied and running render plan id, the apply mode and its reasons instatus.deployedToPods[].- The
HAProxyCfgcarries the render gate's verdict asConfigValidated(HAProxy's own message on a refusal) andConfigPinned(renders are held because it refused two in a row), plus a newhaptic_config_pinnedgauge. - New
haptic agentsubcommand: the HAPTIC agent, which the chart now deploys as theagentcontainer of every HAProxy pod. It owns that pod's file tree and runtime sockets, applies the desired file set in one transaction, runs the runtime commands the controller composed or reloads (paced by--reload-interval-min), restores the last known good set when HAProxy rejects the result, and exportshaptic_agent_*metrics on its own port (agent page). - New
haptic diffanswers "will this change reload?" before the change is applied: it renders a candidate configuration, or reads a running pod's applied plan with--from pod://<namespace>/<pod>, and compares the two with the decision the controller makes per pod. The first line is the verdict —runtime,file_onlyorreload— followed by a reason for every change that cannot run at runtime and the runtime commands it composed.--testrenders both sides with avalidationTest's fixtures,--output jsongates a pipeline on the result. - New
haptic agent stateprints one HAProxy pod's agent state: the applied, running, worker-ops and last-known-good plan ids, the runtime inventory, the deletes still outstanding, and the last apply's mode, error stage and reload outcome. Run it withkubectl exec … -c agent -- haptic agent state;--verifyre-hashes the tree first so the digests are observations,--fileslists every file with its digest and size, and--output jsonprints the raw/v1/state. - New
/debug/heapdumpendpoint on the debug port writes aruntime/debugheap dump: every heap object, the pointer edges between them, and the roots.pprofreports where memory was allocated but not what still holds it, so this is what identifies a retainer. The heap is collected before the dump is written, since an uncollected dump is dominated by unreachable objects that by definition have no retainer. Read it with a heap-dump reader such as heapspurs. Writing the dump stops the world for its duration — health checks and admission both stall — so a concurrent second request is refused with409, and the endpoint answers on loopback only, like/debug/pprof. The dump is roughly heap-sized and is staged in$TMPDIR, which counts against the pod'sephemeral-storage; it refuses with507rather than filling the filesystem, andHAPTIC_HEAPDUMP_DIRredirects it to a mounted volume. The written dump is checked for completeness, since the Go runtime ignores write errors while dumping and would otherwise return a truncated object graph with a200. - Type-preserving collection pipeline for templates:
map,filter,reject,flat_map,unique,unique_byandgroup_by, chained with|. Each stage keeps its input's element type, so typed field access still works at the end of a chain and a renamed field fails the config load instead of silently emptying a map file. See ADR-0018. - Arrow closures
x => exprwith parameter and result types inferred, accepted anywhere a function is expected. - Collection pipelines with inline stage functions compile to a single fused loop, so a chain costs what the equivalent hand-written loop costs.
sort_byaccepts afunc(a, b T) intcomparator alongside its JSONPath criteria, and accepts typed slices.- Nested shapes inside a watched resource are nameable as type expressions (
resources.endpoints.Endpoints,resources.gateways.SpecListeners), so a helper or closure can declare types below the resource root. - A struct whose every field is its zero value is falsy, so an optional object is tested with
if ingress.Spec.DefaultBackend.Serviceinstead of adig()probe. Usenot/and/orrather than!/&&/||on struct operands. appendis Go's own builtin, soappend(dst, src...)works and a widening spread is a compile error instead of a render-time panic. The engine's replacement native is gone rather than renamed: a slice reached throughanyis asserted at the boundary,append(gf["hosts"].([]any), h). Templates appending to anany-typed value must add that assertion, andappend(nil, x)must start from a typed slice — the engine names each such call site at config load.- New
untar_gz(archive)expands a gzip-compressed tar into a map of entry path to content, for templates that build configuration out of a fetched archive. Expansion is all-or-nothing and bounded by entry count and size, path traversal is rejected, and a corrupt archive returns an error instead of failing the render. - New
recordEvent(resource, reason, message)records a KubernetesWarningEvent against any watched resource; the leader forwards them to the API server, so they surface underkubectl describe. - New
currentFilesglobal exposes the accepted map files, general files and crt-lists to reconciliation renders and the published output snapshot to admission renders. Paired with the newrandBytes(n), a template can self-rotate on-disk state with no external component. SSL certificates and CA files are excluded. - New
renderModeglobal ("admission"or"reconcile") lets a check fail loud under the admission webhook while only warning during a live reconcile. - New
admissionSubjectglobal names the watched object and everywatchedResourcesalias affected by its admission request, so a route-scoped check denies only the violating resource. - Admission responses surface template-recorded Warning events as AdmissionReview warnings (capped at 10 plus a suppressed count), so
kubectl applyprints the consequences of a change. - A configuration is now one
HAProxyTemplateConfigplus any number ofHAProxyTemplateLibraryobjects, referenced in order throughspec.libraryRefs(ADR-0017). The config carries what an operator writes —podSelector,watchedResources,templatingSettings,validators— and libraries carry the bulk (templateSnippets,validationTests,maps,files,sslCertificates,k8sResources,haproxyConfig,templatingSettings), leaving the config at 1.3% of etcd's per-object limit. A library carries nopodSelector,watchedResourcesordataplane, so it cannot redefine the controller's operational identity. The config merges last, so its inline content is always the override point. - Each
libraryRefsentry names arevisionthe referenced object must report; the controller compares the two strings and never recomputes either, so a half-applied set is visible as a mismatch and holds the last-good configuration. The config'sownerReferenceis stamped onto every library it references. --crd-name/CRD_NAMEname a singleHAProxyTemplateConfig; which libraries it pulls in, and in what order, comes from itsspec.libraryRefs. Startup waits for the config and every reference to resolve.- New
haptic config view --inputprints the merged input configuration — the config plus every library it references — as opposed to the rendered HAProxy outputconfig viewshows. haptic validate -fis repeatable and accepts multi-document files, assembling every config and library it finds byspec.libraryRefs, sohelm template … > all.yaml && haptic validate -f all.yamlvalidates exactly what the controller assembles. New--dump-mergedprints the merged spec and exits.- New
haptic preflight -f values.yamlruns a configuration through the controller's own checks before it is deployed: it renders the image-embedded chart with your values and runs the load gate (structural validation plus everyvalidationTest, includinghaproxy -c), thenvector validateon the sidecar config andvarnishd -Con the Varnish configuration. Schemas come from the cluster you deploy to;--schema-dirruns it fully offline.--expect-chart-version/HAPTIC_EXPECT_CHART_VERSIONhard-fails when the embedded chart is not the chart being installed. - New
haptic apply-crdsserver-side applies the CRDs bundled in the image-embedded chart (or--chart/$HAPTIC_CHART_DIR), closing the gap Helm leaves for CRDs installed from a chart'scrds/directory. It never deletes and never touches CRD.status. HAProxyTemplateConfig.statusreportsobservedGenerationand a standardValidatedcondition, stamped on every object of the merged set with that object's own generation, plus anObservedprinter column — so GitOps health checks work on whichever object was applied. A startup load-gate rejection is recorded asValidated=Falsewith reasonLoadGateFailedand the failing tests before the controller crash-loops.- Every validation test is checked for render determinism: the runner renders twice and compares the config and every auxiliary file, so a template whose output depends on map-iteration order fails its own suite. Six such sites in the bundled chart were fixed.
- New
validationTests[*].currentServersdeclares a previous deployment's servers as data (backend → server →{address, port}), which templates read ascurrentConfig.ServerIndex. It replacescurrentConfig, whose raw HAProxy text needed a config parser to read the same server index out of. - New
haptic validate --snapshot-dir <dir>writes every validation test's renderedhaproxy.cfg, map files, general files and certificates to<dir>/<test>/, so the output of two checkouts can be compared withdiff -r. - New
spec.validators[i].dataFilessends a validator the files a validated file references, alongside it in the same request and markedkind: "data". The result cache keys on their content too. - Every changed render is judged by
haproxy -cand by the configured pluggable validators. Leader reconciliation, watched-resource admission, and HTTP-store promotion share this gate; identical output returns from content caches. See ADR-0020. --dump-renderedincludesk8sResourcesand status patches, not justhaproxy.cfg, maps, files and certificates.- New
haptic_runtime_map_divergence_total{map}counts runtime maps whose post-apply read-back disagreed with the desired content and forced a reload fallback. Steady growth means endpoint churn is quietly reloading HAProxy. - New
haptic_runtime_backend_fallback_total{reason}counts runtime backend batches a pod reloaded instead of running.reason="name_collision"is a fresh backend whose name a not-yet-deleted one still holds; steady growth means deferred backend deletes are lagging behind routes reusing their names. - New fleet-convergence and staleness metrics:
haptic_haproxy_fleet_size,haptic_haproxy_fleet_converged,haptic_last_full_sync_timestamp_secondsandhaptic_deployment_consecutive_failures. - A browser playground for
HAProxyTemplateConfigruns the controller's production render path client-side in WebAssembly — nothing is uploaded. It shows thehaproxy.cfg, map files, decoded certificates, status patches, recorded events and rendered Kubernetes resources a config produces, with per-line provenance, a reload-vs-runtime impact verdict, and an annotation-migration report. Published per version at/playground/<version>/. - The playground runs a config's
spec.validationTestsin a tests tab. A browser has nohaproxybinary, sohaproxy_validassertions fall back to the pure-Go syntax and schema check, labelledsyntax + schema. - The user docs and landing page embed the playground inline: runnable, editable examples — including challenges with revealable solutions — each with an "open in the full playground" link. Deep-link an example with
/playground/?preset=<id>. - Measured render cost against object count is documented in Performance: roughly 0.11–0.15 ms per Ingress, with the path maps growing faster than the object count and reaching about 40% of the render at 5,000 Ingresses. The same figure is the per-admission cost, since the webhook renders the whole configuration once per admitted object.
- The docs site serves every page as raw Markdown at the page URL plus
index.md, withllms.txtandllms-full.txtindexes. Every page carries an edit link and a report-a-problem link that prefills a GitLab issue.
Changed¶
- The config-load validation gate runs its bundled
validationTests'haproxy -cchecks concurrently (across the worker pool, bounded by the pod's CPU allocation) instead of serializing them behind a single slot. On a multi-core pod this cuts controller startup and config-change reinitialization time noticeably when the merged configuration carries manyhaproxy_validtests — measured ~5.6× faster (12.8s → 2.3s) on a 411-check suite on 16 cores. - Losing the leader Lease no longer reinitializes the controller replica. Election is re-entered in place: the leader-only components stop and the same instances restart on re-acquisition, while the replica's admission webhook and resource stores keep serving throughout. Previously a leadership transition tore the whole iteration down, which retired the webhook's validators for the length of a full resync and denied admission requests routed to that replica with "retry after controller initialization".
- HAProxy's
haproxy -cno longer runs on the reconcile wall clock. A render is dispatched to the fleet immediately and the same check runs concurrently, on a semaphore slot of its own so the admission webhook never queues behind it (controller.config.controller.renderGateIntervalcaps its duty cycle). Coverage is unchanged — the webhook and the configuration-load gate still check synchronously, and every reconcile render is still checked. A refusal is closed on both ends: every pod that took the plan without its own HAProxy loading it is asked to restore its last known good file set, and every later render is held until one passes. A pod whose own binary reloaded the plan is left alone. The verdict surfaces as theHAProxyCfg'sConfigValidated/ConfigPinnedconditions and as aRenderRefusedByHAProxyKubernetes Event on the config. A render that accepts HTTP-store content for the first time still takes the check synchronously, before that content becomes the accepted version. See ADR-0022. - BREAKING: the HAPTIC agent replaces the HAProxy Data Plane API. The controller compares the render with what each pod reports and sends that pod either runtime commands or a file set plus a reload; map entries, certificate and CA content, crt-list entries and server address, weight or state now reach the running worker without a reload, and on HAProxy 3.4 so does adding or removing a route. A pod's HAProxy binary is what judges a configuration, so any directive it accepts can be deployed. See ADR-0022; the chart migration is below.
- BREAKING: readiness semantics on the HAProxy pod. HAProxy's
/readyon the stats port stays the pod's readiness probe, and it answers 200 only once a rendered configuration is running. The agent adds astartupProbeon/readyzand alivenessProbeon/healthz;/readyzreports that the agent can accept applies and stays true after one it rejected, so a fleet-wide rejection can never drain the Service or fence off the repair. A pod whose first apply fails keeps answering 503 and never joins the Service. - BREAKING:
currentConfigexposes onlycurrentConfig.ServerIndex[<backend>][<server>].Address/.Port— the parsed sections (Backends,Frontends,Global, …) are gone. It is no longer a parse of the deployed configuration but the servers the render itself declared, so a template that read another field fails at the configuration-load gate; runhaptic preflightbefore upgrading to see it beforehand. - BREAKING: the controller binary is now
haptic(washaptic-controller):haptic run|validate|preflight|apply-crds|config|benchmark|version, installed at/usr/local/bin/hapticin the image and published ashaptic-<version>-<os>-<arch>on the releases page. Kubernetes object names (haptic-controllerDeployment, ServiceAccount, labels) are unchanged. CI jobs and scripts that invoke the binary directly must use the new name. - The controller-wide watcher debounce default is
100ms(was2s), for every watched kind including an operator's own resources: bursts are already coalesced while a render is in flight and reload pacing lives in the agent, so the longer refractory window only added propagation latency. The config-reinitialisation debounce stays2s; per-resourcedebounceIntervaloverrides are unchanged. - The controller no longer throttles its own apiserver requests by default: client-side rate limiting is disabled (
--kube-client-qps/KUBE_CLIENT_QPSdefault-1), so it relies on the apiserver's API Priority & Fairness (APF) — the posture controller-runtime and kgateway ship. Previously each of the controller's clients carried a private 50-QPS bucket, so under a large rolling fleet or bulk apply the dynamic client's status writes queued behind client-side throttling into multi-second latency while its other buckets sat idle; and because a client-side-throttled request blocks rather than returning429, client-go's automatic retry never engaged. APF returns429 + Retry-After, which client-go retries. Set--kube-client-qpsabove0to reinstate a client-side cap, applied as one budget shared across all the controller's clients. - The bundled SPOA hub is v0.13.0. A hub config change now reloads only the plugins whose library or params changed and carries the rest over untouched, so a WAF- or schema-annotated Ingress edit no longer resets the rate-limit plugin's store connection and circuit breaker; and plugin log records reach the hub log (
"target":"plugin",plugin=<name>,span.req_id) for plugins built against plugin-api 0.8 — the bundled rate-limit plugin (v0.4.1) is, and it now logs one warning per lost store connection. v0.4.1 also fixes the plugin staying in the local fallback for good after a shared-store outage: the periodic lease refresh could take the breaker's single half-open probe without reporting it, so no exact check or borrow ever reached the recovered store until a reload. - A watch event that only echoes the controller's own status write no longer triggers a reconciliation: the watcher refreshes its store from the event but recognises the write by the resourceVersion the API server returned for it. Each render's status patches came back as three to four watch events, and every one of them re-rendered the whole configuration and queued behind the pipeline; under sequential Gateway route creation that was ~200 ms of the per-route propagation time and most of the controller's render CPU. External status writes still trigger as before.
regex_searchreuses a bounded set of compiled patterns per engine instead of recompiling repeated patterns on every call.- The template engine releases what a pooled VM held from its previous render instead of truncating around it, so a render's object graph is no longer pinned for the process lifetime. On a 1000-route churn this cuts peak RSS from 2603-3527 MiB to 1476 MiB and retained heap from 600-1096 MiB to 78 MiB, while rendering 7.5% faster — fewer live references is less for the collector to mark.
- Watched-resource admission reuses the reconciliation pipeline's compiled templates and caches only its validation verdict instead of retaining a second parsed HAProxy configuration.
- Endpoint caches share one parsed desired HAProxy configuration after post-reload comparison proves equivalence, instead of retaining an equivalent parsed read-back per pod.
- The event commentator retains scalar correlation metadata instead of complete event payloads, so its 500-entry history no longer pins configuration objects, rendered resources, or status patches.
- An unparseable or set-based
watchedResources[].labelSelectornow fails the configuration load, naming the resource and the offending selector, instead of being silently discarded — which widened the watch to every object of that kind cluster-wide with no diagnostic. Set-based syntax (in,notin,!,!=) is rejected rather than supported;==is now accepted as equality instead of being read as the value=nginx. This hard-fails at startup and inhaptic validate; live reload keeps its existing fail-open behaviour and logs. http.Fetch's refresh cadence option is nowinterval;delaykeeps working as a deprecated alias, and setting both is an error. The option governs how often content is re-checked, never a wait before the first fetch.- A validation test's per-test
extraContextdeep-merges into the globaltemplatingSettings.extraContextinstead of replacing whole top-level subtrees. A map carrying__replace__: trueopts back into wholesale replacement. - A validation test's shared
_globalblock contributes an isolatedextraContextbaseline, folded under each test below any per-test override. - Changing a general file no longer reloads HAProxy when it carries
spec.files.<name>.reloadOnPush: false; templates registering a file at render time pass the same flag as a fourth argument tofileRegistry.Register. The default staystrue. - Removing a general file reloads HAProxy only when the rendered configuration or a crt-list still names it.
Removed¶
- The Data Plane API client, its seven generated per-version clients, the configuration comparator and the sync orchestrator. Nothing has called them since the agent cutover; the controller image is ~86,000 lines smaller and no longer ships an OpenAPI client generator (
oapi-codegen,oapi-codegen/runtimeandkin-openapileavego.mod). - Syntax and schema validation of the rendered configuration.
haproxy -cis a strict superset of both for loadability, and the schema half only checked conformance to the Data Plane API's model, which nothing speaks any more. What is lost is that model conformance and nothing else: the webhook, the configuration-load gate andrendergateall still runhaproxy -c, the pluggable output validators still run synchronously on the reconcile path, and the browser playground keeps the syntax + schema check as itshaproxy_validanswer because a browser has no HAProxy binary. No production binary parses HAProxy configuration any more, andmake lintfails if one starts. - The deprecated
validationTests[*].currentConfigfield. Declare a previous deployment withcurrentServersinstead — it reaches templates as the samecurrentConfig.ServerIndex. - Data Plane API-era metrics that the agent cutover left unwired:
haptic_dataplane_api_operations_total,haptic_runtime_fast_path_fires_total,haptic_runtime_fast_path_applies_total,haptic_runtime_fast_path_failures_total,haptic_runtime_fast_path_server_updates_totalandhaptic_deploy_runtime_divergence_total. The "Where the old metrics went" table on the Monitoring page names each replacement. - Data Plane API-endpoint capability keys from the template
capabilitiesmap:supports_waf,supports_waf_global,supports_waf_profiles,supports_udp_lb_acls,supports_udp_lb_server_switching,supports_keepalived,supports_udp_load_balancing,supports_bot_management,supports_git_integration,supports_dynamic_update,supports_aloha,supports_advanced_logging,supports_pingandis_enterprise. They were always false; a template that reads one is unchanged, since a missing key is falsy. The map gainssupports_ssl_ca_files,supports_ssl_crl_files,supports_quic_initial_rules,supports_log_profiles,supports_tracesandsupports_acme_providers, which the fleet's HAProxy version does decide. - Per-pod server-side configuration validation. The Data Plane API's
validate_cmdranhaproxy -con each pod before activating a push; the agent has no equivalent, because the pod's own binary rejects a configuration it can't parse at reload and the agent then restores the last known good file set. The webhook and the configuration-load gate keep the fullhaproxy -c, andrendergateruns it on every render. - The standalone
deployer.NewDeploymentSchedulerconstructor. UseNewDeployStackso structural and runtime writes share endpoint state. - The four-slot parsed-config cache in the dataplane parser, and the
haptic_parser_cache_hits_total,haptic_parser_cache_misses_total,haptic_parser_cache_hits_by_source_totalandhaptic_parser_cache_misses_by_source_totalmetrics that reported it. The cache existed to let the desired config be parsed once and reused across HAProxy replicas, but the deploy path already carries that parse by reference (ValidationCompletedEvent.ParsedConfig), so its one beneficiary never consulted it — measured at 122 hits against 1509 misses on a live cluster, while pinning four fully-parsed configs (~900 MB of live heap at 1000 HTTPRoutes, since a parsed config is 35-90x its text size). Parsing is now always direct.
Fixed¶
- The controller no longer rebuilds the watched-resource reflect types on every render.
buildResourcesValueandBuildPerResourceStoreTypereconstructed theresourcesstruct type and each per-resource store type viareflect.StructOfper render, though those types are fixed after startup — the dominant allocation under render churn (measured 69% of allocation on a live controller; total render allocation dropped 62% once cached). This transient churn was the memory pressure behind controller OOMKills under a high-concurrency admission burst; caching the types restores headroom without changing rendered output. - On a reload-gated fleet (HAProxy < 3.4) under rapid add/remove churn, a config change could silently stop propagating: a route deletion was skipped as a no-op and the fleet stayed on the previous config. Content-addressed renders recur — an add and its delete hash to the same plan every cycle — and the skip-unchanged gate compared each render to the last fully deployed config, so a recurring render matching that value was dismissed even after a paced deploy had moved the fleet past it. The gate now skips only when the fleet is actually settled on that config, never while a later render is still converging.
- A backend's per-server weights are now applied on the runtime add-server path. A server added over the runtime API carried no weight, so any backend that distributes traffic by per-server weight — weighted TCPRoute backendRefs, which become one
balance roundrobinbackend — collapsed to an equal split, and a weight-0 server, which must receive no traffic, was added with HAProxy's default weight 1 and took traffic. Theadd servercommand now carries the server's weight, includingweight 0. (HTTPRoute and GRPCRoute weighting is unaffected: it distributes across separate backends through a weighted map, not per-server weights.) - The leader no longer crashes (nil dereference in the config publisher) when leadership is lost while a publish is pending:
retry.OnErrorreports an interrupting context cancellation as the last retriable error, which is nil when none preceded it, so an interrupted HAProxyCfg or auxiliary-file write came back as a success without a result. It's now an error naming the cancellation. - Status writes to auxiliary-file CRs (
HAProxyMapFile,HAProxyGeneralFile,HAProxyCRTListFile) are no longer re-stamped when unchanged, eliminating a client rate-limiter PATCH storm under endpoint churn that could starve ingress-status writes. Their content-hashed names make every re-stamp byte-identical;status.deployedToPodsis unaffected — every value change is still written, the periodic drift-prevention re-sync still re-stamps each live pod authoritatively, and pod departure, an aux-file CR deletion, and a leadership change each invalidate the cache. - Controller memory could grow without bound under sustained route churn. The status applier's mailbox collapses only uninterrupted runs of the same event type, and under churn its three types arrive strictly alternating (render →
resources.applied, deploy →deployment.completed), so nothing ever collapsed while the applier was slower than the arrival rate: a 2,048-deep backlog of full status-patch sets held 6 GB of live heap (12 GB RSS) on a 1,900-route churn. The applier now coalesces each declared type across the whole queue, so the backlog is bounded by the number of event types. - Status stamping at iteration start no longer targets the referenced
HAProxyTemplateLibraryobjects as if they wereHAProxyTemplateConfigs, which logged a NotFound warning per library on every leader start. - Version-cache hits no longer force an unnecessary HAProxy reload, and concurrent runtime updates can no longer leave a reusable stale endpoint observation.
- A template function that renders more than 256 distinct parallel
go rendertargets now fails to compile instead of silently rendering the wrong target's body. The compiled render operand is a single byte, so target 256 onward wrapped modulo 256 and produced full-length output for the wrong macro with no error. (The bundled chart reuses one macro per runtime loop, so it never approached the limit.) - Local Docker builds and tagged-release controller binaries now report their source-input hash instead of
unknown. - The current-config store no longer re-parses an unchanged HAProxy configuration when only auxiliary files changed. It keyed the decision on
spec.checksum, which covers the config and every map file, certificate and CRT-list — so ordinary endpoint churn rewriting a map file re-parsed byte-identical config text. A parsed configuration costs 35–90× its text, which made this one of the larger allocations on the churn path that OOMKilled the controller at its default memory limit. The parse decision now uses a hash of the config text; the checksum remains a fast path for the case where nothing changed at all. - Losing leadership now disables the drift-prevention timer without firing and re-arming it on the former leader.
- Writing a watched resource in any served apiVersion other than the one the controller resolved is no longer denied. The admission webhook's rules are rendered from a watched resource's full
apiVersionscandidate list, but the controller registered a validator — and keyed its resource lookup — only for the single version it resolved to, so withfailurePolicy: Failevery other served version was rejected permanently, advising the operator to "retry after controller initialization" for an initialization that had already finished. On a default install this denied everygateway.networking.k8s.io/v1beta1HTTPRoute write cluster-wide while the identicalv1object was admitted. Both sides now cover every version the cluster serves; a resource nobody watches is still refused, and objects in the added versions get the same full render +haproxy -cvalidation. - The controller no longer accumulates deployed renders without bound under endpoint churn. Every applied config was queued for publication holding its full rendered bytes, but the queue drained one entry per
configPublishInterval(10s) while the runtime-raw lane skipsminDeploymentIntervalby design — so pod churn filled it far faster than it emptied, at ~687 KB per entry. Queued entries are now dropped once every pod has demonstrably moved past their checksum, which bounds the queue by the number of checksums live across the fleet instead of by the deploy rate. A checksum any pod still reports is never dropped, sostatus.deployedToPods[].checksumstays resolvable againstspec. - On-demand exact lookups now return every resource sharing an index key even when only part of that bucket is warm in the cache.
- Template resource lookups with no keys or more keys than
indexBynow fail the render instead of silently omitting resources. - Watched-resource index components now preserve their boundaries, so values containing
/no longer collide with another key or leak into partialFetchresults; empty and Unicode values use the same semantics in both store modes. - Back-to-back renders now read
currentFilesfrom the exact last accepted output instead of an asynchronously updated debug cache, so self-referential files don't rematerialize from stale input. - Reconciliation, admission, and all-replica proposal validation now read
currentFilesfrom one completely committedHAProxyCfgauxiliary reference set, including certificate Secret metadata. Partial publications, legacy in-place Secret updates, and a missing set ID after modern publication fail closed instead of freezing or reusing ambiguous input. A replica that saw a pre-set-ID publication change while it was a standby accepts the visible snapshot when it becomes leader instead of failing every render until a restart; during a rolling upgrade from a pre-set-ID controller every new leader used to go503and be liveness-killed once. http.Fetchno longer serves a response fetched with old credentials after its authentication changes. Each render must declare one authentication and option set per URL; source changes fence in-flight refreshes and replace, re-arm, or stop that URL's timer. Rejected admission candidates no longer replace the live source or timer.- Cold and source-replacement HTTP responses remain render-local until the exact complete output passes every validator, then the whole candidate set is accepted atomically. Failed or canceled renders refetch instead of caching unvalidated bytes, and source replacement immediately revalidates pending content that survives a retired refresh-validation batch.
- Pluggable-validator responses with a missing, unknown, or contradictory aggregate
resultnow fail the render as protocol errors instead of potentially admitting invalid output; malformed responses are never cached. - Cancelling validation now terminates queued or running
haproxy -cchecks and prevents their result from entering the success cache. - Kubernetes read errors, typed watched-resource conversion failures, and ambiguous
GetSinglelookups now fail rendering instead of producing a partial configuration from an empty result. - Interrupted validation can no longer produce a valid verdict or crash the controller during template cancellation, and a retired leader term no longer publishes reconciliation results after cancellation.
- Watched-resource admission now overlays configured
watchedResourcesaliases instead of the Kubernetes plural. Multiple aliases for one group, version, and resource tuple and label or field selector transitions render the same proposed state their watchers will store. - Deployment timeouts now cancel blocked Dataplane calls out of band and retain scheduler ownership until the exact deployment attempt terminates, so stale completions cannot release or overwrite a newer deployment.
- Watched resources no longer remain accessible through an old
indexBykey after the indexed value changes. Bothstore: fullandstore: on-demandmove the resource atomically, and remove its old entry if the updated object no longer yields a complete key. - On-demand watched-resource reads no longer restore stale cache entries when an informer update or delete races a cache-hit renewal or live API fetch.
- A superseded configuration validation can no longer restart the controller with an older candidate, discard credential or schema changes, or make the next iteration refetch an unvalidated newer CR. Reloads carry the exact accepted snapshot, failed replacements retry live state, and schema re-resolution passes the complete config gate before activation.
- Failed controller-iteration retries no longer renew the 90-second reinitialization grace window indefinitely; unresolved failures now make
/healthzreturn 503 after one fixed grace episode. - Config publication now retries incomplete output from immutable render snapshots and commits success only while its generation and leader term remain authoritative; superseded retries stop promptly, terminal failures release the queue, and leadership reacquisition starts clean workers.
- Config publication now heals same-checksum drift, prunes obsolete auxiliary resources, gives long, colliding, or invalid artifacts stable Kubernetes-safe names, and rejects ambiguous file identities before deployment.
- HAProxy pod discovery now proves both the Dataplane API version and HAProxy binary series, publishes rotated credentials in order, re-probes replacements, container restarts, image changes, or URL changes, preserves Enterprise edition detection, honors per-endpoint retry backoff, formats IPv6 endpoints correctly, retires stale deployments, clients, and caches, and binds deployment status to the pod runtime epoch.
- An incomplete runtime-only deployment now invalidates its shared baseline and retries structurally, preventing a failed server update from being omitted by the next runtime diff.
- Rendered-resource apply and orphan-prune failures no longer publish successful infrastructure status. An incomplete desired set also no longer drives pruning, so a transient API or discovery failure cannot delete a resource that is still rendered.
- A full critical EventBus subscriber buffer now fails and rebuilds the controller iteration instead of logging the lost coordination event and continuing with partial state. Drop counters survive the rebuild; explicitly lossy observability subscribers remain non-fatal.
- Controller reinitialization and leadership loss now join every owned worker, informer, timer, and admission call before closing dependencies or starting replacement work. Obsolete leader terms can no longer publish configuration after losing authority, and invalid leader-election timings fail the configuration gate before startup.
- HTTP refresh validation now binds verdicts and timer callbacks to exact content revisions. Late results and retired refreshers can no longer promote or strand newer pending content.
- Validation-suite deadlines now cancel the active main, auxiliary-file, Kubernetes-resource, and determinism renders instead of taking effect only between tests.
- Admission no longer accepts changed invalid output merely because the live baseline is also invalid. The recovery exception now requires both renders to complete and produce identical content checksums.
- Webhook validator registration is atomic and fails controller initialization when any configured kind can't be mapped or the dry-run validator is absent. A request routed to an unregistered kind is denied with HTTP 503 instead of admitted.
- Pluggable-validator errors now block every render path before publication or deployment, including deletes, HTTP refreshes, startup, and ordinary reconciliation. Malformed file globs reject the merged config, and manager construction failures abort the controller iteration instead of silently disabling the feature.
- Config and credential updates observed while an iteration starts are replayed after initialization. Exact bootstrap versions remain suppressed, while newer concurrent updates can no longer be lost.
/debug/vars/auxfiles,/debug/vars/stateand/debug/vars/allno longer serve auxiliary-file contents. They serialized every rendered TLS private key and thetls-ticket-keyssession-ticket file to anything that reached the controller's loopback interface — which any sidecar in the controller Pod, or anyone withpods/portforward, can do. Paths, filenames and counts are unchanged, so the endpoints keep their debugging value.pkg/httpstorerefuses a redirect that downgrades anhttps://source to plaintext, and drops every credential header when a redirect changes host. Fetched content becomes HAProxy configuration and Coraza rules, so a plaintext hop was a config-injection path; and becausenet/httpkeys itsAuthorizationstrip on the host alone, a same-host downgrade also re-sent the credential in cleartext while API keys set throughAuthTypeHeadersurvived any cross-origin redirect. The store's trust model — TLS posture, size bounds, and the absence of checksum pinning — is now written down inpkg/httpstore/README.md.- A configuration change mixing a runtime-eligible server field with a reload-only one is no longer misclassified as runtime-only. The lane classifier counted every modified server as runtime-eligible, so such a diff was written to disk with
skip_reload, only partly applied to the live worker, and then recorded as activated — leaving the on-disk and in-memory configurations divergent until an unrelated reload.haptic-controller's reload-impact prediction had the same error. - A backend whose only change is an
http-errorrule is deployed instead of silently dropped. The update was gated on a field list that cannot name fields living outsideBackendBase, so the change produced zero operations. - The
gostatement is disabled in templates. It was enabled with a comment claiming the chart's parallel rendering needed it; parallel rendering uses a different construct, and no template in the repo used the statement form — which was the only way to create a goroutine outside the render's control, whose panic took the process down.pkg/templating/README.mdnow records the sandbox posture. - The controller no longer leaks four informers per iteration. Watches on HAPTIC's own output CRDs were started on a channel that was closed only when cache sync failed, so every configuration change left another set running for the process's lifetime.
- The admission webhook no longer closes idle API-server connections after 10s.
net/httpfalls back toReadTimeoutwhen noIdleTimeoutis set, so the webhook dropped pooled connections well before the API server's 90s client-go default did, and the API server's next request raced the close into anEOF— surfacing as intermittentfailed calling webhookerrors onkubectl apply. The server now sets a 120s idle timeout, above the client's, so the API server is always the side that closes. - Documentation described capabilities HAPTIC doesn't have. The most consequential: an admission webhook for
HAProxyTemplateConfig(removed by ADR-0016 — a bad config is caught by the leader's first render, the startup load gate, andpreflight, not at apply time); anHAProxyTemplateConfigper template library (they areHAProxyTemplateLibraryobjects referenced from one config'sspec.libraryRefs); the access log reaching thehaproxycontainer's stdout (it goes to the Vector sidecar by default); and the HTTPS listener inheriting HAProxy's default ciphers (the SSL library ships a TLS policy with a TLSv1.2 floor). Also corrected non-existent chart values (controller.networkPolicy.ingress.dataplaneApi.*,rbac.create), wrong defaults (governance.enabled, leader-election timings, alert and exclusion counts), and template examples that could not compile. - The template reference documents all registered helper functions — 17 appeared nowhere in the docs, including
dig_string,selectattr,make_guid,sort_ints, and theresource/jsonpathGet/jsonpathSetgovernance helpers — plus thecurrentFilesglobal,sort_by's comparator form, and the collection-pipeline stages. Thesort_bypipe rule was documented backwards: a direct call needs two variables, while a pipe stage keeps only the first result. spec.libraryRefs, theHAProxyTemplateLibrarykind,spec.validators[i].dataFiles,validationTests.<name>.currentFiles, theeventsassertion target, the reservedhaproxy-podsand_globaltest entries, thehaptic benchmarksubcommand, and four registered metrics (haptic_runtime_map_divergence_total,haptic_deploy_runtime_divergence_total, and the two by-source parser-cache counters) are now documented.- An HTTP-sourced map or blocklist no longer freezes at its last accepted content when a validation verdict is lost. A pending validation that produces no verdict within five minutes is abandoned and the URL refreshes again; a panic in the proposal validator now publishes a failure instead of nothing.
- A new HAProxy pod no longer waits for an unrelated change to receive its configuration when a pod-index update is dropped under load. Pod discovery now re-reads the pod store on every drift-prevention tick.
- A deployment can no longer push one render's configuration alongside a different render's parsed form, which computed the runtime-server diff against a configuration that was not the one being sent. The scheduler now checks that a validation verdict describes the render it holds.
- Events discarded because the startup buffer was full are now counted in
haptic_events_dropped_critical_totaland reach the critical-drops alert. They were previously logged only, so the one drop path that can lose the controller's bootstrap events reported zero drops. - Fixed three data races in the event bus, all unsynchronised reads of the subscriber list against leader-election and scatter-gather subscription churn: the subscriber-count gauge, the startup/leadership event replay, and the drop callback. The drop callback also now runs with no bus lock held, so a callback that publishes cannot deadlock its own publisher.
- Deleting one EndpointSlice no longer evicts every EndpointSlice of the same Service. Resources sharing a non-unique index key — the chart indexes EndpointSlices by namespace plus
kubernetes.io/service-name— shared one store bucket, and a delete dropped the whole bucket, emptying the backend until an unrelated event happened to repopulate it. Deletes are now scoped to the deleted resource's namespace and name, for bothstore: fullandstore: on-demand. - Watched resources are no longer modified while the informer still owns them. Field filtering and the float-to-int conversion moved into the informer's transform hook, which runs before the object is cached and before any handler sees it. On-demand kinds keep their body-stripping projection; full kinds get a normalise-only transform, because for them the stored body is what templates read.
- The HAProxy config parser cache no longer thrashes under churn. Post-reload read-backs and current-config reads carry a
_versionheader that changes on every push, so they could never hit the cache but still evicted the desired config — a live cluster measured a 2.4% hit rate. Single-use parses now bypass the cache. - A runtime-map
addthat finds its key already present converges it withset mapinstead of failing. - Fixed a controller panic when a watched config or credentials object was absent as the watcher finished its initial sync: the callback received a typed nil pointer in a non-nil interface, so its
obj == nilguard did not fire. A component panic now also logs a stack trace. Programmed(and the IngressloadBalancer.ingressaddress) is set only when every HAProxy replica has taken the configuration, not when at least one has. A partial deploy surfaces thedeployFailedvariant instead of advertising an address the fleet does not uniformly serve.- Fixed
HAProxyCfg.status.deployedToPods[].checksumadvertising contentspecnever carried. Deployed checksums were dropped by a one-slot latest-wins queue (one in 31 on a real run) and could be overtaken by validation publishes (measured 5.2s). Distinct deployed checksums are now queued in arrival order and drained first. - Fixed the mid-flight runtime apply corrupting an HAProxy pod's on-disk configuration. It now patches the configuration the in-flight deploy actually wrote, so the only on-disk difference is the runtime-eligible server line, and records that push as proving nothing about the running state — the next sync reloads rather than trusting an empty diff.
- Routes no longer silently 404 for 15–30 seconds after a deploy reports success under churn (#84). The runtime-bypass fast path now pushes the last reload-activated config patched with only the runtime-eligible server changes, a structural deploy verifies the on-disk config after its reload (counted as
haptic_deploy_runtime_divergence_total), a headerless on-disk config always forces a reload, and a superseded render abandons its retry instead of re-pushing a stale body. - A failed or timed-out deployment invalidates the deploy scheduler's lane baseline, so a queued runtime-only render re-dispatches as a full structural sync instead of restamping the version header over structural content the workers never loaded (#76).
- A retryable deploy failure self-reschedules with bounded exponential backoff instead of waiting up to a minute for the drift check, and a fully-failed deploy reports
Programmed=Falsewith a reason instead of freezing the last-known status (#72). - A runtime server update against a backend the loaded config does not have yet (
No such backend/No such server) fails fast to the scheduled structural deploy instead of being retried for up to 2 seconds per apply. - The live config gate's validationTests budget scales with suite size (a 25s floor plus ~100ms per test) instead of a fixed 25s cap, so a large all-passing suite is no longer rejected as partially-validated; fail-closed on genuinely incomplete runs is unchanged (#77).
HAProxyTemplateConfigadmission uses the same budget, capped by the configurable config-admission deadline. HAProxyTemplateConfigadmission no longer turns an internal render timeout into a denial that can block the very config update needed for recovery: it has a separate configurable deadline and admits timed-out checks with a warning for the load gate to enforce. Watched-resource admission keeps its tighter fail-closed budget.- An admission request for a kind with no registered validator is logged and counted as
unregisteredonhaptic_webhook_validations_totalbefore being denied with HTTP 503. This means theValidatingWebhookConfigurationand registered validators have diverged. - Every
HAProxyTemplateConfigadmission decision is logged at INFO, including a clean allow — previously indistinguishable from a webhook the API server never reached. - The controller honors the chart's
WEBHOOK_PORTinstead of always binding9443, and an invalid metrics or webhook port fails startup instead of silently falling back. haptic benchmarkgains--schema-dir/HAPTIC_SCHEMA_DIRand resolves the effective spec likevalidate, so it can benchmark configs that use typed-resource access — as the bundled chart does.- Normal configuration reloads no longer emit a false
Metrics component failed: context cancelederror. - The docs site's changelog page is generated from
CHANGELOG.mdat build time, so/docs/dev/always shows the current[Unreleased]changes.
Helm chart¶
Added¶
- New
controller.kubeClient.qps/controller.kubeClient.burstset the controller's client-side apiserver rate limit. The default (qps: -1) disables client-side throttling and relies on apiserver Priority & Fairness; a positiveqpsreinstates a shared client-side cap. - Gateway API per-route filters are driven by map files instead of per-route directives: header modifiers (rule- and backendRef-level),
RequestRedirect,URLRewrite,spec.rules[].timeoutsandRequestMirroreach become a map entry keyed by the route's rule id, read by one static block. Adding, changing or removing any of them is a runtime map update rather than a change tohaproxy.cfg, so it deploys without reloading HAProxy. Two exceptions keep a configuration line: the first route in the cluster to name a given header, and the two shapes one map value cannot describe — a rule matching several path prefixes withReplacePrefixMatch, and a rule whose mirrors sample at different percentages. See the gateway library page. - Gateway route filter values now pass render-time validators: a
RequestRedirectstatusCodeoutside {301, 302, 303, 307, 308}, a scheme other than http/https, a hostname or mirror target that is not a DNS name, a port outside 1-65535, a header name outside the HTTP token charset, and atimeoutsvalue that is not a duration are each reported against the route (denied at admission, warned and skipped on reconcile). - New base macros for libraries that want the same treatment:
RegisterMap(name, lines, opts)writes a map file and declares its entry order, andHeaderModifierRules(direction, keyExpr, mapPath, setNames, addNames, delNames)emits one header line per header name rather than per resource. Both are documented under Map files. - The rendered
globaldeclarestune.bufsize, and on HAProxy 3.4tune.cli.max-payload-size— the ceiling on what one runtime CLI batch can carry, which the controller sizes its batches from. Override withcontroller.config.templatingSettings.extraContext.tune.bufsize/.cliMaxPayloadSize. - The rendered
globaland the HAProxy bootstrap config carry a worker runtime socket,stats socket <baseDir>/haproxy-worker.sock mode 600 level admin. The master socket relays only the first command of a;-joined line and holds no per-connection session state, so batched commands,waitand payload commands need a socket on the worker. - HTTPRoute and GRPCRoute
backendRefs[].filters[]of typeRequestHeaderModifierandResponseHeaderModifierare applied per backend, keyed by rule id and backend name. - New Vector sidecar on every HAProxy pod, enabled by default (
vector.enabled). It receives the HAProxy access log over a mounted UNIX datagram socket (vector.socketPath) and prints it to stdout, so records surface underkubectl logs <pod> -c vector, and re-exports the SPOA hub's metrics alongside its own on one port (vector.metricsPort, default 9598). HAProxy's own exporter is not re-exported: Prometheus scrapes it directly on thestatsport, where it applies the chart's exclusion policy itself (seeextraContext.prometheusExporterbelow). Setvector.enabled=falseto remove the sidecar. - The Vector config follows the SPOA hub's delivery path: HAPTIC renders it and pushes it into general storage, where
--watch-configpicks it up without a restart. A bootstrap ConfigMap seeds the file before regular containers start, a post-start reload closes the initial read/watch race, and a PID 1 supervisor restarts an exited or unresponsive Vector child without withdrawing healthy HAProxy traffic. - New per-request metrics derived from the access log (
vector.requestMetrics, on with the sidecar): one counter and six histograms keyed by route, with the upstream call split into connect, response headers and full response. Names and labels matchingress-nginx's, so its dashboards work — setprefix: nginx_ingress_controllerfor a literal drop-in. Opt out per label (terminationStateLabel,pathLabel,hostLabel) or per family;cardinalityLimitcaps label values at 500 per metric. - New
vector.sizeMetricsPort(9599) exports the two size histograms, since vector's exporter takes one bucket list per sink.haproxy.monitoring.podMonitordeclares both endpoints. - Metrics can be derived from the access log declaratively in
vector.logMetrics, for in-path components with no scrape endpoint of their own. Each entry names a log field and how to project it (kind: enum→ a counter tagged with the value,kind: numeric→ a counter incremented by it), with an optionalrequirespath switching it off. Four ship for the cache tier:haptic_cache_status_total{status},haptic_cache_age_seconds_total,haptic_cache_uncacheable_total{reason}, andhaptic_degraded_cache_total. - New
extraContext.prometheusExportersets the query HAProxy's Prometheus exporter applies to a scrape that sends none, so every scraper gets the chart's exclusion policy — the bundled PodMonitor and a hand-written job alike — and a scraper's own query still wins.excludeMaintServers(defaulttrue) passes HAProxy's?no-maint, dropping the empty reserved-slot servers (67% of the series on a measured fleet, no metric name disappears);excludeMetricsis a map of named exclusions, each withenabled, exactfamilies(sent asmetrics=-<name>) and an optionalrequiresextraContext path. The shipped exclusions drop the never-resettinghaproxy_*_max_*gauges and four more families — 31 families, roughly a third of a scrape;haproxy_backend_agg_server_statusis deliberately kept as the free-slot census, andbackendHttpCompressionships off because compression is on by default. - New
haptic_denied_total{reason}counts every rejection by the control that made it. The*_unavailablereasons are the ones to alert on: they mean a control could not reach its dependency, not that a client hit a limit. - New
haproxy.monitoring.podMonitor(default off): one PodMonitor for every metrics endpoint on the HAProxy pod — HAProxy's exporter on thestatsport, vector's endpoints while the sidecar is on, and the SPOA hub'smetricsport when the hub is on and vector is off (with vector on, vector re-exports the hub over loopback). A hub pinned to a loopback bind that would be scraped directly fails the render with guidance rather than emitting a dead target. - New
HAProxyAccessLogRecordsDroppedalert (on by default) fires when HAProxy discards access-log records because the Vector sidecar stopped draining the socket. The socket absorbs ~167 records at the defaultrmem, about 170ms of stall at 1000 req/s. - New
HAProxyControllerConfigPinnedandHAProxyAgentRecoveryReloadFailedalerts (both on by default). The first fires while HAProxy has refused two renders in a row, so nothing new reaches the pods; the second when a pod's recovery reload failed after a rollback, leaving its worker on a set nobody described.HAProxyAgentInvariantViolatedno longer double-fires on the second one. - New
controller.config.controller.renderGateInterval(default1s) caps how much CPU the render gate'shaproxy -cruns can take from the admission webhook. - New
HAProxyFleetDivergedalert (toggleable) and a Grafana fleet-convergence panel. - Structured JSON access logs. Every frontend emits one JSON object per request (or per connection, on TCP frontends) using HAProxy's native
%{+json}o, with the log target onformat rawso records carry no syslog prefix. The core record covers request identity, the five HAProxy timers (request_time_ms,queue_time_ms,connect_time_ms,response_time_ms,total_time_ms—-1when a phase never happened), retries, the termination state,resource(the<namespace>/<name>that owns the matched route), anddenied_by, which names the gate that blocked a request. HAProxy's own process and health-check lines on that stream are not JSON, so a collector must tolerate them. - Template libraries contribute access-log fields for the features you configure, through a new
log-fields-*extension point: WAF verdict, rate-limit budget, external-auth status, request-schema outcome, cache status, authenticated consumer, TLS version/SNI/resumption, mTLS result, matched Gateway rule, captured request headers, and the real peer behind a forwarding header. A field is emitted only when its feature is in use. - New
accessLog.fieldsadds custom JSON fields from a YAML hash of field name to HAProxy sample expression, andaccessLog.maxLineBytes(default16384) bounds the record. Names and expressions are validated in both Helm and the render. - New
accessLog.targetsroutes the access log away from the container's stdout to an access-controlled destination — the record carriesclient_ip, which is personal data. Each entry takes anaddress(stdout,stderr,fd@<n>,<host>:<port>, a socket path, orring@<name>),format,facilityandlevel, or aringblock for a buffered TCP client that queues records while the collector is down. HAProxy's process messages keep a separate stdout target. Combinations HAProxy accepts and then silently gets wrong — alevelaboveinfo, a Unix-socket ring server, an undeclaredring@<name>, duplicate targets, an empty map, an undersized ring buffer — are rejected at render time. - New opt-in
accessLog.suppress.successfuldrops records for 2xx/3xx requests that no gate denied, keeping every denial, 4xx and 5xx. - Access-log records omit fields whose value is empty (
vector.omitEmptyLogFields, on by default), measured 27% smaller on a real fleet. Set it tofalseif you feed a strongly typed index or have queries written asfield == "". - The access log and trace spans name the backend pod (
server_pod), its Kubernetes Service and namespace. Servers are named after their pods, soserver_podis the chosen server's name (%s) directly — no address→name map. - The access log and trace spans identify which HAPTIC pod served the request (
instance_pod,instance_node) and which address it arrived on (destination_ip). The pod identity comes from the downward API, read once at startup. - New
listener_portaccess-log field: the port the routing lookup was keyed on. Host and path map keys are scoped by it, so a route that matches nothing is now distinguishable from one that matched a different listener's keys. It is logged as a JSON string on every frontend; remove anyaccessLog.fields.listener_portyou added, since colliding with a built-in name is rejected at render time. - New
waf_matched_varaccess-log field names the request fields a WAF rule matched on (ARGS_GET:id,REQUEST_LINE) — never their values. Withwaf_rule_idit gives both halves of a false positive without Coraza's audit log, which writes client IPs and request bodies. - New
cache_ageandcache_uncacheable_reasonaccess-log fields: how old the served object was, and why a response was not stored (content_type_excluded,too_large,set_cookie,status_not_cacheable,origin_refused_sharing). - New
routeandbytes_inaccess-log fields, backing thepathlabel andrequest_size. Each renders only when something reads it. - Hub log lines correlate with the access log: every SPOE message carries
req_id=unique-idand the bundled hub adopts it, so a hub warning and the access-log record for the request that caused it share one key. - New opt-in distributed tracing, owned by HAProxy itself rather than a sidecar (
extraContext.tracing). HAProxy adopts a valid inbound W3Ctraceparent(honouring its sampling decision), mints one otherwise, takes an edge sampling decision (tracing.sampleRate), and setstraceparentbefore the backend sees the request, so your services join the trace;span_id,parent_span_idandtrace_flagsjoin the access log. Settracing.otlp.endpointto also export HAPTIC's own spans: the Vector sidecar turns access-log records into OTLP/HTTP spans, which covers every request including HAProxy-generated 502/503/504s, denials, redirects and errorfiles. Off by default. - Traces carry a span per request phase: a
SERVERspan for the request and aServer sessionCLIENTspan for the upstream call. Spans are named{method} {host}{route}andServer session [backend], withhttp.routecarrying the matched path template (*marks a prefix match) and falling back to the owning resource in brackets, or the bare method, when nothing matched. Phase timings ride along as attributes rather than child spans. - Trace spans carry the request detail the access log has: protocol version, response body size, server port, TLS version/SNI/resumption, consumer, frontend, idle time, and the HAPTIC decision fields (WAF, external auth, rate limit, schema validation, cache, mTLS, Gateway rule id), each named
haptic.+ its access-log field name. No client IP is exported — correlate with the access log throughhaptic.req_id. - Exported spans identify which HAPTIC deployment produced them, through the OpenTelemetry resource attributes
service.namespace,service.version,k8s.namespace.nameandk8s.deployment.name, plusk8s.cluster.namefrom the newextraContext.tracing.otlp.clusterName(no default; Kubernetes exposes no cluster name to a pod). - New
extraContext.tlscipher and protocol policy sets a forward-secret default on every HTTPS bind:tls.ciphers,tls.ciphersuites(TLS 1.3) andtls.minVersion(TLS 1.2 floor). The list is cert-agnostic, so it serves RSA-only, ECDSA-only and dual deployments; each sub-key is overridable, and an empty string omits its directive. - New
extraContext.tls.hsts.enabled(default off) sendsStrict-Transport-Securityon every TLS response, tunable withmaxAge/includeSubdomains/preloadand still overridable per host by thehstsannotation. The render warns when it is enabled without an HTTP→HTTPS redirect. - New
extraContext.tls.sessionTickets.enabled(default off) turns on fleet-wide TLS session resumption: every HAProxy pod shares one session-ticket encryption key, covering TLS 1.2 (RFC 5077) and TLS 1.3 (RFC 8446 PSK). The key is generated in-cluster and self-rotates daily through a 3-key sliding window with one hitless reload. - New
defaultSSLCertificate.ecdsaSecretNamemakes the default certificate dual:secretNamepoints at the RSA Secret andecdsaSecretNameat an ECDSA one, so HAProxy serves ECDSA to modern clients and RSA to the rest on the no-SNI path. Per-host dual certificates already worked. - New
extraContext.ingressDefaultSSLRedirect(default off) redirects every HTTPS-served Ingress host from HTTP to HTTPS with one global toggle;ingressDefaultSSLRedirectCode(default308) sets the status code. Only hosts actually served over HTTPS are redirected. - New
extraContext.proxyProtocol(enabled,httpPort8081,httpsPort8444) adds binds that require a PROXY protocol header, for HAPTIC behind a layer-4 load balancer that would otherwise hide every client behind its own address. They are additional listeners —haproxy.ports.http/httpsstay open, because HAProxy has no optional-header bind. Enabling adds the ports to the Service, the container, and the NetworkPolicy. - New
extraContext.requestBuffering(enabled, default on;waitTimeout, default10s) makes HAProxy wait for the request body before taking a backend connection — the standard slow-POST defence. Only requests declaring aContent-Lengthare held, so gRPC and chunked streaming uploads are excluded by construction. The newhaproxy-haptic.org/request-bufferingannotation (on/off) overrides it per route through a reload-free map. - Shared response cache: an opt-in, chart-deployed Varnish tier (
cache.varnish.enabled) that HAProxy shards by URL with bounded-load consistent hashing, so the cache is shared across the whole fleet. Deployed through the controller'sk8sResourceswith a PodDisruptionBudget, soft node spreading, an optional HorizontalPodAutoscaler and a release-scoped NetworkPolicy. Per-route control viacache-enable,cache-ttl,cache-negative-ttl(for404/410),cache-key(consumer/header/cookie/query/src),cache-exclude-content-types,cache-exclude-pathsandcache-max-object-size; responses carryX-Cache: HIT/MISS/STALE. The cache key includes the resolved backend, an origin'sCache-Control: no-store/privateorVary: *is honoured, a route whose key cannot be expressed downstream is markedCache-Control: private, and credential checks run on the client leg so a cached response is never served to a caller who presented none. - Staleness and revalidation control for the shared cache:
cache-stale-while-revalidateserves a stale response while it refreshes in the background,cache-stale-if-errorreaches the stale copy only when a refresh fails,cache-ttl: autofollows the origin'sCache-Control/Expires,cache-revalidatekeeps an expired object for a conditional refresh, andcache-strip-set-cookiedrops aSet-Cookiethat would make a public asset uncacheable. - Traffic shaping on the native library:
upload-bandwidth-limitcaps bytes per second received from the client alongside the existingdownload-bandwidth-limit, andbandwidth-limit-scopechooses who shares the budget —stream(default),clientorservice. The shared scopes add a backend stick-table, so combining them with the per-sourcerate-limit-*caps fails the render with an actionable message instead of a config HAProxy refuses to start. - Shared request-rate limiting: opt-in
haproxy-haptic.org/rate-limit-requestsannotations enforce one budget across the fleet through the bundled rate-limit SPOA plugin. The chart-managed default store is HA Valkey with Sentinel (three pods, failover, PodDisruptionBudget, NetworkPolicy); bring your own withrateLimit.shared.externalStore.urls.downAfterMillisecondsdefaults to5000and the chart refuses a value at or below3334, which cannot survive a Sentinel TILT window. - JSON request-body validation: opt-in
haproxy-haptic.org/request-schema-*annotations validate POST/PUT/PATCH bodies against JSON Schemas from ConfigMaps or Secrets through the bundled api-gateway SPOA plugin. HAProxy rejects oversized bodies before the SPOE round-trip.extraContext.requestBodyInspection.haproxyBufferconfigures the shared buffer capacity. - A HAProxy PodDisruptionBudget, enabled by default, preserves at least one load balancer during voluntary disruptions and validates against the static or KEDA minimum fleet size.
- Cache and request-schema dependency failures now have
cache_degradedandschema_degradedaccess-log fields and counters. - New
haptic-annotationstemplate library exposes HAPTIC's nativehaproxy-haptic.org/*vocabulary — a superset of the haproxytech, haproxy-ingress and nginx-ingress libraries with one annotation per capability. Enabled by default, and the recommended vocabulary for new configs; the vendor libraries remain for migrating existing annotations. - API-gateway annotations on the native library (pure HAProxy config, no plugin): API-key authentication (
api-key-secret/api-key-header/api-key-query/api-key-consumer-header) backed by a reload-free key→consumer map; stateless request gating (allowed-methods,require-content-type,require-headers,mock-response,fixed-response); and request correlation IDs (request-id/request-id-header/request-id-accept-inbound). - JWT, HMAC and consumer-group annotations on the native library: asymmetric JWT verification (
jwt-secret/jwt-algorithm/jwt-issuer/jwt-audience/jwt-required-claims/jwt-forward-claims) with an alg-confusion guard andexp/nbf/iss/audchecks; HMAC request-signature verification (hmac-secret/hmac-algorithm/hmac-header/hmac-signed-string); and consumer-group authorization (consumer-groups-secret/allowed-consumer-groups). All fail closed with503when their Secret is missing. - Response compression annotations on the native library (
compress-enable/compress-algorithm/compress-types);brotliandzstdfail the render, being unavailable in the community HAProxy build. - Reusable Coraza WAF policies for native Ingress annotations: administrators define bounded policies in chart values or RBAC-protected cross-namespace ConfigMaps (
waf.policies.configMapRefs), and Ingress authors select one withhaproxy-haptic.org/waf-policy, optionally overriding enforcement withwaf-mode. Mode, vendor rule snippets and raw HAProxy escape hatches are centrally authorized and default-deny. - Structured, self-service-safe CRS tuning on WAF policies:
allowedMethods,paranoiaLevel,anomalyThreshold.inbound/.outbound,crsSettings(content-type allowlist,maxFileSize,maxNumArgs,totalArgLength) andruleExclusions(by rule id or tag, optionally scoped to a literal path). They lower to chart-owned SecLang spliced at the crs-setup position, so no tenant-authored regex runs on the request path and enforcement itself cannot be disabled. - Namespaced self-service WAF policy authoring (
waf.policies.selfService): withenabled: true, every namespace defines policies for its own Ingresses in a well-knownwaf-policiesConfigMap. Policies are namespace-scoped, trusted-catalog names win and collisions are rejected loudly, a broken policy fails only that namespace's selecting routes closed with503plus a Warning Event,secLangneeds the separateallowSecLanggrant, per-namespace and total budgets cut deterministically, and a self-servicedetectpolicy cannot weaken adefault-onbaseline. - New
extraContext.waf.crs.urlreplaces the OWASP CRS compiled into the coraza plugin with one fetched over HTTP, so a CRS release can be adopted without rebuilding the plugin image. HAPTIC writes each rule file to general storage and substitutes only the@crs-setup.conf.exampleand@owasp_crs/*.confincludes, leaving rule ORDER alone. Refresh is a conditional GET onrefreshInterval(default 1h); a fetch failure never fails the render and falls back to the deployed ruleset, then to the plugin's embedded CRS. Adopting a ruleset reloads neither HAProxy nor the hub. - Org-wide policy guardrails (
extraContext.governance.*): cluster admins declare generic, JSONPath-driven rules that namespace teams cannot omit.rulesis a map keyed by a name you choose, so rules a library ships and rules you add merge instead of replacing each other. Each rule targets a watched resource by name and either injects adefaultwhen a value is absent — flowing into the same render — or validates the present value withrequired,min/max(withonViolation: clamporreject),allowed,pattern,anyOforsatisfiedBy: tls.enforcement: rejectdenies a violating resource at admission and records aGovernanceViolationWarning Event for existing ones;enforcement: auditwarns only.exemptNamespacesskips infra namespaces. Disabled by default. - New pre-rollout validation gate (
preRolloutValidation.enabled, default on): apre-install/pre-upgradeJob renders the chart embedded in the controller image with the release's own values and runs the full load gate — includinghaproxy -c— before any object is applied, so a failing configuration fails the release with the previous one still serving. Argo CD runs it asPreSync. The Job hard-fails on chart/image version drift. - A
pre-install/pre-upgradeJob applies the bundled CRDs on every install and upgrade (crds.upgradeJob.enabled, default true), runninghaptic apply-crdsunder its own scoped RBAC (customresourcedefinitions, neverdelete) and removed on success. This makeshelm upgradeand GitOps sync pick up additive CRD schema changes, which Helm never applies for CRDs incrds/. - New
haproxy.dataplane.logLevel(defaultinfo) replaces a hardcodedtracefor the Dataplane API sidecar, which logged ~671 lines for a single startup plus config cycle. - New
haproxy.org/cors-respond-to-options: when"true", HAProxy answers the CORS preflight with a 204 instead of forwarding it, matching the upstream HAProxy Kubernetes Ingress Controller. - New
spoaHub.plugins.<name>.adaptiveConcurrency(default false per plugin) lets the hub resize a plugin's admission semaphore from live request latency, withmaxConcurrencyas the ceiling rather than a fixed limit. - The controller webhook timeout is configurable as
controller.webhook.timeoutSeconds(default10); the chart validates Kubernetes' limit and derives a one-second-shorter controller deadline. - Each vendor annotation library declares machine-readable migration coverage, used to generate the per-source annotation-support tables in the migration guide.
- Ingress request/response header modifiers are driven by map files instead of per-backend or per-host directives: the
request-set-header/response-set-header(haproxytech, haproxy-haptic.org),custom-request-headers/custom-response-headers(nginx) andheaders(haproxy-ingress) annotation values become entries ining-reqhdr.map/ing-reshdr.mapkeyed by the resolved backend, read by one static line per header name. Adding or changing a header on a route that already uses that name is a runtime map update, not a change tohaproxy.cfg. - The settable per-backend timeouts move to a map:
timeout-serverandtimeout-tunnel(all four annotation libraries; nginx'sproxy-read-timeout/proxy-send-timeoutcollapse into the server timeout) become integer-millisecond entries inbackend-timeouts.map, read by one uniformhttp-request set-timeoutline every ingress backend carries. Editing a timeout is a map-only change. The non-settable timeouts (connect,queue,http-request,http-keep-alive,check) stay as backend directives. A value that is not an HAProxy duration is reported against the ingress at render time. - The new author contract for reload-free routing — the
Backend()slots (profile/body/servers/serverLines), the when-does-a-backend-reload table, and where to put a directive — is documented for custom CRDs, Ingress and Gateway alike on the Reload-free routing page.
Changed¶
- The controller's RBAC is tightened to least privilege. Grants the controller exercises only in the release namespace move from the cluster-wide
ClusterRoleto the namespacedRole:podsget/list/watch(the HAProxy-pod watch is pinned to the release namespace); the leader-electionleases(the Lease and its fencing epoch live in the controller's own namespace); thehaproxytemplateconfigs/haproxytemplatelibrarieswatch, owner-stamp and status writes; and the output CRDs (haproxycfgs,haproxygeneralfiles,haproxycrtlistfiles,haproxymapfiles) with their status — previously only two of the four were namespaced. The hardcoded cluster-widenamespacesgrant is removed — the only namespaces watch is the gateway library's, already granted through the watched-resources rules when that library is enabled.customresourcedefinitionsread, cross-namespaceeventswrites, and the gateway library's cross-namespaceservices/gatewayclasseswrites stay cluster-wide because their targets are cluster-scoped or in other namespaces. A goldenhelm unittestsnapshot pins the rendered grants so future drift is a deliberate update. - The admission webhook now validates
Gateway,BackendTLSPolicy,TLSRouteandTCPRoute(previously stored unchecked and only caught later at the config-load gate), so a malformed one is rejected atkubectl applyinstead of poisoning the whole config. The rules cover the object spec onCREATE/UPDATEonly, never thestatussubresource, so the controller's own status writes don't re-enter admission.GatewayClassstays unvalidated by design — the controller emits its own via Server-Side Apply, so an admission rule would intercept that write. - The chart-size guard band is raised to 960,000 bytes to accommodate the map-driven gateway filters and their required validation tests; a release-split is mandated above 975,000 (see
scripts/check-chart-release-size.py). - The nginx (
custom-request-headers/custom-response-headers), haproxy-ingress (headers) and haproxy-haptic.org (request-set-header/response-set-header) header annotations now apply to the ingress's own backends rather than to every request for the ingress's hosts. For a host owned by one ingress this is the same set of requests; for a host shared across ingresses it is more precise, and it fixes the case where a wildcard or host-less ingress silently applied no header. haproxytech'srequest-set-header/response-set-headerwere already backend-scoped. - A header set by two ingresses under differing capitalization is emitted once, spelled the lexicographically smallest of the two, so the config no longer depends on ingress order.
- A Gateway
RequestRedirectpreserves the query string the request arrived with. HAProxy's ownkeep-querydoes not exist on 3.0, so the query is composed explicitly. - A
backendRef-levelResponseHeaderModifiernow overrides a rule-level one for the same header, as the specification requires. Both run in the frontend, backendRef-level second; the old placement put backendRef-level modifiers in the backend section, where HAProxy evaluateshttp-responserules before the frontend's, so the rule-level modifier won. - Two Gateway routes that spell one header name differently share a single configuration line, spelled the lexicographically smallest of the spellings in use. Header names are case-insensitive, so this is the same header.
- Every map the bundled libraries write is registered even when it has no entries, and declares whether its entry order matters. A map created on the first entry and deleted with the last costs two reloads; an unordered map can instead take a runtime append.
map-path-regex-500-gatewayemits its entries in Gateway API precedence order.map_regreturns the first match, and the previous emitter wrote routes in list order.- The Gateway advanced-matcher pass skips routes the two route maps already resolve, instead of processing every route and emitting nothing for the map-decidable majority. This cuts the frontend render for the common plain-route case at scale; the emitted config is unchanged.
- BREAKING: the HAProxy pod's
dataplanecontainer is nowagentand runs the HAPTIC image, so HAProxy pods pull from the controller's registry;haproxy.podSpec.imagePullSecretsdefaults to the controller's. The<release>-haproxy-dataplaneService, itsdataplaneport name and port 5555 are unchanged — a Deployment's selector can't be changed in place — and the credentials Secret keeps itsdataplane_username/dataplane_passwordkeys. The pod'sglobalsection gains a workerstats socket, which is what carries every runtime command. - The HAProxy pod exposes an
agent-metricsport, scraped by the bundled PodMonitor.haptic_agent_*counters report what each pod did with an apply. - Four
controller.config.dataplanefields changed meaning without changing name:minDeploymentIntervalis the shortest interval between two reloads of one pod (the chart passes it to the agent),driftPreventionIntervalis how often a pod re-hashes its tree,reloadVerificationTimeoutis how long the agent waits for a reload, andsyncTimeoutis how long the controller waits for a pod to answer. - Every
backendsection the bundled libraries emit is declared through the baseBackend()macro (util-backend), which builds the section text from a record it hands the controller. A library writing abackendsection by hand keeps working; only what goes throughBackend()is described to the controller as data.BackendServers()returns server records instead of server lines, so a library that called it must pass the result toBackend()asserversrather than showing it. - Ingress and Gateway API backends are now dynamic-eligible: their defaults-legal directives (
default-server, server/tunnel timeouts, retries, session cookie, the BackendTLS fail-closed 503,balance) move into the shared profile, and namespace/service move into a newbackend-service.map(read by the access log'snamespace/servicefields viavar(txn.backend_name)), so a plain backend's section is onlyfrom/guid/serverlines — on HAProxy 3.4 such a route is added or removed at runtime without a reload. Stick-table rate limiting, bandwidth/compression filters and raw operator injections keep their backends structural. The per-installationdynamic-cookie-key(extraContext.dynamicCookieKey, defaulthaptic-dynamic-cookie) replaces the per-backendsha256(<backend>)so cookie backends of the same shape share one profile. - Every backend inherits from a content-addressed named
defaults haptic-be-<hash>profile (backend <name> from haptic-be-<hash>). Backends of the same shape — mode, balance, hash-type,default-serverkeywords and shared directive lines — share one profile section, so on HAProxy 3.4 a route of an existing shape can be added at runtime without a reload. The anonymousdefaultsbecomes the rule-free parentdefaults haptic-base; a trailing, never-referenceddefaults haptic-implicit from haptic-baseis what proxies without an explicitfrominherit.tune.defaults.purgeis never emitted (it would make named defaults un-inheritable byadd backend). - The SSL passthrough loopback's
serverline no longer runs into the next snippet's comment (send-proxy-v2# gateway/...). - Behaviour change for hash-based load balancing:
balance/hashTypeare structuredBackend()arguments (carried by the profile, not a rawbalanceline), and the chart injectshash-type consistentby default for every hash-family algorithm (source,uri,url_param,hdr(),rdp-cookie,hash <expr>) — including thehaproxy.org/load-balance,haproxy-haptic.org/load-balance,haproxy-ingress.github.io/balance-algorithm,nginx.ingress.kubernetes.io/load-balance: ip_hashandupstream-hash-byannotations. Consistent hashing lets a pod be added/removed at runtime without a reload (plainadd serveris refused on a map-based hash), and only remaps its own share of keys on a pod change instead of rehashing every key. Annotation users who relied on map-based (modulo-N) distribution get consistent hashing instead;static-rr(and an explicithashType: map-basedfor a chart author) opt out and reload on pod churn. - Backends hold one
serverper endpoint, named after the pod (server <pod> <ip>:<port>), instead of a fixed pool ofSRV_Nslots with unroutable placeholders. The rendered file always equals the current pod set (ADR-0011), soshow servers state, logs and per-server metrics read the pod name, and a rolling update is an add/remove of named servers over the runtime API. Not-ready and terminating endpoints stay in the config asdisabledservers (they take no traffic), so a readiness flip is a runtimeset server staterather than a del+add. - Gateway API backends are rendered only for resolvable
backendRefs; abackendRefwhose Service doesn't exist (or isn't permitted by aReferenceGrant) no longer produces a placeholder-only backend section, halving the config for routes with an invalid sibling ref. - BREAKING: the chart renders one
HAProxyTemplateLibraryper enabled template library, named<controller.configName>-<library>, plus<controller.configName>for your owncontroller.config, which references them throughspec.libraryRefs. Anything that post-processeshelm templateoutput expecting exactly one object, or reads the whole config out of one, must be updated —haptic config view --inputprints the merged configuration andvalidate -faccepts a multi-document stream. Only the config object is yours to edit; override a snippet by name undercontroller.config.templateSnippets. The split exists because the single merged object had reached 99.4% of Kubernetes' ~1.5 MiB per-object limit withnginx-ingressenabled. - BREAKING: controller workload values moved under
controller.*(image,replicaCount, probes,resources,serviceAccount,rbac,service,securityContext,extraEnv/volumes/sidecars,autoscaling,podDisruptionBudget,monitoring,networkPolicy,webhook), and flatextraContextkeys are restructured intodiagnostics/statusPatches/annotationCompatibility/tlstrees. The routing diagnostic response headers (formerlydebug, on by default) are now opt-in viadiagnostics.routingHeaders.enabled. A value left at its old path fails the render with a message naming the new one. - BREAKING: one owner per runtime setting. Rename
controller.crdName→controller.configName,controller.debugPort→controller.ports.healthz,controller.config.dataplane.port→haproxy.ports.dataplane,controller.config.routing.regexMatchOrder→controller.config.templatingSettings.extraContext.routing.regexMatchOrder, andcontroller.defaultSSLCertificate→ top-leveldefaultSSLCertificate. Remove the no-opcontroller.config.controller.healthzPortandmetricsPort. Legacy paths fail with an explicit migration error. - BREAKING:
haproxy.enterprise.versionwas removed.haproxyVersionnow selects the controller compatibility series, Enterprise image revision and derived binary path together; an emptyhaproxy.image.repositoryderives the registry fromhaproxy.enterprise.enabled. - BREAKING: the access log is JSON on every frontend.
option httplog/option tcplogoutput is gone and the log target usesformat raw, so any pipeline parsing the previous text shape must be updated; records carry their owntsfield (microsecond precision) in place of the syslog timestamp. Overrideutil-log-format-http/util-log-format-tcpviacontroller.config.templateSnippetsto keep a text format. - BREAKING: request correlation IDs are opaque RFC 9562 UUIDv7 values, emitted for every request rather than only when an Ingress opts into
request-id. The previous format embedded the client IP — personal data — in a value forwarded upstream and copied into application logs.request-id-accept-inboundnow preserves a client-supplied id only when it matches^[A-Za-z0-9._:-]{1,128}$.unique-id-formatmoved tobase.yaml; use adefaults-settings-*snippet above band 150 for a custom format. - BREAKING: the vendor annotation libraries (
haproxytech,haproxyIngress,nginxIngress) are disabled by default; the nativehaproxy-haptic.org/*library is the only annotation library on by default. Enable the matching library explicitly if you use those prefixes. The native vocabulary is a superset of all three, and prefixes from different families can coexist on one Ingress as long as each feature comes from a single family. - Ingress is served over HTTPS by default: any Ingress binds
haproxy.ports.httpsand terminates TLS with the default certificate — nospec.tlsrequired. SetextraContext.ingressDefaultHTTPS: falseto keep Ingress plaintext-only. - HAProxy compresses responses by default. A bundled governance rule injects
haproxy-haptic.org/compress-enable: "true"on any Ingress that does not set it, so the default is visible in the rendered config; an Ingress setting it either way keeps its own value. Only responses the backend left uncompressed are touched, and HAProxy addsVary: Accept-Encodingitself. Note the BREACH tradeoff: a route whose HTTPS response carries a secret and reflects attacker-controlled input should opt out withcompress-enable: "false", or fleet-wide viagovernance.rules.haptic-compress-enable.enabled=false. defaultssetsretry-on conn-failure empty-response response-timeout, closing the half of the pod-termination raceoption redispatchcould not — clients sawSH--502s during every rolling update. Retries that replay the request are limited to idempotent methods (RFC 9110), matching nginx-ingress; this narrowshaproxy-haptic.org/retry-on, which previously replayed non-idempotent methods too. SetextraContext.retryNonIdempotent: trueto restore it, orextraContext.retryOnto change the condition list.- HAProxy
nbthreadis derived from the CPU limit, not the CPU request. With no limit the directive is omitted, so HAProxy auto-detects all node cores; withhaproxy.resources.limits.cpuset it rendersnbthread = ceil(limit). Previously the default 250m-request pod was pinned to a single thread. Sethaproxy.nbthreadto pin a value. - The shared Varnish cache uses the canonical HAProxy+Varnish "sandwich" topology: Varnish fetches misses from a dedicated internal backend-fetch frontend (
cache.varnish.loopbackPort, default8090) instead of looping back through the client-facing one, so the WAF, rate limiter, external auth and routing run exactly once per request. The loopback is routed to the client leg's chosen backend through an internal header, so rewritten and weighted routes stay correct. The port is internal-only and restricted to Varnish by the NetworkPolicy. - The coraza and api-gateway plugins default to adaptive concurrency with a memory-derived ceiling, so WAF concurrency needs no manual tuning: the hub finds the right concurrency from live request latency and
maxConcurrencybecomes a ceiling derived fromspoaHub.resources(~16 at the 256Mi default). SetadaptiveConcurrency: falseor a literalmaxConcurrencyto opt out. - The SPOA hub sidecars set
GOGCfrom the newspoaHub.hub.goGCPercent(default300) and deriveGOMEMLIMITas a soft cap at 90% of the container memory limit, trimming GC frequency on the request path. - The bundled SPOA hub is v0.12.0. It processes pipelined SPOE NOTIFY frames concurrently and bounds in-flight frames with a process-wide backpressure gate, so HAProxy is backpressured instead of the hub shedding 503s under saturation (0% 503 across a c=16→96 WAF saturation sweep); it accepts the files a validated config references, so a ruleset included by path is checked at admission; it adopts the access log's
req_id; and it adds SPOE data-path latency metrics (spoa_backpressure_wait_seconds,spoa_ack_flush_duration_seconds,spoa_ack_batch_size,spoa_inflight_frames,spoa_plugin_adaptive_limit). - The bundled coraza plugin is v0.10.0. It recompiles its WAF in place when the rule files its
Includedirectives name change (which is what makes a fetched CRS take effect), and adds per-matched-rule metrics (plugin_coraza_rule_hits_total{phase, rule_id, severity, app}) that count every rule that fired, including on allowed traffic — the signal a detect-mode policy needs before flipping to deny. - The bundled rate-limit SPOA plugin is v0.3.1. Shared limits fall back from Valkey to bounded per-sidecar enforcement for lease and exact GCRA, then fail open if the plugin can't answer;
rateLimit.shared.failClosed=truekeeps strict denial, and degraded decisions are observable throughrate_limit_degradedand plugin metrics. A shared-store connect that never resolves no longer pins a sidecar in that fallback for good after a Sentinel failover. - Request mirroring no longer allocates a per-target SPOE message slot. Each mirror source appends its target to a per-request list the single static
mirrormessage ships to the plugin, so adding or removing a target changes only the HAProxy frontend — neverspoe.confor the hub TOML — and there is no cap on mirror targets. RemovedspoaHub.haproxy.mirror.minMessageSlots. - Cache-enabled GET/HEAD requests bypass Varnish when no semantically healthy shard is available; other methods always go directly to the application. Local and origin-loop probes distinguish a live cache process from one unable to reach HAProxy, while a two-stage dispatch retries cache failures before response delivery directly.
cache.haproxy.responseTimeoutMssets the Varnish-hop inactivity timeout. - Each SPOE message has its own HAProxy engine and processing deadline, and the shared backend performs SPOP health checks, so a slow authentication dependency no longer gives rate limiting, WAF, schema validation, or mirroring its multi-second failure budget.
- Dataplane API, SPOA hub, and Vector children run under exit and health-check supervisors without sidecar readiness probes. HAProxy's activated-config endpoint is the chart's only configured readiness probe; whole-container failures still affect Kubernetes pod readiness.
- Managed Valkey uses
noeviction, native shared rate-limit annotation rules whose refill horizon exceeds the plugin's one-hour state lifetime fail validation, and external-store sharding is rejected because the bundled plugin shares one circuit breaker across its shards. - Mirror requests use their own bounded target budget (
2000ms, zero retries by default) instead of inheriting application timeouts and retry counts. - Request-schema validation allows a request the plugin could not judge, matching the WAF and rate limiter. Set
apiGateway.requestSchemaValidation.defaultFailOpen: falsefor the old posture. - A missing auth or CA Secret no longer denies an Ingress at admission. The webhook renders against the watched-resource store, which lags the API server, so "Secret missing" and "Secret not observed yet" are indistinguishable and applying a Secret with its Ingress could be denied intermittently. The route still fails closed with
503and recovers once the store catches up. Rejection is kept for cases that cannot race: a malformed value,auth-type: basicwith noauth-secret, and an empty group list. - A misconfigured routing or presentation annotation on an already-present Ingress warns instead of bricking the fleet: a bad redirect code, malformed rewrite, or invalid CORS/canary/mirror/compression value records a Warning Event and skips just that feature during a live reconcile, while admission still rejects the change. Security features (authentication, mTLS, WAF, rate limiting, body validation,
X-Forwarded-For) still hard-fail in every mode. - Cross-family annotation collisions are caught: an Ingress configuring the same feature through two annotation families is rejected at admission with a message naming the feature, the families and the annotations, and an existing one records an
AnnotationFamilyConflictWarning Event. The check fires even when the values agree, since the result would otherwise depend on library render order. Different features across families stay allowed. - The governance rule engine moved to its own
governancetemplate library (on by default) fromingress-annotations-compat, where disabling the Ingress annotation scaffold silently took governance with it. Rules andexemptNamespacesare unchanged. cache-enablecan be combined withhmac-secretandconsumer-groups-secret: both are now enforced on the client leg, before the cache is consulted, asapi-key-secretalready was.consumer-groups-secretadditionally requiresapi-key-secreton a cached route, since only api-key resolves the consumer identity on the client leg.- The chart's cluster-scoped objects — the controller
ClusterRole/ClusterRoleBinding, theValidatingWebhookConfigurationand the CRD hook's RBAC — are named with the release namespace appended, so one release name can be installed into two namespaces. TheIngressClassand the CRDs keep their stable names. - The
HAProxyTemplateConfigadmission webhook entry is removed from the chart, andapply-crdsdeletes it from live webhook configurations before manifests apply.controller.webhook.haproxyTemplateConfig.*values are gone. - The template libraries moved from
charts/haptic/libraries/into conditional subcharts undercharts/haptic/charts/. Helm does not store subchart source in the release Secret, cutting the estimated payload by 287 KB. Rendered output is byte-identical; no values change. - Migration coverage is build-time tooling data instead of a CRD field. The playground loads per-source assets generated from each annotation library, and rendered config and library objects no longer carry the 88,909-byte metadata copy.
- The Vector sidecar no longer re-exports HAProxy's Prometheus exporter; Prometheus scrapes it directly on the
statsport, and HAProxy applies the exclusion policy itself. Measured standalone at 2,500 backends, the re-export was 1.3 GB steady / 2.3 GB peak of vector's memory — 90% of it — and no allocator or scrape-interval setting moved that.vector.excludeMetricsandvector.excludeMaintServerMetricsmoved toextraContext.prometheusExporter(patternis gone: the exporter filters by exact family name), andvector.podMonitorandspoaHub.monitoring.podMonitormerged intohaproxy.monitoring.podMonitor; the old keys fail the render with the new location. vector.resourcesdefaults to a256Mirequest and1Gilimit. Its memory tracks the request-metrics series (traffic shape, bounded bycardinalityLimit), measured at 146 MB idle and 364 MB steady / 698 MB peak with 5,000 distinct routes at 500 records/s. The supervisor restarts an exited or unresponsive child, while a whole-container OOM can still briefly affect pod readiness. The container also setsMALLOC_ARENA_MAX=2to bound glibc's per-thread arenas.- The Vector and SPOA-hub configs no longer reload HAProxy when their content changes. Both are watched by their own sidecar and never opened by HAProxy, so a WAF policy edit or an access-log field change reaches the sidecar without touching HAProxy.
spoe.confstill reloads, since HAProxy'sfilter spoereads it. - The internal h2c demultiplexer frontend sets
option dontlog-normal; every connection it accepts is logged by the inner HTTP frontend that handles the request. Abnormal terminations still log. - Shared source-IP rate limits run before Coraza and request-schema validation, so rejected floods do not consume API-gateway processing; consumer-keyed quotas run after native API-key/JWT authentication instead of falling back to source IP before the identity existed.
spoaHub.plugins.<name>.enabledtemplate strings must resolve to exactlytrueorfalse; any other value fails the render instead of being silently coerced to disabled.- The chart fails the render when
controller.config.dataplane.mapsDir,sslCertsDirorgeneralStorageDirpoint somewhere the bundled HAProxy pod cannot honour — a value outside the mounted volumes silently landed rendered files where the sidecars could not read them. Unaffected withhaproxy.enabled=false. - The chart fails the render when
controller.config.dataplane.minDeploymentIntervalorreloadVerificationTimeoutis not a Go duration of at most 60s. The chart passes both to the agent, which refuses a larger value and exits at startup, so every HAProxy pod would crash-loop. Unaffected withhaproxy.enabled=false, where neither key becomes an agent flag. - An unknown key under
haproxy.agentfails the render naming it, alongside the existing guard on thehaproxy.dataplane.*keys it replaced. - The
app-root.map,mtls-error.mapandhsts.mapbuilders collapse into one shared macro. Rendered output is unchanged.
Removed¶
- The
SRV_Nserver-slot pool and its192.0.2.1:1 disabledplaceholders, thepod-names.mapfile, and thecontroller.config.templatingSettings.extraContext.serverSlots.{increment,minFree}knobs are gone — pod-named servers make them unnecessary.extraContext.serverSlotsfails the render (operator input) with a message; thehaproxy.org/scale-server-slotsandhaproxy-haptic.org/scale-server-slotsIngress annotations are tolerated but no longer do anything (warned via a Warning Event, since a tenant value must not abort a render). - BREAKING:
haproxy.dataplane.*moved tohaproxy.agent.*:logLevel,resources,extraEnvandservicekeep their meaning under the new prefix.haproxy.dataplane.validateConfig,haproxy.dataplane.debugSocketPath,haproxy.dataplane.aclFormatandhaproxy.dataplaneBinconfigured the Data Plane API and are gone with it. The chart fails the render on every removed key, naming its replacement, rather than ignoring it. - BREAKING: the
otelSPOA hub plugin is gone, along withspoaHub.plugins.otel. Tracing no longer needs it: HAProxy mints and propagates W3C trace context itself and the Vector sidecar derives OTLP spans from the access log, without a SPOE round-trip per request. RemovespoaHub.plugins.otelfrom your values — the chart fails the render with a message naming the replacement, since a leftover block would be written into a hub config for a plugin the image no longer contains.
Fixed¶
- An install without the SPOA hub failed to load its configuration, and shared-rate-limit and request-schema annotations were silently ignored. The haptic-annotations library imported a spoa-hub-only macro at compile time, so any install without the hub (no Gateway API CRDs, no SPOA plugin) could not compile its config at all. With that fixed, the opt-in guards for
haproxy-haptic.org/rate-limit-requestsandhaproxy-haptic.org/request-schema-*— which previously lived only in map producers the hub renders — now fail loud on a hub-less install instead of dropping the annotation (fail-open on a rate limit and on request validation). - The vendor annotation libraries left several config-injection sites unguarded after the map/named-defaults refactors. A capture-group
rewrite-target(haproxy-ingress,nginx-ingress) wrote its raw value into areplace-pathline — a newline injected a directive — while its literal twin was already guarded;nginx-ingressalso leftproxy-connect-timeout,permanent-redirect-code/temporal-redirect-codeandupstream-hash-byunvalidated (directive /balanceinjection), and itssession-cookie-hashandauth-tls-secret-not-found render comments interpolated the tenant value.haproxy-ingressdid the same in itssecure-verify-ca-secret/secure-crt-secret/auth-tls-secretcomments. Finally, a basic-auth password hash from a referenced Secret (haproxytech,haproxy-ingress) was checked only against the operator-tunable hash regex, whose default^.*$accepts a space — enough to appendgroupsor other keywords to theuserlistline. Every such value now routes through the shared guards: routing/presentation values are denied at admission and warned-and-skipped under reconcile, the Secret-name comments are name-free, and the hash andupstream-hash-byfetch fail closed. - Map values carrying a space or a
;were silently corrupted on the runtime path. The HAProxy runtime CLI splits a map value at the first space and treats a;as a command separator, so everyhsts.mapvalue withincludeSubDomainswas truncated when it reached a running worker, and a redirect location, app-root path or mTLS error page with a space fared the same. Those values are URL-encoded now and decoded in the configuration withurl_dec(1). - The same corruption affected the ingress-written maps.
reqhdr-host.map(haproxytechset-host, nginxupstream-vhost),reqhdr-connection.map,reqhdr-xfwd-prefix.mapandpath-rewrite.map(nginx and haproxy-ingressrewrite-target) stored their values unencoded, so an upstream Host, a Connection value, a forwarded prefix or a rewrite path with a space or a;broke when applied over the runtime CLI. Every writer URL-encodes and every reader decodes withurl_dec(1). - The Ingress render was superlinear above ~1500 routes. The three path maps each recomputed the whole route-ownership map — once per shard, per map — over every Ingress in the cluster. That work is computed once per render now (
shared.ComputeIfAbsent), which the path maps read from. - A Gateway route matching on a
RegularExpressionpath got no filters, no timeout and noresourcefields in the access log.path-regex.mappointed straight at a backend, sotxn.gw_rule_idwas never set for those routes. It emits the sameGW_ROUTE_IDqualifier as the exact and prefix maps now. - A Gateway
URLRewriteorRequestRedirectwithReplacePrefixMatchagainst the/prefix no longer concatenates the replacement onto the whole path (/xbecame/barxinstead of/bar/x). The remainder is taken from the matched prefix's byte length rather than a regex. - A Gateway header modifier value containing
%is inserted literally instead of being re-read as a log-format string, which could fetch request-scoped state into a tenant's own header. - [security] A Gateway API CORS filter could smuggle HAProxy directives or leak request state into a response header. An HTTPRoute/GRPCRoute
type: CORSfilter'sallowMethods,allowHeaders,exposeHeadersandallowOriginsreached the emittedaccess-control-*headers and the origin-m regACL unguarded, so a%was re-read as a log-format fetch (e.g.exposeHeaders: ["%ci"]leaked the client source IP into a tenant-controlled header) and a"or control character could break out of the directive. List values are now HAProxy-log-format escaped and control characters rejected; an origin with a"or control character is rejected — the offending route's CORS block warns-and-skips on reconcile and is denied at admission, while legitimate methods, headers and origins render unchanged. haproxy.dataplane.validateConfig: trueworks: the Dataplane API hands itsvalidate_cmdthe transaction file inDATAPLANEAPI_TRANSACTION_FILEand substitutes nothing, so the previoushaproxy -c -f %sfailed every push withCannot open configuration file/directory %s. The chart now installs a wrapper script that reads the variable.- [security] A cross-namespace
backendRefwithout a coveringReferenceGrantcould still receive traffic. In a weighted HTTPRoute/GRPCRoute rule the weighted map listed every ref, so an unpermitted ref that sorted first took the rule's whole traffic share; TCPRoute and TLSRoute passthrough backends never checked the grant at all. Every backend selector now shares one validity verdict (util-backend-ref-valid): unpermitted and unresolvable refs get no backend section, their weighted share answers500(gw-invalid-backend), and TCPRoute/TLSRoute drop the ref. - The weight share of an invalid
backendRefin a multi-ref rule now returns500, as the Gateway API spec requires, instead of leaking to the valid siblings or hitting a placeholder backend (503) depending on list order. - The OTLP span transform in the Vector library no longer triggers a VRL
E620compilation warning at every Vector start or reload. - Gateway API request latency and throughput no longer degrade with the number of HTTPRoutes. Every route emitted ~4
http-requestlines into the HTTP frontend — each running a regex for aPathPrefix— and HAProxy evaluated all of them, linearly, on every request. An HTTPRoute whose rules all match on path alone is now resolved by the host/path maps plus two new O(1) maps (route-winner.map,route-backend.map); the per-route chain is emitted only for an HTTPRoute where some rule matches on more than the path, which is the only case whose winner depends on the request. Measured at ~500 routes: 2,022 → 112 frontendhttp-requestlines, 2,165 → 4,562 qps and 0.455 → 0.221 ms p50 at one connection, 23,480 → 52,540 qps and 46.1 → 9.6 ms p99 at 64 connections. - Two Gateway API routes sharing one hostname and path could leave a request with no backend. The set of candidate route IDs was built twice — sorted in the route analysis, in emission order in the path map — and the two were then compared as strings. A group of two or more routes whose IDs sort differently than they are emitted never matched, so the request fell through to the default backend and answered 404. Both sides now read one string, built once. Reachable with two Gateways serving the same hostname and path, which is what
GatewayHTTPListenerIsolationexercises. - nginx-ingress annotation values could smuggle HAProxy directives or map entries. Values reaching a config line, a whitespace-delimited map key/value, or an unquoted ACL operand were interpolated without a guard, so a control character (or a quote/space where the context forbids it) could inject a second directive or routing-map entry that
haproxy -cstill accepts.server-alias,canary-by-header/-value/-pattern/-cookie/-weight,custom-request/-response-headers, andsession-cookie-name/-pathnow warn-and-skip the offending value (deny at admission);auth-url/-signin/-method,auth-realm, andlimit-rps/-rpm/-connectionsfail closed. Header values are HAProxy-log-format escaped and the realm is quote-escaped, and canary regex/exact match values are single-quoted, so a legitimate%,"or backslash (e.g. a\dcanary pattern) is preserved rather than executed or rejected. - Resource-scoped chart validation recognizes every store alias attached to the object under admission, so a duplicate GVR alias cannot bypass a reject-mode check.
- Warning Events were lost whenever the
ingresslibrary was disabled. The ClusterRole'seventsgrant was gated oncontroller.templateLibraries.ingress.enabled, butrecordEventis a generic engine function that thegovernance,haptic-annotations,ingress-annotations-compatand vendor annotation libraries all call independently — so the rendered configuration kept itsrecordEventcall sites while the grant disappeared, and every Event they emitted was refused. The grant no longer follows any single library. - Any pod in the cluster could reach the Dataplane API port and the cache's backend-fetch port with
haproxy.networkPolicy.allowExternalat its defaulttrue. NetworkPolicy rules are additive, so the permissive port-matching-everything rule unioned with the narrow rules meant to fence those ports off, and those rules never restricted anything. Permissive mode now emits the port ranges around both, making each narrow rule the only path to its port. - Client-certificate verification could disappear silently. When the
auth-tls-secretCA could not be resolved, the render dropped theca-file … verify requiredclause but kept serving the host — and because every HTTPS host terminates on one shared bind, the route simply answered with ordinary server-side TLS. An Ingress naming a missing Secret is now rejected at admission, and a route whose Secret is deleted afterwards is denied with503. - An upstream TLS connection could go unverified while the operator believed it was checked. Naming
backend-ca-secretis the request to verify the upstream certificate, but an absent Secret — withoutbackend-verifyalso set explicitly — fell through tossl verify none. It now fails the same way an explicitbackend-verify: onalready did. An unresolvable Secret denies only that route with503, rather than aborting the render for the whole fleet. - Basic auth could serve a route with no authentication at all. Four paths reached a backend with no
http-request authrule and no userlist:auth-type: basicwith an absent or emptyauth-secreton the native library, and an unresolvableauth-secreton thehaproxy-ingress,haproxytechandnginx-ingresslibraries. All four now reject at admission and serve503under reconcile. The nginx path checks the Secret before folding intosatisfy: any. - A client could forge the client-certificate identity headers.
auth-tls-cert-headersetsX-SSL-Client-CN,-DNand-Certonly when a certificate is presented, and nothing removed a caller-supplied copy first — so a request over plain HTTP carried the caller's own values to the application. The headers are now stripped before being set, as isapi-key-consumer-header, which had the same shape. - External-auth response headers (
nginx.ingress.kubernetes.io/auth-response-headers,haproxy-ingress.github.io/auth-headers-succeed, the oauth2-proxy desugaring) are stripped from the inbound request before the auth plugin's value is set. Previously aset-headerskipped for a header the auth reply omitted let a client-supplied copy (e.g.X-Auth-Groups: admin) survive to the backend. - API-key authentication and consumer-group authorization fail closed with
503while their referenced Secrets are absent from the watch cache, instead of rejecting an Ingress during Secret propagation races. - Creating a Gateway can no longer move an existing Gateway's per-pod listener port. Same-slot collisions were resolved in key order, so an arriving Gateway could evict the incumbent and take over its port in the same render — silently scoping clients still addressing that port to a different Gateway's routes, across tenants. Slots are handed out oldest-first, so a Gateway's port depends only on its own key and those of older Gateways.
- Gateway API traffic reports the correct owning resource. The owner-identity cascade read a variable scope nothing writes, so the Gateway branch never fired and requests fell through to
txn.resource_id = "gtw/<namespace>"— the key every per-resource feature map (WAF policy, external auth, rate limit, cache, schema validation) is looked up with. - A single mirrored route no longer breaks every gRPC route on the same frontend. A mirror filter emitted a frontend-wide
option http-buffer-request, which deadlocks a gRPC stream until it dies with a408. All sites now emit aContent-Length-gatedhttp-request wait-for-bodyinstead and consult the per-backend override map, so a route'srequest-buffering: offis not defeated by an unrelated mirrored route. A chunked request body is no longer mirrored. - A body-inspecting WAF policy on a route with a gRPC backend is rejected at admission instead of failing at runtime, and detect mode waits only for a body whose length is declared. Coraza buffers a complete body and ships no protobuf processor, so a gRPC stream could never satisfy it: streaming calls timed out with
408while unary calls kept working. An existing route records aWafBodyPolicyOnGRPCRouteWarning Event rather than being taken down. - When two Ingresses claim the same host, path and path type, the route resolves to the older Ingress (by
creationTimestamp, tiebroken by namespace/name) instead of the arbitrary render-order winner, matching ingress-nginx's oldest-wins. An nginx canary Ingress no longer competes for its main's base route, and the losing Ingress gets aRouteConflictWarning Event. - EndpointSlices that do not expose a Service's requested named target port are no longer rendered as HAProxy servers using the Service port, which could send application traffic to unrelated pods matched by a broad Service selector.
- Installing without cert-manager works out of the box: when cert-manager provisioning is enabled (the default) but its API is absent, the chart generates a self-signed
default-ssl-certSecret instead of silently skipping theCertificate— previously every render failed withTLS Secret not foundand the HAProxy pods never became ready. An existing Secret is left untouched (#78). helm upgradefrom 0.1.0 no longer fails withSecret "…-webhook-cert" … cannot be imported into the current release: invalid ownership metadata. The chart's webhook serving certificate Secret is named<release>-webhook-tls, so it no longer contends with the cert-manager-provisioned Secret 0.1.0 left behind — which can be deleted at leisure. This removes the manual pre-step previously documented for it;controller.webhook.secretNamestill overrides the name.- Enabling several vendor annotation libraries at once no longer exceeds Kubernetes' per-object size limit, which previously failed to install with a bare
etcdserver: request is too largeand was worked around by a documented "enable at most one" restriction. - Forcing
spoaHub.enabled=truewith every plugin disabled is rejected at render time instead of installing a broken combination where the sidecar's own config was orphan-deleted out from under it. - The bundled SPOA hub no longer crashes with SIGSEGV during
dlclosewhen a plugin is removed during a config reload; retired plugin instances are still drained and destroyed. - HAProxy's restrictive NetworkPolicy (
haproxy.networkPolicy.allowExternal: false) allowed only port 8404, so Prometheus could reach neither Vector endpoint. Both Vector ports are now allowed alongside 8404. - The default HAProxy bootstrap configuration applies the same configurable
extraContext.hardStopAfterdrain bound as the rendered configuration, so the initial worker cannot remain stuck after the first reload. - Generated configurations no longer emit
haproxy -cwarnings from proxy-mode mismatches or unsupported TLS settings: TCP frontends usetcplog,forwardforis scoped to HTTP frontends, the quiet status endpoint retains a log target with normal logs suppressed, and the AWS-LC-incompatibletune.ssl.default-dh-paramdefault was removed. - The three shipped log-derived cache metrics (
cache_status_total,cache_age_seconds_total,cache_uncacheable_total) were silently dropped fromvalues.yamlwhile the library and docs still promised them, so cache dashboards went empty on upgrade. Restored with the missingorigin_refused_sharingreason, and a build check fails if a shipped entry goes missing again. - Cache bypass paths strip every private Varnish protocol header before the application sees it and preserve downstream
VaryandCache-Control: privateprotections while Varnish is unavailable or the method is not cacheable. - CORS no longer destroys the cache's
Vary. The rule usedset-header, replacing the whole field, so a CDN in front of HAPTIC saw onlyVary: Originand could serve one caller's keyed response to another. It appends now. - The
nginx.ingress.kubernetes.io/enable-corsandhaproxy-ingress.github.io/cors-enableannotations answer the CORS preflight in HAProxy with a synthetic 204 carrying theAccess-Control-*headers, matching ingress-nginx, instead of forwarding it to the backend. - The
nginx.ingress.kubernetes.io/cors-allow-originandhaproxy-ingress.github.io/cors-allow-originannotations accept a comma-separated allow-list with single-level*.subdomain wildcards, match it against the requestOriginand echo the matched origin back (addingVary: Originfor non-wildcard lists). Previously the value was emitted verbatim, so a multi-origin list produced an invalid header. A malformed origin now fails the render. - The
haproxy.org/cors-*annotations mirror the upstream HAProxy Kubernetes Ingress Controller: a non-wildcardcors-allow-originis a regex matched against the request Origin and the response echoes the matched origin, headers are added viahttp-after-response, and thecors-allow-methods/cors-allow-headers/cors-max-agedefaults track upstream. haproxy-ingress.github.io/auth-secretalso parses anauth-key htpasswd Secret (haproxy-ingress's and ingress-nginx's native format), so a Secret migrated from those controllers authenticates. The previous Secret shape still works.haproxy.org/auth-realmnormalizes spaces to dashes instead of failing the render, and its default realm isProtected-Content. Thesanitize_auth_realmtoggle is removed.nginx.ingress.kubernetes.io/proxy-ssl-secretclient certificates deploy under the correct path: the renderedcrtvalue is the bare filename instead of anssl/-prefixed path that doubled tossl/ssl/<name>and made HAProxy reject the config.- The Gateway API template library no longer requires the Ingress library; its shared hostname-to-map-key helper moved into the always-loaded base library, so
controller.templateLibraries.gatewaycan be enabled withingressdisabled. nginx.ingress.kubernetes.io/limit-rate-afteris documented asdifferentrather than supported. It maps to the bandwidth filter'smin-size— the smallest chunk HAProxy forwards, not nginx's "start throttling after N bytes".- A tenant could hijack another tenant's host and path by putting a newline or space in an Ingress or HTTPRoute path. The path was written verbatim into the shared routing map, whose keys are whitespace-delimited, so a control character or space smuggled a second map line pointing a victim's host+path at the attacker's backend — content
haproxy -caccepts. Ingress and Gateway path values are now guarded before they reach the routing map: denied at admission, and warned-and-skipped with anInvalidPathWarning Event under reconcile so one hostile object can't brick the fleet. - A tenant could hijack routing or inject frontend directives by putting a double quote or newline in an HTTPRoute/GRPCRoute match value or URL-rewrite/redirect path. These values land inside double-quoted HAProxy ACL patterns and
http-requestdirectives in the Gateway frontend; a"broke out and appended attacker-controlled ACL terms (always-match routing hijack) and a newline split the line into an arbitrary directive — all valid config thathaproxy -c, the determinism check and the load gate accept. Gateway API does not charset-validateRegularExpression, header, query-param or rewrite values, so they are now guarded before emission: denied at admission, and warned-and-skipped with anInvalidMatch/InvalidURLRewrite/InvalidRedirectWarning Event under reconcile. - A tenant could inject frontend/backend directives through header, CORS, response-body, TLS-option and OAuth values that were emitted with an inadequate guard. Gateway
RequestHeaderModifier/ResponseHeaderModifierand backendRef header values,haproxy-haptic.org/request-set-header/response-set-headervalues, the shared CORScors-allow-methods/-headers/-max-age/-expose-headersvalues, andfixed-response/mock-responsecontent-types and bodies are now emitted inside double quotes with"/$/\escaped and control characters rejected — closing a single-quote break-out ('evil' #commented out the host-match gate, applying a header to all traffic) and an unescaped$that HAProxy expanded as an environment variable, leaking controller env into a response body. Header and CORS values additionally double any%(a log-format string reads%[...]at request time, so a tenant value could otherwise read request-scoped internal state into its own header). Gateway per-listenerspec.tls.options(TLS min/max version, cipher suites) and the OAuthoauth-uri-prefixare guarded before they reach the shared HTTPS bind line and the auth routing maps. All are denied at admission and warned-and-skipped under reconcile. Secret-absent warning comments (api-key-secret,auth-secret,hmac-secret) no longer interpolate the tenant-controlled Secret name into the rendered config. - The
haproxy.org/*(haproxytech) annotation library could inject frontend/backend directives through many values emitted with an inadequate guard.cors-allow-methods/-headers/-max-ageandrequest-set-header/response-set-headervalues are now emitted inside double quotes with"/$/\escaped and%doubled (a header value is a log-format string), closing the old single-quote break-out; thecors-allow-originregex is single-quoted (strong quoting, so its backslashes survive) and rejects a'or control char;path-rewrite, timeouts,load-balance,check-http/check-interval,cookie-persistence(-no-dynamic) andserver-protoreject control characters (and, for bare-token values, whitespace) before emission. Routing values are denied at admission and warned-and-skipped under reconcile; the security-criticalsrc-ip-header,rate-limit-*andauth-realmfail closed. The Secret-absentserver-ca/server-crtwarning comments no longer interpolate the tenant-controlled Secret name into the rendered config. - The
haproxy-ingress.github.ioannotation library passed tenant values into HAProxy config without an adequate guard. A newline or space inserver-alias/server-alias-regexor a regexpath-typepath smuggled a second host/path map entry (cross-tenant route hijack); a newline, space or quote intimeout-*,health-check-uri/backend-check-interval,session-cookie-name/-keywords/-domain,agent-check-*, theheadersname and theauth-realmsplit its directive; a newline inauth-url/auth-signin/auth-method/oauth-uri-prefix/oauth-headersinjected an auth routing-map entry — all valid config thathaproxy -caccepts. Every such value now routes through the shared guard/escape utilities: map keys/values and directive tokens reject whitespace and control characters, theheadersvalue is emitted inside double quotes with"/$/\escaped and%doubled, andauth-realmis escaped inside its quoted literal. Routing values (aliases, paths, timeouts, cookies, health checks) are denied at admission and warned-and-skipped under reconcile; security values (secure-sni/secure-verify-hostname,ssl-ciphers-backend/ssl-cipher-suites-backend,auth-realm) fail closed.
[0.2.0-alpha.1] - 2026-07-05¶
Added¶
- Support for every Gateway API release, adapted at runtime:
watchedResourcesentries accept an orderedapiVersionscandidate list and anoptionalflag,templateSnippets/validationTestsacceptrequires/requiresFields(features whose resources or schema fields aren't served are stripped at config load), and a CRD watch reinitializes the controller when a watched CRD is installed, upgraded, or removed — no Helm operation or pod restart needed. Resolved versions are exposed viaresources.<name>.APIVersion()and/debug/vars/effectiveConfigResolution;controller validateresolves the same way against--schema-dir. A required resource with no served version fails startup with a clear error instead of hanging. - The config's embedded
validationTestsare now enforced at every gate: the admission webhook denies a failingHAProxyTemplateConfigatkubectl apply, and the controller rejects a failing config at startup and on every change — the last-good config keeps serving, and failing test names surface onstatus.validationErrors. Previously only thecontroller validateCLI ran them. - Kubernetes Events on the
HAProxyTemplateConfig: aWarning/ValidationFailedEvent when a config change fails validation and aNormal/ValidatedEvent on recovery, so failures surface inkubectl describeandkubectl get events. - HAProxy 3.4 support: the controller is built, validated, and released for HAProxy 3.4.
- Typed watched resources: chart templates access watched resources through compile-time typed top-level globals (e.g.
gateways,httproutes), type-checked at controller startup against schemas from the live apiserver or--schema-dir. See ADR-0010. spec.k8sResourcesdeclares full Kubernetes resources reconciled via Server-Side Apply, owned by theHAProxyTemplateConfigCR (garbage-collected on uninstall); a partial-ownership mode (haproxy-haptic.org/ownership: partial) lets chart-owned and haptic-owned entries coexist on one resource.spec.validatorsdeclares pluggable validator sidecars consulted by the admission webhook (per-entry socket, file-glob routing, timeout);/healthzreports each socket's reachability.- The admission webhook hot-reloads its TLS certificate from the mounted Secret, so a cert-manager renewal is served within about a minute without a controller restart. The
--webhook-cert-secret-nameflag is replaced by--webhook-cert-dir(envWEBHOOK_CERT_DIR). - Gateway API
RequestMirrorfilters, including multiple and percentage/fraction mirrors, backed by the bundled mirror SPOA plugin. - Ingress
spec.defaultBackendsupport, both rules-less (catch-all) and combined withspec.rules. - HAProxy responses now carry a
Server: hapticheader. - New metrics:
haptic_haproxy_reloads_total(the canonical reload/SLO signal),haptic_dataplane_api_operations_total, runtime fast-path counters (haptic_runtime_fast_path_*), and per-plugin SPOA metrics via the hub's/metricsendpoint. spec.dataplane.configPublishInterval,reloadVerificationTimeout, andsyncTimeoutare now tunable;spec.watchedResources.<name>.debounceIntervaladds a per-resource batching override.- New
spoa-hubcontainer image bundling the SPOE hub plus seven plugin libraries (mirror, coraza, external-auth, fingerprinting, maxmind, otel, sso-auth), Cosign-signed with a CycloneDX SBOM. - External authentication (
auth-url-style annotations) wired end to end via the spoa-hub external-auth plugin.
Changed¶
- Leader-election timing defaults raised from 15s/10s/2s to 30s/20s/5s (
leaseDuration/renewDeadline/retryPeriod): more headroom against apiserver or CPU-starvation stalls, at the cost of slower crash-failover (up to ~35s; voluntary handoffs still release the lease immediately). Tune viaspec.controller.leaderElection(#57). - Config validation now runs asynchronously with latest-wins coalescing: rapid successive
HAProxyTemplateConfigedits validate only the newest change, and an in-flight validation no longer blocks config processing or leaves a new leader without a config (#55). store: on-demandwatched resources keep far less memory resident: the informer cache strips object bodies down to index and identity fields; templates still read the full object, fetched live on access. See ADR-0012.- All log output now shares one structured logfmt format (client-go and other third-party lines included), and routine per-request/per-reconcile success lines moved from INFO to DEBUG — failures, denials, config changes, and leadership transitions stay at INFO. Set
LOG_LEVEL=DEBUGfor the previous verbosity. - Runtime-eligible server changes (pod-IP rotation, port, admin state) now reach the live HAProxy workers in milliseconds via a runtime fast path instead of waiting for the next scheduled deploy; structural changes still take the rate-limited reload path. See ADR-0011. The reconciler-level debounce was removed — batching is per-watcher via
spec.watchedResources.<name>.debounceInterval(default raised from 100ms to 2s;"0"fires on every change). - Content-only changes now apply to the live HAProxy workers via the runtime API without a reload: map files, TLS certificates (HAProxy 3.2+), SSL CA files (HAProxy 3.2+, e.g. frequent trust-bundle rotation), and frontend
maxconn. Adding or removing files and all other structural changes still reload, and a failed runtime apply falls back to a reload, so the result always converges. controller validatenow requires--schema-dir(orHAPTIC_SCHEMA_DIR) for typed-access templates; the embedded Gateway schema fallback was removed. Production (live apiserver) is unaffected.
Fixed¶
- Fixed controller crash-loop and Gateway status-apply failures on clusters running older Gateway API releases (v1.1–v1.5) whose schemas lack fields the bundled chart exercises (#59).
- A replica that loses the leader-election lease now reinitializes and re-enters the election loop instead of running as a permanent follower — on single-replica deployments, a missed lease renewal could previously stall deployments until a pod restart (#57).
- Runtime map updates are now verified by read-back and fall back to a reload when the live map did not converge — a map write acknowledged but lost by HAProxy could previously latch stale routing until an unrelated reload (#48).
- Eliminated intermittent 503s during rolling restarts of single-replica backends: a newly-Ready pod's endpoint change reaches the live workers within
option redispatch's rescue window even when it coincides with an in-flight structural deploy, runtime applies retry across a concurrent reload, and deleting an unreferenced certificate no longer schedules a stray second reload that briefly blacked out the runtime socket (#67). - Reloads are bounded to one per
minDeploymentIntervalunder concurrent churn, and auxiliary-file updates batch into the main config sync's single reload instead of triggering separate DataPlane API auto-reloads. - Resource status no longer goes stale: statuses are written even when a change produces no config diff (a Gateway with no attached routes no longer sticks at
Programmed=Unknown), and sustained event bursts no longer drop status or deployment events. HAProxyCfg.status.deployedToPodsnow reflects reality: a failed deploy keeps the pod's last successfully-deployed checksum plus alastErrorinstead of reading as converged, failed deployments are retried on the next reconcile instead of waiting for the drift timer, and a startup race no longer leaves a deployed pod unreported.- Cross-pod config drift is now detected and repaired: the deployer tracks each pod's actual post-sync state, and a render race no longer leaves a newer render undeployed.
- Credentials-Secret rotation now takes effect immediately, and content updates to watched Secrets/ConfigMaps are no longer dropped.
- gRPC: requests hitting the default backend get a proper trailers-only
grpc-status: 12response instead of a connection-breaking 404, GRPCRoute method matching works on TLS + ALPN h2 and plaintext h2c, and gRPC over h2c multiplexes with HTTP/1.1 on the same bind. - Gateway TLS listeners on non-default ports now bind correctly.
- The renderer fails fast when the config references a map file it didn't register (previously the file was deleted as unreferenced, breaking every subsequent reload), and each render sees one consistent snapshot of the watched-resource stores.
store: on-demandresources no longer trigger a full list or one API fetch per resource on per-key reads, drift cycles, and the/debug/varsendpoints.- Ingress
pathType: Exactnow preserves trailing slashes (/foo/no longer matches/foo). controller validaterenders with the same context as production (capabilities,extraContextpromotion), so a template can no longer pass validation yet render differently in production.- HAProxy 3.3 configs are now validated against the v3.3 DataPlane API schema (was v3.2), and DataPlane API versions newer than the newest bundled client clamp down to it instead of falling back to the oldest v3.0 client.
- Watched-resource admission (e.g. Ingress) now gets the same ~9s internal validation deadline as
HAProxyTemplateConfigadmission (was 5s), so large-config dry-runs aren't prematurely admitted without validation. - Status patches, admission overlays, and owned-resource applies resolve a resource's plural via the cluster RESTMapper, so a CRD with an irregular plural no longer targets a nonexistent GroupVersionResource.
- Hardened component lifecycle: re-acquired leadership no longer stacks orphaned event subscriptions (previously endless "critical drops" log spam), event-loop panics recover instead of silently killing the loop, and the controller no longer schedules a spurious iteration restart right after startup.
- HTTP content stores no longer misclassify an empty
200 OKas304 Not Modified(stale content), SSL-certificate Secret publishing retries on write conflicts, and backends whose only change is on thedefault-serverline now update instead of staying stale. - HAProxy file paths in rendered configs use slash-only semantics regardless of host OS, fixing the controller binary on Windows (no-op on Linux).
- Bundled SPOA hub updated to v0.7.3: config reloads quiesce in-flight SPOE dispatches before retiring the old plugin set, fixing lost mirror/OTLP work around reloads (#47); unhandled SPOE messages surface via a WARN log and
spoa_messages_unhandled_total.
Removed¶
- The dormant webhook-cert-rotation pipeline and its
haptic_webhook_cert_expiry_timestamp_seconds/haptic_webhook_cert_rotations_totalmetrics (it never rotated certs) — replaced by the webhook-cert hot-reload above. namespaceSelectoronwatchedResourcesentries (was never wired up). Scope vialabelSelectoror separate controller instances.- Management of HAProxy
programsections — HAProxy removed the section in 3.3 andclient-nativedropped the model. Rendered configs may still containprogramsections on older HAProxy versions, but the controller no longer parses, diffs, or normalizes them.
Helm chart¶
Added¶
- New
nginx-ingresstemplate library fornginx.ingress.kubernetes.io/*annotation compatibility (disabled by default): routing and redirects (rewrite-target,app-root,server-alias,default-backend,from-to-www-redirect,permanent-redirect/temporal-redirectwith code overrides,ssl-redirect/force-ssl-redirect— 308 by default, tunable via thenginxHttpRedirectCodeextraContext var), session affinity (session-cookie-*), rate limiting and throttling (limit-rps/limit-rpm/limit-whitelist,limit-rate/limit-rate-after), backend TLS (proxy-ssl-*), request/response header rewrites (upstream-vhost,x-forwarded-prefix,connection-proxy-header,proxy-cookie-domain/-path,proxy-redirect-from/-to), retries (proxy-next-upstream/-tries), body-size limits (proxy-body-size), auth (auth-*,auth-secret-type,satisfy), and request mirroring (mirror-target, via the spoa-hub mirror plugin). - haproxy-ingress library additions:
oauth: oauth2_proxy(desugars onto the external-auth machinery),rewrite-target, rate limiting (limit-rps/limit-rpm/limit-whitelist— previously advertised but emitting no config),proxy-body-size,ssl-ciphers-backend/ssl-cipher-suites-backend, agent health checks (agent-check-*),default-backend-redirect/-code,server-alias/server-alias-regex, and raw directive injection (config-frontend/config-global/config-defaults). - haproxytech library additions:
rate-limit-whitelistandssl-redirect-port. - Gateway API support extended: TLSRoute (Terminate and Passthrough), TCPRoute (L4 forwarding),
RequestMirrorfilters (via the spoa-hub mirror plugin), static Gateway addresses (per-Gateway LoadBalancer Service, multi-IP),spec.infrastructurepropagation, ListenerSet routing, GEP-91 frontend client-certificate validation, HTTPRoute/GRPCRoute cookie session persistence (GEP-1619), HTTPRouteretry, per-listener TLS options (GEP-2907), and BackendTLSPolicyvalidation.subjectAltNames. spoaHubvalues block and SPOA hub sidecar (hub v0.7.3 plus seven plugin libraries: mirror, coraza, external-auth, fingerprinting, maxmind, otel, sso-auth), auto-rendered when any plugin is enabled; aspoa-hubtemplate library wires the HAProxy-side SPOE config, and the hub's runtime config reloads without a pod restart.- nginx-ingress and haproxy-ingress libraries wire the external-auth annotation family (
auth-url,auth-signin,auth-method,auth-headers-*, client-mTLSauth-tls-*) to the external-auth plugin, plus the Coraza WAF annotations (/waf,modsecurity-snippet) with per-resource opt-in and adefault-ondispatch mode. controller.validatorsblock and a validator sidecar consulted by the admission webhook, auto-enabled and auto-wired with the spoa-hub sidecar, so admission-time WAF/TOML validation works out of the box.- The
HAProxyTemplateConfigadmission webhook now also runs the config's embeddedvalidationTests, denying a failing config atkubectl apply; itstimeoutSecondsis 10, andfailurePolicy: Ignorestill admits-with-warning when the webhook is slow or unreachable. - The ingress library now emits a
WarningEvent (reasonBackendUnresolved) on each Ingress whose backend Service or named port can't be resolved — visible inkubectl describe ingress— so a permanent Service-name typo is distinguishable from a propagation race; the Event is removed automatically once the Service appears (#66). - Four more default PrometheusRule alerts (individually toggleable):
configRejected,haproxyPodsRejected,noHAProxyPods, andcriticalEventsDropped. controller.templateLibraries.gateway.experimentalChannelgates the validationTests that assert Experimental-channel HTTPRoute fields (sessionPersistence,retry) — since Gateway API v1.6 the Standard and Experimental channels ship an identical CRD set, so the channel can't be auto-detected.- Template libraries can declare default
templatingSettings.extraContextvalues, merged at the lowest precedence so operator overrides still win. - Always-on local
peers localinstancesection so opted-in stick-tables survive reloads: rate-limit counters (haproxy.org/rate-limit-*, nginx-ingresslimit-rps/limit-connections, haproxy-ingresslimit-rps/limit-rpm) now persist across config reloads instead of resetting. - New values:
haproxy.initialConfig(HAProxy bootstrap ConfigMap),controller.config.dataplane.{configPublishInterval,reloadVerificationTimeout,syncTimeout},controller.config.extraContext.hardStopAfter(default10s, emitshard-stop-afterso old workers don't accumulate across reloads), andhaproxy.dataplane.aclFormat(dataplane access-log format override). - The chart-static
haptic-haproxyLoadBalancer Service is now rendered via the controller'sspec.k8sResources(Server-Side Apply with anOwnerReference), folding non-default Gateway listener ports into the same Service. - HAProxy NetworkPolicy opens to all TCP ports when
haproxy.networkPolicy.allowExternal: true, so dynamic Gateway listener ports work; restrictive mode is unchanged. - RBAC: the ClusterRole gains
customresourcedefinitionsread access (schema resolution), cluster-wide Event write verbs (ingress library), and cluster-wide plus namespace-scopedserviceswrite verbs (gateway library, per-Gateway Services); ClusterRole and webhook rules now cover every declared API-version candidate of each watched resource.
Changed¶
- BREAKING: path matching order is now selected by
controller.config.routing.regexMatchOrder(default/last), replacing the removedpath-regex-lasttemplate library. Operators withtemplateLibraries.pathRegexLast.enabled: truemust switch torouting.regexMatchOrder: last. See ADR-0005. - BREAKING:
ingressClass.nameandgatewayClass.namedefault fromhaproxytohaptic. Operators replacing an incumbent controller set them back tohaproxy(or update their manifests toingressClassName: haptic/gatewayClassName: haptic). - BREAKING: pod-spec scheduling, runtime, and metadata fields moved under namespaced
podSpec:blocks on both Deployments. Container-, Deployment-, and chart-wide fields are unchanged. Rename the following keys in custom values files:
| Previous | New |
|---|---|
imagePullSecrets |
controller.podSpec.imagePullSecrets |
podAnnotations |
controller.podSpec.podAnnotations |
podLabels |
controller.podSpec.podLabels |
priorityClassName |
controller.podSpec.priorityClassName |
topologySpreadConstraints |
controller.podSpec.topologySpreadConstraints |
podSecurityContext |
controller.podSpec.podSecurityContext |
nodeSelector |
controller.podSpec.nodeSelector |
tolerations |
controller.podSpec.tolerations |
affinity |
controller.podSpec.affinity |
terminationGracePeriodSeconds |
controller.podSpec.terminationGracePeriodSeconds |
dnsPolicy |
controller.podSpec.dnsPolicy |
dnsConfig |
controller.podSpec.dnsConfig |
hostAliases |
controller.podSpec.hostAliases |
runtimeClassName |
controller.podSpec.runtimeClassName |
haproxy.priorityClassName |
haproxy.podSpec.priorityClassName |
haproxy.topologySpreadConstraints |
haproxy.podSpec.topologySpreadConstraints |
haproxy.nodeSelector |
haproxy.podSpec.nodeSelector |
haproxy.tolerations |
haproxy.podSpec.tolerations |
haproxy.affinity |
haproxy.podSpec.affinity |
haproxy.dnsPolicy |
haproxy.podSpec.dnsPolicy |
haproxy.dnsConfig |
haproxy.podSpec.dnsConfig |
haproxy.hostAliases |
haproxy.podSpec.hostAliases |
haproxy.runtimeClassName |
haproxy.podSpec.runtimeClassName |
haproxy.podAnnotations |
haproxy.podSpec.podAnnotations |
haproxy.shareProcessNamespace |
haproxy.podSpec.shareProcessNamespace |
haproxy.terminationGracePeriodSeconds |
haproxy.podSpec.terminationGracePeriodSeconds |
haproxy.podSecurityContext |
haproxy.podSpec.podSecurityContext |
- The gateway library now supports every Gateway API release, resolved at runtime: each Gateway kind is an optional watched resource with an ordered
apiVersionscandidate list (core kinds v1/v1beta1, GRPCRoute v1/v1alpha2, TLSRoute v1/v1alpha3/v1alpha2, TCPRoute v1/v1alpha2, ReferenceGrant v1/v1beta1, BackendTLSPolicy v1/v1alpha3). Installing or upgrading Gateway API CRDs activates support at runtime — nohelm upgradeneeded; features whose fields don't exist in an older release's schemas stay inactive there. The Helm-render-time.Capabilitiesgate is removed. - The GatewayClass object is now created at runtime by the controller (Server-Side Apply, owned by the
HAProxyTemplateConfig) instead of by a Helm template, so it exists exactly when thegatewayclassesCRD is served. Upgrade note:helm upgraderemoves the previously Helm-owned GatewayClass and the controller recreates it within one reconcile; existing Gateways are unaffected. - Per-route annotation policies are now applied by shared frontend rules reading per-backend/per-host map files instead of inline per-backend/per-ingress rules:
proxy-body-size, upstream header overrides (set-host,upstream-vhost,x-forwarded-prefix,connection-proxy-header), literalrewrite-target,ssl-redirect/force-ssl-redirect, host redirects (redirect-to,request-redirect,permanent-redirect/temporal-redirect), HSTS,app-root,auth-tls-error-page,from-to-www-redirect,default-backend-redirect, andssl-redirect-port. Adding or changing these is now a map-only, reload-free update; behavior, status codes, and config-injection guards are unchanged (app-root,auth-tls-error-page, andhsts-max-agegain previously missing injection guards). - The deployed DataPlane API now addresses storage (maps/certs/general files) by relative paths resolved against the HAProxy base dir — the same identifiers HAProxy uses — so map content changes apply via the runtime API without a reload. The
dataplane.mapsDir/sslCertsDir/generalStorageDirvalues are unchanged. - The auto-generated DataPlane API password (
credentials.dataplane.passwordleft empty) is now a random 32-char value instead of a deterministic hash of release name and namespace. Existing installs are unaffected (the password is preserved from the existing Secret onhelm upgrade). GitOps note: under ArgoCD/Flux (no cluster lookup at render time), an empty password regenerates on every sync — setcredentials.dataplane.passwordexplicitly for those setups. - The validating admission webhook now provisions its own self-signed TLS certificate by default (
webhook.certManager.enableddefaults tofalse; validity viawebhook.selfSigned.certValidityDays, default 10y, reused across upgrades), so admission validation works out of the box without cert-manager. cert-manager stays opt-in and is recommended for production (automatic rotation); the self-signed cert is not auto-rotated. Upgrade note: a release that used the previous cert-manager default holds a webhook Secret without Helm ownership metadata; before upgrading, either keep cert-manager (--set webhook.certManager.enabled=true) or delete the old Secret (kubectl delete secret <release>-webhook-cert -n <namespace>) so the chart can recreate it. - Default
haproxyVersionis now3.4(was3.2); the controller and HAProxy pod images default to the HAProxy 3.4 series. OverridehaproxyVersionto stay on an older series. - Template libraries (
gateway,nginx-ingress,haproxy-ingress,haproxytech) are now packaged as conditional subcharts: a release stores only the source of the libraries it enables, so an install with all libraries enabled no longer exceeds the Kubernetes 1 MiB release-Secret limit. No values changes (the samecontroller.templateLibraries.<x>.enabledflags), and the renderedHAProxyTemplateConfigis byte-identical. - The controller's startup probe is now enabled by default (30 × 10s budget): startup runs the config's embedded
validationTests, so time-to-first-healthy can exceed the liveness budget on slow or contended nodes — previously the liveness probe could kill the controller mid-initialization. haproxy.ports.http/httpsdefaults shift from8080/8443to80/443sodst_portequals the Gateway listener port; explicit overrides are kept.- Dataplane
minDeploymentIntervaldefault raised to5s(was2s), throttling reload-inducing structural deploys; endpoint changes still apply instantly via the controller's runtime fast path. EndpointSlice watches keepdebounceInterval: "0"for instant rolling-restart reaction. - HAProxy
defaultstimeout connectlowered from5000to100(100 ms). Backends are pod IPs over the CNI; 100 ms fails fast on a SYN to a just-terminated pod sooption redispatchretries. Operators on slow networks restore5000viaextraContext.timeout_connect. - gateway library: the cluster-wide
configmapswatch now defaults tostore: on-demand(it is only read by name for BackendTLSPolicy CA bundles), keeping references instead of every ConfigMap body resident. extraDeploynow accepts both list and dict formats.haproxy.org/pod-maxconnquantizes the pod count to the next power of 2 to avoid reload cascades on scaling.- Compound resource names (derived from
haptic.fullname) now truncate to the 63-char label limit; releases with very long names will see affected resources renamed on upgrade (runhelm diff upgradefirst). - Removed the "TLS Certificate Expiry" Grafana dashboard panel — the controller no longer emits
haptic_webhook_cert_expiry_timestamp_seconds(the webhook cert is hot-reloaded; see the controller section above).
Security¶
- Annotation values interpolated into the HAProxy config — CIDR lists (allow/deny lists, rate-limit and
satisfywhitelists) and single-value annotations (upstream header overrides,proxy-ssl-*,proxy-cookie-domain/-path, the auth realm) — are now validated against strict charsets, closing a config-injection vector where a crafted annotation containing a newline could inject arbitrary HAProxy directives into the rendered config.
Fixed¶
- Fixed controller crash-loop and Gateway status-patch failures on clusters running older Gateway API releases (v1.1–v1.5) or Standard-channel installs without the TCPRoute CRD (#59).
- Named Service-port references (
port.name) in Ingress, HTTPRoute, GRPCRoute, and SSL-passthrough backends now resolve to the correct numeric port (previously silent 503s or empty backends). When the referenced Service isn't in the controller's store yet, the backend renders degraded (placeholder slots, 503 for that route) and converges once the Service appears, instead of failing the whole render and denying the Ingress at admission (#50); a Service that exists but lacks the named port still fails loudly. defaultsnow setsoption redispatchandbase.yamlfilters out not-ready/terminating endpoints, eliminating the single-replica rolling-restart 503 windows.- Reserved-slot server addresses are now config-driven across reloads (removed the HAProxy server-state-file machinery); placeholders use the unroutable
192.0.2.1:1sentinel. See ADR-0011. haproxy.dataplane.validateConfig: falsenow actually skips the dataplane'shaproxy -c(the flag was misplaced), cutting raw-config push time ~130ms → ~18ms.- haproxytech:
haproxy.org/check,check-interval, andscale-server-slotsare now honored instead of silently ignored, and IP access control reads the canonicalhaproxy.org/allow-list/deny-listannotations (deprecatedwhitelist/blacklisthonored as fallback) — the real annotations previously emitted no ACL. - haproxy-ingress:
maxconn-server,maxqueue-server,initial-weight,backend-check-interval, andhealth-check-port/-fall-count/-rise-countnow render theirdefault-serverkeywords (previously validated but never emitted), the deprecatedwhitelist-source-rangealias is honored, and thepath-typeannotation now actually routes. - Gateway TLS: an unspecified
tls.modedefaults toTerminateper spec (the listener was previously skipped silently), and a BackendTLSPolicy with no resolvable CA returns 503 instead of downgrading to plaintext. - The bundled
validationTestsnow pass in any release namespace (the shared SSL fixture no longer hardcodeshaproxy-haptic/default). - The chart fails fast at install with actionable guidance when
webhook.certManager.enabled=truebut the cert-manager CRDs are absent, instead of leaving the controller pod stuck inContainerCreating. - Basic-auth snippets no longer fail when the referenced auth Secret is briefly absent from the render snapshot.
- PrometheusRule default alerts
HAProxyControllerHighQueueDepth/HAProxyControllerNoLeadernow reference metrics the controller actually emits (the old expressions never fired). networkPolicy.ingress.webhook.fromandnetworkPolicy.egress.kubernetesApidefaults switched toipBlock0.0.0.0/0+::/0, so a host-network apiserver can reach the webhook on clusters enforcing NetworkPolicy (previously silent admission failures).- The validating-webhook configuration now sources
watchedResourcesfrom the merged libraries, so library-declared resources are actually validated.
[0.1.0] - 2026-03-09¶
Added¶
- Template-driven HAProxy configuration: Generate HAProxy configs using Scriggo templates (Go-based, Jinja2-like syntax) with full access to Kubernetes resources, built-in utility functions, and modular template snippets
- Embedded validation tests: Declarative test fixtures and assertions for testing HAProxy configurations within template libraries; run via
haptic-controller validate --test <name> - Dry-run validation webhook: Admission webhook that intercepts CREATE/UPDATE on opted-in watched resources (Ingress, HTTPRoute, GRPCRoute by default), renders templates with the proposed object overlaid on the live store, and rejects the write if rendering or HAProxy validation fails
- Multi-architecture container images:
linux/amd64,linux/arm64,linux/arm/v7 - HAProxy version support: 3.0, 3.1, 3.2, 3.3 — version-specific images tagged accordingly
- Supply chain security: Container images, binaries, and Helm charts signed with Cosign (keyless OIDC); SBOM attestations in SPDX format
- Prometheus metrics: Reconciliation timing, template rendering duration, validation results, and Kubernetes API latencies
- Leader election for high availability: Multiple controller replicas with automatic leader election; hot-standby replicas continue watching and validating; configurable failover timing
- Stall detection: Components detect when blocked and report unhealthy via
/healthz, enabling automatic pod restart via Kubernetes liveness probes - Configurable deployment timeout:
deploymentTimeoutin dataplane config (default: 30s) to recover from stuck deployments - Server slot preservation: Preserve HAProxy server slots during rolling deployments to enable zero-reload runtime API updates via
currentConfigtemplate context - HAProxy Ingress annotation compatibility: 56
haproxy-ingress.github.io/*annotations via the haproxy-ingress template library - Dataplane API concurrency limiting:
maxParallelconfig option to limit concurrent API operations, preventing timeouts for large configurations - CRD content compression: HAProxyCfg content compressed with zstd when exceeding
configPublishing.compressionThreshold(default 1 MiB), reducing etcd storage - HAProxyGeneralFile CRD: Publish general files (error pages, etc.) as Kubernetes custom resources with compression support
- HAProxyCRTListFile CRD: Publish crt-list files as Kubernetes custom resources with compression support
semver_gtetemplate filter: Version comparison for gating features on HAProxy version (e.g.,semver_gte(haproxyVersion, "3.3"))- Template-driven status patches: Templates can register status patches for any Kubernetes resource via
statusPatch()function, with outcome-keyed variants (rendered,deployed,renderFailed,deployFailed) applied automatically based on pipeline phase - Backend diff field diagnostics: Reconciliation log now includes which BackendBase fields caused backend updates, aiding diagnosis of false diffs from parser round-trip asymmetries
- Status patch helper functions:
condition(),transitionTime(), andtoJSON()template functions for building Kubernetes status conditions with stable transition timestamps
Changed¶
- Reconciliation triggering: Leading-edge triggering with a 5s refractory period; no latency for isolated changes, bursts during that window are batched into a single reconciliation
- Parallel Dataplane API operations: Operations execute in parallel within each priority group, reducing sync time for large configurations
- Balance directive:
balance roundrobinmoved todefaultssection to prevent silent behavior change when upgrading to HAProxy 3.3 (which changed the default balance algorithm fromroundrobintorandom) - Go runtime 1.26.1: Green Tea GC replaces manual GOGC tuning
Helm chart¶
Added¶
- Initial Helm chart deploying the controller and HAProxy pods (2 replicas by default)
- Separate controller Service (ClusterIP for operational endpoints) and HAProxy Service (configurable LoadBalancer/ClusterIP)
- Default NetworkPolicy for HAProxy instances
- Leader election support with configurable replica count
- Default SSL certificate configuration via
controller.defaultSSLCertificate - Modular template library system with composable libraries merged at Helm render time (enable/disable via
controller.templateLibraries.<name>.enabled):base.yaml: Core HAProxy template structure with extension pointsingress.yaml: Kubernetes Ingress support (path types: Exact, Prefix, ImplementationSpecific; TLS termination; default backend)gateway.yaml: Gateway API support — HTTPRoute and GRPCRoute are watched and routed; traffic splitting, request/response header modification, URL rewrites, and Gateway/Route status patches are emitted. TLS/TCP/UDP listeners are reflected in each Gateway'ssupportedKindsstatus but TLSRoute/TCPRoute/UDPRoute resources are not watched or routedhaproxytech.yaml:haproxy.org/*annotation compatibility (backend config snippets, SSL passthrough, CORS, basic auth)ssl.yaml: TLS/SSL featureshaproxy-ingress.yaml: 56haproxy-ingress.github.io/*annotation compatibility (enabled by default)
- Gateway API status reporting: Gateway conditions (Accepted, Programmed), listener status, HTTPRoute/GRPCRoute parent status with Accepted and ResolvedRefs conditions
- Ingress status reporting: LoadBalancer addresses propagated to Ingress
.status.loadBalancer - HAProxy built-in Prometheus exporter enabled by default on the status frontend (
/metricson port 8404) - Grafana dashboard annotations for leader transitions and controller pod starts
- Auto-generated Dataplane API credentials stored in a Secret (deterministic 32-char SHA256 of release-name + namespace; preserved across upgrades from the existing Secret)
haproxy.sysctlsfor setting kernel parameters on HAProxy pods via pod-level securityContexthaproxy.podAnnotationsfor custom pod annotations on HAProxy pods (supports Helm template expressions)haproxy.shareProcessNamespaceto enable process namespace sharing between containers (required for signal-based sidecar reload, e.g., SPIFFE/SPIRE mTLS agents)haproxy.shmStats.enabledto persist stats counters across HAProxy reloads via shared memory (requires HAProxy 3.3+); automatically provisions/dev/shmemptyDir volume with auto-calculated sizehaproxy.nbthreadto control HAProxy thread count (auto-calculated from CPU requests by default)haproxy.dataplane.validateConfigto control server-side config validationhaproxy.dataplane.debugSocketPathto enable Unix socket for runtime profiling of the Dataplane API sidecarcontroller.config.dataplane.maxParallelto limit concurrent Dataplane API operationscontroller.statusPatches.enabledto disable status patch writes during migration from another ingress controllerextraDeployfor deploying arbitrary Kubernetes resources alongside the chart (supports Helm templating)extraEnv,haproxy.extraEnv,haproxy.dataplane.extraEnvfor custom environment variables on all containersglobal-settings-*,defaults-settings-*, andfrontend-extra-*extension points for customizing HAProxy global/defaults sections and early frontend directives via template snippetsstatus-patches-*andstatus-extra-*extension points for custom status and Prometheus endpoint configurationtemplatepost-processor type for declarative output transformations inpostProcessingguiddirectives on all frontends, backends, and servers for stable object identification
Changed¶
- Dataplane API credentials consolidated into
credentials.dataplanesection; auto-generated if not provided - Basic auth userlists are named
auth_<secretNs>_<secretName>and deduplicated per Secret; each Ingress references its userlist viahttp_auth(). Differs from the official HAProxy Ingress Controller's per-Ingress{namespace}-{ingressName}naming so multiple Ingresses sharing the same Secret produce a single userlist (significant speedup for bcrypt hashes) - Production-ready default resource requests and limits: controller (100m CPU / 512Mi memory), HAProxy (250m CPU / 1Gi memory), dataplane sidecar (50m CPU / 256Mi memory)
sidecars,extraVolumes,extraVolumeMountsand theirhaproxy.*counterparts support Helm template expressions
Removed¶
image.appendHaproxyVersionvalue (HAProxy version suffix is now always included in controller image tag)haproxy.dataplane.credentialssection (usecredentials.dataplaneinstead)