ADR-0014: One HAProxyTemplateConfig per template library, merged by the controller¶
Status¶
Accepted. Supersedes the cross-library merge step of ADR-0002 (the decentralized
_helm_load loader itself is unchanged) and builds on ADR-0008 (split-library
directories, whose fragment merge is also unchanged).
Context¶
The chart rendered every enabled template library into one
HAProxyTemplateConfig. With nginxIngress enabled that object measured
1,564,098 bytes of stored JSON against etcd's ~1,572,864-byte per-object
limit: 99.4%, with 8,766 bytes of headroom. Every chart change that added
template content risked pushing an install past a ceiling whose failure mode is
etcdserver: request is too large on helm install, with no workaround.
Three things made this hard to see coming:
- No gate. The only size check in the repo (
scripts/check-chart-release-size.py) measures the Helm release Secret. Nothing measured the object. - The ceiling was managed by prose.
tests/e2e/main_test.go,.gitlab-ci.ymlandscripts/dev-env-assets/e2e-values.yamleach carried a "don't enable all three vendor libraries at once" note — a documented product restriction standing in for a structural fix. - The obvious levers were exhausted. Two byte-shaving passes already run at
merge time:
haptic.filterTestsstrips every test and assertiondescription, andhaptic.stripSnippetCommentsstrips Scriggo comments (RULE #2 in the rootCLAUDE.mdrecords that 120 KB of source comments cost the object 665 bytes). What remains is template code and validationTest fixtures, both load-bearing.
Measured contribution per library (compact stored JSON, delta against a core-only render) showed the mass is spread across libraries rather than concentrated:
| library | bytes | % of limit |
|---|---|---|
| core (base + ssl + ingress + ingress-annotations-compat) | 302,789 | 19% |
| gateway | 613,458 | 39% |
| haptic-annotations | 381,062 | 24% |
| nginx-ingress | 261,051 | 17% |
| haproxy-ingress | 218,518 | 14% |
| haproxytech | 67,051 | 4% |
| spoa-hub | 56,168 | 4% |
Decision¶
The chart renders one HAProxyTemplateConfig per enabled template library, plus
one for the operator's own config. The controller merges them at startup, in the
order given by CRD_NAME, later wins.
The worst case becomes the largest single library rather than their sum. With every bundled library enabled the largest object is the gateway config at ~35% of the limit, and the number of enabled libraries stops mattering.
The merge moved out of Helm entirely¶
Helm keeps all per-library work — _helm_load enable/inject/unset,
haptic.filterTests, description and comment stripping, fragment merging within a
split library. Only the final cross-library mustMergeOverwrite moved.
Leaving a merge on both sides would mean two implementations that drift, with
scripts/test-templates.sh validating one and production running the other. So
conversion.MergeSpecs merges the unstructured specs with
mergo.MergeWithOverwrite — literally the call sprig's mustMergeOverwrite
makes, against the same vendored mergo, starting from the same empty accumulator.
The semantics match by construction rather than by careful reimplementation.
Two details are explicit rather than emergent:
migrationCoverageconcatenates. It is a list of per-source declarations, one per contributing library; an overwrite would keep only the last and makemigrate-checksilently under-report. The chart's loader had the same special case.- Underscore-prefixed top-level keys are stripped.
_helm_loadandssl.yaml's_test_tls_*YAML-anchor scratch values are chart-time-only. They used to be dropped implicitly, because the emitter forwarded an explicit allow-list of keys out of the merged accumulator.
Equivalence was verified before the Helm merge was deleted: the merged spec is
byte-identical to the pre-split output for the default, +nginx-ingress and
all-libraries profiles, and all 612 bundled validationTests pass through the new
path.
Order is the argument list¶
Merge order comes from CRD_NAME, generated by the chart from the same ordered
source list the loader iterates. Not resourceVersion, not creationTimestamp, not
name sorting — object names carry no ordering authority, so inserting a library
never renames an existing object.
The merged set is the unit of completeness¶
Four spec fields were +kubebuilder:validation:Required. A library object
legitimately carries nothing but templateSnippets and validationTests, so the
apiserver would have rejected it. They are now optional per object and enforced
after the merge by config.ValidateStructure, which already checked three of
them; the fourth, credentialsSecretRef, the controller never reads
(conversion/converter.go excludes it deliberately).
The consequence is real and accepted: a hand-written single config missing
podSelector now fails at the controller rather than at kubectl apply. The
admission webhook still catches it at apply time wherever it is enabled.
A single config is the degenerate case, not a compatibility mode¶
--crd-name with one value behaves exactly as it did. There is no mode switch
and no dual path.
Consequences¶
Torn reads need no new machinery. A helm upgrade writes N objects
non-atomically. The existing 2 s reinit debounce plus coalesceToLatestParsed
collapse the burst; if a mixed old/new set still slips through, runtime is
fail-open (configloader keeps the previous config serving) and startup is
fail-closed with a 5 s retry. Both self-heal on the next event.
The published version is a composite. The merged object carries the last
source's metadata, so its resourceVersion alone would not change when only a
library changed — and the redundant-reinit guard compares versions for equality.
conversion.CompositeVersion names every member.
Status writes get cheaper. Every status write rewrites the whole object. It
used to rewrite ~1.3 MB; it now rewrites the small operator config, which
MergeSpecs designates by taking metadata from the last source.
Admission judges the merged result. Validating one object of a merged set in
isolation would deny every apply — a library object has no podSelector, an
operator object is missing every library it overrides. The webhook substitutes
the prospective object into the set and validates the merge. An object the
controller does not merge is still validated standalone, exactly as before:
self-contained is the right frame for a config this controller does not assemble,
and skipping it would silently weaken the gate for hand-written configs.
Snippet collisions are now visible. Two libraries defining the same snippet
name used to resolve to the later one silently, and because the merge is deep,
could leave a stale requires: list from the earlier definition. The controller
logs one line per override. It is not an error: an operator overriding a bundled
snippet is the documented escape hatch.
Reading the whole input needs a command. No single object shows the picture
any more. haptic-controller config view --input prints the merged input;
validate --dump-merged does the same offline.
More objects, more watches. One SingleWatcher per config, so a default
install holds 9 instead of 1. Consolidating them behind one label-selected
informer is the upgrade path if that ever matters.
Alternatives considered¶
Drop the bundled validationTests from the runtime object (−38%, and it would
have relieved the release Secret too). Rejected: they are the startup load gate,
which exists precisely because "a restart or upgrade can't quietly serve a
HAProxyTemplateConfig that fails its own tests — the live scatter-gather gate only
blocks a change on an already-running controller, which a fresh pod bypasses"
(pkg/controller/config.go). They are parameterised by the operator's own values
via _helm_skip_test predicates and _global.extraContext, so CI cannot stand in
for them, and unbundling them means nobody runs them.
Compress templateSnippets into the object. pkg/compression (zstd+base64)
already exists and is used for the output CRDs. Rejected: it would make the input
config unreadable and un-kubectl edit-able, destroying the property that the
config is a legible API.
Resolve libraries from the image instead of the cluster. The whole chart is
already baked in at /usr/share/haptic/chart and cmd/controller/chartrender.go
renders it in-process for migrate-check. Rejected for now: the libraries are
parameterised by Helm values, so run would have to perform an in-process Helm
render at startup, and the config would stop being visible in the cluster.
What this does not fix¶
The Helm release Secret has its own ceiling, and splitting does not move it:
the payload is dominated by release.manifest, and N objects carry the same total
bytes. make chart-size-check reports 865,056 / 1,048,576 (82.5%) against a
950,000 gate. Measured marginal cost is ~0.18 payload bytes per byte of template
YAML, so that gate has roughly 470 KB of library content left — against the 8,766
bytes the object ceiling had. When it does bind, the known mechanism is the
image-resolved libraries above.
References¶
pkg/controller/conversion/merge.go— the merge, andCompositeVersioncharts/haptic/templates/_libraries.tpl—haptic.prepareLibraries,haptic.libraryFiles(the order),haptic.libraryConfigNames(theCRD_NAMElist)scripts/check-cr-size.py/make cr-size-check— the gate that was missing- ADR-0002 — decentralized
_helm_loadloader (retained) - ADR-0008 — split-library directory convention (retained)