For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0065-zammad-on-the-argo-dev-box.md.

0065 — Zammad is an optional Argo CD service of the dev box

Status: Accepted. Date: 22 October 2026. Supersedes how decision 0064 installs Zammad; its configuration-by-name stays.

Support (Beto) needs a Zammad to be tested against. Decision 0064 installed one per dev box with make legacy-controller-zammad, a command of the older controller loop, which main is retiring (decision 0045) and which no longer provisions Beto's broker there. The Argo dev box (decisions 0040, 0041, 0047, 0048) deployed Beto and Support but no Zammad.

Decision

Zammad is the dev box's first optional Argo CD service, selected per target with make deploy PROFILES=zammad. gitops/manifests/zammad holds a wrapper chart around the vendored upstream chart (19.1.3, Zammad 7.2.1), pinned by SHA-256 and its images by digest, installed by the component gitops/apps/components/zammad in the bootstrap-owned zammad namespace after the platform.

  • Shared PostgreSQL, private Redis. Zammad's role and database are on the shared PostgreSQL (decision 0048): pharos-infra creates them and publishes zammad-postgresql. A record written before Zammad existed gains the new password once rather than being refused.
  • Elasticsearch as one plain StatefulSet, not the ECK operator, which needs cluster-scoped resources the services project may not create.
  • Baseline Pod Security. Upstream's privileged root chmod of the temporary directory is replaced by an init step that runs as Zammad's own user and gives Rails a private TMPDIR.
  • Set up after every sync, idempotently. A PostSync hook runs the desk's own tools from chart/desk/: the first start from generated logins, the desk's configuration (decision 0064, unchanged), Beto's integration, staff group access and the dev box AAI as sign-in provider.
  • Zammad adopts Beto's token. The pharos namespace's credential Job already generates Beto's ZAMMAD_TOKEN and ZAMMAD_WEBHOOK_SECRET; it publishes a copy of only those two to the zammad namespace. Zammad always generates a token's value, so the hook creates the integration user's token (ticket.agent, 100 days, decision 0063) and sets Beto's value through Rails, on standard input, then checks it signs in. Nothing writes into the pharos namespace and Beto never restarts. The alternative, Zammad issuing a token and patching Beto's Secret and Deployment, needed write access from the zammad namespace into the platform's.
  • Reached through the edge, signed in through the AAI. The catalogue routes https://zammad.localtest.me. The AAI gains an opt-in public client type for a service that keeps no secret: pharos-zammad uses PKCE and gets no offline access or token exchange, as GRNET registers the team's Zammad.
  • The legacy commands are retired with devops/zammad. Support reset is not needed: Zammad's and Support's databases are on the same shared PostgreSQL, and make devbox-reset empties both together.

What the first installation showed

dev-iasonas was converted from the legacy layout to the Argo one for this, and the installation exposed defects that unit tests and renders could not:

  • the route chart's own profile list lacked zammad, and its NetworkPolicy needed Zammad's pod label; the GitOps checks now require every profile list to agree and render the route chart with every profile active;
  • K3s installed before decision 0045 had no registry mirror for in-cluster builds; the k3s role now keeps it on every installation it owns;
  • Support's retry job had no egress to the shared PostgreSQL;
  • the bootstrap could not read its own mounted Secrets (now fsGroup), and Rails refused the shared temporary directory (now its own TMPDIR);
  • a delivery that ended a child's operation for an earlier revision left Argo nothing to do when the new revision changed no resource, so PostSync hooks never ran at that revision; the delivery now starts that one sync.

On 22 October 2026, with those fixed, dev-iasonas linked the local operator to a Zammad customer, filed the operator's ticket in Zammad, imported a reply the test agent wrote there through the webhook, and closed it from Pharos with Zammad closing it too. Support's contract test passed its create, find, claim, token permission and route-table checks; its two ticket-update checks failed only on the adaptor's 10-second timeout. Zammad completed those updates after about 16 seconds of database time: on the dev box the shared PostgreSQL's synchronous writes to single-node Ceph took up to seconds each (pg_test_fsync: about 9 s per fdatasync, 22 ms per fsync). That storage latency affects every service on the shared PostgreSQL and is left to its own investigation.

Consequences

  • A dev box tests Support end to end with one command, and Zammad lives with the platform it serves rather than beside it.
  • On a dev box Beto's token value changes only when its credential is rotated (gitops/manifests/zammad/README.md), not every 90 days as decision 0063 asks of a shared Zammad; its 100-day expiry is renewed by the hook.
  • Zammad adds about 4 GiB of memory to a dev box that selects it.
  • The bootstrap relies on Zammad storing tokens in plain text (7.2.1 does); the contract test and the hook's own sign-in check fail if that changes.
Page source

Source snapshot · Revision f928834c8.