# 0065 — Zammad is an optional Argo CD service of the dev box

Status: Accepted. Date: 22 October 2026. Supersedes how decision 0064 installs
Zammad; its configuration-by-name stays.

Support (Beto) needs a Zammad to be tested against. Decision 0064 installed one
per dev box with `make legacy-controller-zammad`, a command of the older
controller loop, which main is retiring (decision 0045) and which no longer
provisions Beto's broker there. The Argo dev box (decisions 0040, 0041, 0047,
0048) deployed Beto and Support but no Zammad.

## Decision

**Zammad is the dev box's first optional Argo CD service, selected per target with
`make deploy PROFILES=zammad`.** `gitops/manifests/zammad` holds a wrapper chart
around the vendored upstream chart (19.1.3, Zammad 7.2.1), pinned by SHA-256 and
its images by digest, installed by the component `gitops/apps/components/zammad`
in the bootstrap-owned `zammad` namespace after the platform.

- **Shared PostgreSQL, private Redis.** Zammad's role and database are on the
  shared PostgreSQL (decision 0048): pharos-infra creates them and publishes
  `zammad-postgresql`. A record written before Zammad existed gains the new
  password once rather than being refused.
- **Elasticsearch as one plain StatefulSet**, not the ECK operator, which needs
  cluster-scoped resources the services project may not create.
- **Baseline Pod Security.** Upstream's privileged root `chmod` of the temporary
  directory is replaced by an init step that runs as Zammad's own user and gives
  Rails a private `TMPDIR`.
- **Set up after every sync, idempotently.** A PostSync hook runs the desk's own
  tools from `chart/desk/`: the first start from generated logins, the desk's
  configuration (decision 0064, unchanged), Beto's integration, staff group access
  and the dev box AAI as sign-in provider.
- **Zammad adopts Beto's token.** The pharos namespace's credential Job already
  generates Beto's `ZAMMAD_TOKEN` and `ZAMMAD_WEBHOOK_SECRET`; it publishes a copy
  of only those two to the `zammad` namespace. Zammad always generates a token's
  value, so the hook creates the integration user's token (`ticket.agent`, 100
  days, decision 0063) and sets Beto's value through Rails, on standard input, then
  checks it signs in. Nothing writes into the pharos namespace and Beto never
  restarts. The alternative, Zammad issuing a token and patching Beto's Secret and
  Deployment, needed write access from the `zammad` namespace into the platform's.
- **Reached through the edge, signed in through the AAI.** The catalogue routes
  `https://zammad.localtest.me`. The AAI gains an opt-in public client type for a
  service that keeps no secret: `pharos-zammad` uses PKCE and gets no offline
  access or token exchange, as GRNET registers the team's Zammad.
- **The legacy commands are retired** with `devops/zammad`. Support reset is not
  needed: Zammad's and Support's databases are on the same shared PostgreSQL, and
  `make devbox-reset` empties both together.

## What the first installation showed

dev-iasonas was converted from the legacy layout to the Argo one for this, and the
installation exposed defects that unit tests and renders could not:

- the route chart's own profile list lacked `zammad`, and its NetworkPolicy needed
  Zammad's pod label; the GitOps checks now require every profile list to agree and
  render the route chart with every profile active;
- K3s installed before decision 0045 had no registry mirror for in-cluster builds;
  the k3s role now keeps it on every installation it owns;
- Support's retry job had no egress to the shared PostgreSQL;
- the bootstrap could not read its own mounted Secrets (now `fsGroup`), and Rails
  refused the shared temporary directory (now its own `TMPDIR`);
- a delivery that ended a child's operation for an earlier revision left Argo
  nothing to do when the new revision changed no resource, so PostSync hooks never
  ran at that revision; the delivery now starts that one sync.

On 22 October 2026, with those fixed, dev-iasonas linked the local operator to a
Zammad customer, filed the operator's ticket in Zammad, imported a reply the test
agent wrote there through the webhook, and closed it from Pharos with Zammad
closing it too. Support's contract test passed its create, find, claim, token
permission and route-table checks; its two ticket-update checks failed only on
the adaptor's 10-second timeout. Zammad completed those updates after about 16
seconds of database time: on the dev box the shared PostgreSQL's synchronous
writes to single-node Ceph took up to seconds each (`pg_test_fsync`: about 9 s per
`fdatasync`, 22 ms per `fsync`). That storage latency affects every service on the
shared PostgreSQL and is left to its own investigation.

## Consequences

- A dev box tests Support end to end with one command, and Zammad lives with the
  platform it serves rather than beside it.
- On a dev box Beto's token value changes only when its credential is rotated
  (`gitops/manifests/zammad/README.md`), not every 90 days as decision 0063 asks
  of a shared Zammad; its 100-day expiry is renewed by the hook.
- Zammad adds about 4 GiB of memory to a dev box that selects it.
- The bootstrap relies on Zammad storing tokens in plain text (7.2.1 does); the
  contract test and the hook's own sign-in check fail if that changes.