# 0064 — A dev box's Zammad is seeded from a configuration, not restored from the team's Status: Accepted. Date: 14 October 2026. Installation superseded by decision 0065 (Zammad is an optional Argo CD service; the legacy commands below are retired); the configuration by name stands. Support (Beto) is the only context that talks to a third-party application, so testing it needs a Zammad. Only one dev box had one. It was installed by hand (Helm, by one person, in October 2026) and then had the team's production database restored into it, so every other dev box could run the platform but not test Support at all, and no command existed to change that. ## The problem with the way it got there The dev-box Zammad's configuration is the support desk's real one: 12 groups, 8 states, 4 priorities, 11 SLAs, 12 triggers, 7 core workflows, 12 overviews, 13 administrator-editable ticket fields, business hours, 17 report profiles, templates and text modules. That is what Beto is tested against, and `gitops/manifests/pharos/base/beto/zammad.env` names the part of it Pharos depends on. It arrived inside a `pg_dump` of the team's Zammad. That dump also holds 124 real tickets, 48 people, 5 organisations, password hashes, the helpdesk mailbox password, webhook tokens and the instance secret. It cannot be committed, cannot be reviewed, pins one Zammad version's schema, and would give anybody who restored it a system able to email real people about real tickets. Handing it to each new dev box is not an option. ## Decision **A dev box's Zammad is installed from a pinned chart and seeded from a configuration held in the repository as names.** `make legacy-controller-zammad` does both, and nothing of the team's data reaches a dev box. - **Pinned, so every dev box is the same one.** `devops/zammad/release.json` holds the chart versions (zammad 19.1.3, Zammad 7.2.1-0000, eck-operator 3.5.0) and `devops/zammad/values.yaml` the values. The only value that departs from the chart's defaults is Elasticsearch's memory: the trial in decision 0063 failed on the chart's 1 GiB with a 512 MB heap, and the installed instance was corrected by hand to 2 GiB with a 1 GB heap, which is now what the file says. - **Configuration by name.** `devops/zammad/configuration.json` is the desk's setup with every database identifier replaced by the name it stands for, so it applies to any installation: `"ticket.group_id": {"value": ["3"]}` is held as `{"value": ["Pharos SA – Front Desk"]}`. `export-configuration.py` writes it from the team's Zammad and `seed-configuration.py` applies it. It is reviewable as a difference, which a database dump is not. - **The desk's sign-in provider is seeded, and never switched on.** The team's people sign into Zammad through PHAROS AAI (OpenID Connect, the GRNET Keycloak realm). It carries no client secret — that client uses PKCE — so it travels under `sign_in` with a closed list of fields, and the export refuses a provider that would need a secret or an issuer that is not https. What does not travel is `auth_openid_connect`: Zammad builds its callback from its own address (`http_type://fqdn/auth/openid_connect/callback`), and a dev box's address is a loopback port of its own that the provider does not know, so turning it on would offer a button the provider rejects. Seeding therefore leaves the switch exactly as it finds it, which also means a dev box whose callback *was* registered keeps its choice across re-runs. - **Two settings and a logo, and nothing else Zammad calls a setting.** The desk is also the language and the time zone its agents work in (`settings.locale_default`, `settings.timezone_default`) and the logo its people recognise. Everything else under that name belongs to the installation — its own address, its instance secret, the mailbox it fetches from — so the list is closed rather than exported wholesale, and a configuration holding anything else is refused. The logo travels as `devops/zammad/logo.png` beside the file, because it is the one part of the desk that is not text and twenty kilobytes of base64 inside a file meant to be read as a difference would not be reviewable. Zammad's `product_logo` setting holds only a cache-busting number, so the logo is compared as the bytes Zammad serves. - **No person, ticket or organisation travels.** The export keeps the kinds listed above and drops identity, authorship, timestamps, group membership, ownership and the customer. It refuses to write a file holding a secret-looking value, a leftover identifier, or a name the file does not itself define; `check_infrastructure.py` re-checks all three, and that the profile's groups, ticket type, closed states and integration role are in the file with the group access the profile gives them. - **Each dev box has its own three accounts.** Zammad's own first-start wizard creates an administrator and a test agent from passwords generated into the target's private directory; `setup-integration.py` creates the integration user holding Beto's token. None is printed, none is committed, and none is the team's. - **Outgoing mail goes to a sink.** Mailpit runs in the same namespace as the Service named `mailpit`, which is the host the desk's own notification channel already names, and seeding attaches every email address to it. A dev box can therefore read what Zammad sent and send to nobody. Nothing inbound is configured, so no mailbox password is needed, and an instance that does fetch mail is refused as not being a dev box. - **The integration keeps its owner.** `setup-integration.py` still owns what binds one Zammad to one Beto: the `pp_user_uuid` and `pp_ticket_uuid` fields, the integration role and user, and the webhook and trigger carrying Beto's secret. The export leaves all of them out and seeding does not touch them. - **The command finishes the job.** It also gives Beto a token, when the one Beto holds is not one this Zammad knows, and leaves `dev.env` naming the token file rather than holding a copy, so a later rotation needs no change to it. And it gives the administrator and the test agent full access to every active group: Zammad's wizard runs before the desk's groups exist, and in Zammad all ticket access comes from group access (decision 0063), so without it the agent can neither read nor move a ticket and even the administrator opens an empty ticket list. `make legacy-controller-credentials` prints the three logins, so being private and root-owned does not make them unreachable. ## What installing one from nothing taught us The decision above was written before any dev box had been built this way. The first real run, on 14 October 2026, took eight attempts, because each of these is something only a live Zammad says. All are handled and covered by a test: | What Zammad does | What it means for seeding | |---|---| | Its name uniqueness ignores case | The desk's "Closed" **is** the state a fresh Zammad calls "closed", renamed. Objects are matched case-insensitively, so the one that is there is corrected rather than duplicated, which a create is refused for (422) | | It answers with fields it maintains itself | Whether a ticket field may be deleted, what stops that, every option a field has ever had, and an overview's numbered slug are neither exported nor compared; seeding them never converged | | A group owns its signature and address | Those two are seeded before any group exists, so their own lists of groups are not written back | | It refuses a condition naming a field it has not reloaded | The ticket fields are migrated, and Zammad reloaded, before anything selects on them ("Invalid object selector conditions") | | Its nginx answers 502 while it reloads | That is "not yet", not "no", so it is waited out | | Its first-start wizard runs only when asked | Nothing in the chart asks; the command calls `getting_started/auto_wizard`, without which an installation has no administrator at all | | Which refusal it gives differs between installations | Set up, it refuses an unauthenticated setup query with a 401 on one instance and a 403 on another, so any refusal counts as "already set up" and the administrator's own login is what proves which Zammad it is | | Its objects refer to each other within one kind | A state names the state it moves on to, so each write is made resolvable at once rather than when the whole kind is done | | An agent without group access can do nothing | Not Zammad's fault but its model (decision 0063): the wizard runs before the desk's groups exist, so the administrator and the test agent are both given access afterwards | | Its logo endpoint authorises by the setting its path names | `PUT /settings/image/product_logo` answers 500 (`no policy for nil`), although the action ignores which setting it is told: the path carries the setting's own number | | It masks anything secret-looking inside a setting | A settings hash answers with keys ending `secret`, `token`, `_key` or `bind_pw` starred, so an export that carried them would commit a placeholder, not a value. Only a closed list of fields travels, and a secret-looking one is refused rather than written | | Its chart's init Job races its own Pods, and helm cannot wait for it | The Job synchronises translations while the railsserver does the same, and the loser dies on a duplicate `translations_pkey`, crash-loops and never reaches the container that sets `es_url`. The chart cannot make it a helm hook, because the Job needs the Pods helm would be waiting for, so a failed Job leaves an installation that looks healthy and has no search index. The install sets the search backend itself and a re-run repairs one that came up without it | | A column it holds as NULL answers as an empty hash | A report profile selecting everything is `NULL` in its database and `{}` over its API. The file holds what the API answers, which is what an export reads and a check compares | | It keeps no checksum of the logo it serves | `product_logo` holds the second the upload happened, so a logo can only be compared as bytes. Zammad serves them without a login, and answers with its own logo when nothing was uploaded, so there is always something to compare | ## Options we rejected | Option | Why not | |---|---| | Ship the team's dump and restore it on each dev box | Personal data, the instance secret and a live mailbox password in every checkout; unreviewable; pins one schema | | Ship the dump with the data deleted and secrets scrubbed | Still hundreds of unreviewable tables, and one missed table leaks; no way to show that nothing personal remains | | Install Zammad with the chart's defaults and configure it by hand | What Beto is tested against is the desk's groups, states and types; defaults have none of them, so every tester would differ | | Keep `zammad_migrate/scripts/zammad-migrate.sh` as the only path | It replaces one existing instance's database from a snapshot the operator already holds. It stays for that, and is not a way to give a new dev box a Zammad | | Install Zammad as part of `make legacy-controller-setup` | Zammad adds about 4 GiB of memory and 21 GiB of volumes; dev boxes not testing Support should not pay it | | Add Zammad to `gitops/` with the platform's own workloads | Zammad is a third-party application a dev box borrows, not a Pharos workload, and the overlays describe what Pharos deploys | ## Verified (14 October 2026) The dev box's Zammad was destroyed and installed again from nothing: the Helm releases uninstalled, the namespace and its volumes deleted, and the ECK operator removed. `make legacy-controller-zammad` then rebuilt it, and a second run changed nothing (138 checks, no difference), which is what makes it safe to re-run. - **Same versions**: chart `zammad-19.1.3`, Zammad `7.2.1-0000`, `eck-operator-3.5.0`, Elasticsearch 9.5.0, PostgreSQL 18.4, Redis 8.10.2, Memcached 1.6.45 — the ones the instance had before, with Elasticsearch now on 2 GiB and a 1 GB heap from the values file rather than by hand. - **Same configuration**: all 12 groups, the desk's 8 states with their state types, 4 priorities, 11 SLAs, 13 triggers, 12 overviews, 7 core workflows, 5 macros and 17 report profiles, with Zammad's own leftover states and priorities deactivated rather than deleted. `setup-integration.py --check` reports no difference from the profile. - **None of the team's data**: 1 ticket (Zammad's own example), 5 users (the three this dev box created, plus Zammad's) and 2 organisations, against 124 tickets, 48 users and 5 organisations before. - **It searches**: `es_url`, `es_user` and `es_password` name the Elasticsearch beside it, and the Ticket index exists. This is not cosmetic: Beto finds a ticket it has already filed by searching for its `pp_ticket_uuid`, which is what keeps a timed-out filing from being filed a second time, so an installation without it breaks a Support guarantee while looking healthy. - **Support works against it**: all six of Beto's Zammad contract tests pass, covering filing, finding, claiming, listing, replying, closing into the profile's close state, following a ticket through the partner groups and losing sight of one moved outside them, and the token being refused administration. Zammad's "Pharos ticket sync" webhook reaches Beto across namespaces (HTTP 200), and 98 notification messages landed in Mailpit, addressed to the dev box's own agent. One thing the rebuild also showed: closing a ticket on a cold Zammad can take longer than Beto's per-attempt budget, so the first close after an install may be reported as unreachable although Zammad applied it. It passes once the instance is warm. Beto's budget against a cold third-party system is left as a question for the Support iteration, not for this decision. ## Verified again, from nothing (14 October 2026, second rebuild) Everything above was destroyed again — both Helm releases, both namespaces and all three volumes — and rebuilt by one `make legacy-controller-zammad`, which succeeded on its first attempt. The rebuild is what found the two things this decision now describes: the chart's init Job dying on a translations collision and leaving no search index, and a calendar's `last_sync` being Zammad's own bookkeeping rather than the desk's. - **The first start carries the desk's own settings.** Zammad's defaults for the language and time zone happen to be the desk's, so those two alone prove nothing; `product_name` and `fqdn` ride in the same wizard list and did move off Zammad's defaults (`Zammad Helpdesk`, `zammad.example.com`), and the payload Zammad was handed holds all four. Seeding then reported the language and time zone as already right rather than correcting them. - **159 checks, no difference**, including the logo byte for byte, the sign-in provider, and all 17 report profiles. - **None of the team's data**: 1 ticket, 5 users, 2 organisations. - **All six contract tests pass** once the crash-looping init Job has given up; while it was restarting, four of them timed out against the loaded instance. ## Consequences - The configuration is a copy, so it drifts when the team changes their Zammad. Re-running `export-configuration.py` and committing the difference is the same review step the profile already has, and `seed-configuration.py --check` reports the drift on any dev box. - Seeding deactivates, rather than deletes, the states, priorities, SLAs, triggers, macros, overviews, core workflows and templates the configuration does not name, because the desk replaced Zammad's own wholesale. Nothing is removed from a Zammad, so a mistake is visible and reversible. - A dev box's Zammad has no ticket history, so testing anything about old tickets still needs the team's snapshot and the migrate script. - **Installing one where another stood invalidates what Beto recorded.** A SupportCustomer link holds a Zammad user ID and a Ticket a Zammad ticket ID, and a dev box's Zammad carries none of the team's people, so a rebuilt one issues both from 1 again. The old records then name identifiers that mean somebody else, and it does not fail cleanly: while the new Zammad has issued fewer identifiers than Beto holds a filing fails visibly, and after that it is made under whoever owns that number now. `make legacy-controller-support-reset` empties them and links the local operator again. A Zammad *restored* from a dump of the same database keeps every identifier and needs none of this; whether identifiers were kept is the only question that matters. - The export covers what the API exposes by name, which an audit against the team's own backup (14 October 2026) confirmed is all of the desk there is to carry. What it leaves out is either theirs rather than the desk's (5 organisations, 48 people, 124 tickets and the 43 tags on them), empty in their Zammad (public links, scheduler jobs, checklist templates, and a knowledge base holding no category or answer), Zammad's own in every installation (10 `base -` core workflows, 52 uneditable ticket fields), or bound to one host (their "Local Operaton ticket lifecycle" webhook, which posts to `host.docker.internal`). The tag vocabulary is not carried because the desk's macros create it: Zammad makes a tag the first time one is applied. - The logo travels as a PNG. The export refuses a desk whose logo is an SVG or another format rather than committing something it cannot apply, and refuses one serving Zammad's own logo, which says the instance is not the desk. - A dev box signs in with its own generated administrator and test agent, not through PHAROS AAI. Turning that on needs the dev box's own callback registered at the AAI, which is a change at the identity provider rather than in this repository; `auth_third_party_auto_link_at_inital_login` is the other setting to consider then, and seeding writes neither. - The desk's language is English (`en-us`), which is what the team's Zammad holds and what every one of its users had chosen. A dev box in another language is a per-person choice under Avatar → Profile → Language, not a configuration change. Zammad follows the browser only for somebody who has not chosen one, which is why a dev box whose administrator was made by hand could appear in a language the desk is not set to. - The three generated logins live in the target's private directory. Losing that directory loses the administrator password, and the Zammad has to be reinstalled or its password reset from the Rails console. - The chart passes the first-start configuration to Zammad as a Secret in the namespace, so the generated administrator password also stays there after the wizard has run and deleted its own copy of the file. It is a dev box's own password, held where that dev box's other credentials already are. - `make legacy-controller-zammad-urls` serves Zammad and Mailpit on loopback only while it runs, so no dev box leaves a support desk reachable afterwards.