#0064 — A dev box's Zammad is seeded from a configuration, not restored from the team's

Status: Accepted. Date: 14 October 2026. Installation superseded by decision 0065 (Zammad is an optional Argo CD service; the legacy commands below are retired); the configuration by name stands.

Support (Beto) is the only context that talks to a third-party application, so testing it needs a Zammad. Only one dev box had one. It was installed by hand (Helm, by one person, in October 2026) and then had the team's production database restored into it, so every other dev box could run the platform but not test Support at all, and no command existed to change that.

#The problem with the way it got there

The dev-box Zammad's configuration is the support desk's real one: 12 groups, 8 states, 4 priorities, 11 SLAs, 12 triggers, 7 core workflows, 12 overviews, 13 administrator-editable ticket fields, business hours, 17 report profiles, templates and text modules. That is what Beto is tested against, and gitops/manifests/pharos/base/beto/zammad.env names the part of it Pharos depends on.

It arrived inside a pg_dump of the team's Zammad. That dump also holds 124 real tickets, 48 people, 5 organisations, password hashes, the helpdesk mailbox password, webhook tokens and the instance secret. It cannot be committed, cannot be reviewed, pins one Zammad version's schema, and would give anybody who restored it a system able to email real people about real tickets. Handing it to each new dev box is not an option.

#Decision

A dev box's Zammad is installed from a pinned chart and seeded from a configuration held in the repository as names. make legacy-controller-zammad does both, and nothing of the team's data reaches a dev box.

• Pinned, so every dev box is the same one. devops/zammad/release.json holds the chart versions (zammad 19.1.3, Zammad 7.2.1-0000, eck-operator 3.5.0) and devops/zammad/values.yaml the values. The only value that departs from the chart's defaults is Elasticsearch's memory: the trial in decision 0063 failed on the chart's 1 GiB with a 512 MB heap, and the installed instance was corrected by hand to 2 GiB with a 1 GB heap, which is now what the file says.
• Configuration by name. devops/zammad/configuration.json is the desk's setup with every database identifier replaced by the name it stands for, so it applies to any installation: "ticket.group_id": {"value": ["3"]} is held as {"value": ["Pharos SA – Front Desk"]}. export-configuration.py writes it from the team's Zammad and seed-configuration.py applies it. It is reviewable as a difference, which a database dump is not.
• The desk's sign-in provider is seeded, and never switched on. The team's people sign into Zammad through PHAROS AAI (OpenID Connect, the GRNET Keycloak realm). It carries no client secret — that client uses PKCE — so it travels under sign_in with a closed list of fields, and the export refuses a provider that would need a secret or an issuer that is not https. What does not travel is auth_openid_connect: Zammad builds its callback from its own address (http_type://fqdn/auth/openid_connect/callback), and a dev box's address is a loopback port of its own that the provider does not know, so turning it on would offer a button the provider rejects. Seeding therefore leaves the switch exactly as it finds it, which also means a dev box whose callback was registered keeps its choice across re-runs.
• Two settings and a logo, and nothing else Zammad calls a setting. The desk is also the language and the time zone its agents work in (settings.locale_default, settings.timezone_default) and the logo its people recognise. Everything else under that name belongs to the installation — its own address, its instance secret, the mailbox it fetches from — so the list is closed rather than exported wholesale, and a configuration holding anything else is refused. The logo travels as devops/zammad/logo.png beside the file, because it is the one part of the desk that is not text and twenty kilobytes of base64 inside a file meant to be read as a difference would not be reviewable. Zammad's product_logo setting holds only a cache-busting number, so the logo is compared as the bytes Zammad serves.
• No person, ticket or organisation travels. The export keeps the kinds listed above and drops identity, authorship, timestamps, group membership, ownership and the customer. It refuses to write a file holding a secret-looking value, a leftover identifier, or a name the file does not itself define; check_infrastructure.py re-checks all three, and that the profile's groups, ticket type, closed states and integration role are in the file with the group access the profile gives them.
• Each dev box has its own three accounts. Zammad's own first-start wizard creates an administrator and a test agent from passwords generated into the target's private directory; setup-integration.py creates the integration user holding Beto's token. None is printed, none is committed, and none is the team's.
• Outgoing mail goes to a sink. Mailpit runs in the same namespace as the Service named mailpit, which is the host the desk's own notification channel already names, and seeding attaches every email address to it. A dev box can therefore read what Zammad sent and send to nobody. Nothing inbound is configured, so no mailbox password is needed, and an instance that does fetch mail is refused as not being a dev box.
• The integration keeps its owner. setup-integration.py still owns what binds one Zammad to one Beto: the pp_user_uuid and pp_ticket_uuid fields, the integration role and user, and the webhook and trigger carrying Beto's secret. The export leaves all of them out and seeding does not touch them.
• The command finishes the job. It also gives Beto a token, when the one Beto holds is not one this Zammad knows, and leaves dev.env naming the token file rather than holding a copy, so a later rotation needs no change to it. And it gives the administrator and the test agent full access to every active group: Zammad's wizard runs before the desk's groups exist, and in Zammad all ticket access comes from group access (decision 0063), so without it the agent can neither read nor move a ticket and even the administrator opens an empty ticket list. make legacy-controller-credentials prints the three logins, so being private and root-owned does not make them unreachable.

#What installing one from nothing taught us

The decision above was written before any dev box had been built this way. The first real run, on 14 October 2026, took eight attempts, because each of these is something only a live Zammad says. All are handled and covered by a test:

#Options we rejected

#Verified (14 October 2026)

The dev box's Zammad was destroyed and installed again from nothing: the Helm releases uninstalled, the namespace and its volumes deleted, and the ECK operator removed. make legacy-controller-zammad then rebuilt it, and a second run changed nothing (138 checks, no difference), which is what makes it safe to re-run.

• Same versions: chart zammad-19.1.3, Zammad 7.2.1-0000, eck-operator-3.5.0, Elasticsearch 9.5.0, PostgreSQL 18.4, Redis 8.10.2, Memcached 1.6.45 — the ones the instance had before, with Elasticsearch now on 2 GiB and a 1 GB heap from the values file rather than by hand.
• Same configuration: all 12 groups, the desk's 8 states with their state types, 4 priorities, 11 SLAs, 13 triggers, 12 overviews, 7 core workflows, 5 macros and 17 report profiles, with Zammad's own leftover states and priorities deactivated rather than deleted. setup-integration.py --check reports no difference from the profile.
• None of the team's data: 1 ticket (Zammad's own example), 5 users (the three this dev box created, plus Zammad's) and 2 organisations, against 124 tickets, 48 users and 5 organisations before.
• It searches: es_url, es_user and es_password name the Elasticsearch beside it, and the Ticket index exists. This is not cosmetic: Beto finds a ticket it has already filed by searching for its pp_ticket_uuid, which is what keeps a timed-out filing from being filed a second time, so an installation without it breaks a Support guarantee while looking healthy.
• Support works against it: all six of Beto's Zammad contract tests pass, covering filing, finding, claiming, listing, replying, closing into the profile's close state, following a ticket through the partner groups and losing sight of one moved outside them, and the token being refused administration. Zammad's "Pharos ticket sync" webhook reaches Beto across namespaces (HTTP 200), and 98 notification messages landed in Mailpit, addressed to the dev box's own agent.

One thing the rebuild also showed: closing a ticket on a cold Zammad can take longer than Beto's per-attempt budget, so the first close after an install may be reported as unreachable although Zammad applied it. It passes once the instance is warm. Beto's budget against a cold third-party system is left as a question for the Support iteration, not for this decision.

#Verified again, from nothing (14 October 2026, second rebuild)

Everything above was destroyed again — both Helm releases, both namespaces and all three volumes — and rebuilt by one make legacy-controller-zammad, which succeeded on its first attempt. The rebuild is what found the two things this decision now describes: the chart's init Job dying on a translations collision and leaving no search index, and a calendar's last_sync being Zammad's own bookkeeping rather than the desk's.

• The first start carries the desk's own settings. Zammad's defaults for the language and time zone happen to be the desk's, so those two alone prove nothing; product_name and fqdn ride in the same wizard list and did move off Zammad's defaults (Zammad Helpdesk, zammad.example.com), and the payload Zammad was handed holds all four. Seeding then reported the language and time zone as already right rather than correcting them.
• 159 checks, no difference, including the logo byte for byte, the sign-in provider, and all 17 report profiles.
• None of the team's data: 1 ticket, 5 users, 2 organisations.
• All six contract tests pass once the crash-looping init Job has given up; while it was restarting, four of them timed out against the loaded instance.

#Consequences

• The configuration is a copy, so it drifts when the team changes their Zammad. Re-running export-configuration.py and committing the difference is the same review step the profile already has, and seed-configuration.py --check reports the drift on any dev box.
• Seeding deactivates, rather than deletes, the states, priorities, SLAs, triggers, macros, overviews, core workflows and templates the configuration does not name, because the desk replaced Zammad's own wholesale. Nothing is removed from a Zammad, so a mistake is visible and reversible.
• A dev box's Zammad has no ticket history, so testing anything about old tickets still needs the team's snapshot and the migrate script.
• Installing one where another stood invalidates what Beto recorded. A SupportCustomer link holds a Zammad user ID and a Ticket a Zammad ticket ID, and a dev box's Zammad carries none of the team's people, so a rebuilt one issues both from 1 again. The old records then name identifiers that mean somebody else, and it does not fail cleanly: while the new Zammad has issued fewer identifiers than Beto holds a filing fails visibly, and after that it is made under whoever owns that number now. make legacy-controller-support-reset empties them and links the local operator again. A Zammad restored from a dump of the same database keeps every identifier and needs none of this; whether identifiers were kept is the only question that matters.
• The export covers what the API exposes by name, which an audit against the team's own backup (14 October 2026) confirmed is all of the desk there is to carry. What it leaves out is either theirs rather than the desk's (5 organisations, 48 people, 124 tickets and the 43 tags on them), empty in their Zammad (public links, scheduler jobs, checklist templates, and a knowledge base holding no category or answer), Zammad's own in every installation (10 base - core workflows, 52 uneditable ticket fields), or bound to one host (their "Local Operaton ticket lifecycle" webhook, which posts to host.docker.internal). The tag vocabulary is not carried because the desk's macros create it: Zammad makes a tag the first time one is applied.
• The logo travels as a PNG. The export refuses a desk whose logo is an SVG or another format rather than committing something it cannot apply, and refuses one serving Zammad's own logo, which says the instance is not the desk.
• A dev box signs in with its own generated administrator and test agent, not through PHAROS AAI. Turning that on needs the dev box's own callback registered at the AAI, which is a change at the identity provider rather than in this repository; auth_third_party_auto_link_at_inital_login is the other setting to consider then, and seeding writes neither.
• The desk's language is English (en-us), which is what the team's Zammad holds and what every one of its users had chosen. A dev box in another language is a per-person choice under Avatar → Profile → Language, not a configuration change. Zammad follows the browser only for somebody who has not chosen one, which is why a dev box whose administrator was made by hand could appear in a language the desk is not set to.
• The three generated logins live in the target's private directory. Losing that directory loses the administrator password, and the Zammad has to be reinstalled or its password reset from the Rails console.
• The chart passes the first-start configuration to Zammad as a Secret in the namespace, so the generated administrator password also stays there after the wizard has run and deleted its own copy of the file. It is a dev box's own password, held where that dev box's other credentials already are.
• make legacy-controller-zammad-urls serves Zammad and Mailpit on loopback only while it runs, so no dev box leaves a support desk reachable afterwards.

Page source

Source snapshot · Revision f928834c8.