--- title: 'DD-0060: Development Seeds for Waldur, GitLab and SSH Access' --- # DD-0060 — Development seeds for Waldur, GitLab and SSH Access **Related ADR:** [ADR-0060](../decisions/0060-development-seeds.md). **Status:** Draft for review, with dev-box evidence on Waldur 8.1.3 and GitLab 19.4.2. **Date:** 2026-10-22. ## Use ```sh make seed # the Waldur seed, as below make waldur-seed # dx/common/waldur-seed.yaml make gitlab-seed # dx/common/gitlab-seed.yaml, after make seed make waldur-seed SEED=my.yaml # another file make ssh-access-seed # dx/common/ssh-access-seed.yaml make ssh-access-seed CHECK=1 # fail instead of writing on drift ``` The commands use the target and SSH alias of the root `.env`, like `make credentials`. They print one line per seed entry, `created` or `present`, and a total. A repeated run prints only `present` lines and sends no change. Members must be users of `devops/aai/users.json`. Waldur names a user after the subject of the sign-in, so the Waldur seed looks members up by the subject the local AAI gives them. When a member has no account in the service yet, the command changes nothing, signs that member in once with `dx/common/scripts/service-login.sh` and applies the seed again. The sign-in needs the short addresses (`make devbox-browser-setup`) or `PHAROS_SERVICE_EDGE=1`. The GitLab seed names no members: a Project's members reach its GitLab groups through Source Control, once they have signed in to GitLab. ## Seed files Each file has a JSON schema beside it (`waldur-seed.schema.json`, `gitlab-seed.schema.json`). The command validates the file before it connects. ```yaml # dx/common/waldur-seed.yaml version: 1 customer: Pharos # must exist; the installation creates it projects: - name: Project A # matched by exact name within the customer description: Development project A members: - {user: alice, role: manager} # admin | manager | member orders: - offering: llm-access-offe # provider offering slug plan: LLM access plan # optional when one plan is active name: project-a-llm # resource name; matched to skip a repeat attributes: {models: [qwen2.5-0.5b-cpu]} limits: {max_budget: 1, tpm_limit: 100000, rpm_limit: 100} - name: Project Cloud # a cloud order only members: - {user: alice, role: manager} orders: - {offering: cloud, name: project-cloud, limits: {cpu: 2, memory: 4, storage: 5}} - name: Project HPC # an HPC order only members: - {user: bob, role: manager} orders: - {offering: hpc-icarus, name: project-hpc} ``` The `offerings` section lists the cloud and HPC offerings under the identity of their owners: the Site Agent's installation hook registers `cloud` and the Icarus laboratory (`make metallo-icarus`) registers `hpc-icarus`. The seed finds an offering by slug and does not change it, so it never duplicates an installed one, with two exceptions: it adds a plan to an offering that has none (Waldur activates only an offering with a plan), and it activates one that is still a draft; a paused or archived offering is left as an operator set it. It creates the provider organisation, service provider, category, offering (with the components its orders limit: the cloud offering's cpu, memory and storage, which Waldur refuses an order without), plan and activation only when the offering is missing. An offering whose slug belongs to another organisation stops the run. Keep the section in step with its owners, because a copy is only used on a dev box where they have not run. An order names its offering by slug. The seed and the owners can run in either order: the provider names and slugs are the owners' (`Pharos cloud provider`, `pharos-cloud`; `Icarus-hpc provider`), and each owner finds an existing organisation, offering, category and plan by them. `make metallo-icarus` also brings the HPC offering's cpu component and plugin options to its own values. On a box where `make metallo-icarus` has not run, nothing processes the HPC order of Project HPC, so it is placed and approved but never completed. ```yaml # dx/common/gitlab-seed.yaml version: 2 root: pharos-projects # optional: Source Control's root group, this by default lfs: # LFS files added to every repository: path -> bytes weights/tiny.bin: 1048576 projects: - project: Project A # a Project of the Waldur seed, by its exact name repositories: - kind: repos # models | datasets | spaces | recipes | upskillings | repos name: sample # receives dx/common/gitlab-seed/sample plus the LFS files ``` The GitLab seed adds repositories to the groups Source Control made for the seeded Projects, at `///`, where Themelio reads them. It finds a Project's group among the subgroups of the root by the name Source Control gives it, ` ()`, where the Project ID is the group's path, and then the kind subgroup. The schema refuses a Project name that Source Control would rewrite in that name. All groups are found before anything is written: while a Project or kind group is missing, or two groups carry the same Project name, the run changes nothing and names the Project. Source Control makes the groups on its next pass after Beto takes the Project from Waldur, so a first run straight after `make seed` may have to be repeated. Every GitLab repository receives the text files of `dx/common/gitlab-seed/sample`: a README, a CSV, a Python script and a `.gitattributes` that tracks `*.bin` and `*.safetensors` in LFS. The LFS files' bytes are generated deterministically on the dev box, so the platform repository stores no binaries and every run produces the same object IDs. LFS paths must end in `.bin` or `.safetensors` and be at most 100 MiB. ## SSH access browser fixture ```yaml # dx/common/ssh-access-seed.yaml version: 1 site: hpc # the single Site of the Site API and of SSH access currentAup: v1 projects: - {id: project-a, name: Project A} members: - user: alice # a user of devops/aai/users.json member: 00000000-0000-4000-8000-0000000a11ce # Party member stand-in; keep it stable uidNumber: 5000001 # the UID the laboratory's Waldur pool assigned revision: 3 acceptedAup: v1 projects: [project-a, project-b] actors: # actors that are not AAI sign-ins: operator and CLI fixtures - {actor: 00000000-0000-4000-8000-000000000001, member: 00000000-0000-4000-8000-0000000a11ce} ``` The SSH access fixture stands in for Beto's Party adaptors until they replace it. The browser API acts as the **Party** of the signed-in session, and a Party appears at a person's first sign-in: `services/api/adaptors/authentication_sessions.go` calls `PartyID` the subject the SSH context sees. So the fixture needs an actor entry keyed by that Party, not by the subject the local AAI derives from the user name. Without it every SSH access route answers `403 member_not_found` and the Site list is empty. The Party is created per installation, at the first sign-in, so it cannot live in the committed manifests. `make ssh-access-seed` reads it, read-only, from `pharos.party_authentication_identities` on the dev box selected by the root `.env` (`SELECT` only; nothing is written there), keyed by the subject the local AAI gives the user (`uuid5(NAMESPACE_URL, 'pharos-lab:pharos:' + user)`). A user who has not signed in yet has no Party: the seed writes the rest, prints that the SSH access page will answer `member_not_found` for them, and asks for a sign-in and a second run. The delivered manifests therefore carry only the actors that are not sign-ins; a dev box's run adds the Party entries of the users who have signed in. `make ssh-access-seed` validates the file, takes each member's `posix_username` from `devops/aai/users.json` and the Projects' names from the seed, and renders one `owners.json` value with one member per seeded user, one actor per sign-in Party and the actor fixtures of the seed. It writes that value into `gitops/manifests/pharos/base/ssh-access/workload.yaml`, which `make deploy` delivers. The ConfigMap belongs to the `pharos` Argo application, which self-heals it, so the seed and the manifests are the only writable path: nothing edits the running ConfigMap or a database by hand. A second run reports `present` for every member and changes nothing; `CHECK=1` exits non-zero when the delivered manifests differ from the seed. `NO_PARTIES=1` renders the form the repository carries, without reading a dev box. The seed never guesses a UID: `uidNumber` must equal the POSIX identity the laboratory's Waldur pool assigned (`dx/metallo/icarus-hpc`), because the Site agent refuses to write a key for a member whose UID disagrees with the directory entry. `revision` follows the member's Site identity: bump it when the username or UID changes. ## What is created and when an entry counts as present | Entry | Present when | Created with | |---|---|---| | Waldur Project | The customer has a Project of that exact name | `POST projects/` | | Waldur member | The user holds that role in the Project | `POST projects/{uuid}/add_user/` | | Waldur order | The Project has a resource of that name on the offering that is not Terminated, or an open order for it | `POST marketplace-orders/` on the public offering and plan, then consumer and provider approval | | GitLab Project and kind group | Source Control made them | Never by the seed: it stops and changes nothing | | GitLab repository | The project exists and has a branch | `POST projects` in the kind group, private, LFS enabled, then the LFS uploads and one commit on `main` | The seed never changes an existing entry. A different role, description or limit on an existing entry stays as it is. A repository that exists without branches, after an interrupted run, receives the sample on the next run. GitLab repository content is written without Git: the toolbox image has none. The seed uploads each LFS object through the Git LFS batch API first, which links it to the project, and then commits the text files and LFS pointers through `POST projects/:id/repository/commits`. A clone with Git LFS restores the objects. ## Credentials and process ```mermaid sequenceDiagram participant W as Workstation participant H as Dev-box host (sudo) participant C as Provider container W->>H: SSH: agent program and validated seed (argv) H->>H: Check cluster node UIDs and maintenance H->>C: kubectl exec: REST program on stdin, seed in the environment C->>C: Mint token, run program, revoke token C-->>H: One JSON report line H-->>W: Sanitised report or a fixed error message ``` - **Workstation:** `dx/common/scripts/dev_seed.py` validates the seed, adds the GitLab sample files and sends one program over SSH (`credential_agent.py`, `seed_agent.py`, the service's agent and its REST program). SSH runs without agent or port forwarding, as `make credentials` does. - **Host:** `seed_agent.py` checks that the cluster's node UIDs match the target's binding and that the platform is not under maintenance. Any failure crosses SSH as a fixed message, never as subprocess output. - **Waldur API container:** `waldur_seed_agent.py` runs one `bash -c` that mints a token for the staff user `pharos-seed` (`token_lifetime` 900 seconds) with `waldur shell`, runs `waldur_seed_api.py` with the token in its environment and deletes the user's tokens afterwards. - **GitLab toolbox container:** `gitlab_seed_agent.py` runs one `gitlab-rails runner` that creates a `root` personal access token named `pharos-seed` (scope `api`, expiry one day, the shortest GitLab allows), spawns `gitlab_seed_api.py` with the token in its environment and revokes the token in an `ensure` block. The program first revokes any active `pharos-seed` token left by an interrupted earlier run. It reaches GitLab at `gitlab-webservice-default.gitlab.svc:8181`. LFS upload addresses name the external URL, so the program sends them to the same path on the internal address. - **Reports:** the REST programs print one JSON line with the entries' labels. A rejected request reports the HTTP status and the names of the rejected fields, never the values or the response body. ## Verification - `dx/common/tests/test_waldur_seed.py`: - checks every Waldur request body against the 8.1.3 OpenAPI schemas extracted to `dx/common/tests/fixtures/waldur-8.1.3-seed-requests.json`; - checks that the default seed applies once and a second run sends no change; - checks that missing members, customers and offerings change nothing and name the entry; - checks that invalid seeds and unknown users are refused. - `dx/common/tests/test_gitlab_seed.py`, against a GitLab double that holds the groups as Source Control makes them (it checks the seed's logic, not GitLab): - checks that the default seed adds its repository to the Project's kind group once, writes no group or member, and that a second run sends no change; - checks that only the group named after the Project and its own ID is used, and that a missing root, Project group or kind group, or two groups for one Project, change nothing, not even a token; - checks that the earlier group form, members, unknown kinds and Project names Source Control would rewrite are refused, and that the default Project is one the Waldur seed makes; - checks that LFS objects are uploaded before the commit that names them and that only an earlier run's token is revoked; - checks that the token lives in one Rails process and is passed to Python in its environment. - `dx/common/tests/test_ssh_access_seed.py`: - checks that the delivered `ssh-access-fixture` ConfigMap is exactly the rendered seed, and that a second run changes nothing; - checks that a signed-in user is keyed by the Party their first sign-in created, with her `posix_username`, UID and Projects, that a changed Party replaces the old actor, that a user without a sign-in is reported and keyed by nothing, that the fixture never keys an actor by an AAI subject, and that the operator actors still name their member; - checks that unknown users, undefined Projects, a shared member identifier, an actor without a member and an out-of-range UID are refused, and that the drift check fails on a changed manifest. - `services/metallo/tests/ssh_access/test_browser_fixture.py` reads the delivered ConfigMap through the real fixture adaptor and checks the same members, the Site binding, the Projects, the AUP answer, that a Party actor reaches its member, that no actor is an AAI subject, and that a stranger is no member. `services/metallo/tests/test_icarus_hpc.py` keeps the end-to-end lane pinned to alice's revision. - Dev box `dev-ikons`, 22 October 2026: - **Waldur 8.1.3:** a seed with one Project, a member and an LLM access order created all three, and the site agent completed the order. A second run created nothing, and no `pharos-seed` token remained. The test Project and its Slurm accounts were then removed. - **Waldur after the GitLab change:** a throwaway seed was applied twice (2 created, then 2 present), and its Project was deleted. - **GitLab 19.4.2, first run:** 4 entries were created in 1 min 48 s, after `bob` signed in once. A second run created nothing, in 55 s. - **GitLab as `bob` (Developer):** with an ordinary HTTPS Git client and Git LFS, `bob` cloned `playground/sample`. The LFS file was restored with the expected SHA-256. A push to a new branch with a new LFS file succeeded, and a push to `main` was refused. - **GitLab tokens:** no active `pharos-seed` token remained. These GitLab runs used the earlier playground form, a top-level group with its own members. Not covered: - The Project form of the GitLab seed on a dev box: a repository in a Source Control Project group, shown in Themelio. - Waldur 8.1.2. - GitLab versions other than 19.4.2. - A seed with many groups or large LFS files. ## Limits and later work - The Rails start in the GitLab toolbox takes one to two minutes on a loaded dev box, and the toolbox requests only 50m CPU. - The seed finds a Project's group by the name Source Control gives it (`group_name` in `services/metallo/contexts/source_control/domain/gitlab.py`); a change to that name must change the seed. It creates repositories through GitLab's API because Source Control has no command that creates one; when it gains one, the seed should use it. - A first run straight after `make seed` can find no group yet: Source Control makes it on its next pass. The run changes nothing and says so; run it again. - Mirroring the retained Themelio's repositories from `gitlab.grnet.gr` is not included. It needs a personal token on that server and can transfer large LFS objects.