> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # Waldur GitLab Sync When a [Pharos AI Factory](/terms/pharos-ai-factory.md) administrator approves a resource allocation in [Waldur](/terms/waldur.md), the platform must provision the corresponding infrastructure: a Kubernetes namespace with resource quotas **and** a matching set of GitLab groups so that project members can push [models](/terms/model.md), [datasets](/terms/dataset.md), [spaces](/terms/space.md), repos, recipes, and upskillings. This page documents how that synchronisation works end-to-end, how it is deployed in staging/production, and how to test it locally. ## Architecture overview The sync pipeline involves three systems connected by polling and REST calls: ```mermaid flowchart TD A["Waldur UI - Marketplace"] B["Waldur K8s Site Agent"] C["GitLab Root Actor"] D["GitLab"] E["Kubernetes"] A -- "Waldur REST API\n(polled every 2 min)" --> B B -- "POST /waldur/*\n(Bearer token auth)" --> C C -- "GitLab REST API v4\n(root PAT)" --> D B -- "K8s API\n(ServiceAccount)" --> E subgraph site_agent ["Site Agent Deployments"] direction LR B1["order_process\nprovisions resources"] B2["membership_sync\nsyncs team members"] B3["report\nusage reporting"] end B --- site_agent ``` **No webhooks are used.** The site agent polls the Waldur API at fixed intervals and drives all downstream actions synchronously. ### What happens when an admin approves a project 1. A Pharos admin approves an order in the Waldur marketplace UI. 2. The **order_process** deployment polls the Waldur API (every 2 minutes) and picks up the approved order. 3. The site agent creates a **Kubernetes namespace** with a `ResourceQuota` matching the Waldur limits (CPU, memory, storage) and labels it with Waldur project/resource UUIDs. 4. The site agent calls the GitLab Root Actor's `POST /waldur/project_create` endpoint, passing the project slug, UUIDs, backend ID, and limits. 5. The GitLab Root Actor creates **six subgroups** on GitLab — one under each top-level project-type group: ``` models/ datasets/ spaces/ repos/ recipes/ upskillings/ ``` 6. Waldur metadata (project UUID, resource UUID, backend ID, storage quota, limits) is written to each subgroup as GitLab **custom attributes**. 7. The site agent then calls `POST /waldur/add_project_member` for each initial team member from the Waldur order. ### What happens when team membership changes 1. The **membership_sync** deployment polls Waldur (every 2 minutes) for team membership changes. 2. For each added user, the site agent: - Creates a Kubernetes `ServiceAccount` and `RoleBinding` in the project namespace. - Calls `POST /waldur/add_project_member` with the user's email and offering username. - The GitLab Root Actor looks up the user by email. If found, the user is added to all six subgroups. If not, a GitLab group invitation is sent instead. - A member mapping (`offering_username` → `email`) is stored as a custom attribute on each subgroup, so future removal can work even without the email in the request. 3. For each removed user, the reverse process runs via `POST /waldur/delete_project_member`. ### What happens when a project is decommissioned 1. The site agent calls `POST /waldur/project_delete`. 2. The GitLab Root Actor deletes all six subgroups and their contents. 3. The site agent deletes the Kubernetes namespace. ### Usage reporting The **report** deployment polls every 10 minutes, reads `ResourceQuota.status.used` from each managed namespace, converts backend units to Waldur units (using `unit_factor` from the config), and reports totals back to Waldur. ## Production / staging deployment ### Waldur K8s Site Agent The site agent runs in the `waldur-site-agent-cloud` namespace as three separate Kubernetes Deployments, each running a different mode of the `waldur_site_agent` command: | Deployment | Command | Poll interval | | ------------------------------------ | -------------------------------------- | ------------- | | `waldur-agent-cloud-order-process` | `waldur_site_agent -m order_process` | 2 min | | `waldur-agent-cloud-membership-sync` | `waldur_site_agent -m membership_sync` | 2 min | | `waldur-agent-cloud-report` | `waldur_site_agent -m report` | 10 min | Configuration is mounted from a Kubernetes secret (`waldur-site-agent-cloud-config`) containing a YAML file with: ```yaml offerings: - name: "Kubernetes namespaces via Site Agent (cloud)" waldur_api_url: "" waldur_api_token: "" waldur_offering_uuid: "" backend_settings: gitlab_root_actor_url: "http://gitlab-root-actor.themelio.svc.cluster.local" gitlab_root_actor_token: "" gitlab_root_actor_timeout_seconds: 10 quota_name: "waldur-quota" quota_mapping: cpu: ["requests.cpu", "limits.cpu"] # 1 Waldur CPU = 1000m memory: ["requests.memory", "limits.memory"] # 1 Waldur GiB = 1024 MiB storage: ["requests.storage"] # 1 Waldur GiB = 1 GiB backend_components: cpu: measured_unit: "Core" unit_factor: 1000 memory: measured_unit: "GiB" unit_factor: 1024 storage: measured_unit: "GiB" unit_factor: 1 ``` The site agent's `ClusterRole` grants it full access to namespaces, resource quotas, service accounts, roles, and role bindings across the cluster. ### GitLab Root Actor The GitLab Root Actor runs in the `themelio` namespace as a single Deployment (`gitlab-root-actor`) behind a ClusterIP service on port 80 (internal port 3000). It is **not** exposed to the public internet. Secrets are mounted from `gitlab-root-pat-secret`: | Secret key | Maps to env var | Description | | ---------------------------------- | ---------------------------------- | -------------------------------------------- | | `GITLAB_TOKEN` | `GITLAB_ROOT_ACCESS_TOKEN` | GitLab root PAT with `api` scope | | `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN` | `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN` | Shared secret matching the site agent config | The `GITLAB_URL` in staging points to the cluster-internal GitLab service: `http://gitlab-webservice-default.gitlab.svc.cluster.local:8181` ### Prerequisites on GitLab The following **top-level groups** must exist before the sync can operate: - `models`, `datasets`, `spaces`, `repos`, `recipes`, `upskillings` If any are missing, every sync call fails with `GitLab project type group was not found`. ## End-to-end testing on the dev environment This section walks through testing the full pipeline — from the Waldur UI through the site agent to GitLab — on the dev environment (`dev.waldur.pharos.grnet.gr` + `dev.gitlab.pharos.grnet.gr`). ### Step 1: Verify the pipeline is wired up Before creating any orders, confirm every link in the chain is healthy. **1.1. Check site agent pods are running:** ```bash kubectl -n waldur-site-agent-cloud get pods ``` You should see three pods (one per deployment) in `Running` state: - `waldur-agent-cloud-order-process-*` - `waldur-agent-cloud-membership-sync-*` - `waldur-agent-cloud-report-*` If any pods are in `CrashLoopBackOff` or missing, check the logs: ```bash kubectl -n waldur-site-agent-cloud logs -l app=waldur-agent-cloud-order-process --tail=50 ``` **1.2. Check the GitLab Root Actor is running:** ```bash kubectl -n themelio get pods -l app=gitlab-root-actor ``` Verify it responds to health checks from inside the cluster: ```bash kubectl -n themelio exec deploy/gitlab-root-actor -- \ curl -s http://localhost:3000/-/health ``` **1.3. Verify the offering UUID is configured:** Check that the site agent config secret has a non-empty offering UUID: ```bash kubectl -n waldur-site-agent-cloud get secret waldur-site-agent-cloud-config \ -o jsonpath='{.data.waldur-site-agent-config\.yaml}' | base64 -d | grep waldur_offering_uuid ``` If the UUID is empty, the site agent will poll but never pick up orders. Populate it via the Ansible vault and re-run the playbook. **1.4. Verify the offering exists in Waldur:** Log into `https://dev.waldur.pharos.grnet.gr` as an admin. Navigate to **Provider > Offerings** and confirm there is a Kubernetes offering matching the UUID from step 1.3. Note the offering name — you will use it when creating an order. **1.5. Verify GitLab has the required top-level groups:** ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups?top_level_only=true&per_page=100" \ | jq '.[].path' ``` All six must be present: `models`, `datasets`, `spaces`, `repos`, `recipes`, `upskillings`. Create any missing ones manually in the GitLab admin UI before proceeding. ### Step 2: Understand the Waldur side **2.1. Log into Waldur as admin:** Go to `https://dev.waldur.pharos.grnet.gr` and sign in with your admin credentials. **2.2. Locate the K8s marketplace offering:** Navigate to **Provider > Offerings**. Find the Kubernetes namespace offering (the one whose UUID matches the site agent config). Open it and note: - The **components** it exposes (typically: CPU, Memory, Storage) and their units. - Whether **auto-approve** is enabled or disabled. If disabled (the default for Pharos), you will need to manually approve orders after submission. - The **offering users** policy — Pharos uses `anonymized` username generation and `project_slug` for account names. **2.3. Find or create a test project in Waldur:** Navigate to **Projects**. Either use an existing test project or create a new one (e.g. "E2E Sync Test"). Note the project slug — this becomes the GitLab subgroup name (e.g. `models/`). **2.4. Ensure the test project has team members:** Add at least one team member to the project with a real email address that you can verify on GitLab. This member will be synced during Step 4. ### Step 3: Create and approve an order **3.1. Submit an order:** In the Waldur UI, navigate to the test project and go to **Marketplace > Order**. Select the Kubernetes namespace offering and fill in the required limits: - **CPU** (cores) - **Memory** (GiB) - **Storage** (GiB) Submit the order. **3.2. Approve the order (if auto-approve is off):** Navigate to **Provider > Orders** (or the admin order queue). Find the pending order and approve it. **3.3. Wait for the site agent to pick it up:** The `order_process` deployment polls every 2 minutes. You can watch the logs in real time: ```bash kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-order-process ``` Look for log lines indicating: - The order was detected and processing started. - A Kubernetes namespace was created (with the sanitised project slug). - A `ResourceQuota` was applied with the requested limits. - The GitLab Root Actor was called (`POST /waldur/project_create`). - Initial team members were added (`POST /waldur/add_project_member`). **3.4. Verify Kubernetes resources:** ```bash # Find the namespace (named after the project slug) kubectl get ns -l waldur.io/managed=true # Check the ResourceQuota kubectl -n get resourcequota waldur-quota -o yaml ``` Confirm the quota limits match what you ordered (accounting for `unit_factor`: 1 CPU = 1000m, 1 GiB memory = 1024 MiB). **3.5. Verify GitLab subgroups:** ```bash # Check one of the six subgroups exists curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F" | jq '.full_path' # Verify custom attributes carry the Waldur metadata curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/custom_attributes" | jq ``` Expected custom attributes on each subgroup: - `waldur_project_uuid` - `waldur_project_slug` - `waldur_resource_uuid` - `waldur_resource_backend_id` - `waldur_storage_gib` - `waldur_limits_json` Also verify the subgroups exist under all six parent groups (not just `models`). **3.6. Verify initial team members on GitLab:** If the Waldur order included team members, they should already be added to the GitLab subgroups: ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/members" | jq '.[].username' ``` If the member's email does not match an existing GitLab account, check for pending invitations instead: ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/invitations" | jq ``` ### Step 4: Test membership sync **4.1. Add a team member in Waldur:** In the Waldur UI, go to the test project's team settings and add a new member. Use an email address for someone who has (or does not have) a GitLab account on `dev.gitlab.pharos.grnet.gr`, depending on which path you want to test. **4.2. Wait for the membership sync:** The `membership_sync` deployment polls every 2 minutes: ```bash kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-membership-sync ``` Look for log lines about adding the user to GitLab groups. **4.3. Verify the member was added to GitLab:** ```bash # If the user exists on GitLab — check membership curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/members" \ | jq '.[] | {username, access_level}' # If the user does not exist — check invitations curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/invitations" \ | jq '.[] | {email, access_level}' ``` The member should appear across all six subgroups with access level 50 (Owner), which is the default the site agent sends. **4.4. Verify the member mapping is stored:** ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/custom_attributes" \ | jq '.[] | select(.key | startswith("waldur_member_"))' ``` Each added member gets a `waldur_member_` attribute containing `{"email":"..."}`. ### Step 5: Cleanup and teardown **5.1. Remove the team member in Waldur:** In the Waldur UI, remove the member you added in Step 4 from the project team. **5.2. Wait for membership sync and verify removal:** ```bash kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-membership-sync ``` After the sync runs, verify the member is gone from GitLab: ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F/members" \ | jq '.[].username' ``` The `waldur_member_*` custom attribute should also be deleted. **5.3. Terminate the order in Waldur:** In the Waldur UI, navigate to the resource and terminate/delete it. This triggers the order deletion flow. **5.4. Wait for the site agent to process the deletion:** ```bash kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-order-process ``` Look for: - `POST /waldur/project_delete` call to the GitLab Root Actor. - Kubernetes namespace deletion. **5.5. Verify GitLab subgroups are deleted:** ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F" \ | jq '.message' # Expected: "404 Group Not Found" ``` Repeat for all six parent groups to confirm complete cleanup. **5.6. Verify the Kubernetes namespace is deleted:** ```bash kubectl get ns # Expected: "not found" ``` ## Running locally for testing When testing locally you bypass the site agent and call the GitLab Root Actor directly with `curl`, simulating the HTTP requests the site agent would make. This is useful for verifying the sync logic against a remote dev GitLab instance without needing Waldur or Kubernetes. ### Start the GitLab Root Actor ```bash cd services/gitlab-root-actor GITLAB_URL=https://dev.gitlab.pharos.grnet.gr \ GITLAB_ROOT_ACCESS_TOKEN=$(cat ~/.themelio/remote-gitlab-pat) \ GITLAB_ROOT_ACTOR_INTERNAL_TOKEN=test-internal-token \ pnpm dev ``` Verify the service is up: ```bash curl http://localhost:3000/-/health # {"status":"ok","timestamp":"..."} ``` ### Simulate the full project lifecycle Set a convenience variable for the auth header: ```bash export TOKEN=test-internal-token ``` **1. Create a project** (simulates what `order_process` does after approving an order): ```bash curl -s -X POST http://localhost:3000/waldur/project_create \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{ "project_slug": "astro-lab", "project_name": "Astro Lab", "project_uuid": "aaaaaaaa-1111-2222-3333-444444444444", "resource_uuid": "bbbbbbbb-5555-6666-7777-888888888888", "resource_backend_id": "waldur-res-001", "storage_gib": 200 }' | jq ``` Verify in GitLab: six new subgroups should appear (e.g. `models/astro-lab`, `datasets/astro-lab`, ...). Each carries custom attributes with the Waldur metadata: ```bash curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \ "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2Fastro-lab/custom_attributes" | jq ``` **2. Add a member** (simulates what `membership_sync` does when a user is added to the Waldur project team): ```bash curl -s -X POST http://localhost:3000/waldur/add_project_member \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{ "project_slug": "astro-lab", "offering_username": "researcher1", "email": "researcher1@university.edu", "access_level": 30 }' | jq ``` If the email matches an existing GitLab account, the user appears as Developer (30) in all six subgroups. Otherwise, pending invitations are created. **3. Promote a member** (re-add with a higher access level): ```bash curl -s -X POST http://localhost:3000/waldur/add_project_member \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{ "project_slug": "astro-lab", "offering_username": "researcher1", "email": "researcher1@university.edu", "access_level": 40 }' | jq ``` **4. Remove a member** (simulates `membership_sync` removal): ```bash curl -s -X POST http://localhost:3000/waldur/delete_project_member \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{ "project_slug": "astro-lab", "offering_username": "researcher1", "email": "researcher1@university.edu" }' | jq ``` **5. Decommission the project** (simulates order deletion): ```bash curl -s -X POST http://localhost:3000/waldur/project_delete \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{"project_slug": "astro-lab"}' | jq ``` All six subgroups and their contents are permanently deleted. ### Testing idempotency Both `project_create` and `project_delete` are idempotent: - Re-running `project_create` with the same slug reuses existing groups and overwrites metadata. - Re-running `project_delete` for an already-deleted slug succeeds silently (missing groups are skipped). ### Testing validation and auth ```bash # Missing auth → 401 curl -s -X POST http://localhost:3000/waldur/project_create \ -H "Content-Type: application/json" \ -d '{"project_slug":"test","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq # Wrong auth → 401 curl -s -X POST http://localhost:3000/waldur/project_create \ -H "Content-Type: application/json" \ -H "Authorization: Bearer wrong-token" \ -d '{"project_slug":"test","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq # Invalid slug (uppercase) → 422 curl -s -X POST http://localhost:3000/waldur/project_create \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{"project_slug":"INVALID","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq # Missing required fields → 422 curl -s -X POST http://localhost:3000/waldur/project_create \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{"project_slug":"test"}' | jq ``` ## GitLab Root Actor API reference All `/waldur/*` endpoints require `Content-Type: application/json` and `Authorization: Bearer `. ### `GET /-/health` No authentication. Returns `{"status":"ok","timestamp":"..."}`. ### `POST /waldur/project_create` | Field | Required | Description | | --------------------- | -------- | ------------------------------------------------------------------- | | `project_slug` | yes | 1-63 lowercase alphanumeric or hyphens; no leading/trailing hyphens | | `project_name` | no | Human-readable name (defaults to slug) | | `project_uuid` | yes | Waldur project UUID | | `resource_uuid` | yes | Waldur resource UUID | | `resource_backend_id` | yes | Waldur resource backend ID | | `storage_gib` | no | Storage quota in GiB (can be `null`) | | `limits` | no | Additional Waldur limits (stored as JSON) | ### `POST /waldur/project_delete` | Field | Required | Description | | -------------- | -------- | ----------------- | | `project_slug` | yes | Project to delete | ### `POST /waldur/add_project_member` | Field | Required | Description | | ------------------- | -------- | ------------------------------------------------------------------------------------------- | | `project_slug` | yes | Target project | | `offering_username` | yes | Waldur offering username (mapping key) | | `email` | yes | User email | | `access_level` | no | GitLab access level: 10=Guest, 20=Reporter, 30=Developer, 40=Maintainer, 50=Owner (default) | ### `POST /waldur/delete_project_member` | Field | Required | Description | | ------------------- | -------- | --------------------------------------- | | `project_slug` | yes | Target project | | `offering_username` | yes | Waldur offering username | | `email` | no | Falls back to stored mapping if omitted | ## Error handling | HTTP status | Meaning | Example cause | | ----------- | ---------------- | -------------------------------------------- | | 401 | Unauthorized | Missing or wrong `Authorization` header | | 422 | Validation error | Invalid slug, missing required fields | | 500 | Internal error | GitLab API failure, missing top-level groups | ### Common errors **`GitLab project type group was not found`** — A top-level group is missing on GitLab. Create it before retrying. **`Unauthorized Waldur root actor request`** — The `Authorization` header does not match the configured `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN`. Uses constant-time comparison to prevent timing attacks. **`project_slug must be 1-63 lowercase alphanumeric characters or hyphens, and cannot start or end with a hyphen`** — Fix the slug in Waldur and retry. **`Cannot invite : email is missing`** — The `add_project_member` call requires an `email` field. **`Waldur GitLab subgroup was not found`** — The project has not been created yet. Run `project_create` first. ## Observability The GitLab Root Actor logs structured JSON (Pino/Fastify) to stdout. Every operation logs `project_slug`, `resource_backend_id`, `resource_uuid`, `offering_username`, `email`, `group_path`, and `group_id` as applicable. ```bash # Filter logs for a specific project when running locally pnpm dev 2>&1 | jq 'select(.project_slug == "astro-lab")' ``` The Waldur K8s Site Agent logs to stdout as well. In production, both services' logs are collected by the cluster's log aggregation pipeline.
Page source Source snapshot · Revision `f928834c8`.