Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

Waldur GitLab Sync

When a Pharos AI Factory administrator approves a resource allocation in Waldur, the platform must provision the corresponding infrastructure: a Kubernetes namespace with resource quotas and a matching set of GitLab groups so that project members can push models, datasets, spaces, repos, recipes, and upskillings.

This page documents how that synchronisation works end-to-end, how it is deployed in staging/production, and how to test it locally.

Architecture overview

The sync pipeline involves three systems connected by polling and REST calls:

No webhooks are used. The site agent polls the Waldur API at fixed intervals and drives all downstream actions synchronously.

What happens when an admin approves a project

A Pharos admin approves an order in the Waldur marketplace UI.
The order_process deployment polls the Waldur API (every 2 minutes) and picks up the approved order.
The site agent creates a Kubernetes namespace with a ResourceQuota matching the Waldur limits (CPU, memory, storage) and labels it with Waldur project/resource UUIDs.
The site agent calls the GitLab Root Actor's POST /waldur/project_create endpoint, passing the project slug, UUIDs, backend ID, and limits.

The GitLab Root Actor creates six subgroups on GitLab — one under each top-level project-type group:

models/<project-slug>
datasets/<project-slug>
spaces/<project-slug>
repos/<project-slug>
recipes/<project-slug>
upskillings/<project-slug>

Waldur metadata (project UUID, resource UUID, backend ID, storage quota, limits) is written to each subgroup as GitLab custom attributes.
The site agent then calls POST /waldur/add_project_member for each initial team member from the Waldur order.

What happens when team membership changes

The membership_sync deployment polls Waldur (every 2 minutes) for team membership changes.
For each added user, the site agent:
- Creates a Kubernetes ServiceAccount and RoleBinding in the project namespace.
- Calls POST /waldur/add_project_member with the user's email and offering username.
- The GitLab Root Actor looks up the user by email. If found, the user is added to all six subgroups. If not, a GitLab group invitation is sent instead.
- A member mapping (offering_username → email) is stored as a custom attribute on each subgroup, so future removal can work even without the email in the request.
For each removed user, the reverse process runs via POST /waldur/delete_project_member.

What happens when a project is decommissioned

The site agent calls POST /waldur/project_delete.
The GitLab Root Actor deletes all six subgroups and their contents.
The site agent deletes the Kubernetes namespace.

Usage reporting

The report deployment polls every 10 minutes, reads ResourceQuota.status.used from each managed namespace, converts backend units to Waldur units (using unit_factor from the config), and reports totals back to Waldur.

Production / staging deployment

Waldur K8s Site Agent

The site agent runs in the waldur-site-agent-cloud namespace as three separate Kubernetes Deployments, each running a different mode of the waldur_site_agent command:

Deployment	Command	Poll interval
`waldur-agent-cloud-order-process`	`waldur_site_agent -m order_process`	2 min
`waldur-agent-cloud-membership-sync`	`waldur_site_agent -m membership_sync`	2 min
`waldur-agent-cloud-report`	`waldur_site_agent -m report`	10 min

Configuration is mounted from a Kubernetes secret (waldur-site-agent-cloud-config) containing a YAML file with:

offerings:
  - name: "Kubernetes namespaces via Site Agent (cloud)"
    waldur_api_url: "<Waldur API endpoint>"
    waldur_api_token: "<Waldur API token>"
    waldur_offering_uuid: "<marketplace offering UUID>"

    backend_settings:
      gitlab_root_actor_url: "http://gitlab-root-actor.themelio.svc.cluster.local"
      gitlab_root_actor_token: "<shared secret>"
      gitlab_root_actor_timeout_seconds: 10
      quota_name: "waldur-quota"
      quota_mapping:
        cpu: ["requests.cpu", "limits.cpu"] # 1 Waldur CPU = 1000m
        memory: ["requests.memory", "limits.memory"] # 1 Waldur GiB = 1024 MiB
        storage: ["requests.storage"] # 1 Waldur GiB = 1 GiB

    backend_components:
      cpu:
        measured_unit: "Core"
        unit_factor: 1000
      memory:
        measured_unit: "GiB"
        unit_factor: 1024
      storage:
        measured_unit: "GiB"
        unit_factor: 1

The site agent's ClusterRole grants it full access to namespaces, resource quotas, service accounts, roles, and role bindings across the cluster.

GitLab Root Actor

The GitLab Root Actor runs in the themelio namespace as a single Deployment (gitlab-root-actor) behind a ClusterIP service on port 80 (internal port 3000). It is not exposed to the public internet.

Secrets are mounted from gitlab-root-pat-secret:

Secret key	Maps to env var	Description
`GITLAB_TOKEN`	`GITLAB_ROOT_ACCESS_TOKEN`	GitLab root PAT with `api` scope
`GITLAB_ROOT_ACTOR_INTERNAL_TOKEN`	`GITLAB_ROOT_ACTOR_INTERNAL_TOKEN`	Shared secret matching the site agent config

The GITLAB_URL in staging points to the cluster-internal GitLab service: http://gitlab-webservice-default.gitlab.svc.cluster.local:8181

Prerequisites on GitLab

The following top-level groups must exist before the sync can operate:

models, datasets, spaces, repos, recipes, upskillings

If any are missing, every sync call fails with GitLab project type group <name> was not found.

End-to-end testing on the dev environment

This section walks through testing the full pipeline — from the Waldur UI through the site agent to GitLab — on the dev environment (dev.waldur.pharos.grnet.gr + dev.gitlab.pharos.grnet.gr).

Step 1: Verify the pipeline is wired up

Before creating any orders, confirm every link in the chain is healthy.

1.1. Check site agent pods are running:

kubectl -n waldur-site-agent-cloud get pods

You should see three pods (one per deployment) in Running state:

waldur-agent-cloud-order-process-*
waldur-agent-cloud-membership-sync-*
waldur-agent-cloud-report-*

If any pods are in CrashLoopBackOff or missing, check the logs:

kubectl -n waldur-site-agent-cloud logs -l app=waldur-agent-cloud-order-process --tail=50

1.2. Check the GitLab Root Actor is running:

kubectl -n themelio get pods -l app=gitlab-root-actor

Verify it responds to health checks from inside the cluster:

kubectl -n themelio exec deploy/gitlab-root-actor -- \
  curl -s http://localhost:3000/-/health

1.3. Verify the offering UUID is configured:

Check that the site agent config secret has a non-empty offering UUID:

kubectl -n waldur-site-agent-cloud get secret waldur-site-agent-cloud-config \
  -o jsonpath='{.data.waldur-site-agent-config\.yaml}' | base64 -d | grep waldur_offering_uuid

If the UUID is empty, the site agent will poll but never pick up orders. Populate it via the Ansible vault and re-run the playbook.

1.4. Verify the offering exists in Waldur:

Log into https://dev.waldur.pharos.grnet.gr as an admin. Navigate to Provider > Offerings and confirm there is a Kubernetes offering matching the UUID from step 1.3. Note the offering name — you will use it when creating an order.

1.5. Verify GitLab has the required top-level groups:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups?top_level_only=true&per_page=100" \
  | jq '.[].path'

All six must be present: models, datasets, spaces, repos, recipes, upskillings. Create any missing ones manually in the GitLab admin UI before proceeding.

Step 2: Understand the Waldur side

2.1. Log into Waldur as admin:

Go to https://dev.waldur.pharos.grnet.gr and sign in with your admin credentials.

2.2. Locate the K8s marketplace offering:

Navigate to Provider > Offerings. Find the Kubernetes namespace offering (the one whose UUID matches the site agent config). Open it and note:

The components it exposes (typically: CPU, Memory, Storage) and their units.
Whether auto-approve is enabled or disabled. If disabled (the default for Pharos), you will need to manually approve orders after submission.
The offering users policy — Pharos uses anonymized username generation and project_slug for account names.

2.3. Find or create a test project in Waldur:

Navigate to Projects. Either use an existing test project or create a new one (e.g. "E2E Sync Test"). Note the project slug — this becomes the GitLab subgroup name (e.g. models/<slug>).

2.4. Ensure the test project has team members:

Add at least one team member to the project with a real email address that you can verify on GitLab. This member will be synced during Step 4.

Step 3: Create and approve an order

3.1. Submit an order:

In the Waldur UI, navigate to the test project and go to Marketplace > Order. Select the Kubernetes namespace offering and fill in the required limits:

CPU (cores)
Memory (GiB)
Storage (GiB)

Submit the order.

3.2. Approve the order (if auto-approve is off):

Navigate to Provider > Orders (or the admin order queue). Find the pending order and approve it.

3.3. Wait for the site agent to pick it up:

The order_process deployment polls every 2 minutes. You can watch the logs in real time:

kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-order-process

Look for log lines indicating:

The order was detected and processing started.
A Kubernetes namespace was created (with the sanitised project slug).
A ResourceQuota was applied with the requested limits.
The GitLab Root Actor was called (POST /waldur/project_create).
Initial team members were added (POST /waldur/add_project_member).

3.4. Verify Kubernetes resources:

# Find the namespace (named after the project slug)
kubectl get ns -l waldur.io/managed=true

# Check the ResourceQuota
kubectl -n <namespace> get resourcequota waldur-quota -o yaml

Confirm the quota limits match what you ordered (accounting for unit_factor: 1 CPU = 1000m, 1 GiB memory = 1024 MiB).

3.5. Verify GitLab subgroups:

# Check one of the six subgroups exists
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>" | jq '.full_path'

# Verify custom attributes carry the Waldur metadata
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/custom_attributes" | jq

Expected custom attributes on each subgroup:

waldur_project_uuid
waldur_project_slug
waldur_resource_uuid
waldur_resource_backend_id
waldur_storage_gib
waldur_limits_json

Also verify the subgroups exist under all six parent groups (not just models).

3.6. Verify initial team members on GitLab:

If the Waldur order included team members, they should already be added to the GitLab subgroups:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/members" | jq '.[].username'

If the member's email does not match an existing GitLab account, check for pending invitations instead:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/invitations" | jq

Step 4: Test membership sync

4.1. Add a team member in Waldur:

In the Waldur UI, go to the test project's team settings and add a new member. Use an email address for someone who has (or does not have) a GitLab account on dev.gitlab.pharos.grnet.gr, depending on which path you want to test.

4.2. Wait for the membership sync:

The membership_sync deployment polls every 2 minutes:

kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-membership-sync

Look for log lines about adding the user to GitLab groups.

4.3. Verify the member was added to GitLab:

# If the user exists on GitLab — check membership
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/members" \
  | jq '.[] | {username, access_level}'

# If the user does not exist — check invitations
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/invitations" \
  | jq '.[] | {email, access_level}'

The member should appear across all six subgroups with access level 50 (Owner), which is the default the site agent sends.

4.4. Verify the member mapping is stored:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/custom_attributes" \
  | jq '.[] | select(.key | startswith("waldur_member_"))'

Each added member gets a waldur_member_<offering_username> attribute containing {"email":"..."}.

Step 5: Cleanup and teardown

5.1. Remove the team member in Waldur:

In the Waldur UI, remove the member you added in Step 4 from the project team.

5.2. Wait for membership sync and verify removal:

kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-membership-sync

After the sync runs, verify the member is gone from GitLab:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/members" \
  | jq '.[].username'

The waldur_member_* custom attribute should also be deleted.

5.3. Terminate the order in Waldur:

In the Waldur UI, navigate to the resource and terminate/delete it. This triggers the order deletion flow.

5.4. Wait for the site agent to process the deletion:

kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-order-process

Look for:

POST /waldur/project_delete call to the GitLab Root Actor.
Kubernetes namespace deletion.

5.5. Verify GitLab subgroups are deleted:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>" \
  | jq '.message'
# Expected: "404 Group Not Found"

Repeat for all six parent groups to confirm complete cleanup.

5.6. Verify the Kubernetes namespace is deleted:

kubectl get ns <namespace>
# Expected: "not found"

Running locally for testing

When testing locally you bypass the site agent and call the GitLab Root Actor directly with curl, simulating the HTTP requests the site agent would make. This is useful for verifying the sync logic against a remote dev GitLab instance without needing Waldur or Kubernetes.

Start the GitLab Root Actor

cd services/gitlab-root-actor

GITLAB_URL=https://dev.gitlab.pharos.grnet.gr \
GITLAB_ROOT_ACCESS_TOKEN=$(cat ~/.themelio/remote-gitlab-pat) \
GITLAB_ROOT_ACTOR_INTERNAL_TOKEN=test-internal-token \
pnpm dev

Verify the service is up:

curl http://localhost:3000/-/health
# {"status":"ok","timestamp":"..."}

Simulate the full project lifecycle

Set a convenience variable for the auth header:

export TOKEN=test-internal-token

1. Create a project (simulates what order_process does after approving an order):

curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "project_name": "Astro Lab",
    "project_uuid": "aaaaaaaa-1111-2222-3333-444444444444",
    "resource_uuid": "bbbbbbbb-5555-6666-7777-888888888888",
    "resource_backend_id": "waldur-res-001",
    "storage_gib": 200
  }' | jq

Verify in GitLab: six new subgroups should appear (e.g. models/astro-lab, datasets/astro-lab, ...). Each carries custom attributes with the Waldur metadata:

curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2Fastro-lab/custom_attributes" | jq

2. Add a member (simulates what membership_sync does when a user is added to the Waldur project team):

curl -s -X POST http://localhost:3000/waldur/add_project_member \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "offering_username": "researcher1",
    "email": "researcher1@university.edu",
    "access_level": 30
  }' | jq

If the email matches an existing GitLab account, the user appears as Developer (30) in all six subgroups. Otherwise, pending invitations are created.

3. Promote a member (re-add with a higher access level):

curl -s -X POST http://localhost:3000/waldur/add_project_member \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "offering_username": "researcher1",
    "email": "researcher1@university.edu",
    "access_level": 40
  }' | jq

4. Remove a member (simulates membership_sync removal):

curl -s -X POST http://localhost:3000/waldur/delete_project_member \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "offering_username": "researcher1",
    "email": "researcher1@university.edu"
  }' | jq

5. Decommission the project (simulates order deletion):

curl -s -X POST http://localhost:3000/waldur/project_delete \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"project_slug": "astro-lab"}' | jq

All six subgroups and their contents are permanently deleted.

Testing idempotency

Both project_create and project_delete are idempotent:

Re-running project_create with the same slug reuses existing groups and overwrites metadata.
Re-running project_delete for an already-deleted slug succeeds silently (missing groups are skipped).

Testing validation and auth

# Missing auth → 401
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -d '{"project_slug":"test","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq

# Wrong auth → 401
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer wrong-token" \
  -d '{"project_slug":"test","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq

# Invalid slug (uppercase) → 422
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"project_slug":"INVALID","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq

# Missing required fields → 422
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"project_slug":"test"}' | jq

GitLab Root Actor API reference

All /waldur/* endpoints require Content-Type: application/json and Authorization: Bearer <token>.

`GET /-/health`

No authentication. Returns {"status":"ok","timestamp":"..."}.

`POST /waldur/project_create`

Field	Required	Description
`project_slug`	yes	1-63 lowercase alphanumeric or hyphens; no leading/trailing hyphens
`project_name`	no	Human-readable name (defaults to slug)
`project_uuid`	yes	Waldur project UUID
`resource_uuid`	yes	Waldur resource UUID
`resource_backend_id`	yes	Waldur resource backend ID
`storage_gib`	no	Storage quota in GiB (can be `null`)
`limits`	no	Additional Waldur limits (stored as JSON)

`POST /waldur/project_delete`

Field	Required	Description
`project_slug`	yes	Project to delete

`POST /waldur/add_project_member`

Field	Required	Description
`project_slug`	yes	Target project
`offering_username`	yes	Waldur offering username (mapping key)
`email`	yes	User email
`access_level`	no	GitLab access level: 10=Guest, 20=Reporter, 30=Developer, 40=Maintainer, 50=Owner (default)

`POST /waldur/delete_project_member`

Field	Required	Description
`project_slug`	yes	Target project
`offering_username`	yes	Waldur offering username
`email`	no	Falls back to stored mapping if omitted

Error handling

HTTP status	Meaning	Example cause
401	Unauthorized	Missing or wrong `Authorization` header
422	Validation error	Invalid slug, missing required fields
500	Internal error	GitLab API failure, missing top-level groups

Common errors

GitLab project type group <name> was not found — A top-level group is missing on GitLab. Create it before retrying.

Unauthorized Waldur root actor request — The Authorization header does not match the configured GITLAB_ROOT_ACTOR_INTERNAL_TOKEN. Uses constant-time comparison to prevent timing attacks.

project_slug must be 1-63 lowercase alphanumeric characters or hyphens, and cannot start or end with a hyphen — Fix the slug in Waldur and retry.

Cannot invite <username>: email is missing — The add_project_member call requires an email field.

Waldur GitLab subgroup <path> was not found — The project has not been created yet. Run project_create first.

Observability

The GitLab Root Actor logs structured JSON (Pino/Fastify) to stdout. Every operation logs project_slug, resource_backend_id, resource_uuid, offering_username, email, group_path, and group_id as applicable.

# Filter logs for a specific project when running locally
pnpm dev 2>&1 | jq 'select(.project_slug == "astro-lab")'

The Waldur K8s Site Agent logs to stdout as well. In production, both services' logs are collected by the cluster's log aggregation pipeline.

Page source

Source snapshot · Revision f928834c8.

#Waldur GitLab Sync

#Architecture overview

#What happens when an admin approves a project

#What happens when team membership changes

#What happens when a project is decommissioned

#Usage reporting

#Production / staging deployment

#Waldur K8s Site Agent

#GitLab Root Actor

#Prerequisites on GitLab

#End-to-end testing on the dev environment

#Step 1: Verify the pipeline is wired up

#Step 2: Understand the Waldur side

#Step 3: Create and approve an order

#Step 4: Test membership sync

#Step 5: Cleanup and teardown

#Running locally for testing

#Start the GitLab Root Actor

#Simulate the full project lifecycle

#Testing idempotency

#Testing validation and auth

#GitLab Root Actor API reference

#GET /-/health

#POST /waldur/project_create

#POST /waldur/project_delete

#POST /waldur/add_project_member

#POST /waldur/delete_project_member

#Error handling

#Common errors

#Observability