# Waldur GitLab Sync

When a [Pharos AI Factory](../../terms/pharos-ai-factory) administrator
approves a resource allocation in [Waldur](../../terms/waldur), the platform
must provision the corresponding infrastructure: a Kubernetes namespace with
resource quotas **and** a matching set of GitLab groups so that project
members can push [models](../../terms/model), [datasets](../../terms/dataset),
[spaces](../../terms/space), repos, recipes, and upskillings.

This page documents how that synchronisation works end-to-end, how it is
deployed in staging/production, and how to test it locally.

## Architecture overview

The sync pipeline involves three systems connected by polling and REST calls:

```mermaid
flowchart TD
    A["Waldur UI - Marketplace"]
    B["Waldur K8s Site Agent"]
    C["GitLab Root Actor"]
    D["GitLab"]
    E["Kubernetes"]

    A -- "Waldur REST API\n(polled every 2 min)" --> B
    B -- "POST /waldur/*\n(Bearer token auth)" --> C
    C -- "GitLab REST API v4\n(root PAT)" --> D
    B -- "K8s API\n(ServiceAccount)" --> E

    subgraph site_agent ["Site Agent Deployments"]
        direction LR
        B1["order_process\nprovisions resources"]
        B2["membership_sync\nsyncs team members"]
        B3["report\nusage reporting"]
    end

    B --- site_agent
```

**No webhooks are used.** The site agent polls the Waldur API at fixed
intervals and drives all downstream actions synchronously.

### What happens when an admin approves a project

1. A Pharos admin approves an order in the Waldur marketplace UI.
2. The **order_process** deployment polls the Waldur API (every 2 minutes)
   and picks up the approved order.
3. The site agent creates a **Kubernetes namespace** with a `ResourceQuota`
   matching the Waldur limits (CPU, memory, storage) and labels it with
   Waldur project/resource UUIDs.
4. The site agent calls the GitLab Root Actor's `POST /waldur/project_create`
   endpoint, passing the project slug, UUIDs, backend ID, and limits.
5. The GitLab Root Actor creates **six subgroups** on GitLab — one under
   each top-level project-type group:

   ```
   models/<project-slug>
   datasets/<project-slug>
   spaces/<project-slug>
   repos/<project-slug>
   recipes/<project-slug>
   upskillings/<project-slug>
   ```

6. Waldur metadata (project UUID, resource UUID, backend ID, storage quota,
   limits) is written to each subgroup as GitLab **custom attributes**.
7. The site agent then calls `POST /waldur/add_project_member` for each
   initial team member from the Waldur order.

### What happens when team membership changes

1. The **membership_sync** deployment polls Waldur (every 2 minutes) for
   team membership changes.
2. For each added user, the site agent:
   - Creates a Kubernetes `ServiceAccount` and `RoleBinding` in the project
     namespace.
   - Calls `POST /waldur/add_project_member` with the user's email and
     offering username.
   - The GitLab Root Actor looks up the user by email. If found, the user
     is added to all six subgroups. If not, a GitLab group invitation is
     sent instead.
   - A member mapping (`offering_username` → `email`) is stored as a custom
     attribute on each subgroup, so future removal can work even without
     the email in the request.
3. For each removed user, the reverse process runs via
   `POST /waldur/delete_project_member`.

### What happens when a project is decommissioned

1. The site agent calls `POST /waldur/project_delete`.
2. The GitLab Root Actor deletes all six subgroups and their contents.
3. The site agent deletes the Kubernetes namespace.

### Usage reporting

The **report** deployment polls every 10 minutes, reads
`ResourceQuota.status.used` from each managed namespace, converts
backend units to Waldur units (using `unit_factor` from the config), and
reports totals back to Waldur.

## Production / staging deployment

### Waldur K8s Site Agent

The site agent runs in the `waldur-site-agent-cloud` namespace as three
separate Kubernetes Deployments, each running a different mode of the
`waldur_site_agent` command:

| Deployment                           | Command                                | Poll interval |
| ------------------------------------ | -------------------------------------- | ------------- |
| `waldur-agent-cloud-order-process`   | `waldur_site_agent -m order_process`   | 2 min         |
| `waldur-agent-cloud-membership-sync` | `waldur_site_agent -m membership_sync` | 2 min         |
| `waldur-agent-cloud-report`          | `waldur_site_agent -m report`          | 10 min        |

Configuration is mounted from a Kubernetes secret
(`waldur-site-agent-cloud-config`) containing a YAML file with:

```yaml
offerings:
  - name: "Kubernetes namespaces via Site Agent (cloud)"
    waldur_api_url: "<Waldur API endpoint>"
    waldur_api_token: "<Waldur API token>"
    waldur_offering_uuid: "<marketplace offering UUID>"

    backend_settings:
      gitlab_root_actor_url: "http://gitlab-root-actor.themelio.svc.cluster.local"
      gitlab_root_actor_token: "<shared secret>"
      gitlab_root_actor_timeout_seconds: 10
      quota_name: "waldur-quota"
      quota_mapping:
        cpu: ["requests.cpu", "limits.cpu"] # 1 Waldur CPU = 1000m
        memory: ["requests.memory", "limits.memory"] # 1 Waldur GiB = 1024 MiB
        storage: ["requests.storage"] # 1 Waldur GiB = 1 GiB

    backend_components:
      cpu:
        measured_unit: "Core"
        unit_factor: 1000
      memory:
        measured_unit: "GiB"
        unit_factor: 1024
      storage:
        measured_unit: "GiB"
        unit_factor: 1
```

The site agent's `ClusterRole` grants it full access to namespaces,
resource quotas, service accounts, roles, and role bindings across the
cluster.

### GitLab Root Actor

The GitLab Root Actor runs in the `themelio` namespace as a single
Deployment (`gitlab-root-actor`) behind a ClusterIP service on port 80
(internal port 3000). It is **not** exposed to the public internet.

Secrets are mounted from `gitlab-root-pat-secret`:

| Secret key                         | Maps to env var                    | Description                                  |
| ---------------------------------- | ---------------------------------- | -------------------------------------------- |
| `GITLAB_TOKEN`                     | `GITLAB_ROOT_ACCESS_TOKEN`         | GitLab root PAT with `api` scope             |
| `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN` | `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN` | Shared secret matching the site agent config |

The `GITLAB_URL` in staging points to the cluster-internal GitLab service:
`http://gitlab-webservice-default.gitlab.svc.cluster.local:8181`

### Prerequisites on GitLab

The following **top-level groups** must exist before the sync can operate:

- `models`, `datasets`, `spaces`, `repos`, `recipes`, `upskillings`

If any are missing, every sync call fails with
`GitLab project type group <name> was not found`.

## End-to-end testing on the dev environment

This section walks through testing the full pipeline — from the Waldur UI
through the site agent to GitLab — on the dev environment
(`dev.waldur.pharos.grnet.gr` + `dev.gitlab.pharos.grnet.gr`).

### Step 1: Verify the pipeline is wired up

Before creating any orders, confirm every link in the chain is healthy.

**1.1. Check site agent pods are running:**

```bash
kubectl -n waldur-site-agent-cloud get pods
```

You should see three pods (one per deployment) in `Running` state:
- `waldur-agent-cloud-order-process-*`
- `waldur-agent-cloud-membership-sync-*`
- `waldur-agent-cloud-report-*`

If any pods are in `CrashLoopBackOff` or missing, check the logs:

```bash
kubectl -n waldur-site-agent-cloud logs -l app=waldur-agent-cloud-order-process --tail=50
```

**1.2. Check the GitLab Root Actor is running:**

```bash
kubectl -n themelio get pods -l app=gitlab-root-actor
```

Verify it responds to health checks from inside the cluster:

```bash
kubectl -n themelio exec deploy/gitlab-root-actor -- \
  curl -s http://localhost:3000/-/health
```

**1.3. Verify the offering UUID is configured:**

Check that the site agent config secret has a non-empty offering UUID:

```bash
kubectl -n waldur-site-agent-cloud get secret waldur-site-agent-cloud-config \
  -o jsonpath='{.data.waldur-site-agent-config\.yaml}' | base64 -d | grep waldur_offering_uuid
```

If the UUID is empty, the site agent will poll but never pick up orders.
Populate it via the Ansible vault and re-run the playbook.

**1.4. Verify the offering exists in Waldur:**

Log into `https://dev.waldur.pharos.grnet.gr` as an admin. Navigate to
**Provider > Offerings** and confirm there is a Kubernetes offering
matching the UUID from step 1.3. Note the offering name — you will use
it when creating an order.

**1.5. Verify GitLab has the required top-level groups:**

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups?top_level_only=true&per_page=100" \
  | jq '.[].path'
```

All six must be present: `models`, `datasets`, `spaces`, `repos`,
`recipes`, `upskillings`. Create any missing ones manually in the
GitLab admin UI before proceeding.

### Step 2: Understand the Waldur side

**2.1. Log into Waldur as admin:**

Go to `https://dev.waldur.pharos.grnet.gr` and sign in with your admin
credentials.

**2.2. Locate the K8s marketplace offering:**

Navigate to **Provider > Offerings**. Find the Kubernetes namespace
offering (the one whose UUID matches the site agent config). Open it and
note:

- The **components** it exposes (typically: CPU, Memory, Storage) and
  their units.
- Whether **auto-approve** is enabled or disabled. If disabled (the
  default for Pharos), you will need to manually approve orders after
  submission.
- The **offering users** policy — Pharos uses `anonymized` username
  generation and `project_slug` for account names.

**2.3. Find or create a test project in Waldur:**

Navigate to **Projects**. Either use an existing test project or create
a new one (e.g. "E2E Sync Test"). Note the project slug — this becomes
the GitLab subgroup name (e.g. `models/<slug>`).

**2.4. Ensure the test project has team members:**

Add at least one team member to the project with a real email address
that you can verify on GitLab. This member will be synced during
Step 4.

### Step 3: Create and approve an order

**3.1. Submit an order:**

In the Waldur UI, navigate to the test project and go to
**Marketplace > Order**. Select the Kubernetes namespace offering and
fill in the required limits:

- **CPU** (cores)
- **Memory** (GiB)
- **Storage** (GiB)

Submit the order.

**3.2. Approve the order (if auto-approve is off):**

Navigate to **Provider > Orders** (or the admin order queue). Find the
pending order and approve it.

**3.3. Wait for the site agent to pick it up:**

The `order_process` deployment polls every 2 minutes. You can watch the
logs in real time:

```bash
kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-order-process
```

Look for log lines indicating:
- The order was detected and processing started.
- A Kubernetes namespace was created (with the sanitised project slug).
- A `ResourceQuota` was applied with the requested limits.
- The GitLab Root Actor was called (`POST /waldur/project_create`).
- Initial team members were added (`POST /waldur/add_project_member`).

**3.4. Verify Kubernetes resources:**

```bash
# Find the namespace (named after the project slug)
kubectl get ns -l waldur.io/managed=true

# Check the ResourceQuota
kubectl -n <namespace> get resourcequota waldur-quota -o yaml
```

Confirm the quota limits match what you ordered (accounting for
`unit_factor`: 1 CPU = 1000m, 1 GiB memory = 1024 MiB).

**3.5. Verify GitLab subgroups:**

```bash
# Check one of the six subgroups exists
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>" | jq '.full_path'

# Verify custom attributes carry the Waldur metadata
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/custom_attributes" | jq
```

Expected custom attributes on each subgroup:
- `waldur_project_uuid`
- `waldur_project_slug`
- `waldur_resource_uuid`
- `waldur_resource_backend_id`
- `waldur_storage_gib`
- `waldur_limits_json`

Also verify the subgroups exist under all six parent groups (not just
`models`).

**3.6. Verify initial team members on GitLab:**

If the Waldur order included team members, they should already be added
to the GitLab subgroups:

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/members" | jq '.[].username'
```

If the member's email does not match an existing GitLab account, check
for pending invitations instead:

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/invitations" | jq
```

### Step 4: Test membership sync

**4.1. Add a team member in Waldur:**

In the Waldur UI, go to the test project's team settings and add a new
member. Use an email address for someone who has (or does not have) a
GitLab account on `dev.gitlab.pharos.grnet.gr`, depending on which path
you want to test.

**4.2. Wait for the membership sync:**

The `membership_sync` deployment polls every 2 minutes:

```bash
kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-membership-sync
```

Look for log lines about adding the user to GitLab groups.

**4.3. Verify the member was added to GitLab:**

```bash
# If the user exists on GitLab — check membership
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/members" \
  | jq '.[] | {username, access_level}'

# If the user does not exist — check invitations
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/invitations" \
  | jq '.[] | {email, access_level}'
```

The member should appear across all six subgroups with access level 50
(Owner), which is the default the site agent sends.

**4.4. Verify the member mapping is stored:**

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/custom_attributes" \
  | jq '.[] | select(.key | startswith("waldur_member_"))'
```

Each added member gets a `waldur_member_<offering_username>` attribute
containing `{"email":"..."}`.

### Step 5: Cleanup and teardown

**5.1. Remove the team member in Waldur:**

In the Waldur UI, remove the member you added in Step 4 from the
project team.

**5.2. Wait for membership sync and verify removal:**

```bash
kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-membership-sync
```

After the sync runs, verify the member is gone from GitLab:

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>/members" \
  | jq '.[].username'
```

The `waldur_member_*` custom attribute should also be deleted.

**5.3. Terminate the order in Waldur:**

In the Waldur UI, navigate to the resource and terminate/delete it. This
triggers the order deletion flow.

**5.4. Wait for the site agent to process the deletion:**

```bash
kubectl -n waldur-site-agent-cloud logs -f -l app=waldur-agent-cloud-order-process
```

Look for:
- `POST /waldur/project_delete` call to the GitLab Root Actor.
- Kubernetes namespace deletion.

**5.5. Verify GitLab subgroups are deleted:**

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2F<project-slug>" \
  | jq '.message'
# Expected: "404 Group Not Found"
```

Repeat for all six parent groups to confirm complete cleanup.

**5.6. Verify the Kubernetes namespace is deleted:**

```bash
kubectl get ns <namespace>
# Expected: "not found"
```

## Running locally for testing

When testing locally you bypass the site agent and call the GitLab Root
Actor directly with `curl`, simulating the HTTP requests the site agent
would make. This is useful for verifying the sync logic against a remote
dev GitLab instance without needing Waldur or Kubernetes.

### Start the GitLab Root Actor

```bash
cd services/gitlab-root-actor

GITLAB_URL=https://dev.gitlab.pharos.grnet.gr \
GITLAB_ROOT_ACCESS_TOKEN=$(cat ~/.themelio/remote-gitlab-pat) \
GITLAB_ROOT_ACTOR_INTERNAL_TOKEN=test-internal-token \
pnpm dev
```

Verify the service is up:

```bash
curl http://localhost:3000/-/health
# {"status":"ok","timestamp":"..."}
```

### Simulate the full project lifecycle

Set a convenience variable for the auth header:

```bash
export TOKEN=test-internal-token
```

**1. Create a project** (simulates what `order_process` does after
approving an order):

```bash
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "project_name": "Astro Lab",
    "project_uuid": "aaaaaaaa-1111-2222-3333-444444444444",
    "resource_uuid": "bbbbbbbb-5555-6666-7777-888888888888",
    "resource_backend_id": "waldur-res-001",
    "storage_gib": 200
  }' | jq
```

Verify in GitLab: six new subgroups should appear (e.g.
`models/astro-lab`, `datasets/astro-lab`, ...). Each carries custom
attributes with the Waldur metadata:

```bash
curl -s -H "PRIVATE-TOKEN: $(cat ~/.themelio/remote-gitlab-pat)" \
  "https://dev.gitlab.pharos.grnet.gr/api/v4/groups/models%2Fastro-lab/custom_attributes" | jq
```

**2. Add a member** (simulates what `membership_sync` does when a user
is added to the Waldur project team):

```bash
curl -s -X POST http://localhost:3000/waldur/add_project_member \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "offering_username": "researcher1",
    "email": "researcher1@university.edu",
    "access_level": 30
  }' | jq
```

If the email matches an existing GitLab account, the user appears as
Developer (30) in all six subgroups. Otherwise, pending invitations are
created.

**3. Promote a member** (re-add with a higher access level):

```bash
curl -s -X POST http://localhost:3000/waldur/add_project_member \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "offering_username": "researcher1",
    "email": "researcher1@university.edu",
    "access_level": 40
  }' | jq
```

**4. Remove a member** (simulates `membership_sync` removal):

```bash
curl -s -X POST http://localhost:3000/waldur/delete_project_member \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project_slug": "astro-lab",
    "offering_username": "researcher1",
    "email": "researcher1@university.edu"
  }' | jq
```

**5. Decommission the project** (simulates order deletion):

```bash
curl -s -X POST http://localhost:3000/waldur/project_delete \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"project_slug": "astro-lab"}' | jq
```

All six subgroups and their contents are permanently deleted.

### Testing idempotency

Both `project_create` and `project_delete` are idempotent:

- Re-running `project_create` with the same slug reuses existing groups
  and overwrites metadata.
- Re-running `project_delete` for an already-deleted slug succeeds
  silently (missing groups are skipped).

### Testing validation and auth

```bash
# Missing auth → 401
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -d '{"project_slug":"test","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq

# Wrong auth → 401
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer wrong-token" \
  -d '{"project_slug":"test","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq

# Invalid slug (uppercase) → 422
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"project_slug":"INVALID","project_uuid":"u","resource_uuid":"r","resource_backend_id":"b"}' | jq

# Missing required fields → 422
curl -s -X POST http://localhost:3000/waldur/project_create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"project_slug":"test"}' | jq
```

## GitLab Root Actor API reference

All `/waldur/*` endpoints require `Content-Type: application/json` and
`Authorization: Bearer <token>`.

### `GET /-/health`

No authentication. Returns `{"status":"ok","timestamp":"..."}`.

### `POST /waldur/project_create`

| Field                 | Required | Description                                                         |
| --------------------- | -------- | ------------------------------------------------------------------- |
| `project_slug`        | yes      | 1-63 lowercase alphanumeric or hyphens; no leading/trailing hyphens |
| `project_name`        | no       | Human-readable name (defaults to slug)                              |
| `project_uuid`        | yes      | Waldur project UUID                                                 |
| `resource_uuid`       | yes      | Waldur resource UUID                                                |
| `resource_backend_id` | yes      | Waldur resource backend ID                                          |
| `storage_gib`         | no       | Storage quota in GiB (can be `null`)                                |
| `limits`              | no       | Additional Waldur limits (stored as JSON)                           |

### `POST /waldur/project_delete`

| Field          | Required | Description       |
| -------------- | -------- | ----------------- |
| `project_slug` | yes      | Project to delete |

### `POST /waldur/add_project_member`

| Field               | Required | Description                                                                                 |
| ------------------- | -------- | ------------------------------------------------------------------------------------------- |
| `project_slug`      | yes      | Target project                                                                              |
| `offering_username` | yes      | Waldur offering username (mapping key)                                                      |
| `email`             | yes      | User email                                                                                  |
| `access_level`      | no       | GitLab access level: 10=Guest, 20=Reporter, 30=Developer, 40=Maintainer, 50=Owner (default) |

### `POST /waldur/delete_project_member`

| Field               | Required | Description                             |
| ------------------- | -------- | --------------------------------------- |
| `project_slug`      | yes      | Target project                          |
| `offering_username` | yes      | Waldur offering username                |
| `email`             | no       | Falls back to stored mapping if omitted |

## Error handling

| HTTP status | Meaning          | Example cause                                |
| ----------- | ---------------- | -------------------------------------------- |
| 401         | Unauthorized     | Missing or wrong `Authorization` header      |
| 422         | Validation error | Invalid slug, missing required fields        |
| 500         | Internal error   | GitLab API failure, missing top-level groups |

### Common errors

**`GitLab project type group <name> was not found`** — A top-level group
is missing on GitLab. Create it before retrying.

**`Unauthorized Waldur root actor request`** — The `Authorization`
header does not match the configured `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN`.
Uses constant-time comparison to prevent timing attacks.

**`project_slug must be 1-63 lowercase alphanumeric characters or
hyphens, and cannot start or end with a hyphen`** — Fix the slug in
Waldur and retry.

**`Cannot invite <username>: email is missing`** — The
`add_project_member` call requires an `email` field.

**`Waldur GitLab subgroup <path> was not found`** — The project has not
been created yet. Run `project_create` first.

## Observability

The GitLab Root Actor logs structured JSON (Pino/Fastify) to stdout.
Every operation logs `project_slug`, `resource_backend_id`,
`resource_uuid`, `offering_username`, `email`, `group_path`, and
`group_id` as applicable.

```bash
# Filter logs for a specific project when running locally
pnpm dev 2>&1 | jq 'select(.project_slug == "astro-lab")'
```

The Waldur K8s Site Agent logs to stdout as well. In production, both
services' logs are collected by the cluster's log aggregation pipeline.