#ADR-0003: Waldur Proposal Workflow

Status: Proposed

Date: 2026-05-07

Deciders: TBD

Relates to: ADR-0000 (Friends & Family MVP) — "Proposal Workflow" exploration item; Waldur Installation & Management (future ADR) — functional requirements FR-PRO-01 through FR-PRO-04.

#Context

The Themelio platform currently has a 5-step project wizard (/projects/new) that collects project metadata, PI/organization details, scientific rationale, and resource requests. However, the entire proposal lifecycle is client-side only — drafts and submissions are stored in localStorage via projectsStore.ts, and no data reaches the backend. The form data goes nowhere.

The Friends & Family MVP (ADR-0000) explicitly defers the proposal workflow as a post-MVP item:

The resource request process will be connected to Waldur's proposal management system, replacing the current browser-only prototype.

Meanwhile, Waldur provides a full Proposals API (/api/proposal-proposals/) with a formal review workflow (Draft → Submitted → In Review → Accepted/Rejected), and the existing waldur-k8s-site-agent already provisions Kubernetes namespaces and GitLab groups when Waldur orders are approved. However, this integration only works in the Waldur → Themelio direction (Waldur pushes approved orders to the site-agent). There is no Themelio → Waldur path for users to submit proposals.

Integrating with the Waldur Proposals API would:

1. Make proposals persistent and shared — admins can review them; users see status across devices.
2. Connect proposals to the existing Waldur order and provisioning pipeline — approved proposals can trigger resource allocation via the site agents already deployed (waldur-k8s-site-agent, GitLab sync via gitlab-root-actor).
3. Preserve the rejection/resubmission UX that Themelio requires but Waldur's project model does not natively support.

#Key Constraints

1. Waldur proposals are terminal on rejection — the API does not support resubmission of a rejected proposal. A new proposal must be created.
2. Users should not context-switch to Waldur — the ProjectWizard should be the single interface for proposal submission.
3. Waldur proposals have no generic attributes field — the Proposal model uses structured fields only (no JSONField). Themelio-specific fields that don't map to Waldur's schema need an alternative store.

#Current architecture gaps

• No backend for proposals. The web service has no API route or service for proposal persistence.
• No Waldur API client in the web layer. The Waldur API token is a privileged credential; it currently lives only in the site-agent deployment configs.
• Field mismatch. The Themelio form captures 7 scientific rationale fields, rich resource breakdowns (HPC partitions, cloud quotas, platform services), and team member details that don't map 1:1 to Waldur's proposal schema.
• Rejection semantics differ. Waldur hides rejected projects from the requesting user. Themelio must show rejected proposals and allow editing/resubmission.

#Decision

#1. Introduce a dedicated waldur-root-actor service

A new Fastify service (following the pattern of gitlab-root-actor and k8s-root-actor) that acts as the sole intermediary between Themelio and the Waldur Proposals API. It holds the Waldur API token and exposes an internal REST API consumed by the web service.

┌──────────────┐      ┌─────────────────────┐      ┌──────────────────┐
│  Themelio Web │─────►│  waldur-root-actor   │─────►│  Waldur API      │
│  (Next.js)   │ HTTP │  (Fastify, port 3003)│ HTTP │  (dev/staging)   │
└──────────────┘      └─────────────────────┘      └──────────────────┘

Why a dedicated service:

• Credential isolation. The Waldur API token stays out of the web service, matching the existing pattern where gitlab-root-actor holds the GitLab root token and k8s-root-actor holds K8s credentials.
• Single responsibility. Proposal lifecycle logic (field mapping, resubmission lineage, status polling) lives in one place.
• Testability. The service can be tested against Waldur's dev instance independently of the web frontend.

#2. Use Waldur as the primary draft store; GitLab group for overflow fields only

On first draft save, waldur-root-actor creates a Waldur proposal in draft state (POST /api/proposal-proposals/). The Waldur Proposals API natively supports drafts — proposals start in DRAFT state and can be updated via POST /api/proposal-proposals/{uuid}/update_project_details/ before submission. This makes Waldur the primary persistence layer from the first save, not just after submission.

What lives in Waldur (structured fields):

What lives in GitLab custom attributes (overflow — fields with no Waldur equivalent):

A GitLab group is still pre-created alongside the Waldur draft to store these overflow fields. However, the group is much lighter — it holds only Themelio-specific metadata (~500 bytes total) rather than the full proposal data blob (~2-5 KB).

When Waldur approves the proposal and the site-agent provisions resources, it creates subgroups (models/, datasets/, spaces/, etc.) inside this pre-existing group — no data migration needed. The draft group becomes the project group.

This approach means proposals are server-side persisted from the first save, visible across devices, and survive browser state loss. The localStorage draft in the wizard is retained only as a transient editing buffer.

#3. On form submission, finalize and submit the existing Waldur draft

The Waldur proposal already exists in DRAFT state (created in Section 2). When the user clicks "Submit for Review", the web service calls waldur-root-actor, which:

1. Updates the draft with final field values (POST /api/proposal-proposals/{uuid}/update_project_details/)
2. Attaches resource requests (POST /api/proposal-proposals/{uuid}/resources/)
3. Submits checklist answers for scientific rationale fields (POST /api/proposal-proposals/{uuid}/submit_answers/)
4. Adds team members as proposal users (POST /api/proposal-proposals/{uuid}/add_user/)
5. Submits the proposal (POST /api/proposal-proposals/{uuid}/submit/)
6. Updates the GitLab group overflow custom attributes if changed

#4. Calls and Rounds are admin-provisioned in Waldur

The admin creates Calls for Proposals and Rounds directly in the Waldur UI (e.g. "Daedalus 2026 H2 — National Call"). Themelio fetches available open rounds via GET /api/call-rounds/ and presents them in the project wizard (Step 1 or as a pre-step).

Themelio does not create or manage Calls/Rounds programmatically. This matches the existing pattern where offerings and service providers are set up by operators, not through Themelio.

#5. Store scientific rationale in Waldur checklists

The Themelio form's scientific rationale fields (7 structured fields) do not have direct Waldur equivalents. The recommended approach is to use Waldur's checklist system:

• A checklist template is pre-configured in Waldur (admin-provisioned alongside the Call) with questions mapping to each scientific rationale field.
• On submission, waldur-root-actor calls POST /api/proposal-proposals/{uuid}/submit_answers/ with the structured responses.
• The checklist approach preserves data in a queryable, reviewable format within Waldur's UI.

#6. Map resource requests to Waldur proposal resources

Each enabled resource type in the form becomes a separate proposal resource linked to a pre-configured offering:

HPC Resources → SLURM Offering

{
  "requested_offering": "<slurm-offering-uuid>",
  "limits": {
    "node_hours": 250000
  },
  "attributes": {
    "partition": "gpu-h100"
  },
  "description": "HPC resource request"
}

Cloud Resources → Kubernetes Offering

{
  "requested_offering": "<k8s-offering-uuid>",
  "limits": {
    "cpu": 32,
    "ram": 128,
    "storage": 2000
  },
  "description": "Cloud resource request — <cloudJustification>"
}

Platform services (Kubeflow, MLflow, Open OnDemand, SSH) are stored as boolean attributes on the Cloud resource request or as a separate platform offering resource, depending on how the Waldur offerings are structured by the admin.

#7. Field mapping: Themelio form → Waldur Proposals API

#Proposal creation (POST /api/proposal-proposals/)

#Organization and PI (add_user + update_project_details)

#Scientific rationale → Checklist answers

See Section 5 above.

#Resources → Proposal resources

See Section 6 above.

#Fields with no Waldur equivalent (stored in GitLab group overflow)

#8. Handle rejection and resubmission via new-proposal lineage

Waldur does not support resubmitting a rejected proposal. To implement the Themelio requirement (user sees rejection, edits, resubmits), the following approach is used:

1. On rejection: waldur-root-actor detects the state change (via polling or on-demand fetch). The proposal's REJECTED state is read directly from Waldur; Themelio presents it as "Needs Revision".
2. User edits and resubmits: The web frontend pre-fills the form from the rejected Waldur proposal's structured fields (via GET /api/proposal-proposals/{uuid}/) plus the overflow data from GitLab custom attributes. On resubmission, waldur-root-actor creates a new Waldur proposal in DRAFT state, with a description noting the lineage (e.g., "Resubmission of proposal {old-uuid}, sequence #2").
3. Lineage tracking: The old proposal UUID is appended to waldur_proposal_history on the GitLab group. The waldur_proposal_uuid attribute is updated to the new proposal UUID. The user sees a continuous history in Themelio, even though Waldur treats each as a separate proposal.
4. Admin visibility: The lineage note in the proposal description provides context to reviewers in Waldur that this is a revised submission.

From the user's perspective, this is a single continuous flow. From Waldur's perspective, each submission is a fresh proposal.

#9. Poll for Waldur state changes

Waldur uses STOMP message queues for event notifications, but proposal state change events are not confirmed to be covered. Since proposal review is a human-timescale process (hours to days), polling GET /api/proposal-proposals/?state=... every 5-10 minutes is sufficient and simpler than maintaining a persistent WebSocket/STOMP connection.

Load estimate: Even with 500 active proposals, a single poll request every 5 minutes produces ~0.003 RPS against the Waldur API — negligible. The poller fetches only proposals with state changes since the last poll (filtered by modified timestamp), keeping response payloads small.

Two sync mechanisms are used:

• On-demand fetch: When a user views their proposals list or a proposal detail page, the web service calls waldur-root-actor, which queries Waldur directly for current status. Since Waldur is the source of truth, no local status cache is needed.
• Background polling: waldur-root-actor periodically polls for status changes to trigger side effects (e.g., sending notifications on approval/rejection). This can start as a simple interval and be upgraded to STOMP subscription later if Waldur confirms proposal events are published.

#10. Sequence: proposal submission

#11. Error handling and partial failure

The submission sequence (Section 10) involves multiple Waldur API calls. Since the Waldur draft already exists, failures during submission are recoverable:

1. Draft update fails (update_project_details): The draft exists in Waldur with stale data. Return an error; the user can retry. No data is lost.
2. Resource attachment or checklist submission fails: The Waldur proposal exists but is incomplete. On retry (POST /proposals/:id/submit), waldur-root-actor queries the proposal's current resources and checklist state, skips already-completed steps, and retries only the failed ones.
3. Final submit call fails (POST .../submit/): All data is attached but the proposal remains in Waldur DRAFT state. The service returns an error; the user can retry submission without data loss.
4. GitLab overflow attribute update fails after successful Waldur submission: The overflow metadata is stale but all primary proposal data is safely in Waldur. The next on-demand fetch will reconcile.
5. Initial draft creation fails (POST /api/proposal-proposals/): No Waldur state exists yet. Return an error; the wizard retains the form data in localStorage as a fallback. The user can retry saving the draft.

The general principle: Waldur is the source of truth for all proposal data; GitLab custom attributes store only overflow fields. Any divergence between the two is resolved by re-reading Waldur state.

#12. Sequence: proposal approval → resource provisioning

#13. waldur-root-actor internal API

The service exposes the following endpoints, consumed only by the Themelio web service:

All endpoints return 401 if the internal service token is missing or invalid.

#14. Authentication and authorization

Following the pattern established by gitlab-root-actor, the waldur-root-actor uses a shared internal token for service-to-service authentication:

• The web service includes an Authorization: Bearer <token> header on every request to waldur-root-actor.
• The token is configured via the WALDUR_ROOT_ACTOR_INTERNAL_TOKEN environment variable (same pattern as GITLAB_ROOT_ACTOR_INTERNAL_TOKEN).
• A requireAuth() guard validates the token on every route, returning 401 if missing or invalid.

User identity is passed by the web service as a request parameter (e.g., user email or GitLab user ID). The web service is trusted to provide the correct identity because the internal token proves it is the legitimate caller. waldur-root-actor does not independently verify end-user sessions.

Authorization is scoped per endpoint:

• GET /proposals and GET /proposals/:id filter results to proposals owned by the requesting user (matched via created_by on the Waldur proposal).
• PUT /proposals/:id/draft, POST /proposals/:id/submit, and POST /proposals/:id/resubmit verify that the requesting user is the proposal owner before proceeding.
• GET /rounds is accessible to any authenticated caller (no per-user filtering).

#15. Configuration

The service requires the following environment variables:

#Data Flow

User fills wizard → POST /proposals/draft (waldur-root-actor)
  → Creates Waldur proposal in DRAFT state (POST /api/proposal-proposals/)
  → Creates GitLab group for overflow fields (acronym, tags, platform services, etc.)
  → Returns draft ID (Waldur UUID) to UI

User saves progress → PUT /proposals/{id}/draft (waldur-root-actor)
  → Updates Waldur draft (POST .../update_project_details/)
  → Updates GitLab overflow attributes if changed

User clicks "Submit for Review"
  → POST /proposals/{id}/submit (waldur-root-actor)
  → Finalizes Waldur draft (POST .../update_project_details/)
    → Attaches resource requests (POST .../resources/) for HPC and K8s offerings
    → Submits checklist answers (POST .../submit_answers/) for scientific rationale
    → Adds team members (POST .../add_user/)
    → Submits proposal (POST .../submit/)

User views proposal status → GET /proposals/{id} (waldur-root-actor)
  → Reads state directly from Waldur (GET /api/proposal-proposals/{uuid}/)
  → Reads overflow fields from GitLab group custom attributes
  → Returns merged view

Poller detects state change in Waldur:
  → Approved: site-agent provisions subgroups inside existing GitLab group
  → Rejected: user sees "Needs Revision", edits, resubmits → new Waldur proposal, same GitLab group

#Open Items

#Blocking (must be resolved before implementation)

• round_uuid resolution: Creating a Waldur proposal (even in draft) requires a round UUID. Either configure a default round via env var, or dynamically fetch open rounds via GET /api/call-rounds/ and present a selection in the wizard. The choice affects the wizard UX (Step 1 or pre-step). This is now blocking for draft creation, not just submission.
• Offering UUIDs: HPC and K8s offering UUIDs must be configured (WALDUR_HPC_OFFERING_UUID, WALDUR_K8S_OFFERING_UUID) before any resource attachment can work.
• User UUID mapping: Adding team members requires Waldur user UUIDs. Options: (a) lookup by email via GET /api/users/?email=..., or (b) store Waldur UUID during OIDC login. Option (a) is simpler but adds a Waldur API call per team member; option (b) requires changes to the auth flow.
• GitLab group naming: See the warning note in Section 2 — no slug/path field exists in the wizard today.
• update_project_details field coverage: Confirm which fields the update_project_details action accepts. If it only covers a subset of proposal fields, some data may need to be set at creation time only. To be validated against the Waldur dev instance.

#Non-blocking (can be resolved during or after implementation)

• Checklist template sync: The checklist template is configured per-call in Waldur. Themelio either fetches and renders it dynamically, or maintains a parallel form. Dynamic fetching is recommended to avoid drift.
• Site-agent coordination: The site-agent currently creates the GitLab project group. With pre-created groups, the site-agent must detect the existing group and create subgroups inside it rather than creating a new top-level group.
• Orphaned drafts: Creating Waldur proposals in draft state means abandoned drafts if a user never submits. Waldur drafts may need periodic cleanup (depends on whether Waldur handles this natively). The corresponding GitLab overflow groups should be cleaned up in tandem.

#Consequences

#Positive

• Proposals become real. The current localStorage-only system is replaced with persistent, server-side storage — drafts and submitted proposals in Waldur, overflow fields in GitLab custom attributes.
• Waldur as single source of truth. All primary proposal data (title, summary, resources, rationale, status) lives in Waldur from the first save. GitLab stores only Themelio-specific overflow (~500 bytes), not a full data mirror.
• No new database or infrastructure. Waldur's native draft support and GitLab custom attributes cover all persistence needs.
• End-to-end automation. An approved proposal automatically triggers the existing site-agent pipeline (K8s namespace creation, GitLab group provisioning, member sync) without manual intervention.
• Admin review in Waldur. Admins can review proposals, checklists, and resource requests in Waldur's native UI without depending on Themelio for the review workflow.
• Credential isolation. The Waldur API token is confined to a single service, following established patterns.
• Full proposal history. Resubmission lineage is preserved in GitLab overflow attributes, giving users a continuous view even though Waldur treats each submission as separate.

#Negative

• New service to deploy and maintain. Adds operational overhead (one more Fastify service, health checks, deployment manifests).
• Checklist template coupling. The scientific rationale mapping depends on a pre-configured checklist template in Waldur. Template changes require coordinated updates in both Waldur and waldur-root-actor.
• Resubmission complexity. The lineage tracking for rejected/resubmitted proposals adds state (the history array) that lives in GitLab, outside Waldur. This is minimal (~100 bytes per submission) but still a secondary store.
• Orphaned drafts. Creating Waldur proposals in draft state means abandoned drafts accumulate. The corresponding GitLab groups for overflow also become orphaned. Needs periodic cleanup in both systems.
• Dual-store coordination. Although GitLab stores only overflow fields, waldur-root-actor must keep the GitLab group and Waldur proposal in sync (e.g., updating waldur_proposal_uuid on resubmission). Failure to sync results in stale overflow data, though primary proposal data remains correct in Waldur.

#Risks

• Waldur API stability. The Proposals API is newer than the core Marketplace API. Breaking changes could require service updates.
• Round availability. If no open round exists when a user tries to submit, the form must handle this gracefully (disable submit, show message).
• Checklist template drift. If the Waldur checklist template and the Themelio form diverge, submissions could fail silently or lose data.

#Out of Scope

• Waldur Call/Round management from Themelio. Calls and Rounds are admin-provisioned directly in Waldur.
• Review workflow in Themelio. Proposal review and approval happen in Waldur's UI. Themelio only displays status.
• Document attachments. The Waldur API supports PDF attachments (attach_document). This can be added later if the scientific case requires uploaded documents.
• STOMP-based status sync. Initial implementation uses polling + on-demand fetch. STOMP subscription can be added for real-time notifications if Waldur confirms proposal events are published.