> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # DD-0012: Job and Deployment Management System — Implementation **Related ADR:** [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system.md) **Status:** Draft **Date:** 2026-06-02 **Authors:** TBD ## Overview This document specifies the implementation of a PostgreSQL-based persistent job and deployment management system for Themelio. The system provides clear separation between short-running jobs and long-running deployments, with historical data retention, secret-based security for external updates, and explicit destroy functionality. :::note Implementation status (reconciled 2026-06-07) This design is **implemented** in `k8s-root-actor` (PostgreSQL via knex, the `jobs`/`deployments`/`*_logs` tables, the CRUD/destroy endpoints, secret hashing, and status tracking) and in the `web` service (repo- and namespace-scoped proxy endpoints, GitLab write-access checks, secret passthrough). The formal **Status** above stays `Draft` only because ADR-0012 is still `Proposed` (a DD is not marked `Implemented` ahead of its ADR being `Accepted`). Two details below were corrected to match the shipped code: - The secret header is a **single unified `x-action-secret`** header (not separate `x-job-secret` / `x-deployment-secret` headers). - `job_logs` / `deployment_logs` carry an extra **`source_status`** column that stores the granular template status (e.g. `SLURM_JOB_OUTPUT`, `JOB_FINISHED`). **Log/status update path (reconciled 2026-06-09):** the original "external system → web `PUT …/jobs/{id}` with `X-Resource-Secret`" design was superseded. The as-built path is: the pod posts to a **flat web endpoint, `POST /api/{jobs,deployments}/{id}/notifications`** (keyed by the integer DB row id), authenticated by the **`x-action-secret`** header; the web service is a secret-authenticated passthrough (no GitLab check) that proxies to k8s-root-actor **`POST /{jobs,deployments}/{id}/notifications`**, which validates the secret against `secret_hash`. The repo-scoped `PUT …/jobs/{id}` endpoints described below were never wired and have been removed — read the verb as `POST … /notifications` and the header as `x-action-secret` throughout the rest of this document. ::: ## Goals - Replace ephemeral Valkey storage with persistent PostgreSQL - Separate job and deployment lifecycles with appropriate APIs - Enable historical data retention for analytics and audit - Provide secret-based authentication for external log/status updates - Implement explicit destroy with history preservation - Maintain single database owner pattern (k8s-root-actor only) ## Design ### 1. Architecture The system consists of two main components: 1. **k8s-root-actor** - Owns PostgreSQL database, manages all job/deployment data 2. **Web service** - Provides repo-scoped API endpoints, delegates data operations to k8s-root-actor **Communication Flow:** - Web service → k8s-root-actor: HTTP requests for all CRUD operations - External systems (Kubernetes/SLURM) → Web service → k8s-root-actor: PUT log-and-status with secret - k8s-root-actor → PostgreSQL: Direct database connection ### 2. Database Schema **Jobs Table:** ```sql CREATE TABLE jobs ( id SERIAL PRIMARY KEY, job_id VARCHAR(255) UNIQUE NOT NULL, gitlab_project_id VARCHAR(512) NOT NULL, -- URL-encoded "//" namespace VARCHAR(255) NOT NULL, repo VARCHAR(255) NOT NULL, branch_name VARCHAR(255) NOT NULL, action_name VARCHAR(255) NOT NULL, tag VARCHAR(255), -- Optional tag from actions.yaml status VARCHAR(50) NOT NULL, -- pending, running, succeeded, failed, destroyed secret_hash VARCHAR(255) NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT NOW(), updated_at TIMESTAMP NOT NULL DEFAULT NOW(), started_at TIMESTAMP, completed_at TIMESTAMP, destroyed_at TIMESTAMP, config JSONB NOT NULL, result JSONB, INDEX idx_gitlab_project (gitlab_project_id), INDEX idx_namespace_tag (namespace, tag), INDEX idx_status (status), INDEX idx_created (created_at) ); ``` **Deployments Table:** ```sql CREATE TABLE deployments ( id SERIAL PRIMARY KEY, deployment_id VARCHAR(255) UNIQUE NOT NULL, gitlab_project_id VARCHAR(512) NOT NULL, -- URL-encoded "//" namespace VARCHAR(255) NOT NULL, repo VARCHAR(255) NOT NULL, branch_name VARCHAR(255) NOT NULL, action_name VARCHAR(255) NOT NULL, tag VARCHAR(255), -- Optional tag from actions.yaml status VARCHAR(50) NOT NULL, -- pending, preparing, running, stopped, destroyed secret_hash VARCHAR(255) NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT NOW(), updated_at TIMESTAMP NOT NULL DEFAULT NOW(), deployed_at TIMESTAMP, destroyed_at TIMESTAMP, url VARCHAR(255), config JSONB NOT NULL, result JSONB, INDEX idx_gitlab_project (gitlab_project_id), INDEX idx_namespace_tag (namespace, tag), INDEX idx_status (status), INDEX idx_created (created_at) ); ``` **Job Logs Table:** ```sql CREATE TABLE job_logs ( id SERIAL PRIMARY KEY, job_id VARCHAR(255) NOT NULL REFERENCES jobs(job_id), timestamp TIMESTAMP NOT NULL DEFAULT NOW(), level VARCHAR(20) NOT NULL, -- info, warning, error message TEXT NOT NULL, source_status VARCHAR(255), -- granular template status (e.g. SLURM_JOB_OUTPUT, JOB_FINISHED) INDEX idx_job_timestamp (job_id, timestamp) ); ``` **Deployment Logs Table:** ```sql CREATE TABLE deployment_logs ( id SERIAL PRIMARY KEY, deployment_id VARCHAR(255) NOT NULL REFERENCES deployments(deployment_id), timestamp TIMESTAMP NOT NULL DEFAULT NOW(), level VARCHAR(20) NOT NULL, message TEXT NOT NULL, source_status VARCHAR(255), -- granular template status INDEX idx_deployment_timestamp (deployment_id, timestamp) ); ``` ### 3. API Endpoints #### Web Service Endpoints **Jobs (scoped to repository):** | Method | Path | Description | Access Control | | ------ | -------------------------------------------------- | --------------------------------------------- | ------------------------------------- | | GET | `/api/[repoType]/[namespace]/[repo]/jobs` | List jobs for this repository (with status) | GitLab write access check | | GET | `/api/[repoType]/[namespace]/[repo]/jobs/{job_id}` | Get job details (includes status and logs) | GitLab write access check | | POST | `/api/jobs/{row_id}/notifications` | Update log and status (flat, not repo-scoped) | Secret validation (`x-action-secret`) | | DELETE | `/api/[repoType]/[namespace]/[repo]/jobs/{job_id}` | Destroy job | GitLab write access check | **Jobs (scoped to namespace):** | Method | Path | Description | Access Control | | ------ | --------------------------------- | -------------------------------------------------------------- | ------------------------- | | GET | `/api/jobs/namespace/{namespace}` | List all jobs under namespace (with status, filterable by tag) | GitLab write access check | **Deployments (scoped to repository):** | Method | Path | Description | Access Control | | ------ | ---------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------- | | GET | `/api/[repoType]/[namespace]/[repo]/deployments` | List deployments for this repository (with status) | GitLab write access check | | GET | `/api/[repoType]/[namespace]/[repo]/deployments/{deployment_id}` | Get deployment details (includes status and logs) | GitLab write access check | | POST | `/api/deployments/{row_id}/notifications` | Update log and status (flat, not repo-scoped) | Secret validation (`x-action-secret`) | | DELETE | `/api/[repoType]/[namespace]/[repo]/deployments/{deployment_id}` | Destroy deployment | GitLab write access check | **Deployments (scoped to namespace):** | Method | Path | Description | Access Control | | ------ | ---------------------------------------- | --------------------------------------------------------------------- | ------------------------- | | GET | `/api/deployments/namespace/{namespace}` | List all deployments under namespace (with status, filterable by tag) | GitLab write access check | **Note:** Job and deployment creation happens via the existing action endpoints: - `POST /api/[repoType]/[namespace]/[repo]/action/[branchName]` (web service) - `POST /:repoType/:namespace/:repoId/:branchName/action` (k8s-root-actor) #### k8s-root-actor Endpoints **Jobs:** - `GET /jobs/{job_id}` - Get job details - `GET /jobs` - List jobs (accepts `gitlab_project_id` or `namespace` query parameter, optional `tag` filter) - `POST /jobs/{row_id}/notifications` - Update job log and status (validates `x-action-secret`) - `DELETE /jobs/{job_id}` - Destroy job **Deployments:** - `GET /deployments/{deployment_id}` - Get deployment details - `GET /deployments` - List deployments (accepts `gitlab_project_id` or `namespace` query parameter, optional `tag` filter) - `POST /deployments/{row_id}/notifications` - Update deployment log and status (validates `x-action-secret`) - `DELETE /deployments/{deployment_id}` - Destroy deployment ### Existing Action Endpoint Updates The existing action endpoints must be updated to integrate with the new PostgreSQL-based job/deployment system. #### Web Service Action Endpoint **Current Location:** `services/web/src/app/api/[repoType]/[namespace]/[repo]/action/[branchName]/route.ts` **Updates Required:** 1. Keep existing POST handler for action execution 2. Update to handle response from k8s-root-actor that includes `job_id`/`deployment_id` and `secret` 3. Return these to the caller for future status/log/destroy operations **Request Flow:** ``` POST /api/[repoType]/[namespace]/[repo]/action/[branchName] Body: { actionName, parameters } ↓ Web service validates GitLab access, fetches actions.yaml ↓ Web service calls k8s-root-actor action endpoint ↓ k8s-root-actor executes action, creates job/deployment record in PostgreSQL ↓ k8s-root-actor returns { type: "job"|"deployment", id: "...", secret: "...", status: "pending" } ↓ Web service returns response to caller ``` **Response Format:** ```json { "type": "job" | "deployment", "id": "job-abc123" | "deployment-xyz789", "status": "pending", "created_at": "2026-06-02T10:00:00Z" } ``` **Note:** The secret is generated and stored by k8s-root-actor but is NOT returned in the response. The secret must be obtained and managed separately for use in PUT log-and-status requests. #### k8s-root-actor Action Endpoint **Current Location:** `services/k8s-root-actor/src/index.ts` **Current Endpoint:** `POST /:repoType/:namespace/:repoId/:branchName/action` **Updates Required:** 1. Before creating Kubernetes resources, determine if action type is `slurm-job` (job) or `k8s-deployment`/`k8s-inferenceservice` (deployment) 2. Generate secret and create job/deployment record in PostgreSQL with hashed secret 3. Store the `gitlab_project_id` (URL-encoded `//`) 4. After creating Kubernetes resources, return the job/deployment ID and secret 5. On status changes, update PostgreSQL record **Updated Request Flow:** ``` POST /:repoType/:namespace/:repoId/:branchName/action Body: { actionName, actionConfig, gitlabToken } ↓ k8s-root-actor resolves action references ↓ k8s-root-actor determines resource type (job vs deployment) ↓ k8s-root-actor generates secret, creates record in PostgreSQL: - For slurm-job: INSERT INTO jobs (job_id, gitlab_project_id, status, secret_hash, ...) - For k8s-deployment/inferenceservice: INSERT INTO deployments (...) ↓ k8s-root-actor creates Kubernetes/SLURM resources ↓ k8s-root-actor returns { type, id, status } ``` **Note:** The secret is generated and stored by k8s-root-actor but must be distributed to external systems through a separate mechanism (not via the web service response). **Integration Points:** | Current Action Type | New Resource Type | Table | ID Prefix | | ---------------------- | ----------------- | ------------- | ------------- | | `slurm-job` | Job | `jobs` | `job-` | | `k8s-deployment` | Deployment | `deployments` | `deployment-` | | `k8s-inferenceservice` | Deployment | `deployments` | `deployment-` | **Status Tracking:** k8s-root-actor must update the PostgreSQL record when: - Job starts: `status = "running"`, `started_at = NOW()` - Job completes: `status = "succeeded"` or `"failed"`, `completed_at = NOW()` - Deployment becomes ready: `status = "running"`, `deployed_at = NOW()` - Deployment fails: `status = "failed"` - Job/deployment destroyed: `status = "destroyed"`, `destroyed_at = NOW()` ### 4. Data Flow #### Creation Flow (via Action Endpoints) ``` User UI ↓ (trigger action) Web service POST /api/[repoType]/[namespace]/[repo]/action/[branchName] ↓ (call k8s-root-actor) k8s-root-actor POST /:repoType/:namespace/:repoId/:branchName/action ↓ (determine job/deployment type, generate secret, store record) PostgreSQL (store with hashed secret) ↓ (create resources) k8s-root-actor creates Kubernetes/SLURM resources ↓ (return id) Web service → User (id) ``` #### Status Update Flow (External) ``` External system (Kubernetes/SLURM) ↓ (send log/status update with secret) Web service PUT /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id} ↓ (validate GitLab access, pass secret to k8s-root-actor) k8s-root-actor PUT /jobs/{id} or /deployments/{id} ↓ (validate secret, update record) PostgreSQL (update status and add log entry) ↓ (return success) Web service → External system (success) ``` #### User Polling Flow ``` User UI ↓ (poll for status) Web service GET /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id} ↓ (validate GitLab write access) k8s-root-actor GET /jobs/{id} or /deployments/{id} ↓ (query database) PostgreSQL (return record) ↓ (return data) Web service → User (status/details) ``` #### List Jobs/Deployments Flow ``` User UI ↓ (list jobs/deployments) Web service GET /api/[repoType]/[namespace]/[repo]/jobs or /deployments ↓ (validate GitLab write access, construct gitlab_project_id) k8s-root-actor GET /jobs or /deployments?gitlab_project_id={encoded} ↓ (query database) PostgreSQL (return filtered records) ↓ (return data) Web service → User (list) ``` #### Destroy Flow ``` User UI ↓ (request destroy) Web service DELETE /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id} ↓ (validate GitLab write access) k8s-root-actor DELETE /jobs/{id} or /deployments/{id} ↓ (destroy resources) k8s-root-actor destroys Kubernetes/SLURM resources ↓ (update record) PostgreSQL (set destroyed_at, status=destroyed) ↓ (return success) Web service → User (success) ``` ### 5. Secret Management **Generation (k8s-root-actor only):** ```typescript // Generate 256-bit random secret const secret = crypto.randomBytes(32).toString('hex'); const secretHash = crypto.createHash('sha256').update(secret).digest('hex'); // Store hash in database // Secret is NOT returned to caller - must be managed separately ``` **Validation (k8s-root-actor only):** ```typescript // On POST /{kind}/{id}/notifications endpoint — a single unified header for both kinds const secret = request.headers['x-action-secret']; const secretHash = crypto.createHash('sha256').update(secret).digest('hex'); // Compare with stored hash in database ``` **Passthrough (Web service):** ```typescript // Extract secret from request const secret = request.headers['x-action-secret']; // Pass secret to k8s-root-actor without validation const response = await fetch(k8sRootActorUrl, { headers: { 'x-action-secret': secret } }); ``` **Secret Distribution:** The secret is generated by k8s-root-actor but must be distributed to external systems (Kubernetes/SLURM) through a separate mechanism (not specified in this document). External systems use this secret to authenticate PUT log-and-status requests. ### 6. GitLab Project ID Format The `gitlab_project_id` is a URL-encoded string: `//` Examples: - `models/namespace/my-model` - `datasets/username/my-dataset` - `spaces/org-name/my-space` This format is used for: - Database filtering (query jobs/deployments by project) - Repository-scoped endpoint routing - GitLab access validation ## Data Model ### Job Status Values | Status | Description | | ----------- | ------------------------------------------------------------------------------------- | | `pending` | Job created, not yet started | | `preparing` | Cloning repos and running the action's `pre` commands (login node, before submission) | | `queued` | Submitted to SLURM, waiting for resources to be allocated | | `running` | SLURM is executing the job | | `succeeded` | Job completed successfully | | `failed` | Job failed | | `destroyed` | Job was destroyed | ### Deployment Status Values | Status | Description | | ----------- | ----------------------------------------------------- | | `pending` | Deployment created, not yet deployed | | `preparing` | Cloning repos and running the action's `pre` commands | | `running` | Deployment is running | | `stopped` | Deployment was stopped | | `destroyed` | Deployment was destroyed | ### Log Level Values - `info` - Informational messages - `warning` - Warning messages - `error` - Error messages ## API Design **Note:** Job and deployment creation happens via the existing action endpoints. ### Get Job Details Response **GET** `/api/[repoType]/[namespace]/[repo]/jobs/{job_id}` **Response:** ```json { "job_id": "job-abc123", "gitlab_project_id": "models/namespace/my-model", "namespace": "namespace", "repo": "my-model", "branch_name": "main", "action_name": "train", "tag": "training", "status": "running", "created_at": "2026-06-02T10:00:00Z", "started_at": "2026-06-02T10:01:00Z", "completed_at": null, "destroyed_at": null, "logs": [ { "timestamp": "2026-06-02T10:01:00Z", "level": "info", "message": "Job started" }, { "timestamp": "2026-06-02T10:02:00Z", "level": "info", "message": "Training epoch 1/10" } ] } ``` ### List Jobs Response **GET** `/api/[repoType]/[namespace]/[repo]/jobs` **Response:** ```json { "jobs": [ { "job_id": "job-abc123", "action_name": "train", "tag": "training", "status": "running", "created_at": "2026-06-02T10:00:00Z" }, { "job_id": "job-def456", "action_name": "evaluate", "tag": "evaluation", "status": "succeeded", "created_at": "2026-06-02T09:00:00Z" } ] } ``` **GET** `/api/jobs/namespace/{namespace}?tag=training` **Response:** (same format, filtered by namespace and optional tag) ### List Deployments Response **GET** `/api/[repoType]/[namespace]/[repo]/deployments` **Response:** ```json { "deployments": [ { "deployment_id": "deployment-xyz789", "action_name": "serve", "tag": "inference", "status": "running", "url": "https://my-app.example.com", "created_at": "2026-06-02T10:00:00Z" } ] } ``` **GET** `/api/deployments/namespace/{namespace}?tag=inference` **Response:** (same format, filtered by namespace and optional tag) ### Update Log and Status Request **POST** `/api/jobs/{row_id}/notifications` (flat, keyed by the integer DB row id; deployments use `/api/deployments/{row_id}/notifications`) Headers: - `x-action-secret: ` (unified for both jobs and deployments) ```json { "status": "running", "log": { "level": "info", "message": "Training started" } } ``` **Response:** ```json { "success": true } ``` **Error Responses:** - `401 Unauthorized` - Secret not provided - `403 Forbidden` - Invalid secret (k8s-root-actor validation) ## Testing Strategy ### Unit Tests 1. **Database Models:** - Test CRUD operations for jobs and deployments - Test log insertion and querying - Test gitlab_project_id filtering 2. **Secret Management:** - Test secret generation and hashing - Test secret validation - Test invalid secret rejection ### Integration Tests 1. **End-to-End Flow:** - Trigger action via web service action endpoint - Verify k8s-root-actor creates job/deployment record in PostgreSQL - Verify response includes id and secret - Update status via PUT endpoint with secret - Verify status persisted in PostgreSQL - Query via GET endpoint - Verify response includes status and logs - Destroy and verify record marked destroyed, historical data preserved 2. **Action Endpoint Integration:** - Test slurm-job action → creates job record - Test k8s-deployment action → creates deployment record - Test k8s-inferenceservice action → creates deployment record - Verify gitlab_project_id correctly stored - Verify secret generated and returned - Verify Kubernetes resources created after database record 3. **Security Testing:** - Access without GitLab write access → 403 - PUT without secret → k8s-root-actor returns 401 Unauthorized - PUT with invalid secret → k8s-root-actor returns 403 Forbidden - PUT with valid secret → success - Web service passes secret to k8s-root-actor without validating 4. **Repository Scoping:** - Verify endpoint routing by repoType/namespace/repo - Verify gitlab_project_id filtering - Test cross-repository access denial ### Manual Validation 1. **Database Setup:** - PostgreSQL accessible from k8s-root-actor - Schema created and indexed correctly - Connection pooling configured 2. **API Validation:** - All endpoints respond correctly - Error handling works as expected - Secrets validated properly ## Deployment & Rollout ### Phase 1: Database Setup 1. Provision PostgreSQL database 2. Create schema and tables 3. Configure connection pooling 4. Verify k8s-root-actor connectivity ### Phase 2: k8s-root-actor Updates 1. Add database connection code 2. Add internal endpoints 3. Test database operations 4. Deploy to staging ### Phase 3: Web Service Updates 1. Add repo-scoped endpoints 2. Add GitLab access validation 3. Add k8s-root-actor client calls 4. Test integration 5. Deploy to staging ## Open Questions 1. **GitLab Access Caching:** How should web service cache GitLab access checks to avoid repeated API calls? ## References - [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system.md)
Page source Source snapshot · Revision `f928834c8`.