---
title: "DD-0012: Job and Deployment Management System — Implementation"
---

# DD-0012: Job and Deployment Management System — Implementation

**Related ADR:** [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system)

**Status:** Draft

**Date:** 2026-06-02

**Authors:** TBD

## Overview

This document specifies the implementation of a PostgreSQL-based persistent job and deployment management system for Themelio. The system provides clear separation between short-running jobs and long-running deployments, with historical data retention, secret-based security for external updates, and explicit destroy functionality.

:::note Implementation status (reconciled 2026-06-07)
This design is **implemented** in `k8s-root-actor` (PostgreSQL via knex, the `jobs`/`deployments`/`*_logs` tables, the CRUD/destroy endpoints, secret hashing, and status tracking) and in the `web` service (repo- and namespace-scoped proxy endpoints, GitLab write-access checks, secret passthrough). The formal **Status** above stays `Draft` only because ADR-0012 is still `Proposed` (a DD is not marked `Implemented` ahead of its ADR being `Accepted`). Two details below were corrected to match the shipped code:

- The secret header is a **single unified `x-action-secret`** header (not separate `x-job-secret` / `x-deployment-secret` headers).
- `job_logs` / `deployment_logs` carry an extra **`source_status`** column that stores the granular template status (e.g. `SLURM_JOB_OUTPUT`, `JOB_FINISHED`).

**Log/status update path (reconciled 2026-06-09):** the original "external system → web `PUT …/jobs/{id}` with `X-Resource-Secret`" design was superseded. The as-built path is: the pod posts to a **flat web endpoint, `POST /api/{jobs,deployments}/{id}/notifications`** (keyed by the integer DB row id), authenticated by the **`x-action-secret`** header; the web service is a secret-authenticated passthrough (no GitLab check) that proxies to k8s-root-actor **`POST /{jobs,deployments}/{id}/notifications`**, which validates the secret against `secret_hash`. The repo-scoped `PUT …/jobs/{id}` endpoints described below were never wired and have been removed — read the verb as `POST … /notifications` and the header as `x-action-secret` throughout the rest of this document.
:::

## Goals

- Replace ephemeral Valkey storage with persistent PostgreSQL
- Separate job and deployment lifecycles with appropriate APIs
- Enable historical data retention for analytics and audit
- Provide secret-based authentication for external log/status updates
- Implement explicit destroy with history preservation
- Maintain single database owner pattern (k8s-root-actor only)

## Design

### 1. Architecture

The system consists of two main components:

1. **k8s-root-actor** - Owns PostgreSQL database, manages all job/deployment data
2. **Web service** - Provides repo-scoped API endpoints, delegates data operations to k8s-root-actor

**Communication Flow:**
- Web service → k8s-root-actor: HTTP requests for all CRUD operations
- External systems (Kubernetes/SLURM) → Web service → k8s-root-actor: PUT log-and-status with secret
- k8s-root-actor → PostgreSQL: Direct database connection

### 2. Database Schema

**Jobs Table:**
```sql
CREATE TABLE jobs (
  id SERIAL PRIMARY KEY,
  job_id VARCHAR(255) UNIQUE NOT NULL,
  gitlab_project_id VARCHAR(512) NOT NULL, -- URL-encoded "<repoType>/<namespace>/<repoId>"
  namespace VARCHAR(255) NOT NULL,
  repo VARCHAR(255) NOT NULL,
  branch_name VARCHAR(255) NOT NULL,
  action_name VARCHAR(255) NOT NULL,
  tag VARCHAR(255), -- Optional tag from actions.yaml
  status VARCHAR(50) NOT NULL, -- pending, running, succeeded, failed, destroyed
  secret_hash VARCHAR(255) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
  started_at TIMESTAMP,
  completed_at TIMESTAMP,
  destroyed_at TIMESTAMP,
  config JSONB NOT NULL,
  result JSONB,
  INDEX idx_gitlab_project (gitlab_project_id),
  INDEX idx_namespace_tag (namespace, tag),
  INDEX idx_status (status),
  INDEX idx_created (created_at)
);
```

**Deployments Table:**
```sql
CREATE TABLE deployments (
  id SERIAL PRIMARY KEY,
  deployment_id VARCHAR(255) UNIQUE NOT NULL,
  gitlab_project_id VARCHAR(512) NOT NULL, -- URL-encoded "<repoType>/<namespace>/<repoId>"
  namespace VARCHAR(255) NOT NULL,
  repo VARCHAR(255) NOT NULL,
  branch_name VARCHAR(255) NOT NULL,
  action_name VARCHAR(255) NOT NULL,
  tag VARCHAR(255), -- Optional tag from actions.yaml
  status VARCHAR(50) NOT NULL, -- pending, preparing, running, stopped, destroyed
  secret_hash VARCHAR(255) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
  deployed_at TIMESTAMP,
  destroyed_at TIMESTAMP,
  url VARCHAR(255),
  config JSONB NOT NULL,
  result JSONB,
  INDEX idx_gitlab_project (gitlab_project_id),
  INDEX idx_namespace_tag (namespace, tag),
  INDEX idx_status (status),
  INDEX idx_created (created_at)
);
```

**Job Logs Table:**
```sql
CREATE TABLE job_logs (
  id SERIAL PRIMARY KEY,
  job_id VARCHAR(255) NOT NULL REFERENCES jobs(job_id),
  timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
  level VARCHAR(20) NOT NULL, -- info, warning, error
  message TEXT NOT NULL,
  source_status VARCHAR(255), -- granular template status (e.g. SLURM_JOB_OUTPUT, JOB_FINISHED)
  INDEX idx_job_timestamp (job_id, timestamp)
);
```

**Deployment Logs Table:**
```sql
CREATE TABLE deployment_logs (
  id SERIAL PRIMARY KEY,
  deployment_id VARCHAR(255) NOT NULL REFERENCES deployments(deployment_id),
  timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
  level VARCHAR(20) NOT NULL,
  message TEXT NOT NULL,
  source_status VARCHAR(255), -- granular template status
  INDEX idx_deployment_timestamp (deployment_id, timestamp)
);
```

### 3. API Endpoints

#### Web Service Endpoints

**Jobs (scoped to repository):**

| Method | Path | Description | Access Control |
|--------|------|-------------|----------------|
| GET | `/api/[repoType]/[namespace]/[repo]/jobs` | List jobs for this repository (with status) | GitLab write access check |
| GET | `/api/[repoType]/[namespace]/[repo]/jobs/{job_id}` | Get job details (includes status and logs) | GitLab write access check |
| POST | `/api/jobs/{row_id}/notifications` | Update log and status (flat, not repo-scoped) | Secret validation (`x-action-secret`) |
| DELETE | `/api/[repoType]/[namespace]/[repo]/jobs/{job_id}` | Destroy job | GitLab write access check |

**Jobs (scoped to namespace):**

| Method | Path | Description | Access Control |
|--------|------|-------------|----------------|
| GET | `/api/jobs/namespace/{namespace}` | List all jobs under namespace (with status, filterable by tag) | GitLab write access check |

**Deployments (scoped to repository):**

| Method | Path | Description | Access Control |
|--------|------|-------------|----------------|
| GET | `/api/[repoType]/[namespace]/[repo]/deployments` | List deployments for this repository (with status) | GitLab write access check |
| GET | `/api/[repoType]/[namespace]/[repo]/deployments/{deployment_id}` | Get deployment details (includes status and logs) | GitLab write access check |
| POST | `/api/deployments/{row_id}/notifications` | Update log and status (flat, not repo-scoped) | Secret validation (`x-action-secret`) |
| DELETE | `/api/[repoType]/[namespace]/[repo]/deployments/{deployment_id}` | Destroy deployment | GitLab write access check |

**Deployments (scoped to namespace):**

| Method | Path | Description | Access Control |
|--------|------|-------------|----------------|
| GET | `/api/deployments/namespace/{namespace}` | List all deployments under namespace (with status, filterable by tag) | GitLab write access check |

**Note:** Job and deployment creation happens via the existing action endpoints:
- `POST /api/[repoType]/[namespace]/[repo]/action/[branchName]` (web service)
- `POST /:repoType/:namespace/:repoId/:branchName/action` (k8s-root-actor)

#### k8s-root-actor Endpoints

**Jobs:**
- `GET /jobs/{job_id}` - Get job details
- `GET /jobs` - List jobs (accepts `gitlab_project_id` or `namespace` query parameter, optional `tag` filter)
- `POST /jobs/{row_id}/notifications` - Update job log and status (validates `x-action-secret`)
- `DELETE /jobs/{job_id}` - Destroy job

**Deployments:**
- `GET /deployments/{deployment_id}` - Get deployment details
- `GET /deployments` - List deployments (accepts `gitlab_project_id` or `namespace` query parameter, optional `tag` filter)
- `POST /deployments/{row_id}/notifications` - Update deployment log and status (validates `x-action-secret`)
- `DELETE /deployments/{deployment_id}` - Destroy deployment

### Existing Action Endpoint Updates

The existing action endpoints must be updated to integrate with the new PostgreSQL-based job/deployment system.

#### Web Service Action Endpoint

**Current Location:** `services/web/src/app/api/[repoType]/[namespace]/[repo]/action/[branchName]/route.ts`

**Updates Required:**

1. Keep existing POST handler for action execution
2. Update to handle response from k8s-root-actor that includes `job_id`/`deployment_id` and `secret`
3. Return these to the caller for future status/log/destroy operations

**Request Flow:**
```
POST /api/[repoType]/[namespace]/[repo]/action/[branchName]
Body: { actionName, parameters }
  ↓
Web service validates GitLab access, fetches actions.yaml
  ↓
Web service calls k8s-root-actor action endpoint
  ↓
k8s-root-actor executes action, creates job/deployment record in PostgreSQL
  ↓
k8s-root-actor returns { type: "job"|"deployment", id: "...", secret: "...", status: "pending" }
  ↓
Web service returns response to caller
```

**Response Format:**
```json
{
  "type": "job" | "deployment",
  "id": "job-abc123" | "deployment-xyz789",
  "status": "pending",
  "created_at": "2026-06-02T10:00:00Z"
}
```

**Note:** The secret is generated and stored by k8s-root-actor but is NOT returned in the response. The secret must be obtained and managed separately for use in PUT log-and-status requests.

#### k8s-root-actor Action Endpoint

**Current Location:** `services/k8s-root-actor/src/index.ts`

**Current Endpoint:** `POST /:repoType/:namespace/:repoId/:branchName/action`

**Updates Required:**

1. Before creating Kubernetes resources, determine if action type is `slurm-job` (job) or `k8s-deployment`/`k8s-inferenceservice` (deployment)
2. Generate secret and create job/deployment record in PostgreSQL with hashed secret
3. Store the `gitlab_project_id` (URL-encoded `<repoType>/<namespace>/<repoId>`)
4. After creating Kubernetes resources, return the job/deployment ID and secret
5. On status changes, update PostgreSQL record

**Updated Request Flow:**
```
POST /:repoType/:namespace/:repoId/:branchName/action
Body: { actionName, actionConfig, gitlabToken }
  ↓
k8s-root-actor resolves action references
  ↓
k8s-root-actor determines resource type (job vs deployment)
  ↓
k8s-root-actor generates secret, creates record in PostgreSQL:
  - For slurm-job: INSERT INTO jobs (job_id, gitlab_project_id, status, secret_hash, ...)
  - For k8s-deployment/inferenceservice: INSERT INTO deployments (...)
  ↓
k8s-root-actor creates Kubernetes/SLURM resources
  ↓
k8s-root-actor returns { type, id, status }
```

**Note:** The secret is generated and stored by k8s-root-actor but must be distributed to external systems through a separate mechanism (not via the web service response).

**Integration Points:**

| Current Action Type | New Resource Type | Table | ID Prefix |
|---------------------|-------------------|-------|-----------|
| `slurm-job` | Job | `jobs` | `job-` |
| `k8s-deployment` | Deployment | `deployments` | `deployment-` |
| `k8s-inferenceservice` | Deployment | `deployments` | `deployment-` |

**Status Tracking:**

k8s-root-actor must update the PostgreSQL record when:
- Job starts: `status = "running"`, `started_at = NOW()`
- Job completes: `status = "succeeded"` or `"failed"`, `completed_at = NOW()`
- Deployment becomes ready: `status = "running"`, `deployed_at = NOW()`
- Deployment fails: `status = "failed"`
- Job/deployment destroyed: `status = "destroyed"`, `destroyed_at = NOW()`

### 4. Data Flow

#### Creation Flow (via Action Endpoints)

```
User UI
  ↓ (trigger action)
Web service POST /api/[repoType]/[namespace]/[repo]/action/[branchName]
  ↓ (call k8s-root-actor)
k8s-root-actor POST /:repoType/:namespace/:repoId/:branchName/action
  ↓ (determine job/deployment type, generate secret, store record)
PostgreSQL (store with hashed secret)
  ↓ (create resources)
k8s-root-actor creates Kubernetes/SLURM resources
  ↓ (return id)
Web service → User (id)
```

#### Status Update Flow (External)

```
External system (Kubernetes/SLURM)
  ↓ (send log/status update with secret)
Web service PUT /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id}
  ↓ (validate GitLab access, pass secret to k8s-root-actor)
k8s-root-actor PUT /jobs/{id} or /deployments/{id}
  ↓ (validate secret, update record)
PostgreSQL (update status and add log entry)
  ↓ (return success)
Web service → External system (success)
```

#### User Polling Flow

```
User UI
  ↓ (poll for status)
Web service GET /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id}
  ↓ (validate GitLab write access)
k8s-root-actor GET /jobs/{id} or /deployments/{id}
  ↓ (query database)
PostgreSQL (return record)
  ↓ (return data)
Web service → User (status/details)
```

#### List Jobs/Deployments Flow

```
User UI
  ↓ (list jobs/deployments)
Web service GET /api/[repoType]/[namespace]/[repo]/jobs or /deployments
  ↓ (validate GitLab write access, construct gitlab_project_id)
k8s-root-actor GET /jobs or /deployments?gitlab_project_id={encoded}
  ↓ (query database)
PostgreSQL (return filtered records)
  ↓ (return data)
Web service → User (list)
```

#### Destroy Flow

```
User UI
  ↓ (request destroy)
Web service DELETE /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id}
  ↓ (validate GitLab write access)
k8s-root-actor DELETE /jobs/{id} or /deployments/{id}
  ↓ (destroy resources)
k8s-root-actor destroys Kubernetes/SLURM resources
  ↓ (update record)
PostgreSQL (set destroyed_at, status=destroyed)
  ↓ (return success)
Web service → User (success)
```

### 5. Secret Management

**Generation (k8s-root-actor only):**
```typescript
// Generate 256-bit random secret
const secret = crypto.randomBytes(32).toString('hex');
const secretHash = crypto.createHash('sha256').update(secret).digest('hex');
// Store hash in database
// Secret is NOT returned to caller - must be managed separately
```

**Validation (k8s-root-actor only):**
```typescript
// On POST /{kind}/{id}/notifications endpoint — a single unified header for both kinds
const secret = request.headers['x-action-secret'];
const secretHash = crypto.createHash('sha256').update(secret).digest('hex');
// Compare with stored hash in database
```

**Passthrough (Web service):**
```typescript
// Extract secret from request
const secret = request.headers['x-action-secret'];
// Pass secret to k8s-root-actor without validation
const response = await fetch(k8sRootActorUrl, {
  headers: { 'x-action-secret': secret }
});
```

**Secret Distribution:**
The secret is generated by k8s-root-actor but must be distributed to external systems (Kubernetes/SLURM) through a separate mechanism (not specified in this document). External systems use this secret to authenticate PUT log-and-status requests.

### 6. GitLab Project ID Format

The `gitlab_project_id` is a URL-encoded string: `<repoType>/<namespace>/<repoId>`

Examples:
- `models/namespace/my-model`
- `datasets/username/my-dataset`
- `spaces/org-name/my-space`

This format is used for:
- Database filtering (query jobs/deployments by project)
- Repository-scoped endpoint routing
- GitLab access validation

## Data Model

### Job Status Values

| Status | Description |
|--------|-------------|
| `pending` | Job created, not yet started |
| `preparing` | Cloning repos and running the action's `pre` commands (login node, before submission) |
| `queued` | Submitted to SLURM, waiting for resources to be allocated |
| `running` | SLURM is executing the job |
| `succeeded` | Job completed successfully |
| `failed` | Job failed |
| `destroyed` | Job was destroyed |

### Deployment Status Values

| Status | Description |
|--------|-------------|
| `pending` | Deployment created, not yet deployed |
| `preparing` | Cloning repos and running the action's `pre` commands |
| `running` | Deployment is running |
| `stopped` | Deployment was stopped |
| `destroyed` | Deployment was destroyed |

### Log Level Values

- `info` - Informational messages
- `warning` - Warning messages
- `error` - Error messages

## API Design

**Note:** Job and deployment creation happens via the existing action endpoints.

### Get Job Details Response

**GET** `/api/[repoType]/[namespace]/[repo]/jobs/{job_id}`

**Response:**
```json
{
  "job_id": "job-abc123",
  "gitlab_project_id": "models/namespace/my-model",
  "namespace": "namespace",
  "repo": "my-model",
  "branch_name": "main",
  "action_name": "train",
  "tag": "training",
  "status": "running",
  "created_at": "2026-06-02T10:00:00Z",
  "started_at": "2026-06-02T10:01:00Z",
  "completed_at": null,
  "destroyed_at": null,
  "logs": [
    {
      "timestamp": "2026-06-02T10:01:00Z",
      "level": "info",
      "message": "Job started"
    },
    {
      "timestamp": "2026-06-02T10:02:00Z",
      "level": "info",
      "message": "Training epoch 1/10"
    }
  ]
}
```

### List Jobs Response

**GET** `/api/[repoType]/[namespace]/[repo]/jobs`

**Response:**
```json
{
  "jobs": [
    {
      "job_id": "job-abc123",
      "action_name": "train",
      "tag": "training",
      "status": "running",
      "created_at": "2026-06-02T10:00:00Z"
    },
    {
      "job_id": "job-def456",
      "action_name": "evaluate",
      "tag": "evaluation",
      "status": "succeeded",
      "created_at": "2026-06-02T09:00:00Z"
    }
  ]
}
```

**GET** `/api/jobs/namespace/{namespace}?tag=training`

**Response:** (same format, filtered by namespace and optional tag)

### List Deployments Response

**GET** `/api/[repoType]/[namespace]/[repo]/deployments`

**Response:**
```json
{
  "deployments": [
    {
      "deployment_id": "deployment-xyz789",
      "action_name": "serve",
      "tag": "inference",
      "status": "running",
      "url": "https://my-app.example.com",
      "created_at": "2026-06-02T10:00:00Z"
    }
  ]
}
```

**GET** `/api/deployments/namespace/{namespace}?tag=inference`

**Response:** (same format, filtered by namespace and optional tag)

### Update Log and Status Request

**POST** `/api/jobs/{row_id}/notifications` (flat, keyed by the integer DB row id; deployments use `/api/deployments/{row_id}/notifications`)

Headers:
- `x-action-secret: <secret>` (unified for both jobs and deployments)

```json
{
  "status": "running",
  "log": {
    "level": "info",
    "message": "Training started"
  }
}
```

**Response:**
```json
{
  "success": true
}
```

**Error Responses:**
- `401 Unauthorized` - Secret not provided
- `403 Forbidden` - Invalid secret (k8s-root-actor validation)

## Testing Strategy

### Unit Tests

1. **Database Models:**
   - Test CRUD operations for jobs and deployments
   - Test log insertion and querying
   - Test gitlab_project_id filtering

2. **Secret Management:**
   - Test secret generation and hashing
   - Test secret validation
   - Test invalid secret rejection

### Integration Tests

1. **End-to-End Flow:**
   - Trigger action via web service action endpoint
   - Verify k8s-root-actor creates job/deployment record in PostgreSQL
   - Verify response includes id and secret
   - Update status via PUT endpoint with secret
   - Verify status persisted in PostgreSQL
   - Query via GET endpoint
   - Verify response includes status and logs
   - Destroy and verify record marked destroyed, historical data preserved

2. **Action Endpoint Integration:**
   - Test slurm-job action → creates job record
   - Test k8s-deployment action → creates deployment record
   - Test k8s-inferenceservice action → creates deployment record
   - Verify gitlab_project_id correctly stored
   - Verify secret generated and returned
   - Verify Kubernetes resources created after database record

2. **Security Testing:**
   - Access without GitLab write access → 403
   - PUT without secret → k8s-root-actor returns 401 Unauthorized
   - PUT with invalid secret → k8s-root-actor returns 403 Forbidden
   - PUT with valid secret → success
   - Web service passes secret to k8s-root-actor without validating

3. **Repository Scoping:**
   - Verify endpoint routing by repoType/namespace/repo
   - Verify gitlab_project_id filtering
   - Test cross-repository access denial

### Manual Validation

1. **Database Setup:**
   - PostgreSQL accessible from k8s-root-actor
   - Schema created and indexed correctly
   - Connection pooling configured

2. **API Validation:**
   - All endpoints respond correctly
   - Error handling works as expected
   - Secrets validated properly

## Deployment & Rollout

### Phase 1: Database Setup

1. Provision PostgreSQL database
2. Create schema and tables
3. Configure connection pooling
4. Verify k8s-root-actor connectivity

### Phase 2: k8s-root-actor Updates

1. Add database connection code
2. Add internal endpoints
3. Test database operations
4. Deploy to staging

### Phase 3: Web Service Updates

1. Add repo-scoped endpoints
2. Add GitLab access validation
3. Add k8s-root-actor client calls
4. Test integration
5. Deploy to staging

## Open Questions

1. **GitLab Access Caching:** How should web service cache GitLab access checks to avoid repeated API calls?

## References

- [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system)