Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0012: Job and Deployment Management System — Implementation

Related ADR: ADR-0012: Job and Deployment Management System

Status: Draft

Date: 2026-06-02

Authors: TBD

#Overview

This document specifies the implementation of a PostgreSQL-based persistent job and deployment management system for Themelio. The system provides clear separation between short-running jobs and long-running deployments, with historical data retention, secret-based security for external updates, and explicit destroy functionality.

#Goals

• Replace ephemeral Valkey storage with persistent PostgreSQL
• Separate job and deployment lifecycles with appropriate APIs
• Enable historical data retention for analytics and audit
• Provide secret-based authentication for external log/status updates
• Implement explicit destroy with history preservation
• Maintain single database owner pattern (k8s-root-actor only)

#Design

#1. Architecture

The system consists of two main components:

1. k8s-root-actor - Owns PostgreSQL database, manages all job/deployment data
2. Web service - Provides repo-scoped API endpoints, delegates data operations to k8s-root-actor

Communication Flow:

• Web service → k8s-root-actor: HTTP requests for all CRUD operations
• External systems (Kubernetes/SLURM) → Web service → k8s-root-actor: PUT log-and-status with secret
• k8s-root-actor → PostgreSQL: Direct database connection

#2. Database Schema

Jobs Table:

CREATE TABLE jobs (
  id SERIAL PRIMARY KEY,
  job_id VARCHAR(255) UNIQUE NOT NULL,
  gitlab_project_id VARCHAR(512) NOT NULL, -- URL-encoded "<repoType>/<namespace>/<repoId>"
  namespace VARCHAR(255) NOT NULL,
  repo VARCHAR(255) NOT NULL,
  branch_name VARCHAR(255) NOT NULL,
  action_name VARCHAR(255) NOT NULL,
  tag VARCHAR(255), -- Optional tag from actions.yaml
  status VARCHAR(50) NOT NULL, -- pending, running, succeeded, failed, destroyed
  secret_hash VARCHAR(255) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
  started_at TIMESTAMP,
  completed_at TIMESTAMP,
  destroyed_at TIMESTAMP,
  config JSONB NOT NULL,
  result JSONB,
  INDEX idx_gitlab_project (gitlab_project_id),
  INDEX idx_namespace_tag (namespace, tag),
  INDEX idx_status (status),
  INDEX idx_created (created_at)
);

Deployments Table:

CREATE TABLE deployments (
  id SERIAL PRIMARY KEY,
  deployment_id VARCHAR(255) UNIQUE NOT NULL,
  gitlab_project_id VARCHAR(512) NOT NULL, -- URL-encoded "<repoType>/<namespace>/<repoId>"
  namespace VARCHAR(255) NOT NULL,
  repo VARCHAR(255) NOT NULL,
  branch_name VARCHAR(255) NOT NULL,
  action_name VARCHAR(255) NOT NULL,
  tag VARCHAR(255), -- Optional tag from actions.yaml
  status VARCHAR(50) NOT NULL, -- pending, preparing, running, stopped, destroyed
  secret_hash VARCHAR(255) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
  deployed_at TIMESTAMP,
  destroyed_at TIMESTAMP,
  url VARCHAR(255),
  config JSONB NOT NULL,
  result JSONB,
  INDEX idx_gitlab_project (gitlab_project_id),
  INDEX idx_namespace_tag (namespace, tag),
  INDEX idx_status (status),
  INDEX idx_created (created_at)
);

Job Logs Table:

CREATE TABLE job_logs (
  id SERIAL PRIMARY KEY,
  job_id VARCHAR(255) NOT NULL REFERENCES jobs(job_id),
  timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
  level VARCHAR(20) NOT NULL, -- info, warning, error
  message TEXT NOT NULL,
  source_status VARCHAR(255), -- granular template status (e.g. SLURM_JOB_OUTPUT, JOB_FINISHED)
  INDEX idx_job_timestamp (job_id, timestamp)
);

Deployment Logs Table:

CREATE TABLE deployment_logs (
  id SERIAL PRIMARY KEY,
  deployment_id VARCHAR(255) NOT NULL REFERENCES deployments(deployment_id),
  timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
  level VARCHAR(20) NOT NULL,
  message TEXT NOT NULL,
  source_status VARCHAR(255), -- granular template status
  INDEX idx_deployment_timestamp (deployment_id, timestamp)
);

#3. API Endpoints

#Web Service Endpoints

Jobs (scoped to repository):

Jobs (scoped to namespace):

Deployments (scoped to repository):

Deployments (scoped to namespace):

Note: Job and deployment creation happens via the existing action endpoints:

• POST /api/[repoType]/[namespace]/[repo]/action/[branchName] (web service)
• POST /:repoType/:namespace/:repoId/:branchName/action (k8s-root-actor)

#k8s-root-actor Endpoints

Jobs:

• GET /jobs/{job_id} - Get job details
• GET /jobs - List jobs (accepts gitlab_project_id or namespace query parameter, optional tag filter)
• POST /jobs/{row_id}/notifications - Update job log and status (validates x-action-secret)
• DELETE /jobs/{job_id} - Destroy job

Deployments:

• GET /deployments/{deployment_id} - Get deployment details
• GET /deployments - List deployments (accepts gitlab_project_id or namespace query parameter, optional tag filter)
• POST /deployments/{row_id}/notifications - Update deployment log and status (validates x-action-secret)
• DELETE /deployments/{deployment_id} - Destroy deployment

#Existing Action Endpoint Updates

The existing action endpoints must be updated to integrate with the new PostgreSQL-based job/deployment system.

#Web Service Action Endpoint

Current Location: services/web/src/app/api/[repoType]/[namespace]/[repo]/action/[branchName]/route.ts

Updates Required:

1. Keep existing POST handler for action execution
2. Update to handle response from k8s-root-actor that includes job_id/deployment_id and secret
3. Return these to the caller for future status/log/destroy operations

Request Flow:

POST /api/[repoType]/[namespace]/[repo]/action/[branchName]
Body: { actionName, parameters }
  ↓
Web service validates GitLab access, fetches actions.yaml
  ↓
Web service calls k8s-root-actor action endpoint
  ↓
k8s-root-actor executes action, creates job/deployment record in PostgreSQL
  ↓
k8s-root-actor returns { type: "job"|"deployment", id: "...", secret: "...", status: "pending" }
  ↓
Web service returns response to caller

Response Format:

{
  "type": "job" | "deployment",
  "id": "job-abc123" | "deployment-xyz789",
  "status": "pending",
  "created_at": "2026-06-02T10:00:00Z"
}

Note: The secret is generated and stored by k8s-root-actor but is NOT returned in the response. The secret must be obtained and managed separately for use in PUT log-and-status requests.

#k8s-root-actor Action Endpoint

Current Location: services/k8s-root-actor/src/index.ts

Current Endpoint: POST /:repoType/:namespace/:repoId/:branchName/action

Updates Required:

1. Before creating Kubernetes resources, determine if action type is slurm-job (job) or k8s-deployment/k8s-inferenceservice (deployment)
2. Generate secret and create job/deployment record in PostgreSQL with hashed secret
3. Store the gitlab_project_id (URL-encoded <repoType>/<namespace>/<repoId>)
4. After creating Kubernetes resources, return the job/deployment ID and secret
5. On status changes, update PostgreSQL record

Updated Request Flow:

POST /:repoType/:namespace/:repoId/:branchName/action
Body: { actionName, actionConfig, gitlabToken }
  ↓
k8s-root-actor resolves action references
  ↓
k8s-root-actor determines resource type (job vs deployment)
  ↓
k8s-root-actor generates secret, creates record in PostgreSQL:
  - For slurm-job: INSERT INTO jobs (job_id, gitlab_project_id, status, secret_hash, ...)
  - For k8s-deployment/inferenceservice: INSERT INTO deployments (...)
  ↓
k8s-root-actor creates Kubernetes/SLURM resources
  ↓
k8s-root-actor returns { type, id, status }

Note: The secret is generated and stored by k8s-root-actor but must be distributed to external systems through a separate mechanism (not via the web service response).

Integration Points:

Status Tracking:

k8s-root-actor must update the PostgreSQL record when:

• Job starts: status = "running", started_at = NOW()
• Job completes: status = "succeeded" or "failed", completed_at = NOW()
• Deployment becomes ready: status = "running", deployed_at = NOW()
• Deployment fails: status = "failed"
• Job/deployment destroyed: status = "destroyed", destroyed_at = NOW()

#4. Data Flow

#Creation Flow (via Action Endpoints)

User UI
  ↓ (trigger action)
Web service POST /api/[repoType]/[namespace]/[repo]/action/[branchName]
  ↓ (call k8s-root-actor)
k8s-root-actor POST /:repoType/:namespace/:repoId/:branchName/action
  ↓ (determine job/deployment type, generate secret, store record)
PostgreSQL (store with hashed secret)
  ↓ (create resources)
k8s-root-actor creates Kubernetes/SLURM resources
  ↓ (return id)
Web service → User (id)

#Status Update Flow (External)

External system (Kubernetes/SLURM)
  ↓ (send log/status update with secret)
Web service PUT /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id}
  ↓ (validate GitLab access, pass secret to k8s-root-actor)
k8s-root-actor PUT /jobs/{id} or /deployments/{id}
  ↓ (validate secret, update record)
PostgreSQL (update status and add log entry)
  ↓ (return success)
Web service → External system (success)

#User Polling Flow

User UI
  ↓ (poll for status)
Web service GET /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id}
  ↓ (validate GitLab write access)
k8s-root-actor GET /jobs/{id} or /deployments/{id}
  ↓ (query database)
PostgreSQL (return record)
  ↓ (return data)
Web service → User (status/details)

#List Jobs/Deployments Flow

User UI
  ↓ (list jobs/deployments)
Web service GET /api/[repoType]/[namespace]/[repo]/jobs or /deployments
  ↓ (validate GitLab write access, construct gitlab_project_id)
k8s-root-actor GET /jobs or /deployments?gitlab_project_id={encoded}
  ↓ (query database)
PostgreSQL (return filtered records)
  ↓ (return data)
Web service → User (list)

#Destroy Flow

User UI
  ↓ (request destroy)
Web service DELETE /api/[repoType]/[namespace]/[repo]/jobs/{id} or /deployments/{id}
  ↓ (validate GitLab write access)
k8s-root-actor DELETE /jobs/{id} or /deployments/{id}
  ↓ (destroy resources)
k8s-root-actor destroys Kubernetes/SLURM resources
  ↓ (update record)
PostgreSQL (set destroyed_at, status=destroyed)
  ↓ (return success)
Web service → User (success)

#5. Secret Management

Generation (k8s-root-actor only):

// Generate 256-bit random secret
const secret = crypto.randomBytes(32).toString('hex');
const secretHash = crypto.createHash('sha256').update(secret).digest('hex');
// Store hash in database
// Secret is NOT returned to caller - must be managed separately

Validation (k8s-root-actor only):

// On POST /{kind}/{id}/notifications endpoint — a single unified header for both kinds
const secret = request.headers['x-action-secret'];
const secretHash = crypto.createHash('sha256').update(secret).digest('hex');
// Compare with stored hash in database

Passthrough (Web service):

// Extract secret from request
const secret = request.headers['x-action-secret'];
// Pass secret to k8s-root-actor without validation
const response = await fetch(k8sRootActorUrl, {
  headers: { 'x-action-secret': secret }
});

Secret Distribution: The secret is generated by k8s-root-actor but must be distributed to external systems (Kubernetes/SLURM) through a separate mechanism (not specified in this document). External systems use this secret to authenticate PUT log-and-status requests.

#6. GitLab Project ID Format

The gitlab_project_id is a URL-encoded string: <repoType>/<namespace>/<repoId>

Examples:

• models/namespace/my-model
• datasets/username/my-dataset
• spaces/org-name/my-space

This format is used for:

• Database filtering (query jobs/deployments by project)
• Repository-scoped endpoint routing
• GitLab access validation

#Data Model

#Job Status Values

#Deployment Status Values

#Log Level Values

• info - Informational messages
• warning - Warning messages
• error - Error messages

#API Design

Note: Job and deployment creation happens via the existing action endpoints.

#Get Job Details Response

GET /api/[repoType]/[namespace]/[repo]/jobs/{job_id}

Response:

{
  "job_id": "job-abc123",
  "gitlab_project_id": "models/namespace/my-model",
  "namespace": "namespace",
  "repo": "my-model",
  "branch_name": "main",
  "action_name": "train",
  "tag": "training",
  "status": "running",
  "created_at": "2026-06-02T10:00:00Z",
  "started_at": "2026-06-02T10:01:00Z",
  "completed_at": null,
  "destroyed_at": null,
  "logs": [
    {
      "timestamp": "2026-06-02T10:01:00Z",
      "level": "info",
      "message": "Job started"
    },
    {
      "timestamp": "2026-06-02T10:02:00Z",
      "level": "info",
      "message": "Training epoch 1/10"
    }
  ]
}

#List Jobs Response

GET /api/[repoType]/[namespace]/[repo]/jobs

Response:

{
  "jobs": [
    {
      "job_id": "job-abc123",
      "action_name": "train",
      "tag": "training",
      "status": "running",
      "created_at": "2026-06-02T10:00:00Z"
    },
    {
      "job_id": "job-def456",
      "action_name": "evaluate",
      "tag": "evaluation",
      "status": "succeeded",
      "created_at": "2026-06-02T09:00:00Z"
    }
  ]
}

GET /api/jobs/namespace/{namespace}?tag=training

Response: (same format, filtered by namespace and optional tag)

#List Deployments Response

GET /api/[repoType]/[namespace]/[repo]/deployments

Response:

{
  "deployments": [
    {
      "deployment_id": "deployment-xyz789",
      "action_name": "serve",
      "tag": "inference",
      "status": "running",
      "url": "https://my-app.example.com",
      "created_at": "2026-06-02T10:00:00Z"
    }
  ]
}

GET /api/deployments/namespace/{namespace}?tag=inference

Response: (same format, filtered by namespace and optional tag)

#Update Log and Status Request

POST /api/jobs/{row_id}/notifications (flat, keyed by the integer DB row id; deployments use /api/deployments/{row_id}/notifications)

Headers:

• x-action-secret: <secret> (unified for both jobs and deployments)

{
  "status": "running",
  "log": {
    "level": "info",
    "message": "Training started"
  }
}

Response:

{
  "success": true
}

Error Responses:

• 401 Unauthorized - Secret not provided
• 403 Forbidden - Invalid secret (k8s-root-actor validation)

#Testing Strategy

#Unit Tests

1. Database Models:

   • Test CRUD operations for jobs and deployments
   • Test log insertion and querying
   • Test gitlab_project_id filtering

2. Secret Management:

   • Test secret generation and hashing
   • Test secret validation
   • Test invalid secret rejection

#Integration Tests

1. End-to-End Flow:

   • Trigger action via web service action endpoint
   • Verify k8s-root-actor creates job/deployment record in PostgreSQL
   • Verify response includes id and secret
   • Update status via PUT endpoint with secret
   • Verify status persisted in PostgreSQL
   • Query via GET endpoint
   • Verify response includes status and logs
   • Destroy and verify record marked destroyed, historical data preserved

2. Action Endpoint Integration:

   • Test slurm-job action → creates job record
   • Test k8s-deployment action → creates deployment record
   • Test k8s-inferenceservice action → creates deployment record
   • Verify gitlab_project_id correctly stored
   • Verify secret generated and returned
   • Verify Kubernetes resources created after database record

3. Security Testing:

   • Access without GitLab write access → 403
   • PUT without secret → k8s-root-actor returns 401 Unauthorized
   • PUT with invalid secret → k8s-root-actor returns 403 Forbidden
   • PUT with valid secret → success
   • Web service passes secret to k8s-root-actor without validating

4. Repository Scoping:

   • Verify endpoint routing by repoType/namespace/repo
   • Verify gitlab_project_id filtering
   • Test cross-repository access denial

#Manual Validation

1. Database Setup:

   • PostgreSQL accessible from k8s-root-actor
   • Schema created and indexed correctly
   • Connection pooling configured

2. API Validation:

   • All endpoints respond correctly
   • Error handling works as expected
   • Secrets validated properly

#Deployment & Rollout

#Phase 1: Database Setup

1. Provision PostgreSQL database
2. Create schema and tables
3. Configure connection pooling
4. Verify k8s-root-actor connectivity

#Phase 2: k8s-root-actor Updates

1. Add database connection code
2. Add internal endpoints
3. Test database operations
4. Deploy to staging

#Phase 3: Web Service Updates

1. Add repo-scoped endpoints
2. Add GitLab access validation
3. Add k8s-root-actor client calls
4. Test integration
5. Deploy to staging

#Open Questions

1. GitLab Access Caching: How should web service cache GitLab access checks to avoid repeated API calls?

#References

• ADR-0012: Job and Deployment Management System