For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0012-job-deployment-management-system.md.

ADR-0012: Job and Deployment Management System

Status: Proposed

Date: 2026-06-02

Deciders: TBD

Context

The current action/job system in Themelio has several architectural limitations that hinder scalability and operational efficiency.

Current State

The current system consists of:

  1. Web Service (Next.js) handles user-facing action execution via:

    • Three identical route handlers for models/repos/spaces delegating to handleActionRoute
    • Job tracking via Valkey (Redis) with 24-hour TTL
    • Endpoints: PUT /api/jobs/[id], GET /api/jobs/[id], GET /api/jobs/namespace/[namespace]

  2. k8s-root-actor (Fastify) creates Kubernetes resources:

    • Single endpoint: POST /:repoType/:namespace/:repoId/:branchName/action
    • Three action types: slurm-job, k8s-deployment, k8s-inferenceservice
    • No direct database access

  3. Data Flow:

    • User triggers action → web service fetches actions.yaml from GitLab
    • Creates temporary GitLab PAT with broad scopes (read_repository, write_repository, api)
    • Creates job entry in Valkey (24-hour TTL)
    • Calls k8s-root-actor which creates Kubernetes resources
    • Logs pushed back via /api/jobs/[id]

Current Problems

1. Ephemeral Storage

  • All job data stored in Valkey with 24-hour TTL
  • No historical data beyond 24 hours
  • No archiving or audit trail
  • Lost data on Valkey restart/crash

2. No Clear Separation Between Jobs and Deployments

  • Both "jobs" (short-running batch tasks) and "deployments" (long-running services) use the same actions.yaml system
  • UI/UX confusion for users
  • Inappropriate lifecycle management (deployments shouldn't have "completion")

3. Limited Observability

  • No job/deployment history
  • No filtering or search capabilities
  • No analytics or reporting

4. No Destroy Functionality

  • Jobs/deployments cannot be explicitly destroyed
  • Resources may orphan if tracking lost
  • No controlled teardown

5. Security Concerns

  • Temporary PATs created with overly broad scopes
  • No fine-grained authorization for job/deployment operations
  • Anyone with the job ID can access logs via current endpoints
  • No per-resource authentication tokens

6. Unified Endpoints

  • Jobs and deployments mixed in same API structure
  • No differentiation in queries or listing

7. Direct Web Service Database Access

  • Web service needs Valkey for jobs AND would need PostgreSQL for persistence
  • k8s-root-actor has no database access

Considered Options

Option A: PostgreSQL-based Persistent Job/Deployment System

Architecture Overview:

Introduce a PostgreSQL database for persistent storage of jobs and deployments with a clear separation of concerns. Only k8s-root-actor accesses the database directly. Web service delegates all data operations to k8s-root-actor.

Key Components:

  1. PostgreSQL Schema:

    • jobs table: short-running batch tasks (slurm jobs, one-off scripts)
    • deployments table: long-running services (web apps, APIs)
    • job_logs table: job log entries
    • deployment_logs table: deployment log entries
    • Secret stored as hashed field in job/deployment record

  2. Separated API Endpoints:

    • Jobs: /api/[repoType]/[namespace]/[repo]/jobs/* (list, get, update, destroy)
    • Deployments: /api/[repoType]/[namespace]/[repo]/deployments/* (list, get, update, destroy)
    • Creation happens via existing action endpoints

  3. Database Access Pattern:

    • Only k8s-root-actor connects to PostgreSQL
    • Web service calls k8s-root-actor endpoints for all CRUD operations
    • Web service polls for status updates

  4. Secret-Based Security:

    • Each job/deployment creation generates a unique secret token (generated by k8s-root-actor)
    • Secret hash stored in job/deployment record
    • Only PUT log/status endpoints require secret validation
    • Secret validation happens in k8s-root-actor only
    • Web service passes secret through to k8s-root-actor
    • List/get/logs/destroy endpoints check GitLab project write access

  5. GitLab Project Integration:

    • Every job/deployment linked to GitLab project ID (URL-encoded <repoType>/<namespace>/<repoId>)
    • Database filters by gitlab_project_id
    • API layer validates user's write access to project

  6. Destroy Functionality:

    • Soft delete: mark record as destroyed, keep historical data
    • Delete Kubernetes/SLURM resources
    • Update status to destroyed

Pros:

  • Persistent historical data - full audit trail and analytics
  • Clear separation - jobs vs deployments with appropriate lifecycles
  • Improved security - per-resource secrets for log/status updates
  • Better observability - filtering, search, history
  • Single database owner - k8s-root-actor owns all data access
  • Controlled teardown - explicit destroy with history retention

Cons:

  • Additional infrastructure - PostgreSQL setup and maintenance
  • More service hops - web service → k8s-root-actor for data access

Decision

We will implement Option A: PostgreSQL-based Persistent Job/Deployment System.

This solution addresses all identified problems while maintaining architectural clarity. The single database owner pattern (k8s-root-actor) prevents consistency issues, and the polling-based approach maintains simplicity while enabling proper access control.

See also: DD-0012: Implementation details

Consequences

Positive:

  • Full historical data for analytics and audit
  • Clear separation between jobs and deployments
  • Improved security with per-resource secrets for external log/status updates
  • Explicit destroy with history retention
  • Single database owner pattern
  • Polling-based approach maintains simplicity

Negative:

  • PostgreSQL infrastructure requirement
  • More service hops for data queries
  • Web service must validate GitLab write access on each request

Neutral:

  • Web service delegates all data operations to k8s-root-actor
  • k8s-root-actor becomes the single source of truth
  • Existing action YAML system remains largely unchanged
  • Polling replaces real-time notifications (simpler but less responsive)

Open Questions

  1. GitLab Access Caching: How should web service cache GitLab access checks to avoid repeated API calls?

References

Page source

Source snapshot · Revision f928834c8.