--- title: "ADR-0012: Job and Deployment Management System" --- # ADR-0012: Job and Deployment Management System **Status:** Proposed **Date:** 2026-06-02 **Deciders:** TBD ## Context The current action/job system in Themelio has several architectural limitations that hinder scalability and operational efficiency. ### Current State The current system consists of: 1. **Web Service (Next.js)** handles user-facing action execution via: - Three identical route handlers for models/repos/spaces delegating to `handleActionRoute` - Job tracking via Valkey (Redis) with 24-hour TTL - Endpoints: `PUT /api/jobs/[id]`, `GET /api/jobs/[id]`, `GET /api/jobs/namespace/[namespace]` 2. **k8s-root-actor (Fastify)** creates Kubernetes resources: - Single endpoint: `POST /:repoType/:namespace/:repoId/:branchName/action` - Three action types: `slurm-job`, `k8s-deployment`, `k8s-inferenceservice` - No direct database access 3. **Data Flow**: - User triggers action → web service fetches `actions.yaml` from GitLab - Creates temporary GitLab PAT with broad scopes (read_repository, write_repository, api) - Creates job entry in Valkey (24-hour TTL) - Calls k8s-root-actor which creates Kubernetes resources - Logs pushed back via `/api/jobs/[id]` ### Current Problems **1. Ephemeral Storage** - All job data stored in Valkey with 24-hour TTL - No historical data beyond 24 hours - No archiving or audit trail - Lost data on Valkey restart/crash **2. No Clear Separation Between Jobs and Deployments** - Both "jobs" (short-running batch tasks) and "deployments" (long-running services) use the same `actions.yaml` system - UI/UX confusion for users - Inappropriate lifecycle management (deployments shouldn't have "completion") **3. Limited Observability** - No job/deployment history - No filtering or search capabilities - No analytics or reporting **4. No Destroy Functionality** - Jobs/deployments cannot be explicitly destroyed - Resources may orphan if tracking lost - No controlled teardown **5. Security Concerns** - Temporary PATs created with overly broad scopes - No fine-grained authorization for job/deployment operations - Anyone with the job ID can access logs via current endpoints - No per-resource authentication tokens **6. Unified Endpoints** - Jobs and deployments mixed in same API structure - No differentiation in queries or listing **7. Direct Web Service Database Access** - Web service needs Valkey for jobs AND would need PostgreSQL for persistence - k8s-root-actor has no database access --- ## Considered Options ### Option A: PostgreSQL-based Persistent Job/Deployment System **Architecture Overview:** Introduce a PostgreSQL database for persistent storage of jobs and deployments with a clear separation of concerns. Only k8s-root-actor accesses the database directly. Web service delegates all data operations to k8s-root-actor. **Key Components:** 1. **PostgreSQL Schema:** - `jobs` table: short-running batch tasks (slurm jobs, one-off scripts) - `deployments` table: long-running services (web apps, APIs) - `job_logs` table: job log entries - `deployment_logs` table: deployment log entries - Secret stored as hashed field in job/deployment record 2. **Separated API Endpoints:** - Jobs: `/api/[repoType]/[namespace]/[repo]/jobs/*` (list, get, update, destroy) - Deployments: `/api/[repoType]/[namespace]/[repo]/deployments/*` (list, get, update, destroy) - Creation happens via existing action endpoints 3. **Database Access Pattern:** - Only k8s-root-actor connects to PostgreSQL - Web service calls k8s-root-actor endpoints for all CRUD operations - Web service polls for status updates 4. **Secret-Based Security:** - Each job/deployment creation generates a unique secret token (generated by k8s-root-actor) - Secret hash stored in job/deployment record - Only PUT log/status endpoints require secret validation - Secret validation happens in k8s-root-actor only - Web service passes secret through to k8s-root-actor - List/get/logs/destroy endpoints check GitLab project write access 5. **GitLab Project Integration:** - Every job/deployment linked to GitLab project ID (URL-encoded `//`) - Database filters by gitlab_project_id - API layer validates user's write access to project 6. **Destroy Functionality:** - Soft delete: mark record as destroyed, keep historical data - Delete Kubernetes/SLURM resources - Update status to destroyed **Pros:** - **Persistent historical data** - full audit trail and analytics - **Clear separation** - jobs vs deployments with appropriate lifecycles - **Improved security** - per-resource secrets for log/status updates - **Better observability** - filtering, search, history - **Single database owner** - k8s-root-actor owns all data access - **Controlled teardown** - explicit destroy with history retention **Cons:** - **Additional infrastructure** - PostgreSQL setup and maintenance - **More service hops** - web service → k8s-root-actor for data access --- ## Decision **We will implement Option A: PostgreSQL-based Persistent Job/Deployment System.** This solution addresses all identified problems while maintaining architectural clarity. The single database owner pattern (k8s-root-actor) prevents consistency issues, and the polling-based approach maintains simplicity while enabling proper access control. **See also:** [DD-0012: Implementation details](../content/dd/0012-job-deployment-management-system.md) --- ## Consequences **Positive:** - Full historical data for analytics and audit - Clear separation between jobs and deployments - Improved security with per-resource secrets for external log/status updates - Explicit destroy with history retention - Single database owner pattern - Polling-based approach maintains simplicity **Negative:** - PostgreSQL infrastructure requirement - More service hops for data queries - Web service must validate GitLab write access on each request **Neutral:** - Web service delegates all data operations to k8s-root-actor - k8s-root-actor becomes the single source of truth - Existing action YAML system remains largely unchanged - Polling replaces real-time notifications (simpler but less responsive) --- ## Open Questions 1. **GitLab Access Caching:** How should web service cache GitLab access checks to avoid repeated API calls? --- ## References - [DD-0012: Job and Deployment Management System — Implementation](../content/dd/0012-job-deployment-management-system.md)