Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.
DD-0013: actions.yaml Automation System — Implementation
Related ADR: ADR-0013: actions.yaml Automation System
Status: Draft
Date: 2026-06-04
Authors: TBD
ADR-derived technical notes
These notes are reconciled against the current code (
services/webandservices/k8s-root-actor). Where the ADR description and the implementation diverge, the code wins and the gap is flagged. The Design section below distinguishes as-built from target throughout.
YAML Schema Specification
The actions.yaml file at the root of a GitLab repository defines a map of named actions. The fields the runtime actually consumes today are a subset of the documented schema:
The TypeScript shape that k8s-root-actor deserializes into is ActionConfig in services/k8s-root-actor/src/types/action.ts. Note env is a string[] of KEY=value entries (not a map), and options keys are cpus | ram | gpus | gpuLabel | port | modelName. gpuLabel (k8s-deployment / k8s-inferenceservice only) selects the GPU resource key — for example, nvidia.com/a100 for a whole A100 or nvidia.com/a100-mig-1g.5gb for a MIG slice; an unsupported value is rejected with HTTP 400. There is no Zod/JSON-Schema validation today — js-yaml output is cast to ActionConfig.
Field semantics worth calling out:
parameters— each parameter definition follows the JSON Schema specification (type,title,default,enum, etc.). The web UI builds its input controls from these definitions, and the target validation (see Design §7) checks user-supplied values against the JSON Schema before substitution.repos— an optional list of extra repositories to clone into the workspace. The repository that contains theactions.yaml(the "main repo") is always cloned first, independently of this list. For any repository whose URL is on the Themelio GitLab domain, a short-lived user GitLab PAT is automatically minted and injected (oauth2:<token>@…) so the clone can authenticate; non-Themelio URLs are cloned as-is. See the cloning logic inservices/k8s-root-actor/src/templates/k8s-common/prepare.sh.hbsand Design §4.2.env—KEY=valueentries parsed byparseActionEnv. As of 2026-06-07 all three action types consumeenv:k8s-deployment/k8s-inferenceservicerender it into the manifest env block (yamlq-quoted), andslurm-jobemitsexport NAME='value'lines (formatEnvExports,shq-quoted) into the remotescript.sh.
Parameter substitution (as-built — known gap)
Update (2026-06-07): closed. This naive replace was superseded by validated, context-escaped substitution (
validateParameterValue+shq/yamlqinaction-executor.ts). See Design §7. The snippet below is the original pre-hardening code, kept for history.
Substitution happens in the web service, before k8s-root-actor is called, in services/web/src/utils/actions/action-executor.ts:
This is a naive global string replace on the raw YAML text — values are not shell-quoted, not YAML-escaped, and not validated against the declared parameter type. This is an injection risk and is treated as a gap to close (see Design §7), not the target behavior.
Template system (Handlebars)
Kubernetes manifests are rendered from Handlebars templates under services/k8s-root-actor/src/templates/ (compiled on every render in src/utils/template-renderer.ts — there is no template cache). The templates that actually exist:
- slurm-job:
slurm-job/script.sh.hbs,slurm-job/k8s-job.yaml.hbs - k8s-deployment:
k8s-deployment/app.yaml.hbs, plus sharedk8s-common/prepare.sh.hbs,k8s-common/PersistentVolumeClaim.yaml.hbs,k8s-common/Service.yaml.hbs,k8s-common/VirtualService.yaml.hbs,k8s-common/Gateway.yaml.hbs - k8s-inferenceservice:
k8s-inferenceservice/app.yaml.hbs, plus the samek8s-common/*shared templates
The registered helpers (in template-renderer.ts) are:
There is no base64/toLower helper; base64 encoding is done in TypeScript (Buffer.from(...).toString('base64')) before the value is passed into the template. shq exists but is not applied to user parameters — closing that loop is part of the security target.
Reference resolution
Server-side references are resolved by populateRefs() in services/k8s-root-actor/src/utils/actions-yaml-references.ts. Reference files live under services/k8s-root-actor/src/actions-yaml-references/ (.yaml, .yml, .json); one ships today: gradio-space.yaml. Path traversal is blocked by validatePathInDirectory() (resolved path must stay within the references dir). Nested references are resolved (the merged object is re-processed recursively), but there is no cycle detection.
SLURM specifics
The slurm-job k8s Job SSHes to a single external SLURM cluster using the Kubernetes secret themelio-hpc-ssh (keys: id_ed25519, known_hosts, ssh_user, ssh_host, remote_script_path), mounted read-only at /ssh. The remote script is sbatch-submitted inside a screen session. Both the cluster and secret name are hardcoded in slurm-job/k8s-job.yaml.hbs. This single-cluster/single-account model is a deliberate temporary solution; a future per-project SLURM-account binding is anticipated (see Design §4.1).
GitLab token scopes (as-built)
Implemented (2026-06-07): the web service mints a short-lived GitLab PAT named after the action with scopes ['read_repository', 'write_repository'] only (action-executor.ts). The broad api scope was dropped — actions only clone/push repository content. (Originally as-built this also included api.)
Overview
This document specifies how Themelio turns a declarative actions.yaml file at the root of a GitLab repository into running compute. The web service discovers actions, collects parameters, performs substitution, persists a job record, and proxies execution to k8s-root-actor; k8s-root-actor resolves references, renders Handlebars templates, and applies Kubernetes/SLURM resources for one of three action types (slurm-job, k8s-deployment, k8s-inferenceservice). It documents the system as built today, the target design it should converge to (secure substitution, ADR-0012-backed persistence and destroy, typed parameter UI, UI-guided generation), and the gap between them so an implementer knows exactly what remains.
Most of the "target / gap" items below have since shipped. The sections retain their as-built→target framing for history, but the current state is:
The formal Status stays Draft because ADR-0013 is still Proposed.
Goals and Non-Goals
Goals
- Provide an accurate, implementer-facing specification of the three action types and their end-to-end execution path, naming the actual files and symbols.
- Close the parameter-handling gap: validate parameter values against their declared
typeand quote them safely (using the existingshq/yamlqhelpers) so user input cannot break out of the script or manifest. - Render every declared parameter
type(string,number,boolean,enum,file) with an appropriate input control in the web UI. - Create the ADR-0012 record at action execution and surface per-resource destroy in the UI.
- Auto-generate
actions.yamlduring model and space creation only, driven by the web UI: ahuggingface-format model →k8s-inferenceservice, agradiospace →k8s-deployment.
Non-Goals
- Re-specifying ADR-0012's persistence schema. The
jobs/deployments/*_logstables, secret model, and CRUD/destroy endpoints are defined in DD-0012; this DD only describes how actions integrate with them. - Multi-step workflows or dependencies between actions.
- Multiple SLURM clusters or dynamic cluster/account selection (single hardcoded cluster + single shared SLURM account via the
themelio-hpc-sshsecret). This is a deliberate temporary stance — binding projects/namespaces to specific SLURM accounts is future work. - Per-action quota enforcement. Resource quotas are owned by an external quota mechanism managed by a Waldur site agent at the namespace/SLURM-account level; this system does not cap or duplicate
cpus/ram/gpusenforcement per action. - Container image caching across executions.
Design
1. Architecture & responsibilities
Two services collaborate; the boundary is that the web service pre-resolves the action config and only sends the single selected action to k8s-root-actor.
2. actions.yaml format & parsing
Two parse paths exist, both using js-yaml:
- Listing for the UI —
loadActionsYaml()(services/web/src/utils/actions/action-loader.ts) fetches…/repository/files/actions.yaml/raw?ref={branch}and projects each entry to aCustomAction(name,title,label,parameters). Missing/invalid files degrade gracefully to[]. This is invoked from the resource-detail GET routes (e.g.services/web/src/app/api/models/[namespace]/[repo]/route.ts), so the action list rides along on the model/space payload — there is no dedicated/actions/[branch]endpoint. - Execution —
executeAction()(action-executor.ts) re-fetches the raw YAML at the target branch, substitutes parameters, parses, and selectsactions[actionName].
Target: introduce a shared schema validator (Zod) for ActionConfig used by executeAction before dispatch, returning a 400 with field-level errors instead of casting js-yaml output.
3. Action execution flow (as-built)
4. Action types
All three handlers receive the resolved Action (src/types/action.ts) and base64-encode rendered scripts before embedding them in manifests.
4.1 slurm-job — src/lib/action-slurm-job.ts
Renders slurm-job/script.sh (with repos, pre, post, env as export NAME='value' lines via formatEnvExports, script, and cpus/ram/gpus defaulting to 1/4/0), base64-encodes it, and creates a Kubernetes batch/v1 Job from slurm-job/k8s-job.yaml. The Job container scps the script to the SLURM head node via the themelio-hpc-ssh secret and launches it under screen + sbatch. No PVC, Service, or external URL. Maps to ADR-0012's jobs table.
A single SLURM cluster and a single shared SLURM account back all slurm-job actions, with the cluster and themelio-hpc-ssh secret hardcoded in the template. This is an explicitly temporary arrangement: there is no per-project or per-action cluster/account selection today. The intended future direction is to bind each Themelio project/namespace to a specific SLURM account, so the substitution and secret-resolution paths should be written with that eventual per-project mapping in mind rather than hardcoding assumptions that would be costly to unwind.
4.2 k8s-deployment — src/lib/action-k8s-deployment.ts
Computes a stable name via buildStableActionName([repoType, namespace, repoId, branchName, actionName]) (sanitized prefix + 8-char SHA1), then:
kubectlResourceExists(...)→ throwsK8sResourceAlreadyExistsError(surfaced as 409 conflict) if the deployment exists.- Creates a
10GiPVC ({name}-workspace). - Renders
k8s-common/prepare.shas an init script: always clones the main repo (the one holdingactions.yaml, at{THEMELIO_URL}/{repoType}/{namespace}/{repoId}.git) withoauth2:{gitlabToken}@injected, then clones each entry inrepos— injectingoauth2:{gitlabToken}@for Themelio-hosted URLs and cloning others as-is — and finally runspre. (sync_or_clone_repore-syncs an existing checkout instead of re-cloning.) - Applies
k8s-deployment/app.yaml(init container + main container running the base64 script,envfromparseActionEnv,cpus/ram/gpus/port). - Applies Service, Istio VirtualService, and Gateway (
service-gateway) for an external URL.
Returns { deploymentName, pvcName, url, internalURL }. Maps to ADR-0012's deployments table.
4.3 k8s-inferenceservice — src/lib/action-k8s-inferenceservice.ts
Same stable-name + init-script approach, but applies a KServe InferenceService (k8s-inferenceservice/app.yaml, vLLM serving) with a PVC mounted at /mnt/models, env from parseActionEnv appended to the model-server env block, plus VirtualService and Gateway. Maps to ADR-0012's deployments table.
5. Template system
Templates are read and compiled on every call to loadTemplate (no caching — edits to .hbs files take effect immediately) and rendered with renderTemplate(name, context). Helpers available: join, shq, yamlq, capitalize, toUpperCase (see notes above). Scripts are base64-encoded in TypeScript before templating; manifests are piped to kubectl via stdin by src/utils/kubectlCommands.ts.
Target / gap: route every user-supplied parameter through shq (for shell context) or yamlq (for manifest scalars) at substitution time, so quoting is structural rather than relying on the author of actions.yaml. (Implemented 2026-06-07 — see §7.)
6. Reference resolution
populateRefs() recursively walks the config; any object with a ref key is replaced by the merged content of the referenced file (#/references/<name> or a relative path under src/actions-yaml-references/), with the action's inline keys overriding the reference. Path traversal is rejected (PathTraversalError → 400); a missing file is ReferenceNotFoundError → 404; a parse failure is ReferenceParseError → 500. Nested references resolve. Implemented (2026-06-07): nesting is bounded at a maximum depth of 3 — populateRefs tracks the resolution depth and the visited-path set, throwing ReferenceDepthError past three levels and CircularReferenceError on a revisited path. Shipped references: gradio-space.yaml (a k8s-deployment that bootstraps uv, installs requirements.txt, and runs app.py) and huggingface-inference.yaml (a k8s-inferenceservice for vLLM serving).
7. Parameter substitution & security (gap)
Update (2026-06-07): implemented.
action-executor.tsnow validates each value against its definition and appliesshq(shell) /yamlq(YAML) escaping. The "As-built" text below is the original pre-hardening state, kept for history.
As-built: raw-text regex replace, no quoting, no validation (notes above). A parameter value like "; rm -rf / #" or one containing YAML control characters can break out of the intended context.
Target:
- Validate each provided value against its parameter's JSON Schema definition (
type,enummembership,required); reject unknown parameter keys. - Apply context-appropriate escaping when substituting —
shqfor values landing in shellscript/pre/post,yamlqfor values landing in manifest scalars (options,env). - Perform validation, escaping, and parameter resolution entirely in the web service (
executeAction).k8s-root-actoris never responsible for parameter handling — it receives an already-resolvedactionConfigand the request contract between the two services is unchanged.
8. Persistence integration (ADR-0012)
Per ADR-0012, the three action types collapse into two ADR-0012 resource kinds:
For now slurm-job is the only job type; both Kubernetes action types are deployments. Action execution is the creation point for the corresponding ADR-0012 record. The record schema, secret model, status enums, gitlab_project_id keying, and the read/update/list/destroy endpoints are all defined in DD-0012 and are not restated here.
Update (2026-06-07): implemented (the split moved server-side). The legacy Valkey helpers (
createJobEntry/pushJobLog/getRepoJobs) were removed.executeActionno longer creates any record itself — it proxies the resolved action tok8s-root-actor, which owns the id+secret, branches onactionConfig.type(slurm-job→jobs,k8s-deployment/k8s-inferenceservice→deployments), and persists the correct ADR-0012 record. The "As-built gap" text below predates this.
As-built gap: services/web/src/utils/actions/action-executor.ts predates the ADR-0012 job/deployment split — it unconditionally calls createJobEntry / pushJobLog / getRepoJobs (Valkey-backed "job" records) for every action type, so a k8s-deployment or k8s-inferenceservice is wrongly tracked as a job. Target: executeAction must branch on actionConfig.type and create a job record for slurm-job versus a deployment record for k8s-deployment/k8s-inferenceservice, routing each through the matching ADR-0012 layer rather than the legacy Valkey job helpers.
9. Destroy lifecycle
Update (2026-06-07): implemented for both kinds.
k8s-root-actorexposesDELETE /jobs/{id}andDELETE /deployments/{id}(DD-0012) with per-type resource teardown;DeploymentManager.tsxandJobManager.tsxboth render working Delete buttons. The "As-built" text below predates this.
As-built: no destroy. DeploymentManager.tsx renders a hardcoded disabled delete button; k8s-root-actor exposes no DELETE.
Target: destroy is per-resource (not a separate action type) and runs through DD-0012's destroy flow and DELETE endpoints. The actions-specific obligation is teardown of the exact resources each action type created: for k8s-deployment the Deployment + Service + VirtualService + Gateway + PVC, for k8s-inferenceservice the InferenceService + VirtualService + Gateway + PVC, for slurm-job the Kubernetes Job — deleted by their stored stable names.
10. UI-guided creation
Automatic actions.yaml generation is scoped to two repo types only — model and space — and is driven entirely by the web UI during repository creation. Datasets and generic repos never get an auto-generated actions.yaml.
Update (2026-06-07): implemented.
RepositoryForm.tsxoffers the type/format selector andapi_createActionsFilecommits the pre-filledactions.yamlon create (gradio →k8s-deployment, huggingface →k8s-inferenceservice). The "As-built" text below predates this.
As-built: the model/space create flows (services/web/src/app/(auth)/{models,spaces}/new/page.tsx) only create the repo and a README; no actions.yaml is generated.
Target: the create-model and create-space UIs offer a type/format selector and commit a pre-filled actions.yaml based on the choice. The selectable values are intentionally minimal for now:
Each is a single-choice selector today (gradio for spaces, huggingface for models) but is modelled as an enumeration so additional space types / model formats can be added later without reworking the flow. The committed actions.yaml is immediately runnable, and the user can edit it afterward.
11. Parameter input UI
Update (2026-06-07): implemented.
ActionParametersModal.tsxnow renders dedicated controls per type (text, number, checkbox, select, file upload) with default pre-fill and client-siderequired/enumvalidation. The "As-built" text below predates this.
As-built: ActionParametersModal.tsx renders type={param.type === 'string' ? 'text' : param.type}, so only string (and partially number) produce valid inputs; boolean, enum, and file emit invalid HTML type attributes. No required-field enforcement or default pre-fill.
Target: dedicated controls derived from each parameter's JSON Schema — text (string), number input (number/integer), toggle/checkbox (boolean), select for a constrained enum, file upload (file) — with default pre-fill and client-side required/enum validation mirroring the server-side validation from §7.
Data Model
Current ActionConfig (k8s-root-actor)
The web-side CustomAction/ActionParameter shapes (services/web/src/utils/actions/action-types.ts) carry only the UI-relevant fields (name, title, label, parameters).
Target: a validated schema (Zod) sharing these field names, additionally typing parameters[*] as JSON Schema parameter definitions (type, title, default, required, enum) so both the UI and executeAction validate against one source of truth.
Persistent records (jobs, deployments, job_logs, deployment_logs) and their status enums are owned by ADR-0012 — see DD-0012 § Data Model.
API Design
Web service
Execute an action
POST /api/{models|datasets|spaces|repos}/[namespace]/[repo]/action/[branchName]
Request:
Response (proxied from k8s-root-actor):
Errors: 401 (missing PRIVATE-TOKEN), 409 (A job for this action is already running), 500 (action not found / file not found / k8s failure).
Listing actions is not a standalone endpoint — actions are embedded in the resource-detail GET payload (GET /api/models/[namespace]/[repo], etc.) via loadActionsYaml.
k8s-root-actor
POST /:repoType/:namespace/:repoId/:branchName/action
Request:
Responses:
Target additions: the action endpoint also creates the persisted record, and destroy is served by dedicated DELETE endpoints — both contracts are defined in DD-0012.
Testing Strategy
Unit
populateRefs:#/references/and relative resolution; nested refs up to depth 3 (and rejection past it, which also covers cycles);PathTraversalErroron../escapes;ReferenceNotFoundError/ReferenceParseError.- Parameter validation + escaping (target §7): values are checked against their JSON Schema (
enummembership rejected when out of range,requiredenforced,typecoerced/validated); shell/YAML metacharacters in values are neutralized byshq/yamlq. buildStableActionName/sanitizeK8sName: deterministic, DNS-safe, ≤ name limits.- Template rendering: each
*.hbsrenders valid YAML for representative contexts.
Integration (local Minikube)
- Each action type end-to-end:
slurm-jobcreates abatch/v1Job;k8s-deploymentcreates PVC + Deployment + Service + VirtualService + Gateway and serves at the external URL;k8s-inferenceservicecreates a KServe InferenceService. - Re-running a live
k8s-deploymentreturns 409. - Target: action execution creates the expected ADR-0012 record, and destroy removes exactly the resources the action created (per §9). Record/status assertions belong to DD-0012's test suite.
Manual acceptance criteria
- A repo with
actions.yamlshows its actions inResourceActions; a parameterized action opens the modal with correct controls per type; submitting runs the action and the UI redirects to#jobswhere status updates appear. - A malicious parameter value cannot alter the executed command or manifest (target).
Deployment & Rollout
Phased, ordered by risk and by dependency on ADR-0012 landing first:
- Security hardening (no new infra): add
ActionConfig/parameter schema validation and safeshq/yamlqsubstitution in the web service; drop the GitLab PATapiscope (mint withread_repository/write_repositoryonly); enforce the depth-3 reference nesting limit. - Persistence + destroy (requires ADR-0012): adopt the ADR-0012 record layer for action jobs/deployments; implement per-action resource teardown on destroy; enable the destroy button.
- Typed parameter UI: ship boolean/enum/file controls and client-side validation in
ActionParametersModal. - UI-guided creation: generate
actions.yamlduring model and space creation only — ahuggingfacemodel →k8s-inferenceservice, agradiospace →k8s-deployment.
Each phase is independently shippable; phase 2 must not start before ADR-0012's schema and endpoints exist.
References
- ADR-0013: actions.yaml Automation System — decision and rationale
- ADR-0012: Job and Deployment Management System — persistence decision
- DD-0012: Job and Deployment Management System — Implementation — PostgreSQL schema, secret model, CRUD/destroy endpoints
- KServe InferenceService CRD
- SLURM Documentation
- Handlebars.js
- In-repo:
services/web/src/utils/actions/{action-executor,action-loader,action-route-handler}.ts,services/web/src/components/{actions/ResourceActions,repo/ActionParametersModal,jobs/JobManager,deployments/DeploymentManager}.tsx,services/k8s-root-actor/src/index.ts,services/k8s-root-actor/src/lib/action-*.ts,services/k8s-root-actor/src/utils/{template-renderer,actions-yaml-references}.ts,services/k8s-root-actor/src/templates/,services/k8s-root-actor/src/actions-yaml-references/gradio-space.yaml
Page source
Source snapshot · Revision f928834c8.