--- title: "DD-0021: Themelio CLI — Implementation" --- # DD-0021: Themelio CLI — Implementation **Related ADR:** [ADR-0021: Themelio Command-Line Interface](/adr/0021-themelio-cli) **Status:** Draft **Date:** 2026-07-06 **Authors:** TBD ## ADR-derived technical notes Implementation detail that surfaced while authoring ADR-0021. A complete working specification also exists at `plans/themelio-cli/SPEC.md` and is the primary input for this DD. - **Package / stack:** Python 3.11+, console entry point `themelio`, distributed via a `curl … | sh` standalone installer **and** `pipx install themelio-cli` / `uvx themelio`. Stack: **Typer** (commands), **httpx** (HTTP + SSE), **rich** (tables/progress), **PyYAML** (pre-parse of `actions.yaml`), **keyring** (token storage). `git`/`git-lfs` are the user's, not a runtime dependency. - **Auth / config precedence:** `--host`/`--token` flags → `THEMELIO_HOST`/ `THEMELIO_TOKEN` → `~/.themelio/config` (or keyring) → built-in default host `https://dev.themelio.pharos.grnet.gr`. Single `PRIVATE-TOKEN` header. - **git credential helper:** `git-setup` prints `git config --global credential.https://.helper '!themelio git-credential'` by default; `--apply` writes it. Hidden `themelio git-credential` speaks git's credential-helper protocol (emits `username=oauth2` + `password=`). - **Content commands print git**, never shell out; no single-file API shortcut. - **New endpoint** `POST /api/actions/validate` reuses the web action library and delegates `ref:` resolution to `k8s-root-actor` (`populateRefs`). ## Overview This document specifies how to build the `themelio` command-line client: a standalone Python package that is a thin, typed client of the Themelio web API, with a `themelio ` grammar, keyring-backed `api`-scoped PAT auth, git hand-off for content transfer (via a `git-setup` credential helper), and action-driven compute that yields observe-only jobs and deployments. It also specifies the one new backend endpoint the CLI depends on, `POST /api/actions/validate`. ## Goals and Non-Goals ### Goals - Installable two ways: a zero-prerequisite `curl … | sh` standalone installer, and `pipx install themelio-cli` / `uvx themelio` for Python users — both yielding a `themelio` command covering auth, repository lifecycle, actions, and job/deployment observation against the existing web API. - Resource-first command grammar with the shared verb set, implemented cleanly on top of Typer/Click. - One credential (`api` PAT) for both API and git; stored in the OS keyring. - Scriptable output (`--json`, `--quiet`) and stable exit codes. - The `POST /api/actions/validate` endpoint, reusing existing web/k8s-root-actor logic. ### Non-Goals - Moving repository bytes through the CLI (delegated to the user's `git`/`git-lfs`). - A browser device-login flow (deferred; see Open Questions). - Repo deletion, file-level commits, branches/tags, and a local content cache (unsupported by the backend today). ## Design ### 1. Dependencies | Package | Purpose | |---|---| | `typer` (with `rich` extra) | CLI framework (Click under the hood); type-hinted commands, help, completion | | `httpx` | HTTP client + streaming (SSE) with connection reuse and timeouts | | `rich` | Human-readable tables, progress, colored errors (pulled in by Typer) | | `pydantic` v2 | Typed models for API responses; validation of the config file | | `PyYAML` | Offline pre-parse of `actions.yaml` before `actions validate` | | `keyring` | OS keyring storage of the token (Secret Service / Keychain / WinCred) | | `tomli-w` | Write `~/.themelio/config` (read via stdlib `tomllib`) | | `platformdirs` | Resolve config/cache dirs cross-platform | | Dev: `pytest`, `respx`, `pytest-httpx`, `ruff`, `mypy` | Tests (httpx mocking), lint, type-check | | Build: `hatchling` (wheel/sdist) · `pyinstaller` (standalone single-file binaries) | PyPI package + per-platform executables for the curl installer | Build backend: **hatchling**, `src/` layout, `requires-python = ">=3.11"`. ### 2. File structure ``` themelio-cli/ ├── pyproject.toml # metadata, deps, [project.scripts] themelio = "themelio.cli:app" ├── README.md ├── src/themelio/ │ ├── __init__.py # __version__ │ ├── __main__.py # enables `python -m themelio` │ ├── cli.py # root Typer app; global options; assembles sub-apps │ ├── config.py # Settings model + host/token resolution & persistence │ ├── keyring_store.py # keyring get/set/delete with file fallback │ ├── client.py # ThemelioClient: httpx wrapper, PRIVATE-TOKEN, error mapping │ ├── errors.py # exception hierarchy ↔ exit codes (plain exceptions) │ ├── context.py # AppContext (settings/fmt/repo) + repo resolution helpers │ ├── output.py # render()/detail(): rich tables vs --json/--quiet │ ├── normalize.py # strip GitLab plumbing; expose namespace/repo refs │ ├── models.py # pydantic response models (Repo, Job, Deployment, Action…) │ ├── restypes.py # resource-type constants + plural/singular mapping │ ├── grammar.py # ResourceGroup: the parser │ ├── gitcmd.py # repo git URL, clone/upload command builders │ ├── sse.py # SSE line iterator over httpx streaming (future --follow) │ └── commands/ │ ├── __init__.py │ ├── auth.py # login · logout · whoami · token · list-tokens · revoke │ ├── git_setup.py # git-setup (+ hidden git-credential) │ ├── resource.py # make_resource_app(): shared verbs per {type} │ ├── actions.py # make_actions_app(): ls · describe · run · validate │ ├── compute.py # make_compute_app(): shared jobs/deployments verbs │ ├── jobs.py # thin wrapper: make_compute_app("jobs", …) │ └── deployments.py # thin wrapper: make_compute_app("deployments", …) └── tests/ ├── conftest.py # temp config, disabled-keyring, invoke helper (respx) ├── test_grammar.py # repo-first vs repo-less parsing ├── test_config.py # precedence + keyring fallback + 0600 ├── test_auth.py · test_resource.py · test_actions.py ├── test_jobs.py · test_deployments.py └── test_git_setup.py # printed config + credential-helper protocol I/O ``` The error hierarchy is plain exceptions (not `click.ClickException`) because the pinned Typer vendors its own Click; a root `RootGroup(TyperGroup)` catches `ThemelioError` in `invoke()` and raises `typer.Exit(code)`, giving stable exit codes under both real runs and `CliRunner`. ### 3. Command grammar (`grammar.py`) The resource-first grammar `themelio ` is not Click's default (`group verb args`). A repo reference always contains exactly one `/`; verbs never do. We exploit that with a custom group that pops a leading repo token into the context before resolving the verb: ```python import re from typer.core import TyperGroup REPO_RE = re.compile(r"^[^/\s]+/[^/\s]+$") class ResourceGroup(TyperGroup): def parse_args(self, ctx, args): # `models vkefallinos/bert info` → repo popped, subcommand = info # `models ls` / `models actions validate` → nothing popped if args and REPO_RE.match(args[0]) and args[0] not in self.commands: ctx.ensure_object(dict)["repo"] = args.pop(0) return super().parse_args(ctx, args) ``` - Each resource sub-app is `typer.Typer(cls=ResourceGroup)`; commands read the repo via `ctx.obj["repo"]` (Typer injects `ctx: typer.Context`). - `ls` (no repo) and the `actions`/`jobs`/`deployments` groups dispatch normally because their first token has no `/`. - Those sub-apps are mounted on the resource app, so `models vkefallinos/bert actions run train` (or `… jobs ls`) pops the repo at the resource level and reaches the nested verb with `ctx.obj["repo"]` set; `models actions validate f` never pops a repo, and repo-scoped verbs raise a usage error (exit 2) when `repo` is absent. ### 4. Root app & resource factory `cli.py` builds the root Typer app, defines global options as a top-level callback, and mounts sub-apps: ```python app = typer.Typer(no_args_is_help=True) @app.callback() def main(ctx, host: str = typer.Option(None, envvar="THEMELIO_HOST"), token: str = typer.Option(None, envvar="THEMELIO_TOKEN"), json_: bool = typer.Option(False, "--json"), quiet: bool = typer.Option(False, "-q", "--quiet"), yes: bool = typer.Option(False, "-y", "--yes")): ctx.obj = AppContext(settings=resolve_settings(host, token), fmt=..., assume_yes=yes) app.add_typer(auth.app, name="auth") app.command("git-setup")(git_setup.git_setup) for t in ("models", "datasets", "spaces", "repos"): app.add_typer(make_resource_app(t), name=t) app.add_typer(jobs.app, name="jobs") # top-level (--repo / --namespace) app.add_typer(deployments.app, name="deployments") ``` `models`/`spaces`/`repos` additionally mount the same `jobs.app`/`deployments.app` as repo-scoped shortcuts (e.g. `themelio models vkefallinos/bert jobs ls`), so the verbs run with the repo already in context. `make_resource_app(resource_type)` (in `commands/resource.py`) returns a `ResourceGroup` Typer with the shared verbs, closing over `resource_type` so all four types share one implementation. `has_compute = type in {models, spaces, repos}` mounts the `actions`, `jobs`, and `deployments` sub-apps (datasets get none); `has_search = type in {models, datasets}` toggles `--search`. The `jobs`/ `deployments` sub-apps are the **same** Typer apps as the top-level `jobs.app`/ `deployments.app` — mounting them under a `ResourceGroup` means the repo is popped into `ctx.obj["repo"]` before their verbs run, so those verbs read the repo from context instead of requiring `--repo`. Verb → endpoint mapping: | Verb | Call | |---|---| | `ls [--search q] [--author] [--limit] [--offset]` | `GET /api/{type}` (author → `?namespace=`; with `--search`, `…/search?search=`) | | `create [--private] [--desc] [--type] [--sdk]` | `POST /api/repos/create` (split `ns/repo`; **singular** `type`, `name`, `namespace`) | | `fork --to ` | `POST /api/fork/{type}s/{ns}/{repo}` (catch-all), body `{ namespacePath }` | | `info` | `GET /api/{type}/{ns}/{repo}` | | `card [--metadata] [--text]` | `GET /api/{type}/{ns}/{repo}/readme` | | `ls-files [--revision] [--path]` | `GET /api/{type}/{ns}/{repo}/tree/{rev}` | | `settings get` / `settings set …` | `GET` / `PUT /api/{type}/{ns}/{repo}/settings` | | `download` / `upload` | print git commands (§7) | ### 5. HTTP client (`client.py`) A thin wrapper around one reused `httpx.Client`: ```python class ThemelioClient: def __init__(self, settings): self._c = httpx.Client(base_url=settings.host, headers={"PRIVATE-TOKEN": settings.token}, timeout=httpx.Timeout(30.0, read=None)) def request(self, method, path, **kw) -> Any: r = self._c.request(method, path, **kw) if r.is_success: return r.json() if r.content else None raise error_from_response(r) # maps status → typed exception def stream_sse(self, path): ... # see §9 ``` `error_from_response` maps `401/403 → AuthError`, `404 → NotFoundError`, `409 → ConflictError`, `400 → ValidationError` (carrying `fieldErrors`), else `ThemelioError`. Requests without a resolved token short-circuit to `AuthError` ("run `themelio auth login`"). ### 6. Config & credential storage (`config.py`, `keyring_store.py`) - `Settings` is a pydantic model (`host: HttpUrl`, `token: SecretStr | None`). - `resolve_settings()` applies the ADR precedence: flags → env → config file → default host `https://dev.themelio.pharos.grnet.gr` (token has no default). - Config file `~/.themelio/config` (TOML, mode `0600`) stores `host` only when the token lives in the keyring; if no keyring backend exists it also stores `token`. - `keyring_store` wraps `keyring.get/set/delete_password("themelio", host)`, catching `keyring.errors.NoKeyringError` to fall back to the config file. `logout` clears both. ### 7. Content transfer (`gitcmd.py`, `commands/resource.py`, `git_setup.py`) - `repo_git_url(host, type, ns, repo) -> f"{host}/{type}s/{ns}/{repo}.git"`. - `download`/`upload` **print** the exact commands (never execute git): clone → `git clone `; upload → `cd && git add -A && git commit -m … && git push`, prefixed with a one-time `themelio git-setup --apply && git lfs install` hint. - `git-setup` (`commands/git_setup.py`): - Default: print `git config --global credential.https://.helper '!themelio git-credential'`. - `--apply`: run it via `subprocess.run([... "git", "config", ...])`. - Hidden `themelio git-credential` command implements the credential-helper protocol: read `key=value` lines from stdin until blank; if `host` matches the configured host, print `protocol=https`, `host=`, `username=oauth2`, `password=`. `store`/`erase` verbs are no-ops (the token is owned by `themelio auth`, not git). ### 8. Actions (`commands/actions.py`) - `actions ls` / `describe` read the `actions` array embedded in `GET /api/{type}/{ns}/{repo}` (there is no list endpoint); `describe` renders the whole action as a panel — its type, the job/deployment it yields, and every parameter (type, required, default, enum options, description). Note the info payload exposes only `name/title/label/type/parameters` per action; action-level `options` (cpus/gpus/ image/script) are not returned, so `describe` cannot show them. - `actions run [--param k=v]… [--branch main] [--follow]` → `POST /api/{type}/{ns}/{repo}/action/{branch}` with `{action, parameters}` (route exists for **models/spaces/repos** only). The web route **relays the k8s-root-actor response verbatim** (201 create / 409 already-running); the id is embedded in that upstream payload, so the CLI extracts it defensively (tries `id`/`jobId`/`deploymentId`/`name` and a nested `job`/`deployment`). Whether it is a **job** or **deployment** is decided by the action `type` (`slurm-job` → job, else deployment); on `--follow` the CLI points at the relevant `logs` command. - `actions validate [file=./actions.yaml]` → offline `yaml.safe_load` sanity check, then `POST /api/actions/validate` with the raw content; prints `errors`/`warnings` and exits non-zero when `valid` is false. `actions` is read/run-only: there is no `init` verb. Authoring `actions.yaml` (and any commit to it) is done in the repository itself, not through the CLI. ### 9. Jobs, deployments & log streaming (`commands/jobs.py`, `deployments.py`, `sse.py`) - Observe-only verbs: `ls [--repo|--namespace] [--status]`, `get --repo`, `logs --repo [--follow]`, `wait --repo [--timeout]`, `destroy --repo`. These map to the repo-scoped `…/{jobs,deployments}[/{id}]` routes and the namespace routes `/api/{jobs,deployments}/namespaces/{ns}`. - **Repo resolution:** each verb resolves the repo from `ctx.obj["repo"]` first (set when reached via the ` jobs|deployments …` shortcut), else the `--repo` option. The same `jobs.app`/`deployments.app` therefore serve both the top-level and the per-resource shortcut with no duplicated code; `--namespace` listing is only valid on the top-level form. - `wait` is client-side polling of `get` until a terminal status, honoring `--timeout`; exit non-zero if any target failed. - `logs`: there is **no separate log-stream endpoint**; the `get` response (`…/{jobs,deployments}/{id}`) embeds an ordered `logs` array of `{id, timestamp, level, message, source_status}`. `logs` reads that array and prints each line timestamped, coloring `error`/`warn` and dimming empty-message status transitions. This is identical for **jobs and deployments** — both k8s-root-actor `GET /jobs/:id` and `GET /deployments/:id` return the ordered logs. `--follow` re-fetches every 2 s and prints only lines not seen (tracked by `id`), stopping at a terminal status. `--json` emits the raw `logs` array. (The `spaces/{ns}/{repo}/logs/run` route — a hardcoded stub — is **not** used. `sse.py` remains for a future true-stream endpoint, when `--follow` can switch from polling to SSE.) ### 10. Output & exit codes (`output.py`, `errors.py`) - `render(data, *, columns)` picks human tables (rich) by default, raw JSON with `--json`, and id-only lines with `--quiet`. - **Platform-native view (`normalize.py`).** The web API leaks its GitLab backing store — list items carry a GraphQL global id (`gid://gitlab/Project/74`) rather than a usable reference, plus `gitlabUrl`/`namespaceName`/`cardData`/`slug` and, on jobs/ deployments, `gitlab_project_id`/`gitlab_user_id`. The CLI strips these in **both** human and `--json` output and surfaces the canonical `namespace/repo` reference (from the `slug`) as `id`, so `ls` leads with a copy-pasteable `Repository` column and `-q` prints refs usable in the next command. Topics/tags are preserved verbatim. Jobs/deployments tables read the real API fields (`action_name` → **Action**, `created_at` → **Created**). - **Collection vs repo verbs.** `ls` is a collection verb (whole type); giving it a repo (` ls`) is rejected with exit 2 pointing at `ls-files`. On the `jobs`/`deployments` repo-scoped mount, `--repo`/`--type` are hidden (the repo comes from the path and the type is fixed); they remain on the top-level mount. - The root `RootGroup.invoke()` catches the exception hierarchy and converts it to a rendered message + `typer.Exit(code)`: `0` ok · `2` usage · `3` auth · `4` not-found · `5` conflict · `6` validation (renders `fieldErrors` per parameter) · `1` other. ## Data Model Config file (`~/.themelio/config`, TOML): ```toml host = "https://dev.themelio.pharos.grnet.gr" token = "…" # present only when no OS keyring backend is available ``` Keyring entry: service `themelio`, username ``, password ``. Pydantic response models in `models.py` mirror the API payloads the CLI renders (`Repo`, `FileEntry`, `TokenInfo`, `WhoAmI`, `Job`, `Deployment`, `Action`, `ActionParameter`, `ValidationResult`). Unknown fields are ignored (`model_config = ConfigDict(extra="ignore")`) so the CLI tolerates API additions. ## API Design ### New: `POST /api/actions/validate` (web service) New route `services/web/src/app/api/actions/validate/route.ts`. Auth: standard `PRIVATE-TOKEN` (no repo scope — validates content). Request: ```json { "content": "", "action": "train-model" } ``` Response: ```json { "valid": false, "errors": [{ "action": "train-model", "path": "parameters.epochs.type", "message": "unsupported type 'number' (use string|enum)" }], "warnings": [{ "action": "train-model", "path": "script", "message": "$foo references an undefined parameter" }] } ``` Implementation (`src/lib/actions/action-validator.ts` holds the reusable logic; the route orchestrates): 1. Require `private-token` (401 if absent). Parse via a new `parseActionsForValidation()` that calls `js-yaml` **directly** and surfaces errors — unlike the executor's `parseActionsYaml`/`loadActionsYaml`, which swallow them and return `{}`. Malformed YAML or a non-mapping top level → `400`. 2. `validateActionsStructure()` reuses `action-parameters.ts` (`SUPPORTED_PARAMETER_TYPES`, `validateParameterValue`) and `action-types.ts` for structural + per-parameter checks: parameter `type` ∈ `string|enum`, enum `options`/`enum` present and default a member, action `type` ∈ `slurm-job|k8s-deployment|k8s-inferenceservice`, and `$name` (incl. hyphens) reference warnings (undefined refs / unused params). Invalid **content** returns `200` with `valid:false` + issues (only malformed input is a 400). 3. For each action carrying a `ref:`, `kubeFetch('/actions/validate', …)` (base `K8S_ROOT_ACTOR_URL` from `src/config.ts`) posts `{ actionConfig }` to the **new `k8s-root-actor` route `POST /actions/validate`** (`src/routes/actions-validate.ts`, registered in `src/index.ts`). That route runs `populateRefs` (`src/utils/actions-yaml-references.ts`), maps its error classes to structured `{ path, message }` issues, and checks the resolved `type` plus a config-free GPU coherence rule (`gpuLabel` required when `gpus` set). The `gpuLabel` allow-list check is intentionally left to the executor at run time so the validate route stays decoupled from DB-bound config. Its issues merge into the web response. The web authoring UI ("Setup deployment") can call the same endpoint, keeping one source of truth. ### Existing endpoints consumed `whoami-v2` (unwrapped `UserProfile`); `settings/tokens[/{id}]` (POST body `{name, scopes[], expiresAt?}` → `{message, token}`; GET → bare array); `{type}` (list; author is `?namespace=`); `{type}/search?search=` (models/datasets only — **no** `spaces/search`); `{type}/{ns}/{repo}[/readme|/tree/{rev}|/settings]`; `repos/create` (singular `type`); `fork/{…path}` (catch-all, body `{namespacePath}`); `{type}/{ns}/{repo}/action/{branch}` (models/spaces/repos; response relayed from k8s-root-actor); and the `{jobs,deployments}` repo- and namespace-scoped routes, whose `…/{id}` detail embeds the ordered `logs` array the `logs` verb reads. The `spaces/{ns}/{repo}/logs/run` stub is **not** consumed. No changes required to any of these; the CLI is coded against these verified shapes. ## Testing Strategy - **Grammar** (`test_grammar.py`): `ResourceGroup.parse_args` pops repos only when a `/` token precedes a verb; `ls` and `actions validate` never pop; missing-repo on a repo-scoped verb exits 2. - **Commands** (`test_resource/actions/jobs/deployments.py`): drive Typer's `CliRunner` with `respx`-mocked httpx routes; assert method+path+body and rendered output for human/`--json`/`--quiet`, and exit-code mapping for 401/404/409/400. - **Config** (`test_config.py`): precedence order; keyring fallback via a monkeypatched `NoKeyringError`. - **git-setup** (`test_git_setup.py`): printed config string; `git-credential` reads a stdin request and emits `username=oauth2` + password for the matching host only. - **Endpoint**: web-side `action-validator.test.ts` (node:test) covering malformed/non-mapping YAML, unsupported action/parameter types, enum options/default, `$name` warnings, and the single-action filter; plus k8s-root-actor `routes/actions-validate.test.ts` (Fastify `inject`) exercising `populateRefs` success, a missing ref, an unsupported type, and a missing body. - CI: CLI — `ruff`, `mypy`, `pytest` (3.11/3.12) + wheel build; backend — `pnpm test` in `services/web` and `services/k8s-root-actor`. ## Deployment & Rollout - **Package:** `themelio-cli` in its own repository (Python is outside the pnpm monorepo build). Two distribution channels, both producing the same `themelio` command: - **Python users:** wheel/sdist on the internal index / GitLab Package Registry — `pipx install themelio-cli` or `uvx themelio` (isolated environments). - **Zero-prerequisite:** `curl -LsSf https:///cli/install.sh | sh` (and `install.ps1` for Windows). The script detects OS/arch, downloads the matching **PyInstaller** single-file executable from the release artifacts, installs it to `~/.local/bin/themelio` (`%LOCALAPPDATA%` on Windows), `chmod +x`, and prints PATH guidance. `themelio --version` self-reports the channel. - **CI build matrix** (GitLab CI) produces `pyinstaller --onefile` binaries for linux `x86_64`/`aarch64`, macOS `arm64`/`x86_64`, and Windows `x86_64`, plus the wheel; the `install.sh`/`install.ps1` scripts and binaries are attached to each tagged release. The credential helper works under both channels because git invokes `themelio git-credential` from PATH regardless of how it was installed. - **Backend:** the `POST /api/actions/validate` route ships with the normal web deploy; the companion `k8s-root-actor` `POST /actions/validate` route ships with its service. Both are additive and backward-compatible. - **Prerequisite already shipped:** the Settings token UI exposes the `api` scope. - **Phasing** (from the spec): M1 read + `auth` + `git-setup`; M2 write + actions (needs the validate endpoint); M3 log streaming, keyring, completions, polish. - **Rollback:** removing the CLI has no server impact; the two new endpoints can be withdrawn independently since nothing else depends on them. ## Open Questions 1. ~~Distribution channel?~~ **Resolved:** ship both — PyInstaller single-file binaries behind a `curl … | sh` installer (à la `hf`) **and** a wheel via `pipx`/`uvx`. Open sub-question: which host serves `/cli/install.sh` and the release artifacts (the web host vs the GitLab release page). 2. Should `POST /api/actions/validate` also accept a repo + `ref` to validate an already-committed `actions.yaml`, in addition to raw content? 3. Device-login flow (reusing OIDC + impersonation tokens) as a future `auth login` mode — out of scope for v1, tracked in ADR-0021. ## References - [ADR-0021: Themelio Command-Line Interface](/adr/0021-themelio-cli) - [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system) - [ADR-0013: actions.yaml Automation System](/adr/0013-actions-yaml-automation-system) - Full command specification: `plans/themelio-cli/SPEC.md` - [Typer](https://typer.tiangolo.com/) · [httpx](https://www.python-httpx.org/) · [keyring](https://github.com/jaraco/keyring) · [Rich](https://rich.readthedocs.io/) - [git credential helper protocol](https://git-scm.com/docs/gitcredentials)