# Pharos dev box setup Run these commands from your workstation, in the repository root. The [developer guide](../../docs/development.md) covers editing, takeover, Tilt, hand-back and committed delivery. These commands implement decision 0045. ## Prerequisites Install Git, Make, SSH and uv on the workstation. All root deployment commands transfer the committed source to a Linux SSH runner; follow the [runner preparation](../../docs/development.md#macos-delivery). The runner needs rootful Docker Engine, Git, Python 3, Make and Bash. Its pinned tools container provides Ansible, kubectl, Helm and the deployment libraries. Workstation Docker is required only for Docker takeover, not deployment. Copy `.env.example` to `.env` in the repository root to save your workstation defaults; see [workstation settings](../../docs/development.md#workstation-defaults). Docker Desktop and Colima are qualified for macOS workstation takeover; follow the developer guide's [Colima file-sharing setup](../../docs/development.md#colima-file-sharing). Use your own Debian 12 AMD64 VM with systemd and passwordless sudo for the SSH user. The qualified VM shape is 16 vCPU, 32 GiB RAM, an 80 GiB root disk and a separate empty 100 GiB disk for Ceph. Test SSH access and verify its host key first. The tools container keeps Ansible control sockets in its private `/tmp` so long checkout paths remain usable. ```sh ssh your-vm-alias true make devbox-setup TARGET=dev-yourname SSH=your-vm-alias STORAGE_DEVICE=/dev/vdb ``` Use the actual dedicated storage disk on your VM. `TARGET` starts with `dev-` and a lowercase letter, followed by lowercase letters, digits or hyphens. The first deploy saves its SSH and disk bindings; later runs need only `TARGET`. The deployment command uses the committed checkout. ## Dedicated SSH credentials The workstation uses its normal SSH client to reach the Linux runner. Container commands use a separate key for the selected dev box. Personal `~/.ssh` is never selected automatically. Every controller SSH connection disables agent and X11 forwarding. Command-only helpers clear inherited forwards and disable connection sharing; explicit tunnels use their own private connection and control socket. User-configured shared connections are never reused. Existing owned tunnels restart with these settings when used again. Run `make devbox-setup`, `make deploy` or **Set up dev box** in Pharos DX from **your workstation**. Secure access is prepared remotely before deployment or reset changes. No manual SSH-directory creation is needed for the normal flow. Planning and **Check dev box** inspect prerequisites without creating credentials. The workstation first checks remote tools, Docker availability and passwordless sudo. Missing dependencies produce an exact Debian/Ubuntu install command; package installation remains manual. The initial workstation SSH connection must already trust both the dev box and any separate runner. An unknown or changed host key stops setup; verify it with the VM administrator or console before continuing. Secure preparation then: 1. Generates one Ed25519 key on the runner under `~/.local/share/pharos-delivery//ssh/`, as its ordinary SSH user. 2. Adds only that public key to the target user's `authorized_keys`, preserving existing entries. It disables agent/X11 forwarding, PTYs and user SSH startup scripts while allowing the port forwarding needed by the tools. 3. Reads the dev box's public host key and connection address through the verified workstation connection, then pins them in a minimal runner configuration. 4. Tests the dedicated connection and passwordless sudo from the runner. The private key stays on the runner. The workstation coordinates separate runners through its own trusted connections to both hosts; it never forwards its SSH agent. Retries reuse the generated key and authorisation. Interrupted preparation resumes; missing generated configuration is restored from the pinned record. Changed host keys, changed target identity, replaced/missing recorded keys and conflicting configuration require review rather than automatic replacement. Keys have the administrative authority of the dev-box login, with an empty passphrase for unattended use. Their files are private (`0600`) in a private directory (`0700`). Only `id_ed25519`, `known_hosts` and optional `config` enter tools containers as individual read-only mounts. Config accepts only `Host`, `HostName`, `User` and `Port`. Linked files, shared-write directories and unsafe file permissions are rejected. The dev box needs a standard Ed25519 host public key at `/etc/ssh/ssh_host_ed25519_key.pub`, and its SSH connection address must be reachable from the runner and its containers. Loopback connections cannot supply that address. ### Manually managed access and recovery Existing manually prepared credentials are preserved and tested. For a custom network route or dedicated directory, set `RUNNER_SSH_DIR` in the workstation's `.env`, for example `RUNNER_SSH_DIR=~/pharos-keys/dev-yourname`. This selects **manually managed** credentials on the runner: provide its dedicated key, verified host-key record and minimal config there. An explicit override is validated and never generated or edited automatically. The path is saved with the delivery binding and applies to both CLI and Pharos DX. For direct container checks such as `make devbox-storage-test` on a Linux host, the default remains `/.local/ssh/`, which Git ignores. These lower-level commands require prepared credentials, or `METALLO_SSH_DIR` pointing at the dedicated runner directory. Use the root setup/deploy commands for automatic preparation. Local Minikube and image builds do not need SSH credentials. `make devbox-forward`, `make urls`, the takeover and the credential, seed, support link and GitLab qualification commands use the normal SSH client, unless `METALLO_SSH_DIR` selects such a directory. After an intentional VM replacement or credential rotation, verify the new host identity first. Revoke the old `pharos-` public-key entry on the old target where accessible, and preserve or remove the corresponding dedicated runner directory after reviewing it. Retrying setup then prepares a fresh key. Never remove unrelated `authorized_keys` entries or copy personal private keys to the runner. Existing deployment/node bindings have their own recovery checks. On upgrade, the launcher rebuilds older tools images and removes the selected target's legacy persistent SSH tunnel before requesting dedicated credentials. It also removes every other target's owned legacy tunnel, so a retired box's tunnel does not restart forever. The next connection recreates it with individual file mounts; changing the path or replacing an input also replaces the tunnel. The entrypoint removes the old SSH symlink from its private tools home without touching the link's destination. Finish any older deployment and stop older tools/Tilt sessions before upgrading; already-running containers retain their mounts until they exit. The tools still administer rootful Docker through its socket. Docker access and sudo remain privileged host access. Keep the runner dedicated and free of unrelated credentials: everyone administering its Docker daemon can access host files. Individual SSH mounts reduce routine credential exposure within that existing trust boundary. ## Guided setup and clean reset Use **Dev box** in Pharos DX, or `make devbox-setup` with your saved `.env` target. Both use the same reviewed controller and deliver the committed checkout through the same SSH delivery runner as `make deploy`. The runner needs Git, Python 3, Make, Bash and rootful Docker Engine. Its SSH user needs passwordless sudo; [dedicated SSH credentials](#dedicated-ssh-credentials) are prepared automatically, including when the runner is the dev box itself. These prerequisites also apply to the guided flow on Linux and Windows/WSL2. Save `STORAGE_DEVICE=/dev/vdb` in `.env` for first setup, using your VM's actual empty dedicated disk. Existing installations reuse their saved disk binding. Setup repeats installation and readiness checks. The first upgrade from the previous provider layout discards application data once; the review displays this scope. Later setups preserve the newly installed application data. Delivery checks each selected service's rendered setup hooks against Argo's last operation, including its source inputs. Pending hooks run even when Argo reports no ordinary resource changes. Selecting an optional service refreshes both the shared infrastructure and platform credential producers at the same commit. Completed hooks with matching inputs are reused on repeat setup. ```sh make devbox-setup # Check and install the committed source ``` For a clean reinstall, run `make devbox-reset` in an interactive terminal. It shows the deletion plan and asks you to type the target name before proceeding. Choose this only when you intend to discard application data. ### Explicit plans for inspection and automation Pharos DX presents planning and confirmation in its UI. The corresponding CLI commands let you inspect setup without installing, or separate a reset review from its execution: ```sh make devbox-plan # Optional setup preview ``` For a non-interactive reset, review the plan first and use its returned ID: ```sh make devbox-plan MODE=reset make devbox-reset PLAN= CONFIRM_TARGET=dev-yourname ``` The plan response includes its `id`, target, SSH alias, disk, commit, namespaces, volume count and object-bucket count. Plans expire after ten minutes and are checked again before execution. In an interactive terminal, `make devbox-reset` can create the plan and ask you to type the target name. Non-interactive reset requires the reviewed plan and exact target confirmation shown above. Progress is emitted as JSON events, with raw deployment output in a private `.local/pharos-dx/.log` file. Setup/reset runs in a background process on your workstation after review. Closing the terminal or interrupting its command stops observation; the same reviewed operation continues, including its final health checks. Repeat `make devbox-setup` or `make devbox-reset` from the same checkout, with the same target settings, to reconnect. Pharos DX's **Check dev box** does the same on the matching tab. This follows the original commit and operation; reconnecting to reset never repeats its deletion phase. Progress and the outcome are retained privately alongside the deployment log. Keep the workstation running and connected during deployment. If the background worker itself is killed or the workstation restarts, inspect the private log and review a new plan. A runner lock can still belong to surviving remote work. Earlier versions and deployments started from another checkout cannot be reattached through this journal. Let their runner operation finish before retrying. A competing Make command shows the latest 40 lines from the runner's private log and a command to follow it. Pharos DX shows that command; raw deployment output stays outside the webview. The SSH runner saves Make's stdout and stderr in `~/.local/share/pharos-delivery//delivery.log`, owned by the SSH user with mode `0600`. Each delivery appends its start time, commit and final exit status, so the previous outcome remains available after a retry. Output continues to be saved if the observing SSH connection closes. Follow it **on the runner** with: ```sh tail -n 40 -f ~/.local/share/pharos-delivery/dev-yourname/delivery.log ``` This log is available for deliveries started with the updated runner. For older runs, `sudo lslocks -o PID,COMMAND,PATH` identifies the process holding `operation.lock`; inspect its deployment container's logs. Keep that lock file in place while work runs. A retained log provides observation; the same-checkout background session still owns setup's final workspace checks. Reset deletes the reviewed service Applications and namespaces (`pharos`, `themelio-compat`, `gitlab`, `waldur`, `pharos-aai`, `pharos-infra`), their persistent volumes and object buckets. This removes application databases, messages, sessions, uploads and AAI realm state. It retains the VM, SSH access, Kubernetes, Ceph, host certificates, build cache and saved private runner configuration. Existing saved AAI credentials can be reused; cluster provider credentials are regenerated. No application data migration or backup is performed. Reset suspends the root and child Applications before recording route and Argo bootstrap recovery. This lets the existing admission guards permit namespace cleanup and reinstall; the final delivery restores Argo ownership. During this fenced recovery, Argo recreates policies in the replaced namespaces and preserves the identities of surviving routes and gateways. Takeover must be handed back first. During setup/reset a maintenance record blocks unrelated delivery and takeover. Successful completion requires every selected platform Application to be Synced and Healthy at the reviewed commit. If work fails, the maintenance record and private log remain: inspect the log, correct the cause, then check a new plan. Choose setup to resume installation, or review reset again if application deletion remains incomplete. Kubernetes finalisers are preserved; the command never force-removes a stuck volume or namespace. ## What deployment installs `make deploy` installs K3s, Ceph block/filesystem/S3 storage, Istio, the HTTPS host edge, Argo CD and shared Keycloak AAI. It builds application images inside the cluster from the delivered commit, then installs GitLab, Waldur, retained Themelio (`themelio-compat`) and the new platform (`pharos`) through Argo CD. It also checks storage and routes. The provider databases stay on the VM. Deployment is idempotent: repeat `make deploy` to apply the committed full-stack configuration and recover interrupted work. Application data survives repeated runs on the current layout; the [one-time old-layout conversion](#shared-infrastructure-and-private-file-ownership) discards old application data. Full deployment still runs host, cluster and readiness checks. Use `make reconcile` for everyday committed application changes; it builds changed images and skips delivery entirely when inputs are unchanged and the deployed applications are healthy. Private AAI overrides use a local checksum receipt, checked against the installed Secret and the applied AAI hook inputs. Changed or removed overrides, CA rotation and missing receipts require ordinary delivery. Provider credentials due for renewal within seven days also require delivery, even when the service images are unchanged. Both commands hand back this checkout's takeover of the target first and resume it after a successful delivery. A fresh checkout has no cached Waldur seed, so its first migrations can take about an hour. A delivery including Waldur has a 90-minute health gate and must finish Synced and Healthy at the delivered commit. Later installations can reuse the credential-free seed captured by a successful deployment. ```sh make deploy TARGET=dev-yourname ``` The Icarus-hpc laboratory uses `make metallo-icarus TARGET=dev-NAME SSH=your-dev-box` and its [laboratory guide](../metallo/icarus-hpc/README.md); deployment of the platform alone does not install the HPC host services. ### Incremental source delivery `make deploy` and `make reconcile` reuse Git history already held by the runner and cluster. Each hop reports its source bundle size; a receiver at the selected commit needs no bundle. First delivery, unavailable local base history and rollback to an ancestor use a complete bundle. Received source must still match the selected commit, and a modified managed checkout blocks delivery. Temporary source bundles are removed after transfer. Image fingerprints select the builds needed on AMD64 and ARM64. Database administration packages each registered PostgreSQL context's migrations; ordinary context implementation edits leave its image unchanged. Adding an owner requires its migration directory in both `devops/database-admin/Dockerfile` and `gitops/manifests/images.yaml`; verification checks this coverage. ## Prepare workstation development Takeover runs outside the deployment tools container. Install uv, kubectl and Helm on the workstation. The [developer guide](../../docs/development.md) lists native and Docker runtime requirements and the one-time native daemon setup. Tilt 0.37.8 is optional; the controller follows saved files without it. Set `RUNTIME=docker` in `.env` to use Docker for the following commands. Save `TARGET=dev-yourname` and `SSH=your-vm-alias` there too to omit them below. ```sh make takeover TARGET=dev-yourname SSH=your-vm-alias make takeover-status TARGET=dev-yourname make takeover-logs TARGET=dev-yourname SERVICE=compat-web make tilt TARGET=dev-yourname ``` With Docker Engine on Linux, the controller starts a session relay container on Docker's default bridge once the takeover journal exists. The workstation dials it over `docker exec` and opens SSH reverse forwards to the takeover's loopback API tunnel and to its own tunnel to the dev box's public-only listener. Telepresence and the application containers reach those through the relay, so no inbound firewall rule is needed, but containers on the default bridge must reach each other (Docker's default). Hand-back removes the relay. `compat-web` is the retained frontend at `compat/themelio/services/web`, in the `themelio-compat` Application and namespace. `web` is the new platform shell. The controller selects affected services, reloads them and renews an Argo CD lease. One developer owns a dev box's takeover at a time. `make deploy` and `make reconcile` hand back this checkout's takeover and resume it afterwards. Takeover preserves Beto's enrolled Waldur Project API and event broker connections, and the enabled Support and LLM provider paths. It checks both sides of the installed owner policies before granting those paths to the development tunnel. A provider's namespace-wide egress is narrowed to its declared ingress Pod selector. ## Browser access and login Setup and deployment prepare a background SSH connection for public browser access. `make urls` connects or reconnects it before listing addresses; in Pharos DX, select **Your services → Service details → URLs** and click a link. This works on macOS, Linux and Windows with WSL2. No separate SSH command or open terminal is needed. VPN and verified SSH access must remain available. After a restart or connection loss, repeat `make urls` or click a service URL to reconnect. Public HTTPS uses local port 8443 and the dev box's public-only listener on loopback port 8444. Argo CD, Ceph and Waldur administration routes return 403. Takeover reuses this public connection, so `https://platform.localtest.me` reaches the replacement web workload. Port 28000 remains its direct local listener. Hand-back keeps public HTTPS available for the deployed services. For administration access, hand back local services and run: ```sh make devbox-forward ``` Administration links in Pharos DX and this command switch to the operator edge and returns immediately. It requires no local Docker engine. The next takeover restores public-only access automatically, before starting application code. An unrelated process using port 8443 is reported and left running. To close browser access when finished, use **Disconnect browser access** in Pharos DX or `make devbox-disconnect`. Hand back any active takeover first. Only one dev box can use local port 8443 at a time; disconnect before switching. On Windows, run Make inside the configured WSL2 distribution. Pharos DX starts the same controller through WSL2 and opens the native Windows browser. Keep WSL's localhost forwarding enabled; the app checks that Windows can reach the listener before opening a link. See [Microsoft's WSL networking guide](https://learn.microsoft.com/en-us/windows/wsl/networking#accessing-linux-networking-apps-from-windows-localhost). Command-only SSH connections clear inherited port forwards and disable agent/X11 forwarding and shared connections. Managed tunnels use a fresh private control socket and add only their requested forwards, preserving SSH aliases and jump hosts. Operator access still grants administration routes to local processes, including containers able to reach the workstation's host gateway. | Service | Workstation URL | |---|---| | Pharos web | `https://platform.localtest.me` | | Pharos Go API | `https://api.localtest.me/api/v1/` | | Shared AAI | `https://pharos.127.0.0.1.sslip.io:8443/auth` | | Retained Themelio (enable local HTTPS below) | `https://themelio.localtest.me` | | GitLab | `https://gitlab.localtest.me` | | Waldur | `https://waldur.localtest.me` | | Direct platform web during takeover | `http://localhost:28000` | | Tilt dashboard | `http://127.0.0.1:10350` | Run `make devbox-browser-setup` to configure browser trust for your VM's public CA. Open **Credentials** in Pharos DX or run `make credentials` for the available accounts. Explicitly copy or reveal one value, for example `make credentials COPY=waldur.staff`. The platform shell uses `pharos.operator`; the retained frontend uses the shared AAI. See [credential access](CREDENTIALS.md) for database passwords, initial administrator accounts and access requirements. Use one dev-box HTTPS forward at a time on port 8443. Stop the previous forward when switching VMs. ### Upgrade existing takeover access Hand back with **Hand back all services** or `make handback`, update the checkout, and close any manually opened edge forward. Managed connections switch automatically. Run **Dev box → Check dev box → Set up dev box** or `make deploy` to install the public-only edge, then take over again. **Deploy this commit** / `make reconcile` does not apply this host configuration. The controller verifies public sign-in and management denial before starting local services; an older or unhealthy edge produces these upgrade instructions. Recognised old takeover forwards are retired automatically. ### Short service addresses Use `https://platform.localtest.me`, `https://api.localtest.me`, `https://themelio.localtest.me`, `https://gitlab.localtest.me` and `https://waldur.localtest.me` after enabling local HTTPS: ```sh # Existing dev box: install the new route, TLS certificate and sign-in callback. make deploy make devbox-browser-setup ``` Fresh setup already installs the route. The command verifies the current public CA over SSH, prepares local HTTPS and requests OS certificate-trust approval when needed. Repeat runs reuse verified trust. No hosts-file change is needed. Themelio keeps the short address during navigation and after sign-in or logout; AAI itself still uses its existing address. The short address is canonical: browser visits to the old address redirect here. Existing Git/LFS remotes work. The API host exposes `/api/v1/`, `/auth/` and `/healthz`; other paths return 404 at the edge. For `https://ceph.localtest.me`, `https://argo.localtest.me` and Waldur `/admin/`, run `make handback`, then `make devbox-forward`. Public-only access refuses administration. Run the command as your normal user on macOS or Linux, or inside the configured WSL2 distribution on Windows. Setup requests approval for port 443 and CA trust when needed. The port bootstrap drops privilege and exits; a normal-user relay forwards encrypted bytes to the existing local port 8443. It neither decrypts traffic nor runs SSH as root. An occupied port is reported without stopping its owner. The relay survives closing the terminal. After a restart, rerun setup. After a VPN interruption, `make urls` or Pharos DX's service URLs reconnect SSH and open the canonical address. Enable local HTTPS before using short addresses. `make devbox-disconnect` closes SSH. Release port 443 with: ```sh make devbox-browser-stop ``` Short addresses follow the single dev box on port 8443. Disconnect before switching. On Windows, WSL localhost forwarding must be enabled, and port 443 must also be free on the Windows host. ### Git certificate trust `make devbox-browser-setup` also configures Git's user-level `http..sslCAInfo` and `sslVerify=true` for the development Themelio and GitLab HTTPS origins, including their older port-8443 addresses. This covers Git builds that use a separate CA store, so ordinary clones, pulls, pushes and Git LFS can use the verified development CA without per-command flags. Other Git origins keep their existing configuration. On Windows, setup configures WSL Git and, when `git.exe` is on the Windows PATH, native Windows Git with a Windows-readable copy of the public CA. Repeat setup after switching dev boxes, moving this checkout, or installing Windows Git. Existing repository-level or environment overrides take precedence; remove an old clone's local `http..sslCAInfo` override if it still points at a previous CA path. ## Return and deliver ```sh git add git commit -m 'feat: describe your change' make reconcile TARGET=dev-yourname ``` Reconcile delivers the commit and builds only changed images. An active takeover is handed back first and resumes after a successful delivery with the same runtime and selection; after a failure, the command prints the `make takeover` command to run once delivery succeeds. Use deploy for Ansible or host platform configuration. Committed AAI changes use `make reconcile`. Hand-back restores the delivered workloads and removes the lease; repeat it after an interruption. Closing Tilt leaves the watcher running, so use hand-back when finished. ## Controller maintenance and verification The older `metallo-devbox` application loop has its own [controller maintenance guide](CONTROLLER-LOOP.md). Its `legacy-controller-setup`, `legacy-controller-takeover`, `legacy-controller-full-cluster`, `legacy-controller-tilt` and `legacy-controller-stop` commands manage that separate namespace. Run `pnpm verify` and the applicable real infrastructure lanes in [TESTING.md](../../TESTING.md). Cold and seeded installation passed on clean Debian 12 AMD64 VMs of the shape above. The developer guide records the [runtime and browser coverage](../../docs/development.md#readiness) and provides [repeatable acceptance commands](../../docs/development.md#acceptance-checks). ### Shared infrastructure and private file ownership Personal dev boxes use one PostgreSQL server and one RabbitMQ server in `pharos-infra`. Keycloak, GitLab, Waldur, retained Themelio and the Pharos contexts have separate database identities and databases. Pharos and Waldur use separate broker scopes; Waldur's event subscriptions receive their own virtual hosts. Pharos sessions and retained Themelio share persistent Valkey with separate ACL users and key prefixes. Argo, GitLab and ephemeral realtime history retain their own cache servers. See [decision 0048](../../docs/decisions/0048-shared-development-infrastructure.md) for the boundaries and compatibility choices. To upgrade an existing personal dev box, hand back any takeover, update the checkout and run `make deploy`. Its one-time conversion discards the old application databases, messages, sessions, volumes, GitLab repositories and uploaded files. It preserves the VM, Kubernetes, Ceph, certificates and build cache. Shared PostgreSQL moves to 18.6 and GitLab to its compatible 19.5.1 release. The ownership record checkpoints cleanup before the first deletion and installation before Argo creates new data. If interrupted, repeat `make deploy` with the same saved target: it resumes without deleting the new installation. A completed conversion is skipped on subsequent deployments. Use `make devbox-reset` only when you explicitly want to discard the current application data again. Writable tools containers use your host UID/GID. SSH delivery drops to the SSH user before creating checkout, cache or private-state files. Docker socket access is granted only to the running process through its supplementary group. The launcher detects tools images without the current identity contract and rebuilds them. Before reading saved configuration, an isolated helper repairs recognised legacy state, including a checkout's linked `.local` directory and its recorded SSH runner binding. The helper has no network or Docker socket. Private modes, credentials and existing lock inodes are preserved; external symlink targets, unrelated `.local` content and other users' files stay outside its scope. Root-owned hard links are refused. `make repair-ownership` remains available for an explicit repair from the normal developer account. Normal deploy and takeover perform this preparation automatically, including after a failed command.