#Distributed setup

Use ADR-0051 and its implementation/evidence for scope. The personal TARGET=dev-NAME workflow remains in development.

Commands with ENV do not load the checkout's personal .env. Explicit personal TARGET or SSH selections still fail before access to an installation. Ordinary dev-box commands retain their saved settings, runner bindings and private state.

#Inputs on one administration machine

Run administration through an SSH runner that holds the environment's private inputs and state, for example a small VM inside the site:

make deploy-plan ENV=icarus-cloud DEPLOY_RUNNER=pharos-admin@site-runner
make deploy ENV=icarus-cloud

The first command records the runner for that ENV; later commands reuse it, and another runner or local execution is refused until its private .local/deployments/<ENV>/runner.json is deliberately removed. The workstation sends only the committed revision; untracked and ignored files never travel. The runner account must be unprivileged and have git, python3, uv, kubectl, helm and SSH access to the inventory hosts; commands check their tools first. Its output is also kept in ~/.local/share/pharos-deployments/<ENV>/delivery.log, so an operation continues after the SSH session ends. INVENTORY, KUBECONFIG, STATE_DIR and PHAROS_PRIVATE_ROOT then name files on the runner. Without DEPLOY_RUNNER and a recorded binding, the workstation is the administration machine.

Keep native inventory and its group_vars under ~/.config/pharos/environments/<ENV>/inventory/hosts.yml, or select another private file with INVENTORY. Directories use 0700 and private files 0600. Keep vault unlock credentials in the operator's secret facility. Never commit connection credentials, private keys or administrator kubeconfig.

The native groups are kube-master, kube-master-primary, kube-node, etcd and k8s_cluster; equivalent modern Kubespray names are supported. Conflicting aliases fail. Switches and other non-cluster groups remain valid. Per-host storage_devices preserve existing references for observation; fresh installation requires explicit /dev/disk/by-id/ devices. Label three selected storage/monitor hosts with storage-node: "True" and storage-control: "True"; selected AMD64 service hosts use pharos.platform/services: "true".

Administrator kubeconfig defaults to <private-root>/<ENV>/admin/kubeconfig. KUBECONFIG selects another private file. PHAROS_PRIVATE_ROOT changes the common private root; STATE_DIR selects a private record directory. These are paths, never credential values. No private-state symlink or laptop-to-runner copy is required. Set pharos_api_access: ssh on the primary only when API access needs its native SSH binding; otherwise direct access is used.

A complete site also needs a private external-AAI JSON file named by the selected primary host's pharos_external_aai_file. The administrative path must be mode 0600 and bind pharos-control-plane, pharos-tools and pharos-gitlab (and pharos-open-webui when the site serves chat sign-in), with the issuer and an explicit public CA certificate. Do not put credentials in inventory, committed policy, logs or the checkout. The plan records only the canonical input fingerprint; delivery refuses missing or changed input before opening the cluster.

#Commands

make deploy-plan ENV=icaruscloudlab
make deploy-plan ENV=icaruscloudlab OFFLINE=1
make ansible-apply ENV=icaruscloudlab
make deploy ENV=icaruscloudlab
make reconcile ENV=icaruscloudlab
make distributed-qualify ENV=icaruscloudlab
make hpc-site-central ENV=icaruscloudlab
make renew-tokens ENV=icaruscloudlab

renew-tokens renews the platform's own GitLab and Waldur tokens (Source Control's and the compatibility service's) without a delivery, under the environment lock. It reads the expiry the Secrets record and renews only a token in its last week, and finishes a rotation an earlier run left half done. Every workload that reads a renewed Secret is rolled, and a replaced GitLab token is revoked only after all of them run Ready on the new token and the provider accepts it; until then the rotation stays marked for the next run. It fails when GitLab or Waldur is not Synced and Healthy. Run it daily on the runner with the user timer in devops/platform/systemd/. The timer runs the revision and runner modules the last successful deploy or reconcile pinned, under the runner's checkout lock, so it needs one successful delivery first:

cp devops/platform/systemd/pharos-token-renewal@.* ~/.config/systemd/user/
systemctl --user enable --now pharos-token-renewal@icarus-cloud.timer
loginctl enable-linger "$USER"   # keep user timers running without a session

hpc-site-central provisions the platform half of the environment's HPC Site (its Waldur offering, the Site agent's account and the Site API bearers). It runs where the private inputs are, on the runner when one is set or recorded, under the same environment lock, and keeps its files in that machine's ~/.config/pharos/environments/<ENV>/hpc-site. The offering stays Draft unless OFFERING_STATE=Active is given.

The plan writes private evidence on the administration machine only. Offline mode performs no SSH/API queries and is accepted only for a plan. Commit tracked source before mutation; the command freezes that revision for the whole operation. ansible-apply installs one whole fresh topology, followed by selected foundation layers. It refuses unrecorded existing Kubernetes/etcd state and never wipes devices. A pending operation may be retried with its unchanged inputs; inspect its private stage log when it fails. On a ready foundation, ansible-apply also reapplies changed storage/mesh/Argo policy, but refuses node, group, Kubernetes version or CNI changes. The environment lock prevents overlapping operations. If Kubernetes itself stopped after occupying the hosts but before recording its cluster identity, inspect that installation and restore its verified private access/identity first. The command refuses to adopt an unanchored partial cluster.

Kubespray uses its own pinned tools under devops/kubespray; the host/foundation tools use devops with the collections pinned in devops/ansible/requirements.yml, installed privately under ~/.cache/pharos; operator collections are not used. Do not override Kubespray's version assertion or downgrade the personal workflow's Ansible. State/source caches remain private on the administration machine.

icaruscloudlab remains foundation-only until it has approved public origins and external identity. A three-node lab with one control plane/etcd member is not HA.

GPU hosts use the explicit host/cluster ownership in the NVIDIA GPU Operator guide. Host driver, toolkit and containerd runtime work is Ansible-only on selected GPU hosts. The operator and its operands are an opt-in Argo component, never part of the dev-box composition.

When delivery is enabled, Argo installs the closed complete-site composition: Argo CD, routes, shared infrastructure, GitLab, Waldur, LLM, Pharos, the paused Site Runtime and themelio-compat. The compatibility service is mandatory until all its routes have moved to the new Themelio. Development AAI, Mailpit, browser aliases and Waldur seeds are excluded.

KServe and LeaderWorkerSet are included in this composition. Delivery refreshes their bootstrap prerequisites and grants on existing foundations as well as fresh installations. See platform model serving for architecture-specific CPU images and preservation of external prerequisites.

Public routing uses the same catalogue as a dev box, with explicit HTTPS/443 origins and existing certificates. The root receives exact active profiles, catalogue checksum, Ceph certificate trust and built image digests. Unknown or missing Applications, development identity or an incomplete child set fail before the root is applied.

While make deploy runs, the GPU (gpu-dra, gpu-operator), Kubeflow and llm-inference Applications can read Unknown or Missing ("namespace … is not managed") for a while. That is expected: the Argo step writes the cluster views for the platform's own namespaces only, and delivery widens them for the selected compositions after the image builds, before the root names those Applications (platform_deploy/delivery.py, share_namespaces). Do not patch the views by hand. That step clears the state; the readiness gate at the end must still report every Application Synced and Healthy at the delivered revision.

Qualification creates an owned namespace and checks cross-node storage, actual network denial and image distribution. Success deletes it with its exact UID; failure retains it and the private report for inspection. The registry probe builds one real platform image without starting its application or giving it credentials, then pulls that digest on all three nodes.

#GitLab LFS capacity

The distributed GitLab profile keeps three idle webservice/Workhorse replicas and allows its HPA to reach 18 under load. It uses 900-second Rails waits, a 3600-second Workhorse proxy-header timeout and 16 concurrent Git LFS transfers. Pinned GitLab 19.4.2 reached 154 MiB/s at 18 replicas with four clients; eight clients reached 213 MiB/s, or 99% of the matched direct-RGW ceiling. A four-client 1 GiB sequential test reached 252 MiB/s upload and 292 MiB/s download.

The 19.5.1 images remain pinned to amd64 pending equivalent ARM64 qualification. The preceding 19.4.2 images failed at startup on the cluster's ARM64 nodes with 64 KiB pages (jemalloc: Unsupported system page size), so the distributed overlay pins every GitLab component to kubernetes.io/arch=amd64. The Site Runtime preparation image pins checksum-verified Git LFS 3.8.1 rather than the Bookworm package. The matched direct-download client test measured an 11% median improvement over 3.6.1, while hashing and disk writes were ruled out as the remaining bottleneck. The previous immutable batch-image digest is the rollback; replacing the download with apt-get install git-lfs restores the distro client for a subsequent build. Keep LFS downloads proxied until a public TLS, presigning and NetworkPolicy contract exists; the object endpoint is currently internal HTTP. GitLab excludes LFS objects from attachment and push-size limits, and chart 10.5.1 has no per-object LFS cap.

#Existing site

make deploy-plan ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
make distributed-qualify ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
make cutover-plan ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig

These existing-foundation operations read identities/readiness and capture a private review. They do not invoke Kubespray, discover/wipe disks or replace application writers. Optional private pharos_expected_cluster_uid and pharos_expected_ceph_fsid bindings reject a different site. Follow the cutover runbook after 0058 supplies the complete identity/application composition and a maintenance window.

cutover-plan writes a read-only maintenance review and preserves the foundation record. After approving that private capture and retiring the reviewed writers, make foundation-adopt ENV=icarus-cloud ... rechecks the source/input/topology fingerprints and live Kubernetes, node and Ceph identities before recording the existing foundation. The Kubernetes API and every kubelet must match the pinned release. Follow the cutover runbook for the exact sequence and repeat adoption after a policy change. Kubespray foundations retain their installer-owned record.

Approved origins use HTTPS/443 with a distinct hostname per service; certificates stay under their existing renewal owner. The Pharos browser origin uses one platform.<domain> name for the web application and its same-origin API, native /auth/oidc/callback and realtime routes; a separate direct API hostname is not selected by implication. The native Site API uses the distinct site.<domain> host, with only /v1/site/commands/ and /v1/site/ssh-access/ forwarded to the HTTPS site-api Service. It performs scoped bearer authentication and does not use browser OIDC or register a Site API redirect URI. The gateway verifies the backend certificate for site-api.pharos.svc with the internal Site API trust.

Every distributed installation declares a kubeflow and an llm-chat origin. The retained web's sidebar links to both, and to the pharos origin's Support view, so delivery stops when either origin is missing. A new kubeflow origin also needs its https://<kubeflow-origin>/oauth2/callback registered at the AAI, as the Kubeflow identity decision describes.

Icarus-cloud currently selects the canonical *.staging.pharosfactory.ai application names, while Argo, storage and the AAI issuer retain separately sourced bindings. Danias owns DNS changes for *.pharosfactory.ai; DNS and certificate existence remain qualification gates. The Site Agent and installer use these public base origins without appending /v1/site/:

SITE_API_URL=https://site.staging.pharosfactory.ai
SSH_ACCESS_SITE_API_URL=https://site.staging.pharosfactory.ai
central.site_api_url=https://site.staging.pharosfactory.ai

Management routes keep the same restrictions.

Choose one certificate owner before installing the public edge. With the default edge_certificate_mode: existing, an external issuer such as the CSLab inventory's Certbot layout renews the certificates; the role verifies trust, hostname, expiry and key match for edge_certificate_files and writes no port-80 listener. With edge_certificate_mode: certbot the role owns the names instead: it installs /etc/nginx/conf.d/pharos-platform-http.conf, which answers /.well-known/acme-challenge/ from edge_acme_webroot and redirects every other request to HTTPS, requests one lineage per approved host with certbot certonly --webroot, then installs the 443 configuration. Set edge_acme_email and review with edge_acme_staging: true before a real site switches its names. A renewal hook (nginx -t && systemctl reload nginx) is recorded in each lineage, and the run checks it with certbot renew --dry-run --run-deploy-hooks, because a plain dry run runs neither the recorded directory nor the deploy hooks. edge_acme_server selects an alternative ACME directory, which is how a laboratory's own test CA is exercised without a public name.

make deploy installs the edge itself, right before it applies the root: it writes the edge's environment from the policy's origins (exactly the edge schema's keys) to the private state directory and runs edge-distributed.yml on the inventory's platform-edge hosts. A certificate for a newly selected origin is issued before the HTTPS configuration names it. The certificate owner above stays the private inventory's choice, and the delivery run skips the certbot renew dry-run probe; run the playbook with edge_acme_renewal_check: true to check renewal on purpose.

The challenge listener must be the only server on the host that declares listen 80 default_server, and the 443 file must be the only listen 443 ssl default_server; retire a legacy virtual host in the same step rather than alongside it, because nginx -t rejects the duplicate. The role retires the nginx package's own welcome vhost, whose symlink also claims port 80, and fails on any other occupant instead of removing it. A listener that fails that check is rolled back before the run stops: the previous file is restored, a first installation is removed and the retired welcome symlink is put back, so the host keeps a configuration every later nginx -t, reload and restart accepts. A pending reload survives a failed reload or an interruption after validation. Retrying with an unchanged listener completes that reload before requesting certificates.

Outbound mail is one smtp block in the environment policy: the relay host, port, TLS mode, the EHLO name the relay will see, the sender, the public API address a member's client uses and the address blocks the sending pods may reach. An enabled block must have all of them, including at least one address block; a malformed one is refused before any site uses it. Sending workloads reach that relay directly over STARTTLS and verify the relay's own certificate, so the address the relay permits is the one it observes. The blocks are declared rather than resolved while rendering, because a policy cannot name a host and a DNS answer would not be deterministic: smtp.egress takes exact networks, refuses a default route, and the delivery renders one owned smtp-egress NetworkPolicy in the pharos namespace only while smtp.enabled is true, withdrawing it otherwise. The base manifests stay site-neutral, so each environment declares its own relay. The delivery renders the block into Metallo's Deployment beside the workloads it places, as LLM_ACCESS_SMTP_HOST, _PORT, _TLS, _FROM and _EHLO plus LLM_ACCESS_PUBLIC_API_URL, and renders nothing while smtp.enabled is false. The SMTP public_api origin must match edge.origins.llm-api; delivery appends /v1 for the client base shown in personal-key emails. The workload therefore reads one reviewed source: it keeps mailing off without a host and refuses to start without a sender and the public API address. The EHLO name matters because a container's own name is not what the relay knows the site by. Waldur and Zammad take the same values when their profiles are enabled: Zammad's desk sends its outgoing mail to that relay, with the policy's TLS mode and certificate verification, in place of the Mailpit sink it bundles, and only while smtp.enabled is true. A password, if one is ever needed, follows the external-identity pattern: a private file with mode 0600 rendered to a per-namespace Secret.

Waldur receives the relay, sender, STARTTLS or implicit TLS mode, and EHLO name through its chart's mail values. It uses Django's standard certificate verification. Its API, worker and scheduler restart when those values change. Disabling the SMTP policy removes the relay from the generated Waldur values. Waldur's namespace currently restricts ingress only; the pharos SMTP egress policy above applies to Metallo.

Project invitations use the HTML and plain-text templates in gitops/manifests/waldur/chart/files/invitations. The generator inserts this environment's Themelio /dashboard URL. The recipient signs in with the invited address and accepts the pending invitation there. Other invitation scopes retain Waldur's own acceptance link. The post-sync bootstrap applies the two overrides through waldur override_templates on every sync, so existing installations receive template changes. Other stored templates are preserved. Template customisation does not enable a disabled notification.

To activate this on an existing site:

1. Include the generator and chart changes in the site's reviewed deployment revision and deliver it through the normal environment workflow. The SMTP policy already enabled in Icarus Cloud needs no additional policy change. Wait for Waldur's API, worker and scheduler rollouts and the post-sync pharos-bootstrap Job to succeed.
2. With the site's administrator kubeconfig, check the worker's effective EMAIL_HOST, EMAIL_PORT, EMAIL_USE_TLS, EMAIL_USE_SSL, DEFAULT_FROM_EMAIL and django.core.mail.utils.DNS_NAME.get_fqdn(). Confirm users.invitation_created is enabled under Waldur's Notifications administration and that the two stored invitation templates contain the current dashboard URL.
3. Create one project invitation to an explicitly approved test recipient. Confirm receipt of both HTML and plain-text content, the sender, expiry, dashboard link and acceptance after AAI sign-in. Check the worker and relay logs if delivery fails. A successful TCP or STARTTLS handshake establishes connectivity; mailbox receipt and project membership establish the journey.
4. After delivery is qualified, review affected invitations and resend those still pending. Replace expired invitations through the normal invitation workflow. SMTP activation affects Waldur notification mail generally, so review its existing pending work before rollout.

The separate TLS client (the smtp-proxy role) is retained for a host-local sender only; it was never wired to the policy and is not part of this path. It validates upstream TLS, delays relay DNS until connection time and never shares an Nginx startup dependency.

Test the edge configuration without sending mail:

uv run --project devops --frozen python scripts/distributed_edge_smoke.py