# Distributed setup

Use [ADR-0051](decisions/0051-distributed-inventory-setup.md) and
[its implementation/evidence](dd/0051-distributed-inventory-setup.md) for scope.
The personal `TARGET=dev-NAME` workflow remains in [development](development.md).

Commands with `ENV` do not load the checkout's personal `.env`. Explicit personal
`TARGET` or `SSH` selections still fail before access to an installation. Ordinary
dev-box commands retain their saved settings, runner bindings and private state.

## Inputs on one administration machine

Run administration through an SSH runner that holds the environment's private
inputs and state, for example a small VM inside the site:

```sh
make deploy-plan ENV=icarus-cloud DEPLOY_RUNNER=pharos-admin@site-runner
make deploy ENV=icarus-cloud
```

The first command records the runner for that `ENV`; later commands reuse it, and
another runner or local execution is refused until its private
`.local/deployments/<ENV>/runner.json` is deliberately removed. The workstation
sends only the committed revision; untracked and ignored files never travel. The
runner account must be unprivileged and have `git`, `python3`, `uv`, `kubectl`,
`helm` and SSH access to the inventory hosts; commands check their tools first. Its output is also kept in
`~/.local/share/pharos-deployments/<ENV>/delivery.log`, so an operation continues
after the SSH session ends. `INVENTORY`, `KUBECONFIG`, `STATE_DIR` and
`PHAROS_PRIVATE_ROOT` then name files on the runner. Without `DEPLOY_RUNNER` and a
recorded binding, the workstation is the administration machine.

Keep native inventory and its `group_vars` under
`~/.config/pharos/environments/<ENV>/inventory/hosts.yml`, or select another
private file with `INVENTORY`. Directories use 0700 and private files 0600.
Keep vault unlock credentials in the operator's secret facility. Never commit
connection credentials, private keys or administrator kubeconfig.

The native groups are `kube-master`, `kube-master-primary`, `kube-node`, `etcd`
and `k8s_cluster`; equivalent modern Kubespray names are supported. Conflicting
aliases fail. Switches and other non-cluster groups remain valid. Per-host
`storage_devices` preserve existing references for observation; fresh installation
requires explicit `/dev/disk/by-id/` devices. Label three selected storage/monitor
hosts with `storage-node: "True"` and `storage-control: "True"`; selected AMD64
service hosts use `pharos.platform/services: "true"`.

Administrator kubeconfig defaults to `<private-root>/<ENV>/admin/kubeconfig`.
`KUBECONFIG` selects another private file. `PHAROS_PRIVATE_ROOT` changes the common
private root; `STATE_DIR` selects a private record directory. These are paths,
never credential values. No private-state symlink or laptop-to-runner copy is
required. Set `pharos_api_access: ssh` on the primary only when API access needs
its native SSH binding; otherwise direct access is used.

A complete site also needs a private external-AAI JSON file named by the selected
primary host's `pharos_external_aai_file`. The administrative path must be mode
0600 and bind `pharos-control-plane`, `pharos-tools` and `pharos-gitlab` (and `pharos-open-webui` when the site serves chat sign-in), with the
issuer and an explicit public CA certificate. Do not put credentials in inventory,
committed policy, logs or the checkout. The plan records only the canonical input
fingerprint; delivery refuses missing or changed input before opening the cluster.

## Commands

```sh
make deploy-plan ENV=icaruscloudlab
make deploy-plan ENV=icaruscloudlab OFFLINE=1
make ansible-apply ENV=icaruscloudlab
make deploy ENV=icaruscloudlab
make reconcile ENV=icaruscloudlab
make distributed-qualify ENV=icaruscloudlab
make hpc-site-central ENV=icaruscloudlab
make renew-tokens ENV=icaruscloudlab
```

`renew-tokens` renews the platform's own GitLab and Waldur tokens (Source
Control's and the compatibility service's) without a delivery, under the
environment lock. It reads the expiry the Secrets record and renews only a token
in its last week, and finishes a rotation an earlier run left half done. Every
workload that reads a renewed Secret is rolled, and a replaced GitLab token is
revoked only after all of them run Ready on the new token and the provider accepts
it; until then the rotation stays marked for the next run. It fails when GitLab or
Waldur is not Synced and Healthy. Run it daily on the runner with the user timer in
`devops/platform/systemd/`. The timer runs the revision and runner modules the last
successful `deploy` or `reconcile` pinned, under the runner's checkout lock, so it
needs one successful delivery first:

```sh
cp devops/platform/systemd/pharos-token-renewal@.* ~/.config/systemd/user/
systemctl --user enable --now pharos-token-renewal@icarus-cloud.timer
loginctl enable-linger "$USER"   # keep user timers running without a session
```

`hpc-site-central` provisions the platform half of the environment's HPC Site (its
Waldur offering, the Site agent's account and the Site API bearers). It runs where
the private inputs are, on the runner when one is set or recorded, under the same
environment lock, and keeps its files in that machine's
`~/.config/pharos/environments/<ENV>/hpc-site`. The offering stays Draft unless
`OFFERING_STATE=Active` is given.

The plan writes private evidence on the administration machine only. Offline mode performs no SSH/API
queries and is accepted only for a plan. Commit tracked source before mutation;
the command freezes that revision for the whole operation. `ansible-apply` installs
one whole fresh topology, followed by selected foundation layers. It refuses
unrecorded existing Kubernetes/etcd state and never wipes devices. A pending
operation may be retried with its unchanged inputs; inspect its private stage log
when it fails. On a ready foundation, `ansible-apply` also reapplies changed
storage/mesh/Argo policy, but refuses node, group, Kubernetes version or CNI
changes. The environment lock prevents overlapping operations.
If Kubernetes itself stopped after occupying the hosts but before recording its
cluster identity, inspect that installation and restore its verified private
access/identity first. The command refuses to adopt an unanchored partial cluster.

Kubespray uses its own pinned tools under `devops/kubespray`; the host/foundation
tools use `devops` with the collections pinned in `devops/ansible/requirements.yml`,
installed privately under `~/.cache/pharos`; operator collections are not used. Do not override Kubespray's version assertion or downgrade the
personal workflow's Ansible. State/source caches remain private on the administration machine.

`icaruscloudlab` remains foundation-only until it has approved public origins and
external identity. A three-node lab with one control plane/etcd member is not HA.

GPU hosts use the explicit host/cluster ownership in the
[NVIDIA GPU Operator guide](gpu-operator.md). Host driver, toolkit and containerd
runtime work is Ansible-only on selected GPU hosts. The operator and its operands
are an opt-in Argo component, never part of the dev-box composition.

When delivery is enabled, Argo installs the closed complete-site composition:
Argo CD, routes, shared infrastructure, GitLab, Waldur, LLM, Pharos, the paused
Site Runtime and `themelio-compat`. The compatibility service is mandatory until
all its routes have moved to the new Themelio. Development AAI, Mailpit, browser
aliases and Waldur seeds are excluded.

KServe and LeaderWorkerSet are included in this composition. Delivery refreshes
their bootstrap prerequisites and grants on existing foundations as well as
fresh installations. See [platform model serving](../gitops/manifests/kserve/README.md)
for architecture-specific CPU images and preservation of external prerequisites.

Public routing uses the same catalogue as a dev box, with explicit HTTPS/443
origins and existing certificates. The root receives exact active profiles,
catalogue checksum, Ceph certificate trust and built image digests. Unknown or
missing Applications, development identity or an incomplete child set fail before
the root is applied.

While `make deploy` runs, the GPU (`gpu-dra`, `gpu-operator`), Kubeflow and
`llm-inference` Applications can read Unknown or Missing ("namespace … is not
managed") for a while. That is expected: the Argo step writes the cluster views
for the platform's own namespaces only, and delivery widens them for the selected
compositions after the image builds, before the root names those Applications
(`platform_deploy/delivery.py`, `share_namespaces`). Do not patch the views by
hand. That step clears the state; the readiness gate at the end must still report
every Application Synced and Healthy at the delivered revision.

Qualification creates an owned namespace and checks cross-node storage, actual
network denial and image distribution. Success deletes it with its exact UID;
failure retains it and the private report for inspection. The registry probe
builds one real platform image without starting its application or giving it
credentials, then pulls that digest on all three nodes.

## GitLab LFS capacity

The distributed GitLab profile keeps three idle webservice/Workhorse replicas and
allows its HPA to reach 18 under load. It uses 900-second Rails waits, a
3600-second Workhorse proxy-header timeout and 16 concurrent Git LFS transfers.
Pinned GitLab 19.4.2 reached 154 MiB/s at 18 replicas with four clients; eight
clients reached 213 MiB/s, or 99% of the matched direct-RGW ceiling. A four-client
1 GiB sequential test reached 252 MiB/s upload and 292 MiB/s download.

The 19.5.1 images remain pinned to amd64 pending equivalent ARM64 qualification.
The preceding 19.4.2 images failed at startup on the cluster's ARM64 nodes with
64 KiB pages (`jemalloc: Unsupported system page size`), so the distributed
overlay pins every GitLab component to `kubernetes.io/arch=amd64`.
The Site Runtime preparation image pins checksum-verified Git LFS 3.8.1 rather
than the Bookworm package. The matched direct-download client test measured an
11% median improvement over 3.6.1, while hashing and disk writes were ruled out
as the remaining bottleneck. The previous immutable batch-image digest is the
rollback; replacing the download with `apt-get install git-lfs` restores the
distro client for a subsequent build.
Keep LFS downloads proxied until a public TLS, presigning and NetworkPolicy
contract exists; the object endpoint is currently internal HTTP. GitLab excludes
LFS objects from attachment and push-size limits, and chart 10.5.1 has no
per-object LFS cap.

## Existing site

```sh
make deploy-plan ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
make distributed-qualify ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
make cutover-plan ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
```

These existing-foundation operations read identities/readiness and capture a
private review. They do not invoke Kubespray, discover/wipe disks or replace
application writers. Optional private `pharos_expected_cluster_uid` and
`pharos_expected_ceph_fsid` bindings reject a different site. Follow the
[cutover runbook](../devops/runbooks/icarus-cloud-cutover.md) after 0058 supplies
the complete identity/application composition and a maintenance window.

`cutover-plan` writes a read-only maintenance review and preserves the foundation
record. After approving that private capture and retiring the reviewed writers,
`make foundation-adopt ENV=icarus-cloud ...` rechecks the source/input/topology
fingerprints and live Kubernetes, node and Ceph identities before recording the
existing foundation. The Kubernetes API and every kubelet must match the pinned
release. Follow the [cutover runbook](../devops/runbooks/icarus-cloud-cutover.md#record-the-approved-foundation)
for the exact sequence and repeat adoption after a policy change. Kubespray
foundations retain their installer-owned record.

Approved origins use HTTPS/443 with a distinct hostname per service; certificates
stay under their existing renewal owner. The Pharos browser origin uses one
`platform.<domain>` name for the web application and its same-origin API, native
`/auth/oidc/callback` and realtime routes; a separate direct API hostname is not
selected by implication. The native Site API uses the distinct `site.<domain>`
host, with only `/v1/site/commands/` and `/v1/site/ssh-access/` forwarded to the
HTTPS `site-api` Service. It performs scoped bearer authentication and does not
use browser OIDC or register a Site API redirect URI. The gateway verifies the
backend certificate for `site-api.pharos.svc` with the internal Site API trust.

Every distributed installation declares a `kubeflow` and an `llm-chat` origin. The
retained web's sidebar links to both, and to the `pharos` origin's Support view,
so delivery stops when either origin is missing. A new `kubeflow` origin also needs
its `https://<kubeflow-origin>/oauth2/callback` registered at the AAI, as
the [Kubeflow identity decision](decisions/0067-kubeflow-gitops-composition.md#identity)
describes.

Icarus-cloud currently selects the canonical `*.staging.pharosfactory.ai`
application names, while Argo, storage and the AAI issuer retain separately
sourced bindings. Danias owns DNS changes for `*.pharosfactory.ai`; DNS and
certificate existence remain qualification gates. The Site Agent and installer
use these public base origins without appending `/v1/site/`:

```sh
SITE_API_URL=https://site.staging.pharosfactory.ai
SSH_ACCESS_SITE_API_URL=https://site.staging.pharosfactory.ai
central.site_api_url=https://site.staging.pharosfactory.ai
```

Management routes keep the same restrictions.

Choose one certificate owner before installing the public edge. With the default
`edge_certificate_mode: existing`, an external issuer such as the CSLab inventory's
Certbot layout renews the certificates; the role verifies trust, hostname, expiry
and key match for `edge_certificate_files` and writes no port-80 listener. With
`edge_certificate_mode: certbot` the role owns the names instead: it installs
`/etc/nginx/conf.d/pharos-platform-http.conf`, which answers
`/.well-known/acme-challenge/` from `edge_acme_webroot` and redirects every other
request to HTTPS, requests one lineage per approved host with
`certbot certonly --webroot`, then installs the 443 configuration. Set
`edge_acme_email` and review with `edge_acme_staging: true` before a real site
switches its names. A renewal hook (`nginx -t && systemctl reload nginx`) is
recorded in each lineage, and the run checks it with `certbot renew --dry-run
--run-deploy-hooks`, because a plain dry run runs neither the recorded directory
nor the deploy hooks.
`edge_acme_server` selects an alternative ACME directory, which is how a
laboratory's own test CA is exercised without a public name.

`make deploy` installs the edge itself, right before it applies the root: it
writes the edge's environment from the policy's origins (exactly the edge
schema's keys) to the private state directory and runs `edge-distributed.yml` on
the inventory's `platform-edge` hosts. A certificate for a newly selected origin is
issued before the HTTPS configuration names it. The certificate owner above stays
the private inventory's choice, and the delivery run skips the `certbot renew`
dry-run probe; run the playbook with `edge_acme_renewal_check: true` to check
renewal on purpose.

The challenge listener must be the only server on the host that declares
`listen 80 default_server`, and the 443 file must be the only
`listen 443 ssl default_server`; retire a legacy virtual host in the same step
rather than alongside it, because `nginx -t` rejects the duplicate. The role
retires the nginx package's own welcome vhost, whose symlink also claims port 80,
and fails on any other occupant instead of removing it. A listener that fails that
check is rolled back before the run stops: the previous file is restored, a first
installation is removed and the retired welcome symlink is put back, so the host
keeps a configuration every later `nginx -t`, reload and restart accepts. A pending
reload survives a failed reload or an interruption after validation. Retrying with
an unchanged listener completes that reload before requesting certificates.

Outbound mail is one `smtp` block in the environment policy: the relay host,
port, TLS mode, the EHLO name the relay will see, the sender, the public API
address a member's client uses and the address blocks the sending pods may reach.
An enabled block must have all of them, including at least one address block; a
malformed one is refused before any site uses it. Sending workloads reach that relay directly over
STARTTLS and verify the relay's own certificate, so the address the relay permits
is the one it observes. The blocks are declared rather than resolved while
rendering, because a policy cannot name a host and a DNS answer would not be
deterministic: `smtp.egress` takes exact networks, refuses a default route, and
the delivery renders one owned `smtp-egress` NetworkPolicy in the `pharos`
namespace only while `smtp.enabled` is true, withdrawing it otherwise. The base
manifests stay site-neutral, so each environment declares its own relay.
The delivery renders the block into Metallo's Deployment beside the workloads it
places, as `LLM_ACCESS_SMTP_HOST`, `_PORT`, `_TLS`, `_FROM` and `_EHLO` plus
`LLM_ACCESS_PUBLIC_API_URL`, and renders nothing while `smtp.enabled` is false.
The SMTP `public_api` origin must match `edge.origins.llm-api`; delivery appends `/v1`
for the client base shown in personal-key emails. The workload therefore reads one
reviewed source: it keeps mailing off without a host
and refuses to start without a sender and the public API address. The EHLO name
matters because a container's own name is not what the relay knows the site by. Waldur and Zammad take the same
values when their profiles are enabled: Zammad's desk sends its outgoing mail to
that relay, with the policy's TLS mode and certificate verification, in place of
the Mailpit sink it bundles, and only while `smtp.enabled` is true. A password,
if one is ever needed,
follows the external-identity pattern: a private file with mode 0600 rendered to
a per-namespace Secret.

Waldur receives the relay, sender, STARTTLS or implicit TLS mode, and EHLO name
through its chart's mail values. It uses Django's standard certificate
verification. Its API, worker and scheduler restart when those values change.
Disabling the SMTP policy removes the relay from the generated Waldur values.
Waldur's namespace currently restricts ingress only; the `pharos` SMTP egress
policy above applies to Metallo.

Project invitations use the HTML and plain-text templates in
[`gitops/manifests/waldur/chart/files/invitations`](../gitops/manifests/waldur/chart/files/invitations).
The generator inserts this environment's Themelio `/dashboard` URL. The recipient
signs in with the invited address and accepts the pending invitation there.
Other invitation scopes retain Waldur's own acceptance link. The post-sync
bootstrap applies the two overrides through `waldur override_templates` on every
sync, so existing installations receive template changes. Other stored templates
are preserved. Template customisation does not enable a disabled notification.

To activate this on an existing site:

1. Include the generator and chart changes in the site's reviewed deployment
   revision and deliver it through the normal environment workflow. The SMTP
   policy already enabled in Icarus Cloud needs no additional policy change.
   Wait for Waldur's API, worker and scheduler rollouts and the post-sync
   `pharos-bootstrap` Job to succeed.
2. With the site's administrator kubeconfig, check the worker's effective
   `EMAIL_HOST`, `EMAIL_PORT`, `EMAIL_USE_TLS`, `EMAIL_USE_SSL`,
   `DEFAULT_FROM_EMAIL` and `django.core.mail.utils.DNS_NAME.get_fqdn()`.
   Confirm `users.invitation_created` is enabled under Waldur's Notifications
   administration and that the two stored invitation templates contain the
   current dashboard URL.
3. Create one project invitation to an explicitly approved test recipient.
   Confirm receipt of both HTML and plain-text content, the sender, expiry,
   dashboard link and acceptance after AAI sign-in. Check the worker and relay
   logs if delivery fails. A successful TCP or STARTTLS handshake establishes
   connectivity; mailbox receipt and project membership establish the journey.
4. After delivery is qualified, review affected invitations and resend those
   still pending. Replace expired invitations through the normal invitation
   workflow. SMTP activation affects Waldur notification mail generally, so
   review its existing pending work before rollout.

The separate TLS client (the `smtp-proxy` role) is retained for a host-local
sender only; it was never wired to the policy and is not part of this path. It
validates upstream TLS, delays relay DNS until connection time and never shares
an Nginx startup dependency.

Test the edge configuration without sending mail:

```sh
uv run --project devops --frozen python scripts/distributed_edge_smoke.py
```