# Pharos HPC site installer Installs and runs the Pharos agents on an HPC login node, as root. Plain bash; the only Python is the agents' own virtual environment. It creates service users, unpacks one release under `/opt/pharos-hpc/releases/`, writes configuration under `/etc/pharos-hpc`, and starts the systemd units. It never changes LDAP, Slurm policy, or anything else on the host. ## Before you start - A release bundle (`pharos-hpc-site-vX.Y.Z.tar.gz`) from the Pharos release workflow, with its `.sha256` file. - LDAP, Slurm, NSS/SSSD and munge on this host, and one CPython **3.13** interpreter for the agents' own virtual environments ([below](#python-313)). This installer never installs Python: the wheelhouse carries `cp313` wheels only. `check` stops with `approved Python is missing: ` when the interpreter is not there, with `Python 3.13 is required, is ` when it is another version, and with `cannot create a virtual environment` when it cannot make one. - The Site API and Waldur tokens issued by the central step, and the two trust bundles, as regular files in one folder: `site-agent-token`, `site-agent-ldap-password`, `ssh-access-site-token`, `site-agent-ca.crt`, `site-api-ca.crt` ([the CA files](#the-two-ca-files)). `configure` takes that folder and copies what it finds. - The offering UUID of this site's Waldur offering. - A real `/scratch/pharos` directory for the member workspaces (`RUNTIME_WORKSPACE_ROOT`), and the Slurm client on this host. The installer creates neither. - The Slurm coordinator grant for the Runtime's own identity, which the installer only checks: `sacctmgr add coordinator account= names=pharos-runtime` and `sacctmgr modify user pharos-runtime where account= set MaxSubmitJobs=0` ([the Slurm Runtime](#the-slurm-runtime)). - The Slurm coordinator grant for the **Site agent's** identity. Waldur creates a Project's native account by asking its Slurm client to add it, and Slurm allows that only to admins, operators and coordinators of the parent account; without it the first order fails with *Only admins/operators/coordinators can add accounts*. `check` reads the row and stops with the command when it is missing: ```sh sudo sacctmgr add coordinator account= names=pharos-agent ``` The agent that runs on the site is not necessarily this installer's: the Icarus administrators run it from `/opt/pharos-hpc-agents/venv`, so only the Slurm grant belongs to this check. ### Python 3.13 The installer installs no Python and no `venv` module. Debian 12 ships 3.11, and Debian's `python3` has no `ensurepip` until `python3-venv` is installed, so pick one of these cases and leave the result in `site.env`. **Debian 13**, with the system interpreter: ```sh sudo apt-get install -y python3 python3-venv ``` ```text /etc/pharos-hpc/site.env: PYTHON=/usr/bin/python3 ``` **Debian 12, or a pinned interpreter**: install the same python-build-standalone build as the Icarus laboratory, `3.13.16+20261009`, and unpack it under `/opt/pharos-hpc/python/3.13`. ```sh build=https://github.com/astral-sh/python-build-standalone/releases/download/20261009 archive=cpython-3.13.16%2B20261009-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz curl -sSfLO "$build/$archive" printf '%s %s\n' 24801ea31c7d406d559a17e2226325c723d50c56900b9c0da7b62adaae2affc9 "$archive" \ | sha256sum -c - sudo install -d -m 0755 /opt/pharos-hpc/python/3.13 sudo tar xzf "$archive" -C /opt/pharos-hpc/python/3.13 --strip-components=1 /opt/pharos-hpc/python/3.13/bin/python3 --version # Python 3.13.16 ``` ```text /etc/pharos-hpc/site.env: PYTHON=/opt/pharos-hpc/python/3.13/bin/python3 ``` The checksum is the one pinned in `dx/metallo/icarus-hpc/ansible/roles/icarus_hpc/defaults/main.yml`. That build is x86_64; on another architecture take the matching asset of the same release. A later 3.13 patch release is accepted as well: the release records the patch it was built with. ### The two CA files `site-agent-ca.crt` is the trust bundle for `CENTRAL_WALDUR_API_URL`; `site-api-ca.crt` is the bundle for `CENTRAL_SITE_API_URL`. When both endpoints are publicly trusted — staging uses Let's Encrypt — each file is a plain copy of the host's own bundle: ```sh sudo install -m 0644 /etc/ssl/certs/ca-certificates.crt /path/to/tokens/site-agent-ca.crt sudo install -m 0644 /etc/ssl/certs/ca-certificates.crt /path/to/tokens/site-api-ca.crt ``` With a private CA, the same two files carry that CA's certificate instead, after its fingerprint is checked against the one the platform publishes; put it in both, because the two endpoints may have different issuers. `jwt-ca.crt` is not needed. `configure` copies it when it happens to be in the folder, but nothing in this installer reads it and `check` does not ask for it. ### The Slurm Runtime The Runtime is the site half of a run: it takes the commands the control plane leaves in the Site API, submits them to Slurm as the member, moves data in and out of the member's workspace, and reports the result. It has its own virtual environment inside the release, its own service identity and its own two journals, and it runs as `pharos-runtime` under `pharos-slurm-runtime.service`. - **Journals.** `delivery.sqlite` and `jobs.sqlite` live in `/var/lib/pharos-hpc/runtime`, never inside a release, so a release switch or a rollback keeps a member's accepted work. `activate` creates the pair as `pharos-runtime` on the first run; `check` refuses a single journal or a pair that belongs to another user. - **Workspace root.** `RUNTIME_WORKSPACE_ROOT` is the existing directory that holds member workspaces — `/scratch/pharos` on Icarus. The installer does not create or change it, and the Runtime writes only its own `.runtime` box inside it. `check` requires it to be `root:pharos-users` mode `1770`, so members create their own entries and the sticky bit keeps them apart, and it requires the Runtime's identity to be in `pharos-users`: ```sh sudo chown root:pharos-users /scratch/pharos && sudo chmod 1770 /scratch/pharos sudo usermod -aG pharos-users pharos-runtime ``` A site that names scratch by allocation sets `RUNTIME_WORKSPACE_ALLOCATION_PREFIX` to the Site agent's `allocation_prefix`. Each member's root is then `//`, and the slug is the Slurm account without that prefix. An account without the prefix is refused. The account directory above the member root comes from the Site's own provisioning (the Waldur ``order_process`` that creates the shared allocation directory); the member's stage job creates the member root below it (see 'Member workspaces'), and `check` does not look inside the root. Icarus names its accounts `pharos-a-` — `pharos-a-ssh-e2e-20` today — so its agent gets `NATIVE_SLURM_ALLOCATION_PREFIX=pharos-a-` and its Runtime the same string in `RUNTIME_WORKSPACE_ALLOCATION_PREFIX`; `check` warns when the two differ, because the Runtime then refuses every account the agent creates. Changing a prefix renames accounts, so a running site keeps the prefix it already has. A site that keeps scratch under the **whole** Slurm account, as `/nfs/scratch//`, sets `RUNTIME_WORKSPACE_LAYOUT=account` with `RUNTIME_WORKSPACE_ROOT` pointing at the shared parent (`/nfs/scratch`) and no allocation prefix; nothing is stripped from the account name. `member` keeps the older `/` form. The layouts are mutually exclusive: `allocation` requires the prefix, and `member`/`account` refuse one. Because the shared parent is not the Runtime's own directory, that mode only requires the Runtime to traverse it. The `/` directory is the Site's; each `//` is created by the member's own stage job as that member's `0700` directory and verified per Attempt, never taken over. The credential box (`.runtime`) holds the askpass helper and the repository CA, so it stays the Runtime's own directory. It remains below `RUNTIME_WORKSPACE_ROOT` unless `RUNTIME_CREDENTIAL_ROOT` names another directory: a site that moves the workspace root to a shared scratch sets it to a directory it owns (for example the existing pharos scratch directory), so the Runtime never creates `.runtime` beside a site-wide root. The sandbox drop-in grants write access to the credential box, not to member scratch. **Changing the layout is a state change.** A saved Attempt records the workspace path it planned, and the Runtime refuses a record whose path differs from the current layout, so every Attempt must be terminal (or explicitly disposed by an operator) before the setting changes. The same happens to a stage the Runtime rendered but had not submitted yet: it is compared byte for byte with the fresh render before the submit, and a difference is reported (`Prepared Slurm stage changed before submission`) instead of submitting a script the record never held. An Attempt interrupted between saving a stage and submitting it must therefore finish under the release that saved it, or be disposed of, before an upgrade changes how a stage is rendered. - **Slurm identity.** The Runtime runs `squeue`, `sacct` and `scancel` as its own Unix user, with munge, so it needs the coordinator role on the site's root account and must never submit: ```sh sudo sacctmgr add coordinator account= names=pharos-runtime sudo sacctmgr modify user pharos-runtime where account= set MaxSubmitJobs=0 ``` `check` reads both and stops with the exact command to run when either is missing. The installer never changes Slurm policy. - **slurmrestd transport.** `NATIVE_SLURM_REST_URL` is the API origin and `configure` still requires it; `NATIVE_SLURM_REST_UNIX_SOCKET` optionally names slurmrestd's Unix socket, for example `/run/slurmrestd/slurmrestd.socket`, and is written to `runtime.env` as `SLURM_REST_UNIX_SOCKET`. The Runtime itself accepts an origin **or** a socket: started with `SLURM_REST_URL=` empty and `SLURM_REST_UNIX_SOCKET` set it uses the socket alone, and a non-empty plain `http://` origin that is not a loopback is refused (`Plain HTTP is permitted only for a loopback slurmrestd`; a loopback `http://` origin also needs `SLURM_REST_ALLOW_INSECURE_LOOPBACK=true`). A socket-only site therefore keeps the installer's URL setting and starts the Runtime with those two variables on its command line after the env file, until `configure` can express a socket without an origin. It replaces only how the bytes travel: the request path stays `/slurm/v0.0.42/...` and the Runtime still sends the member's own `X-SLURM-USER-NAME` and `X-SLURM-USER-TOKEN`, so the socket must run with `AuthAltTypes=auth/jwt`. `rest_auth/local` is a different mechanism that authenticates the peer's Unix uid, and this Runtime does not use it. The socket's owner and mode are the site's settings; the Runtime only connects. Turning the setting on is a `configure` run and a restart of the Runtime, which keeps its journals and does not resubmit anything. - **Staging.** `NATIVE_STAGING_PARTITION`, `NATIVE_STAGING_CPUS`, `NATIVE_STAGING_MEMORY_MIB` and `NATIVE_STAGING_TIME_MINUTES` describe the job that moves data, so that partition is the one that must reach the repository and the object store. The defaults are generic (1 CPU, 512 MiB, ten minutes); a site that stages large repository objects raises the memory, because staging them can exceed 512 MiB and die with an out-of-memory kill (Slurm reports `OUT_OF_MEMORY` and the Attempt fails). While `NATIVE_STAGING_PARTITION` equals `NATIVE_COMPUTE_PARTITION`, `check` warns that the compute partition carries the stage jobs and their egress; the Runtime works either way. Icarus (9 November 2026) has an Internet-facing staging partition beside the offline compute one: `NATIVE_COMPUTE_PARTITION=gh200`, `NATIVE_STAGING_PARTITION=pharos-staging`, `NATIVE_STAGING_MEMORY_MIB=4096`. With 500 MB Git-LFS objects, three of the four qualification Runs were OOM-killed at the 512 MiB default and completed on a retry at 4096; the fourth (the initial NTUAGPU Run) had already passed at 512 MiB. 4096 MiB is the qualified Icarus setting; the minimum needed was not measured. `configure` writes those as `SLURM_PARTITION`, `SLURM_STAGING_PARTITION` and `SLURM_STAGING_MEMORY_MIB`; a Runtime started by hand uses the same three variables. The staging partition and its resources are saved with the Attempt when it is prepared, so a change needs a Runtime restart **and a new Attempt**: an Attempt that is already prepared keeps the partition and resources it saved. A stage job runs with `PATH=/usr/bin:/bin` (its environment also carries the Run-scoped repository credentials), so `git-lfs` must be visible there: `/usr/bin/git-lfs`, confirmed 3.6.1 on Icarus, not only an installation under `/usr/local/bin`. The stage's `git lfs pull` fails otherwise, even though a login shell finds `git lfs`. - **GPUs and wall time.** `NATIVE_SLURM_GPU_GRES` maps an accelerator class to the GRES the compute partition offers, for example `gh200=gpu:nvidia_gh200_144g_hbm3e`; `NATIVE_SLURM_MAX_WALL_SECONDS` is the longest wall time one compute job may ask for, for example `259200` (72 hours). `configure` writes them to `runtime.env` as `SLURM_GPU_GRES` and `SLURM_MAX_WALL_SECONDS`; empty or absent means the Site offers no GPUs and allows any time. Stage jobs never get a GPU and keep their own time limit. The class must agree on both sides: the action's `gpuClass`, the map here, and `gpu.` in the allocation limits. `check` refuses a malformed value and names the key, and warns when the map names a GRES that `NATIVE_COMPUTE_PARTITION` does not offer. For Icarus, from the documentation at `https://icarus.hpc.ntua.gr/docs/` (`resource-queue-and-basic-tier-limits`, `compute-tiers`, `job-script-template`): `NATIVE_COMPUTE_PARTITION=gh200`, `NATIVE_SLURM_GPU_GRES=gh200=gpu:nvidia_gh200_144g_hbm3e` (two GH200 GPUs per node) and `NATIVE_SLURM_MAX_WALL_SECONDS=259200`. The 72 node-hour limit is 72 hours on one node, and one Run is one node: the rendered job sets no node count and the domain refuses `distributed` and `jobArray`. The QoS is the allocation's default `standard`, so nothing sets `--qos`. The node figures on those pages, 144 cores and 960000 MiB, are shown as an example and are not confirmed against the running cluster; neither have the templates been run there. - **Repository CA.** If the GitLab the stage jobs clone from does not have a public certificate, `NATIVE_REPOSITORY_CA_FILE` names the CA file on this host; `configure` writes it to `runtime.env` as `REPOSITORY_CA_FILE` and the Runtime hands it to the stage job as `GIT_SSL_CAINFO`. Empty or absent writes no line, and git uses the host trust bundle. `check` refuses a relative path or a file it cannot read. - **Owner connections.** Accepted work, current Project/Run authority, Run-scoped repository grants and the per-action member token all come from the Site API over the Runtime's own scoped bearer; there is no separate credential per connection to fill in. `activate` refuses rather than leaving a unit that starts and exits, and `check` reports the Runtime's own message. - **Member workspaces.** Workspaces are member-owned directories with no filesystem ACLs, and the Runtime holds no member authority: it never traverses, rewrites or deletes member-private files, and it needs no sudo, no impersonation and no world-writable scratch. The member's stage job starts in the account directory the Site's provisioning already made (`/`), and its own script creates the member's root, `/scratch/pharos/` by default, as that member's `0700` directory: `mkdir -m 0700` and an explicit `chmod g-s`, because an account directory created with the setgid bit would otherwise pass it on. A root that already exists is used only when it is the member's own `0700` directory; a symlink, a foreign owner, an unsafe mode or an inherited setgid bit is refused (`untrusted scratch root`, exit 70) and never taken over. The Attempt and its `main_repo` are then created with `mkdir -m 0700` without `-p`, so no site-created Prolog is needed for a first Run. Slurm opens a job's output file before the batch script runs, so a stage-in without a Run-log spool declares `/dev/null` and the script opens the member's own log (`/pharos--in.log`, `0600` in the `0700` root) once the root exists; a clone that fails before that point leaves no log, which is the early-error visibility limit of this bootstrap. A configured Run-log spool keeps its existing spool path instead. The one thing the Runtime writes is its own credential box, below. - **The credential box.** The Run-scoped Git username and token never touch the scratch: the Runtime puts them in the stage job's own Slurm environment when it submits the job, which Slurm shows only to the job's owner, the member, and to its administrators. A site that sets `AccountingStoreFlags=job_env` also stores job environments in slurmdbd; leave it unset. The box, `/scratch/pharos/.runtime//git-/`, holds only what is not secret: the askpass helper that hands Git those two values from the environment, and the repository's CA. It is owned by `pharos-runtime`, mode `0711` at the box, `0710` at the Attempt and `0750` at the stage directory, and members reach it through the `pharos-users` group rather than an ACL. The Runtime deletes it when the stage releases the grant. Before activation, the Runtime creates or validates `.runtime` as its own Unix user. `configure` records that exact directory in `/etc/pharos-hpc/runtime-workspace.conf`; activation installs it as `pharos-slurm-runtime.service.d/50-pharos-workspace.conf`. This grants write access to the credential box under `ProtectSystem=strict`; member scratch remains read-only to the service. `RUNTIME_WORKSPACE_ROOT` must be an absolute, normalised path using letters, numbers, `/`, `_`, `.` and `-`. - **Accounting comments.** The Runtime identifies a job that has left the queue by the comment it wrote at submission, `pharos:`, read from `sacct`. slurmdbd keeps that comment only when the site's `AccountingStoreFlags` includes `job_comment` (for example `AccountingStoreFlags=job_comment,steps`); `job_env` is a different, secret-bearing flag and stays unset. Without `job_comment` a finished job has no readable identity marker, so the Runtime cannot confirm the record and the Attempt waits instead of finishing. The setting applies to jobs submitted after a slurmctld restart, and rows already written without a comment stay that way. The installer never changes Slurm policy: `check` only reads the setting and names it. ## Quick start ```sh sha256sum -c pharos-hpc-site-vX.Y.Z.tar.gz.sha256 tar xzf pharos-hpc-site-vX.Y.Z.tar.gz cd hpc-site-vX.Y.Z sudo install -d -m 0750 /etc/pharos-hpc/secrets sudo install -m 0640 site.env.example /etc/pharos-hpc/site.env sudoedit /etc/pharos-hpc/site.env # fill in every APPROVED_ value sudo ./pharos-hpc install # users and one release sudo ./pharos-hpc configure /path/to/tokens # configuration, secrets, offering components sudo ./pharos-hpc check # read-only, changes nothing sudo ./pharos-hpc activate site-agent ssh-access # link, units, start, health sudo ./pharos-hpc status ``` `configure` copies the token files it finds in the directory you give it into `/etc/pharos-hpc/secrets/` unchanged and sets their modes. Without a directory it uses the files already there. It also reconciles the offering's components in Waldur with the Site agent's configuration, through the upstream `waldur_site_load_components`; a repeat is a no-op. ## Commands | Command | What it does | | --- | --- | | `check` | Reads the release, `site.env`, secrets, host, units and the Waldur offering. Changes nothing. | | `install` | Creates the two service accounts and one release with offline virtual environments. | | `configure [TOKEN_DIR]` | Writes `/etc/pharos-hpc/site-agent.yaml`, `site-agent.env`, `ssh-access.env` and adopts secrets. | | `activate [COMPONENTS]` | Points `current` at this release, installs the units, initialises the Runtime journals, starts and checks them. Default: `ACTIVATION_COMPONENTS`, which carries `site-agent ssh-access runtime`. | | `status` | Active release, `current`/`previous` links, unit state. Read-only. | | `rollback [RELEASE_ID]` | Activates `previous`, or the named installed release. | | `stop` | Stops and disables this site's agents. Keeps releases and configuration. | `activate` and `rollback` return to the previous release by themselves if a unit does not stay active or the offering is not `Active`. ## Configuration Everything lives in `/etc/pharos-hpc/site.env` (`root:root`, `0640`). One `KEY=value` per line, no quotes. `site.env.example` carries every key with a comment. Secrets are files, never values in this file: ``` /etc/pharos-hpc/secrets/ 0750 root:root /etc/pharos-hpc/secrets/site-agent-token 0640 root:root /etc/pharos-hpc/secrets/site-agent-ldap-password 0640 root:root /etc/pharos-hpc/secrets/runtime-site-token 0640 root:pharos-runtime /etc/pharos-hpc/secrets/ssh-access-site-token 0640 root:pharos-ssh-access /etc/pharos-hpc/secrets/site-agent-ca.crt 0640 root:root /etc/pharos-hpc/secrets/site-api-ca.crt 0640 root:root ``` `configure` copies what it finds into this directory and ends by making every file there `0640` in the root-only directory; the source folder may hold them as the central step made them. The two CA files are also copied to `/etc/pharos-hpc/site-agent-ca.crt` and `/etc/pharos-hpc/site-api-ca.crt` with mode `0644`: they are public trust material, and the agents run unprivileged and cannot read the root-only `secrets` directory. The service account names `pharos-agent` and `pharos-ssh-access` are fixed, because the unit files are final files; your UIDs and GIDs are not. Changing a token is a `configure` run, not a new release. ### Directory layout The access agent reads one subtree and finds a member there by `entryUUID`; the Site agent writes the members. Both write the same tree, so both are told where it is: | Setting | Meaning | Default | | --- | --- | --- | | `NATIVE_PEOPLE_BASE` | The subtree the access agent reads. | required | | `NATIVE_SITE_AGENT_LDAP_BASE_DN` | `base_dn` for the Site agent. | `ou=pharos,$NATIVE_LDAP_SUFFIX` | | `NATIVE_SITE_AGENT_LDAP_PEOPLE_OU` | `people_ou`, relative to `base_dn`. | `ou=people` | | `NATIVE_SITE_AGENT_LDAP_GROUPS_OU` | `groups_ou`, relative to `base_dn`. | `ou=groups` | | `NATIVE_SITE_AGENT_LDAP_BIND_DN` | The Site agent's bind DN. | `cn=pharos-agent,ou=services,$NATIVE_LDAP_SUFFIX` | | `NATIVE_SSH_ACCESS_BIND_DN` | The DN the ACL grants `entryUUID` to. | `cn=pharos-ssh-access,ou=services,$NATIVE_LDAP_SUFFIX` | The upstream agent writes a member as `uid=,,`, so the two settings name `NATIVE_PEOPLE_BASE` between them. `check` refuses a pair that names a different tree and refuses a value that is not a distinguished name. Left out, the defaults reproduce the layout this bundle was written for, `ou=people,ou=pharos,$NATIVE_LDAP_SUFFIX`. Icarus nests People above Pharos and binds its agent at the suffix root, so its `site.env` carries: ``` NATIVE_LDAP_SUFFIX=dc=hpc,dc=ntua,dc=gr NATIVE_PEOPLE_BASE=ou=Pharos,ou=People,dc=hpc,dc=ntua,dc=gr NATIVE_SITE_AGENT_LDAP_BASE_DN=dc=hpc,dc=ntua,dc=gr NATIVE_SITE_AGENT_LDAP_PEOPLE_OU=ou=Pharos,ou=People NATIVE_SITE_AGENT_LDAP_BIND_DN=cn=pharos-agent,dc=hpc,dc=ntua,dc=gr ``` [`access.ldif.template`](../../../services/metallo/deploy/ssh-access-agent/directory/access.ldif.template) carries those names as `@PEOPLE_BASE@`, `@BASE_DN@`, `@AGENT_DN@` and `@SSH_AGENT_DN@`; every renderer in the repository fills all four and refuses a template with a placeholder left. Apply the ACL with this site's values, or the access agent reads nothing where it looks and stays `pending`. **One identity, today.** On Icarus the access agent binds as the Site agent's own DN, which can write the whole people tree. A separate, restricted SSH identity — one DN that may write only `sshPublicKey` — is an administrator task; until it exists, `NATIVE_SSH_ACCESS_BIND_DN` names the shared identity and this bundle must not be described as finished for members. ### Reading the directory by hand The agents reach the directory over `ldapi` with SASL EXTERNAL and peer credentials, and need no password; `check` uses the same path. Run these as root: ```sh sudo ldapwhoami -H ldapi://%2Frun%2Fslapd%2Fldapi -Y EXTERNAL sudo ldapsearch -H ldapi://%2Frun%2Fslapd%2Fldapi -Y EXTERNAL -LLL \ -b ou=Pharos,ou=People,dc=hpc,dc=ntua,dc=gr dn ``` The socket path is part of the URI and keeps its slashes encoded (`%2F`); writing it unencoded makes `libldap` read the path as a host and fail with `ldap_sasl_interactive_bind_s: Can't contact LDAP server`. Without `-Y EXTERNAL`, or as a user other than root, the same call answers `authentication required`, because the ACL gives `manage` to local root only. A password-bound administrator is a separate decision, not a fallback: it needs its own bind DN, an `olcAuthzRegexp` mapping for it, and an ACL entry naming it. The installer adds none of these, `check` never asks for a password, and nothing here loosens the TLS or SASL defaults to make one work. ## UID and GID ranges The approved Icarus ranges are split by owner, so two allocators never issue the same number. `check` refuses a project-group range that overlaps the members' primary GIDs. | Use | Range | Allocator | | --- | --- | --- | | Member UIDs | `5000000–5499999` | Waldur POSIX pool | | Members' primary GIDs | `5500000–5549999` | Waldur POSIX pool | | Project group GIDs | `5550000–5599999` | Site agent (`ldap.gid_range_*`) | Set them in `site.env` with `NATIVE_MEMBER_UID_START/END`, `NATIVE_MEMBER_GID_START/END` and `NATIVE_PROJECT_GID_START/END`. Only the project-group range is written into the Site agent's configuration. Waldur keeps the members' pool **against the HPC offering**, not against the provider organisation: one organisation may sell several offerings, and a provider-wide pool would hand their users numbers out of the members' range. A role run therefore resumes above every identifier already handed out and never reissues one, and it adopts a pool the offering or its provider already has instead of adding a second, overlapping one. **Deleting a provider organisation deletes its POSIX pool and every POSIX identity with it.** Those rows are what record which UID and primary GID a member's directory account already carries; Waldur recreates neither, so the members keep their numbers with no record here and the next member gets none at all. Before deleting or merging a provider organisation, move the pool and its identities to the organisation that will carry the offering — or renumber the members with the Site's retention procedure — and only then delete. The qualification teardown in `scripts/qualify_site_agent.py` refuses a deletion that would cascade either one. The example also polls Waldur every minute for orders and membership changes, matching the laboratory and the accepted 120-second access window. Raise `SITE_AGENT_ORDER_PROCESS_PERIOD_MINUTES` and `SITE_AGENT_MEMBERSHIP_SYNC_PERIOD_MINUTES` only if this site's Waldur tolerates a slower cadence. Laboratory tests: switching an offering to `service_provider` makes Waldur clear the offering users' usernames, so set them again before the run - a real site gets them from the central side. ## What is where | Path | Content | | --- | --- | | `/opt/pharos-hpc/releases//` | Code only: `site-agent/` and `ssh-access/` virtual environments, `units/`, `release.yml`, `SHA256SUMS`. | | `/opt/pharos-hpc/current` | Symlink to the active release. | | `/opt/pharos-hpc/previous` | Symlink to the last release, for `rollback`. | | `/opt/pharos-hpc/site-agent` | Symlink through `current`; the agent unit runs from here. | | `/opt/pharos-hpc/ssh-access` | Symlink through `current`; the access agent runs from here. | | `/etc/pharos-hpc/` | `site.env`, generated configuration, `secrets/`. | | `/etc/systemd/system/` | `waldur-site-agent@.service`, `pharos-ssh-access-agent.service`, `pharos-hpc.target`. | | `/var/lib/pharos-hpc/` | The active release ID and the components that were activated. | Units are final files, not templates. They read `EnvironmentFile=/etc/pharos-hpc/.env` and take the access bearer through `LoadCredential=`. The Runtime's scratch write exception is a generated drop-in because its path comes from `site.env`. Activation updates only Pharos's four unit files and this drop-in. Existing unit masks and redirected destinations are refused; other host units and permissions are preserved. Release code and virtual environments remain readable and traversable when the administrator uses `umask 077`. Runtime journals keep mode `0700` on their directory, and secrets retain the private modes listed above. ## Upgrade and rollback One step at a time, and stop if a step says something you did not expect: ```sh cd hpc-site-vX.Y.Z+1 sudo ./pharos-hpc check # host, site.env, secrets, offering, units; changes nothing sudo ./pharos-hpc install # a new release directory; the running one stays sudo ./pharos-hpc configure # only if site.env changed; keeps the secrets it is given sudo ./pharos-hpc activate site-agent ssh-access sudo ./pharos-hpc status # the active release and components, without secrets ``` `rollback` returns both components to the previous release; `activate` with one component moves only that one, which is how a partly upgraded site is finished or undone: ```sh sudo ./pharos-hpc rollback sudo ./pharos-hpc activate ssh-access ``` **Which code is this?** Every Pharos wheel keeps version `0.1.0`, so a version number does not tell two builds apart. `release.yml` in the bundle records `source_revision`, the commit it was built from, `SHA256SUMS` covers every file, and both `.installed.json` in the release directory and `./pharos-hpc status` repeat the release ID. Compare those, not `pip show`, when deciding whether a host runs the fix you expect. **Secrets and the site's own files.** `install` writes only `/opt/pharos-hpc` and `/var/lib/pharos-hpc`; `configure` copies the secret files it is handed into `/etc/pharos-hpc/secrets` and does not rewrite `site.env`. An upgrade therefore does not rotate a token: hand `configure` the same folder, or skip it when only the code changed. An interrupted `install` leaves an incomplete release; install it again with the same `site.env`, or install a new release ID. Never edit a release directory: hashes are verified on every install and rollback. Apply the current `access.ldif.template` (`services/metallo/deploy/ssh-access-agent/directory/` in the bundle) before or during the upgrade, rendered with this site's names ([Directory layout](#directory-layout)). The access agent reads `entryUUID` to decide which entry is a member; a directory whose ACLs predate that rule answers every lookup with nothing and the agent stays `pending` for ever without an error. `check` renders the template with this site's values and warns when it does not grant `entryUUID` to `NATIVE_SSH_ACCESS_BIND_DN`. ## Releases and offline installs `dependency_mode: wheelhouse` releases carry every dependency wheel and install with `--no-index`; `index` releases take dependencies from `PYTHON_INDEX_URL` and still verify their hashes. Both use `--require-hashes`. The bundle's `SHA256SUMS` is checked before and after installation, and the official archives are Sigstore-signed by the release workflow. ## Troubleshooting - `check` first. It names the missing secret, the wrong Python, the refused unit or the offering that is not `Active`. - The access agent stays `pending` while the member exists in the directory: the ACLs do not let it read `entryUUID`, so apply the current `access.ldif.template` (see the upgrade notes) and restart `pharos-ssh-access-agent.service`. - `systemctl status waldur-site-agent@order_process` and `journalctl -u` show the agent's own error; `pharos-hpc status` shows which release is active. - A failed `activate` restores the previous release and says so. If it reports that nothing was active before, run `install` and `configure` again before activating. ## Removing the agent from this host ```sh sudo ./pharos-hpc stop sudo systemctl disable pharos-hpc.target sudo rm /etc/systemd/system/{waldur-site-agent@.service,pharos-ssh-access-agent.service,pharos-slurm-runtime.service,pharos-hpc.target} sudo rm /etc/systemd/system/pharos-slurm-runtime.service.d/50-pharos-workspace.conf sudo rmdir --ignore-fail-on-non-empty /etc/systemd/system/pharos-slurm-runtime.service.d sudo systemctl daemon-reload sudo rm -rf /opt/pharos-hpc /etc/pharos-hpc /var/lib/pharos-hpc ``` This removes the agent and its configuration only. Service accounts, LDAP entries, Slurm associations, member homes and jobs are yours to keep or remove. ## Optional Run logs Native Run logs require the separately reviewed [spool activation](RUN-LOGS.md). The normal installer leaves log dispatch, Slurm hooks and retention unchanged.