# Local AAI A local Keycloak shaped like GRNET AAI, shared by everything on a [Pharos dev box](../README.md): Pharos login, user attributes, offline refresh and the login clients of dev-box components such as Waldur, with your own test users. It runs in the separate `pharos-aai` namespace and does not change the `metallo-devbox` stack. The [accepted decision](../../../docs/decisions/0035-local-aai-laboratory.md) records its boundaries. ## Provider selection Local Keycloak is the default. Pharos DX and Make can select a hosted AAI using registered clients held in a private file. See [choose the sign-in provider](external.md) for the switch, CLI steps, callbacks, trust and switching back. ## Deploy Run `make deploy TARGET=dev-` for the first installation. Thereafter, `make reconcile` delivers committed AAI configuration and image changes alongside the other platform services. The `pharos-aai` Argo Application owns Keycloak and its administration Jobs. Shared PostgreSQL becomes healthy first, followed by stable credential delivery, the private Keycloak database, bootstrap administrator, runtime and realm configuration. The parent waits until each child is Synced and Healthy at the delivered source, with no pending or failed operation, before starting its consumers. Delivery passes a checksum of the private overrides and issuer trust to `pharos-aai`, so Argo reruns these Jobs when either changes. The host edge retains the existing public issuer and development CA. Passwords are generated once in the cluster. Retrying a failed hook or repeating reconciliation preserves user and client secrets. Runtime Keycloak receives only its database credentials. The credential Job may update explicitly named Secret slots in the consumer namespaces; it cannot create or list arbitrary Secrets. Each target exports private probe and Icarus bindings to `.local/local-aai/` after Argo passes its health gate. Icarus remains an independent laboratory and reads these bindings without installing AAI. ## Open it ```sh make devbox-forward TARGET='dev-' ``` Run it on the machine where the browser runs and keep it open. It forwards loopback port 8443 to the target's VM using the workstation's saved `SSH` setting and verified host key. OpenSSH runs directly on the workstation; this command needs no Docker container or separate SSH-key setup. Agent forwarding and connection multiplexing are disabled. Run `make handback` before opening this operator tunnel; an occupied port is refused. Port 8443 serves one target at a time. Press Ctrl+C to close the tunnel, then `make takeover` to return to local development. Trust `.local/local-aai//ca.crt` as a development certificate authority. On Windows, copy that public certificate to a convenient path and use: ```powershell certutil -user -addstore Root C:\path\to\local-aai-ca.crt ``` | Purpose | URL | |---|---| | Issuer | `https://pharos.127.0.0.1.sslip.io:8443/auth/realms/pharos` | | Staging-like issuer | `https://pharos.127.0.0.1.sslip.io:8443/auth/realms/pharos-baseline` | | Account console | `https://pharos.127.0.0.1.sslip.io:8443/auth/realms/pharos/account` | | Administration | `https://pharos.127.0.0.1.sslip.io:8443/auth/admin` | The name resolves to loopback; the tunnel carries it to the VM. Pods reach the same name through the node address, so browsers and services see one issuer URI. The temporary administrator is `lab-bootstrap`; its password is `bootstrap_admin` in the private `.local/local-aai//private.json`. ## Users [`users.json`](../../../devops/aai/users.json) defines the default `alice` and `bob`. To use other identities, copy it to `.local/local-aai//users.json`, edit the copy and repeat `make reconcile`. Each entry needs `username` and `posix_username`; optional fields are `email`, `first_name`, `last_name` and `attributes`. Attribute values are JSON lists. `voperson_external_affiliation` and `entitlements` accept several values and are released as lists. `voperson_id` and `schac_home_organization` take exactly one value and are released as strings. Listed users are created or updated in both realms, reusing their existing generated passwords. Passwords are in the private `.local/local-aai//test-users.json`. Users removed from the file are left in Keycloak; delete them in the administration console. `posixUsername` and the other attributes are visible to users but editable only by administrators. ## Clients and Site metadata [`clients.json`](../../../devops/aai/clients.json) lists exactly what Pharos registers with GRNET AAI: login clients and public Site trust metadata. On staging the same deployments receive the registered values from the vault; on a dev box they receive this Keycloak's. To try a change on one dev box, copy the file to `.local/local-aai//clients.json`, edit the copy and repeat `make reconcile`. The Argo credential Job renders each service's `gitops/manifests//aai-client.yaml` into a Secret in that service's namespace, with the client and this AAI's public CA. Dev-box browser aliases live in [`devbox-redirect-uris.json`](../../../devops/aai/devbox-redirect-uris.json), never in `clients.json`: the tracked registry is the list GRNET registers, and a `localtest.me` name has no place there. The deploy delivers the tracked registry plus those aliases through the private `identity-overrides` Secret, so only this Keycloak registers them. | Entry | Who uses it | Exchange | |---|---|---| | Client `pharos-control-plane` | Pharos web and API (Themelio), including the Icarus control plane: login, offline consent | none | | Client `pharos-gitlab` | GitLab login (OpenID Connect with PKCE); a new registration with GRNET | none | | Client `pharos-open-webui` | Chat sign-in: who the person is. Who may use which model is kept in Open WebUI's own groups (ADR-0020), never in this realm | none | | Client `pharos-tools` | Waldur login (`/api-auth/keycloak/complete/`), later Kubeflow and MLflow; redirect URIs are added as each is ported. Receives `site_username` by default, as staging releases it, for Waldur's `identity_claim` usernames | none | | Site metadata `pharos-icarus` | Icarus issuer/JWKS/CA export; no OIDC client or resource scope is created | none | Client fields: `client_id`; `vars`, the prefix of its deployment variables; `redirect_uris`, absolute `http(s)` URIs, `{public_url}/…` for the issuer origin or `{service_url:}/…` for a service host beside it, where only a final `/*` may be a wildcard; `post_logout_redirect_uris`, where Keycloak may send the browser after a sign-out the client asked for (the same forms, each exact: no wildcard and no `#`; none by default, so a client that sends `post_logout_redirect_uri` without one gets "Invalid redirect uri"); `pkce` (default true); `scopes` (default) and `optional_scopes` from `profile`, `email`, `site_username`, `voperson_id`, `voperson_external_affiliation`, `schac_home_organization` and `entitlements`; and `offline` for offline consent and refresh tokens. Site metadata fields are `id` and `vars`, the prefix of its issuer/JWKS/CA bindings. A new HPC Site is a new metadata entry. Site IDs and variable prefixes must be unique. Every client is confidential, uses the authorisation-code flow with PKCE and has no service account, direct grants or token exchange. The `pharos-baseline` realm keeps only `pharos-control-plane`, with staging's current behaviour: refresh tokens and token exchange disabled. Clients removed from the file are left in Keycloak; delete them in the administration console. For clients still declared, repeated deployment reconciles both default and optional scope assignments, removing access that is no longer configured. `make reconcile` writes each entry's variables, in the vault's format, to the private `.local/local-aai//oidc/.yml`. The names are the ones the deployment roles already read, for example the legacy Themelio role's `themelio_oidc_issuer`, `themelio_oidc_client_id` and `themelio_oidc_client_secret`, and the `shared_oauth2_*` that Waldur, Kubeflow and MLflow read. Site metadata receives `_issuer` and `_jwks_uri`, and no secret. Each file also names `_ca_file`, the development CA: unlike GRNET's public certificate, deployments must trust it, and their Pods reach the issuer through the node address on port 8443, so they need a host alias for the issuer name and an egress rule to that port. ## Realms Both realms follow the staging token contract verified on 26 October 2026: ordinary RS256 access tokens carry `site_username`, have a 3600-second lifetime, no `aud`, and `azp` identifying the login client. Offline consent permits refresh; ID tokens omit `site_username`. `pharos-baseline` contains only the control-plane login client, while `pharos` also registers the service clients. The committed registry enables no token exchange and creates no Icarus resource client. The normal Keycloak user-attribute mapper supplies `site_username` on login and refresh. The custom exchange mapper and its Maven build no longer exist. Old resource clients left by previous deployments have no attached exchange scopes and are not used. Remove retired clients through administration if desired. Other claim shapes retain the observed staging mappings: `voperson_id` appears in ID/access tokens; userinfo carries profile, email and provider attributes. `voperson_external_affiliation` and `entitlements` are arrays; `voperson_id`, `schac_home_organization` and `site_username` are strings. Waldur still reads `site_username` from userinfo. See [ADR 0035](../../../docs/decisions/0035-local-aai-laboratory.md) for Runtime admission, trust and the one-hour residual native-token lifetime. `pharos-control-plane` accepts the retained compatibility redirects and the exact API-owned callbacks at `https://themelio.localtest.me/auth/oidc/callback` and `https://platform.localtest.me/auth/oidc/callback`. Clients and resources come from the registry (see above). ## Verify The direct profile has a disposable real-Keycloak/HTTPS smoke lane: ```sh uv run --project services/metallo --frozen python scripts/aai_direct_smoke.py ``` It verifies PKCE login, offline refresh and production Runtime signature/claim admission in both realms. It removes its container and generated fixture secrets. Native Slurm and the two-VM Pipeline are separate checks. For concurrent two-VM qualification, select private `METALLO_MIXED_HANDOFF`, `METALLO_MIXED_ISSUER` and unused `METALLO_MIXED_SITE_PORT`/`METALLO_MIXED_GITLAB_PORT` values. The client is the ordinary `pharos-control-plane`; no exchange client is needed. Python reconciliation and probe regressions run in `pnpm verify`. With the forward open and SSH access to the saved target: ```sh METALLO_TARGET='dev-' node dx/common/local_aai/tests/identity-probe.cjs METALLO_TARGET='dev-' node dx/common/local_aai/tests/baseline-probe.cjs ``` Both use the first configured user and check the profile shape and its attributes. The probes read only their current client credential over SSH and keep it in memory. They also work after macOS SSH delivery, which leaves the full private credential file on the runner. The identity probe performs a real authorisation-code/PKCE login with offline consent and refresh. It checks userinfo and the ordinary direct access-token shape. The baseline probe confirms the staging-like token shape and denied exchange. Results without credentials are written under `.local/local-aai//evidence`. ## Operate ```sh ssh sudo k3s kubectl -n pharos-aai get pods sudo k3s kubectl -n pharos-aai logs deployment/keycloak ``` Administration jobs create the database and configure realms. Keycloak receives only its database identity; bootstrap credentials stay in administration jobs. Namespace NetworkPolicies permit only the required paths. Private files under `.local/local-aai/` contain credentials; do not copy them into source control. The Icarus laboratory consumes these generated variables for the control-plane client and Site resource. Its identity calls use the same HTTPS issuer and public CA as browser login. Private administrator state is not a consumer interface.