# Icarus Cloud application cutover

This is the one-time, reviewed application cutover prepared by
[DD-0051](../../docs/dd/0051-distributed-inventory-setup.md). Execute it only in
0058's maintenance window, after the complete external identity/application
composition and approved DNS/TLS/mail inputs are ready. Ordinary deploy never
performs it. It retains foundation and discards only explicitly selected old
application data. After data deletion, recovery is a fresh selected installation.

## Refresh and review

Run through the environment's recorded runner with the private native inventory
and verified administrator kubeconfig. Do not substitute the older local NTUA
reference for the live nine-node topology. Refresh effective Istio/Argo versions rather than
trusting old Helm metadata. Confirm Ceph's device identities, including the
previously unlisted device on `openst-nogpu09`, before any later storage change.

```sh
make deploy-plan ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
make cutover-plan ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
```

The private `cutover.json` contains current namespace/name/UID/resourceVersion
for every Application and claim, Kubernetes identities and Ceph FSID. It excludes
Helm values, Secret data and raw hostvars. Review the four observed legacy writers:

Capture is read-only apart from its private review file. It leaves `operation.json`
and cluster ownership unchanged, so it can observe the legacy writers before
retirement. It checks the exact Kubernetes API and kubelet pin and inventory
membership. Review the captured source, input and topology fingerprints along
with the identities; set `approved` to `true` in this private file after review.

| Order | Application | Action |
| --- | --- | --- |
| 1 | `argocd/ntua-apps` | Fence the parent and wait for all active operations to stop. |
| 2 | `argocd/argocd-istio` | Fence it; retain compatible effective Istio. |
| 3 | `argocd/themelio` | Fence the old application writer. |
| 4 | `argocd/waldur-site-agent-cloud` | Fence the old agent writer. |

Missing or additional Applications block Argo replacement until their ownership
and retention are reviewed. Do not delete an unknown Application/controller.
Inventory the exact selected GitLab/Waldur Helm release names/namespaces with
`helm list -A -o json`; record the owning release and current resource UIDs in
the maintenance review. Do not infer a release name from its browser hostname.

Populate `retire_claims` only with captured `{namespace, name, uid, owner,
disposable: true}` entries for the selected old applications. Keep Kubernetes/CNI,
Ceph/CSI/OSDs, GPU configuration, compatible mesh, registry, monitoring/Kubeflow
and shared CNPG/Valkey still serving retained consumers. There is no namespace-wide
or cluster-wide PVC deletion. No raw-device wipe is part of this runbook.
Rehearsal refuses all `gitlab` and `pharos-infra` claim retirement. Use the
[GitLab persistence recovery guide](gitlab-persistence-recovery.md) for its
database, Gitaly, object data and encryption secrets; the shared PostgreSQL claim
also serves retained consumers.

Rehearse the edited review without writing to Kubernetes:

```sh
uv run --project devops --frozen python - <<'PY'
import json, sys
from pathlib import Path
sys.path.insert(0, 'devops/platform/scripts')
from platform_deploy.cutover import rehearse
path = Path.home()/'.local/state/pharos/deployments/icarus-cloud/cutover.json'
print(json.dumps(rehearse(json.loads(path.read_text())), indent=2))
PY
```

## Fence writers before removing ownership

Re-read each captured object and compare UID/resourceVersion immediately before
its change. Use JSON Patch `test` operations for both, followed by `add` of
`/spec/syncPolicy/automated` with `null`. This works with the observed Argo 2.12;
do not rely on the newer `automated.enabled` field to disable that controller.
The patch has this exact structure, with values from the private capture:

```json
[
  {"op":"test","path":"/metadata/uid","value":"<captured UID>"},
  {"op":"test","path":"/metadata/resourceVersion","value":"<current reviewed version>"},
  {"op":"add","path":"/spec/syncPolicy/automated","value":null}
]
```

Use the fully qualified `applications.argoproj.io` resource: Icarus also has
`applications.app.k8s.io`, so its short `applications` name is ambiguous.
Apply it with `kubectl -n argocd patch applications.argoproj.io NAME --type=json --patch-file
/private/fence.json`. Fence the parent first. If an operation is running, terminate
it through the matching Argo administration API/CLI and observe quiescence before
fencing children. Disabled auto-sync alone does not terminate an active operation.
Re-read and test changed resource versions; never overwrite a concurrent update.

Inspect every Application finaliser. Retire Applications without cascading into
retained resources. Remove only a reviewed Argo resource finaliser where orphaning
is intended; unknown finalisers stop the operation. Use deletion preconditions
with the captured UID and the newly reviewed resource version. For Kubernetes
client calls, pass `V1DeleteOptions(preconditions=V1Preconditions(uid=...,
resource_version=...))`; the same condition applies to each approved claim.

Before replacing Argo, prove all old writers and operations are stopped and that
no retained Application depends on the old controller. Review the old controller's
Helm ownership, exact resources and CRDs. Preserve required CRDs and selected
objects; never run two competing controllers. Stop if the installed state differs
from the capture. A generic `helm uninstall argocd` is not this review.

Icarus also keeps the containerd configuration its installer wrote, which predates
the in-cluster registry: the nodes have no host entry for it, so a kubelet cannot
pull a built image. Run `containerd-registry.yml` across `kube_node` from the exact
source before `make deploy`. Each node then reads
`/etc/containerd/certs.d/registry.pharos-build.svc.cluster.local:5000/hosts.toml`,
which serves that host over plain HTTP through the registry Service; the node port
it also declares is not bound on the nodes and cannot be the pull path. containerd
reads the file per pull, so no restart interrupts running workloads. Remove the
file to revert the node to its recorded configuration.

Icarus served Argo from the legacy repository, so its objects carry no Helm
ownership and their controllers have fixed label selectors. The bootstrap role
first removes the four controllers whose selectors the pinned chart changes, then
installs the chart with `helm upgrade --install --namespace argocd --take-ownership
--force`, which adopts the remaining objects into the `argocd` release and
replaces the fields a merge patch would keep. Argo's state stays in its
ConfigMaps, Secrets and claims, which are patched in place; no claim is deleted.
The same role delivers the checksum-verified Git daemon package the private
transport reads. Run the bootstrap playbook from the exact source before the
first `make deploy`; delivery stops at the source hand-off while
`deployment/pharos-platform-git` is not ready.

## Record the approved foundation

After the reviewed legacy writers have retired and the replacement Argo bootstrap
has established its ownership record, record the existing foundation:

```sh
make foundation-adopt ENV=icarus-cloud INVENTORY=/private/ntua/hosts.yml KUBECONFIG=/private/admin/kubeconfig
```

This command requires the approved `cutover.json` to match the current source,
inputs and topology. It rechecks Kubernetes and Ceph identities, the exact API and
kubelet pin, retained Application identities and the absence of legacy writers.
It compares the live ownership version before naming the environment, then writes
`operation.json`. Delivery remains a separate command. A failed check preserves
the previous foundation record; an interrupted write can be retried with the same
review. Keep the original capture throughout the maintenance window.

For a later policy change, capture and approve the new inputs with `cutover-plan`,
then run `foundation-adopt` before `deploy`. The recorded environment permits its
existing composition during this later adoption. Never hand-edit `operation.json`.
A Kubespray foundation continues to use its installer-owned record.

## Selected application and edge replacement

Retire only the reviewed legacy application/agent resources and selected Helm
release objects. Inspect reclaim policies before deleting the explicitly approved
old claims; verify retained claim UIDs and Ceph identity afterwards. Reinstall
the new application state once through 0058's exact-revision Argo composition.
No generic schema/credential migration or automatic Kubernetes/Ceph upgrade runs.
GitLab and shared infrastructure persistence remain retained through this step.
An absent GitLab database after initialisation was claimed stops bootstrap for
the dedicated recovery procedure.

Retain unrelated Nginx virtual hosts. Review their files and SHA-256 digests before
retiring only the selected old platform vhosts. Remove the old SMTP stream include
from Nginx under that reviewed file change; the separate SMTP service owns relay
DNS/TLS. Public edge installation uses existing per-host certificate/key files and
their renewal owner, never the development CA. Validate the complete Nginx config
before reload. Confirm one route writer and management denial.

The approved From address and relay/IP permission require a real delivery check
in 0058. An alias, EHLO or STARTTLS alone does not establish delivery. Send no
test mail without the selected recipient and explicit instruction.

## External identity: issuer and registered callbacks

The site signs in through `https://staging.aai.pharos.grnet.gr/auth/realms/pharos`
with the public Let's Encrypt chain; the private `external-aai.json` holds that issuer,
the discovery/configuration documents and the ISRG Root X1 and X2 certificates. A
different realm answers exactly like a client that does not exist ("Client not found"),
so confirm the issuer before delivery instead of reading the login error as a missing
client.

The delivered `ae46f71fa` composition binds four consumer Secrets from that input:
`themelio-compat/themelio-oidc`, `gitlab/gitlab-oidc`, `waldur/waldur-oidc` and
`llm/llm-oidc`. All four carry the reviewed `pharos.dev/aai-inputs` fingerprint.
Themelio and Waldur copy it into Pod annotations; GitLab uses its first 40 characters
as a Pod label. Those three consumers roll when the input changes. Confirm their
fingerprints and replacement Pods after delivery, for example:

```sh
kubectl -n themelio-compat get secret themelio-oidc -o jsonpath='{.metadata.annotations.pharos\.dev/aai-inputs}'
kubectl -n themelio-compat get pods -o custom-columns='NAME:.metadata.name,AGE:.metadata.creationTimestamp'
```

Open WebUI has no fingerprint in its Pod template at that revision. It reads the
issuer and client credentials from `llm-oidc` into environment variables at startup,
and its init container copies the CA into a trust bundle. A Secret update therefore
requires an explicit rollout before chat can use the new values. After confirming
that `llm-oidc` carries the reviewed fingerprint, run:

```sh
kubectl -n llm get secret llm-oidc -o jsonpath='{.metadata.annotations.pharos\.dev/aai-inputs}'
kubectl -n llm rollout restart deployment/open-webui
kubectl -n llm rollout status deployment/open-webui --timeout=5m
kubectl -n llm get pods -l app.kubernetes.io/name=open-webui -o custom-columns='NAME:.metadata.name,AGE:.metadata.creationTimestamp'
```

Verify chat sign-in through the expected issuer after the replacement Pod is ready.
Themelio's successful rollout alone does not establish that chat has loaded the new
configuration. Repeat this step after an issuer, client credential or CA change until
Open WebUI has an automatic fingerprint rollout trigger.

Ask the provider (not this repository) for these exact callbacks, one client each; the
list is what the code sends, so it is also the list to compare against GRNET:

| Client | Callback |
| --- | --- |
| `pharos-control-plane` (Pharos web and API) | `https://platform.staging.pharosfactory.ai/auth/oidc/callback` |
| `pharos-control-plane` (retained Themelio) | `https://themelio.staging.pharosfactory.ai/api/auth/callback/openid_connect` |
| `pharos-gitlab` | `https://gitlab.staging.pharosfactory.ai/users/auth/openid_connect/callback` |
| `pharos-tools` (Waldur) | `https://waldur.staging.pharosfactory.ai/api-auth/keycloak/complete/` |
| `pharos-open-webui` (chat) | `https://chat.staging.pharosfactory.ai/oauth/oidc/callback` |
| `pharos-zammad` (support desk, optional) | `https://zammad.staging.pharosfactory.ai/auth/openid_connect/callback` |

`pharos-zammad` is a public PKCE client: register it without a secret (client
authentication off), with the `openid`, `email` and `profile` scopes and the valid
post-logout redirect URI `https://zammad.staging.pharosfactory.ai`, exactly, with no
trailing slash: Zammad's sign-out returns there, and Keycloak otherwise answers
"Invalid redirect uri". Put only its `client_id` in
`external-aai.json`. Without it Zammad's staff use the generated logins
(`gitops/manifests/zammad/README.md`).

A callback can be checked before delivery without a secret: request the realm's
`/protocol/openid-connect/auth` with only `client_id`, `redirect_uri`, `response_type` and
`scope=openid`. A registered callback falls through to the sign-in page ("Restart login
cookie not found"); a missing one stops with "Invalid parameter: redirect_uri".
A client that enforces PKCE, such as staging's `pharos-zammad`, instead redirects to its
callback with "Missing parameter: code_challenge_method", which also confirms the
registration; add `code_challenge` and `code_challenge_method=S256` to reach the
sign-in page.

`platform.staging.pharosfactory.ai` still signs in with the local operator access code:
the distributed composition does not include the `api-auth` Component, so the API serves
no browser session route and the web falls back to the development form. This is a
known gap of the first cutover, not an environment input; see the proposal recorded in
the hand-off notes rather than changing the cluster by hand.

## The scheme the edge terminated

The edge terminates TLS and dials the gateway's NodePort over plain HTTP, so the gateway
must trust exactly that one hop (`edge-trusted-hop`, `xff_num_trusted_hops: 1`). Without
it Envoy replaces `X-Forwarded-Proto` with http and the applications build http URLs:
Waldur sends `redirect_uri=http://…` and the provider answers "Invalid parameter:
redirect_uri", GitLab fails its CSRF origin check and returns to `/users/sign_in`.
Confirm after any gateway or edge change, with no credentials:

```sh
curl -s -o /dev/null -D - https://waldur.staging.pharosfactory.ai/api-auth/keycloak/init/ | grep -i location
curl -s -o /dev/null -D - https://gitlab.staging.pharosfactory.ai/users/auth/openid_connect | grep -i location
```

The first must carry `redirect_uri=https%3A%2F%2F…`; the second answers 302 to the AAI only
when the request carries the sign-in form's authenticity token, so fetch `/users/sign_in`
first and post that token, then read the `redirect_uri` in the location.

## Retained edge access and GitLab registration

The retained edge admits `147.102.0.0/16` (NTUA campus) to the management hosts: Argo
CD, the Ceph dashboard and Waldur's administration path. The policy is the routes
chart's `access.management`, rendered as `remoteIpBlocks` on the `edge-*`
AuthorizationPolicies in `istio-system`, so `make deploy` applies it; check it from a
campus address afterwards with `curl -o /dev/null -w '%{http_code}'` against both hosts.

Public GitLab registration is closed twice over: the chart's
`global.appConfig.initialDefaults.signupEnabled` covers a fresh installation, and
delivery sets and reads back the stored setting on every run. Delivery stops before it
hands the root actor a token when GitLab does not print the closed setting, so a failed
run leaves registration open and reports it. Confirm with `/users/sign_in` (no "Register
now"), `/users/sign_up` (302) and `ApplicationSetting.current.signup_enabled` in the
toolbox.

## Findings from the 2026-10-30 cutover

These were found in the first execution and are part of the reviewed procedure now.

- Bootstrap GPU admission before its grants: apply `gpu-crd-admission.yaml` and
  `gpu-role-admission.yaml` from `gitops/manifests/argocd/bootstrap/` with the
  administrator kubeconfig. Wait until both policies report
  `status.observedGeneration == metadata.generation`, a `status.typeChecking`
  result and no `expressionWarnings`, then apply `gpu-operator.yaml`. The Ansible
  role performs this ordering and check. The Argo role refuses to re-apply bootstrap objects while the owning
  Application still auto-syncs, so on an adopted site this file is applied directly
  and the role runs only after a fenced recovery.
- The retained Istio publishes its shared gateway on NodePort 30080, not the fresh
  installation's 30088: `edge.gateway_node_port` names the port in the environment
  policy.
- kube-proxy binds node ports on the node's own addresses only, so the edge dials
  the edge host's address (`edge_mesh_address`) rather than loopback. The packet
  still leaves the node for the gateway pod, so the gateway's source policy holds.
- A management route with an empty `edge.management_cidrs` denies every client.
  Name the networks the retired edge admitted (Argo CD, Ceph dashboard, Waldur
  administration) or the operators lose the interface.
- Follow the approved capture, retirement and `foundation-adopt` sequence in the
  [existing-site guide](../../docs/distributed-setup.md#existing-site) before
  `make deploy`, smoke checks and SMTP. A changed policy needs a new approved
  capture and foundation adoption before delivery.
- Enable SMTP only after the smoke checks: `smtp.enabled: true` makes the workload
  carry `LLM_ACCESS_SMTP_*` and the `smtp-egress` policy admit the relay address on
  its port. Prove the path from inside a sending pod (EHLO, STARTTLS, MAIL, RCPT,
  RSET, QUIT — never DATA) and then send one approved test message.
- Check etcd before the GPU child's manual sync (`etcdctl check perf` with
  `/etc/ssl/etcd/ssl/member-<host>.pem`): the sync applies CRDs and installs
  workloads, so it belongs to a quiet step after the smoke checks, never inside the
  critical path. Sync with a merge patch on `operation.sync` at the delivered
  revision; the child stays fenced otherwise.
- The GPU child's pre-upgrade hook needs three writers the writer policy must name:
  the platform's Argo controller (writes scoped to the rendered CRD names), the
  hook's own ServiceAccount
  (`gpu-operator-upgrade-crd-hook-sa`) and the garbage collector (finalizer of an
  object already being deleted). The CRD scope policy limits the hook and operator accounts to GPU
  definitions; generated role constraints preserve every chart role's permissions
  and binding subjects. A legacy hook binding left in foreground deletion
  by an earlier install is cleared by applying the same reviewed bootstrap file with
  the administrator kubeconfig.

## Stop and recovery conditions

Stop on changed cluster/node/Ceph identity, foreign owners, unexpected finalisers,
retained provider dependencies, unknown application/PVC ownership, invalid TLS,
failed readiness or an unreviewed public route. Leave new writes fenced.

Recovery uses `/etc/kubernetes/super-admin.conf`, which still carries
`system:masters`; from Kubernetes 1.29 kubeadm's `/etc/kubernetes/admin.conf` carries
`kubeadm:cluster-admins` instead, and the GPU writer policy accepts both.

Before old data deletion, restore the reviewed old writers/configuration only
after proving the new writers are stopped. After deletion, recover by reinstalling
the selected new state. Preserve the maintenance capture, exact source/input
revisions, and completed stages. Validate first deployment, repeat preserving new
data/credentials, controlled revision and interrupted retry once under 0058.