# GitLab persistence consistency and recovery GitLab's database, repositories, object data and encryption secrets form one recovery set. Repository identifiers are opaque provider bindings for Pharos; reassigning a numeric GitLab ID can attach an existing binding to different data. The Source Control owner and retention rules are in its [README](../../services/metallo/contexts/source_control/README.md). ## Evidence and scope Finding 9 in [#1199](https://github.com/pharos-ai-factory/pharos-platform/issues/1199), tracked by [#1123](https://github.com/pharos-ai-factory/pharos-platform/issues/1123), reported database projects 1–6 alongside Gitaly repositories for IDs 3 and 7–27. Moving the project sequence past 27 allowed new IDs but left the older repositories unaccounted for. Those observations do not identify the historical restore or reinstall command, and this change does not establish current live consistency. The checked-in deployment stores `gitlabhq_production` in shared `pharos-infra/postgres`, reached through `gitlab/gitlab-postgresql`. Gitaly uses a separate Ceph claim. LFS, uploads, artifacts, packages and backups use separate ObjectBucketClaims. The previous bootstrap created the database whenever it was absent; it could therefore recreate only one part of a retained installation. Generic cutover rehearsal now refuses retirement of claims in `gitlab` and `pharos-infra`. Even selecting both namespaces cannot establish a coherent restore: PostgreSQL serves other consumers, and GitLab's object data and encryption secrets have separate lifecycles. ## Database initialisation safeguard The existing `pharos-infra/infrastructure-generated` Secret records `gitlab_database_initialisation`. Its transitions preserve every credential field. | State | Meaning and permitted action | | --- | --- | | `new` | A fresh infrastructure record was created with no surviving credential copies or GitLab PVC/ObjectBucketClaims. One initialisation attempt may be claimed. | | `attempted` | The attempt was claimed before any SQL was allowed. A replacement Pod requires the database to exist. An absent database needs operator recovery. | | `ready` | The database setup SQL succeeded and the Job recorded completion. Subsequent setup requires the database to exist. | | Missing field on an existing record | Legacy installation: require the database to exist. | The database Job has two init containers followed by a receipt container: 1. `database-prepare` compares and changes `new` to `attempted` through a JSON Patch test. It writes a one-use creation script into the Pod's memory volume. All other states receive only guarded SQL, which contains no GitLab database creation statement. 2. `database-setup` waits for PostgreSQL's TCP endpoint, then consumes the creation script before invoking `psql`. Every later invocation in the same Pod uses guarded SQL. The guard fails before role or password changes when the database is absent. 3. `database-receipt` records `ready` only after the SQL init container succeeds. An interrupted receipt is retryable against the existing database. On fresh delivery, the `pharos-infra` Application is wave -1 and GitLab is wave 3. The root's Application health check waits for the child's current operation to finish. An incomplete database Job therefore keeps fresh GitLab writers from starting. Existing GitLab writers require explicit fencing during recovery; changing an Application's desired revision does not stop running workloads. PVC existence alone says nothing about repository contents. Claims may appear after a valid fresh record and before database initialisation. If the record and every credential copy are absent but claims already exist, bootstrap stops before generating credentials. Pre-provisioned empty claims require the same inspection as retained claims in that ambiguous situation. Managed fresh foundations install Rook with its CRDs and object store before Argo. Argo bootstrap creates the `gitlab` namespace before application delivery. Existing-storage and adopted sites must supply that Ceph provider already. The ObjectBucketClaim API is required by the GitLab base composition. A missing API, namespace, access grant or incomplete inventory stops initialisation; repair the foundation prerequisite rather than treating an unreadable inventory as empty. This safeguard covers the supported bootstrap and cutover paths. It does not detect an older restored database that still exists, missing individual repositories, a storage swap, or detached Ceph assets after their Kubernetes claims and every credential record were deleted. It does not reconcile the reported orphaned repositories; the offline comparison below only reports them. ## Read-only evidence collection Use the environment's verified administrator access and a private evidence directory. The `ntua-staging` account alone has no administrator kubeconfig. Do not copy full Secrets into a ticket or command output. Start with the [cutover capture](icarus-cloud-cutover.md#refresh-and-review). Record the source revision, GitLab version and edition, Helm release, Kubernetes cluster UID, PostgreSQL claim/PV UID and Gitaly claim/PV UID for every storage shard. Record the Ceph volume handles, bucket identities, backup timestamps and encryption-secret identities. A new claim with the same name is a different resource. These commands read metadata and the non-secret initialisation state: ```sh kubectl -n pharos-infra get job database-setup kubectl -n pharos-infra get pods -l app=database-setup kubectl -n pharos-infra get pvc -o wide kubectl -n gitlab get pvc,objectbucketclaims -o wide kubectl -n gitlab get deployments,statefulsets -o wide kubectl -n pharos-infra get secret infrastructure-generated \ -o jsonpath='{.data.gitlab_database_initialisation}' | base64 --decode ``` Inspect the exact Job Pod's init-container status and logs to distinguish API claim failure, an absent database, SQL failure and receipt failure. Do not assume `attempted` proves SQL ran. A missing field can be a legacy installation. Once writers are fenced, export the database's complete project mapping from the verified GitLab toolbox. Keep this output private because project paths may be sensitive: ```sh umask 077 kubectl -n gitlab exec deploy/gitlab-toolbox -- gitlab-rails runner \ 'Project.find_each(batch_size: 100) { |p| puts({id: p.id, path: p.full_path, storage: p.repository_storage, disk_path: p.disk_path}.to_json) }' \ > projects.jsonl ``` Discover every Gitaly storage root from the deployed configuration and mounts. For each root, collect a read-only listing of bare repository directories from the preserved copy and compare it offline with the database mapping, as described in the next section. Project wikis, design repositories, snippets, group wikis and object pools are not automatically orphaned projects. For an unmatched hashed project path, use the upstream [storage-path lookup](https://docs.gitlab.com/administration/repository_storage_paths/) and read its Git config metadata from a preserved copy. For example, `git config --file /path/to/repository.git/config --get gitlab.fullpath` reads a recorded project path where present. Compare it with backup metadata, refs and the expected owner. Never infer ownership solely from a numeric ID or a matching hash: a recreated database can reuse the same ID for a different project. ## Offline inventory comparison `scripts/gitlab_persistence_inventory.py` compares two JSON files you captured. It uses only the Python standard library, reads only the two named files, contacts no cluster, database, Git repository or Gitaly storage, and writes only to stdout. Run it with private output, because the report contains project paths: ```sh umask 077 python3 scripts/gitlab_persistence_inventory.py \ --database database.json --gitaly gitaly.json > inventory-report.json ``` Exit 0 means the declared inventories were compared and no unresolved evidence remains. Exit 1 means findings need investigation. Exit 2 means the input was invalid or incomplete; stdout is empty and stderr names only the failing field location, never input values. Each file is limited to 32 MiB and 200,000 entries. Rejected input includes malformed JSON, duplicate JSON keys, unknown fields, duplicate project IDs or `(storage, disk_path)` keys, undeclared storages, absolute or traversing paths, differing `target`/`capture_id` and differing storage sets. **Database file** (`pharos.gitlab-persistence.database.v1`): ```json {"format": "pharos.gitlab-persistence.database.v1", "target": "ntua-staging", "capture_id": "2026-11-09-restore-set-1", "storages": {"default": {"complete": true, "entries": 1}}, "projects": [{"id": 3, "path": "group/project", "storage": "default", "disk_path": "@hashed/4e/07/4e07...5b"}]} ``` `projects` are the exact objects the Rails command above writes to `projects.jsonl`; `id` is an opaque positive integer and `path` is `full_path`. Build the file from them without retyping rows. Set `storages` to the reviewed, complete list of every storage name in the deployed configuration, not the names found in the projects, so a shard without projects keeps `entries: 0`. Choose one `target` and `capture_id` and reuse them for both files. The counts are calculated; `complete: true` is your assertion that the capture is complete: ```sh jq -s --arg target ntua-staging --arg capture_id 2026-11-09-restore-set-1 \ --argjson storages '["default"]' ' . as $projects | {format: "pharos.gitlab-persistence.database.v1", target: $target, capture_id: $capture_id, storages: ($storages | map({key: ., value: {complete: true, entries: (. as $s | $projects | map(select(.storage == $s)) | length)}}) | from_entries), projects: $projects}' projects.jsonl > database.json ``` A project in a storage that is not listed makes the comparison exit 2. **Gitaly file** (`pharos.gitlab-persistence.gitaly.v1`) has the same header and `repositories` entries of `storage`, `disk_path`, `kind` and optional `fullpath`. `kind` is required. Collect it only from a read-only mount of the preserved copy of one storage root, never from live pods. Gitaly names every bare repository directory with a trailing `.git`. Remove that suffix, and only that one, to get `disk_path`, which is relative to the storage root and so matches the database: | Directory under the storage root | `disk_path` | `kind` | | --- | --- | --- | | `@hashed/ab/cd/.git` | `@hashed/ab/cd/` | `project` | | `@hashed/ab/cd/.wiki.git` | `@hashed/ab/cd/.wiki` | `project-wiki` | | `@hashed/ab/cd/.design.git` | `@hashed/ab/cd/.design` | `design` | | `@snippets/ab/cd/.git` | `@snippets/ab/cd/` | `snippet` | | `@groups/ab/cd/.wiki.git` | `@groups/ab/cd/.wiki` | `group-wiki` | | `@pools/ab/cd/.git` | `@pools/ab/cd/` | `object-pool` | `` is 64 lowercase hexadecimal characters. Any other path, including a legacy named-storage `group/project.git`, has no kind the command may guess: the capture stops and a reviewer decides. Read a single repository's stored path with `git config --file "$root/$disk_path.git/config" --get gitlab.fullpath`, but only after confirming that `config` is a readable regular file (not a symlink): git also exits 1 for a missing file. With the file confirmed, exit 1 means the key is absent and `fullpath` is omitted (unverified); any other exit status is a failure. For each storage name `$storage` and its preserved root `$root`, this lists, classifies and reads `fullpath`, without typing rows. It fails on an unclassifiable path (naming only the input line) or on a missing, unreadable, non-regular or malformed configuration (a generic message; git's stderr is suppressed, so no path or private value is echoed). Then discard the output file and review the listing: ```sh set -o pipefail # bash umask 077 storage=default; root=/mnt/preserved-gitaly/default ( cd "$root" && find . -type d -name '*.git' -prune -print ) \ | sed -e 's|^\./||' -e 's|\.git$||' | LC_ALL=C sort \ | while IFS= read -r path; do config="$root/$path.git/config" if [ -L "$config" ] || [ ! -f "$config" ] || [ ! -r "$config" ]; then echo 'capture failed: a repository configuration is not a readable regular file' >&2 exit 1 fi full=$(git config --file "$config" --get gitlab.fullpath 2>/dev/null) \ || [ $? -eq 1 ] || { echo 'capture failed: a repository configuration is unusable' >&2; exit 1; } printf '%s\t%s\n' "$path" "$full" done \ | jq -R -c --arg storage "$storage" ' def h: "[0-9a-f]{2}/[0-9a-f]{2}/[0-9a-f]{64}"; split("\t") as $f | if ($f | length) != 2 then error("unexpected line") else . end | $f[0] as $path | $f[1] as $full | ($path | if test("^@hashed/" + h + "$") then "project" elif test("^@hashed/" + h + "\\.wiki$") then "project-wiki" elif test("^@hashed/" + h + "\\.design$") then "design" elif test("^@snippets/" + h + "$") then "snippet" elif test("^@groups/" + h + "\\.wiki$") then "group-wiki" elif test("^@pools/" + h + "$") then "object-pool" else error("unclassified repository path") end) as $kind | {storage: $storage, disk_path: $path, kind: $kind} + (if $full == "" then {} else {fullpath: $full} end)' \ > "gitaly-$storage.jsonl" ``` Repeat for every storage (an empty storage gives an empty file), then build the Gitaly file with the same `target`, `capture_id` and `storages` list as the database file. As above, `entries` are calculated and `complete: true` is yours: ```sh jq -s --arg target ntua-staging --arg capture_id 2026-11-09-restore-set-1 \ --argjson storages '["default"]' ' . as $repositories | {format: "pharos.gitlab-persistence.gitaly.v1", target: $target, capture_id: $capture_id, storages: ($storages | map({key: ., value: {complete: true, entries: (. as $s | $repositories | map(select(.storage == $s)) | length)}}) | from_entries), repositories: $repositories}' gitaly-default.jsonl > gitaly.json ``` List every `gitaly-.jsonl` after the program. An entry looks like `{"storage": "default", "disk_path": "@hashed/ab/cd/", "kind": "project", "fullpath": "group/project"}`. The command only compares; it never decides ownership, and no finding authorises deleting, moving or repairing a repository. **Completeness is asserted, not proven.** `complete: true` and `entries` state what the collector believes it captured for each full storage name. The command checks only that both files declare the same non-empty storage set, that every entry lies in a declared storage and that the counts match. Capture from a fenced set, list every Gitaly storage and keep each file's checksum with the evidence. Matching uses only `(storage, disk_path)`. The report counts and lists: - `database_mapping_without_project_repository`: a mapping with no repository at its path. - `database_mapping_matches_non_project_repository`: a mapping whose path holds an auxiliary kind. - `project_repository_without_database_mapping`: a `project` repository that no mapping accounts for. Investigate it; do not call it owned or deletable. - `stored_fullpath_differs_investigate`: a matched path whose stored `gitlab.fullpath` differs from the database path. A recreated database can reuse an ID and hash for another project, but a rename can leave a stale value, so this proves neither. - `stored_fullpath_unverified`: a matched path with no stored `fullpath`; never reported as consistent. - `no_project_mappings_compared`: both captures declare no project entries. Entries carry references such as `database.projects[3]` into the input files. Auxiliary repositories are counted by kind only. Numeric IDs and hash values never select an owner. The result does not qualify a restore and says nothing about encryption secrets, object data or Source Control bindings. Preserve evidence of unmatched and missing repositories, conflicting paths, encryption-key availability, object-data coverage and Source Control bindings. Record relevant deployment/Argo operation times and PostgreSQL restore or initialisation logs if retained. This is the evidence needed to identify the historical action; resource age and the sequence workaround alone are insufficient. ## Operator recovery Keep external writes, GitLab web/SSH/API writes, Sidekiq and competing deployment writers fenced under a reviewed maintenance procedure. Take and verify backups or snapshots of the current complete recovery set before changing it. Preserve shared PostgreSQL consumers and their credentials. Choose a recovery based on verified evidence: - **A matching complete backup exists:** rehearse restoration into isolated storage at the exact GitLab version and edition, with the matching encryption secrets and object data. Follow the [GitLab Helm restore procedure](https://docs.gitlab.com/charts/backup-restore/restore/) and [restore prerequisites](https://docs.gitlab.com/administration/backup_restore/restore_gitlab/). Validate database-to-repository ownership before planning the live cutover. Restoring only the database onto the retained Gitaly volume repeats the unsafe condition. - **Repositories survive without matching database records:** preserve copies and recover their refs and object data into an isolated GitLab instance through a reviewed import procedure. Git repositories alone cannot reconstruct users, permissions, issues or other database-only information. Reconcile canonical Pharos bindings explicitly before resuming consumers; do not invent rows for orphaned numeric IDs or overwrite existing hashed paths. - **A first attempt is uncertain and the database is absent:** establish that no GitLab writer ever ran, no repositories or object data exist, and the same reviewed storage and credentials remain. If any fact is uncertain, preserve `attempted` and use the recovery paths above. Only after proving an unused fresh installation may an operator compare the Secret UID, resourceVersion and encoded `attempted` field and patch that field alone back to `new`. Retry the reviewed Argo operation; its replacement Job claims and consumes a new attempt. This is an exceptional recovery action, never an automatic retry. - **All credential records are lost while claims survive:** restore the matching credentials and encryption secrets from a verified backup before retrying. Preserve claims while establishing their contents. Removing claims or fabricating a fresh record to bypass the refusal is not a recovery procedure. Do not delete orphaned repositories, advance/reset sequences, drop the shared database, rotate credentials, or restore live data as part of diagnosis. The sequence bump is insufficient evidence to close the incident. Before reopening writes, repeat the inventory and ownership comparison, check restored encrypted data and LFS/uploads, and verify clone/read access for retained repositories. In the approved maintenance window, qualify a new project through the platform's normal creation path and verify its GitLab ID, owner and repository contents. Retain the exact tested source and installed revision with the evidence. ## Verification Run the initialisation regressions and cutover fixtures: ```sh uv run --project devops --frozen python -m unittest discover \ -s dx/common/tests -p 'test_gitlab_database_initialisation.py' uv run --project devops --frozen python -m unittest discover \ -s scripts -p 'test_distributed_cutover.py' python3 -m unittest discover -s scripts -p 'test_gitlab_persistence_inventory.py' ``` On the configured Linux runner, in an isolated checkout of the source under review, run: ```sh uv run --project devops --frozen python scripts/check_gitlab_database_initialisation.py ``` The fixture starts private PostgreSQL containers and a network, exercises the deployed SQL and grant-consumption script, and removes its resources. It covers fresh creation, interruption before SQL, missing completion acknowledgement, same-Pod and replacement-Pod retries, data preservation, legacy records and refusal before credential changes. It reads no kubeconfig and performs no live GitLab or staging recovery. Follow [TESTING.md](../../TESTING.md) for full source verification; fixture success does not qualify a live restore.