> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # Backup GitLab Repositories Themelio stores all user-facing artifacts ([models](/terms/model.md), [datasets](/terms/dataset.md), [spaces](/terms/space.md), repos, recipes, upskillings) as Git repositories on a GitLab instance, with binary content managed by [Git LFS](/terms/lfs.md). Backing up the Git side of a Themelio deployment therefore reduces to mirroring every project on the source GitLab instance to a target GitLab instance. This page documents that procedure; GitLab-level metadata (issues, MRs, wiki, registries) and Themelio's non-GitLab state (Keycloak, Redis/Valkey, etc.) are out of scope here and need separate runbooks. The repository ships a script that does exactly that: ``` scripts/backup_gitlab_repos.sh ``` It enumerates projects from the source instance, recreates the same group hierarchy under a configurable root group on the target, and pushes every ref, tag, and LFS blob across. ## What gets transferred Per project, the script transfers: - **Every ref** — all branches and tags (`git push --mirror` semantics). - **All Git history** — full commit graph, not a shallow clone. - **All Git LFS objects** referenced anywhere in the repo (`git lfs fetch --all` followed by `git lfs push --all`). - **Project metadata** required to recreate the project on the target: `name`, `path`, `description`, and `default_branch`. What is **not** transferred: - Issues, merge requests, milestones, labels, wiki, snippets, CI/CD variables, deploy keys, webhooks, runners, or pipeline history. The script is a Git-level mirror; if you also need GitLab-level metadata, use GitLab's project export/import API on top of this. - Project members and permissions. Target projects are created as `private` under the target root group; re-grant access manually. - Container or package registries. ## How the script operates At a high level, each invocation does the following: 1. **Validate inputs and dependencies.** Required CLI tools (`git`, `git-lfs`, `curl`, `jq`, `awk`, `mktemp`) are checked up front. Whitespace is trimmed from all token/URL/group inputs to avoid surprises from copy-paste or `read`-into-env-var workflows. 2. **Validate the source token** by calling `GET /api/v4/user` against the source GitLab. The same is done for the target token unless the run is a dry-run with no target args (see below). 3. **Resolve the target root group.** The script does **not** create the root group; it must already exist on the target. The script _does_ create any missing intermediate subgroups beneath it. 4. **Resolve the source repository list** from `--source-repos`. This accepts: - `*` (or empty) — every project the source token can access. - Comma-separated entries, or a `@/path/to/list.txt` file with one entry per line (`#` comments allowed). - Each entry is either a **full path** (contains `/`, e.g. `models/foo/bar`, matched exactly) or a **keyword** (no `/`, e.g. `surya`, matched as a case-insensitive substring against every accessible project's `path_with_namespace`). 5. **Apply exclusions** from `--exclude-repos` using the same full-path vs. keyword semantics. 6. **Sort and de-duplicate** the resolved list (`LC_ALL=C sort -u`) so migration order is deterministic. 7. **For each project**, in sorted order: 1. Look up source metadata; skip if archived (unless `--include-archived`); skip if the target already has commits and `--skip-existing` is set. 2. Compute the target full path: `/`. 3. Ensure all intermediate target subgroups exist; create the target project if missing (`private`, `lfs_enabled: true`, preserving name/description/default_branch). 4. **Up-to-date short-circuit.** Run `git ls-remote --heads --tags` against both source and target. If every ref on both sides has the exact same SHA, the target is already a perfect mirror of the source — record the project as **up-to-date** and skip the remaining clone/push steps entirely. Otherwise continue. 5. `git clone --mirror` the source. 6. `git lfs fetch --all origin` to populate LFS blobs locally. 7. `git lfs push --all target` — **uploads LFS objects before refs**. GitLab's pre-receive hook rejects refs whose LFS blobs are not already present on the target's LFS server, so this ordering is mandatory. 8. `git push --mirror target` to push every branch and tag. 8. **Print a summary** of created groups, created/reused projects, mirrored projects, up-to-date projects, skipped projects, and failed projects. Exits non-zero if any project failed. The script is idempotent: re-running reuses existing target groups and projects. The up-to-date check (step 7.4) avoids any clone/push work for repositories whose refs already match the source — only repositories whose source has new or rewritten refs (or whose target is missing some refs) actually pay the clone+LFS-fetch+push cost. Even when transfer does happen, `git push --mirror` and `git lfs push --all` are themselves incremental: Git's wire protocol negotiates the missing delta and the LFS server only accepts blobs it does not already store. ## Up-to-date short-circuit and re-run semantics Before each project is mirrored, the script compares heads and tags on both sides via `git ls-remote --heads --tags`: - **All refs match** → the project is recorded as _up-to-date_ and skipped without cloning. This makes scheduled re-runs cheap: only repositories with new commits, new tags, deleted refs, or rewritten history actually pay the clone/LFS-fetch/push cost. - **Refs differ, target has no refs, or target project was just created** → fall through to the normal clone + LFS push + ref push flow. - **Source `ls-remote` fails** (network/auth error) → also fall through; the actual clone will surface the same error in a more useful place. Important caveats: - The check compares **heads and tags**, not arbitrary refs (e.g. `refs/keep-around/...`). For backup purposes this matches every ref that `git push --mirror` would also touch. - Once the script _does_ decide to push, `git push --mirror` is force-push semantics: it **rewrites** target refs to match the source and **deletes** target refs that no longer exist on the source. Treat the target as a read-only backup destination — do not commit to it directly, or your changes will be silently overwritten on the next run. - The up-to-date check itself is read-only and uses the source/target tokens you already provide. It runs whether `--dry-run` is set or not; the only difference in dry-run is that the result is reported in the plan instead of acted on. ## Required tools Install on the host that will run the backup: - `bash` 4+ (the script uses arrays and modern parameter expansion) - `git` and `git-lfs` (run `git lfs install` once after installing) - `curl` - `jq` - `awk`, `mktemp` (typically pre-installed) ## Required tokens - **Source token** — a GitLab personal access token (or admin impersonation token) on the source instance with at least the `read_api` scope plus read access to every project you want to mirror. An admin token mirrors everything; a regular user token only sees projects that user is a member of (or that are public/internal). - **Target token** — a personal access token on the target instance with the `api` scope and at least `Maintainer` role on the target root group (so the script can create subgroups and projects beneath it). ## Arguments Every CLI flag has an environment variable equivalent. CLI flags take precedence over environment variables when both are set. ### Required | Flag | Environment variable | Description | | --------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `--source-url` | `GITLAB_SOURCE_URL` | Base URL of the source GitLab instance, e.g. `https://gitlab.grnet.gr`. | | `--source-token` | `GITLAB_SOURCE_TOKEN` | Source GitLab API token (read access). | | `--target-url` | `GITLAB_TARGET_URL` | Base URL of the target GitLab instance. | | `--target-token` | `GITLAB_TARGET_TOKEN` | Target GitLab API token (write access). | | `--target-root-group` | `GITLAB_TARGET_ROOT_GROUP` | Full path of an **already-existing** group on the target, beneath which the source hierarchy will be recreated, e.g. `themelio/dev_gitlab_backup`. | In `--dry-run` mode the target arguments become **optional** — see below. ### Optional | Flag | Environment variable | Default | Description | | -------------------- | ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--source-repos` | `GITLAB_SOURCE_REPOS` | `*` | Which projects to mirror (see [selection syntax](#selection-syntax)). | | `--exclude-repos` | `GITLAB_EXCLUDE_REPOS` | (none) | Projects to exclude from the resolved list. Same syntax as `--source-repos`. | | `--include-archived` | — | off | Mirror archived projects too. By default they are skipped. | | `--skip-existing` | — | off | Skip projects whose target already contains commits. Useful when re-running after a partial backup and you do not want `git push --mirror` to overwrite refs that may have diverged on the target. | | `--dry-run` | — | off | Resolve the plan and print what _would_ be mirrored, without creating groups/projects or running any git operations. Target arguments become optional in this mode. | | `-h`, `--help` | — | — | Print usage and exit. | ### `DEBUG=1` environment variable Set `DEBUG=1` in the environment to enable extra debug output (group lookup responses, project create payloads, etc.). Useful when diagnosing GitLab API issues. ## Selection syntax The `--source-repos` and `--exclude-repos` flags share the same syntax. Each accepts one of: - **`*`** — every accessible project. - **A comma-separated list** of entries: `'a,b,c'`. - **A file reference** prefixed with `@`: `@/path/to/list.txt`. The file is read line-by-line; blank lines and lines starting with `#` are ignored. Each individual **entry** is interpreted as either: - **A full path** if it contains a `/` — matched **exactly** against a project's `path_with_namespace`. Example: `models/foo/bar`, `themelio/demo-data/repos/pdf-tools`. - **A keyword** if it does not contain a `/` — matched as a **case-insensitive substring** against every accessible project's `path_with_namespace`. Example: `surya` matches `models/dkats/surya-cache`, `repos/dkats/surya`, `spaces/dkats/dkats-surya-pdf-app`, etc. Mixing is fine: `--source-repos 'surya,dkats,themelio/special-repo'`. When applied to `--exclude-repos`, the same rules govern matching: a keyword `kathan` drops every resolved project whose path contains `kathan`, and a full path drops only its exact match. ## Examples ### Dry-run: list everything that would be mirrored ```bash ./scripts/backup_gitlab_repos.sh --dry-run \ --source-url https://gitlab.grnet.gr \ --source-token "$GITLAB_SOURCE_TOKEN" \ --target-url https://backup.gitlab.example \ --target-token "$GITLAB_TARGET_TOKEN" \ --target-root-group themelio/dev_gitlab_backup ``` The output groups projects by their top-level prefix (`datasets/...`, `models/...`, `repos/...`, `spaces/...`) and labels each with its target path and one of: - `target would be created` — no project exists at the target path yet. - `target exists, would mirror updates` — the target project exists but its refs differ from the source. - A separate `Up-to-date (would skip; refs already match source)` section lists projects whose target is already an exact mirror of the source — re-running the script without `--dry-run` performs no work for these. ### Dry-run without target args (preview source list only) ```bash export GITLAB_SOURCE_TOKEN=glpat-xxxx ./scripts/backup_gitlab_repos.sh --dry-run \ --source-url https://gitlab.grnet.gr \ --source-repos 'dkats,louridas' ``` This validates the source token and resolves the source list (with keyword expansion) without touching any target instance — useful to iterate on `--source-repos` / `--exclude-repos` before running the actual transfer. ### Mirror two users' worth of projects, excluding one space ```bash export GITLAB_SOURCE_TOKEN=glpat-source export GITLAB_TARGET_TOKEN=glpat-target ./scripts/backup_gitlab_repos.sh \ --source-url https://gitlab.grnet.gr \ --source-repos 'dkats,louridas' \ --exclude-repos 'spaces/kathan/dkats-surya-pdf-app' \ --target-url https://backup.gitlab.example \ --target-root-group themelio/dev_gitlab_backup ``` Resolves all projects whose path contains `dkats` or `louridas` (case insensitive), drops the explicitly listed space, then mirrors the rest under `themelio/dev_gitlab_backup/`. ### Full backup of every accessible project ```bash ./scripts/backup_gitlab_repos.sh \ --source-url https://gitlab.grnet.gr \ --source-token "$GITLAB_SOURCE_TOKEN" \ --target-url https://backup.gitlab.example \ --target-token "$GITLAB_TARGET_TOKEN" \ --target-root-group themelio/dev_gitlab_backup ``` `--source-repos` defaults to `*`, so every project the source token can see (excluding archived ones) is mirrored. ### Backup from a curated list, skipping repos that already migrated ```bash cat > /tmp/repos.txt <<'EOF' # Models models/dkats/surya-cache models/louridas/surya-cache # Datasets datasets/dkats/pdf-demo-input # Keyword: every space named pdf-app under any user pdf-app EOF ./scripts/backup_gitlab_repos.sh --skip-existing \ --source-url https://gitlab.grnet.gr \ --source-token "$GITLAB_SOURCE_TOKEN" \ --source-repos @/tmp/repos.txt \ --target-url https://backup.gitlab.example \ --target-token "$GITLAB_TARGET_TOKEN" \ --target-root-group themelio/dev_gitlab_backup ``` `--skip-existing` short-circuits projects whose target already has commits, which is useful when iteratively expanding the backup or re-running after a partial failure. ### Re-running a failed or scheduled backup The script is idempotent. After fixing the underlying issue (e.g. LFS disabled on a target group, missing permissions, expired token), simply re-run the same command. Existing groups/projects are reused, the up-to-date check skips repositories that did not change since the previous successful run, and Git push semantics ensure only diverged refs and missing LFS objects are re-transferred for the rest. This makes the script suitable for nightly cron jobs against a stable target instance. ## Verifying a backup After the script completes, confirm a sample of repositories on the target instance: ```bash # List backed-up projects under the target root group curl -sS -H "PRIVATE-TOKEN: $GITLAB_TARGET_TOKEN" \ "$GITLAB_TARGET_URL/api/v4/groups/$(jq -nr --arg p "$GITLAB_TARGET_ROOT_GROUP" '$p|@uri')/projects?include_subgroups=true&per_page=100" \ | jq -r '.[].path_with_namespace' \ | LC_ALL=C sort # Spot-check refs and LFS objects on a single repo git clone "$GITLAB_TARGET_URL//.git" /tmp/check git -C /tmp/check log --oneline -5 git -C /tmp/check lfs ls-files | head ``` ## Troubleshooting ### `Failed to authenticate source token: invalid_token` GitLab is rejecting the token value the script sent. Common causes: - **Trailing whitespace/newline** in the env var (the script trims this since the most recent revision, but verify with `printenv GITLAB_SOURCE_TOKEN | wc -c` — the byte count should be the token length plus 1 for the trailing newline added by `printenv`). - **Token expired or revoked** — generate a new one in _User settings → Access tokens_ with `read_api` scope. - **Wrong instance** — tokens are per-instance. Verify directly: ```bash curl -sS -H "PRIVATE-TOKEN: $GITLAB_SOURCE_TOKEN" \ "$GITLAB_SOURCE_URL/api/v4/user" ``` ### `404 Group Not Found` when creating an intermediate subgroup The target token does not have permission to create subgroups beneath the target root group, or the parent group reference is stale. Confirm the token has at least `Maintainer` on the root group: ```bash curl -sS -H "PRIVATE-TOKEN: $GITLAB_TARGET_TOKEN" \ "$GITLAB_TARGET_URL/api/v4/groups/$(jq -nr --arg p "$GITLAB_TARGET_ROOT_GROUP" '$p|@uri')/members/all" \ | jq '.[] | select(.username == "")' ``` ### `LFS objects are missing. Ensure LFS is properly set up` The target's pre-receive hook rejected the ref push because some LFS blobs were not on its LFS server. Make sure: - `git lfs version` works on the host running the script. - LFS is enabled on the target project (it is by default; the script sets `lfs_enabled: true` at creation). - LFS storage is not full or quota-capped on the target instance — ask the target's GitLab administrator if unsure. - The source token has read access to LFS objects (admin or owner on the source project). If these are all in order and the issue persists for a specific repo, re-run with `DEBUG=1` to see the underlying `git lfs fetch` / `git lfs push` errors. ### Some projects show as `target exists, would mirror` in the dry-run, but I want a clean import Either delete the target projects manually, or run without `--skip-existing` — `git push --mirror` will overwrite the target's refs to match the source. Use this with care: `--mirror` will also **delete** target refs that no longer exist on the source.
Page source Source snapshot · Revision `f928834c8`.