# Backup GitLab Repositories

Themelio stores all user-facing artifacts ([models](../../terms/model),
[datasets](../../terms/dataset), [spaces](../../terms/space), repos, recipes,
upskillings) as Git repositories on a GitLab instance, with binary content
managed by [Git LFS](../../terms/lfs). Backing up the Git side of a Themelio
deployment therefore reduces to mirroring every project on the source GitLab
instance to a target GitLab instance. This page documents that procedure;
GitLab-level metadata (issues, MRs, wiki, registries) and Themelio's
non-GitLab state (Keycloak, Redis/Valkey, etc.) are out of scope here and
need separate runbooks.

The repository ships a script that does exactly that:

```
scripts/backup_gitlab_repos.sh
```

It enumerates projects from the source instance, recreates the same group
hierarchy under a configurable root group on the target, and pushes every
ref, tag, and LFS blob across.

## What gets transferred

Per project, the script transfers:

- **Every ref** — all branches and tags (`git push --mirror` semantics).
- **All Git history** — full commit graph, not a shallow clone.
- **All Git LFS objects** referenced anywhere in the repo (`git lfs fetch
  --all` followed by `git lfs push --all`).
- **Project metadata** required to recreate the project on the target:
  `name`, `path`, `description`, and `default_branch`.

What is **not** transferred:

- Issues, merge requests, milestones, labels, wiki, snippets, CI/CD
  variables, deploy keys, webhooks, runners, or pipeline history. The
  script is a Git-level mirror; if you also need GitLab-level metadata,
  use GitLab's project export/import API on top of this.
- Project members and permissions. Target projects are created as
  `private` under the target root group; re-grant access manually.
- Container or package registries.

## How the script operates

At a high level, each invocation does the following:

1. **Validate inputs and dependencies.** Required CLI tools (`git`,
   `git-lfs`, `curl`, `jq`, `awk`, `mktemp`) are checked up front.
   Whitespace is trimmed from all token/URL/group inputs to avoid
   surprises from copy-paste or `read`-into-env-var workflows.
2. **Validate the source token** by calling `GET /api/v4/user` against
   the source GitLab. The same is done for the target token unless the
   run is a dry-run with no target args (see below).
3. **Resolve the target root group.** The script does **not** create the
   root group; it must already exist on the target. The script *does*
   create any missing intermediate subgroups beneath it.
4. **Resolve the source repository list** from `--source-repos`. This
   accepts:
   - `*` (or empty) — every project the source token can access.
   - Comma-separated entries, or a `@/path/to/list.txt` file with one
     entry per line (`#` comments allowed).
   - Each entry is either a **full path** (contains `/`, e.g.
     `models/foo/bar`, matched exactly) or a **keyword** (no `/`, e.g.
     `surya`, matched as a case-insensitive substring against every
     accessible project's `path_with_namespace`).
5. **Apply exclusions** from `--exclude-repos` using the same full-path
   vs. keyword semantics.
6. **Sort and de-duplicate** the resolved list (`LC_ALL=C sort -u`) so
   migration order is deterministic.
7. **For each project**, in sorted order:
   1. Look up source metadata; skip if archived (unless
      `--include-archived`); skip if the target already has commits and
      `--skip-existing` is set.
   2. Compute the target full path:
      `<target-root-group>/<source-path-with-namespace>`.
   3. Ensure all intermediate target subgroups exist; create the target
      project if missing (`private`, `lfs_enabled: true`, preserving
      name/description/default_branch).
   4. **Up-to-date short-circuit.** Run `git ls-remote --heads --tags`
      against both source and target. If every ref on both sides has
      the exact same SHA, the target is already a perfect mirror of
      the source — record the project as **up-to-date** and skip the
      remaining clone/push steps entirely. Otherwise continue.
   5. `git clone --mirror` the source.
   6. `git lfs fetch --all origin` to populate LFS blobs locally.
   7. `git lfs push --all target` — **uploads LFS objects before
      refs**. GitLab's pre-receive hook rejects refs whose LFS blobs
      are not already present on the target's LFS server, so this
      ordering is mandatory.
   8. `git push --mirror target` to push every branch and tag.
8. **Print a summary** of created groups, created/reused projects,
   mirrored projects, up-to-date projects, skipped projects, and failed
   projects. Exits non-zero if any project failed.

The script is idempotent: re-running reuses existing target groups and
projects. The up-to-date check (step 7.4) avoids any clone/push work
for repositories whose refs already match the source — only
repositories whose source has new or rewritten refs (or whose target
is missing some refs) actually pay the clone+LFS-fetch+push cost. Even
when transfer does happen, `git push --mirror` and `git lfs push --all`
are themselves incremental: Git's wire protocol negotiates the missing
delta and the LFS server only accepts blobs it does not already store.

## Up-to-date short-circuit and re-run semantics

Before each project is mirrored, the script compares heads and tags on
both sides via `git ls-remote --heads --tags`:

- **All refs match** → the project is recorded as *up-to-date* and
  skipped without cloning. This makes scheduled re-runs cheap: only
  repositories with new commits, new tags, deleted refs, or rewritten
  history actually pay the clone/LFS-fetch/push cost.
- **Refs differ, target has no refs, or target project was just created**
  → fall through to the normal clone + LFS push + ref push flow.
- **Source `ls-remote` fails** (network/auth error) → also fall through;
  the actual clone will surface the same error in a more useful place.

Important caveats:

- The check compares **heads and tags**, not arbitrary refs (e.g.
  `refs/keep-around/...`). For backup purposes this matches every ref
  that `git push --mirror` would also touch.
- Once the script *does* decide to push, `git push --mirror` is
  force-push semantics: it **rewrites** target refs to match the source
  and **deletes** target refs that no longer exist on the source.
  Treat the target as a read-only backup destination — do not commit
  to it directly, or your changes will be silently overwritten on the
  next run.
- The up-to-date check itself is read-only and uses the source/target
  tokens you already provide. It runs whether `--dry-run` is set or
  not; the only difference in dry-run is that the result is reported
  in the plan instead of acted on.

## Required tools

Install on the host that will run the backup:

- `bash` 4+ (the script uses arrays and modern parameter expansion)
- `git` and `git-lfs` (run `git lfs install` once after installing)
- `curl`
- `jq`
- `awk`, `mktemp` (typically pre-installed)

## Required tokens

- **Source token** — a GitLab personal access token (or admin
  impersonation token) on the source instance with at least the
  `read_api` scope plus read access to every project you want to mirror.
  An admin token mirrors everything; a regular user token only sees
  projects that user is a member of (or that are public/internal).
- **Target token** — a personal access token on the target instance
  with the `api` scope and at least `Maintainer` role on the target
  root group (so the script can create subgroups and projects beneath
  it).

## Arguments

Every CLI flag has an environment variable equivalent. CLI flags take
precedence over environment variables when both are set.

### Required

| Flag                  | Environment variable        | Description |
|-----------------------|-----------------------------|-------------|
| `--source-url`        | `GITLAB_SOURCE_URL`         | Base URL of the source GitLab instance, e.g. `https://gitlab.grnet.gr`. |
| `--source-token`      | `GITLAB_SOURCE_TOKEN`       | Source GitLab API token (read access). |
| `--target-url`        | `GITLAB_TARGET_URL`         | Base URL of the target GitLab instance. |
| `--target-token`      | `GITLAB_TARGET_TOKEN`       | Target GitLab API token (write access). |
| `--target-root-group` | `GITLAB_TARGET_ROOT_GROUP`  | Full path of an **already-existing** group on the target, beneath which the source hierarchy will be recreated, e.g. `themelio/dev_gitlab_backup`. |

In `--dry-run` mode the target arguments become **optional** — see below.

### Optional

| Flag                  | Environment variable    | Default | Description |
|-----------------------|-------------------------|---------|-------------|
| `--source-repos`      | `GITLAB_SOURCE_REPOS`   | `*`     | Which projects to mirror (see [selection syntax](#selection-syntax)). |
| `--exclude-repos`     | `GITLAB_EXCLUDE_REPOS`  | (none)  | Projects to exclude from the resolved list. Same syntax as `--source-repos`. |
| `--include-archived`  | —                       | off     | Mirror archived projects too. By default they are skipped. |
| `--skip-existing`     | —                       | off     | Skip projects whose target already contains commits. Useful when re-running after a partial backup and you do not want `git push --mirror` to overwrite refs that may have diverged on the target. |
| `--dry-run`           | —                       | off     | Resolve the plan and print what *would* be mirrored, without creating groups/projects or running any git operations. Target arguments become optional in this mode. |
| `-h`, `--help`        | —                       | —       | Print usage and exit. |

### `DEBUG=1` environment variable

Set `DEBUG=1` in the environment to enable extra debug output (group
lookup responses, project create payloads, etc.). Useful when
diagnosing GitLab API issues.

## Selection syntax

The `--source-repos` and `--exclude-repos` flags share the same syntax.
Each accepts one of:

- **`*`** — every accessible project.
- **A comma-separated list** of entries: `'a,b,c'`.
- **A file reference** prefixed with `@`: `@/path/to/list.txt`. The
  file is read line-by-line; blank lines and lines starting with `#`
  are ignored.

Each individual **entry** is interpreted as either:

- **A full path** if it contains a `/` — matched **exactly** against a
  project's `path_with_namespace`.
  Example: `models/foo/bar`, `themelio/demo-data/repos/pdf-tools`.
- **A keyword** if it does not contain a `/` — matched as a
  **case-insensitive substring** against every accessible project's
  `path_with_namespace`.
  Example: `surya` matches `models/dkats/surya-cache`,
  `repos/dkats/surya`, `spaces/dkats/dkats-surya-pdf-app`, etc.

Mixing is fine: `--source-repos 'surya,dkats,themelio/special-repo'`.

When applied to `--exclude-repos`, the same rules govern matching: a
keyword `kathan` drops every resolved project whose path contains
`kathan`, and a full path drops only its exact match.

## Examples

### Dry-run: list everything that would be mirrored

```bash
./scripts/backup_gitlab_repos.sh --dry-run \
  --source-url https://gitlab.grnet.gr \
  --source-token "$GITLAB_SOURCE_TOKEN" \
  --target-url https://backup.gitlab.example \
  --target-token "$GITLAB_TARGET_TOKEN" \
  --target-root-group themelio/dev_gitlab_backup
```

The output groups projects by their top-level prefix
(`datasets/...`, `models/...`, `repos/...`, `spaces/...`) and labels
each with its target path and one of:

- `target would be created` — no project exists at the target path yet.
- `target exists, would mirror updates` — the target project exists but
  its refs differ from the source.
- A separate `Up-to-date (would skip; refs already match source)`
  section lists projects whose target is already an exact mirror of
  the source — re-running the script without `--dry-run` performs no
  work for these.

### Dry-run without target args (preview source list only)

```bash
export GITLAB_SOURCE_TOKEN=glpat-xxxx
./scripts/backup_gitlab_repos.sh --dry-run \
  --source-url https://gitlab.grnet.gr \
  --source-repos 'dkats,louridas'
```

This validates the source token and resolves the source list (with
keyword expansion) without touching any target instance — useful to
iterate on `--source-repos` / `--exclude-repos` before running the
actual transfer.

### Mirror two users' worth of projects, excluding one space

```bash
export GITLAB_SOURCE_TOKEN=glpat-source
export GITLAB_TARGET_TOKEN=glpat-target

./scripts/backup_gitlab_repos.sh \
  --source-url https://gitlab.grnet.gr \
  --source-repos 'dkats,louridas' \
  --exclude-repos 'spaces/kathan/dkats-surya-pdf-app' \
  --target-url https://backup.gitlab.example \
  --target-root-group themelio/dev_gitlab_backup
```

Resolves all projects whose path contains `dkats` or `louridas` (case
insensitive), drops the explicitly listed space, then mirrors the rest
under `themelio/dev_gitlab_backup/<source-path>`.

### Full backup of every accessible project

```bash
./scripts/backup_gitlab_repos.sh \
  --source-url https://gitlab.grnet.gr \
  --source-token "$GITLAB_SOURCE_TOKEN" \
  --target-url https://backup.gitlab.example \
  --target-token "$GITLAB_TARGET_TOKEN" \
  --target-root-group themelio/dev_gitlab_backup
```

`--source-repos` defaults to `*`, so every project the source token
can see (excluding archived ones) is mirrored.

### Backup from a curated list, skipping repos that already migrated

```bash
cat > /tmp/repos.txt <<'EOF'
# Models
models/dkats/surya-cache
models/louridas/surya-cache
# Datasets
datasets/dkats/pdf-demo-input
# Keyword: every space named pdf-app under any user
pdf-app
EOF

./scripts/backup_gitlab_repos.sh --skip-existing \
  --source-url https://gitlab.grnet.gr \
  --source-token "$GITLAB_SOURCE_TOKEN" \
  --source-repos @/tmp/repos.txt \
  --target-url https://backup.gitlab.example \
  --target-token "$GITLAB_TARGET_TOKEN" \
  --target-root-group themelio/dev_gitlab_backup
```

`--skip-existing` short-circuits projects whose target already has
commits, which is useful when iteratively expanding the backup or
re-running after a partial failure.

### Re-running a failed or scheduled backup

The script is idempotent. After fixing the underlying issue (e.g. LFS
disabled on a target group, missing permissions, expired token), simply
re-run the same command. Existing groups/projects are reused, the
up-to-date check skips repositories that did not change since the
previous successful run, and Git push semantics ensure only diverged
refs and missing LFS objects are re-transferred for the rest. This
makes the script suitable for nightly cron jobs against a stable
target instance.

## Verifying a backup

After the script completes, confirm a sample of repositories on the
target instance:

```bash
# List backed-up projects under the target root group
curl -sS -H "PRIVATE-TOKEN: $GITLAB_TARGET_TOKEN" \
  "$GITLAB_TARGET_URL/api/v4/groups/$(jq -nr --arg p "$GITLAB_TARGET_ROOT_GROUP" '$p|@uri')/projects?include_subgroups=true&per_page=100" \
  | jq -r '.[].path_with_namespace' \
  | LC_ALL=C sort

# Spot-check refs and LFS objects on a single repo
git clone "$GITLAB_TARGET_URL/<target-root-group>/<some-project>.git" /tmp/check
git -C /tmp/check log --oneline -5
git -C /tmp/check lfs ls-files | head
```

## Troubleshooting

### `Failed to authenticate source token: invalid_token`

GitLab is rejecting the token value the script sent. Common causes:

- **Trailing whitespace/newline** in the env var (the script trims
  this since the most recent revision, but verify with
  `printenv GITLAB_SOURCE_TOKEN | wc -c` — the byte count should be
  the token length plus 1 for the trailing newline added by `printenv`).
- **Token expired or revoked** — generate a new one in
  *User settings → Access tokens* with `read_api` scope.
- **Wrong instance** — tokens are per-instance.

Verify directly:

```bash
curl -sS -H "PRIVATE-TOKEN: $GITLAB_SOURCE_TOKEN" \
  "$GITLAB_SOURCE_URL/api/v4/user"
```

### `404 Group Not Found` when creating an intermediate subgroup

The target token does not have permission to create subgroups beneath
the target root group, or the parent group reference is stale. Confirm
the token has at least `Maintainer` on the root group:

```bash
curl -sS -H "PRIVATE-TOKEN: $GITLAB_TARGET_TOKEN" \
  "$GITLAB_TARGET_URL/api/v4/groups/$(jq -nr --arg p "$GITLAB_TARGET_ROOT_GROUP" '$p|@uri')/members/all" \
  | jq '.[] | select(.username == "<your-username>")'
```

### `LFS objects are missing. Ensure LFS is properly set up`

The target's pre-receive hook rejected the ref push because some LFS
blobs were not on its LFS server. Make sure:

- `git lfs version` works on the host running the script.
- LFS is enabled on the target project (it is by default; the script
  sets `lfs_enabled: true` at creation).
- LFS storage is not full or quota-capped on the target instance — ask
  the target's GitLab administrator if unsure.
- The source token has read access to LFS objects (admin or owner on
  the source project).

If these are all in order and the issue persists for a specific repo,
re-run with `DEBUG=1` to see the underlying `git lfs fetch` /
`git lfs push` errors.

### Some projects show as `target exists, would mirror` in the dry-run, but I want a clean import

Either delete the target projects manually, or run without
`--skip-existing` — `git push --mirror` will overwrite the target's
refs to match the source. Use this with care: `--mirror` will also
**delete** target refs that no longer exist on the source.