Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#Backup GitLab Repositories

Themelio stores all user-facing artifacts (models, datasets, spaces, repos, recipes, upskillings) as Git repositories on a GitLab instance, with binary content managed by Git LFS. Backing up the Git side of a Themelio deployment therefore reduces to mirroring every project on the source GitLab instance to a target GitLab instance. This page documents that procedure; GitLab-level metadata (issues, MRs, wiki, registries) and Themelio's non-GitLab state (Keycloak, Redis/Valkey, etc.) are out of scope here and need separate runbooks.

The repository ships a script that does exactly that:

scripts/backup_gitlab_repos.sh

It enumerates projects from the source instance, recreates the same group hierarchy under a configurable root group on the target, and pushes every ref, tag, and LFS blob across.

#What gets transferred

Per project, the script transfers:

• Every ref — all branches and tags (git push --mirror semantics).
• All Git history — full commit graph, not a shallow clone.
• All Git LFS objects referenced anywhere in the repo (git lfs fetch --all followed by git lfs push --all).
• Project metadata required to recreate the project on the target: name, path, description, and default_branch.

What is not transferred:

• Issues, merge requests, milestones, labels, wiki, snippets, CI/CD variables, deploy keys, webhooks, runners, or pipeline history. The script is a Git-level mirror; if you also need GitLab-level metadata, use GitLab's project export/import API on top of this.
• Project members and permissions. Target projects are created as private under the target root group; re-grant access manually.
• Container or package registries.

#How the script operates

At a high level, each invocation does the following:

1. Validate inputs and dependencies. Required CLI tools (git, git-lfs, curl, jq, awk, mktemp) are checked up front. Whitespace is trimmed from all token/URL/group inputs to avoid surprises from copy-paste or read-into-env-var workflows.
2. Validate the source token by calling GET /api/v4/user against the source GitLab. The same is done for the target token unless the run is a dry-run with no target args (see below).
3. Resolve the target root group. The script does not create the root group; it must already exist on the target. The script does create any missing intermediate subgroups beneath it.
4. Resolve the source repository list from --source-repos. This accepts:
   • * (or empty) — every project the source token can access.
   • Comma-separated entries, or a @/path/to/list.txt file with one entry per line (# comments allowed).
   • Each entry is either a full path (contains /, e.g. models/foo/bar, matched exactly) or a keyword (no /, e.g. surya, matched as a case-insensitive substring against every accessible project's path_with_namespace).
5. Apply exclusions from --exclude-repos using the same full-path vs. keyword semantics.
6. Sort and de-duplicate the resolved list (LC_ALL=C sort -u) so migration order is deterministic.
7. For each project, in sorted order:
   1. Look up source metadata; skip if archived (unless --include-archived); skip if the target already has commits and --skip-existing is set.
   2. Compute the target full path: <target-root-group>/<source-path-with-namespace>.
   3. Ensure all intermediate target subgroups exist; create the target project if missing (private, lfs_enabled: true, preserving name/description/default_branch).
   4. Up-to-date short-circuit. Run git ls-remote --heads --tags against both source and target. If every ref on both sides has the exact same SHA, the target is already a perfect mirror of the source — record the project as up-to-date and skip the remaining clone/push steps entirely. Otherwise continue.
   5. git clone --mirror the source.
   6. git lfs fetch --all origin to populate LFS blobs locally.
   7. git lfs push --all target — uploads LFS objects before refs. GitLab's pre-receive hook rejects refs whose LFS blobs are not already present on the target's LFS server, so this ordering is mandatory.
   8. git push --mirror target to push every branch and tag.
8. Print a summary of created groups, created/reused projects, mirrored projects, up-to-date projects, skipped projects, and failed projects. Exits non-zero if any project failed.

The script is idempotent: re-running reuses existing target groups and projects. The up-to-date check (step 7.4) avoids any clone/push work for repositories whose refs already match the source — only repositories whose source has new or rewritten refs (or whose target is missing some refs) actually pay the clone+LFS-fetch+push cost. Even when transfer does happen, git push --mirror and git lfs push --all are themselves incremental: Git's wire protocol negotiates the missing delta and the LFS server only accepts blobs it does not already store.

#Up-to-date short-circuit and re-run semantics

Before each project is mirrored, the script compares heads and tags on both sides via git ls-remote --heads --tags:

• All refs match → the project is recorded as up-to-date and skipped without cloning. This makes scheduled re-runs cheap: only repositories with new commits, new tags, deleted refs, or rewritten history actually pay the clone/LFS-fetch/push cost.
• Refs differ, target has no refs, or target project was just created → fall through to the normal clone + LFS push + ref push flow.
• Source ls-remote fails (network/auth error) → also fall through; the actual clone will surface the same error in a more useful place.

Important caveats:

• The check compares heads and tags, not arbitrary refs (e.g. refs/keep-around/...). For backup purposes this matches every ref that git push --mirror would also touch.
• Once the script does decide to push, git push --mirror is force-push semantics: it rewrites target refs to match the source and deletes target refs that no longer exist on the source. Treat the target as a read-only backup destination — do not commit to it directly, or your changes will be silently overwritten on the next run.
• The up-to-date check itself is read-only and uses the source/target tokens you already provide. It runs whether --dry-run is set or not; the only difference in dry-run is that the result is reported in the plan instead of acted on.

#Required tools

Install on the host that will run the backup:

• bash 4+ (the script uses arrays and modern parameter expansion)
• git and git-lfs (run git lfs install once after installing)
• curl
• jq
• awk, mktemp (typically pre-installed)

#Required tokens

• Source token — a GitLab personal access token (or admin impersonation token) on the source instance with at least the read_api scope plus read access to every project you want to mirror. An admin token mirrors everything; a regular user token only sees projects that user is a member of (or that are public/internal).
• Target token — a personal access token on the target instance with the api scope and at least Maintainer role on the target root group (so the script can create subgroups and projects beneath it).

#Arguments

Every CLI flag has an environment variable equivalent. CLI flags take precedence over environment variables when both are set.

#Required

In --dry-run mode the target arguments become optional — see below.

#Optional

#DEBUG=1 environment variable

Set DEBUG=1 in the environment to enable extra debug output (group lookup responses, project create payloads, etc.). Useful when diagnosing GitLab API issues.

#Selection syntax

The --source-repos and --exclude-repos flags share the same syntax. Each accepts one of:

• * — every accessible project.
• A comma-separated list of entries: 'a,b,c'.
• A file reference prefixed with @: @/path/to/list.txt. The file is read line-by-line; blank lines and lines starting with # are ignored.

Each individual entry is interpreted as either:

• A full path if it contains a / — matched exactly against a project's path_with_namespace. Example: models/foo/bar, themelio/demo-data/repos/pdf-tools.
• A keyword if it does not contain a / — matched as a case-insensitive substring against every accessible project's path_with_namespace. Example: surya matches models/dkats/surya-cache, repos/dkats/surya, spaces/dkats/dkats-surya-pdf-app, etc.

Mixing is fine: --source-repos 'surya,dkats,themelio/special-repo'.

When applied to --exclude-repos, the same rules govern matching: a keyword kathan drops every resolved project whose path contains kathan, and a full path drops only its exact match.

#Examples

#Dry-run: list everything that would be mirrored

./scripts/backup_gitlab_repos.sh --dry-run \
  --source-url https://gitlab.grnet.gr \
  --source-token "$GITLAB_SOURCE_TOKEN" \
  --target-url https://backup.gitlab.example \
  --target-token "$GITLAB_TARGET_TOKEN" \
  --target-root-group themelio/dev_gitlab_backup

The output groups projects by their top-level prefix (datasets/..., models/..., repos/..., spaces/...) and labels each with its target path and one of:

• target would be created — no project exists at the target path yet.
• target exists, would mirror updates — the target project exists but its refs differ from the source.
• A separate Up-to-date (would skip; refs already match source) section lists projects whose target is already an exact mirror of the source — re-running the script without --dry-run performs no work for these.

#Dry-run without target args (preview source list only)

export GITLAB_SOURCE_TOKEN=glpat-xxxx
./scripts/backup_gitlab_repos.sh --dry-run \
  --source-url https://gitlab.grnet.gr \
  --source-repos 'dkats,louridas'

This validates the source token and resolves the source list (with keyword expansion) without touching any target instance — useful to iterate on --source-repos / --exclude-repos before running the actual transfer.

#Mirror two users' worth of projects, excluding one space

export GITLAB_SOURCE_TOKEN=glpat-source
export GITLAB_TARGET_TOKEN=glpat-target

./scripts/backup_gitlab_repos.sh \
  --source-url https://gitlab.grnet.gr \
  --source-repos 'dkats,louridas' \
  --exclude-repos 'spaces/kathan/dkats-surya-pdf-app' \
  --target-url https://backup.gitlab.example \
  --target-root-group themelio/dev_gitlab_backup

Resolves all projects whose path contains dkats or louridas (case insensitive), drops the explicitly listed space, then mirrors the rest under themelio/dev_gitlab_backup/<source-path>.

#Full backup of every accessible project

./scripts/backup_gitlab_repos.sh \
  --source-url https://gitlab.grnet.gr \
  --source-token "$GITLAB_SOURCE_TOKEN" \
  --target-url https://backup.gitlab.example \
  --target-token "$GITLAB_TARGET_TOKEN" \
  --target-root-group themelio/dev_gitlab_backup

--source-repos defaults to *, so every project the source token can see (excluding archived ones) is mirrored.

#Backup from a curated list, skipping repos that already migrated

cat > /tmp/repos.txt <<'EOF'
# Models
models/dkats/surya-cache
models/louridas/surya-cache
# Datasets
datasets/dkats/pdf-demo-input
# Keyword: every space named pdf-app under any user
pdf-app
EOF

./scripts/backup_gitlab_repos.sh --skip-existing \
  --source-url https://gitlab.grnet.gr \
  --source-token "$GITLAB_SOURCE_TOKEN" \
  --source-repos @/tmp/repos.txt \
  --target-url https://backup.gitlab.example \
  --target-token "$GITLAB_TARGET_TOKEN" \
  --target-root-group themelio/dev_gitlab_backup

--skip-existing short-circuits projects whose target already has commits, which is useful when iteratively expanding the backup or re-running after a partial failure.

#Re-running a failed or scheduled backup

The script is idempotent. After fixing the underlying issue (e.g. LFS disabled on a target group, missing permissions, expired token), simply re-run the same command. Existing groups/projects are reused, the up-to-date check skips repositories that did not change since the previous successful run, and Git push semantics ensure only diverged refs and missing LFS objects are re-transferred for the rest. This makes the script suitable for nightly cron jobs against a stable target instance.

#Verifying a backup

After the script completes, confirm a sample of repositories on the target instance:

# List backed-up projects under the target root group
curl -sS -H "PRIVATE-TOKEN: $GITLAB_TARGET_TOKEN" \
  "$GITLAB_TARGET_URL/api/v4/groups/$(jq -nr --arg p "$GITLAB_TARGET_ROOT_GROUP" '$p|@uri')/projects?include_subgroups=true&per_page=100" \
  | jq -r '.[].path_with_namespace' \
  | LC_ALL=C sort

# Spot-check refs and LFS objects on a single repo
git clone "$GITLAB_TARGET_URL/<target-root-group>/<some-project>.git" /tmp/check
git -C /tmp/check log --oneline -5
git -C /tmp/check lfs ls-files | head

#Troubleshooting

#Failed to authenticate source token: invalid_token

GitLab is rejecting the token value the script sent. Common causes:

• Trailing whitespace/newline in the env var (the script trims this since the most recent revision, but verify with printenv GITLAB_SOURCE_TOKEN | wc -c — the byte count should be the token length plus 1 for the trailing newline added by printenv).
• Token expired or revoked — generate a new one in User settings → Access tokens with read_api scope.
• Wrong instance — tokens are per-instance.

Verify directly:

curl -sS -H "PRIVATE-TOKEN: $GITLAB_SOURCE_TOKEN" \
  "$GITLAB_SOURCE_URL/api/v4/user"

#404 Group Not Found when creating an intermediate subgroup

The target token does not have permission to create subgroups beneath the target root group, or the parent group reference is stale. Confirm the token has at least Maintainer on the root group:

curl -sS -H "PRIVATE-TOKEN: $GITLAB_TARGET_TOKEN" \
  "$GITLAB_TARGET_URL/api/v4/groups/$(jq -nr --arg p "$GITLAB_TARGET_ROOT_GROUP" '$p|@uri')/members/all" \
  | jq '.[] | select(.username == "<your-username>")'

#LFS objects are missing. Ensure LFS is properly set up

The target's pre-receive hook rejected the ref push because some LFS blobs were not on its LFS server. Make sure:

• git lfs version works on the host running the script.
• LFS is enabled on the target project (it is by default; the script sets lfs_enabled: true at creation).
• LFS storage is not full or quota-capped on the target instance — ask the target's GitLab administrator if unsure.
• The source token has read access to LFS objects (admin or owner on the source project).

If these are all in order and the issue persists for a specific repo, re-run with DEBUG=1 to see the underlying git lfs fetch / git lfs push errors.

#Some projects show as target exists, would mirror in the dry-run, but I want a clean import

Either delete the target projects manually, or run without --skip-existing — git push --mirror will overwrite the target's refs to match the source. Use this with care: --mirror will also delete target refs that no longer exist on the source.