Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#Local Stack: Known Issues and Lessons Learned

Issues, workarounds, and lessons discovered while validating:

cd devops/ansible
make local-up

This document covers both supported local host modes:

• native Debian/Linux hosts using a loop-backed image
• Windows/WSL2 hosts using a Windows VHD passed through with wsl --mount

Canonical automation now lives in code, not only in notes:

• devops/ansible/Makefile is the entrypoint for local bring-up
• make local-up is the supported path for Rook-Ceph local runs (Ceph RGW for object storage, Minikube standard for PVCs)
• WSL VHD settings live in devops/ansible/inventory/local/group_vars/all.yml

#Common Issues (all hosts)

#C1. Inconsistent Ceph state blocks local reconciliation

Problem: If the local Rook/Ceph namespace, CephCluster, OSD deployments, and backing image no longer describe the same cluster, make local-up cannot safely determine whether the existing object-storage data should be preserved.

Expected behavior: Local reconciliation is non-destructive. It never removes finalizers, wipes BlueStore metadata, or resets the OSD image automatically. Helm or Rook may report the inconsistent resources that prevent reconciliation.

Resolution: Use the supported teardown command to explicitly destroy both the Minikube cluster and the Ceph OSD image, then create a fresh local environment:

make local-down ACTION=delete
make local-up

Running minikube delete directly is unsupported because it bypasses the OSD image cleanup performed by the Makefile.

#C2. Empty NBD device files can crash ceph-volume raw list

Problem: If /sys/block/nbd* exists but user-created /dev/nbd* device files are unconnected or invalid, ceph-volume raw list may fail with:

lsblk: /dev/nbd0: not a block device
failed to get devices already provisioned by ceph-volume raw

Root cause: The old workaround of manually creating /dev/nbd* with mknod produced empty device files that lsblk could not inspect correctly.

Fix: Do not create NBD device nodes manually. Let the host kernel and udev provide the real devices. The local automation was updated accordingly.

#C5. GitLab gitaly and gitlab-shell can crash-loop after restart-heavy Helm operations

Problem: After certain Helm upgrades or restart paths, GitLab init containers can fail with permission errors under /etc/gitlab-secrets/....

Root cause: The relevant init containers copy secrets into an emptyDir. A container restart does not clear that volume, so stale files can block the next init run.

Manual recovery:

kubectl delete pod gitlab-gitaly-0 -n gitlab
kubectl delete pod -l app=gitlab-gitlab-shell -n gitlab

#C6. CoreDNS local overrides can fail if Jinja2 sees Python brace syntax

Problem: Inline Python used by the local CoreDNS role can break if f-string brace escaping is rendered by Jinja2 before the shell sees it.

Fix: Avoid f-string brace escaping in embedded Python. Use string concatenation for rendered CoreDNS fragments.

#C8. Ceph OSD liveness probe cycles after restart — transient HEALTH_ERR

Problem: After a Minikube/Docker restart, the Ceph OSD pod (rook-ceph-osd-0) goes through 2–3 kill/restart cycles before stabilizing. The pod liveness probe runs:

ceph --admin-daemon /run/ceph/ceph-osd.0.asok status

with a 5-second timeout. During startup, BlueStore takes time to mount the backing device and register the admin socket. The probe fires before the socket is ready, times out, and Kubernetes kills the container.

Visible symptoms:

• kubectl -n rook-ceph get pods shows OSD as 1/2 (log-collector up, osd container restarting)
• kubectl -n rook-ceph get cephcluster shows HEALTH_ERR / Progressing
• OSD events: Liveness probe failed: command timed out / Container osd failed liveness probe, will be restarted

Expected behavior: Self-heals within ~5 minutes. Persisted data is intact — the OSD logs show AllReplicasActivated with non-zero object counts on every restart, confirming BlueStore data survives. No manual action needed.

What to check if it hasn't recovered after 10 minutes:

kubectl -n rook-ceph get cephcluster        # should reach HEALTH_OK
kubectl -n rook-ceph get pods               # OSD should be 2/2 Running
kubectl -n rook-ceph logs <osd-pod> -c osd --tail=20  # look for AllReplicasActivated

Restart the operator if the OSD is still stuck after 10+ minutes:

kubectl -n rook-ceph rollout restart deployment/rook-ceph-operator

#C9. GitLab gitaly and gitlab-shell probe deadlock on constrained hosts

Problem: After make local-up (or any pod restart), gitlab-gitaly-0 and gitlab-gitlab-shell pods cycle through Running → Completed (exit 0) with incrementing restart counts. The webservice and sidekiq keep running but Git operations fail because gitaly is unavailable.

Root cause: The Helm chart ships liveness and readiness probes with timeoutSeconds: 3 and no initialDelaySeconds for gitaly. On a resource-constrained host (WSL2 / Minikube with limited RAM), the gRPC health check on port 8075 times out before gitaly has finished its BlueStore/repository initialization (can take 60–120 s). With failureThreshold: 3 and periodSeconds: 10, Kubernetes sends SIGTERM after only 30 s, causing a graceful exit (code 0 → "Completed"). The pod restarts into a 5-minute backoff.

Visible symptoms:

gitlab-gitaly-0        0/1  Completed  12  3h
gitlab-gitlab-shell-*  0/1  Completed  11  3h

Events:

Readiness probe failed: timeout: health rpc did not complete within 3s
Container gitaly failed liveness probe, will be restarted

Automated fix: deploy_gitlab/tasks/main.yml now patches both the gitaly StatefulSet and the gitlab-shell Deployment with lenient probe values (initialDelaySeconds: 120, timeoutSeconds: 10, failureThreshold: 10) whenever deployment_environment == 'local'. This runs automatically as part of make local-up.

Manual recovery (if already stuck):

kubectl patch statefulset -n gitlab gitlab-gitaly --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":120},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds","value":30},
  {"op":"add","path":"/spec/template/spec/containers/0/readinessProbe/initialDelaySeconds","value":120},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds","value":30}
]'
kubectl patch deployment -n gitlab gitlab-gitlab-shell --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds","value":30},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds","value":15}
]'
# Force a new rollout so the patched spec is applied immediately
kubectl rollout restart statefulset/gitlab-gitaly -n gitlab
kubectl rollout restart deployment/gitlab-gitlab-shell -n gitlab

#C10. GitLab gitaly and gitlab-shell configure init container Permission Denied on re-run

Problem: After gitlab-gitaly-0 or gitlab-gitlab-shell pods restart (e.g. from probe deadlock / C9), the configure init container crashes with:

cp: cannot create regular file '/init-secrets/shell/.gitlab_shell_secret': Permission denied
# or for gitlab-shell:
cp: cannot create regular file '//init-secrets/ssh/ssh_host_rsa_key': Permission denied

The pod stays in Init:CrashLoopBackOff.

Root cause: The configure init container copies secrets into the gitaly-secrets / shell-secrets emptyDir. On the first run, the files are created with mode 0440/0400 (read-only). The emptyDir persists within the pod lifetime — it is NOT cleared on container restart, only on pod deletion. On subsequent runs, cp without -f tries to open the existing read-only destination for writing → EACCES.

Two separate scripts are affected:

• gitlab-gitaly ConfigMap: cp -v -r -L missing -f (chart v18.4.0 omission)
• gitlab-gitlab-shell ConfigMap: the shell secrets use cp -f correctly, but the SSH host keys are copied with cp -v -r -L and then chmod 0400 — so on re-run those also fail

Automated fix: deploy_gitlab/tasks/main.yml patches both ConfigMaps after each helm upgrade (local environment only) to add -f to all cp calls.

Manual recovery (if already stuck):

kubectl patch configmap -n gitlab gitlab-gitaly \
  --type=merge \
  -p='{"data":{"configure":"set -e\nmkdir -p /init-secrets/gitaly /init-secrets/shell\ncp -f -v -r -L /init-config/.gitlab_shell_secret  /init-secrets/shell/.gitlab_shell_secret\ncp -f -v -r -L /init-config/gitaly_token  /init-secrets/gitaly/gitaly_token\n"}}'

kubectl patch configmap -n gitlab gitlab-gitlab-shell \
  --type=merge \
  -p='{"data":{"configure":"# BEGIN gitlab.scripts.configure.secrets\nset -e\nconfig_dir=\"/init-config\"\nsecret_dir=\"/init-secrets\"\nfor secret in shell ; do\n  mkdir -p \"${secret_dir}/${secret}\"\n  cp -f -v -r -L \"${config_dir}/${secret}/.\" \"${secret_dir}/${secret}/\"\ndone\n# END gitlab.scripts.configure.secrets\nmkdir -p /${secret_dir}/ssh\ncp -f -v -r -L /${config_dir}/ssh_host_* /${secret_dir}/ssh/\nchmod 0400 /${secret_dir}/ssh/ssh_host_*\n"}}'

kubectl delete pod -n gitlab gitlab-gitaly-0
kubectl delete pod -n gitlab -l app=gitlab-shell

#C7. Validation flows to keep repeating

The two local Rook-Ceph validation modes we care about are:

1. existing image / existing disk contents
2. blank image / blank disk contents

For both, the acceptance checks are:

• make local-up completes cleanly
• GitLab object storage goes through Ceph RGW

#Debian / Native Linux Host Specific Issues

#D1. Local Ceph on native Linux uses a loop-backed image on the host

Behavior: On native Linux the local target creates a sparse image file and attaches it with losetup. The detected loop device is then passed into the Minikube node.

Lesson: This path is different from WSL. Native Linux should use the host loop-backed image path in inventory/local/group_vars/all.yml; it should not use the Windows VHD settings.

#D2. Editing Minikube config alone does not change memory on an existing Docker-backed cluster

Problem: Changing ~/.minikube/machines/<profile>/config.json does not update the actual cgroup limit of the existing Minikube container.

Fix:

docker update --memory 12288m --memory-swap 12288m themelio-local

If you want the config file to match reality, update both.

#D3. Desktop packages inside the Debian VM waste memory needed by Minikube

Problem: A Debian VM used only as a local cluster host does not benefit from a full GNOME desktop, but it still consumes substantial RAM.

Fix:

sudo apt remove --purge gnome-shell gnome-session -y
sudo apt autoremove -y
sudo systemctl set-default multi-user.target

#D4. Hyper-V VM startup can fail when the Windows host is already memory-constrained

Problem: Starting a large Debian Hyper-V VM while Windows and WSL2 are already using a lot of RAM can fail with Hyper-V allocation errors.

Lesson: On a 32 GB machine, the local host budget matters. If Minikube runs inside a Debian VM, budget RAM across Windows, WSL2, and the VM before testing large local stacks.

#Windows / WSL2 Host Specific Issues

#W1. The Ceph disk is a Windows VHD and must be passed into WSL as a block device

Supported path: make local-up automates the WSL flow using:

• local_ceph_vhd_windows_path
• local_ceph_vhd_windows_drive
• local_ceph_vhd_device

from devops/ansible/inventory/local/group_vars/all.yml.

The helper path is:

1. elevate a PowerShell child
2. Mount-VHD -Path <vhd> if needed
3. resolve the current DiskNumber with Get-VHD
4. run wsl --mount \\.\PhysicalDriveN --bare

Manual fallback:

Mount-VHD -Path "C:\ceph-disk.vhd"
wsl --mount \\.\PhysicalDrive1 --bare

Lesson: The canonical place for this behavior is now the Makefile and local inventory, not an ad hoc note or shell history entry.

#W2. themelio-local Docker network can become orphaned after restarts

Problem: After Docker or host restarts, Minikube may fail with Address already in use because a stale themelio-local bridge still exists or a sidecar container is still attached to it.

Fix:

docker network disconnect themelio-local docker-hub-cache
docker network rm themelio-local
sudo ip link delete br-<id>

make local-up already tries to release containers from the stopped Minikube network before continuing.

#W3. docker-hub-cache should not be started before Minikube recreates its network

Problem: The cache container depends on the Minikube Docker network. If it comes up before Minikube, it can crash-loop without working network connectivity.

Lesson: Bring the environment up with make local-up; do not try to hand-start the cache before Minikube exists.

#W4. The .wslconfig memory cap limits the whole WSL2 environment

Problem: The Windows-side .wslconfig memory= setting caps all WSL2 distros together. A low cap can starve Minikube even if the laptop has free physical RAM.

Lesson: For local Rook-Ceph testing on WSL2, size the WSL memory cap for the whole workflow, not only for the shell you happen to be using.

#W5. WSL2 cannot reliably SSH directly into a Debian Hyper-V VM on the Default Switch

Problem: WSL2 and the Hyper-V Default Switch live on separate virtual networks. Direct routing from WSL2 to the VM often fails because Windows does not forward that traffic the way you might expect.

Workaround:

ssh.exe -o ConnectTimeout=15 ikons@<vm-ip> "command"

Use the Windows SSH client from WSL when you need to hop from the WSL host into a Hyper-V Debian VM.

#W7. CrashLoopBackOff pods don't recover promptly after make local-up applies inotify fix

Problem: When a laptop restarts, Docker auto-starts the Minikube container and Kubernetes brings up all pods immediately with the default kernel values (fs.inotify.max_user_instances=128). Istio ingressgateway pods and the Themelio root actors crash on startup with EMFILE (too many open files). Kubernetes enters exponential backoff — after 90+ restarts, the delay between retries is ~5 minutes.

When make local-up subsequently runs, it raises the inotify limits early in local-setup.yml. But pods already locked in backoff don't restart immediately; they wait for the backoff timer even though the root cause is now gone. The result: those pods appear still broken long after the playbook completes.

Affected pods (seen with 90–179 restarts):

• istio-system/istio-ingressgateway-web
• themelio/gitlab-root-actor
• themelio/k8s-root-actor

Manual recovery (before the Ansible fix):

kubectl -n istio-system rollout restart deployment/istio-ingressgateway-web
kubectl -n themelio rollout restart deployment/gitlab-root-actor deployment/k8s-root-actor

Automation fix (now in local-setup.yml): A task runs immediately after the sysctl -w step and deletes all CrashLoopBackOff pods cluster-wide, forcing an immediate fresh start with the corrected limits. Safe on fresh installs (no pods = no-op).

#W6. Hyper-V memory resizing uses Set-VMMemory, not Set-VM

Problem: Trying to resize VM memory with Set-VM -DynamicMemory $false fails because that is not the correct cmdlet for memory configuration.

Fix:

Set-VMMemory -VMName "Debian 12" -DynamicMemoryEnabled $false -StartupBytes 16GB