---
title: Local Stack — Known Issues and Lessons Learned
---

# Local Stack: Known Issues and Lessons Learned

Issues, workarounds, and lessons discovered while validating:

```bash
cd devops/ansible
make local-up
```

This document covers both supported local host modes:

- native Debian/Linux hosts using a loop-backed image
- Windows/WSL2 hosts using a Windows VHD passed through with `wsl --mount`

Canonical automation now lives in code, not only in notes:

- `devops/ansible/Makefile` is the entrypoint for local bring-up
- `make local-up` is the supported path for Rook-Ceph local runs (Ceph RGW for object storage, Minikube `standard` for PVCs)
- WSL VHD settings live in `devops/ansible/inventory/local/group_vars/all.yml`

---

## Common Issues (all hosts)

### C1. Inconsistent Ceph state blocks local reconciliation

**Problem**:
If the local Rook/Ceph namespace, CephCluster, OSD deployments, and backing image no
longer describe the same cluster, `make local-up` cannot safely determine whether the
existing object-storage data should be preserved.

**Expected behavior**:
Local reconciliation is non-destructive. It never removes finalizers, wipes BlueStore
metadata, or resets the OSD image automatically. Helm or Rook may report the inconsistent
resources that prevent reconciliation.

**Resolution**:
Use the supported teardown command to explicitly destroy both the Minikube cluster and
the Ceph OSD image, then create a fresh local environment:

```bash
make local-down ACTION=delete
make local-up
```

Running `minikube delete` directly is unsupported because it bypasses the OSD image
cleanup performed by the Makefile.

---

### C2. Empty NBD device files can crash `ceph-volume raw list`

**Problem**:
If `/sys/block/nbd*` exists but user-created `/dev/nbd*` device files are unconnected or
invalid, `ceph-volume raw list` may fail with:

```text
lsblk: /dev/nbd0: not a block device
failed to get devices already provisioned by ceph-volume raw
```

**Root cause**:
The old workaround of manually creating `/dev/nbd*` with `mknod` produced empty device
files that `lsblk` could not inspect correctly.

**Fix**:
Do not create NBD device nodes manually. Let the host kernel and udev provide the real
devices. The local automation was updated accordingly.

---

### C5. GitLab gitaly and gitlab-shell can crash-loop after restart-heavy Helm operations

**Problem**:
After certain Helm upgrades or restart paths, GitLab init containers can fail with
permission errors under `/etc/gitlab-secrets/...`.

**Root cause**:
The relevant init containers copy secrets into an `emptyDir`. A container restart does not
clear that volume, so stale files can block the next init run.

**Manual recovery**:
```bash
kubectl delete pod gitlab-gitaly-0 -n gitlab
kubectl delete pod -l app=gitlab-gitlab-shell -n gitlab
```

---

### C6. CoreDNS local overrides can fail if Jinja2 sees Python brace syntax

**Problem**:
Inline Python used by the local CoreDNS role can break if f-string brace escaping is
rendered by Jinja2 before the shell sees it.

**Fix**:
Avoid f-string brace escaping in embedded Python. Use string concatenation for rendered
CoreDNS fragments.

---

### C8. Ceph OSD liveness probe cycles after restart — transient HEALTH_ERR

**Problem**:
After a Minikube/Docker restart, the Ceph OSD pod (`rook-ceph-osd-0`) goes through 2–3 kill/restart cycles before stabilizing. The pod liveness probe runs:
```
ceph --admin-daemon /run/ceph/ceph-osd.0.asok status
```
with a 5-second timeout. During startup, BlueStore takes time to mount the backing device and register the admin socket. The probe fires before the socket is ready, times out, and Kubernetes kills the container.

**Visible symptoms**:
- `kubectl -n rook-ceph get pods` shows OSD as `1/2` (log-collector up, osd container restarting)
- `kubectl -n rook-ceph get cephcluster` shows `HEALTH_ERR` / `Progressing`
- OSD events: `Liveness probe failed: command timed out` / `Container osd failed liveness probe, will be restarted`

**Expected behavior**:
Self-heals within ~5 minutes. Persisted data is intact — the OSD logs show `AllReplicasActivated` with non-zero object counts on every restart, confirming BlueStore data survives. No manual action needed.

**What to check** if it hasn't recovered after 10 minutes:
```bash
kubectl -n rook-ceph get cephcluster        # should reach HEALTH_OK
kubectl -n rook-ceph get pods               # OSD should be 2/2 Running
kubectl -n rook-ceph logs <osd-pod> -c osd --tail=20  # look for AllReplicasActivated
```

**Restart the operator** if the OSD is still stuck after 10+ minutes:
```bash
kubectl -n rook-ceph rollout restart deployment/rook-ceph-operator
```

---

### C9. GitLab gitaly and gitlab-shell probe deadlock on constrained hosts

**Problem**:
After `make local-up` (or any pod restart), `gitlab-gitaly-0` and `gitlab-gitlab-shell` pods cycle through `Running → Completed` (exit 0) with incrementing restart counts. The webservice and sidekiq keep running but Git operations fail because gitaly is unavailable.

**Root cause**:
The Helm chart ships liveness and readiness probes with `timeoutSeconds: 3` and **no** `initialDelaySeconds` for gitaly. On a resource-constrained host (WSL2 / Minikube with limited RAM), the gRPC health check on port 8075 times out before gitaly has finished its BlueStore/repository initialization (can take 60–120 s). With `failureThreshold: 3` and `periodSeconds: 10`, Kubernetes sends SIGTERM after only 30 s, causing a graceful exit (code 0 → "Completed"). The pod restarts into a 5-minute backoff.

**Visible symptoms**:
```
gitlab-gitaly-0        0/1  Completed  12  3h
gitlab-gitlab-shell-*  0/1  Completed  11  3h
```
Events:
```
Readiness probe failed: timeout: health rpc did not complete within 3s
Container gitaly failed liveness probe, will be restarted
```

**Automated fix**:
`deploy_gitlab/tasks/main.yml` now patches both the gitaly `StatefulSet` and the gitlab-shell `Deployment` with lenient probe values (`initialDelaySeconds: 120`, `timeoutSeconds: 10`, `failureThreshold: 10`) whenever `deployment_environment == 'local'`. This runs automatically as part of `make local-up`.

**Manual recovery** (if already stuck):
```bash
kubectl patch statefulset -n gitlab gitlab-gitaly --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":120},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds","value":30},
  {"op":"add","path":"/spec/template/spec/containers/0/readinessProbe/initialDelaySeconds","value":120},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds","value":30}
]'
kubectl patch deployment -n gitlab gitlab-gitlab-shell --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds","value":30},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/timeoutSeconds","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold","value":10},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds","value":15}
]'
# Force a new rollout so the patched spec is applied immediately
kubectl rollout restart statefulset/gitlab-gitaly -n gitlab
kubectl rollout restart deployment/gitlab-gitlab-shell -n gitlab
```

---

### C10. GitLab gitaly and gitlab-shell configure init container Permission Denied on re-run

**Problem**:
After `gitlab-gitaly-0` or `gitlab-gitlab-shell` pods restart (e.g. from probe deadlock / C9), the `configure` init container crashes with:
```
cp: cannot create regular file '/init-secrets/shell/.gitlab_shell_secret': Permission denied
# or for gitlab-shell:
cp: cannot create regular file '//init-secrets/ssh/ssh_host_rsa_key': Permission denied
```
The pod stays in `Init:CrashLoopBackOff`.

**Root cause**:
The `configure` init container copies secrets into the `gitaly-secrets` / `shell-secrets` emptyDir. On the first run, the files are created with mode `0440`/`0400` (read-only). The emptyDir persists within the pod lifetime — it is NOT cleared on container restart, only on pod deletion. On subsequent runs, `cp` without `-f` tries to open the existing read-only destination for writing → EACCES.

Two separate scripts are affected:
- `gitlab-gitaly` ConfigMap: `cp -v -r -L` missing `-f` (chart v18.4.0 omission)
- `gitlab-gitlab-shell` ConfigMap: the `shell` secrets use `cp -f` correctly, but the SSH host keys are copied with `cp -v -r -L` and then `chmod 0400` — so on re-run those also fail

**Automated fix**:
`deploy_gitlab/tasks/main.yml` patches both ConfigMaps after each helm upgrade (local environment only) to add `-f` to all `cp` calls.

**Manual recovery** (if already stuck):
```bash
kubectl patch configmap -n gitlab gitlab-gitaly \
  --type=merge \
  -p='{"data":{"configure":"set -e\nmkdir -p /init-secrets/gitaly /init-secrets/shell\ncp -f -v -r -L /init-config/.gitlab_shell_secret  /init-secrets/shell/.gitlab_shell_secret\ncp -f -v -r -L /init-config/gitaly_token  /init-secrets/gitaly/gitaly_token\n"}}'

kubectl patch configmap -n gitlab gitlab-gitlab-shell \
  --type=merge \
  -p='{"data":{"configure":"# BEGIN gitlab.scripts.configure.secrets\nset -e\nconfig_dir=\"/init-config\"\nsecret_dir=\"/init-secrets\"\nfor secret in shell ; do\n  mkdir -p \"${secret_dir}/${secret}\"\n  cp -f -v -r -L \"${config_dir}/${secret}/.\" \"${secret_dir}/${secret}/\"\ndone\n# END gitlab.scripts.configure.secrets\nmkdir -p /${secret_dir}/ssh\ncp -f -v -r -L /${config_dir}/ssh_host_* /${secret_dir}/ssh/\nchmod 0400 /${secret_dir}/ssh/ssh_host_*\n"}}'

kubectl delete pod -n gitlab gitlab-gitaly-0
kubectl delete pod -n gitlab -l app=gitlab-shell
```

---

### C7. Validation flows to keep repeating

The two local Rook-Ceph validation modes we care about are:

1. existing image / existing disk contents
2. blank image / blank disk contents

For both, the acceptance checks are:

- `make local-up` completes cleanly
- GitLab object storage goes through Ceph RGW

---

## Debian / Native Linux Host Specific Issues

### D1. Local Ceph on native Linux uses a loop-backed image on the host

**Behavior**:
On native Linux the local target creates a sparse image file and attaches it with
`losetup`. The detected loop device is then passed into the Minikube node.

**Lesson**:
This path is different from WSL. Native Linux should use the host loop-backed image path in
`inventory/local/group_vars/all.yml`; it should not use the Windows VHD settings.

---

### D2. Editing Minikube config alone does not change memory on an existing Docker-backed cluster

**Problem**:
Changing `~/.minikube/machines/<profile>/config.json` does not update the actual cgroup
limit of the existing Minikube container.

**Fix**:
```bash
docker update --memory 12288m --memory-swap 12288m themelio-local
```

If you want the config file to match reality, update both.

---

### D3. Desktop packages inside the Debian VM waste memory needed by Minikube

**Problem**:
A Debian VM used only as a local cluster host does not benefit from a full GNOME desktop,
but it still consumes substantial RAM.

**Fix**:
```bash
sudo apt remove --purge gnome-shell gnome-session -y
sudo apt autoremove -y
sudo systemctl set-default multi-user.target
```

---

### D4. Hyper-V VM startup can fail when the Windows host is already memory-constrained

**Problem**:
Starting a large Debian Hyper-V VM while Windows and WSL2 are already using a lot of RAM
can fail with Hyper-V allocation errors.

**Lesson**:
On a 32 GB machine, the local host budget matters. If Minikube runs inside a Debian VM,
budget RAM across Windows, WSL2, and the VM before testing large local stacks.

---

## Windows / WSL2 Host Specific Issues

### W1. The Ceph disk is a Windows VHD and must be passed into WSL as a block device

**Supported path**:
`make local-up` automates the WSL flow using:

- `local_ceph_vhd_windows_path`
- `local_ceph_vhd_windows_drive`
- `local_ceph_vhd_device`

from `devops/ansible/inventory/local/group_vars/all.yml`.

The helper path is:

1. elevate a PowerShell child
2. `Mount-VHD -Path <vhd>` if needed
3. resolve the current `DiskNumber` with `Get-VHD`
4. run `wsl --mount \\.\PhysicalDriveN --bare`

**Manual fallback**:
```powershell
Mount-VHD -Path "C:\ceph-disk.vhd"
wsl --mount \\.\PhysicalDrive1 --bare
```

**Lesson**:
The canonical place for this behavior is now the Makefile and local inventory, not an ad
hoc note or shell history entry.

---

### W2. `themelio-local` Docker network can become orphaned after restarts

**Problem**:
After Docker or host restarts, Minikube may fail with `Address already in use` because a
stale `themelio-local` bridge still exists or a sidecar container is still attached to it.

**Fix**:
```bash
docker network disconnect themelio-local docker-hub-cache
docker network rm themelio-local
sudo ip link delete br-<id>
```

`make local-up` already tries to release containers from the stopped Minikube network
before continuing.

---

### W3. `docker-hub-cache` should not be started before Minikube recreates its network

**Problem**:
The cache container depends on the Minikube Docker network. If it comes up before Minikube,
it can crash-loop without working network connectivity.

**Lesson**:
Bring the environment up with `make local-up`; do not try to hand-start the cache before
Minikube exists.

---

### W4. The `.wslconfig` memory cap limits the whole WSL2 environment

**Problem**:
The Windows-side `.wslconfig` `memory=` setting caps all WSL2 distros together. A low cap
can starve Minikube even if the laptop has free physical RAM.

**Lesson**:
For local Rook-Ceph testing on WSL2, size the WSL memory cap for the whole workflow, not
only for the shell you happen to be using.

---

### W5. WSL2 cannot reliably SSH directly into a Debian Hyper-V VM on the Default Switch

**Problem**:
WSL2 and the Hyper-V Default Switch live on separate virtual networks. Direct routing from
WSL2 to the VM often fails because Windows does not forward that traffic the way you might
expect.

**Workaround**:
```bash
ssh.exe -o ConnectTimeout=15 ikons@<vm-ip> "command"
```

Use the Windows SSH client from WSL when you need to hop from the WSL host into a Hyper-V
Debian VM.

---

### W7. CrashLoopBackOff pods don't recover promptly after `make local-up` applies inotify fix

**Problem**:
When a laptop restarts, Docker auto-starts the Minikube container and Kubernetes brings up all pods immediately with the default kernel values (`fs.inotify.max_user_instances=128`). Istio ingressgateway pods and the Themelio root actors crash on startup with EMFILE (`too many open files`). Kubernetes enters exponential backoff — after 90+ restarts, the delay between retries is ~5 minutes.

When `make local-up` subsequently runs, it raises the inotify limits early in `local-setup.yml`. But pods already locked in backoff don't restart immediately; they wait for the backoff timer even though the root cause is now gone. The result: those pods appear still broken long after the playbook completes.

**Affected pods** (seen with 90–179 restarts):
- `istio-system/istio-ingressgateway-web`
- `themelio/gitlab-root-actor`
- `themelio/k8s-root-actor`

**Manual recovery** (before the Ansible fix):
```bash
kubectl -n istio-system rollout restart deployment/istio-ingressgateway-web
kubectl -n themelio rollout restart deployment/gitlab-root-actor deployment/k8s-root-actor
```

**Automation fix** (now in `local-setup.yml`):
A task runs immediately after the `sysctl -w` step and deletes all CrashLoopBackOff pods cluster-wide, forcing an immediate fresh start with the corrected limits. Safe on fresh installs (no pods = no-op).

---

### W6. Hyper-V memory resizing uses `Set-VMMemory`, not `Set-VM`

**Problem**:
Trying to resize VM memory with `Set-VM -DynamicMemory $false` fails because that is not the
correct cmdlet for memory configuration.

**Fix**:
```powershell
Set-VMMemory -VMName "Debian 12" -DynamicMemoryEnabled $false -StartupBytes 16GB
```