For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0037-dev-box-storage.md.

0037 — Dev-box storage from the final deployment's Rook/Ceph role

Status: Accepted for development use. Date: 11 October 2026.

Dev-box providers need block volumes and S3 buckets. Without a shared substrate, each one brings its own: an omnibus GitLab keeping LFS on its volume, Garage or MinIO for S3, and local-path volumes for databases. Code written against those local providers then differs from the final deployment, which uses Rook/Ceph.

Each dev-box VM has a second, empty disk for this purpose. The storage role installs Rook/Ceph on it and provides ceph-block volumes, ceph-filesystem volumes and ceph-bucket object buckets through Ceph RGW. The role is a copy of the final deployment's storage role from legacy/devops/ansible/roles/storage, cluster-wide flow only, with two later changes from the GB10 cluster: a 16 MiB CephFS readahead and a 16Gi OSD memory limit. The Minikube flow is not copied; the legacy local setup never installed Ceph.

Only inventory values differ from a multi-node installation: one node is both storage and monitor node, pools keep one replica, and the explicitly named disk is the only OSD. The role adds variables whose defaults keep the original behaviour: single-replica safety, Helm value overrides, verified chart archives and an optional dashboard route. The dev box pins the Ceph and operator images by digest and disables Prometheus integration, which it does not run. As in the final deployment, ceph-block is the default class: K3s no longer restores local-path as a second default, and claims made before storage keep their local-path volumes until they are recreated. It points the CSI node plugins at the dev box's K3s kubelet directory and silences only the single-replica health warnings, so a healthy dev box reports HEALTH_OK. Its OSD keeps the chart's 4Gi memory limit: Rook derives the OSD's cache target from the limit, a 32 GB VM also runs GitLab, Waldur and Keycloak, and measured CephFS reads were no faster with 16Gi. The readahead alone made them about 1.5 to 2 times faster. The role waits for the pool, filesystem and object store to be ready before reporting success. Rook leaves a bucket claim's BUCKET_REGION empty; S3 clients sign with the ceph-bucket class's region.

The dev box's disks are themselves Ceph volumes of the hosting OpenStack cloud, so each in-VM monitor and OSD adds synchronous writes to shared storage.

Kubernetes tasks run in the Metallo controller through the target tunnel; the VM needs neither Helm nor Kubernetes Python libraries. Before Rook starts, a guard on the VM accepts only a whole, unmounted disk without partitions that is empty or already a Ceph OSD, and never the root disk. The first successful run records the disk for the target; moving Ceph to another disk needs an explicit migration.

First setup runs make deploy from the repository root. Its ordered stages install K3s, then storage, then check block, filesystem and S3 storage, and each is safe to repeat. It runs before application setup, so the stack's claims use Ceph without naming a class. Later platform layers join the same command as further stages. New dev-box providers use the default class, ceph-filesystem or ceph-bucket rather than adding local storage providers. Existing laboratories and parallel-operation profiles move to it in their own changes.

Optional storage sources

Dedicated-disk storage remains the default. An explicit STORAGE_MODE=file preallocates a target-owned file on the internal filesystem and attaches it through a persistent loop-device binding. Its size and identity are recorded before use; setup refuses unrelated files, partial allocation and changes to the bound source. This single-node development option has the same single-replica limitation.

STORAGE_MODE=existing validates and binds a healthy Rook/Ceph provider with the three named StorageClasses and the dashboard required by the shared routes. It preserves that installation and the host's default StorageClass. New platform claims explicitly select Ceph. Provider extension and foreign namespace handover are separate operator actions; setup never adopts them implicitly. The personal ARM64 profile records its qualification.

Page source

Source snapshot · Revision f928834c8.