> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # ADR-0004: Xet Content-Addressed Storage for ML Artifacts ## Status Draft ## Context Themelio Projects host ML models, datasets, and spaces that routinely reach tens of GBs to TBs. [ADR-0003](/reference/docs/decisions/0001-lfs-storage-quota-enforcement.md) addresses the immediate need for LFS storage quota enforcement via a custom proxy that intercepts the Git LFS Batch API. This ADR evaluates the Xet protocol as a future storage backend that could complement or eventually supersede the LFS proxy, delivering chunk-level deduplication and significant bandwidth and storage efficiencies. ### What Is the Xet Protocol? Xet is a content-addressable storage (CAS) protocol originally developed by XetHub (acquired by Hugging Face in August 2024). It is now used as the default storage backend on the Hugging Face Hub (1M+ users, 20+ PB across 500K+ repositories as of 2026) and is under active IETF standardization ([draft-denis-xet-03](https://datatracker.ietf.org/doc/draft-denis-xet/03/)). Instead of storing whole files as opaque blobs (as Git LFS does), Xet: 1. **Chunks** files into variable-sized blocks (\~64 KiB target, 8–128 KiB range) using content-defined chunking (Gearhash CDC) 2. **Deduplicates** at the chunk level — both locally and globally across repositories 3. **Compresses** chunks with LZ4 and groups them into **xorbs** (containers up to 64 MiB) 4. **Records** file reconstruction metadata in **shards** describing how to reassemble files from xorb chunk ranges The default algorithm suite is **XET-BLAKE3-GEARHASH-LZ4** (BLAKE3 for hashing, Gearhash for CDC boundaries, LZ4 for compression). ### Why This Matters for Themelio FR-REP-02 already specifies "LFS (or Xet) protocol" with "chunk-level deduplication" as acceptance criteria (AC 2.2.1, AC 2.2.2). The concrete benefits for the platform are: - **Bandwidth savings**: When a user retrains a model and only 5% of weights change, Xet uploads only the changed chunks. For researchers iterating on TB-scale models over EU research networks, this is transformative. - **Storage efficiency under quotas**: With per-project quotas (FR-QTA-01), deduplication means quotas stretch further. If 10 repositories within a project fine-tune the same base model, the shared weights are stored once rather than 10 times. - **Pre-upload enforcement compatibility**: Like the LFS proxy in ADR-0003, a Xet CAS server can enforce quotas before or during upload — at token issuance, at xorb upload, or at shard registration. - **Industry alignment**: Hugging Face is the de facto platform for ML model sharing. Supporting the same protocol positions Themelio for interoperability (e.g., pulling models from the Hub, pushing to Themelio with the same tooling). ### Current Ecosystem State | Component | Status | License | | ---------------------------------------------------------------------------------------------------- | -------------------------------------------- | ----------------- | | [Xet Protocol Specification](https://huggingface.co/docs/xet/index) (v1.0.0) | Published, IETF draft submitted | Public | | [xet-core](https://github.com/huggingface/xet-core) (Rust client, reference impl) | Production, maintained by HF | Apache-2.0 | | [git-xet](https://github.com/huggingface/xet-core/tree/main/git_xet) (Git LFS custom transfer agent) | Stable | Apache-2.0 | | [hf_xet](https://pypi.org/project/hf-xet/) (Python bindings) | Stable, bundled with huggingface_hub >= 0.32 | Apache-2.0 | | [huggingface.js XetBlob](https://github.com/huggingface/huggingface.js) (JS/WASM client) | Production | Apache-2.0 | | [zig-xet](https://github.com/jedisct1/zig-xet) (third-party Zig client) | Complete, protocol-compliant | Open source | | **CAS Server** | **Proprietary (Hugging Face only)** | **Not available** | The critical gap is that **no open-source CAS server implementation exists**. The protocol specification is fully public and sufficient to build against, but the server would need to be built from scratch. ## Decision Themelio adopts the Xet protocol as its target storage backend for ML artifacts, to be implemented when the platform's maturity and resource allocation allow. The LFS proxy (ADR-0003) remains the near-term solution for quota enforcement, and the system should be designed with awareness that Xet may follow. This is not a commitment to a specific timeline. It is a recognition that the Xet protocol offers substantial advantages for Themelio's workloads and that the architecture should not preclude its adoption. ## Approach ### CAS Server: What Needs to Be Built A Xet CAS server implements four HTTP endpoints as defined by the [CAS API specification](https://huggingface.co/docs/xet/api): | Endpoint | Method | Purpose | Auth Scope | | ------------------------------------------ | ------ | ------------------------------------------- | ---------- | | `/v1/reconstructions/{file_id}` | GET | File reconstruction metadata (download) | `read` | | `/v1/chunks/default-merkledb/{chunk_hash}` | GET | Global chunk deduplication query | `read` | | `/v1/xorbs/default/{xorb_hash}` | POST | Upload serialized xorb (chunk container) | `write` | | `/v1/shards` | POST | Upload shard (file registration + metadata) | `write` | The server's responsibilities include: 1. **Xorb storage** in S3-compatible object storage (MinIO), keyed by content hash 2. **Shard storage and file indexing** for reconstruction lookups (PostgreSQL) 3. **Global deduplication index** mapping chunk hashes to xorb locations 4. **Authentication** via token exchange integrated with GitLab/Themelio identity 5. **Ordering enforcement**: reject shard uploads referencing non-existent xorbs 6. **Idempotent uploads**: re-uploading the same content hash is safe (protocol requirement) ### Upload and Download Flows **Upload:** ``` Client: chunk file → compute hashes → query CAS for existing chunks (dedup) → group new chunks into xorbs → compress (LZ4) → serialize → POST /v1/xorbs (upload each new xorb) → build shard (file reconstruction + xorb metadata) → POST /v1/shards (register files) → commit LFS pointer file to Git (contains SHA256) ``` **Download:** ``` Client: GET /v1/reconstructions/{file_id} → receive terms (xorb hash + chunk ranges) → fetch xorb byte ranges via HTTP Range headers (parallel) → decompress chunks → concatenate in term order → file reconstructed ``` ### Quota Enforcement Quota enforcement integrates naturally into the CAS server at multiple points: 1. **At token issuance**: The auth endpoint (analogous to HF Hub's `/xet-write-token/` endpoint) can refuse to issue write tokens when a project's quota is exhausted. This is true pre-upload enforcement. 2. **At xorb upload**: The server tracks cumulative xorb sizes per project and can reject uploads when quota would be exceeded. Since xorbs are at most 64 MiB, the maximum overshoot is bounded and small relative to TB-scale quotas. 3. **At shard registration**: The shard contains file reconstruction metadata including effective file sizes. The server can reject the shard and trigger cleanup of orphaned xorbs. A combined approach (token-level gating + xorb-level tracking) provides robust enforcement with minimal waste, satisfying the same pre-upload enforcement requirement as ADR-0003. **Dedup-aware quota accounting**: The CAS server has full visibility into shared chunks. Unlike LFS (where each file is counted at full size), Xet can count only _new_ storage consumed — deduplicated chunks are free. This is fairer for projects with many forks or incremental model versions. ### Deduplication Scope Initial implementation should scope deduplication **per Themelio Project** (i.e., per GitLab group). This simplifies data isolation, quota accounting, and aligns with the existing per-project quota model from Waldur. > **Note**: Cross-project deduplication (global dedup across all Themelio Projects) could yield significant additional storage savings — particularly for popular base models that many projects fine-tune. However, it introduces complex questions around quota attribution (which project "owns" a shared chunk?) and billing fairness. This should be explored as a separate initiative once per-project Xet storage is operational, ideally in collaboration with Waldur's billing/allocation model to determine a fair accounting approach (e.g., reference-counted, first-writer-owns, or proportional attribution). ### Client Strategy The Xet protocol supports Git integration via the **standalone transfer agent** mechanism: ```ini [lfs] standalonetransferagent = xet [lfs "customtransfer.xet"] path = git-xet direction = both concurrent = true ``` With this configuration, `git-xet` handles all data transfer directly with the CAS server. GitLab only stores pointer files (standard LFS pointer format) in the Git repository. The client never contacts GitLab's LFS Batch API for data transfer, which is the cleanest integration path for GitLab CE. Users would need the `git-xet` binary installed (available via package managers) or use `huggingface_hub` (which bundles `hf_xet`). This is an acceptable requirement given that Themelio's user base works with ML tooling that already depends on these packages. ### Backward Compatibility with Git LFS Clients Standard Git LFS clients (without `git-xet`) cannot natively interact with Xet storage. On the Hugging Face Hub, backward compatibility is provided by a **server-side LFS bridge** — a translation layer that converts LFS Batch API requests into Xet CAS operations. This bridge is part of Hugging Face's proprietary infrastructure and is not included in the open-source Xet protocol. If backward compatibility with plain `git lfs` clients is needed, Themelio would need to implement a similar bridge. This is a non-trivial component (translating between two storage protocols) and should be considered optional — Themelio can mandate Xet-aware clients for repositories backed by Xet storage, while repositories created before the transition continue to use the LFS proxy from ADR-0003. ### Relationship to ADR-0003 ADR-0003 and this ADR share the same deployment topology but are independent decisions: | Aspect | ADR-0003 (LFS Proxy) | ADR-0004 (Xet CAS Server) | | ------------------- | ---------------------- | ------------------------------------ | | Protocol | Git LFS Batch API | Xet CAS API (4 endpoints) | | Storage granularity | File-level | Chunk-level (\~64 KiB) | | Deduplication | None | Content-defined, per-project | | GitLab LFS | Disabled per-project | Disabled per-project | | Storage backend | MinIO (S3) | MinIO (S3) — same or separate bucket | | Quota enforcement | Batch API interception | Token + xorb + shard level | | Client changes | `.lfsconfig` only | `.lfsconfig` + `git-xet` binary | The two can coexist: LFS-backed repositories use the ADR-0003 proxy, Xet-backed repositories use the CAS server. New repositories can default to Xet when the CAS server is ready, while existing repositories continue on LFS indefinitely (or are migrated in the background). Design recommendations for ADR-0003 implementation that would ease a future Xet transition: - Use content-hash-based keys in MinIO (not GitLab project IDs) where possible - Store quota tracking in a database (not in-memory), as the CAS server will reuse it - Keep the proxy's auth mechanism extensible for CAS token exchange ## Complexity Assessment | Component | Effort | Notes | | ----------------------------------------- | -------- | ------------------------------------------------ | | HTTP API (4 endpoints) | Medium | Well-specified, deterministic behavior | | Xorb storage (MinIO) | Low | Store/retrieve blobs by hash key | | Shard parsing + file reconstruction index | High | Binary format parsing, Merkle verification | | Per-project deduplication index | Medium | Chunk hash → xorb location within project scope | | Auth integration (GitLab token exchange) | Medium | Custom token issuer replacing HF Hub's endpoint | | Quota enforcement | Medium | Reuses patterns from ADR-0003 | | LFS bridge (optional) | High | Full LFS Batch API ↔ CAS translation | | **Estimated total** | **High** | Comparable to building a focused storage service | ### Technology Alignment The protocol is HTTP-based, so any language works. Candidates: - **TypeScript/Fastify**: Aligns with existing Themelio services (gitlab-root-actor, k8s-root-actor). Simpler to maintain within the team. - **Rust**: Aligns with xet-core. Better performance for hash computation and binary format parsing. The `cas_client` and `mdb_shard` crates could be reused for testing. ## Consequences ### If adopted: - **New service**: A CAS server is added to the Kubernetes deployment (ArgoCD manifests, health checks, monitoring) - **Repository creation**: New repositories would be configured with `git-xet` transfer agent settings (automatable in `gitlab-root-actor`) - **User tooling**: Users need `git-xet` or `huggingface_hub >= 0.32` — this should be documented and ideally bundled in any Themelio CLI or SDK - **GitLab UI**: LFS file browsing in GitLab's web UI is unavailable for Xet-backed repos (same limitation as ADR-0003). The Themelio web UI would need to handle file previews via the CAS reconstruction API. - **Storage topology**: MinIO stores both LFS objects (from ADR-0003 repos) and xorbs (from Xet repos), potentially in separate buckets - **Monitoring**: CAS server metrics (upload/download throughput, dedup ratio, quota utilization) become important operational signals ### If not adopted: - The LFS proxy from ADR-0003 remains the sole storage enforcement mechanism - No chunk-level deduplication — each file version stored in full - Higher storage consumption and bandwidth usage for iterative ML workflows - Potential divergence from the broader ML ecosystem as Xet adoption grows ## Open Questions 1. **Timeline**: When does the platform's scale and user demand justify the investment? A useful signal would be observing high storage waste from duplicate model weights across repositories. 2. **Build vs. wait**: Hugging Face may open-source their CAS server or a community implementation may emerge. The IETF standardization effort suggests they want ecosystem adoption, but there is no public commitment to open-sourcing the server. 3. **Technology choice**: TypeScript (team familiarity) vs. Rust (protocol alignment, performance). A prototype in TypeScript with performance-critical paths (hashing, shard parsing) delegated to native modules via xet-core WASM could be a pragmatic middle ground. 4. **MinIO bucket structure**: One bucket with hash-prefixed keys, or separate buckets per project? Affects dedup scope and operational simplicity. 5. **Migration strategy**: When Xet is ready, should existing LFS repositories be migrated (background re-chunking), or left on LFS permanently? Migration yields dedup benefits but adds operational complexity. ## References - [Xet Protocol Specification v1.0.0](https://huggingface.co/docs/xet/index) - [CAS API Documentation](https://huggingface.co/docs/xet/api) - [Upload Protocol](https://huggingface.co/docs/xet/upload-protocol) - [Download Protocol](https://huggingface.co/docs/xet/download-protocol) - [Authentication and Authorization](https://huggingface.co/docs/xet/auth) - [Xet Deduplication](https://huggingface.co/docs/hub/xet/deduplication) - [IETF Internet-Draft: draft-denis-xet-03](https://datatracker.ietf.org/doc/draft-denis-xet/03/) - [xet-core (Rust reference implementation)](https://github.com/huggingface/xet-core) - [zig-xet (Zig implementation)](https://github.com/jedisct1/zig-xet) - [Migrating the Hub from Git LFS to Xet](https://huggingface.co/blog/migrating-the-hub-to-xet) - [XetHub joins Hugging Face](https://huggingface.co/blog/xethub-joins-hf) - [Frank Denis — Introduction to XET (Part 1)](https://00f.net/2026/01/19/xet-intro-1/) - [Git LFS Custom Transfers](https://github.com/git-lfs/git-lfs/blob/main/docs/custom-transfers.md) - [ADR-0003: LFS Storage Quota Enforcement](/reference/docs/decisions/0001-lfs-storage-quota-enforcement.md)
Page source Source snapshot · Revision `f928834c8`.