#ADR-0004: Xet Content-Addressed Storage for ML Artifacts

#Status

Draft

#Context

Themelio Projects host ML models, datasets, and spaces that routinely reach tens of GBs to TBs. ADR-0003 addresses the immediate need for LFS storage quota enforcement via a custom proxy that intercepts the Git LFS Batch API. This ADR evaluates the Xet protocol as a future storage backend that could complement or eventually supersede the LFS proxy, delivering chunk-level deduplication and significant bandwidth and storage efficiencies.

#What Is the Xet Protocol?

Xet is a content-addressable storage (CAS) protocol originally developed by XetHub (acquired by Hugging Face in August 2024). It is now used as the default storage backend on the Hugging Face Hub (1M+ users, 20+ PB across 500K+ repositories as of 2026) and is under active IETF standardization (draft-denis-xet-03).

Instead of storing whole files as opaque blobs (as Git LFS does), Xet:

1. Chunks files into variable-sized blocks (~64 KiB target, 8–128 KiB range) using content-defined chunking (Gearhash CDC)
2. Deduplicates at the chunk level — both locally and globally across repositories
3. Compresses chunks with LZ4 and groups them into xorbs (containers up to 64 MiB)
4. Records file reconstruction metadata in shards describing how to reassemble files from xorb chunk ranges

The default algorithm suite is XET-BLAKE3-GEARHASH-LZ4 (BLAKE3 for hashing, Gearhash for CDC boundaries, LZ4 for compression).

#Why This Matters for Themelio

FR-REP-02 already specifies "LFS (or Xet) protocol" with "chunk-level deduplication" as acceptance criteria (AC 2.2.1, AC 2.2.2). The concrete benefits for the platform are:

• Bandwidth savings: When a user retrains a model and only 5% of weights change, Xet uploads only the changed chunks. For researchers iterating on TB-scale models over EU research networks, this is transformative.
• Storage efficiency under quotas: With per-project quotas (FR-QTA-01), deduplication means quotas stretch further. If 10 repositories within a project fine-tune the same base model, the shared weights are stored once rather than 10 times.
• Pre-upload enforcement compatibility: Like the LFS proxy in ADR-0003, a Xet CAS server can enforce quotas before or during upload — at token issuance, at xorb upload, or at shard registration.
• Industry alignment: Hugging Face is the de facto platform for ML model sharing. Supporting the same protocol positions Themelio for interoperability (e.g., pulling models from the Hub, pushing to Themelio with the same tooling).

#Current Ecosystem State

The critical gap is that no open-source CAS server implementation exists. The protocol specification is fully public and sufficient to build against, but the server would need to be built from scratch.

#Decision

Themelio adopts the Xet protocol as its target storage backend for ML artifacts, to be implemented when the platform's maturity and resource allocation allow. The LFS proxy (ADR-0003) remains the near-term solution for quota enforcement, and the system should be designed with awareness that Xet may follow.

This is not a commitment to a specific timeline. It is a recognition that the Xet protocol offers substantial advantages for Themelio's workloads and that the architecture should not preclude its adoption.

#Approach

#CAS Server: What Needs to Be Built

A Xet CAS server implements four HTTP endpoints as defined by the CAS API specification:

The server's responsibilities include:

1. Xorb storage in S3-compatible object storage (MinIO), keyed by content hash
2. Shard storage and file indexing for reconstruction lookups (PostgreSQL)
3. Global deduplication index mapping chunk hashes to xorb locations
4. Authentication via token exchange integrated with GitLab/Themelio identity
5. Ordering enforcement: reject shard uploads referencing non-existent xorbs
6. Idempotent uploads: re-uploading the same content hash is safe (protocol requirement)

#Upload and Download Flows

Upload:

Client: chunk file → compute hashes → query CAS for existing chunks (dedup)
      → group new chunks into xorbs → compress (LZ4) → serialize
      → POST /v1/xorbs (upload each new xorb)
      → build shard (file reconstruction + xorb metadata)
      → POST /v1/shards (register files)
      → commit LFS pointer file to Git (contains SHA256)

Download:

Client: GET /v1/reconstructions/{file_id} → receive terms (xorb hash + chunk ranges)
      → fetch xorb byte ranges via HTTP Range headers (parallel)
      → decompress chunks → concatenate in term order → file reconstructed

#Quota Enforcement

Quota enforcement integrates naturally into the CAS server at multiple points:

1. At token issuance: The auth endpoint (analogous to HF Hub's /xet-write-token/ endpoint) can refuse to issue write tokens when a project's quota is exhausted. This is true pre-upload enforcement.
2. At xorb upload: The server tracks cumulative xorb sizes per project and can reject uploads when quota would be exceeded. Since xorbs are at most 64 MiB, the maximum overshoot is bounded and small relative to TB-scale quotas.
3. At shard registration: The shard contains file reconstruction metadata including effective file sizes. The server can reject the shard and trigger cleanup of orphaned xorbs.

A combined approach (token-level gating + xorb-level tracking) provides robust enforcement with minimal waste, satisfying the same pre-upload enforcement requirement as ADR-0003.

Dedup-aware quota accounting: The CAS server has full visibility into shared chunks. Unlike LFS (where each file is counted at full size), Xet can count only new storage consumed — deduplicated chunks are free. This is fairer for projects with many forks or incremental model versions.

#Deduplication Scope

Initial implementation should scope deduplication per Themelio Project (i.e., per GitLab group). This simplifies data isolation, quota accounting, and aligns with the existing per-project quota model from Waldur.

Note: Cross-project deduplication (global dedup across all Themelio Projects) could yield significant additional storage savings — particularly for popular base models that many projects fine-tune. However, it introduces complex questions around quota attribution (which project "owns" a shared chunk?) and billing fairness. This should be explored as a separate initiative once per-project Xet storage is operational, ideally in collaboration with Waldur's billing/allocation model to determine a fair accounting approach (e.g., reference-counted, first-writer-owns, or proportional attribution).

#Client Strategy

The Xet protocol supports Git integration via the standalone transfer agent mechanism:

[lfs]
    standalonetransferagent = xet
[lfs "customtransfer.xet"]
    path = git-xet
    direction = both
    concurrent = true

With this configuration, git-xet handles all data transfer directly with the CAS server. GitLab only stores pointer files (standard LFS pointer format) in the Git repository. The client never contacts GitLab's LFS Batch API for data transfer, which is the cleanest integration path for GitLab CE.

Users would need the git-xet binary installed (available via package managers) or use huggingface_hub (which bundles hf_xet). This is an acceptable requirement given that Themelio's user base works with ML tooling that already depends on these packages.

#Backward Compatibility with Git LFS Clients

Standard Git LFS clients (without git-xet) cannot natively interact with Xet storage. On the Hugging Face Hub, backward compatibility is provided by a server-side LFS bridge — a translation layer that converts LFS Batch API requests into Xet CAS operations. This bridge is part of Hugging Face's proprietary infrastructure and is not included in the open-source Xet protocol.

If backward compatibility with plain git lfs clients is needed, Themelio would need to implement a similar bridge. This is a non-trivial component (translating between two storage protocols) and should be considered optional — Themelio can mandate Xet-aware clients for repositories backed by Xet storage, while repositories created before the transition continue to use the LFS proxy from ADR-0003.

#Relationship to ADR-0003

ADR-0003 and this ADR share the same deployment topology but are independent decisions:

The two can coexist: LFS-backed repositories use the ADR-0003 proxy, Xet-backed repositories use the CAS server. New repositories can default to Xet when the CAS server is ready, while existing repositories continue on LFS indefinitely (or are migrated in the background).

Design recommendations for ADR-0003 implementation that would ease a future Xet transition:

• Use content-hash-based keys in MinIO (not GitLab project IDs) where possible
• Store quota tracking in a database (not in-memory), as the CAS server will reuse it
• Keep the proxy's auth mechanism extensible for CAS token exchange

#Complexity Assessment

#Technology Alignment

The protocol is HTTP-based, so any language works. Candidates:

• TypeScript/Fastify: Aligns with existing Themelio services (gitlab-root-actor, k8s-root-actor). Simpler to maintain within the team.
• Rust: Aligns with xet-core. Better performance for hash computation and binary format parsing. The cas_client and mdb_shard crates could be reused for testing.

#Consequences

#If adopted:

• New service: A CAS server is added to the Kubernetes deployment (ArgoCD manifests, health checks, monitoring)
• Repository creation: New repositories would be configured with git-xet transfer agent settings (automatable in gitlab-root-actor)
• User tooling: Users need git-xet or huggingface_hub >= 0.32 — this should be documented and ideally bundled in any Themelio CLI or SDK
• GitLab UI: LFS file browsing in GitLab's web UI is unavailable for Xet-backed repos (same limitation as ADR-0003). The Themelio web UI would need to handle file previews via the CAS reconstruction API.
• Storage topology: MinIO stores both LFS objects (from ADR-0003 repos) and xorbs (from Xet repos), potentially in separate buckets
• Monitoring: CAS server metrics (upload/download throughput, dedup ratio, quota utilization) become important operational signals

#If not adopted:

• The LFS proxy from ADR-0003 remains the sole storage enforcement mechanism
• No chunk-level deduplication — each file version stored in full
• Higher storage consumption and bandwidth usage for iterative ML workflows
• Potential divergence from the broader ML ecosystem as Xet adoption grows

#Open Questions

1. Timeline: When does the platform's scale and user demand justify the investment? A useful signal would be observing high storage waste from duplicate model weights across repositories.
2. Build vs. wait: Hugging Face may open-source their CAS server or a community implementation may emerge. The IETF standardization effort suggests they want ecosystem adoption, but there is no public commitment to open-sourcing the server.
3. Technology choice: TypeScript (team familiarity) vs. Rust (protocol alignment, performance). A prototype in TypeScript with performance-critical paths (hashing, shard parsing) delegated to native modules via xet-core WASM could be a pragmatic middle ground.
4. MinIO bucket structure: One bucket with hash-prefixed keys, or separate buckets per project? Affects dedup scope and operational simplicity.
5. Migration strategy: When Xet is ready, should existing LFS repositories be migrated (background re-chunking), or left on LFS permanently? Migration yields dedup benefits but adds operational complexity.

#References

• Xet Protocol Specification v1.0.0
• CAS API Documentation
• Upload Protocol
• Download Protocol
• Authentication and Authorization
• Xet Deduplication
• IETF Internet-Draft: draft-denis-xet-03
• xet-core (Rust reference implementation)
• zig-xet (Zig implementation)
• Migrating the Hub from Git LFS to Xet
• XetHub joins Hugging Face
• Frank Denis — Introduction to XET (Part 1)
• Git LFS Custom Transfers
• ADR-0003: LFS Storage Quota Enforcement