#ADR-0001: LFS Storage Quota Enforcement via Custom Proxy

#Status

Proposed

#Context

Themelio Projects are the primary organizational unit on the platform. Each Themelio Project maps to a GitLab group, with subgroups for repository types (models/, datasets/, spaces/). Institutions are allocated storage quotas (managed via Waldur) that must be enforced across all repository types combined per Themelio Project.

The problem: GitLab CE has no per-group storage quota enforcement. The repository_size_limit setting at group level is a Premium/Ultimate-only feature, and the Groups API exposes no writable quota parameter in CE. The only globally available limit is an instance-wide repository size cap, which cannot differentiate between projects.

LFS is the dominant storage consumer — model weights and datasets routinely reach tens of GBs. Current LFS objects are stored in MinIO (git-lfs S3 bucket) via GitLab's built-in object storage integration. Without enforcement, any user can push unlimited LFS data, making fair multi-tenant resource allocation impossible.

We need a mechanism to:

1. Track cumulative LFS storage per Themelio Project (GitLab group) across all its repositories
2. Enforce a configurable quota ceiling per Themelio Project
3. Reject uploads before data transfer begins — model weights and datasets routinely reach hundreds of GBs to TBs. Post-upload rejection is unacceptable: a user uploading a 200 GB model over several hours only to be told they exceeded quota wastes their time, bandwidth, and temporarily consumes storage for orphaned objects. Pre-upload enforcement is a hard requirement.
4. Provide a clear error message explaining the quota violation and remaining capacity
5. Integrate with Waldur as the source of truth for quota allocations

#How GitLab CE Handles LFS Today

Understanding the current flow is critical for evaluating options:

1. Client calls the Git LFS Batch API (POST /objects/batch) to request upload authorization
2. GitLab Workhorse proxies the batch request to Rails
3. Rails returns upload URLs (either direct-to-object-storage or via Workhorse)
4. Client uploads LFS objects to the provided URLs
5. Workhorse calls LfsStorageController#upload_authorize then upload_finalize
6. Rails creates LfsObject records and associates them with the project

The Git LFS client discovers its server endpoint through a precedence chain:

1. remote.{name}.lfsurl (per-remote git config)
2. lfs.url (global git config)
3. .lfsconfig file (committed to the repository)
4. Default: appends .git/info/lfs to the Git remote URL

#Decision Constraint: Pre-Upload Enforcement Required

Given the scale of LFS objects on the platform (TB-scale model weights and datasets), enforcement must happen before data transfer begins. This is a hard requirement derived from FR-REP-02 (LFS support up to TB scale) and FR-QTA-01 (storage quota lifecycle). A user should never complete a multi-hour upload only to have it rejected.

This constraint eliminates options that rely solely on post-upload validation (server hooks in pre-receive run after LFS objects are already uploaded to object storage). Options B, C, and D are documented below for completeness, but only Options A and E satisfy the pre-upload enforcement requirement.

#Options

#Option A: Custom LFS Proxy Server (Intercept at Batch API Level)

Deploy a standalone service that implements the Git LFS Batch API and sits between Git LFS clients and the actual storage backend (MinIO). The proxy intercepts upload requests, checks cumulative usage against the Themelio Project quota, and either forwards the request or rejects it with a quota-exceeded error.

How it works:

• Each repository's .lfsconfig (or lfs.url) points to the proxy instead of GitLab
• On upload batch requests, the proxy:
  1. Identifies the Themelio Project from the request path (e.g., models/org-name/repo-name)
  2. Queries current usage (from its own tracking DB or GitLab API statistics endpoint)
  3. Compares cumulative usage + incoming object sizes against the Waldur-sourced quota
  4. Returns per-object errors (HTTP 200 with error field) for objects that would exceed quota, or forwards to MinIO
• On download batch requests, the proxy passes through to MinIO (or generates pre-signed URLs)
• GitLab's built-in LFS support must be disabled per-project (otherwise GitLab can't verify external LFS objects and pushes fail)

Candidate implementations:

Pros:

• Full control over quota logic — can enforce per-project, per-repo-type, or combined quotas
• Quota checked before data transfer begins (at Batch API level), saving bandwidth
• Can integrate directly with Waldur API for dynamic quota lookups
• Storage backend can be the same MinIO instance (or a different one)
• The Batch API spec supports per-object error responses, so partial batches can succeed
• Independent of GitLab version — works with any GitLab CE release

Cons:

• GitLab LFS must be disabled per-project — GitLab can't verify objects stored externally, so pushes fail if LFS is enabled alongside .lfsconfig pointing elsewhere
• Loses GitLab UI integration for LFS (file browsing, size display in the web UI)
• New service to deploy, monitor, and maintain
• .lfsconfig must be configured per-repository (can be automated at repo creation time)
• Authentication must be handled by the proxy (can delegate to GitLab tokens or implement its own)
• LFS objects are no longer visible in GitLab's storage statistics

#Option B: GitLab Server Hooks (Pre-Receive Validation)

Use GitLab's server hooks to run a custom pre-receive script that validates LFS pushes against quotas. Server hooks are available in GitLab CE and can be applied globally or per-repository.

How it works:

• A global pre-receive hook script is placed in /var/opt/gitlab/gitaly/custom_hooks/pre-receive.d/
• On every git push, the hook:
  1. Reads stdin for ref updates
  2. Inspects the pushed objects for LFS pointers (parsing .gitattributes and pointer files)
  3. Sums up LFS object sizes from the push
  4. Queries current group storage via GitLab API (GET /api/v4/groups/:id?statistics=true returns lfs_objects_size)
  5. Compares against the Waldur-sourced quota
  6. Exits non-zero with GL-HOOK-ERR: prefixed message to reject the push

Available environment variables in hooks:

• GL_PROJECT_PATH — e.g., models/org-name/repo-name (can extract the Themelio Project)
• GL_USERNAME, GL_ID, GL_REPOSITORY, GL_PROTOCOL
• GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES

Pros:

• No architectural changes — LFS objects continue flowing through GitLab's standard pipeline to MinIO
• GitLab UI continues to show LFS files, sizes, and storage statistics
• No client-side configuration changes (no .lfsconfig needed)
• No new services to deploy (hook runs inside Gitaly)
• Authentication is handled by GitLab as usual

Cons:

• Validation happens after LFS upload — by the time the pre-receive hook runs, LFS objects have already been uploaded to storage. The push is rejected, but the orphaned LFS objects remain in MinIO until GitLab's housekeeping garbage-collects them (this can be slow or unreliable)
• Parsing LFS pointers from stdin in a shell/script during pre-receive is fragile — the hook receives raw ref updates, not file contents, so determining LFS object sizes requires git cat-file calls against the quarantine object directory
• Race conditions: concurrent pushes to different repos in the same project could both pass quota checks before either is finalized
• Hook execution adds latency to every push (even non-LFS pushes), unless the hook can cheaply detect the absence of LFS objects
• Reliance on the GitLab statistics API for current usage introduces a dependency on eventual consistency (statistics may lag)
• Global hooks apply to all repositories including those outside Themelio Projects — needs filtering logic

#Option C: Application-Level Enforcement in Themelio Web Service

Enforce quotas at the Themelio application layer (Next.js API routes or a new microservice) rather than at the Git/LFS protocol level. The web service already mediates repository creation and could be extended to track and enforce storage limits.

How it works:

• Track LFS storage per Themelio Project in a dedicated database (or query GitLab's statistics API periodically)
• Before repository creation, check if the project has remaining quota
• Expose a quota dashboard in the Themelio web UI
• Periodically reconcile actual storage usage with GitLab statistics
• For enforcement: either set GitLab's per-project repository_size_limit via the Projects API (available in CE for individual projects), or flag over-quota projects for admin intervention

Pros:

• No changes to the Git/LFS pipeline
• Quota visibility directly in the Themelio UI
• Can leverage existing Waldur integration for quota source of truth
• Simpler to implement than protocol-level interception
• Can enforce soft limits (warnings) in addition to hard limits

Cons:

• Cannot truly block LFS uploads — GitLab CE has no per-project LFS size limit API, only a total repository_size_limit which includes non-LFS data and is documented as not applying to LFS pushes specifically
• Enforcement is advisory or delayed — a user can push exceeding LFS objects and the system can only react after the fact
• Requires periodic reconciliation to stay accurate
• Doesn't prevent a single large push from blowing through the quota
• Per-project repository_size_limit via the Projects API may not cover LFS objects (GitLab docs state: "This setting does not apply when pushing Git LFS objects")

#Option D: Hybrid — Server Hook + Application Dashboard

Combine Option B (server hooks for hard enforcement) with Option C (application-level tracking and UI).

How it works:

• Global pre-receive hook performs quota validation on push (hard enforcement)
• Themelio web service tracks usage and provides a quota dashboard
• Waldur remains the quota source of truth
• The web service periodically syncs usage data from GitLab statistics API
• Users see their quota status in the UI before pushing

Pros:

• Hard enforcement at the Git protocol level prevents quota violations
• User-facing dashboard provides quota visibility and avoids surprise rejections
• Waldur integration is cleanly separated (web service queries Waldur, hook queries a local cache)
• No client-side configuration changes

Cons:

• Inherits the core cons of Option B: post-upload rejection, orphaned LFS objects, pre-receive parsing complexity
• Two systems to maintain (hook + dashboard)
• Race condition risk from Option B remains
• More moving parts than any single option

#Option E: Custom LFS Proxy + Application Dashboard (Hybrid of A + C)

Deploy a custom LFS proxy (Option A) for hard quota enforcement at the Batch API level, combined with a Themelio web dashboard (Option C) for quota visibility.

How it works:

• LFS proxy enforces quotas before data transfer (pre-upload rejection)
• Themelio web service provides quota dashboard and Waldur integration
• Proxy queries a shared quota state (database or Waldur API) for limits
• Web service syncs usage from the proxy's tracking data

Pros:

• Strongest enforcement — rejects before upload, saving bandwidth and preventing orphaned objects
• Full quota visibility in Themelio UI
• Clean Waldur integration for dynamic quota management
• Batch API per-object errors give users precise feedback on which files exceed quota
• No race conditions if the proxy serializes quota checks per project (or uses optimistic locking)

Cons:

• Most complex option — new LFS proxy service + dashboard + shared quota state
• GitLab LFS disabled per-project — loses web UI LFS browsing
• .lfsconfig management per-repository
• Two new components to deploy and maintain

#Comparison Matrix

#Consequences

The chosen option will affect:

• Repository creation flow: Options A/E require .lfsconfig to be set in every new repository, which must be automated in the gitlab-root-actor or web service
• Deployment topology: Options A/E add a new service to the Kubernetes deployment (ArgoCD manifests, health checks, monitoring)
• User experience: Options A/E lose GitLab web UI LFS browsing; Options B/D may show confusing post-upload rejections
• Waldur integration: All options need a mechanism to fetch quota allocations from Waldur — this can be pull-based (periodic sync) or push-based (Waldur webhook on quota change)
• Migration: Existing repositories with LFS objects need their usage accounted for in the new quota tracking system

#Open Questions

1. What is the acceptable failure mode? Resolved: Pre-upload enforcement is required. Post-upload rejection is unacceptable given TB-scale LFS objects. This narrows viable options to A and E.
2. Is GitLab web UI LFS browsing important? If users primarily interact through git CLI and the Themelio web UI, losing GitLab's native LFS file browser may be acceptable. Both viable options (A, E) disable GitLab's LFS integration.
3. Quota granularity: Should quotas be per Themelio Project (combined across all repo types), or should there be separate sub-quotas per type (models, datasets, spaces)?
4. Quota source of truth latency: Is eventual consistency acceptable (periodic sync from Waldur), or must quota checks be real-time against Waldur?
5. Existing LFS data: How should current LFS usage be accounted for when quotas are first enabled?
6. Proxy technology choice: Should we extend an existing LFS server (Giftless) or build a custom service? Giftless is Python-based and pluggable but lacks built-in quota support. A custom service in Node.js/TypeScript would align with the existing stack.

#References

• Git LFS Batch API Specification
• Git LFS Server Discovery
• Git LFS Server Implementations
• GitLab CE LFS Administration
• GitLab Server Hooks
• GitLab Account and Limit Settings
• GitLab Groups API
• Giftless — Pluggable Git LFS Server (Python)
• git-lfs-s3-proxy

Page source

Source snapshot · Revision f928834c8.