# ADR-0001: LFS Storage Quota Enforcement via Custom Proxy

## Status

Proposed

## Context

Themelio Projects are the primary organizational unit on the platform. Each Themelio Project maps to a GitLab group, with subgroups for repository types (`models/`, `datasets/`, `spaces/`). Institutions are allocated storage quotas (managed via Waldur) that must be enforced across all repository types combined per Themelio Project.

The problem: **GitLab CE has no per-group storage quota enforcement**. The `repository_size_limit` setting at group level is a Premium/Ultimate-only feature, and the Groups API exposes no writable quota parameter in CE. The only globally available limit is an instance-wide repository size cap, which cannot differentiate between projects.

LFS is the dominant storage consumer — model weights and datasets routinely reach tens of GBs. Current LFS objects are stored in MinIO (`git-lfs` S3 bucket) via GitLab's built-in object storage integration. Without enforcement, any user can push unlimited LFS data, making fair multi-tenant resource allocation impossible.

We need a mechanism to:

1. Track cumulative LFS storage per Themelio Project (GitLab group) across all its repositories
2. Enforce a configurable quota ceiling per Themelio Project
3. **Reject uploads before data transfer begins** — model weights and datasets routinely reach hundreds of GBs to TBs. Post-upload rejection is unacceptable: a user uploading a 200 GB model over several hours only to be told they exceeded quota wastes their time, bandwidth, and temporarily consumes storage for orphaned objects. Pre-upload enforcement is a hard requirement.
4. Provide a clear error message explaining the quota violation and remaining capacity
5. Integrate with Waldur as the source of truth for quota allocations

### How GitLab CE Handles LFS Today

Understanding the current flow is critical for evaluating options:

1. **Client** calls the Git LFS **Batch API** (`POST /objects/batch`) to request upload authorization
2. **GitLab Workhorse** proxies the batch request to **Rails**
3. **Rails** returns upload URLs (either direct-to-object-storage or via Workhorse)
4. **Client** uploads LFS objects to the provided URLs
5. **Workhorse** calls `LfsStorageController#upload_authorize` then `upload_finalize`
6. **Rails** creates `LfsObject` records and associates them with the project

The Git LFS client discovers its server endpoint through a precedence chain:
1. `remote.{name}.lfsurl` (per-remote git config)
2. `lfs.url` (global git config)
3. `.lfsconfig` file (committed to the repository)
4. Default: appends `.git/info/lfs` to the Git remote URL

### Decision Constraint: Pre-Upload Enforcement Required

Given the scale of LFS objects on the platform (TB-scale model weights and datasets), **enforcement must happen before data transfer begins**. This is a hard requirement derived from FR-REP-02 (LFS support up to TB scale) and FR-QTA-01 (storage quota lifecycle). A user should never complete a multi-hour upload only to have it rejected.

This constraint eliminates options that rely solely on post-upload validation (server hooks in pre-receive run after LFS objects are already uploaded to object storage). Options B, C, and D are documented below for completeness, but only Options A and E satisfy the pre-upload enforcement requirement.

## Options

### Option A: Custom LFS Proxy Server (Intercept at Batch API Level)

Deploy a standalone service that implements the [Git LFS Batch API](https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md) and sits between Git LFS clients and the actual storage backend (MinIO). The proxy intercepts upload requests, checks cumulative usage against the Themelio Project quota, and either forwards the request or rejects it with a quota-exceeded error.

**How it works:**

- Each repository's `.lfsconfig` (or `lfs.url`) points to the proxy instead of GitLab
- On `upload` batch requests, the proxy:
  1. Identifies the Themelio Project from the request path (e.g., `models/org-name/repo-name`)
  2. Queries current usage (from its own tracking DB or GitLab API `statistics` endpoint)
  3. Compares cumulative usage + incoming object sizes against the Waldur-sourced quota
  4. Returns per-object errors (HTTP 200 with `error` field) for objects that would exceed quota, or forwards to MinIO
- On `download` batch requests, the proxy passes through to MinIO (or generates pre-signed URLs)
- GitLab's built-in LFS support must be **disabled** per-project (otherwise GitLab can't verify external LFS objects and pushes fail)

**Candidate implementations:**

| Project | Language | Storage Backends | Notes |
|---------|----------|-----------------|-------|
| [Giftless](https://github.com/datopian/giftless) | Python | S3, GCS, Azure, local | Pluggable auth and backends, extensible |
| [git-lfs-s3-proxy](https://github.com/twilligon/git-lfs-s3-proxy) | Go | S3-compatible | Lightweight, direct S3 proxy |
| Custom service | Any | MinIO (S3) | Full control, can integrate Waldur natively |

**Pros:**
- Full control over quota logic — can enforce per-project, per-repo-type, or combined quotas
- Quota checked **before** data transfer begins (at Batch API level), saving bandwidth
- Can integrate directly with Waldur API for dynamic quota lookups
- Storage backend can be the same MinIO instance (or a different one)
- The Batch API spec supports per-object error responses, so partial batches can succeed
- Independent of GitLab version — works with any GitLab CE release

**Cons:**
- **GitLab LFS must be disabled** per-project — GitLab can't verify objects stored externally, so pushes fail if LFS is enabled alongside `.lfsconfig` pointing elsewhere
- Loses GitLab UI integration for LFS (file browsing, size display in the web UI)
- New service to deploy, monitor, and maintain
- `.lfsconfig` must be configured per-repository (can be automated at repo creation time)
- Authentication must be handled by the proxy (can delegate to GitLab tokens or implement its own)
- LFS objects are no longer visible in GitLab's storage statistics

### Option B: GitLab Server Hooks (Pre-Receive Validation)

Use GitLab's [server hooks](https://docs.gitlab.com/administration/server_hooks/) to run a custom pre-receive script that validates LFS pushes against quotas. Server hooks are available in GitLab CE and can be applied globally or per-repository.

**How it works:**

- A global pre-receive hook script is placed in `/var/opt/gitlab/gitaly/custom_hooks/pre-receive.d/`
- On every `git push`, the hook:
  1. Reads stdin for ref updates
  2. Inspects the pushed objects for LFS pointers (parsing `.gitattributes` and pointer files)
  3. Sums up LFS object sizes from the push
  4. Queries current group storage via GitLab API (`GET /api/v4/groups/:id?statistics=true` returns `lfs_objects_size`)
  5. Compares against the Waldur-sourced quota
  6. Exits non-zero with `GL-HOOK-ERR:` prefixed message to reject the push

**Available environment variables in hooks:**
- `GL_PROJECT_PATH` — e.g., `models/org-name/repo-name` (can extract the Themelio Project)
- `GL_USERNAME`, `GL_ID`, `GL_REPOSITORY`, `GL_PROTOCOL`
- `GIT_OBJECT_DIRECTORY`, `GIT_ALTERNATE_OBJECT_DIRECTORIES`

**Pros:**
- No architectural changes — LFS objects continue flowing through GitLab's standard pipeline to MinIO
- GitLab UI continues to show LFS files, sizes, and storage statistics
- No client-side configuration changes (no `.lfsconfig` needed)
- No new services to deploy (hook runs inside Gitaly)
- Authentication is handled by GitLab as usual

**Cons:**
- **Validation happens after LFS upload** — by the time the pre-receive hook runs, LFS objects have already been uploaded to storage. The push is rejected, but the orphaned LFS objects remain in MinIO until GitLab's housekeeping garbage-collects them (this can be slow or unreliable)
- Parsing LFS pointers from `stdin` in a shell/script during pre-receive is fragile — the hook receives raw ref updates, not file contents, so determining LFS object sizes requires `git cat-file` calls against the quarantine object directory
- Race conditions: concurrent pushes to different repos in the same project could both pass quota checks before either is finalized
- Hook execution adds latency to every push (even non-LFS pushes), unless the hook can cheaply detect the absence of LFS objects
- Reliance on the GitLab statistics API for current usage introduces a dependency on eventual consistency (statistics may lag)
- Global hooks apply to all repositories including those outside Themelio Projects — needs filtering logic

### Option C: Application-Level Enforcement in Themelio Web Service

Enforce quotas at the Themelio application layer (Next.js API routes or a new microservice) rather than at the Git/LFS protocol level. The web service already mediates repository creation and could be extended to track and enforce storage limits.

**How it works:**

- Track LFS storage per Themelio Project in a dedicated database (or query GitLab's statistics API periodically)
- Before repository creation, check if the project has remaining quota
- Expose a quota dashboard in the Themelio web UI
- Periodically reconcile actual storage usage with GitLab statistics
- For enforcement: either set GitLab's per-project `repository_size_limit` via the Projects API (available in CE for individual projects), or flag over-quota projects for admin intervention

**Pros:**
- No changes to the Git/LFS pipeline
- Quota visibility directly in the Themelio UI
- Can leverage existing Waldur integration for quota source of truth
- Simpler to implement than protocol-level interception
- Can enforce soft limits (warnings) in addition to hard limits

**Cons:**
- **Cannot truly block LFS uploads** — GitLab CE has no per-project LFS size limit API, only a total `repository_size_limit` which includes non-LFS data and is documented as not applying to LFS pushes specifically
- Enforcement is advisory or delayed — a user can push exceeding LFS objects and the system can only react after the fact
- Requires periodic reconciliation to stay accurate
- Doesn't prevent a single large push from blowing through the quota
- Per-project `repository_size_limit` via the Projects API may not cover LFS objects (GitLab docs state: "This setting does not apply when pushing Git LFS objects")

### Option D: Hybrid — Server Hook + Application Dashboard

Combine Option B (server hooks for hard enforcement) with Option C (application-level tracking and UI).

**How it works:**

- Global pre-receive hook performs quota validation on push (hard enforcement)
- Themelio web service tracks usage and provides a quota dashboard
- Waldur remains the quota source of truth
- The web service periodically syncs usage data from GitLab statistics API
- Users see their quota status in the UI before pushing

**Pros:**
- Hard enforcement at the Git protocol level prevents quota violations
- User-facing dashboard provides quota visibility and avoids surprise rejections
- Waldur integration is cleanly separated (web service queries Waldur, hook queries a local cache)
- No client-side configuration changes

**Cons:**
- Inherits the core cons of Option B: post-upload rejection, orphaned LFS objects, pre-receive parsing complexity
- Two systems to maintain (hook + dashboard)
- Race condition risk from Option B remains
- More moving parts than any single option

### Option E: Custom LFS Proxy + Application Dashboard (Hybrid of A + C)

Deploy a custom LFS proxy (Option A) for hard quota enforcement at the Batch API level, combined with a Themelio web dashboard (Option C) for quota visibility.

**How it works:**

- LFS proxy enforces quotas before data transfer (pre-upload rejection)
- Themelio web service provides quota dashboard and Waldur integration
- Proxy queries a shared quota state (database or Waldur API) for limits
- Web service syncs usage from the proxy's tracking data

**Pros:**
- Strongest enforcement — rejects before upload, saving bandwidth and preventing orphaned objects
- Full quota visibility in Themelio UI
- Clean Waldur integration for dynamic quota management
- Batch API per-object errors give users precise feedback on which files exceed quota
- No race conditions if the proxy serializes quota checks per project (or uses optimistic locking)

**Cons:**
- Most complex option — new LFS proxy service + dashboard + shared quota state
- GitLab LFS disabled per-project — loses web UI LFS browsing
- `.lfsconfig` management per-repository
- Two new components to deploy and maintain

## Comparison Matrix

| Criterion | A: LFS Proxy | B: Server Hook | C: App-Level | D: Hook+Dashboard | E: Proxy+Dashboard |
|-----------|:---:|:---:|:---:|:---:|:---:|
| Hard enforcement (blocks uploads) | Pre-upload | Post-upload | No | Post-upload | Pre-upload |
| Orphaned objects on rejection | No | Yes | N/A | Yes | No |
| GitLab UI LFS integration preserved | No | Yes | Yes | Yes | No |
| New services to deploy | 1 | 0 | 0 | 0 | 1 |
| Client config changes needed | Yes (.lfsconfig) | No | No | No | Yes (.lfsconfig) |
| Race condition risk | Low | Medium | N/A | Medium | Low |
| Waldur integration complexity | Medium | Medium | Low | Medium | Medium |
| Per-push latency impact | Low (pre-upload check) | Medium (all pushes) | None | Medium (all pushes) | Low (pre-upload check) |
| Implementation effort | High | Medium | Low | Medium-High | High |
| Maintenance burden | Medium | Low | Low | Medium | Medium |

## Consequences

The chosen option will affect:

- **Repository creation flow**: Options A/E require `.lfsconfig` to be set in every new repository, which must be automated in the `gitlab-root-actor` or web service
- **Deployment topology**: Options A/E add a new service to the Kubernetes deployment (ArgoCD manifests, health checks, monitoring)
- **User experience**: Options A/E lose GitLab web UI LFS browsing; Options B/D may show confusing post-upload rejections
- **Waldur integration**: All options need a mechanism to fetch quota allocations from Waldur — this can be pull-based (periodic sync) or push-based (Waldur webhook on quota change)
- **Migration**: Existing repositories with LFS objects need their usage accounted for in the new quota tracking system

## Open Questions

1. ~~**What is the acceptable failure mode?**~~ **Resolved**: Pre-upload enforcement is required. Post-upload rejection is unacceptable given TB-scale LFS objects. This narrows viable options to A and E.
2. **Is GitLab web UI LFS browsing important?** If users primarily interact through `git` CLI and the Themelio web UI, losing GitLab's native LFS file browser may be acceptable. Both viable options (A, E) disable GitLab's LFS integration.
3. **Quota granularity**: Should quotas be per Themelio Project (combined across all repo types), or should there be separate sub-quotas per type (models, datasets, spaces)?
4. **Quota source of truth latency**: Is eventual consistency acceptable (periodic sync from Waldur), or must quota checks be real-time against Waldur?
5. **Existing LFS data**: How should current LFS usage be accounted for when quotas are first enabled?
6. **Proxy technology choice**: Should we extend an existing LFS server (Giftless) or build a custom service? Giftless is Python-based and pluggable but lacks built-in quota support. A custom service in Node.js/TypeScript would align with the existing stack.

## References

- [Git LFS Batch API Specification](https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md)
- [Git LFS Server Discovery](https://github.com/git-lfs/git-lfs/blob/main/docs/api/server-discovery.md)
- [Git LFS Server Implementations](https://github.com/git-lfs/git-lfs/wiki/Implementations)
- [GitLab CE LFS Administration](https://docs.gitlab.com/administration/lfs/)
- [GitLab Server Hooks](https://docs.gitlab.com/administration/server_hooks/)
- [GitLab Account and Limit Settings](https://docs.gitlab.com/administration/settings/account_and_limit_settings/)
- [GitLab Groups API](https://docs.gitlab.com/api/groups/)
- [Giftless — Pluggable Git LFS Server (Python)](https://github.com/datopian/giftless)
- [git-lfs-s3-proxy](https://github.com/twilligon/git-lfs-s3-proxy)