For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0070-platform-side-security-scanning-of-user-repositories.md.

0070 — Platform-side security scanning of user repositories

Status: Proposed. Detailed design reference: DD-0056 (pending publication). Date: 22 October 2026. Deciders: TBD.

Relates to:

  • 0012 and 0013: the deploy gate is inserted into the k8s-root-actor dispatch route defined there.
  • 0014: that runner served the platform monorepo and is not used for user repositories.
  • 0025: its caller allow-list for gitlab-root-actor exists only in the archived legacy GitOps; see Current state.
  • 0041: GitLab is an Argo CD service, which is where the system hook must be managed.
  • 0043: the receiver, worker and gates extend the retained Themelio (themelio-compat), whose GitLab root token is a dev-box exception.
  • 0048: the Valkey used for the queue is the shared pharos-infra instance.
  • The SecOps architecture proposal (branch docs/adr-00xx-secops, not yet numbered): this decision supersedes its Component 1 (CI-runner-based scanning).

Context

Themelio users push model, dataset and Space repositories to the platform's self-hosted GitLab and run them on shared infrastructure, on Kubernetes or on Daedalus through Slurm. Many model formats (.pkl, .pt, .pth, .bin, .ckpt, .joblib) are pickle-based and execute embedded code when loaded. A malicious model that reaches an inference pod therefore runs code inside the shared cluster. Repositories can also carry conventional malware and credentials committed by mistake.

Proofs of concept. The scanner image is based on python:3.12-slim and pins picklescan 1.0.5, modelscan 0.8.8, fickling 0.1.12 and modelaudit 0.2.52, together with ClamAV. TruffleHog was added for secrets. On real samples, the four model scanners caught a malicious pickle and modelaudit caught a malformed safetensors file. The system-hooks proof of concept on the local GitLab showed that:

  • every push, tag push and branch deletion reached the receiver with the correct project, ref and SHA within a second;
  • a commit status written through the API is shown on the commit like a pipeline result, without a runner;
  • a slow or stopped receiver does not affect the user's push;
  • GitLab waits 10 seconds for a reply and does not retry a failed delivery, so six pushes made while the receiver was down were lost;
  • a push that touches more than three refs fires no push event, while a repository_update event lists all refs (tested with 15 branches and 10 tags) only if repository_update_events is enabled separately on the hook;
  • the hook stayed active after nine consecutive failures;
  • a clean clone at the exact SHA retrieved the LFS content, and picklescan and fickling caught a malicious pickle pushed through LFS.

Current problems

  1. Nothing inspects pushed content, and no service consults a security result before a launch or before a repository is made public.
  2. A scan driven from the user's own repository can be skipped or forged by its owner, who controls .gitlab-ci.yml, can push with ci.skip and holds an API token.
  3. User repositories have no CI runner, and providing one raises a separate policy question on quota, abuse and cost.

Requirements

  1. The repository owner cannot trigger, skip, alter or forge a scan result.
  2. Scanning does not block or delay a push.
  3. Events that GitLab fails to deliver are recovered.
  4. The component that scans user content holds neither the GitLab root token nor Kubernetes API access.
  5. A single enforcement point covers every deployment type.

Decision

Scans are triggered by GitLab system hooks, outside the user's repository. GitLab sends an instance-wide system hook on every push to a receiver operated by Themelio. A Themelio worker clones the exact commit and scans it. The result is stored through gitlab-root-actor, where the gates read it. User CI plays no part, so nothing the repository owner controls takes part in triggering, scanning or recording.

  1. Trigger. A GitLab administrator registers one instance-level system hook with push_events, tag_push_events and repository_update_events enabled, and a secret token. repository_update_events is mandatory, because without it a push that touches more than three refs produces no event.
  2. Receiver and queue. A new receiver service beside the retained Themelio checks the hook token, places the event on a Valkey queue and replies immediately, within GitLab's 10-second timeout.
  3. Worker. A new scanner worker, written in TypeScript on Node.js, consumes the queue and ignores projects outside the six resource groups. It clones the SHA named in the event, with LFS, and runs picklescan, modelscan, fickling, modelaudit, ClamAV and TruffleHog as external commands on a Python 3.12 runtime. The result is PASS, WARN or FAIL. The worker runs as a separate service with network access only to GitLab, no Kubernetes API access, no GitLab root credentials and resource limits.
  4. Verdict store. The worker posts the verdict to POST /mlsecops/verdict on gitlab-root-actor. The route accepts a dedicated MLSECOPS_WORKER_INTERNAL_TOKEN, separate from GITLAB_ROOT_ACTOR_INTERNAL_TOKEN so that each token can be rotated on its own. gitlab-root-actor stores the verdict as the project custom attributes mlsecops_verdict and mlsecops_verdict_meta; the latter holds the commit SHA, a summary, the scanner versions and the time. Gates read only these attributes. The worker also writes a commit status on the scanned SHA for display.
  5. Reconciliation. A periodic job finds commits without a verdict and queues them. It recovers lost deliveries and backfills existing repositories.
  6. Gates. Before dispatching an action on a model repository, k8s-root-actor reads the verdict through GET /mlsecops/verdict/:projectId. It blocks on FAIL, and on a missing verdict in enforce mode; permissive and audit modes are also supported. Every deployment type passes through this route, so the gate covers Kubernetes deployments, inference services and Slurm jobs. web applies the same verdict when a repository is made public.

Consequences

Positive:

  • The repository owner cannot bypass or forge the scan.
  • User repositories need no CI runners and no CI quota policy.
  • The scanner image and scan logic from the first proof of concept are reused.
  • One gate in k8s-root-actor covers every deployment type.

Negative:

  • Four new components to build and operate: receiver, queue consumer, worker and reconciliation job.
  • GitLab does not retry hook deliveries, so complete coverage depends on the reconciliation job.
  • gitlab-root-actor, which holds the root token, gains a write route and two new callers, the worker and k8s-root-actor. themelio-compat has no caller allow-list today, so one must be added rather than extended.
  • k8s-root-actor gains its first outbound dependency, and launches depend on gitlab-root-actor being available.

Neutral:

  • The verdict is stored in GitLab, so no database or schema is added.
  • The CI pipeline and job-token verification of the first proof of concept are retired.

Migration

No existing infrastructure is migrated. Repositories created before scanning are covered by the reconciliation backfill. The proof-of-concept artefacts are removed. Credentials among them are revoked, not only deleted.

Open questions

Q1: Commit status account

Which account writes commit statuses, with which role and on which projects?

Q2: Reconciliation and worker isolation

How does the reconciliation job identify commits without a verdict, and how often does it run? What isolates the worker: namespace, network controls and ephemeral storage for 50 GiB files? Placing the worker in themelio-compat would admit it to gitlab-root-actor under the namespace-wide NetworkPolicy.

Q3: Gate mode and rollout

Which mode does the gate start in, when does it switch to enforce, and how are false positives overridden?

Q4: Enforcement scope of the gates

The worker scans every push across all six resource groups (step 3). The deploy gate as described, however, enforces only on model repositories (step 6), and the visibility gate's scope is left unspecified. Should deploy and visibility enforcement extend to datasets, Spaces and code repositories, or is scanning them for now only defence in depth and future readiness while enforcement stays model-only?

Q5: Cost and rescans

Is scan compute platform overhead or charged to project quota? When are repositories rescanned after a scanner or signature update?

Q6: Queue durability

The queue uses the shared Valkey, while the platform's durable asynchronous hand-offs use RabbitMQ with confirmed delivery. Which broker carries scan jobs, and what are its persistence, deduplication, retry and dead-letter rules?

References

Page source

Source snapshot · Revision f928834c8.