# 0070 — Platform-side security scanning of user repositories Status: Proposed. Detailed design reference: DD-0056 (pending publication). Date: 22 October 2026. Deciders: TBD. Relates to: - [0012](0012-job-deployment-management-system.md) and [0013](0013-actions-yaml-automation-system.md): the deploy gate is inserted into the `k8s-root-actor` dispatch route defined there. - [0014](0014-gitlab-ci-runner-ec2-instance.md): that runner served the platform monorepo and is not used for user repositories. - [0025](0025-scoped-istio-authorization-policies.md): its caller allow-list for `gitlab-root-actor` exists only in the archived legacy GitOps; see [Current state](#current-state). - [0041](0041-gitlab-waldur-services.md): GitLab is an Argo CD service, which is where the system hook must be managed. - [0043](0043-themelio-compat-service.md): the receiver, worker and gates extend the retained Themelio (`themelio-compat`), whose GitLab root token is a dev-box exception. - [0048](0048-shared-development-infrastructure.md): the Valkey used for the queue is the shared `pharos-infra` instance. - The SecOps architecture proposal (branch `docs/adr-00xx-secops`, not yet numbered): this decision supersedes its Component 1 (CI-runner-based scanning). ## Context Themelio users push model, dataset and Space repositories to the platform's self-hosted GitLab and run them on shared infrastructure, on Kubernetes or on Daedalus through Slurm. Many model formats (`.pkl`, `.pt`, `.pth`, `.bin`, `.ckpt`, `.joblib`) are pickle-based and execute embedded code when loaded. A malicious model that reaches an inference pod therefore runs code inside the shared cluster. Repositories can also carry conventional malware and credentials committed by mistake. **Proofs of concept.** The scanner image is based on `python:3.12-slim` and pins picklescan 1.0.5, modelscan 0.8.8, fickling 0.1.12 and modelaudit 0.2.52, together with ClamAV. TruffleHog was added for secrets. On real samples, the four model scanners caught a malicious pickle and modelaudit caught a malformed safetensors file. The system-hooks proof of concept on the local GitLab showed that: - every push, tag push and branch deletion reached the receiver with the correct project, ref and SHA within a second; - a commit status written through the API is shown on the commit like a pipeline result, without a runner; - a slow or stopped receiver does not affect the user's push; - GitLab waits 10 seconds for a reply and does not retry a failed delivery, so six pushes made while the receiver was down were lost; - a push that touches more than three refs fires no push event, while a `repository_update` event lists all refs (tested with 15 branches and 10 tags) only if `repository_update_events` is enabled separately on the hook; - the hook stayed active after nine consecutive failures; - a clean clone at the exact SHA retrieved the LFS content, and picklescan and fickling caught a malicious pickle pushed through LFS. ### Current problems 1. Nothing inspects pushed content, and no service consults a security result before a launch or before a repository is made public. 2. A scan driven from the user's own repository can be skipped or forged by its owner, who controls `.gitlab-ci.yml`, can push with `ci.skip` and holds an API token. 3. User repositories have no CI runner, and providing one raises a separate policy question on quota, abuse and cost. ### Requirements 1. The repository owner cannot trigger, skip, alter or forge a scan result. 2. Scanning does not block or delay a push. 3. Events that GitLab fails to deliver are recovered. 4. The component that scans user content holds neither the GitLab root token nor Kubernetes API access. 5. A single enforcement point covers every deployment type. ## Decision Scans are triggered by GitLab system hooks, outside the user's repository. GitLab sends an instance-wide system hook on every push to a receiver operated by Themelio. A Themelio worker clones the exact commit and scans it. The result is stored through `gitlab-root-actor`, where the gates read it. User CI plays no part, so nothing the repository owner controls takes part in triggering, scanning or recording. ```mermaid flowchart LR U([User]) -- "git push, API or UI commit" --> GL[GitLab] GL -- "1. system hook: push, tag_push, repository_update" --> R[receiver] R -- "2. enqueue, reply at once" --> Q[(Valkey queue)] Q -- "3. job" --> W[scanner worker] W -- "3. clone exact SHA with LFS" --> GL W -- "4. POST /mlsecops/verdict" --> GRA[gitlab-root-actor] GRA -- "4. project custom attributes" --> GL W -. "4. commit status, display only" .-> GL REC[reconciliation job] -. "5. commits without a verdict" .-> Q KRA[k8s-root-actor deploy gate] -- "6. GET /mlsecops/verdict/:id" --> GRA WEB[web visibility gate] -- "6. read verdict" --> GRA ``` 1. **Trigger.** A GitLab administrator registers one instance-level system hook with `push_events`, `tag_push_events` and `repository_update_events` enabled, and a secret token. `repository_update_events` is mandatory, because without it a push that touches more than three refs produces no event. 2. **Receiver and queue.** A new receiver service beside the retained Themelio checks the hook token, places the event on a Valkey queue and replies immediately, within GitLab's 10-second timeout. 3. **Worker.** A new scanner worker, written in TypeScript on Node.js, consumes the queue and ignores projects outside the six resource groups. It clones the SHA named in the event, with LFS, and runs picklescan, modelscan, fickling, modelaudit, ClamAV and TruffleHog as external commands on a Python 3.12 runtime. The result is PASS, WARN or FAIL. The worker runs as a separate service with network access only to GitLab, no Kubernetes API access, no GitLab root credentials and resource limits. 4. **Verdict store.** The worker posts the verdict to `POST /mlsecops/verdict` on `gitlab-root-actor`. The route accepts a dedicated `MLSECOPS_WORKER_INTERNAL_TOKEN`, separate from `GITLAB_ROOT_ACTOR_INTERNAL_TOKEN` so that each token can be rotated on its own. `gitlab-root-actor` stores the verdict as the project custom attributes `mlsecops_verdict` and `mlsecops_verdict_meta`; the latter holds the commit SHA, a summary, the scanner versions and the time. Gates read only these attributes. The worker also writes a commit status on the scanned SHA for display. 5. **Reconciliation.** A periodic job finds commits without a verdict and queues them. It recovers lost deliveries and backfills existing repositories. 6. **Gates.** Before dispatching an action on a model repository, `k8s-root-actor` reads the verdict through `GET /mlsecops/verdict/:projectId`. It blocks on FAIL, and on a missing verdict in `enforce` mode; `permissive` and `audit` modes are also supported. Every deployment type passes through this route, so the gate covers Kubernetes deployments, inference services and Slurm jobs. `web` applies the same verdict when a repository is made public. ## Consequences Positive: - The repository owner cannot bypass or forge the scan. - User repositories need no CI runners and no CI quota policy. - The scanner image and scan logic from the first proof of concept are reused. - One gate in `k8s-root-actor` covers every deployment type. Negative: - Four new components to build and operate: receiver, queue consumer, worker and reconciliation job. - GitLab does not retry hook deliveries, so complete coverage depends on the reconciliation job. - `gitlab-root-actor`, which holds the root token, gains a write route and two new callers, the worker and `k8s-root-actor`. `themelio-compat` has no caller allow-list today, so one must be added rather than extended. - `k8s-root-actor` gains its first outbound dependency, and launches depend on `gitlab-root-actor` being available. Neutral: - The verdict is stored in GitLab, so no database or schema is added. - The CI pipeline and job-token verification of the first proof of concept are retired. ### Migration No existing infrastructure is migrated. Repositories created before scanning are covered by the reconciliation backfill. The proof-of-concept artefacts are removed. Credentials among them are revoked, not only deleted. ## Open questions ### Q1: Commit status account Which account writes commit statuses, with which role and on which projects? ### Q2: Reconciliation and worker isolation How does the reconciliation job identify commits without a verdict, and how often does it run? What isolates the worker: namespace, network controls and ephemeral storage for 50 GiB files? Placing the worker in `themelio-compat` would admit it to `gitlab-root-actor` under the namespace-wide NetworkPolicy. ### Q3: Gate mode and rollout Which mode does the gate start in, when does it switch to `enforce`, and how are false positives overridden? ### Q4: Enforcement scope of the gates The worker scans every push across all six resource groups (step 3). The deploy gate as described, however, enforces only on model repositories (step 6), and the visibility gate's scope is left unspecified. Should deploy and visibility enforcement extend to datasets, Spaces and code repositories, or is scanning them for now only defence in depth and future readiness while enforcement stays model-only? ### Q5: Cost and rescans Is scan compute platform overhead or charged to project quota? When are repositories rescanned after a scanner or signature update? ### Q6: Queue durability The queue uses the shared Valkey, while the platform's durable asynchronous hand-offs use RabbitMQ with confirmed delivery. Which broker carries scan jobs, and what are its persistence, deduplication, retry and dead-letter rules? ## References - GitLab: [System hooks](https://docs.gitlab.com/administration/system_hooks/) · [System hooks API](https://docs.gitlab.com/api/system_hooks/) · [Webhook limits](https://docs.gitlab.com/user/project/integrations/webhooks/) · [Commit statuses API](https://docs.gitlab.com/api/commits/#set-the-pipeline-status-of-a-commit) · [Custom attributes API](https://docs.gitlab.com/api/custom_attributes/) - Scanners: [picklescan](https://github.com/mmaitre314/picklescan) · [fickling](https://github.com/trailofbits/fickling) · [ModelScan](https://github.com/protectai/modelscan) · [ModelAudit](https://github.com/promptfoo/modelaudit) · [ClamAV](https://www.clamav.net/) · [TruffleHog](https://github.com/trufflesecurity/trufflehog) · [safetensors](https://github.com/huggingface/safetensors) - Hugging Face: [Malware scanning](https://huggingface.co/docs/hub/security-malware) · [Pickle scanning](https://huggingface.co/docs/hub/security-pickle)