> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

> **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform.

# Architecture Overview

Themelio is an open-source ML software stack that gives research teams a single place to store models and datasets, deploy inference endpoints and interactive applications, and manage compute and HPC resources for running model training & finetuning, and other ML workflows — all backed by federated institutional authentication.

Under the hood, Themelio wraps GitLab CE as a repository storage and collaboration backend, adds a purpose-built web layer for ML workflows, and orchestrates workflows and deployments on Kubernetes and Slurm. Users never interact with GitLab directly; Themelio handles identity, access control, and API translation so that researchers see a familiar, Hugging Face-style experience.

This page covers the system's moving parts and how they fit together. The pages that follow go deeper into [authentication](/product/architecture/authentication.md), [resource types](/product/requirements/core.md#themelio-catalogue), and [networking](/product/architecture/networking.md).

## System Diagram

The diagram below shows the three main layers: the **platform** (application services running on Kubernetes), **Waldur** (resource management and provisioning), and **compute infrastructure** (HPC, GPU clusters, and storage).

```mermaid
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
graph TB
    %% ── Users ──────────────────────────────────────────
    User["ML Researcher / Data Scientist"]
    GitCli["Git Client (HTTPS · SSH)"]

    %% ── Platform (K8s Cluster) ─────────────────────────
    Ingress["Istio IngressGateway"]

    User -->|browser| Ingress
    GitCli -->|"PAT / SSH key"| Ingress

    subgraph Platform["Kubernetes Cluster"]

        PharosAAI["Pharos AAI\n(Keycloak · eduGAIN · eIDAS · OIDC)"]
        Valkey["Valkey\n(sessions ↔ tokens)"]

        Web["Themelio Web\n(Next.js)"]

        GitlabRA["gitlab-root-actor\n(Fastify)"]
        K8sRA["k8s-root-actor\n(Fastify)"]

        GitLab["GitLab CE (hidden)"]

        KServe["KServe"]
        Kubeflow["Kubeflow"]
        MLflow["MLflow"]
        OOD["Open OnDemand"]
        Grafana["Prometheus · Grafana\nLoki · Mimir · Tempo"]
    end

    Ingress --> Web
    Ingress -->|git smart HTTP| GitLab

    Web -->|OIDC| PharosAAI
    Web -->|session| Valkey
    Web --> GitlabRA
    Web --> K8sRA
    Web -->|"GraphQL · REST\n(impersonation token)"| GitLab

    GitlabRA -->|"REST (root PAT)"| GitLab
    GitlabRA -->|tokens| Valkey

    %% ── Waldur ─────────────────────────────────────────
    subgraph WaldurLayer["Waldur (Resource Management)"]
        Waldur["Proposals Module\nisolation · access control\nquota mgmt · billing"]
        WCloud["cloud-site-agent\n(planned)"]
        WHPC["hpc-site-agent\n(planned)"]
        Waldur --> WCloud & WHPC
    end

    KubeflowRA["kubeflow-root-actor\n(Fastify · planned)"]
    MLflowRA["mlflow-root-actor\n(Fastify · planned)"]

    WCloud --> KubeflowRA
    WCloud --> MLflowRA
    KubeflowRA -->|profile provisioning| Kubeflow
    MLflowRA -->|instance provisioning| MLflow
    WHPC -->|job submission · access provisioning| HPC

    %% ── Compute ────────────────────────────────────────
    subgraph Compute["Compute Land (Daedalus + Cloud)"]
        K8sGPU["K8s Cluster (GPUs)"]
        HPC["HPC (Slurm)"]
        Ceph["Ceph Storage (rook.io)"]
    end

    K8sRA --> K8sGPU
    KServe --> K8sGPU
    Kubeflow --> K8sGPU
    HPC --> OOD
    WHPC -->|infrastructure provisioning| OOD

    %% ── Offerings ──────────────────────────────────────
    EuroHPC["EuroHPC"] --> Compute
    National["National"] --> Compute
    GRNET["GRNET"] --> Compute
    StorageOff["Storage Offering"] --> Ceph
```

## Services

Themelio runs as a bundle of cooperating services, each with a clear responsibility boundary.

### Themelio Web

The primary application — a Next.js 15+ app that serves both the UI and API routes. It handles everything a user sees and touches:

- **UI**: React 19 with server-side rendering. Catalogue pages, file browsers, deployment dashboards, settings.
- **API routes**: Organized by resource type (models, datasets, spaces, repos, recipes, upskilling materials). These routes translate between the Themelio API and GitLab's GraphQL and REST APIs.
- **Authentication**: NextAuth v5 with [OIDC](/terms/oidc.md). Sessions stored in [Valkey](/terms/valkey.md).
- **Middleware**: Intercepts every authenticated request, retrieves the user's GitLab [impersonation token](/terms/impersonation-token.md) from [Valkey](/terms/valkey.md), and injects it as a `PRIVATE-TOKEN` header. Downstream code never handles raw credentials.

The API routes live inside the same Next.js process, avoiding cross-service latency for user-facing operations.

### Cloud Site Agent _(planned)_

A Waldur site agent plugin that provisions cloud-based ML platform resources when a project proposal is accepted:

- **Kubeflow resources**: Delegates profile and namespace provisioning to the Kubeflow Root Actor.
- **MLflow resources**: Delegates tenant and workspace provisioning to the MLflow Root Actor.
- **Order lifecycle**: Handles order processing, membership synchronization, and usage reporting via the Waldur site agent framework.

The cloud site agent does not hold privileged credentials itself — it delegates all privileged operations to the Kubeflow and MLflow root actors, maintaining the same separation of concerns used across the platform.

### HPC Site Agent _(planned)_

A Waldur site agent plugin that provisions HPC infrastructure to give users access to Slurm-based compute resources:

- **Slurm access**: Provisions user accounts and project allocations on HPC clusters.
- **Open OnDemand**: Configures interactive compute sessions (Jupyter, RStudio, virtual desktops) for project members.
- **Storage mounts**: Sets up shared filesystem access between HPC nodes and project storage.
- **Order lifecycle**: Handles order processing, membership synchronization, and usage reporting via the Waldur site agent framework.

### GitLab Root Actor

A Fastify service that holds the GitLab root PAT and manages user lifecycle:

- **User provisioning**: Creates GitLab accounts and group structures on first login.
- **Impersonation tokens**: Creates tokens on sign-in, revokes them on sign-out.
- **Personal access tokens**: Issues scoped tokens for HTTPS git operations.

:::warning

GitLab root PAT should be regularly rotated and handled as one of the most sensitive security concerns.

:::

This service exists because the web layer should never hold root credentials. It acts as a narrow, auditable gateway to GitLab admin operations.

### K8s Root Actor

A Fastify service that holds cluster-admin credentials and manages Kubernetes deployments:

- **Inference endpoints**: Deploys models behind HTTP endpoints with Istio routing.
- **Spaces**: Deploys interactive applications (Gradio, Streamlit, Docker, static sites).
- **Custom actions**: Executes repository-defined workflows (`actions.yaml`) as Kubernetes deployments or Slurm jobs.
- **Lifecycle management**: Tracks deployment state, collects logs, manages cleanup.

Like the GitLab Root Actor, this service isolates privileged operations behind a purpose-built API.

### Kubeflow Root Actor _(planned)_

A Fastify service that holds Kubeflow admin credentials and manages Kubeflow resources on behalf of the cloud site agent:

- **Profile provisioning**: Creates Kubeflow profiles and namespaces for project teams.
- **RBAC management**: Configures role bindings and access policies for notebooks and pipelines.
- **GPU resource allocation**: Manages GPU quota assignments for training workloads.

This service isolates Kubeflow admin operations behind a narrow API, following the same root actor pattern as the GitLab and K8s root actors. It is invoked by the cloud site agent during order processing.

### MLflow Root Actor _(planned)_

A Fastify service that holds MLflow admin credentials and manages MLflow resources on behalf of the cloud site agent:

- **Tenant provisioning**: Creates MLflow tenants and workspaces for project teams.
- **Permission management**: Configures access controls for experiments and model registries.
- **Usage reporting**: Tracks resource consumption for billing and quota enforcement.

Like the other root actors, this service isolates MLflow admin operations behind a purpose-built API. It is invoked by the cloud site agent during order processing.

### Supporting Services

| Service                                      | Role                                                                                                     |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| **[Valkey](/terms/valkey.md)**               | Redis-compatible store for session data and [impersonation token](/terms/impersonation-token.md) mapping |
| **GitLab CE**                                | Git backend, repository storage, CI/CD. Not exposed to users directly                                    |
| **[Waldur](/terms/waldur.md)**               | Resource management platform — proposals, quota enforcement, billing, and site agent orchestration       |
| **[Kubeflow](/terms/kubeflow.md)**           | ML pipelines, notebook servers, and distributed training workloads                                       |
| **[MLflow](/terms/mlflow.md)**               | Experiment tracking, model registry, and artifact storage                                                |
| **[Open OnDemand](/terms/open-ondemand.md)** | HPC portal for interactive compute sessions (Jupyter, RStudio, virtual desktops)                         |

## Infrastructure

### Kubernetes

Kubernetes is the deployment target across all environments. Services run as pods with Kustomize overlays per environment (local, dev, staging, production).

- **Local development**: Minikube with host filesystem mounts for hot-reload, or Docker Compose for a lighter setup
- **Production**: AWS infrastructure provisioned via Terraform (VPC, EC2, ECR, S3)
- **Configuration management**: Ansible (41 roles covering identity, storage, networking, and ML platform components)

### Istio

Istio provides the service mesh:

- **Ingress**: Separate gateways for web traffic and deployed user workloads (spaces, inferences)
- **Routing**: VirtualService-based routing splits git smart HTTP operations from web requests
- **Security**: Mutual TLS between services in the mesh

See [Networking](/product/architecture/networking.md) for details on how routing works.

### GitOps

ArgoCD manages all deployments with automated sync, pruning, and self-healing. Each environment has a root ArgoCD Application that sources its configuration from a specific Kustomize overlay.

Sync waves control deployment order — infrastructure resources deploy before application services.

### Storage and Data

| Component      | Purpose                                                                  |
| -------------- | ------------------------------------------------------------------------ |
| **PostgreSQL** | GitLab persistence                                                       |
| **MinIO**      | S3-compatible object storage for Git LFS (model weights, large datasets) |
| **Ceph**       | Distributed storage for production workloads (via rook.io)               |

## Shared Libraries

| Package                        | Purpose                                        |
| ------------------------------ | ---------------------------------------------- |
| `@themelio/gitlab-client`      | Typed GitLab API client shared across services |
| `@workspace/eslint-config`     | Shared linting rules                           |
| `@workspace/typescript-config` | Shared TypeScript configuration                |

## Key Design Decisions

| Decision                                   | Rationale                                                                                            |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| GitLab CE as the storage backend           | Mature git API, built-in CI/CD, avoids building repository management from scratch                   |
| Impersonation tokens over OAuth delegation | GitLab CE lacks fine-grained OAuth scopes; impersonation provides full API access scoped to the user |
| Separate root actor services               | Isolates privileged credentials behind narrow, auditable APIs                                        |
| Valkey over Redis                          | Drop-in compatible with open-source governance                                                       |
| Istio over plain ingress                   | Smart routing (git vs. web), mTLS, observability hooks                                               |
| ArgoCD + Kustomize over Helm               | Declarative GitOps with environment overlays, less templating complexity                             |
| Next.js 15 for the web layer               | Full-stack development velocity, SSR for data-heavy dashboards, unified deployment                   |
| pnpm workspaces + Turborepo                | Dependency management and build orchestration across the monorepo with intelligent caching           |

<details class="page-source"><summary>Page source</summary>

<a href="/source/docs/content/product/architecture/overview.md.txt">Source snapshot</a> · Revision `f928834c8`.

</details>