Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#Architecture Overview

Themelio is an open-source ML software stack that gives research teams a single place to store models and datasets, deploy inference endpoints and interactive applications, and manage compute and HPC resources for running model training & finetuning, and other ML workflows — all backed by federated institutional authentication.

Under the hood, Themelio wraps GitLab CE as a repository storage and collaboration backend, adds a purpose-built web layer for ML workflows, and orchestrates workflows and deployments on Kubernetes and Slurm. Users never interact with GitLab directly; Themelio handles identity, access control, and API translation so that researchers see a familiar, Hugging Face-style experience.

This page covers the system's moving parts and how they fit together. The pages that follow go deeper into authentication, resource types, and networking.

#System Diagram

The diagram below shows the three main layers: the platform (application services running on Kubernetes), Waldur (resource management and provisioning), and compute infrastructure (HPC, GPU clusters, and storage).

#Services

Themelio runs as a bundle of cooperating services, each with a clear responsibility boundary.

#Themelio Web

The primary application — a Next.js 15+ app that serves both the UI and API routes. It handles everything a user sees and touches:

• UI: React 19 with server-side rendering. Catalogue pages, file browsers, deployment dashboards, settings.
• API routes: Organized by resource type (models, datasets, spaces, repos, recipes, upskilling materials). These routes translate between the Themelio API and GitLab's GraphQL and REST APIs.
• Authentication: NextAuth v5 with OIDC. Sessions stored in Valkey.
• Middleware: Intercepts every authenticated request, retrieves the user's GitLab impersonation token from Valkey, and injects it as a PRIVATE-TOKEN header. Downstream code never handles raw credentials.

The API routes live inside the same Next.js process, avoiding cross-service latency for user-facing operations.

#Cloud Site Agent (planned)

A Waldur site agent plugin that provisions cloud-based ML platform resources when a project proposal is accepted:

• Kubeflow resources: Delegates profile and namespace provisioning to the Kubeflow Root Actor.
• MLflow resources: Delegates tenant and workspace provisioning to the MLflow Root Actor.
• Order lifecycle: Handles order processing, membership synchronization, and usage reporting via the Waldur site agent framework.

The cloud site agent does not hold privileged credentials itself — it delegates all privileged operations to the Kubeflow and MLflow root actors, maintaining the same separation of concerns used across the platform.

#HPC Site Agent (planned)

A Waldur site agent plugin that provisions HPC infrastructure to give users access to Slurm-based compute resources:

• Slurm access: Provisions user accounts and project allocations on HPC clusters.
• Open OnDemand: Configures interactive compute sessions (Jupyter, RStudio, virtual desktops) for project members.
• Storage mounts: Sets up shared filesystem access between HPC nodes and project storage.
• Order lifecycle: Handles order processing, membership synchronization, and usage reporting via the Waldur site agent framework.

#GitLab Root Actor

A Fastify service that holds the GitLab root PAT and manages user lifecycle:

• User provisioning: Creates GitLab accounts and group structures on first login.
• Impersonation tokens: Creates tokens on sign-in, revokes them on sign-out.
• Personal access tokens: Issues scoped tokens for HTTPS git operations.

This service exists because the web layer should never hold root credentials. It acts as a narrow, auditable gateway to GitLab admin operations.

#K8s Root Actor

A Fastify service that holds cluster-admin credentials and manages Kubernetes deployments:

• Inference endpoints: Deploys models behind HTTP endpoints with Istio routing.
• Spaces: Deploys interactive applications (Gradio, Streamlit, Docker, static sites).
• Custom actions: Executes repository-defined workflows (actions.yaml) as Kubernetes deployments or Slurm jobs.
• Lifecycle management: Tracks deployment state, collects logs, manages cleanup.

Like the GitLab Root Actor, this service isolates privileged operations behind a purpose-built API.

#Kubeflow Root Actor (planned)

A Fastify service that holds Kubeflow admin credentials and manages Kubeflow resources on behalf of the cloud site agent:

• Profile provisioning: Creates Kubeflow profiles and namespaces for project teams.
• RBAC management: Configures role bindings and access policies for notebooks and pipelines.
• GPU resource allocation: Manages GPU quota assignments for training workloads.

This service isolates Kubeflow admin operations behind a narrow API, following the same root actor pattern as the GitLab and K8s root actors. It is invoked by the cloud site agent during order processing.

#MLflow Root Actor (planned)

A Fastify service that holds MLflow admin credentials and manages MLflow resources on behalf of the cloud site agent:

• Tenant provisioning: Creates MLflow tenants and workspaces for project teams.
• Permission management: Configures access controls for experiments and model registries.
• Usage reporting: Tracks resource consumption for billing and quota enforcement.

Like the other root actors, this service isolates MLflow admin operations behind a purpose-built API. It is invoked by the cloud site agent during order processing.

#Supporting Services

#Infrastructure

#Kubernetes

Kubernetes is the deployment target across all environments. Services run as pods with Kustomize overlays per environment (local, dev, staging, production).

• Local development: Minikube with host filesystem mounts for hot-reload, or Docker Compose for a lighter setup
• Production: AWS infrastructure provisioned via Terraform (VPC, EC2, ECR, S3)
• Configuration management: Ansible (41 roles covering identity, storage, networking, and ML platform components)

#Istio

Istio provides the service mesh:

• Ingress: Separate gateways for web traffic and deployed user workloads (spaces, inferences)
• Routing: VirtualService-based routing splits git smart HTTP operations from web requests
• Security: Mutual TLS between services in the mesh

See Networking for details on how routing works.

#GitOps

ArgoCD manages all deployments with automated sync, pruning, and self-healing. Each environment has a root ArgoCD Application that sources its configuration from a specific Kustomize overlay.

Sync waves control deployment order — infrastructure resources deploy before application services.

#Storage and Data

#Shared Libraries

#Key Design Decisions

Page source

Source snapshot · Revision f928834c8.