# Pharos Platform

> Guides for using Pharos Platform: Projects, repositories, Runs, HPC and AI services.

## Get started

- [Get started](/guides/getting-started.md): You can browse published repositories before signing in. To join a Project, create repositories or run work, sign in to your organisation's Pharos Platform.

## Projects

- [Projects](/guides/projects.md): A Project brings together a team, its repositories and its allocated resources. Open a Project from your dashboard to find its members, hardware and active Runs.

## Repositories

- [Repositories](/guides/repositories.md): Use Explore to browse published models, datasets, Spaces and other repository categories. Open a repository to read its description, README and files. Available task and licence filters help narrow the catalogue.

## Runs

- [Runs](/guides/runs.md): A Run records an execution of a repository action. Actions can submit batch jobs or start applications and inference services, depending on the Site and your Project's allocations.

## Others

- [0000-agentic-system](/adr/0000-agentic-system.md)
- [ADR-0000: Friends & Family MVP Scope](/adr/0000-mvp-friends-and-family.md): Status: Proposed Date: 2026-05-14 Deciders: TBD Relates to: Product Requirements (Core), Functional Requirements, Architecture Overview
- [ADR-0003: LFS Storage Quota Enforcement via Custom Proxy](/adr/0001-lfs-storage-quota-enforcement.md)
- [ADR-0002: Architecture Documentation Update](/adr/0002-architecture-docs-update.md)
- [ADR-0003: Waldur Proposal Workflow](/adr/0003-waldur-proposal-workflow.md): Status: Proposed Date: 2026-05-07 Deciders: TBD Relates to: ADR-0000 (Friends & Family MVP) — "Proposal Workflow" exploration item; Waldur Installation & Management (future ADR) — functional requirements FR-PRO-01 through FR-PRO-04.
- [ADR-0004: Xet Content-Addressed Storage for ML Artifacts](/adr/0004-xet-content-addressed-storage.md)
- [ADR-0005: AAI → Waldur User Synchronization](/adr/0005-aai-waldur-user-sync.md): Status: Accepted Date: 2026-05-13 Deciders: TBD Relates to: ADR-0003 (Waldur Proposal Workflow) — users must exist in Waldur before they can be added to projects and orders.
- [ADR-0006: Automated Data Seeding Strategy](/adr/0006-automated-seeding-strategy.md): Status: Accepted Date: 2026-05-14 Deciders: TBD Relates to: ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)
- [ADR-0007: GPU Slicing and Quota Enforcement](/adr/0007-gpu-slicing-and-quota-enforcement.md): Status: Draft Date: 2026-05-22 Deciders: TBD
- [ADR-0007: Stress Test Strategy](/adr/0007-stress-test-strategy.md): Status: Draft Date: 2026-05-22 Deciders: TBD Relates to: ADR-0000 (Friends & Family MVP) — stress testing is a prerequisite for MVP confidenceIssue #85 (Observability) — monitoring infrastructure needed during test executionIssue #96 (Define stress test strategy) — parent work item
- [ADR-0009: Shared LLM Inference Backend](/adr/0009-shared-llm-inference-backend.md): Status: Draft Date: 2026-05-22 Deciders: Platform developers (confirmed 2026-06-30) Relates to: ADR-0000 (Friends & Family MVP)ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)ADR-0010 (LLM Gateway and Project Token Quotas)ADR-0011 (Chat UI and Client Access for Shared LLM Services)
- [ADR-0010: LLM Gateway and Project Token Quotas](/adr/0010-project-scoped-llm-gateway-token-quotas-api-access.md): Status: Draft Date: 2026-05-22 Deciders: Platform developers (confirmed 2026-06-30) Relates to: ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)ADR-0009 (Shared LLM Inference Backend)ADR-0011 (Chat UI and Client Access for Shared LLM Services)ADR-0020 (LLM Access Membership Source of Truth)
- [ADR-0011: Chat UI and Client Access for Shared LLM Services](/adr/0011-human-web-ui-and-agent-client-access.md): Status: Draft Date: 2026-05-22 Deciders: Themelio developers (confirmed 2026-06-30) Relates to: ADR-0009 (Shared LLM Inference Backend)ADR-0010 (LLM Gateway and Project Token Quotas)ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)ADR-0020 (LLM Access Membership Source of Truth)
- [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system.md): Status: Proposed Date: 2026-06-02 Deciders: TBD
- [ADR-0013: actions.yaml Automation System](/adr/0013-actions-yaml-automation-system.md): Status: Proposed Date: 2026-06-03 Deciders: TBD Relates to: ADR-0012 (Job and Deployment Management System) — actions.yaml builds on the PostgreSQL persistence layer
- [ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance](/adr/0014-gitlab-ci-runner-ec2-instance.md): Status: Accepted Date: 2026-06-02 Deciders: Thanasis Katsadas, Panos Louridas Relates to: ADR-0015 (GitLab CI Runner EC2 Instance — Ansible Onboarding) — covers Ansible provisioning, runner installation, and GitLab registration once the instance decided here is running.
- [ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding](/adr/0015-gitlab-ci-runner-ec2-ansible-onboarding.md): Status: Proposed Date: 2026-06-03 Deciders: Thanasis Katsadas, Dimitris Katsimpris Relates to: ADR-0014 (GitLab CI Runner on Dedicated AWS EC2 Instance) — this ADR addresses the operational onboarding of the instance type and purpose decided there. Implementation: DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding
- [ADR-0016: Declarative Database List in the Ansible postgres Role](/adr/0016-declarative-database-list-in-postgres-role.md): Status: Proposed Date: 2026-06-04 Deciders: Thanasis Katsadas
- [ADR-0017: Shared Themelio Database with Decoupled Schema Migrations](/adr/0017-shared-themelio-database-decoupled-migrations.md): Status: Proposed Date: 2026-06-08 Deciders: TBD Relates to: ADR-0012 (Job and Deployment Management System) — decided the PostgreSQL job/deployment store this database backs.ADR-0016 (Declarative Database List in the Ansible postgres Role) — the declarative CNPG provisioning this database rides on.
- [ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads](/adr/0018-queued-status-and-over-capacity-handling.md): Status: Proposed Date: 2026-07-02 Deciders: TBD
- [ADR-0019: Container Image Tag Management for GitOps Deployments](/adr/0019-container-image-tag-management.md): Status: Draft Date: 2026-06-26 Deciders: Athanasios Katsadas, TBD
- [ADR-0020: LLM Access Membership Source of Truth](/adr/0020-llm-access-membership-source-of-truth.md): Status: Draft Date: 2026-06-30 Deciders: Themelio developers (confirmed in session, 2026-06-30) Relates to: ADR-0005 (AAI → Waldur User Synchronization)ADR-0010 (Project-Scoped LLM Gateway, Token Quotas, and API Access)ADR-0011 (Human Web UI and Agent Client Access)
- [ADR-0021: Themelio Command-Line Interface](/adr/0021-themelio-cli.md): Status: Draft Date: 2026-07-06 Deciders: TBD Relates to: ADR-0011 (Human Web UI and Agent Client Access) — the CLI is a new programmatic client alongside the web UI; ADR-0012 (Job and Deployment Management System) and ADR-0013 (actions.yaml Automation System) — the CLI drives and observes the jobs/deployments and actions those ADRs introduced.
- [ADR-0023: DRA GPU Allocation with Extended Resource Names](/adr/0023-dra-gpu-extended-resource-names.md): Status: Accepted Date: 2026-07-06 Deciders: Themelio developers Relates to: ADR-0007 (GPU Slicing and Quota Enforcement) - decided the GPU slicing and quota model; this ADR decides how Themelio should consume NVIDIA DRA while preserving that model.
- [ADR-0025: Scoped Istio Authorization Policies](/adr/0025-scoped-istio-authorization-policies.md): Status: Draft Date: 2026-07-16 Deciders: TBD Related: ADR-0024 Baseline L3/L4 Network Segmentation (historical reference unavailable)
- [Architectural Decision Records](/adr/index.md)
- [Architecture and ownership](/architecture/index.md): Use the repository's architecture, modelling conventions and context catalogue as the current shared reference. Context READMEs own the detailed operations, invariants and implementation limits.
- [Documentation version](/build.md): Source revision: f928834c82b9d99e12575f9bd3f311c16c8f138b. Branch: detached. Local changes: none. Built at: 2026-11-09T12:32:21.242Z. Reviewed against integration/current at 1f3f0e84d7271ecedd9b3274dfb9c5870e8282ff on 2026-11-09. This records the documentation source. Feature availability depends on the installation and Project. Developers can find implementation and qualification guidance under Contributing.
- [Themelio compatibility and parallel operation](/compatibility/index.md): The retained application lives under compat/themelio and runs as themelio-compat. The new service scaffold lives under services/themelio. Both can appear in the same environment. Their deployment names and responsibilities must remain explicit in operational instructions.
- [Contributing to Pharos Platform](/contributing/index.md): This section is for developers and operators working on the platform. For day-to-day use, start with the user guides.
- [DD-0005: AAI → Waldur User Sync — Implementation](/dd/0005-aai-waldur-user-sync.md): Related ADR: ADR-0005: AAI → Waldur User Synchronization Status: Draft Date: 2026-05-19 Authors: TBD
- [DD-0006: Automated Seeding Strategy — Implementation](/dd/0006-automated-seeding-strategy.md): Related ADR: ADR-0006: Automated Data Seeding Strategy This document provides the full technical specification for implementing the automated seeding strategy described in ADR-0006. It is intended to be followed by an engineer or an automated agent.
- [DD-0009: Shared LLM Inference Backend — Implementation](/dd/0009-shared-llm-inference-backend.md): Related ADR: ADR-0009: Shared LLM Inference Backend Status: Draft Date: 2026-06-30 Authors: TBD
- [DD-0010: LLM Gateway and Project Token Quotas — Implementation](/dd/0010-project-scoped-llm-gateway-token-quotas-api-access.md): Related ADR: ADR-0010: LLM Gateway and Project Token Quotas Status: Draft Date: 2026-06-30 Authors: TBD
- [DD-0011: Chat UI and Client Access — Implementation](/dd/0011-human-web-ui-and-agent-client-access.md): Related ADR: ADR-0011: Chat UI and Client Access for Shared LLM Services Status: Draft Date: 2026-06-30 Authors: TBD
- [DD-0012: Job and Deployment Management System — Implementation](/dd/0012-job-deployment-management-system.md): Related ADR: ADR-0012: Job and Deployment Management System Status: Draft Date: 2026-06-02 Authors: TBD
- [DD-0013: actions.yaml Automation System — Implementation](/dd/0013-actions-yaml-automation-system.md): Related ADR: ADR-0013: actions.yaml Automation System Status: Draft Date: 2026-06-04 Authors: TBD
- [DD-0014: GitLab CI Runner EC2 Instance — Manual Operator Steps](/dd/0014-gitlab-ci-runner-ec2-instance.md): Related ADR: ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance Status: Draft Date: 2026-06-03 Authors: Thanasis Katsadas
- [DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding](/dd/0015-gitlab-ci-runner-ec2-ansible-onboarding.md): Related ADR: ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding Status: Draft Date: 2026-06-03 Authors: Dimitris Katsimpris
- [DD-0016: Declarative Database List in the Ansible postgres Role — Implementation](/dd/0016-declarative-database-list-in-postgres-role.md): Related ADR: ADR-0016: Declarative Database List in the Ansible postgres Role Status: Draft Date: 2026-06-04 Authors: Thanasis Katsadas
- [DD-0017: Shared Themelio Database with Decoupled Schema Migrations — Implementation](/dd/0017-shared-themelio-database-decoupled-migrations.md): Related ADR: ADR-0017: Shared Themelio Database with Decoupled Schema Migrations Status: Draft Date: 2026-06-08 Authors: TBD
- [DD-0018: Queued Status and Over-Capacity Handling — Implementation](/dd/0018-queued-status-and-over-capacity-handling.md): Related ADR: ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads Status: Draft Date: 2026-07-02 Authors: TBD
- [DD-0019: Container Image Tag Management — Implementation](/dd/0019-container-image-tag-management.md): Related ADR: ADR-0019: Container Image Tag Management for GitOps Deployments Status: Draft Date: 2026-06-29 Authors: TBD
- [DD-0020: LLM Access Membership Source of Truth — Implementation](/dd/0020-llm-access-membership-source-of-truth.md): Related ADR: ADR-0020: LLM Access Membership Source of Truth Status: Draft Date: 2026-06-30 Authors: TBD
- [DD-0021: Themelio CLI — Implementation](/dd/0021-themelio-cli.md): Related ADR: ADR-0021: Themelio Command-Line Interface Status: Draft Date: 2026-07-06 Authors: TBD
- [DD-0023: DRA GPU Extended Resource Implementation](/dd/0023-dra-gpu-extended-resource-names.md): Related ADR: ADR-0023: DRA GPU Allocation with Extended Resource Names Status: Draft Date: 2026-07-09 Authors: TBD
- [DD-0025: Scoped Istio Authorization Policies — Implementation](/dd/0025-scoped-istio-authorization-policies.md): Related ADR: ADR-0025: Scoped Istio Authorization Policies Status: Draft Date: 2026-07-20 Authors: TBD
- [Design Documents](/dd/index.md): Design Documents (DDs) are detailed technical specifications that describe how to implement a decision recorded in an ADR. Where an ADR answers "what did we decide and why?", a Design Document answers "how exactly do we build it?" Each DD is linked from its corresponding ADR and is intended to be actionable by an engineer or an automated agent.
- [Development and operations](/development/index.md)
- [Compatibility glossary](/glossary.md): Terms retained from the Themelio documentation. See the current modelling guide for platform conventions. aaidaedalusdatasetedugaineidasgated resourcegitlab root actorgrnetimpersonation tokenk8s root actorkatibkubeflowlfsmlflowmodelmyaccessidnode hoursoidcopen ondemandpharos aaipharos ai factoryrbacrciamreciperesourceslurmspacethemelio cataloguethemelio explorethemelio projectthemelio webthemelioupskilling materialvalkeywaldur site agentwaldur <a href="/source/docs/content/glossary.md.txt">Source snapshot</a> · Revision f928834c8.
- [HPC access](/guides/hpc-access.md): The HPC access settings show whether your membership, policy acceptance and Site account allow you to register an SSH public key.
- [Using Pharos Platform](/guides/index.md): Pharos Platform brings your team's repositories, compute and AI services together in a Project. Start with the task you want to complete. Available services depend on your Project's allocations, your permissions and the Site's configuration. The platform shows preparation and access messages when a service needs attention. <a href="/source/docs/content/guides/index.md.txt">Source snapshot</a> · Revision f928834c8.
- [LLM access](/guides/llm-access.md): A Project's LLM allocation determines its available models, shared budget and rate limits. Open the Project dashboard to inspect your allocation and available chat and API access.
- [Support](/guides/support.md): Open Support to create and follow your own tickets when your installation has the service enabled.
- [Educational sources and attribution](/learn/attribution.md): The learning path adapts eight chapters from the multilingual architecture reference, commit b6b40bd039a135676ff5d52cd923aa2a019156fc (22 October 2026). Shared definitions and explanations are retained or adapted; café examples are replaced with reviewed Pharos examples. The original authorship remains in the source repository. docs/education-provenance.json records the source paths, revision and SHA-256 hashes. Copying between repositories does not transfer Git history. The retained Themelio pages have a separate exact git mv relocation recorded in docs/provenance.json. Pharos code examples come from the current checkout and are reviewed with the <a href="/source/docs/integration-baseline.json.txt">integration baseline</a>. Build-time extraction keeps the shown excerpt aligned with its source. The platform's own policies and context guides govern implementation.
- [Consistency and reliable delivery](/learn/consistency-and-delivery.md): Business consistency starts with identifying what must be true together. Pharos protects an aggregate transition in one owner transaction. Work involving other aggregates proceeds through explicit pending states and durable hand-offs.
- [CQRS: decisions and views](/learn/cqrs.md): Command Query Responsibility Segregation separates the model used to change state from the model used to read it. A command expresses intent; a query asks for information. Separate models can share one database and one process.
- [Event-driven architecture: facts and reactions](/learn/event-driven-architecture.md): An event-driven system connects parts of an application through facts that have occurred. A publisher records a fact; subscribers decide what it means for their own responsibilities.
- [Event sourcing: choosing the source of truth](/learn/event-sourcing.md): Event sourcing stores an aggregate's authoritative changes as an ordered event stream. Reconstructing state applies those events in sequence. State reconstruction must reproduce recorded decisions deterministically.
- [Hexagonal architecture: ports and adaptors](/learn/hexagonal-architecture.md): Hexagonal architecture exposes meaningful application interactions and expresses external needs through ports. Adaptors connect those capabilities to technologies. Source dependencies point towards application and domain policy.
- [Learn the architecture](/learn/index.md): Start with the decisions the platform must protect: who a Party is, which source owns a Project, which memberships have been observed, and what evidence permits an operation. Then follow those decisions through persistence and delivery.
- [Strategic DDD: language and ownership](/learn/strategic-ddd.md): Domain-driven design connects software design to a shared understanding of the business. Strategic design identifies the models a system needs, their boundaries and their relationships. The purpose is to make business meaning explicit enough that people can discuss and change the software coherently.
- [Tactical DDD: behaviour and consistency](/learn/tactical-ddd.md): Tactical DDD provides building blocks for implementing a domain model. Start with the decisions and rules, then choose the objects needed to express them.
- [Testing the architectural claims](/learn/testing-and-exercises.md): Choose a test by the claim it must establish. Domain tests, adaptor tests and real-service qualification provide different evidence.
- [Pharos Platform](/platform/index.md): Work with your team in a Project, discover models and datasets, and use your allocated compute and AI services. Start with the user guides, or go straight to signing in, Projects, repositories or Runs. Developers and operators can find architecture, setup, testing and implementation status under Contributing. <a href="/source/docs/content/platform/index.md.txt">Source snapshot</a> · Revision f928834c8.
- [Authentication](/product/architecture/authentication.md): This page covers how Themelio handles authentication — identity providers, the login and logout flow, token lifecycle, and session management. For a broader view of how authentication fits into the system, see the Architecture Overview. See also the Pharos AAI Architecture for a deep dive into the authentication and authorization layer.
- [Authorization Policies](/product/architecture/authorization-policies.md): This page explains what Istio authorization policies do in Themelio and what to do about them whenever a new service is added. For the traffic and mTLS picture see Networking.
- [Catalogue & Discovery](/product/architecture/flows/catalogue-and-discovery.md): This page describes how users browse, search, and discover resources on the platform — models, datasets, spaces, repos, recipes, and upskilling materials. For what each resource type is and what you can do with it, see Resource Types.
- [Deployment](/product/architecture/flows/deployment.md): This page describes how users deploy model inference endpoints, interactive spaces, and custom workloads. All deployments are orchestrated by k8s-root-actor, which manages Kubernetes resources and Istio routing on behalf of users.
- [Overview](/product/architecture/flows/overview.md): This section documents the key user-facing flows in Themelio. Each page covers a distinct functional area with diagrams showing how services interact. Catalogue & Discovery — Browsing, searching, and filtering models, datasets, spaces, and other resources.Repository Operations — Creating, forking, browsing repositories, and working with git (clone, push, LFS).Deployment — Deploying model inference endpoints, spaces, and custom actions via actions.yaml.Resource Management — Project proposals, compute provisioning, quota enforcement, and Waldur integration. For authentication flows, see the Authentication page. For an overview of the resource types these flows operate on, see Resource Types. <a href="/source/docs/content/product/architecture/flows/overview.md.txt">Source snapshot</a> · Revision f928834c8.
- [Repository Operations](/product/architecture/flows/repository-operations.md): This page describes how users create, browse, fork, and manage repositories. It also covers how git operations (clone, push, pull) work through the platform and how large files are handled. Repositories are the core unit of content on the platform — every model, dataset, space, recipe, and upskilling resource is backed by a GitLab repository. See Resource Types for what each type adds on top of this shared foundation.
- [Resource Management](/product/architecture/flows/resource-management.md): This page describes how compute resources are provisioned and managed — from project proposals to Kubernetes namespaces with enforced quotas.
- [GitLab Groups & Visibility](/product/architecture/gitlab-groups-and-visibility.md): This document describes how Themelio uses GitLab groups and subgroups to organize repositories (models, datasets, spaces), and the visibility rules that govern access.
- [Networking](/product/architecture/networking.md): This page describes how traffic reaches Themelio's services and how deployed workloads (model inference endpoints, spaces) get routable URLs.
- [Observability](/product/architecture/observability.md): This page describes the monitoring, logging, and tracing capabilities available to operators running a Themelio deployment.
- [Overview](/product/architecture/overview.md): Themelio is an open-source ML software stack that gives research teams a single place to store models and datasets, deploy inference endpoints and interactive applications, and manage compute and HPC resources for running model training & finetuning, and other ML workflows — all backed by federated institutional authentication. Under the hood, Themelio wraps GitLab CE as a repository storage and collaboration backend, adds a purpose-built web layer for ML workflows, and orchestrates workflows and deployments on Kubernetes and Slurm. Users never interact with GitLab directly; Themelio handles identity, access control, and API translation so that researchers see a familiar, Hugging Face-style experience. This page covers the system's moving parts and how they fit together. The pages that follow go deeper into authentication, resource types, and networking.
- [Self-Hosting](/product/architecture/self-hosting.md): This page is a guide for operators who want to deploy Themelio for their institution or research group. It covers the deployment model, prerequisites, initial setup, and ongoing operations.
- [Authoring actions.yaml](/product/guides/actions-yaml.md): An actions.yaml file lives at the root of a Themelio repository and declares one or more actions that anyone with write access to the repository can run from the Themelio web UI. An action packages a piece of compute — a training script, a long-running app, an inference service — so it can be launched on demand, with parameters, without anyone touching a terminal. This guide shows how to write one. It covers the file format, the three action types, parameters, references, and how an action is run and torn down.
- [The themelio CLI](/product/guides/themelio-cli.md): themelio is a Hugging Face–style command-line client for the Themelio platform. It drives the same operations as the web UI — authentication, browsing and creating repositories for models/datasets/spaces, running actions.yaml actions, and observing the resulting jobs and deployments — from your terminal, so you can script Themelio from CI or an agent.
- [Getting started](/product/introduction/getting-started.md)
- [Introduction to Themelio](/product/introduction/index.md): This page will help you understand how Themelio lowers barriers to AI adoption—enabling both non‑experts and developers to quickly discover, experiment with, and integrate cutting‑edge AI models into their own research or projects.
- [Vision](/product/introduction/vision.md): Vision and Mission Pharos AI Factory represents Greece's commitment to becoming a leader in trustworthy artificial intelligence and high-performance computing in Southeastern Europe. As one of the first seven AI Factories established under the EuroHPC Joint Undertaking, Themelio positions itself as a lighthouse of trustworthy AI in Southeastern Europe. The factory's mission is to foster an ethically-grounded, technologically advanced AI ecosystem that bridges academia, industry and the public sector. Rather than operating as a traditional research institution, Themelio functions as a comprehensive hub that supports innovation, entrepreneurship, and research excellence across multiple sectors. Strategic Focus Areas Pharos AI concentrates its efforts on three critical sectors that offer transformative potential: Health — leveraging AI to develop innovative medical applications and improve healthcare outcomes across Greece and the EU. Culture and Language — creating comprehensive AI services specifically designed for the Greek language, developing data spaces, preparing data for AI use, and developing open AI models. This includes support for SMEs through specialized tools and development environments (sandboxes). Sustainability — addressing climate change and environmental challenges through satellite data analysis for natural resource management and mitigation of climate-related impacts. Customer Value Proposition Pharos AI delivers value to four primary customer segments: For Researchers and Academic Institutions, Pharos AI offers opportunities to collaborate on AI projects with direct access to the Daedalus supercomputer, establishing a research hub that connects with other European infrastructures and EU initiatives. For Public Sector Organizations, the factory develops AI applications that address societal challenges, particularly in healthcare and environmental management. For Startups and SMEs, the factory provides access to cutting-edge AI tools, high-performance computing infrastructure, large-scale datasets, and specialized human expertise. This substantially reduces the barrier to entry for innovation and enables scaling of AI solutions without requiring massive internal infrastructure investments. For Citizens, Pharos AI delivers AI-driven advancements in health, culture preservation, and sustainability that contribute to Greece's and the EU's broader digital transformation goals. The factory specifically enhances competitiveness by providing advanced computational resources, access to big data, and specialized human expertise. Additionally, it supports the development of reliable AI services aligned with European AI regulations (particularly the AI Act). Business Impact and Strategic Outcomes With €30 million in funding—split equally between the EuroHPC Joint Undertaking and national resources—Pharos AI will operate from March 2025 over a three-year period to establish Greece's AI infrastructure. The factory helps Greek startups and SMEs access advanced AI tools and computing power they couldn't afford otherwise, creating opportunities for new businesses to emerge and grow while strengthening Greece's technology sector. By focusing on practical solutions for critical industries, Pharos AI uses satellite data to tackle environmental challenges and develops climate technologies that make a real difference in sustainability efforts. The program builds Greece's AI talent pipeline through university partnerships, skills development programs, and initiatives to attract AI experts to the country. As part of a continent-wide network of AI Factories, Pharos AI collaborates with Digital Innovation Hubs, data centers, and European initiatives like DeployAI and LLMs4EU, helping Greece participate in larger European AI projects. Being one of the first European AI Factories positions Greece at the center of Europe's AI infrastructure development, enabling the country to both contribute to and benefit from European AI advancement. Governance and Implementation The project is managed and operated by GRNET (National Infrastructures for Research and Technology) under the auspices of the Greek Ministry of Digital Governance. The consortium includes leading institutions such as the National Centre for Scientific Research DEMOKRITOS, the National Technical University of Athens, the Athena Research Center, and numerous affiliated universities and research organizations. This institutional framework ensures that Pharos AI operates with both technical excellence and strategic alignment with Greek national priorities and European AI policy objectives. <a href="/source/docs/content/product/introduction/vision.md.txt">Source snapshot</a> · Revision f928834c8.
- [ADR-0025 Validation](/product/operations/adr-0025-validation.md): This document records the commands and manual checks used to validate ADR-0025 in staging. Execute it with two people or with an operator and an agent: run one numbered validation, inspect its evidence, record the result, and only then continue. Do not execute the whole document as an unattended script. The validation covers only behavior required by ADR-0025: native Istio security controls, proof that the protected application flows still work, and regression checks for the gateways and applications affected by those controls. A failed required validation keeps ADR-0025 in Draft.
- [Backup GitLab Repositories](/product/operations/backup-gitlab-repos.md): Themelio stores all user-facing artifacts (models, datasets, spaces, repos, recipes, upskillings) as Git repositories on a GitLab instance, with binary content managed by Git LFS. Backing up the Git side of a Themelio deployment therefore reduces to mirroring every project on the source GitLab instance to a target GitLab instance. This page documents that procedure; GitLab-level metadata (issues, MRs, wiki, registries) and Themelio's non-GitLab state (Keycloak, Redis/Valkey, etc.) are out of scope here and need separate runbooks. The repository ships a script that does exactly that: It enumerates projects from the source instance, recreates the same group hierarchy under a configurable root group on the target, and pushes every ref, tag, and LFS blob across.
- [Overview](/product/operations/index.md): Operator-facing runbooks for maintaining a running Themelio instance: backup and restore procedures, migration playbooks, and other day-2 tasks. Backup GitLab repositories — mirror every Git repository (refs, branches, tags, and Git LFS objects) from a source GitLab instance to a target GitLab instance, preserving the group hierarchy.Waldur GitLab Sync — create and manage GitLab groups, members, and metadata in response to Waldur project lifecycle events.Local Stack Known Issues — issues and lessons for the full local stack (make local-up), covering both WSL2/VHD and native Linux/loop-device hosts.ADR-0025 Validation — execute and record the scoped Istio host, workload-identity, ambient mTLS, observability, and affected application regression validations one gate at a time. <a href="/source/docs/content/product/operations/index.md.txt">Source snapshot</a> · Revision f928834c8.
- [Kubernetes Upgrade Compatibility Audit](/product/operations/k8s-upgrade-audit.md): A reusable prompt for driving an AI agent (Claude Code or similar) through a read-only compatibility audit of this repository before a Kubernetes upgrade. It cross-checks every version-pinned component — Kubespray, Helm charts, operators, CNI, manifests, and client libraries — against a target Kubernetes version and reports what must change, and in what order.
- [Local Stack — Known Issues and Lessons Learned](/product/operations/local-known-issues.md): Issues, workarounds, and lessons discovered while validating: This document covers both supported local host modes: native Debian/Linux hosts using a loop-backed imageWindows/WSL2 hosts using a Windows VHD passed through with wsl --mount Canonical automation now lives in code, not only in notes: devops/ansible/Makefile is the entrypoint for local bring-upmake local-up is the supported path for Rook-Ceph local runs (Ceph RGW for object storage, Minikube standard for PVCs)WSL VHD settings live in devops/ansible/inventory/local/group_vars/all.yml
- [Access pgAdmin](/product/operations/pgadmin-access.md): pgAdmin runs inside the Kubernetes cluster in the databases namespace. It is not exposed publicly; access it through an SSH tunnel and a Kubernetes port-forward on the control-plane node.
- [Waldur GitLab Sync](/product/operations/waldur-gitlab-sync.md): When a Pharos AI Factory administrator approves a resource allocation in Waldur, the platform must provision the corresponding infrastructure: a Kubernetes namespace with resource quotas and a matching set of GitLab groups so that project members can push models, datasets, spaces, repos, recipes, and upskillings. This page documents how that synchronisation works end-to-end, how it is deployed in staging/production, and how to test it locally.
- [Core Specifications](/product/requirements/core.md): Version: 0.2 | Status: Under Review | Date: 2026-2-20 Project: Themelio / Daedalus This page describes the Core Specifications based on the high-level write up of use cases written by Panos Louridas, and reviewed and commented by the technical staff of GRNET and NTUA teams involved in the project.
- [Functional Requirements](/product/requirements/functional-requirements.md): Version: 0.2 | Status: Under Review | Date: 2026-2-20 This document contains the Functional Requirements for Themelio. For complete context including executive vision, scope, and user personas, please refer to the Core Specifications and the Introduction to Themelio.
- [User Stories](/product/requirements/user-stories.md): Version: 0.2 | Status: Under Review | Date: 2026-2-20 This document contains the User Stories for Themelio. For complete context including executive vision, scope, and user personas, please refer to the Core Specifications and the Introduction to Themelio.
- [Architecture](/reference/ARCHITECTURE.md): The platform's five top-level service boundaries are Web, API, Metallo, Themelio and Beto. Metallo owns Execution Core for internal Run/Attempt identity and persistence, Site Coordination for ordered execution commands and recorded results, and Site Runtime for Site-local delivery, native recovery and member/workspace rules. Pipeline Coordination owns Pipeline records and their ordered child steps, with internal start, read and cancellation-request operations through published owner command/query interfaces and context-owned persistence ports. Its private PostgreSQL database commits each snapshot, receipt and pending coordinator work atomically. Its leased coordinator submits stable step launches through Execution Core and records Run progress through its published queries and completion events. Workload Portfolio owns Workloads and their immutable revisions. Execution Core and Pipeline Coordination read saved revisions through Workload Portfolio's published query interfaces. Source Control owns repository/provider bindings and delegated GitLab effects. Its first slice owns consumer projections of Beto Project/Party facts and reconciles Project groups and direct memberships. It never creates provider accounts. Its dedicated process owns repository requests, publication and recovery. Its context guide records owner bootstrap, activation requirements and qualification limits. Beto owns Party Governance, the Project context and the Support context, with separate logical PostgreSQL databases and credentials. Project owns its Waldur Project and membership acquisition ports. Support links Pharos users to Zammad customers and files and follows their tickets. Metallo consumes canonical queries/events when its receiving capabilities are enrolled; its other provider capabilities remain separate. Beto also owns Marketplace, which has its Order domain, database and private OrderPlaced event, and serves PlaceOrder and ListProjectOrders over RabbitMQ and the API's routes with stand-in catalogue and buyer ports. SSH Access owns members' SSH public keys per Site, their lifecycle and the Site work that applies them. Resource Provisioning owns allocation and native-account evidence under its working boundary. Member Credentials owns sealed offline consent and native-action token refresh. Themelio has no populated bounded contexts. Process names do not pre-assign future domain authority. The context catalogue describes each registered context's responsibilities and owner, active proposals and unresolved boundaries. The Source Control context belongs to Metallo; its scope and remaining boundary decisions are recorded in the catalogue and its context README. LLM Access belongs to Metallo and owns its Project LLM allocation and provider fulfilment model; see its context guide. The shared rules below apply as capabilities are introduced. Context READMEs record active operations, adaptor choices, implementation limits and verification. Reusable foundations and empty registries alone do not establish a live capability.
- [Realtime browser messages](/reference/CLIENT-SUBSCRIPTIONS.md): Centrifugo with Protobuf is the browser update transport. Contexts publish typed messages to an explicit domain or security audience. The API calculates every channel from the authenticated principal's current authority. Browsers neither name nor select channels. Run applications have a separate Project-private serving gateway. It forwards application HTTP, SSE and WebSocket traffic with continuously checked member authority. Platform projections and notifications use the rules in this document.
- [Contributing](/reference/CONTRIBUTING.md): Use British English. Read AGENTS.md, ARCHITECTURE.md, DDD.md, CLIENT-SUBSCRIPTIONS.md and TESTING.md before architectural changes. Identify the owner, aggregate and command/query before editing. Keep business policy inside its context, inward dependencies explicit, and each service's source independent. Use native types and typed outcomes; generated contracts remain outside domain/application packages. Create reviewable branches from the selected baseline. Use Conventional Commits, for example feat(metallo): add a context command. Record notable changes in a <a href="/source/changelog.d/README.md.txt">PR changelog fragment</a>. Release-preparation PRs assemble CHANGELOG.md. Follow the documentation scope when updating guides or recording architectural decisions. Number a new decision after the highest number on main and in open pull requests. Paginate the complete changed-file lists of open pull requests; large infrastructure changes can place decisions beyond the first page. pnpm check:architecture rejects two decisions with the same number; whichever change reaches main second takes the next free number. The exact historical 0000 and 0007 pairs listed in the ADR catalogue retain their original identifiers and are the only exceptions. Before hand-off, select checks using Choosing checks. Documentation- and instruction-only edits use the focused documentation lane. Other changes require pnpm verify and the applicable real infrastructure lane. Use the configured dev box for infrastructure-dependent checks; follow the dev-box testing workflow. Report omitted PostgreSQL, browser, UAT, Minikube/k3s and deployment checks. Review generated diffs after pnpm generate:contracts and keep dependency locks committed. Generated credentials, tool caches and build outputs stay ignored. Checked-in generated web clients and schemas live beneath generated/ directories in their owning package. Update generators and imports when moving their outputs. Next.js requires each project's next-env.d.ts bootstrap at the project root; its transient route types and build output remain in ignored .next/ directories.
- [Domain-driven design](/reference/DDD.md): Execution Core owns the Run aggregate and its child Attempts. Site Coordination owns each configured environment's command stream, with commands and results as children of its stream aggregate. Site Runtime owns Site-local delivery progress, native recovery and member/workspace rules. Its private SQLite journals record local recovery facts; Run/Attempt authority remains in Execution Core and command/result authority remains in Site Coordination. It has no platform PostgreSQL database. Support owns the SupportCustomer aggregate, keyed by Pharos user ID, and the Ticket, TicketMessage and TicketSync aggregates. Choose further bounded contexts from the business language and authority they need, rather than from the three service names. A service is a deployment boundary; a bounded context is a model boundary. Pipeline Coordination owns the Pipeline aggregate and its ordered child steps. The domain records immutable Workload revision references, execution environments and advancement conditions. Repository references select a branch whose latest contents are fetched when each action prepares. Its coordinator records Run references through owner interfaces; Execution Core retains Run/Attempt authority. Internal start, read and cancellation-request handlers use explicit Project capability and exact Workload revision queries, with current access checked before receipt replay. PostgreSQL commits its aggregate snapshot, command receipt and pending coordinator work together. The coordinator saves stable step launches before submission, follows Runs through owner interfaces and consumes typed completion facts with atomic receipts and progress. Its work leases fence stale workers. Execution Core retains Run creation and retry authority. Workload Portfolio owns the Workload aggregate and its immutable, numbered revisions. Each revision keeps the execution specification and required inputs that Execution Core plans from. Other contexts read saved revisions through Workload Portfolio's published query interfaces and check member access themselves. SSH Access owns one MemberSshAccess aggregate per member and Site, with the member's keys as children. Each submission, resubmission or revocation issues a new key generation; Site results are fenced by generation, and applied changes count only after a configured propagation window. An aggregate owns one immediate consistency boundary. Entities retain identity; value objects compare by value. Children can change only through their aggregate root. Identify the invariant, command/query and owning fact before introducing persistence. Model aggregate boundaries from business consistency and independent lifecycles. If an immediate invariant spans proposed roots, reconsider that boundary. Changes to separate children still compete for their root's version. Workflows spanning aggregates have explicit allowed pending states and an owner for each recovery step. For example, an accepted launch awaiting Run creation is a valid pending state when its intent is durable and recoverable. Describe the conditions for progress, retry and terminal outcomes in the owning context. Commands use explicit canonical identity and expected versions. A retry with the same material input returns its recorded outcome. Reusing a command identity for different input is a conflict. Expected business rejection is typed and recorded; infrastructure outages and corrupt restored authority remain retryable failures. Duplicate or no-op transitions emit no new domain events. Keep transport deduplication separate from business uniqueness: each consumer has its own receipt, while a stable business key protects the same occurrence arriving under different message identities. Correlation IDs group related work; deduplication uses command/message identities and explicit business keys. Queries use named inputs and handlers, application-owned read models and separate read repository ports. They have no authoritative side effects. Owner adaptors decode stored or provider data and map it into those read models. Preserve root revisions when reading versioned authority, and make missing-resource behaviour explicit. Pagination is explicit: a query must return the complete collection or a traversable cursor. Domain code has no framework, transport, database, provider, implicit clock or random-number dependency. Time and identities arrive as explicit values. A domain event is a context-local fact. Application mapping decides whether it needs private durable delivery or a public versioned integration contract. Required reactions across aggregates happen asynchronously through the same receipt/outbox mechanism. Do not invoke a second aggregate's command handler inside the first command transaction. Learn facts from another context through its published integration events or query interfaces. Maintain consumer-owned projections where local reads are needed. Between services, use the versioned Protobuf contracts and messaging conventions defined in ADR 0046. Database co-location permits no foreign SQL access. External IDs remain opaque provider bindings; canonical Pharos identity is independent of them. Model ownership does not require local persistence. A supporting context can translate an authoritative third-party model through its own ports and adaptors without owning a database. Whenever it persists state, including delivery receipts or projections, that storage belongs exclusively to the context under the persistence rules. Go and Python use idiomatic native types while preserving the same consistency, rejection and delivery rules. Generated wire types remain in outer adaptor or contract packages. Provider-independent application DTOs remain context-owned. Static types describe valid shapes; boundary decoders must also validate external input and restored state at runtime. Malformed requests follow the boundary's validation policy. Corrupt stored authority remains a retryable state failure, distinct from an expected business rejection. Keep provider, transport and snapshot mappings in the owning context's adaptors; shared infrastructure carries technical mechanisms. Use British English and adaptor. Name domain/application modules after their context, adaptors after their capability/technology, and executable composition roots after the process they run. Classify driving/driven relative to the port. See ARCHITECTURE.md for the repository map and docs/adding-a-context.md for registration steps. Themelio's Python scaffold uses a central command executor around plain handlers and named write repository ports. An aggregate save stages private domain facts. An event reaction persists its receiving receipt and exact outgoing command bytes; a separate command consumer executes that command. Publication handlers map private facts into integration events. Realtime intent is explicit and commits with its owner transaction. These mechanisms introduce no Themelio business model. <a href="/source/DDD.md.txt">Source snapshot</a> · Revision f928834c8.
- [Testing](/reference/TESTING.md)
- [API-owned login and compatibility contracts](/reference/contracts/authentication/README.md): Status: implemented behind explicit installation enrolment. The public browser protocol is included in the assembled browser OpenAPI contract and generated web client. The API runtime guide records the route, worker and credential configuration. Retained Themelio requires API-owned authentication; the NextAuth bridge has been retired. Limited sessions receive only their own Party readiness notifications. The <a href="/source/services/beto/docs/party-project/authentication-transition.md.txt">transition plan</a> records the coordinated cutover and endpoint migration.
- [Beto shared contract base](/reference/contracts/beto/messaging/README.md): PR #723's contract generation and broker provisioning use the targeted realtime behaviour from PR #727. The integration starts at #723 commit bbd25b5ed, brings in main at 04fd56025, then merges #727 at b62067c6e.
- [Events contracts](/reference/contracts/events/README.md): Versioned integration and private-event envelope specifications. The shared catalogue declares published owner payloads; private events stay with their service. Party Governance's contract guide defines its first integration facts and consumer recovery requirements. Register typed owner codecs and authorised consumers before enabling delivery. <a href="/source/contracts/events/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [HTTP contracts](/reference/contracts/http/README.md): site-execution.v1.yaml defines the six Site execution start/cancel messages as OpenAPI components. Its paths is empty: 1C establishes the message contract; Site transport and delivery are implemented in Task 4. site-coordination.v1.yaml adds command drafts, paginated command reads and typed success/failure results. It references the six existing execution schemas. Site Coordination assigns sequences and stores commands/results in its private database. Its paths also remains empty until authenticated Runtime transport. Batch observations may include an optional bounded failure after terminal failure and confirmed cleanup. It carries stage, opaque native job/state and separate process/derived-step exit facts. Existing observations and stored outcomes retain their shape, and existing Protobuf Run fields carry the member summary without new wire identities. Upgrade Site API and Execution Core before the Runtime: older strict Site consumers reject the added field. This does not retroactively enrich already terminal Runs. site-ssh-access.v1.yaml declares the SSH access agent's pull, snapshot and acknowledge routes with native bearer authentication; the Site comes from the credential's grant. See the SSH Access context. scripts/generate_site_execution.py generates the three sets of strict Python transport models in Metallo's generated adaptors with pinned tooling. pnpm generate:contracts regenerates them; pnpm verify checks drift. Application DTOs have no generated model dependency. The owning Execution Core codec additionally checks cross-field rules, canonical identities, non-null fields and duplicate JSON keys. api.openapi.json is the Go API's browser-facing contract. Each business path references its owning context's fragment under contracts/<context>/http_api/, names that context in x-owner and has an operationId. Shared errors live in contracts/shared/http_api/; Execution Core's routes keep Themelio's {error, code} errors and per-parameter 400 body. scripts/http_contracts.py bundles the fragments into the API's embedded copy and generates the browser's tagged Orval client; pnpm generate:contracts regenerates both and pnpm verify checks drift. The API registers only declared business operations, each through a typed binding. Technical AAI paths use the same browser document and their dedicated API adaptor. An empty security list marks an operation as needing no session, and a list holding only CLIKey marks it operator-only: a browser session is refused with 403 operator_only. Commands are POST operations on their target's {id} and carry the shared headers in contracts/shared/http_api/parameters.json (Idempotency-Key, If-Match, X-Correlation-ID); they answer the shared CommandOutcome. Support's tickets, messages and customer links are in <a href="https://github.com/pharos-ai-factory/pharos-platform/tree/f928834c82b9d99e12575f9bd3f311c16c8f138b/contracts/support/http_api">contracts/support/http_api</a>. See the browser client guide. The Project member and invitation preview is an agreed standalone frontend contract with generated clients and Go API fixture bindings. Explicit development configuration serves its examples through the normal authenticated HTTP path. Its default API activation accompanies the backend implementation. llm-access.v1.yaml is owned by LLM Access and defines what the Waldur Site Agent sends for an order and what it reads back: one order message with a request ID, a result, and the allocation status that says when the gateway matches. scripts/generate_site_execution.py generates llm_access_http.py from it with the other owned HTTP contracts. Its paths are mounted only when LLM_ACCESS_INTAKE_KEY is set (the usage path also needs LLM_ACCESS_USAGE_KEY, and the members and member-revocation paths LLM_ACCESS_SEAL_KEY with key issuing), under their own credential (see the context README). <a href="/source/contracts/http/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [Marketplace requests, version 1](/reference/contracts/marketplace/messaging/README.md): Sources: proto/pharos/marketplace/v1/requests.proto (the envelope), commands.proto (the command inputs), queries.proto (the query inputs) and replies.proto (the query results). Commands answer with the shared Outcome. Run pnpm generate:contracts after edits. Generated Go bindings live in Beto's outer contract package, Marketplace's messaging adaptor and the API's messaging adaptor. PlaceOrder places one Order for a Project. It carries project_id and pack_id, both required. The Order ID is the command's target. The buyer is the command's actor, which the API sets from the session, and the price comes from the catalogue, so neither is in the payload. The Marketplace README lists its refusals. ListProjectOrders lists one Project's Orders for one of its buyers. It carries project_id (required), after_order_id and page_size (default 50, at most 100). The caller is the request's verified actor. The reply's ProjectOrderPage holds each Order as it is stored now, ordered by Order ID, with its version as revision; next_after_order_id is set when another page follows. The provider plan reference is not part of it. A refusal answers with a RequestError code. The envelope follows the new-owner numbering: the request's command = 10 and query = 11, the command's place_order = 10, the query's list_project_orders = 10, and the reply's outcome = 10, error = 11 and order_page = 12. The contract is declared in contracts/beto/messaging/catalogue.json with the API as its consumer. broker.json declares the query and command requests with the API as their only caller, allowed to carry the actor. <a href="/source/contracts/marketplace/messaging/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [MCP contract](/reference/contracts/mcp/README.md): The experimental MCP gateway exposes every x-owner business operation in the assembled Go API OpenAPI specification. The reviewed <a href="/source/contracts/mcp/manifest.json.txt">manifest</a> gives each operation one stable tool name, scope and effect classification. It currently contains 54 operations, including writes, deletions and the two operator-only Support routes. The manifest is deliberately explicit. Generation fails when an API operation is added or removed without a parity decision, a scope crosses its owner, a write is classified as a read, or an input admits an authentication header. Authentication/session endpoints, internal relays, diagnostics and realtime connections do not become tools. Run pnpm generate:contracts after editing sources or generators. pnpm check:http checks the bundled specification, generated tool schemas, Go DTOs, HTTP client and closed typed dispatcher. Generated files live in services/api/adaptors/{mcpcontract,mcpclient}/generated.
- [Private Party authentication contracts](/reference/contracts/party_governance/authentication/README.md): Party Governance owns canonical identity resolution and registration. The API authentication edge verifies the browser's OIDC transaction and presents enrolled evidence through this private broker protocol. Session issuance, AUP acceptance and provider account readiness remain separate decisions.
- [Party profile HTTP contract](/reference/contracts/party_governance/http_api/README.md): GET /api/v1/me/profile and PUT /api/v1/me/profile use the current canonical Party session and the existing acceptable-use gate. Technical operator and CLI credentials cannot act as a Party. The API forwards the verified Party ID and authentication generation through its trusted broker caller metadata; Beto rechecks them before reading or changing the profile. PUT replaces every editable field. Empty strings clear optional details. The Idempotency-Key UUID names the command and If-Match names the profile revision, including zero for an absent profile. Retrying an uncertain response preserves both headers and the body. An explicit X-Correlation-ID may group attempts. Authentication generation and profile revision have separate purposes. GET /api/v1/parties/{partyId}/profile is public. Beto returns only profiles whose owner selected public; absent and private profiles both return not_found. The API additionally rejects mismatched identity or unsafe reply data and refuses to serialise a private profile publicly. Responses contain the canonical Party ID, profile revision and editable fields. Source AAI attributes, provider IDs, email and authentication state are excluded. An absent self profile has revision zero, private visibility and empty fields. Social fields contain bounded handles or paths for the named provider. Clients construct links using fixed provider hosts. The website field allows only absolute HTTP(S) URLs without credentials. Text limits count Unicode code points; only the biography permits line feeds and tabs. The owner contract defines the exact <a href="/source/contracts/party_governance/http_api/schemas.json.txt">field limits</a>. <a href="/source/contracts/party_governance/http_api/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [Party Governance contracts](/reference/contracts/party_governance/messaging/README.md): These version 1 contracts define Beto's canonical Party boundary. They are available to API and Metallo consumers through independently generated bindings. The first implementation was staged as contracts, domain and application commits. Owner persistence, broker registration, scoped callers and integration publication now compose those layers. <a href="/source/contracts/party_governance/authentication/proto/pharos/party_governance/authentication/v1/authentication.proto.txt">Private authentication ingress</a> has its own protocol because registration has no canonical target yet. Legacy Themelio login, provider writers and AUP remain active. The role-separated sources under proto/pharos/party_governance/v1/ retain the existing package, message names, field numbers and presence: requests.proto holds public wrappers and errors, queries.proto inputs, replies.proto results, and commands.proto application commands. Run pnpm generate:contracts after source changes. The specialised HTTP projection uses scripts/generate_api_party.py and services/api/adaptors/http/party/generated; its validation remains separate from the generic HTTP mapping output.
- [Private provider-binding ingress](/reference/contracts/party_governance/provider_binding/README.md): Party Governance records one canonical provider binding from qualified ownership evidence. The API relays the retained compatibility writer's request through request.party_governance.provider_binding.v1. Only its enrolled broker principal can publish this kind. Browser sessions cannot call this private operation. The <a href="/source/contracts/party_governance/provider_binding/proto/pharos/party_governance/provider_binding/v1/provider_binding.proto.txt">wire contract</a> and <a href="/source/contracts/authentication/legacy.openapi.yaml.txt">private HTTP relay</a> preserve command_id, correlation_id, party_id, proof_ref and the complete immutable signed assertion. Transport request IDs may change. The proof reference is sha256: plus the lowercase SHA-256 digest of the exact assertion bytes. Unknown/duplicate fields, malformed UUIDs and unsupported versions are rejected. Replies contain one closed typed outcome and its exact transport correlation. The <a href="/source/contracts/authentication/provider_binding_assertion.schema.json.txt">signed ownership fact</a> has header alg: EdDSA, typ: pharos-party-provider-binding-v1+jwt and an enrolled kid. Its issuer is themelio-compat; its audience is party_governance.provider_binding.v1. The signed jti, correlation_id and party_id match the request. It also binds the independently allocated canonical binding_id, expected_revision, auth_version, exact provider/instance/external identity, accepted API authentication evidence reference, exact AAI issuer/subject and integer-second verified_at. This is a durable ownership fact without an expiry claim. It establishes no authentication or provider access grant. The writer first resolves accepted API evidence and verifies the provider's exact federated account identity. Email, username, display attributes and a caller-selected external ID cannot establish association. It persists the complete command and signed fact before sending. Lost acknowledgement is resolved by repeating those exact bytes. Beto enrols the writer's public key with exact attester, provider, instance and AAI issuer scope. Every intake, including receipt replay, checks current canonical Party identity, eligibility and authentication generation against the fact. verified_at may be at most thirty seconds in the future. An old fact cannot acquire a later generation after disablement/re-enablement. The existing binding transaction enforces expected revisions and retained external identity uniqueness, committing its root, receipt, outcome and event intents together. Retired bindings reject reuse and reassignment. Successful recording returns the current established binding with its positive JSON-safe revision. An unavailable outcome can follow a successful commit; it does not permit a new command or another binding ID. The private read relay uses the existing canonical owner query, verifies the requested Party and enrolled provider scope, and preserves established/retired/missing semantics. The compatibility account worker marks ready only after a committed established binding. The legacy AUP policy read and credential issuance resolve that binding through this owner query before provider effects. The writer holds THEMELIO_COMPAT_BINDING_SIGNING_KEY (or _FILE) and THEMELIO_COMPAT_BINDING_SIGNING_KEY_ID. Beto holds only its verification key. HTTP BindingWriter and BindingReader credentials are separately scoped and never admitted by public ingress. scripts/generate_party_provider_binding.py reproduces API/Beto wire bindings. Generated output and schemas declare the protocol; activation additionally needs the explicitly enrolled owner worker, broker permissions and private API relay. <a href="/source/contracts/party_governance/provider_binding/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [Persistence contracts](/reference/contracts/persistence/README.md): Shared technical current-state, receipt, outbox and realtime schema foundations. Contexts own separate databases and additive migration manifests. The shared 0001 and 0002 files are frozen. Beto's service-owned services/beto/contracts/persistence/0003_delivery.sql adds private-command and reply outboxes plus explicit snapshot/rejection publication. Its additive 0004_reply_routes.sql preserves the originating API connection's address with each command reply; existing rows retain the legacy owner route. The administrator applies and checksums these before Beto context migrations. Metallo's additive 0004_realtime_audiences.sql lets one revision record a separate publication for each principal without weakening event identity. Runtime only verifies the checksums. Storage-free Project has no migration enrolment. <a href="/source/contracts/persistence/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [Project members and invitations](/reference/contracts/project/http_api/README.md): The default Go API serves these reads through the Project owner. The development fixture mode remains available through explicit configuration. Generated clients and disposable fixture checks do not establish support in a deployed backend. Both reads require a current Party identity and a canonical Project UUID. Browsers use the Party session; the enrolled MCP gateway can use its operation-bound assertion under the same current owner permission checks. Responses use Cache-Control: private, no-store. Neither operation accepts query parameters. These reads add no mutation, realtime subscription or automatic polling.
- [Project member and invitation contract preview](/reference/contracts/project/http_api/member-invitation-preview/README.md): Development fixture contract. This standalone <a href="/source/contracts/project/http_api/member-invitation-preview/api.openapi.json.txt">OpenAPI specification</a> and <a href="/source/contracts/project/http_api/member-invitation-preview/schemas.json.txt">schemas</a> let frontend work proceed with generated clients and fixture responses. The same shapes are now present in the active HTTP contract; the isolated preview client and fixture assets remain available for development. The preview supplies contract sources, two isolated Orval outputs and Go API fixture bindings. An explicitly configured development API serves these examples through the normal HTTP/session path. The default API uses the real Project owner reads. The field names are agreed independently of the broader #1066 naming work. Both reads require a current Party identity and a canonical Project UUID. Browsers use the Party session; the enrolled MCP gateway can use its operation-bound assertion. The preview retains the active contract's authentication declarations and current owner checks. Responses use Cache-Control: private, no-store. Neither operation accepts query parameters. These reads add no mutation, realtime subscription or automatic polling.
- [Project queries and integration events, version 1](/reference/contracts/project/messaging/README.md): Sources: proto/pharos/project/v1/requests.proto (the envelope), queries.proto (directory and canonical query inputs) and replies.proto (their typed results). Run pnpm generate:contracts after edits. Generated Go bindings live in Beto's outer contract package and the API's messaging adaptor. Metallo receives generated Python query/event bindings. Each service compiles schemas into its own package; consumer wiring remains explicit and is deferred in this slice. GetProjects declares its owner, public visibility and Projects success response through the pharos.contracts.v1.contract schema option. These build-time annotations preserve the existing wire fields. Generation validates the association against ProjectReply.payload, including its RequestError alternative. The retained directory operation is GetProjects. A ProjectRequest has contract version 1, context project, a UUID request_id and query.get_projects. The query has no parameters. A ProjectReply repeats the version, context and request identity, with either projects.items or error.code.
- [Realtime contracts](/reference/contracts/realtime/README.md): Publication is the common typed browser-message envelope. It carries one DeliveryTarget: canonical Project ID, canonical Party ID or principal subject. The publisher selects the audience and payload; the restricted gateway validates the publisher's owner and permitted target kinds before deriving the Centrifugo channel. Channels are absent from publisher APIs. The projection field retains its wire number and contains a registered context-specific Protobuf message. operation_rejected carries command and correlation identity plus the recorded rejection and requires a principal target. Add fields compatibly, regenerate with pnpm generate:contracts and verify stored bytes separately from generated-file drift. Execution Core's RunProjection is delivered to its canonical Project audience. The Run ID, Project ID and revision remain in the typed message so clients can apply only newer state. Its additive member_sign_in_required field marks a terminal consent refusal. Clients display one sign-in-again notice per Run to the matching member_id; duplicate delivery or another Run's progress does not add a second notice. Older stored projections omit the field and decode to false. The existing Project audience and wire identities remain unchanged. Support's SupportTicketProjection and SupportMessageProjection (support.proto) are delivered to the owning user's principal target; its customer and ticket-sync aggregates are private and never published. Party Governance's PartyProfileProjection (party_governance.proto) carries the saved profile and social links to the owning principal. Its aggregate kind is party_profile, its aggregate ID is the canonical Party ID and the envelope revision orders updates. Private profiles use the same owner-only delivery. Public readers use the visibility-filtered HTTP query. The Party readiness channel never carries profile data. The API's technical authentication authority publishes SessionReadinessChanged (authentication.proto, envelope owner authentication, kind session_readiness) to the canonical Party target. Its revision increases per Party. This is an invalidation notification: each browser reads its own current session snapshot. It carries no session reference, provider identifier, credentials or permission. An additive payload leaves the existing envelope and projection wire numbers unchanged. Old clients reject this unrecognised message; deploy the new browser and API/gateway together for automatic readiness progression. <a href="/source/contracts/realtime/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [SSH Access contracts](/reference/contracts/ssh_access/README.md): The SSH Access context answers the browser's #678 routes through the API. The routes and their JSON schemas are in <a href="/source/contracts/ssh_access/http_api/ssh_access.paths.json.txt">http_api/</a>; pnpm generate:contracts bundles them into the API's OpenAPI document under /api/v1/ssh-access/sites.
- [Support requests, version 1](/reference/contracts/support/messaging/README.md): Source: <a href="/source/contracts/support/messaging/proto/pharos/support/v1/requests.proto.txt">requests.proto</a>, which imports <a href="/source/contracts/support/messaging/proto/pharos/support/v1/commands.proto.txt">commands.proto</a> and <a href="/source/contracts/support/messaging/proto/pharos/support/v1/queries.proto.txt">queries.proto</a>. Run pnpm generate:contracts after edits. Private commands are declared alongside public commands in commands.proto; their visibility annotations keep them outside the API's published operations. Commands carry the shared CommandMetadata: command ID (the Idempotency-Key), target aggregate, expected version, correlation and the acting user (actor_id). The API sets user_id in queries and actor_id in commands from the session; a browser never supplies either. Command replies carry the shared Outcome; query replies carry their response or the shared RequestError. Broker roles and permissions are generated from <a href="/source/contracts/support/messaging/broker.json.txt">broker.json</a> into services/beto/services/topology/permissions.generated.json. <a href="/source/contracts/support/messaging/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [Pharos documentation site](/reference/docs/README.md): From the repository root: Preview binds to http://127.0.0.1:3001. pnpm docs:dev prepares sources and starts the development server. After editing a source, run pnpm --filter @pharos/docs prepare:site again, or restart docs:dev, to refresh the generated tree.
- [Add a context](/reference/docs/adding-a-context.md): Decide its owning service, business language, aggregate boundary and command/ query outcomes. Create services/<service>/contexts/<owner> with domain, application and driving/driven adaptor modules. The domain has no technology imports. A co-located context has independent database authority.Register { "owner": "<owner>", "service": "<service>" } in contracts/contexts.json. A read-only context without owned storage declares "persistence": "none"; it needs no database or migration manifest. Site-local SQLite contexts declare "persistence": "sqlite" and own their private journals without platform PostgreSQL credentials or migration enrolment. PostgreSQL contexts retain the default mode and add a migration manifest and SQL under adaptors/persistence/migrations. Runtime credentials cannot migrate schemas. Provision its database with the shared baseline schema and a restricted role, then apply scripts/migrate.py; schema checks reject owner/checksum mismatch. Beto also applies its service-owned delivery migration before the context migrations. Storage-free registrations stay outside this enrolment. A Beto context without broker contracts, events or subscriptions declares "messaging": "none"; it receives no broker user or credential.Add context-specific typed Protobuf payloads, independent application DTOs, event metadata in contracts/events/catalogue.json, and the owning service's transport payload encoder/decoder. Root envelopes stay business-neutral. Declare Beto request contracts in contracts/beto/messaging/catalogue.json with the owner's queries file pharos/<owner>/v1/queries.proto, its commands file pharos/<owner>/v1/commands.proto (either may be null, not both) and their binding consumers (api and/or metallo). Declared owners must differ even with underscores removed, since generated Go names drop them: model2 and model_2 would both generate Model2Query and model2v1. The contract is split by role: requests.proto holds the <Owner>Request, <Owner>Query, <Owner>Command and <Owner>Reply wrappers and RequestError; queries.proto the query inputs; replies.proto what queries return; and commands.proto the commands. Each file declares package pharos.<owner>.v1 and go_package github.com/pharos-ai-factory/pharos-platform/services/beto/contracts/requests/generated/pharos/<owner>/v1;<owner>v1, with underscores dropped from the final <owner>v1. Generation checks both and that each file holds exactly its role's messages. The files must exist under contracts/<owner>/messaging/proto/ and in no other context's proto root, and an undeclared role file is refused. Number a new owner's envelope this way: the request's command = 10 and query = 11; the reply's outcome = 10, error = 11 and results from 12; and each wrapper's alternatives from 10. Existing Project and Party Governance envelopes retain their field numbers; their frozen wire fixtures qualify compatibility separately. A declared contract generates Beto and API bindings before registration and receives no broker authority. A registered owner with requests names them and their callers in contracts/<owner>/messaging/broker.json; each kind needs its declared file and no caller is implied. Governance authenticates RabbitMQ UserId against generated owner-specific policies. Only the API may carry actors or use private authentication/provider-binding protocols; allow_actor defaults to false. Production callers remain API-only until downstream enrolment is reviewed and qualified. Regenerate and review the scoped topology permissions. Host provisioning reads the generated runtime_bindings; caller pharos_<name> uses <NAME>_BROKER_URL. Add each binding to the projected Secret of Argo's broker provision Job; beto_broker_deployment.py checks the list. Prove allowed delivery and refused authority. In Beto, generate command/query values, responses and owner mappings from annotated Protobuf. Declare private events in the service's event catalogue and consumers in services/beto/contracts/subscriptions.json. The generator produces typed event codecs, subscription names, paired command queues and replay membership. See the Beto adaptor map.Add a typed browser projection, its registered encoder and scoped gateway publisher credential only when the UI needs live state. Persist exact bytes and revision atomically; register server-side channel authorisation and owner reconciliation queries before mounting the feature. In Metallo, explicitly register None for an internal aggregate whose browser feature is deferred; persistence then saves its state and receipts without realtime rows. Unknown projection registrations remain errors. Beto and Themelio handlers explicitly publish through RealtimeBusPort; their requested snapshot or rejection must match committed state or recorded outcome.Compose its ports, driving adaptors and workers in the service's services folder. Keep DI registrations here and inject plain ports into application handlers. Go persistence uses support.Open("<owner>") and explicit consumer binding; Python uses its owning service's context/subscription registry. Themelio uses Dependency Injector under services/composition; extend its empty contract generator before registering the first context. Its service-local delivery migrations are applied by the administrator. See the <a href="/source/services/themelio/README.md.txt">Themelio scaffold guide</a>. No foreign implementation import or synchronous second-root command is permitted. Beto commands use services/commandregistration and a handler factory with a per-invocation owner write repository. The shared transaction owns receipts, version checks and outgoing intent. Integration publication and durable command reactions run through their declared subscribers. List each Beto owner whose request worker the binary composes, with its request kinds, in services/beto/services/beto/di/owners.json. Broker generation and provisioning refuse a registered owner with requests that the file does not list, and a listed owner that is not registered; a DI test checks the file against the composed workers.When a browser feature needs access, explicitly bind its public route in the API, keeping its backend key separate from browser credentials. A Beto owner's HTTP translation is enrolled in <a href="/source/services/api/adaptors/http/mappings/README.md.txt">services/api/adaptors/http/mappings/&lt;owner&gt;.json</a>; it needs the owner's catalogue entry for api and creates no route. The enrolment check also runs Party Governance's existing specialised generator. Add workload secret references, restricted broker topic permissions, the administrator-only topology bootstrap and exact NetworkPolicy edges. No database or broker admin key enters an application.Regenerate contracts/checksum metadata, add owner-native behaviour tests and real transaction/delivery/recovery tests, then run all applicable lanes from TESTING.md. Add browser multi-window/history-gap tests for live projections. The skeleton deliberately makes registration explicit. A new folder or catalogue entry alone grants no routes, queue, credential, publisher, subscription or business authority. API command bindings Declare each POST route and its JSON schema in the owner's HTTP contract. Register a CommandBinding using the API's independently generated owner request/reply types. It reads {id}, Idempotency-Key, If-Match, optional X-Correlation-ID and the validated JSON body. The API supplies the authenticated session subject (or the separate provisioned CLI actor) as actor_id; headers/body cannot choose it. The typed request uses a fresh transport ID on every attempt while retaining the command ID. Owner errors and committed outcomes map to HTTP status codes. RouteCommandBinding also serves DELETE routes without a body, builders that read other route values, and creates named by their Idempotency-Key (KeyTarget). Queries that check member access carry the same subject in QueryMetadata. Beto validates actor metadata and exposes it as CommandContext.ActorID to the handler. Authorisation remains owner policy. Present actors contribute to the idempotency fingerprint; changing actor produces a conflict. Empty actors are reserved for trusted internal/legacy commands and convey no user authority. Project's published request surface remains query-only; technical fixtures exercise this public command path. <a href="/source/docs/adding-a-context.md.txt">Source snapshot</a> · Revision f928834c8.
- [CI runner roles](/reference/docs/ci-runners.md): The ARM64 CI fleet uses three disjoint scheduling roles. The desired allocation is recorded in <a href="/source/devops/ci-runner-roles.json.txt">devops/ci-runner-roles.json</a>. Control workflows select their role alongside self-hosted, Linux, ARM64 and pharos-ubuntu-24.04. General and infrastructure workflows select portable pools described below, preserving the ARM role allocation. Control workers deliberately omit pharos-ci, fencing them from older workflows that selected the whole CI fleet. This keeps ordinary verification from consuming the infrastructure workers and keeps short control jobs from joining the test queue. Each runner still executes one Actions job at a time. Role labels affect subsequent assignments; existing jobs finish normally. Older workflow revisions using broad labels can still consume general and infrastructure workers while those runs drain.
- [Contexts, responsibilities and ownership](/reference/docs/context-catalogue.md): Reviewed against integration/current on 9 November 2026; the exact source is in <a href="/source/docs/integration-baseline.json.txt">the integration baseline</a>. This catalogue describes registered ownership and current integration boundaries. Context READMEs retain operation details, activation requirements and qualification evidence. <a href="/source/contracts/contexts.json.txt">contracts/contexts.json</a> registers 14 contexts. ARCHITECTURE.md records the shared architectural rules. A registered context may still have development stand-ins or disabled processes.
- [DD-0038: Platform Edge and Mesh](/reference/docs/dd/0038-platform-edge-catalogue.md): Related ADR: ADR-0038 — Platform service catalogue and shared host edge. Status: Draft for review. The A1/A2 implementation and owned development deployment are qualified; wider deployment and approval remain pending. Date: 2026-10-13. Author: Implementation/documentation prepared for ikons; platform review pending.
- [DD-0039: Development Argo CD Bootstrap and Ownership](/reference/docs/dd/0039-dev-box-argocd-ownership.md): Related ADR: ADR-0039. Status: Draft for review. The authorised dev-ikons implementation is qualified; wider adoption, review and production approval remain pending. Date: 2026-10-13.
- [DD-0041: GitLab and Waldur as Argo CD Services](/reference/docs/dd/0041-gitlab-waldur-services.md): Related ADR: ADR-0041. Status: Draft for review. Qualified on the authorised dev-ikons dev box; wider adoption, staging and production remain pending. Date: 2026-10-14. Waldur pin updated to 8.1.4 on 2026-11-07.
- [DD-0043: The Retained Themelio as the themelio-compat Service](/reference/docs/dd/0043-themelio-compat-service.md): Related ADR: ADR-0043. Status: Draft for review. Qualified on the authorised dev-ikons dev box; other dev boxes remain pending. Staging and production do not run it. Date: 2026-10-14.
- [DD-0044 — The shared LLM stack as the llm service](/reference/docs/dd/0044-llm-stack-service.md): Related ADR: 0044. Status: draft for review. Date: 20 October 2026.
- [DD-0045: Development Loop Interface: Image Catalogue, Deployment Receipt and Takeover Lease](/reference/docs/dd/0045-development-loop.md): Related ADR: ADR-0045. Status: Draft for review. The lease, mode, expiry and catalogue decisions were reviewed with danias before implementation; dev-box qualification follows each pull request. Date: 2026-10-14.
- [DD-0051 — Distributed inventory setup](/reference/docs/dd/0051-distributed-inventory-setup.md): Related decision: ADR-0051. Status: Implementation in progress; evidence below distinguishes code from live qualification. Date: 20 October 2026.
- [DD-0055 — Member SSH access: browser surface, laboratory and delivery](/reference/docs/dd/0055-hpc-ssh-key-provisioning-browser-and-operations.md): Continuation of DD-0055 — Member SSH access to Icarus-hpc; section numbers are kept so existing references stay valid.
- [DD-0055: Member SSH access to Icarus-hpc](/reference/docs/dd/0055-hpc-ssh-key-provisioning.md): Related ADR: ADR-0055. Status: Draft for review. The dev-ikons laboratory and e2e lane are qualified; Icarus-hpc qualification remains outstanding. Date: 2026-10-21. Terminology (Pharos platform, Icarus-cloud, Icarus-hpc, Site agent, access agent, Site Runtime) is defined in ADR-0055.
- [DD-0058 — Complete distributed services composition](/reference/docs/dd/0058-distributed-services-composition.md): Related decision: ADR-0058. Status: Implementation in progress; deterministic evidence is recorded below. Date: 24 October 2026.
- [DD-0059: Production Icarus-hpc provisioning and run installation](/reference/docs/dd/0059-icarus-hpc-provisioning-run-installation.md): Related ADR: ADR-0059. Status: Draft. Deterministic installation and gate implementation is in review; production activation and native qualification are outstanding. The installer is plain bash since 3 November 2026 (§1–§4, §8); the earlier Ansible role is removed. Date: 2026-10-25. Amended 2026-11-03.
- [DD-0060: Development Seeds for Waldur, GitLab and SSH Access](/reference/docs/dd/0060-development-seeds.md): Related ADR: ADR-0060. Status: Draft for review, with dev-box evidence on Waldur 8.1.3 and GitLab 19.4.2. Date: 2026-10-22.
- [DD-0067: Kubeflow GitOps composition](/reference/docs/dd/0067-kubeflow-gitops-composition.md): Related ADR: ADR-0067. Status: Draft for review. No manifests are implemented by this document. Date: 2026-10-29.
- [Design documents](/reference/docs/dd/README.md): Design documents describe implementation, migration, verification and recovery for the related architecture decision. Decisions retain their canonical paths under ../decisions; a companion DD uses the same number and filename here. Links run in both directions so a design cannot be mistaken for an independent architectural decision. Implementation evidence and review approval are separate. A deployed development design may remain Draft while wider adoption and production qualification are pending. DD-0038 — Platform edge and mesh, related to ADR-0038.DD-0039 — Development Argo CD bootstrap and ownership, related to ADR-0039; draft for review, with owned development implementation and qualification evidence.DD-0041 — GitLab and Waldur as Argo CD services, related to ADR-0041; draft for review, with dev-box qualification evidence.DD-0043 — The retained Themelio as the themelio-compat service, related to ADR-0043; draft for review, with dev-box qualification evidence.DD-0044 — The shared LLM stack as the llm service, related to ADR-0044; draft for review.DD-0045 — Development loop interface, related to ADR-0045; draft for review.DD-0055 — Member SSH access to Icarus-hpc, with browser surface, laboratory and delivery, related to ADR-0055; draft for review.DD-0058 — Complete distributed services composition, related to ADR-0058; implementation in progress, with deterministic evidence.DD-0059 — Production Icarus-hpc provisioning and run installation, related to ADR-0059; draft for review, with deterministic installer evidence and production activation gated.DD-0060 — Development seeds for Waldur and GitLab, related to ADR-0060; draft for review, with dev-box evidence.DD-0067 — Kubeflow GitOps composition, related to ADR-0067; draft for review, with no manifests implemented yet. <a href="/source/docs/dd/README.md.txt">Source snapshot</a> · Revision f928834c8.
- [0000-agentic-system](/reference/docs/decisions/0000-agentic-system.md)
- [ADR-0000: Friends & Family MVP Scope](/reference/docs/decisions/0000-mvp-friends-and-family.md): Status: Proposed Date: 2026-05-14 Deciders: TBD Relates to: Product Requirements (Core), Functional Requirements, Architecture Overview
- [ADR-0001: LFS Storage Quota Enforcement via Custom Proxy](/reference/docs/decisions/0001-lfs-storage-quota-enforcement.md)
- [ADR-0002: Architecture Documentation Update](/reference/docs/decisions/0002-architecture-docs-update.md)
- [ADR-0003: Waldur Proposal Workflow](/reference/docs/decisions/0003-waldur-proposal-workflow.md): Status: Proposed Date: 2026-05-07 Deciders: TBD Relates to: ADR-0000 (Friends & Family MVP) — "Proposal Workflow" exploration item; Waldur Installation & Management (future ADR) — functional requirements FR-PRO-01 through FR-PRO-04.
- [ADR-0004: Xet Content-Addressed Storage for ML Artifacts](/reference/docs/decisions/0004-xet-content-addressed-storage.md)
- [ADR-0005: AAI → Waldur User Synchronization](/reference/docs/decisions/0005-aai-waldur-user-sync.md): Status: Accepted Date: 2026-05-13 Deciders: TBD Relates to: ADR-0003 (Waldur Proposal Workflow) — users must exist in Waldur before they can be added to projects and orders.
- [ADR-0006: Automated Data Seeding Strategy](/reference/docs/decisions/0006-automated-seeding-strategy.md): Status: Accepted Date: 2026-05-14 Deciders: TBD Relates to: ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)
- [ADR-0007: GPU Slicing and Quota Enforcement](/reference/docs/decisions/0007-gpu-slicing-and-quota-enforcement.md): Status: Draft Date: 2026-05-22 Deciders: TBD
- [ADR-0007: Stress Test Strategy](/reference/docs/decisions/0007-stress-test-strategy.md): Status: Draft Date: 2026-05-22 Deciders: TBD Relates to: ADR-0000 (Friends & Family MVP) — stress testing is a prerequisite for MVP confidenceIssue #85 (Observability) — monitoring infrastructure needed during test executionIssue #96 (Define stress test strategy) — parent work item
- [ADR-0009: Shared LLM Inference Backend](/reference/docs/decisions/0009-shared-llm-inference-backend.md): Status: Draft Date: 2026-05-22 Deciders: Platform developers (confirmed 2026-06-30) Relates to: ADR-0000 (Friends & Family MVP)ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)ADR-0010 (LLM Gateway and Project Token Quotas)ADR-0011 (Chat UI and Client Access for Shared LLM Services)
- [ADR-0010: LLM Gateway and Project Token Quotas](/reference/docs/decisions/0010-project-scoped-llm-gateway-token-quotas-api-access.md): Status: Draft Date: 2026-05-22 Deciders: Platform developers (confirmed 2026-06-30) Relates to: ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)ADR-0009 (Shared LLM Inference Backend)ADR-0011 (Chat UI and Client Access for Shared LLM Services)ADR-0020 (LLM Access Membership Source of Truth)
- [ADR-0011: Chat UI and Client Access for Shared LLM Services](/reference/docs/decisions/0011-human-web-ui-and-agent-client-access.md): Status: Draft Date: 2026-05-22 Deciders: Themelio developers (confirmed 2026-06-30) Relates to: ADR-0009 (Shared LLM Inference Backend)ADR-0010 (LLM Gateway and Project Token Quotas)ADR-0003 (Waldur Proposal Workflow)ADR-0005 (AAI → Waldur User Synchronization)ADR-0020 (LLM Access Membership Source of Truth)
- [ADR-0012: Job and Deployment Management System](/reference/docs/decisions/0012-job-deployment-management-system.md): Status: Proposed Date: 2026-06-02 Deciders: TBD
- [ADR-0013: actions.yaml Automation System](/reference/docs/decisions/0013-actions-yaml-automation-system.md): Status: Proposed Date: 2026-06-03 Deciders: TBD Relates to: ADR-0012 (Job and Deployment Management System) — actions.yaml builds on the PostgreSQL persistence layer
- [ADR-0014: GitLab CI Runner on Dedicated AWS EC2 Instance](/reference/docs/decisions/0014-gitlab-ci-runner-ec2-instance.md): Status: Accepted Date: 2026-06-02 Deciders: Thanasis Katsadas, Panos Louridas Relates to: ADR-0015 (GitLab CI Runner EC2 Instance — Ansible Onboarding) — covers Ansible provisioning, runner installation, and GitLab registration once the instance decided here is running.
- [ADR-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding](/reference/docs/decisions/0015-gitlab-ci-runner-ec2-ansible-onboarding.md): Status: Proposed Date: 2026-06-03 Deciders: Thanasis Katsadas, Dimitris Katsimpris Relates to: ADR-0014 (GitLab CI Runner on Dedicated AWS EC2 Instance) — this ADR addresses the operational onboarding of the instance type and purpose decided there. Implementation: DD-0015: GitLab CI Runner EC2 Instance — Ansible Onboarding
- [ADR-0016: Declarative Database List in the Ansible postgres Role](/reference/docs/decisions/0016-declarative-database-list-in-postgres-role.md): Status: Proposed Date: 2026-06-04 Deciders: Thanasis Katsadas
- [ADR-0017: Shared Themelio Database with Decoupled Schema Migrations](/reference/docs/decisions/0017-shared-themelio-database-decoupled-migrations.md): Status: Proposed Date: 2026-06-08 Deciders: TBD Relates to: ADR-0012 (Job and Deployment Management System) — decided the PostgreSQL job/deployment store this database backs.ADR-0016 (Declarative Database List in the Ansible postgres Role) — the declarative CNPG provisioning this database rides on.
- [ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads](/reference/docs/decisions/0018-queued-status-and-over-capacity-handling.md): Status: Proposed Date: 2026-07-02 Deciders: TBD
- [ADR-0019: Container Image Tag Management for GitOps Deployments](/reference/docs/decisions/0019-container-image-tag-management.md): Status: Draft Date: 2026-06-26 Deciders: Athanasios Katsadas, TBD
- [ADR-0020: LLM Access Membership Source of Truth](/reference/docs/decisions/0020-llm-access-membership-source-of-truth.md): Status: Draft Date: 2026-06-30 Deciders: Themelio developers (confirmed in session, 2026-06-30) Relates to: ADR-0005 (AAI → Waldur User Synchronization)ADR-0010 (Project-Scoped LLM Gateway, Token Quotas, and API Access)ADR-0011 (Human Web UI and Agent Client Access)
- [ADR-0021: Themelio Command-Line Interface](/reference/docs/decisions/0021-themelio-cli.md): Status: Draft Date: 2026-07-06 Deciders: TBD Relates to: ADR-0011 (Human Web UI and Agent Client Access) — the CLI is a new programmatic client alongside the web UI; ADR-0012 (Job and Deployment Management System) and ADR-0013 (actions.yaml Automation System) — the CLI drives and observes the jobs/deployments and actions those ADRs introduced.
- [ADR-0023: DRA GPU Allocation with Extended Resource Names](/reference/docs/decisions/0023-dra-gpu-extended-resource-names.md): Status: Accepted Date: 2026-07-06 Deciders: Themelio developers Relates to: ADR-0007 (GPU Slicing and Quota Enforcement) - decided the GPU slicing and quota model; this ADR decides how Themelio should consume NVIDIA DRA while preserving that model.
- [ADR-0025: Scoped Istio Authorization Policies](/reference/docs/decisions/0025-scoped-istio-authorization-policies.md): Status: Draft Date: 2026-07-16 Deciders: TBD Related: ADR-0024 Baseline L3/L4 Network Segmentation (referenced in the original record; absent from the archived tree)
- [0034 — fresh service-first skeleton](/reference/docs/decisions/0034-fresh-skeleton.md): Status: Accepted. Date: 30 September 2026. A new fresh branch from main, replacing the existing application with an empty skeleton based on the sibling multilingual reference. The branch starts at main 8046ea0f1 in the existing repository and is intended to merge back into main. The complete previous main tree is moved with git mv into legacy/, preserving its exact blobs and modes. The active root is replaced by the skeleton. The archive is excluded from active builds and checked against its recorded Git inventory. The original working checkout, its uncommitted work and running installations remain independent of this branch. The selected topology is Next.js Web, Go API, Python Metallo, Go Themelio and Go Beto. Each domain service owns direct service-local folders, including an empty contexts directory. Preserve framework-independent cores, one-root transactions, canonical identity, receipts, outboxes, retry/dead-letter/replay, typed Protobuf contracts and server-authorised Centrifugo. No production business authority is assigned by these scaffolds. The multilingual reference contributes independently built runtime foundations; its demo contexts, business payloads, routes and UI are removed. Shared envelopes and empty codec registries provide extension points, with no default permissive payload decoder or channel authority. Current-state persistence remains the choice. Infrastructure source is architecture-reset at b228782dd, rather than main. Adapt its pinned GitHub Actions, ARM64 runner labels and Rocky development k3s bootstrap. The k3s role retains checksum-verified downloads, SELinux, secret at-rest encryption, firewall controls and guarded persistent mounts. Separate fresh ownership markers prevent adoption of its existing data. Provider registrations, product releases and Ceph-specific storage are omitted because this skeleton owns no corresponding capability. Shared PVCs use each cluster's default development storage class. Minikube uses a dedicated containerd/Calico profile. Development k3s uses the same Kustomize base with a registry-image overlay. Full CI and optional signed image publication are separate from cluster activation. The Docker integration harness is disposable test infrastructure derived from the Kubernetes workload source. Verification requires deterministic checks and real PostgreSQL/broker/browser lanes. A rendered manifest or a passing Docker test does not qualify Minikube, k3s, remote CI or production. Context onboarding must add the relevant owner, transaction, transport, security and browser recovery evidence. Clarification, 9 October 2026: an internal aggregate can precede its browser feature. Metallo uses an explicit None projection registration to save that aggregate and its receipts without a realtime contract or notification rows. Unregistered aggregate keys remain errors. Once a browser projection is enabled, its typed bytes and delivery intent commit atomically with the state change. Instantiate and run assigns launch browser notifications to Milestone 2 (7A). Amendment, 16 October 2026: Themelio uses Python, following the Operations service at multilingual reference commit a9d7bf6. It retains direct service-local folders, with Dependency Injector in services/composition and separate persistence, messaging, realtime and HTTP adaptors. Shared technical input/response models are schema-generated. Central command execution, durable private command reactions, private-domain publication and explicit realtime publication follow that baseline. The service owns no contexts or business registrations. Its versioned delivery extension is applied only to Themelio databases by administration tooling; Metallo and Beto retain their current persistence contracts. Existing Pharos wire envelopes, authentication and deployment boundaries remain in use. Amendment, 26 October 2026: Web consumes the API's assembled browser OpenAPI document through a committed Orval Fetch and TanStack Query client. Contract generation produces the embedded Go document, typed browser operations, models and test fixtures together and drift checks compare the complete generated tree. AAI login start remains a native browser form navigation so the identity-provider redirect and callback-cookie protocol stay under browser control. <a href="/source/docs/decisions/0034-fresh-skeleton.md.txt">Source snapshot</a> · Revision f928834c8.
- [0035 — Shared dev-box AAI](/reference/docs/decisions/0035-local-aai-laboratory.md): Status: Accepted for development use. Date: 6 October 2026. Member-token policy amended on 26 October 2026 after a verified staging authorisation-code/PKCE probe; offline-session limit measured on staging on 28 October 2026. Developers and dev-box components need an OpenID Connect issuer shaped like GRNET AAI, with their own test users, before registered staging clients exist. Every dev box gets one from dx/common/local_aai, as shared infrastructure next to its storage (decision 0037): make deploy installs it after storage and make reconcile refreshes it. It is development infrastructure; it assigns no production context authority and adds nothing to the context, route, event or subscription registries. It runs in the separate pharos-aai namespace with its own PostgreSQL on the default ceph-block class. The dev-box k3s role owns K3s; the local_aai role adds only a development CA and an Nginx HTTPS edge. This original edge ownership is superseded by decision 0038: a shared platform role and catalogue own it. Browsers and Pods use one issuer URI; laboratories retain their own namespaces and data. clients.json declares login clients and public Site trust metadata. pharos-control-plane serves the Pharos web and API, including Icarus, with one-time offline consent. Slurm uses this same client's ordinary access token; there is no exchange request, dedicated exchange client or required resource client. Site entries still export issuer/JWKS/CA configuration without a secret. The committed registry has no exchange configuration or custom mapper. Both pharos and pharos-baseline reproduce the verified staging member-token shape: RS256 access tokens with site_username, a 3600-second lifetime, no aud, azp identifying the login client, refresh tokens and exchange disabled. The baseline contains only the control-plane login client; pharos also serves tools. Deployments receive each client's and resource's values as private variables under the names the vault provides on staging, so moving to GRNET changes only the registered values and URLs. The development CA is the one difference deployments must accept. Nothing asks for an administration credential or registers clients by hand. site_username is released in ordinary access tokens and userinfo, but not ID tokens, using Keycloak's standard user-attribute mapper. Its value remains assigned by an administrator. An owner retains the offline refresh token and refreshes before each native operation. Only the access token reaches Slurm, separately from durable Site commands; the native Site API bearer remains a different identity. Runtime checks the signature, issuer, exact login client (azp), member binding, access-token type, identity scope and a validity window of at most 3600 seconds. A missing audience is accepted; a supplied audience must name only the login client. Slurm independently verifies its trusted signing keys, expiry and native username. Waldur continues to consume the claim from userinfo. A staging measurement on 28 October 2026 fixed the shape of this mechanism: the offline session has a maximum lifespan counted from consent, and a refresh does not extend it (a refresh about 150 seconds after consent returned refresh_expires_in 86249 against the then-current 86400, the remaining time rather than a fresh 86400). Refresh tokens rotate on every use. The value the issuer reports is already the nearer of the idle timeout and the maximum lifespan, so it is the consent's single deadline; the access token's 3600-second expires_in says nothing about it and the Member Credentials store therefore never uses it as one. GRNET accepted the requested wider window on 29 October 2026: a 30-day maximum lifespan counted from consent and a 7-day idle timeout. devops/aai/identity_config.py sets those defaults in DEFAULT_OFFLINE_SESSION and keeps both numbers as parameters (OFFLINE_SESSION_MAX_SECONDS, OFFLINE_SESSION_IDLE_SECONDS), so the laboratory realm and a provider change carry no code change. A refresh inside the 7-day idle window keeps the session alive and moves its deadline forward; only the 30-day maximum, counted from consent, can end a session that is in use. Accepted work that runs for up to 14 days therefore fits, provided at least one native action or consent refresh happens inside every 7-day idle window. Nothing waits for a native action to keep the Session alive. The Member Credentials owner runs one periodic worker beside its revocation sweep (#879): it refreshes every active consent whose idle deadline falls inside the next two intervals, seals the rotated refresh token before the new value is used, and keeps no access token at all. The interval is configurable (MEMBER_CREDENTIALS_REFRESH_SECONDS, three days by default) and must leave one interval of margin inside the agreed idle timeout. It refreshes only a consent of a member who still holds Project execution authority, so a revoked, superseded, lost or expired consent, and the consent of a member who left the Project, is never kept alive. A refresh that the issuer refuses ends the consent in the state that refusal names, with no retry of that token and no token in a log. The 30-day maximum is still the issuer's: a refresh only ever stores the deadline the issuer reports, capped at the agreed maximum, so the worker extends an idle Run towards that maximum and never past it. Confirmation that the provider applied the wider window is a repeat of the 28 October measurement against a GRNET test client: a session must still answer a refresh after more than 24 hours, and a refresh inside the idle window must return a refresh_expires_in close to the idle timeout rather than a falling remainder of 86400. It is run only against our own test account and recorded as metadata. The laboratory realm follows those values rather than its earlier 395-day value, and a qualification must not assume a wider window than the issuer reports for the Session it holds. A second measurement on the same day records what a repeat login does. It opens its own offline session with a full window; the earlier session is neither extended nor asked for a new consent prompt. Every login carrying offline_access therefore replaces the consent a Run already uses: the incoming offline token becomes the Run's token and the previous one is revoked at the issuer before the next native action. The sealed offline token belongs to the Member Credentials context in Metallo, which refreshes and revokes it; no Runtime or browser sees it. A refused refresh ends the consent and defers the Run with a fixed reason. Evidence is the metadata-only record offline-session-second-login-20261028.jsonl from the same staging audit. Personal credentials are also the only permitted model. Each Slurm job runs as the member who submitted it, with that member's own access token. Impersonating the member through a service account is rejected: DAEDALUS does not permit it, and neither the FirecREST nor the LUMI model offers it. Monitoring, accounting and cancellation for policy reasons use a Slurm coordinator role instead, so they never depend on a member's token; submission always carries the member's own access token. This deliberately replaces the earlier resource-bound, short-lived exchange policy. The token is a shared login bearer, with up to one hour of residual native validity after revocation: Slurm's local JWT checks do not introspect consent or restrict the token to one API. Runtime access checks and refresh refusal prevent new authorised operations; they cannot revoke a copied JWT already issued. A future requirement for narrower credentials needs a separately agreed policy, not an implicit prerequisite for running with the actual staging contract. Bootstrap administrator credentials stay in administration jobs. Keycloak receives only its restricted database identity. Test users come from an editable file; generated passwords and credentials stay in private ignored files. This is not an export of GRNET's realm and does not replace GRNET AAI as the production issuer.
- [0036 — Isolated Icarus runtime laboratory](/reference/docs/decisions/0036-icarus-runtime-laboratory.md): Status: Superseded by DD-0055 and the Icarus-hpc laboratory. Historical only; its implementation was removed from the repository. Date: 5 October 2026. Its exchange policy is superseded by the 26 October 2026 amendment to 0035; the description below records the original experiment. The user authorised implementation and deployment of the proposed local AAI and Slurm proof on their Debian development VM. This decision governs the laboratory under dx/metallo/icarus-runtime; it assigns no production context authority. Run/Attempt ownership and durable Site commands are exercised through an immutable revision of the external Metallo reference slice. A reproducible overlay adds canonical initiating-member identity, TLS, member consent, ephemeral credentials, Slurm execution, scratch handover and typed member projections. Production services cannot import laboratory code. Their context and route registries remain empty. The reference retains its shared execution database across its experimental Workload, Execution and Site packages. This is inherited experimental scope, not permission for production contexts to share database authority. Member consent uses a separate private database. Runtime database roles cannot create schemas; migrations and provider topology creation run as explicit administration jobs. A Metallo dev-box VM hosts the control-plane workloads, in the separate icarus-lab namespace with its own PostgreSQL, and native Slurm daemons. The dev-box role owns K3s. Identity comes from the local AAI of decision 0035; only the processes that call it are admitted across the namespace boundary. An ordinary pharos-runtime account opens TLS Site connections and submits member JWTs to loopback slurmrestd. It receives no offline token, provider administrator credential, SSH execution key, sudo rule or Kubernetes administration credential. Test Unix accounts are entries under ou=pharos in a host OpenLDAP directory, resolved through SSSD. Only pharos-agent may write that subtree, without passwords; ACLs, slapo-unique and slapo-constraint bound what it writes, and no directory administrator password exists. Project associations remain administration fixtures. The local Keycloak extension emits site_username only for the Icarus exchange. The broker explicitly translates an allow-listed resource URI to stock Keycloak's audience parameter. Exact staging RFC 8707 behaviour needs separate qualification. Slurm signature/expiry validation is complemented by runtime issuer, audience and identity admission; native audience/issuer enforcement is not assumed. The browser uses NextAuth authorisation-code/PKCE login and server-selected Centrifugo Protobuf subscriptions. Tokens are delivered separately from durable commands. Submission intents recover native bindings conservatively; an uncertain submission is never blindly repeated. Results are explicitly exported by the member through scratch ACLs. A bare test job precedes external Git/Harbor/S3 and Apptainer integration, which require their own configured resources and evidence.
- [0037 — Dev-box storage from the final deployment's Rook/Ceph role](/reference/docs/decisions/0037-dev-box-storage.md): Status: Accepted for development use. Date: 11 October 2026. Dev-box providers need block volumes and S3 buckets. Without a shared substrate, each one brings its own: an omnibus GitLab keeping LFS on its volume, Garage or MinIO for S3, and local-path volumes for databases. Code written against those local providers then differs from the final deployment, which uses Rook/Ceph. Each dev-box VM has a second, empty disk for this purpose. The storage role installs Rook/Ceph on it and provides ceph-block volumes, ceph-filesystem volumes and ceph-bucket object buckets through Ceph RGW. The role is a copy of the final deployment's storage role from legacy/devops/ansible/roles/storage, cluster-wide flow only, with two later changes from the GB10 cluster: a 16 MiB CephFS readahead and a 16Gi OSD memory limit. The Minikube flow is not copied; the legacy local setup never installed Ceph. Only inventory values differ from a multi-node installation: one node is both storage and monitor node, pools keep one replica, and the explicitly named disk is the only OSD. The role adds variables whose defaults keep the original behaviour: single-replica safety, Helm value overrides, verified chart archives and an optional dashboard route. The dev box pins the Ceph and operator images by digest and disables Prometheus integration, which it does not run. As in the final deployment, ceph-block is the default class: K3s no longer restores local-path as a second default, and claims made before storage keep their local-path volumes until they are recreated. It points the CSI node plugins at the dev box's K3s kubelet directory and silences only the single-replica health warnings, so a healthy dev box reports HEALTH_OK. Its OSD keeps the chart's 4Gi memory limit: Rook derives the OSD's cache target from the limit, a 32 GB VM also runs GitLab, Waldur and Keycloak, and measured CephFS reads were no faster with 16Gi. The readahead alone made them about 1.5 to 2 times faster. The role waits for the pool, filesystem and object store to be ready before reporting success. Rook leaves a bucket claim's BUCKET_REGION empty; S3 clients sign with the ceph-bucket class's region. The dev box's disks are themselves Ceph volumes of the hosting OpenStack cloud, so each in-VM monitor and OSD adds synchronous writes to shared storage. Kubernetes tasks run in the Metallo controller through the target tunnel; the VM needs neither Helm nor Kubernetes Python libraries. Before Rook starts, a guard on the VM accepts only a whole, unmounted disk without partitions that is empty or already a Ceph OSD, and never the root disk. The first successful run records the disk for the target; moving Ceph to another disk needs an explicit migration. First setup runs make deploy from the repository root. Its ordered stages install K3s, then storage, then check block, filesystem and S3 storage, and each is safe to repeat. It runs before application setup, so the stack's claims use Ceph without naming a class. Later platform layers join the same command as further stages. New dev-box providers use the default class, ceph-filesystem or ceph-bucket rather than adding local storage providers. Existing laboratories and parallel-operation profiles move to it in their own changes.
- [0038 — Platform service catalogue and shared host edge](/reference/docs/decisions/0038-platform-edge-catalogue.md): Status: Accepted for the authorised development deployment; platform-wide adoption and PR review remain pending. Date: 2026-10-12. Extended for A2 on 2026-10-13. Laboratory routes leave the catalogue on 2026-10-14; see Laboratory routes. Deciders: ikons for the development scope. Wider platform review remains pending. Related DD: DD-0038 — Platform edge and mesh.
- [0039 — Development Argo CD bootstrap and ownership](/reference/docs/decisions/0039-dev-box-argocd-ownership.md): Status: Proposed for review. The authorised development implementation is qualified; wider platform approval remains pending. Date: 2026-10-13. Deciders: ikons authorises the development implementation and live test. Wider platform approval remains pending. Related DD: DD-0039.
- [0040 — Platform layout: Ansible for the cluster, Argo CD for services](/reference/docs/decisions/0040-platform-layout.md): Status: Proposed. The design was agreed in review on 13 October 2026 and is implemented in stages. Date: 13 October 2026.
- [0041 — GitLab and Waldur as Argo CD services](/reference/docs/decisions/0041-gitlab-waldur-services.md): Status: Proposed. Implements stage 3 of 0040; the implementation is in DD-0041. Date: 13 October 2026. Waldur pin updated to 8.1.3 on 22 October 2026.
- [0042 — preserve authorship when reactivating legacy source](/reference/docs/decisions/0042-legacy-source-relocation.md): Status: Accepted in #626 (6 October 2026) as its decision 0007; renumbered 0042 when the Themelio relocation was carried into the platform stack (14 October 2026). The relocation is commit 77bfeecfe; its deployment is decision 0043. The maintained compatibility application will use files moved from legacy/ into compat/themelio/ with git mv. Preserve their contents, executable modes and relative package structure in the relocation commit. Existing implementation continues its history at the new paths. Git remains the record of line authorship. Git detects renames from snapshots. Keeping relocation separate from subsequent edits makes that detection straightforward for blame, log and review tools. Inspect the staged diff for exact renames and verify representative files with git blame and git log --follow. The relocation commit records who performed the move; unchanged source lines retain their earlier attribution. Do not squash the move together with source rewrites or regenerate files during relocation. This amends decision 0034 only where it requires every original file to remain under legacy/. The original complete tree remains in Git at the recorded archive baseline, described by docs/legacy-manifest.json. Files remaining under legacy/ retain exact blob/mode verification and remain excluded from builds. Maintained files receive ordinary source checks at their new locations. Before relocation, extend verification to recognise an explicit mapping of original path, destination, original blob and mode. Preserve the original manifest. Require each original file to be accounted for either at its unchanged archive location or by a verified relocation. Reject unrecorded disappearance, duplicate mappings and unexpected archive edits. Verify the original relocation against Git history; subsequent compatibility edits are reviewed through normal commits. Record later retirements explicitly without removing their historical provenance. The mechanical move may update relocation and large-file-review metadata. Source, lockfile and build changes follow in separate commits. Prepare the checker and its negative fixtures first so the move can pass verification without weakening the archive checks. This decision settles source movement and history preservation. Building and deploying the retained Themelio are in decision 0043.
- [0043 — The retained Themelio as the themelio-compat service](/reference/docs/decisions/0043-themelio-compat-service.md): Status: Proposed. Implements stage 4 of 0040; the implementation is in DD-0043. Decision 0045 amends decisions 1, 3 and 6: the service is installed on every dev box, and its images are built in the cluster. Date: 14 October 2026.
- [0044 — The shared LLM stack as the llm Argo CD service](/reference/docs/decisions/0044-llm-stack-service.md): Status: Proposed. Applies 0040 to the shared LLM access epic; the implementation is in DD-0044. Date: 20 October 2026.
- [0045 — The development loop: deploy, reconcile and takeover](/reference/docs/decisions/0045-development-loop.md): Status: Proposed. Implements stage 6 of 0040; the interfaces are in DD-0045. Date: 14 October 2026.
- [0046 — Service communication and composition](/reference/docs/decisions/0046-service-communication-and-composition.md): Status: Accepted. Date: 13 October 2026. All services follow the same communication and dependency injection boundaries. Business commands, queries and integration events between services use versioned, typed Protobuf messages over RabbitMQ. Driving adaptors translate wire contracts into context-owned application types. This keeps transport and framework concerns outside domain and application code. Browser-facing HTTP, external integration protocols and private health and diagnostics endpoints remain at their designated boundaries. Browser updates use the server-authorised Centrifugo protocol defined in CLIENT-SUBSCRIPTIONS.md. Dependency registrations and lifecycle hooks belong in service composition roots. Application handlers receive explicit, provider-independent ports. Go composition uses Fx; other languages implement the same boundaries with idiomatic tooling. Worker cancellation and draining precede resource disposal, including after partial startup failure. Broker topology is provisioned through administration tooling. Runtime roles have scoped permissions. Publications use durable messages, mandatory routing and publisher confirms. Queries use correlated request/reply messages; the consumer acknowledges a query delivery only after its reply is confirmed and routable. Uncertain reads can be repeated. Command and event consumers acknowledge after the owner transaction commits state, receipts, recorded outcomes and outgoing intents. Retries preserve message identity and use bounded retry, dead-letter and explicit replay policies. Contexts without owned storage declare persistence: none. PostgreSQL remains the default for persisted contexts. Administration and migration generation omit storage-free registrations. This extends decision 0034, “fresh service-first skeleton”. These rules apply across services. Context documentation and published contracts specify individual operations, payloads and implementation status. Architecture checks and transport integration tests verify the shared boundaries and delivery guarantees as each service adopts them.
- [0047 — Dev-box consolidation: ownership, Argo AAI and shared data](/reference/docs/decisions/0047-dev-box-consolidation.md): Status: proposed Date: 16 October 2026. Source requirements: the development hand-off of 15 October 2026, inspected against main at 297a7d50b6dc6d2e034d7f4ec97f9d0598ee609a.
- [0048 — Shared infrastructure on personal development boxes](/reference/docs/decisions/0048-shared-development-infrastructure.md): Status: accepted
- [0049 — The Kubernetes Site Runtime as the pharos-site-runtime service](/reference/docs/decisions/0049-site-runtime-service.md): Status: Proposed. Applies 0040 and 0041 to the Instantiate and Run epic's Kubernetes Site Runtime (subtask 5B). Date: 22 October 2026.
- [0051 — Inventory-driven distributed platform setup](/reference/docs/decisions/0051-distributed-inventory-setup.md): Status: Proposed; implementation and qualification are recorded in DD-0051. Date: 20 October 2026. Stacked on fix/workstation-env at d0837f84e.
- [0055 — Member SSH access to Icarus-hpc](/reference/docs/decisions/0055-hpc-ssh-key-provisioning.md): Status: Proposed. Date: 21 October 2026. The implementation is in DD-0055. Deciders: ikons; review by Metallo owners (dkatsimpris) and the Icarus-hpc administrators (jimsiak). This decision builds on the September prototype (#137, Approach C). It keeps the prototype's fingerprint identity, lock ordering, withdrawal handling and compensation for an add that lands after its withdrawal. It moves key ownership from the Waldur offering user to the member, in a Metallo context, and delivers changes to Icarus-hpc through the Site API instead of a CSV queue.
- [0056 — Verified platform releases from one deployment catalogue](/reference/docs/decisions/0056-verified-platform-releases.md): Status: Proposed. Date: 22 October 2026.
- [0057 — Targeted realtime messages](/reference/docs/decisions/0057-targeted-realtime-messages.md): Status: accepted
- [0058 — Complete distributed services composition](/reference/docs/decisions/0058-distributed-services-composition.md): Status: Proposed; implementation and evidence are recorded in DD-0058. Date: 24 October 2026. Stacked on ADR-0051 and ADR-0056.
- [0059 — Production Icarus-hpc provisioning and run installation](/reference/docs/decisions/0059-icarus-hpc-provisioning-run-installation.md): Status: Proposed. Date: 25 October 2026. Amended 1 November 2026 (decision 3 and the required Site-agent mode set; see Amendment). Implementation and remaining qualification are in DD-0059. Stacked on ADR/DD-0055 and the verified Slurm Runtime work in #778 (native Slurm Runtime delivery).
- [0060 — Development seeds for Waldur and GitLab](/reference/docs/decisions/0060-development-seeds.md): Status: Proposed. The commands, seed formats and safeguards are in DD-0060. Date: 22 October 2026. GitLab seed revisited on 4 November 2026, once Source Control made the Project groups (decision 5).
- [0061 — Support context and development broker provisioning](/reference/docs/decisions/0061-support-context-and-broker-provisioning.md): Status: Accepted. Date: 7 October 2026. Broker provisioning superseded by decision 0046 on 21 October 2026. 21 October 2026. The Support context and its link rules stand. Its broker provisioning below does not: Support now follows decision 0046. Its broker roles, queues and permissions are generated from contracts/support/messaging/broker.json and services/beto/contracts/subscriptions.json into services/beto/services/topology/permissions.generated.json, and the administration process provisions them; contracts/events/topology/beto.json, the Go amqp.Plan and the import_definitions step are gone. The .dead retention policy is not carried over; dead letters are replayed with main's replay tool. The text below is kept as the record of the original decision. Beto owns the Support context. Its SupportCustomer aggregate, keyed by the Pharos user ID, links each Pharos user to a customer in the support service (today Zammad, whose user ID stays an opaque binding). The flow follows the Miro frame "Final create user in support service flow simple method": a link is pending, linked, failed (support service unreachable, retried) or conflict (email owned by another Pharos user, or HTTP 422; a person resolves it and the retry after 24 hours picks the fix up). A support-retry CronJob retries links with no attempt limit; unresolvedSince feeds an age alert instead. Support is the first context with broker delivery inside a cluster. Beto's composition root therefore lists contexts, delivery contexts and realtime contexts separately, as Metallo does. Support has delivery and no realtime projection. Runtime broker users have no topology or administration rights. Each delivery context gets a pharos_<owner> user: configure ^$; write only its delivery exchange and pharos.events; read only ref.<owner>.*; topic permissions to publish only domain|integration.<owner>.* and read its own private events plus public ones. Development provisioning imports these users as salted SHA-256 hashes, their permissions, the topology and Support's .dead retention policy (7 days, 10,000 messages, oldest dropped) with one rabbitmqctl import_definitions read from stdin into a private broker-local file. This reuses the mechanism of the hello-world decision on feat/metallo-hello-world (its 0002, not yet merged); the two scripts should merge, and this record renumbered, when that branch lands. The broker topology is declared once, in Beto's Go amqp.Plan. Declare uses it and topology -definitions prints it; pnpm generate:contracts writes contracts/events/topology/beto.json, which provisioning imports. A Go test fails when the committed file or the Go catalogue drifts from the code and the shared contracts/events/catalogue.json. Only support.customer_link_requested is published: mandatory routing would block the outbox for an event without a consumer. Linked, conflict and failure outcomes are aggregate state until something consumes them. No UserCreated event exists yet, so a development-only Beto endpoint stands in for it. <a href="/source/docs/decisions/0061-support-context-and-broker-provisioning.md.txt">Source snapshot</a> · Revision f928834c8.
- [0062 — Support tickets, iteration 1: direct requests to Zammad](/reference/docs/decisions/0062-support-tickets-direct-zammad.md): Status: Accepted. Date: 7 October 2026. Acting on behalf of the customer (X-On-Behalf-Of) was replaced by an agent-only token in decision 0063; the rest stands. A linked Pharos user creates support tickets and reads them as a conversation. Iteration 1 keeps no ticket state in Pharos. Every request goes straight to Zammad while the user waits, on behalf of the user's linked Zammad customer (Zammad's X-On-Behalf-Of), so Zammad only ever shows or creates that customer's tickets. Zammad is authoritative for tickets and their messages. This is a deliberate exception to the durable hand-off rule in AGENTS.md. If Zammad is slow or Beto stops during a creation, nothing records the attempt. Creation therefore retries only when Zammad clearly never received the request (connection refused, 429, 503) and only within five seconds; after a timeout it reports that the ticket may exist and does not retry. A user who tries again anyway may create a duplicate, which an agent merges. Iteration 2 replaces this with a ticket aggregate, an outbox hand-off and a search before create. Only users whose SupportCustomer is linked may create, list or open tickets. The user comes from the session: the API authenticates the browser, then forwards X-Pharos-Subject with Beto's key; forwarded headers are rebuilt, so a browser cannot name a user. The ticket's customer is always the user's SupportCustomer, never a browser value and never Zammad's guess:<email>. Closing follows the same pattern: Beto reads the ticket as the customer, then sends only state: closed. Acting as the customer, Zammad would also accept title, priority and owner changes, so no browser field is ever passed through. Closing a closed ticket changes nothing, so closure retries timeouts, 5xx, 408 and 429 within five seconds, unlike creation. Replies post a public, plain-text message as the customer, after reading the ticket as the customer. Like creation, a reply is not safe to repeat: it is retried only when Zammad never received it, and a timeout is reported as "may have arrived". Closed tickets accept no replies in iteration 1. Zammad would take a customer's reply on a closed ticket but leave it closed, so agents could miss it; reopening by replying is a separate later flow. The ticket list and the conversation are read from Zammad when they open. No tickets read model or Zammad webhook exists in iteration 1; both belong to iteration 2 with live updates. Internal notes are removed in Beto as well as by Zammad. Agent messages are converted to plain text and the browser renders text only. This binds the first public backend route: /api/v1/support/* on the API maps to /v1/support/* on Beto. Beto accepts in-cluster traffic only from the API. <a href="/source/docs/decisions/0062-support-tickets-direct-zammad.md.txt">Source snapshot</a> · Revision f928834c8.
- [0063 — Zammad integration: an agent-only token, no administrator permission](/reference/docs/decisions/0063-zammad-integration-token.md): Status: Accepted. Date: 9 October 2026. Amended 13 October 2026 (see the end). A trial against the test Zammad (7.2.1) passed every check below before the adaptor was reworked; its checks are now part of contract_test.go. Beto calls Zammad with one API token held in its pod. Iteration 1 (decision 0062) used a token with admin.user and ticket.agent and the X-On-Behalf-Of header, so Zammad acted as the customer and kept customers apart. AGENTS.md says administrator credentials never enter runtime containers; no decision covered this.
- [0064 — A dev box's Zammad is seeded from a configuration, not restored from the team's](/reference/docs/decisions/0064-dev-box-zammad-from-a-configuration.md): Status: Accepted. Date: 14 October 2026. Installation superseded by decision 0065 (Zammad is an optional Argo CD service; the legacy commands below are retired); the configuration by name stands. Support (Beto) is the only context that talks to a third-party application, so testing it needs a Zammad. Only one dev box had one. It was installed by hand (Helm, by one person, in October 2026) and then had the team's production database restored into it, so every other dev box could run the platform but not test Support at all, and no command existed to change that.
- [0065 — Zammad is an optional Argo CD service of the dev box](/reference/docs/decisions/0065-zammad-on-the-argo-dev-box.md): Status: Accepted. Date: 22 October 2026. Supersedes how decision 0064 installs Zammad; its configuration-by-name stays. Support (Beto) needs a Zammad to be tested against. Decision 0064 installed one per dev box with make legacy-controller-zammad, a command of the older controller loop, which main is retiring (decision 0045) and which no longer provisions Beto's broker there. The Argo dev box (decisions 0040, 0041, 0047, 0048) deployed Beto and Support but no Zammad.
- [0066 — Optional dev-box services use reviewed lifecycle selection](/reference/docs/decisions/0066-reviewed-optional-dev-box-services.md): Status: Accepted. Date: 27 October 2026. Extends decision 0065. Decision 0065 made Zammad the first optional Argo CD service and selected it through PROFILES=zammad. That is sufficient for installation, but the ordinary deployment path cannot safely infer that omitting a profile authorises deletion. Keeping a service forever after trying it wastes dev-box resources; removing it through a full reset also discards unrelated platform data.
- [0067 — Kubeflow is one pinned Argo CD composition](/reference/docs/decisions/0067-kubeflow-gitops-composition.md): Status: Proposed. Date: 29 October 2026. The live Icarus staging cluster runs Kubeflow community distribution 26.03.1, installed by the legacy kubeflow-deploy Ansible role and then corrected by hand: NetworkPolicies, Pod Security labels and the Istio global deny-all policy were deleted, Dex was replaced by GRNET OIDC through oauth2-proxy, and the ingress gateway was patched to NodePorts. That installation works, but it is not the composition the platform now uses, and the post-apply corrections are not reviewable or repeatable.
- [0068 — Platform acceptance tests with Playwright and Gherkin](/reference/docs/decisions/0068-platform-acceptance-tests.md): Status: Accepted. Date: 31 October 2026. Integration clarified: 3 November 2026.
- [0069 — Project-private Run services](/reference/docs/decisions/0069-project-private-run-services.md): Status: accepted, 6 November 2026. Approved in the #1123 implementation discussion; live GPU qualification will take place on staging.
- [0070 — Platform-side security scanning of user repositories](/reference/docs/decisions/0070-platform-side-security-scanning-of-user-repositories.md): Status: Proposed. Detailed design reference: DD-0056 (pending publication). Date: 22 October 2026. Deciders: TBD. Relates to: 0012 and 0013: the deploy gate is inserted into the k8s-root-actor dispatch route defined there.0014: that runner served the platform monorepo and is not used for user repositories.0025: its caller allow-list for gitlab-root-actor exists only in the archived legacy GitOps; see Current state.0041: GitLab is an Argo CD service, which is where the system hook must be managed.0043: the receiver, worker and gates extend the retained Themelio (themelio-compat), whose GitLab root token is a dev-box exception.0048: the Valkey used for the queue is the shared pharos-infra instance.The SecOps architecture proposal (branch docs/adr-00xx-secops, not yet numbered): this decision supersedes its Component 1 (CI-runner-based scanning).
- [0071 — Generated MCP access to the Go API](/reference/docs/decisions/0071-generated-mcp-api-edge.md): Status: Proposed for the experimental/mcp branch.
- [Architecture decision records](/reference/docs/decisions/README.md): This catalogue brings the archived ADRs and the current platform decisions into one sequence. The historical records retain their original status and rationale. Read them in the context of the platform they describe; the current architecture starts with 0034 — fresh service-first skeleton.
- [Developer guide](/reference/docs/development.md): Start here to set up your personal dev box and work on Pharos. Run the commands below from the repository root. Your dev box is a Linux VM running the platform and its databases through Argo CD. Your workstation holds your checkout and editor. Takeover runs selected services on your workstation while they use the dev box's dependencies; saving a file reloads the affected service. Hand-back returns those services to the VM. Pharos DX gives you the same controls in a Workspace window and tray.
- [Distributed setup](/reference/docs/distributed-setup.md): Use ADR-0051 and its implementation/evidence for scope. The personal TARGET=dev-NAME workflow remains in development. Commands with ENV do not load the checkout's personal .env. Explicit personal TARGET or SSH selections still fail before access to an installation. Ordinary dev-box commands retain their saved settings, runner bindings and private state.
- [NVIDIA GPU Operator ownership](/reference/docs/gpu-operator.md): Status: live in Icarus Cloud. On 26 October 2026 Argo CD adopted the existing installation at source revision 808f1a31f87712282892be97116579f77517f950 and applied GPU Operator v26.7.2. This page records the ownership boundary and the live adoption evidence. It does not claim a fresh Debian 13 installation.
- [Icarus Cloud release readiness](/reference/docs/icarus-cloud-release-readiness.md): Status: not ready for a platform-v* release selection. The live cluster can run its current workloads, but its Argo, application and operator state is not the source state that the verified release path expects. This is a read-only audit snapshot taken and revalidated on 27 October 2026. Re-run the checks before a maintenance operation; the matrix records evidence, not continuing authority to mutate the cluster.
- [Large-file review](/reference/docs/large-file-review.md): All active handwritten files remain below 450 lines. The exact exceptions in large-file-review.json cover generated Protobuf bindings, package-manager lockfiles and the generated archive inventories. Their tools own their structure; splitting them by hand would create another source of truth. pnpm generate:contracts reproduces the bindings and frozen installs check dependency resolution. pnpm check:large-file-review enforces explicit review coverage. A future handwritten exception requires a responsibility decision recorded in that JSON file and an explanatory note here. Archived files over 450 lines each have an explicit archival decision. Their responsibility is to preserve the exact previous main tree for reference; splitting or rewriting them would damage that provenance. The integrity check verifies those blobs and modes while active architecture checks exclude legacy/. The GB10 serving sources carried over unchanged from the CSLab fork register four further exceptions: the LLMInferenceService template, the Open WebUI deployment task sequence, the model catalogue and the verbatim GB10 notes. Each records one responsibility that was qualified on the cluster as a whole. <a href="/source/docs/large-file-review.md.txt">Source snapshot</a> · Revision f928834c8.
- [Browser OpenAPI client](/reference/docs/openapi-client.md): The web application uses Orval 8 with its Fetch and TanStack Query v5 client. The Go API's assembled OpenAPI document is authoritative. Generate the document and browser client together: pnpm generate:contracts includes the same step. Generated browser files are committed under services/web/src/adaptors/generated/http/. pnpm check:http regenerates them in isolation and checks added, removed and changed files. These commands use the frozen devops Python environment, including PyYAML.
- [Verified platform releases](/reference/docs/platform-releases.md): Decision 0056 separates official release publication from personal development and production activation.
- [Combined PR qualification](/reference/docs/pr-integration.md): integration/current is the shared candidate for testing the PRs selected in <a href="/source/integration/prs.json.txt">integration/prs.json</a>. A full refresh starts from the current main, merges the integration tooling, then merges the selected PR heads in dependency order. The original PRs retain their bases and reviews. The selected batch excludes archived branches and unrelated drafts. Explicitly authorised drafts are listed in draft_exceptions, each with its relevance to issue #1123 or acceptance PR #1132. A selected PR becoming a draft needs either removal or an authorised exception before rebuilding. Run the helper from a dedicated clean worktree. It requires Git, Python 3, an authenticated gh and an origin pointing at this repository. The helper creates a new integration/build-* branch in that worktree. Preserve any active build before starting another. For additions and descendant-only source changes against the same main, use python3 scripts/pr_integration.py update. It starts from the published candidate, retains its supplemental MCP source and recorded qualification corrections, and merges new or advanced PR heads in dependency order. Updated pinned patches apply only their new source delta. The command refuses rewritten sources, removed PRs, a changed main, changed patch bases or replacement rules; these require refresh. Publication still uses publish, an immutable snapshot and a lease. Every updated candidate starts with qualification pending. Same-repository open PRs use their actual Git branch heads. When GitHub's PR metadata lags behind the branch, the manifest preserves the reported older SHA separately. Fetching and publication recheck the actual source. status compares the committed manifest with the selected PR heads, their bases and states, the latest main and the tooling branch. A changed source reports STALE and exits 1. New PRs require an explicit addition to the selection. Update the selection and helper on chore/pr-integration-tools, push that branch, then refresh. Builds include its exact commit. A source update, rebase or selection change requires a new candidate. A full refresh after a rebase or removal avoids retaining removed commits or a dropped PR from an earlier candidate. A closed, unmerged PR requires removal from the selection. A merged PR's source remains in the manifest; Git recognises commits already present in the base. When a category aggregate supersedes a stacked PR's base, base_replacements maps that base branch name to the selected aggregate PR number. The builder requires the aggregate to contain the base's current commit and merges it before the dependent PR. The manifest records the replacement and exact base commit; changes to either make the candidate stale. For a PR based on a deployed integration snapshot, patch_bases pins its base branch to an exact commit. The builder verifies that the pinned commit is an ancestor of the PR head, then applies the base-to-head patch with three-way conflict handling. This preserves the PR's own changes without importing the older snapshot's complete selection. The manifest records the original head, base, patch hash and resulting commit. Resolve a patch conflict by staging the complete resolution and running continue; let the helper commit it. A changed source base requires an explicit selection update. The moving integration target may be rebuilt independently; its latest SHA does not replace the original source base. Use pr_patch_bases to pin an individual PR when several PRs share a moving target but started from different snapshots. Its PR-number keys override patch_bases for those PRs. A stacked follow-up to a patched PR can likewise pin its parent's source head, applying only the follow-up. update accepts pins for new selections; changing an existing PR's effective pin requires refresh.
- [Fresh skeleton verification](/reference/docs/verification.md): Recorded on 30 September 2026 for the replacement built from main 8046ea0f1. The active runtime foundations derive from the multilingual reference workspace at 7f8d8ab; infrastructure conventions derive from architecture-reset at b228782dd in this repository. Disposable Docker projects created for these checks were removed. The original working checkout and its uncommitted files were preserved. The fresh branch is intended to replace the active main tree when merged into the existing repository. <a href="/source/docs/verification.md.txt">Source snapshot</a> · Revision f928834c8.
- [API-owned login and sessions](/reference/services/api/adaptors/authentication/README.md): The optional Run serving gateway issues a host-only Run handle through a nonce-bound bootstrap. Every request and open connection revalidates its parent session, Party and AUP through AuthoriseBrowserFence. The handle contains no parent login token. See the <a href="/source/gitops/manifests/pharos-site-runtime/serving/README.md.txt">activation guide</a>. The API implements the authentication protocols behind API_AUTHENTICATION_ENABLED=true. The retained web requires this enrolment. The NextAuth <a href="/source/services/api/adaptors/party/README.md.txt">bridge endpoints</a> have been retired. Enabling API authentication also closes the isolated development password's browser session issuer; the separately enrolled CLI retains its own technical scope.
- [Experimental MCP service](/reference/services/api/services/mcp/README.md): This branch provides a native Go Streamable HTTP gateway and an API-owned OAuth broker adapted from architecture-reset. The gateway uses generated OpenAPI types and calls the current Go API. The broker uses oidc-provider and openid-client for OAuth and AAI federation. The contract guide defines tool parity, inputs, scopes, command recovery and the private assertion protocol.
- [Beto boundaries](/reference/services/beto/adaptors/README.md): Start with the Project directory for its query, read model and provider limitations. Shared adaptors contain technical mechanisms. Context adaptors own wire-to-application mapping and read repositories. Project's static repository lives under contexts/project/adaptors/persistence/mock. Foundation packages retain domain/application primitives, configuration, diagnostics and cancellation timing. Fx registrations stay in services/beto/di. Project has no write commands or events. Its named query handler is registered in the composition root through a generated response-bound registration. Its schema owns plain input/response models, query/reply identifiers and mechanical wire mappings. The shared registry routes plain application values through QueryBusPort. Registration covers every declared request before workers start. Commands use services/commandregistration to bind a plain FeatureHandler factory to one AggregateTransaction. Handlers receive CommandContext, generated input and an owner WriteRepository.Get/Save capability. The transaction owns locking, version checks, durable receipts, outcomes and outgoing intent. Repository and realtime bus capabilities expire when the invocation ends. CommandStore, CommandPort, Fact and the old HTTP command/query helpers have been removed. Saving stages private domain facts through the owner's mapper. An integration publication subscriber commits its own receipt and encoded event, preserving the source aggregate revision and occurrence time. A command reaction commits exact private command bytes before a paired consumer executes them. Public commands commit their exact reply bytes alongside their outcome; the reply relay confirms delivery independently. Query replies retain confirmed publication before ACK. Each owner's query queue has four consumers, each with one outstanding request and its own reply publisher. Independent reads can progress while another waits on a provider. Cancellation drains all consumers and leaves unfinished reads available for redelivery. Command queues retain one consumer. Realtime publication is explicit. Each message selects a canonical Project or Party audience, or a principal subject for a private outcome. Command snapshots must match committed state; rejection messages must match the recorded outcome and use a principal target. The gateway derives channels from the target. Subscribers have a separate receipt/outbox boundary. Saving a root alone creates no browser publication.
- [Marketplace](/reference/services/beto/contexts/marketplace/README.md): Marketplace in Beto lets a Project's buyer buy fixed provider packs, such as ten H200 hours or a block of storage. A provider fulfils each order behind a provider port; today the provider is Waldur. The <a href="/source/services/beto/docs/epics/marketplace/adr.md.txt">epic</a> plans the tasks, and <a href="/source/services/beto/docs/epics/marketplace/place-order.md.txt">Place an order</a> designs the first one. It has its domain, the PlaceOrder and ListProjectOrders contracts and handlers, its <a href="/source/services/beto/contexts/marketplace/adaptors/persistence/README.md.txt">database</a> and its private OrderPlaced event. Beto serves both over RabbitMQ to the API, with stand-in catalogue and buyer ports, and the API exposes them as routes.
- [Party Governance](/reference/services/beto/contexts/party_governance/README.md): Party Governance in Beto owns canonical Party authentication and verified provider identity associations. The <a href="/source/services/beto/docs/party-project/plan.md.txt">implementation plan</a> and <a href="/source/services/beto/docs/party-project/identity-model.md.txt">identity model</a> describe its wider scope. Its runtime owns the isolated pharos_party_governance database, authenticated broker queries, private authentication ingress, attribute delivery and canonical integration publication. The <a href="/source/services/beto/contexts/party_governance/adaptors/messaging/authentication/README.md.txt">authentication boundary</a> describes enrolment. The retained login edge remains optional until its configured AAI flow is qualified; an unenrolled edge cannot register a Party.
- [Project context](/reference/services/beto/contexts/project/README.md): The proposed <a href="/source/services/beto/docs/party-project/plan.md.txt">Party Governance and Project evolution plan</a> defines the next persistent model, phased Waldur handover, owner contracts and draft OpenAPI replacements for Themelio routes. The source-managed canonical slice and retained directory have separate contracts.
- [Support](/reference/services/beto/contexts/support/README.md): Support links each Pharos user to their customer in the support service (today: Zammad) and keeps their support tickets. Pharos records tickets and messages first and sends them to Zammad afterwards; Zammad's changes come back through a webhook and a re-read. The flows are modelled on the Miro board "Pharos Platform" (link, ticket and sync frames). Support follows decision 0046: the API reaches it over RabbitMQ, and every change is one command on one aggregate. Support answers only the callers its contracts/support/messaging/broker.json declares (the API's broker user, for queries and commands), checked on every request as Project and Party Governance do.
- [Execution Core — internal launch and execution planning](/reference/services/metallo/contexts/execution_core/README.md): The stateless ServiceAccess query authorises current runs.output.view members against the latest active Kubernetes service Attempt. Its public query requires explicit publication in the accepted execution material and current accepted-work continuation authority. Neither query creates a durable grant. The API and Runtime use these queries for Run services; bounded file/log/prediction permits retain their existing protocol. This slice records a planned Run and its child Attempts with the initiating Pharos member and Project, then resolves their provisioned account for the unique Site. The Run owns the consistency boundary; Attempts have no separate write repository.
- [LLM Access — order fulfilment core (3B, first slice)](/reference/services/metallo/contexts/llm_access/README.md): This folder holds the pure core of subtask 3B: how one approved Waldur order becomes a Project's allocation (models, budget, limits) and the desired gateway and chat state derived from it. The context is registered and activated: the Metallo service opens its private database and relays its outbox, the workload mounts its restricted credentials, and allocations persist through the generic aggregate store (state, version, command receipts and the reconcile intent in one transaction). The reconcile intent, llm_access.allocation.reconcile.v1, is an event private to this context; the aggregate has no browser projection, because nothing reads one, and a shared realtime contract is added when a browser feature needs it. A snapshot whose version differs from its row is refused as corrupt. Delivery runs as outbox relay, then the broker queue ref.llm_access.gateway, then the consumer, then the Reconciler, then LiteLLM. The consumer is optional by configuration: With neither LLM_ACCESS_GATEWAY_URL nor LLM_ACCESS_GATEWAY_KEY set there is no consumer. The relay still publishes and the queue waits. This is the base deployment's setting, which has no LiteLLM; the devbox overlay, which runs the llm service, sets both.With both set the service consumes. With only one set it refuses to start.The key is a LiteLLM key limited to /team/new, /team/update, /team/info, /team/block, /team/unblock and /key/block, created by a LiteLLM administrator outside the container (never the master key, and never a key that may call /key/generate, which can mint unrestricted keys). /key/block is the only key route and there is no /key/unblock, so the credential cannot undo a block. Measured: with it the key could not unblock, generate, update, delete, rotate, list or read keys, create users, read spend logs or register models. Its user has the proxy admin role, because LiteLLM refuses a team update that raises max_budget from any other key and an order may raise the allowance; the route list is what limits it. Ending an allocation does not depend on a change to its limits being accepted: its keys are blocked first, and the team is blocked before its limits are updated. Mount it as the file LLM_ACCESS_GATEWAY_KEY_FILE; it is read on each call, so replacing the file rotates it. A deployment that sets them also has to allow the Metallo pod network egress to the gateway, which the base policies do not; the devbox overlay does both.A gateway failure is retried by the broker a few times, a second apart, and then the message goes to ref.llm_access.gateway.dead, where it waits for the explicit replay (python -m metallo.services.replay llm_access.gateway). Recording the allocation as reconciled happens only after the gateway call succeeds. A recovery sweep (adaptors/sweep.py) starts with the consumer and reconciles allocations whose gateway state is behind their entitlement, which also covers a dead-lettered message. The Site Agent intake (contracts/http/llm-access.v1.yaml) is optional by configuration too: With LLM_ACCESS_INTAKE_KEY unset there are no routes. With it set, POST /llm-access/orders and GET /llm-access/allocations/{allocationId} are mounted, and the key must resolve like any other secret (LLM_ACCESS_INTAKE_KEY_FILE is preferred).The key guards only /llm-access. The service key does not open it and the intake key opens nothing else, so the agent that holds it cannot read sessions or Execution Core. It must be at least 32 characters and differ from the service key.LLM_ACCESS_MODELS, a comma-separated list of the served model names, is then required: orders may only name models in it. The base deployment sets neither the key nor the list; the devbox overlay, which runs the llm service, sets both, and a platform check keeps the list equal to the models the gateway serves.The deployment has to allow the agent's network access to Metallo, which the base policies do not.projectAlias preserves the Waldur Project slug's case. Aliases accept up to 63 ASCII letters, digits and internal hyphens. Stored allocations, gateway team names and chat prefixes retain that spelling. Allocation identity comes from the resource UUID; Project aliases can coincide. An applied alias is immutable, including its case. Existing lower-case snapshots retain their representation. Older Metallo versions cannot restore allocations created with mixed-case aliases. Key records (domain/key.py, application/keys.py, adaptors/keys.py) are a second aggregate of this context: a personal key's member, allocation and models, with no secret. They publish no event yet. Nothing issues gateway keys until subtask 4B, which records the gateway's hashed token on the record. Ending an allocation blocks every key that has a token, revoked ones included, and then its team. The token can block the key, so it is never in a projection, a route or an error. Key issuing (domain/provision.py, application/issuing.py, adaptors/issuer.py) creates a recoverable gateway key. It chooses the key locally, saves it sealed in a private provision and binds its hashed token before calling LiteLLM. An interrupted call therefore replays the exact key, while termination can block it whether creation completed or not. The caller settles the provision, erasing the sealed value, after it has durably kept or consumed the secret. LLM_ACCESS_ISSUER_KEY is the key of one ordinary internal gateway user, never a proxy admin. A gateway administrator creates the user once, with a model list that names no real model (that is what bounds any personal key it makes) and no budget or rate limit (every Project's keys count toward it), and creates its key with allowed_routes: ["/key/generate"], which only an administrator can set. Mount it as LLM_ACCESS_ISSUER_KEY_FILE.LLM_ACCESS_ISSUER_USER is that user's ID. The Reconciler adds it to every new team as a plain member and grants members /key/generate in the same team call, so it needs no extra route. A team created before this is set needs the user added by an administrator. With the issuer key set, the gateway URL and the issuer user must be set too.Measured: that user can generate keys only for its own teams, cannot name another user or set allowed_routes, and a key it makes with key_type llm_api (always sent) cannot generate, read or list keys. A model list or budget above the team's is accepted but capped by the team at request time. The caller-supplied key and replay behaviour were qualified against LiteLLM v1.96.0. LLM_ACCESS_SEAL_KEY protects both provisions and pending mail deliveries.A leaked issuer credential can create extra keys inside any Project's team, capped by that team and blockable by blocking the team. The Reconciler cannot find such keys, because no record knows them. Chat access (application/chat.py, adaptors/openwebui.py) gives a Project's members the models its allocation entitles in Open WebUI and removes them when it ends. The Reconciler runs it as the last step of each reconcile, after the gateway, and only when chat is configured: LLM_ACCESS_CHAT_URL is Open WebUI as Metallo reaches it, LLM_ACCESS_CHAT_KEY the admin API key described below (mount LLM_ACCESS_CHAT_KEY_FILE), and LLM_ACCESS_CHAT_GATEWAY_URL the gateway's address as Open WebUI reaches it, the one that goes into every Project's connection. All three or none. Chat needs the gateway and key issuing to be configured, or the service refuses to start. The base policies do not let Metallo reach Open WebUI.A Project key is a key record with no member. An interrupted creation resumes from its sealed provision without making another key. Once issuing returns, the provision is erased; if a later chat step fails, the next attempt retires that recorded key and issues the next. The key asks for no model list, so it follows the team's current models.Per allocation: a group in Open WebUI named for the allocation (the Project alias and the first eight characters of its ID, which is also the model prefix; its description binds it to the allocation, and a group with that name that belongs to something else is a conflict, not overwritten), a connection to the gateway with the Project's key and that name as prefix, and a grant to the group on each name.model. Open WebUI shows a model only to users a grant names and keeps an ungranted model admin-only (measured), so access is added connection first, grants second, and removed grants first, connection second.Group membership follows the member routes (4A, ADR-0020). Adding a member records a chat_member that keeps their address and puts them into the allocation's Open WebUI group; revoking them takes them out. The sign-in provider only says who a person is: Metallo writes no group to its realm. Open WebUI knows a person by their address only after their first sign-in, so a member who has not signed in yet stays pending and the recovery sweep retries until they have; a failure raises and the request is retried. Group membership is read per request, so a removal ends access in a session that is already open. Group management from the sign-in claim is off, so a sign-in never changes groups. Without chat configured, keys are still issued and the member record persists in both authority modes. Group changes wait until chat is configured, retaining the last applied account so that a pending removal can then finish. In beto mode the recovery sweep remains active for key authority. Who belongs is decided in one of two ways, by LLM_ACCESS_CHAT_MEMBERSHIP. site-agent (the default) lets the Site Agent's member routes decide, as above. beto lets Beto's membership decide (#769): LLM Access keeps its own copy of Beto's Project and Party facts (LLM_ACCESS_VIEWS_BROKER_URL is its enrolled identity, and PROJECT_SOURCE_SCOPE the source it loads for), the member routes only record who the Site Agent named (sign-in subject, address and Waldur user UUID), and the sweep makes the group follow the people Beto's rule admits to the allocation's Project. An allocation is the Project whose Waldur ID it carries, and a named person is in it when Beto's membership holds their Waldur user UUID. A person Beto names that the Site Agent has not named, or one without a Waldur user UUID, is never added. A Project the loaded view does not hold is left as it is, with a warning from the sweep, and a held Project that does not name a person refuses them. The view must have finished loading before anything changes. A person Beto takes away after admitting them also has their personal keys stopped at the gateway and revoked, before they leave the group. Every refused member is checked for old keys, including someone never admitted to chat or without a usable Waldur user UUID. This repairs keys issued before the membership check existed; pending deliveries are abandoned and their sealed secrets erased. The sweep also discovers allocations through their personal keys, covering older issuance that failed before saving its first member record. Once the view holds the Project, personal keys with no member record are unpaired and revoked. The member directory is rechecked under the key-issuance lock before this repair, so concurrent authorised issuance keeps its newly saved link. An incomplete view or an unheld Project still preserves existing keys. Before switching an existing installation from site-agent to beto, complete the member-link preflight and backfill. It inventories every active personal key, including keys whose provisioning has not finished. Missing member links or usable Waldur UUIDs block a switch that preserves authorised members' existing keys. The account is the one whose sign-in subject Open WebUI recorded as the member's Waldur reference (the platform AAI issues it as the subject) and whose address is the member's: an account with the same name or address is never used. The record also keeps the Open WebUI account the group was last told about: when the member's address or subject moves to an account that has not signed in yet, the earlier account leaves the group at once and only the add waits. The record remembers the add requests it has taken into account, so an add replayed after the member left, or an older one still in flight, cannot put them back; only a new request does.The credential is an Open WebUI admin API key limited to four path prefixes: /openai/config, /api/v1/groups, /api/v1/models/model/access and /api/v1/users (to find a signed-in account by address and read the subject it signed in with; the allowlist is by path, so it also covers the other user routes of an admin key). Set it up on the Open WebUI side with ENABLE_API_KEYS, endpoint restrictions on and those prefixes as the allowed endpoints (the Open WebUI role has variables for them; once saved they live in its database, so an existing install changes them through /api/v1/auths/admin/config). With it the key could not read users or the admin config, add users, delete models, chat or read its own key (measured). It can read every connection's key through the config, so it is held by this adaptor alone and is never the gateway's credential.The connection list is replaced as a whole and its settings are keyed by position, so it is read, changed and written under a lock, and removing one connection renumbers the others. Usage reporting (application/usage.py, adaptors/usage.py) answers GET /llm-access/allocations/{id}/usage?period=YYYY-MM for the Site Agent's report mode, which posts the figures to Waldur. Metallo holds no Waldur credential: The month is summed from the gateway's per-UTC-day figures for the allocation's team (/team/daily/activity), never from the gateway's own budget, which resets on its own schedule, so a request that finishes after midnight is counted in its own month. The figures are cumulative: asking again, or twice, gives the same or a larger answer, which makes duplicate delivery, interruption and restart harmless. A later month is refused, an earlier one is allowed, and a month before the allocation existed is zero.Waldur components: spend is the priced one (TOTAL_ACCOUNT_USAGE); requests (served requests, a cancelled stream counts for what it produced), inputTokens, outputTokens and totalTokens are metered. failedRequests costs nothing and is not reported. The agent should report the month just ended once more after it closes, because the last figures arrive a little late.The route exists only with LLM_ACCESS_USAGE_KEY (mount LLM_ACCESS_USAGE_KEY_FILE), the gateway URL and the intake. The key is a gateway key limited to the one route: a gateway administrator creates a proxy-admin user's key with allowed_routes: ["/team/daily/activity"]. Measured: it reads any team's figures and was refused everything else. It reads no prompts. The raw /spend/logs route was rejected because its team filter did not filter (it returned every team's entries) and its entries have fields for messages and responses. Member budget view (application/member_budgets.py) answers the API's typed ListProjectBudgets query. GET /api/v1/projects/{projectId}/llm-budgets requires a current Party session and Beto's project.view permission. The owner checks access before and after the read, resolves the canonical Project's enrolled Waldur binding, and returns all matching allocations with their revisions. The product decision for M18-34 is the existing monetary budget in EUR. Configured Pharos model prices and offering components use virtual EUR; the default development prices are documented in <a href="/source/devops/docs/llm-serving.md.txt">LLM serving</a>. The view reports the approved budget, measured spend and remaining amount for the current UTC calendar month. Remaining budget stops at zero when spend exceeds the allowance; the measured spend remains unchanged. TPM and RPM are separate optional rate limits. Measured token counts describe usage and are not a monthly token allowance. Missing measurement configuration or a gateway failure produces null usage and remaining amounts while retaining the approved budget. A successful empty usage report produces measured zero. The measurement timestamp and next monthly reset are explicit; recent requests can arrive after the snapshot. The retained web reads on entry and on the member's explicit refresh, without background polling. The Metallo process runs this query consumer with its dedicated LLM_ACCESS_REQUESTS_BROKER_URL, existing context database, scoped usage key and Beto caller. The API receives neither gateway credentials nor provider bindings. Enable it only with an enrolled Waldur PROJECT_SOURCE_SCOPE. External model prices denominated in another currency must be converted into the configured euro accounting prices before use; this view does not relabel dollar prices. Dashboard chat entry (application/member_chat.py) answers the typed GetProjectChatAccess query at GET /api/v1/projects/{projectId}/llm-chat-access. It checks current project.view and runs.launch authority, then reuses the chat membership policy over the caller's established provider accounts in the context-owned Beto view. The local membership revision must match the current Project revision. Access and binding checks run again before returning the result. An incomplete view is retryable; an inaccessible Project remains missing. The result contains canonical Project/allocation IDs, revisions and permitted Project-prefixed model names. Each allocation is ready, preparing, blocked or unavailable. Ready means its entitlement is reconciled and this member's current chat account has its requested membership applied. An eligible member can open an active allocation while it is preparing: first OIDC sign-in creates the Open WebUI account the existing recovery sweep needs. This read writes no membership or provider grant. Blocked allocations disclose no model grants or entry link. Current Project access and the existing provider revocation workflow remain authoritative for access from an already open chat session. LLM_ACCESS_CHAT_PUBLIC_URL is the reviewed public HTTPS origin, with no path, query, fragment or credentials. GitOps derives it from the development edge catalogue or the distributed environment's llm-chat origin. The owner adaptor uses Open WebUI's /oauth/oidc/login entry; the enrolled callback remains /oauth/oidc/callback. LLM_ACCESS_CHAT_URL retains its internal provider purpose. An unpublished chat has no entry link. The dashboard reads on entry, explicit refresh and click, without polling. Metering outages do not affect this query. Key deliveries (domain/delivery.py, application/deliveries.py, adaptors/sealing.py) are the foundation for sending a personal key to its owner. The gateway shows a key's secret once, so until the key is mailed it is kept in a pending delivery, sealed (AES-256-GCM, bound to that delivery) together with its token and the owner's address. The record is internal: it is never projected to the browser and publishes no event, and the sealed value is erased as soon as the delivery is settled as sent or abandoned. A pending delivery cannot lose its secret and a settled one cannot hold one; saved state that breaks either rule is refused as corrupt. Nothing creates a delivery yet: the route that does comes in the next change. Personal keys (application/member_keys.py, domain/delivery.py, adaptors/sealing.py) give a member of a Project their own key. POST /llm-access/allocations/{id}/members takes a request ID, the member's Waldur reference (a binding the service never rewrites; it is the offering username, which a federated deployment issues as the AAI address and which may carry an @; the member's ID and the key's ID are derived from it) and their email address: In beto membership mode, application/member_admission.py checks the allocation's Project and the member's Waldur binding against Beto's loaded authority before creating or resuming a key, returning an existing key or rotating it. A refusal returns member_not_authorised; an incomplete view, missing Project or pending authority remains retryable and issues nothing. A retry or rotation may use the member's already recorded binding when the request omits it. The check applies even when Open WebUI is not configured. The member link is saved before creating or resuming a key through a record-only operation: it contacts no chat provider and reconciles no other member. site-agent mode retains its existing authority and records the member through the request-aware operation after the key and delivery are durable, including when chat is disabled.The key record takes the allocation's models and Project. Before LiteLLM is called, a random key is sealed into a provision and its hashed token is recorded. After LiteLLM confirms that exact key, its secret, token and address are sealed into the pending delivery and the provision is erased. Nothing in the clear is stored, projected or published.A retry never creates a second key: it resumes the provision or delivery. Failure to save the delivery leaves the same sealed provision for retry; an alias collision on the first provider call is refused rather than mistaken for replay. Two identical requests at once are serialised by one lock that the routes, the recovery sweep and the chat share: a re-entrant thread lock under a named database advisory lock, so the second pod of a rolling update waits for the first.Optional chat synchronisation follows the durable key and delivery. A provider outage, conflicting group or another member's failed group change may make the request retry, while the issued key and sealed delivery remain recoverable. The chat record retains pending work and a retry reuses the same key and delivery.LLM_ACCESS_SEAL_KEY (mount LLM_ACCESS_SEAL_KEY_FILE) is 32 random bytes as 64 hex characters or base64. It is a different secret from every other credential: losing it loses pending deliveries and leaking it exposes them. The route exists only with it, the intake, the gateway and key issuing; the service refuses to start with the sealing key but without them. Without it a deployment that names the gateway still starts — orders, usage and the chat group do not need it — but it records seal-key-missing for llm_access/member_keys in /diagnostics and logs it, because it then issues no personal key and stops none for a member Beto takes away. A restart cannot bring a missing key back; restoring the secret can.The key is then mailed (subtask 4C, below). A delivery stays pending until it is. Member revocation (application/member_revocation.py) takes a leaving member's keys away. POST /llm-access/allocations/{id}/members/revoke takes a request ID and the member's Waldur reference. Each personal key the member holds on the allocation is stopped at the gateway first, then revoked, and a delivery never mailed is ended and its sealed secret and address erased. It is safe to repeat, shares the lock of the route that adds members and touches nothing else (other members, the Project key, other allocations). Ending chat access is not built. Member rotation (MemberKeys.rotate_member, KeyMailing) replaces a member's key in two steps. POST /llm-access/allocations/{id}/members/rotate issues the next generation key and seals it for mailing, leaving the current key live; asking again meanwhile returns the same next key. When that key's mail has gone out, the mailer stops and revokes the member's other live keys on the allocation before settling the delivery, so the member always has a working key. An unmailed next key is abandoned after the delivery limit and the current key stays. Mailing keys (application/mailing.py, adaptors/mailer.py, adaptors/mailing.py) sends each pending key to its owner from a dedicated platform account. Metallo does not run a mail server: it needs an SMTP account on one the platform may use, and nothing starts without it. LLM_ACCESS_SMTP_HOST turns it on, and then these are required or the service refuses to start: LLM_ACCESS_SMTP_FROM, the sender address, and LLM_ACCESS_PUBLIC_API_URL, the gateway address a member's client uses, which the email shows. Personal keys (the sealing key, the gateway and key issuing) must be configured too, because mailing sends what they keep.LLM_ACCESS_SMTP_PORT (587), LLM_ACCESS_SMTP_TLS (starttls, the default, or ssl; none is accepted only for a loopback mail catcher), and LLM_ACCESS_SMTP_USERNAME with LLM_ACCESS_SMTP_PASSWORD (mount LLM_ACCESS_SMTP_PASSWORD_FILE), which go together. The password is read at every send, so rotating it needs no restart. The server's certificate is checked.LLM_ACCESS_DELIVERY_TTL_HOURS (72) and LLM_ACCESS_MAIL_INTERVAL_SECONDS (30). The dev-box root derives the emailed base from its reviewed llm-api route and appends /v1; it enables the loopback Mailpit sender only while that profile is selected. Distributed delivery requires smtp.public_api to match the reviewed edge.origins.llm-api and sends the same /v1 base. LLM_ACCESS_GATEWAY_URL retains the internal address used for provider operations. A pass mails every pending key that is due and settles it as sent, which erases the sealed secret and address. A failure is counted and keeps the secret, so the retry sends the same key; the delays grow (1, 3, 7, 15, 30 and 60 minutes after the key was made, then one more hour for each further failure). A key still pending after its time limit is given up: the key is stopped at the gateway, its record is revoked and its secret is erased, so no key is left that its owner never received. The member's next request then makes a new one. If the mail went out but settling failed, the same key is mailed again, never a second key. A failure is reported by its class alone: the address, the secret and the mail server's own text are never stored or logged. Personal key self-service Settings → LLM Keys selects a canonical Project and allocation. The member can request an emailed personal key, replace it, revoke it and explicitly refresh its status. The token settings page links to this entry. Key material is delivered by the existing mail worker; the page, API, request receipts and browser messages contain no secret, gateway token, address or provider account. PersonalAccess is the member/Project/allocation request aggregate in LLM Access's private database. application/personal_access.py records its command identity, allocation version, reserved key identities and a digest of the existing trusted member delivery binding. The snapshot and command reply commit atomically. A retry preserves the original material input; a new request while one is pending reuses that pending work. Recovery finishes the reserved key rather than selecting another generation after a lost acknowledgement. MemberKeys and KeyMailing retain issuance, sealed provision/delivery, rotation and revocation authority. The member query and command use fresh Beto Project visibility and execution authority with the enrolled Party/provider-account view. Issuance also requires a settled, active allocation and the trusted member binding already supplied to LLM Access. Missing delivery details are reported as preparing; the browser cannot submit an email or provider identity. Usage measurement is independent of access. The HTTP definitions are in <a href="/source/contracts/llm_access/http_api/keys.paths.json.txt">keys.paths.json</a>. The existing LLM_ACCESS_REQUESTS_BROKER_URL enrols query and command consumers, the durable command-reply relay and recovery; no new credential is required. Valid pending phases are requested (gateway work unacknowledged), mail_pending (key prepared, email unacknowledged) and revoking (gateway revocation unacknowledged). mail_sent means SMTP acceptance, not inbox receipt. revoked acknowledges the recorded key revocations; denied records withdrawn authority, changed material scope or an unusable reserved key. Provider outages retain the pending request. The owner retries every five seconds; the browser only reads on entry or Refresh. Acceptance, issuance, revocation and mailing share the existing member-key lock across threads and service replicas. The mail worker reloads each scanned delivery and key under that lock and checks current membership before opening its sealed payload. Self-service deliveries also check current canonical authority and the accepted allocation/contact binding. A recorded revocation fences later mail and stops its exact keys; replay cannot revoke a subsequently issued key. Revocation preserves chat membership. Replacing retires the old key after SMTP accepts the new email. An explicit revocation or denied request also settles any sealed provision. test_personal_keys.py covers recovery, retries, scope/authority changes and secret absence. llm_personal_requests.py, included in pnpm test:integration -- --group metallo, checks private PostgreSQL receipts and RabbitMQ replay. The browser fixture is pnpm exec playwright test --config acceptance/browser/personal-keys.config.ts. Live qualification additionally requires an owned trusted member binding, its Project/allocation and authorised mailbox access; the earlier Site Agent acceptance helper requires adaptation for Beto membership mode. Fixture results do not establish live gateway request or revocation behaviour. The allowance period is one calendar month (UTC), restored by the gateway's own 1mo reset: an order carries 1mo and anything else is refused. The reset happens at 00:00 UTC on the first of the month but is applied by the gateway up to about ten minutes late (measured), so a Project that ran out stays blocked that long into the new month. Details are in the design note (subtask 5B). Allowance enforcement is not strict. The gateway checks a team's budget before a request and learns the cost after it, so the request that crosses the cap is still served and a burst can go further. It is documented, with the measurements, in the design note (subtask 5A), and the overshoot shows in the usage figures. Subtask 5A stays open until admission with cost reservation exists. domain/ has no technology imports: Order evidence, the ProjectAllocation aggregate, and fulfil, the decision function.adaptors/ composes the aggregate store (postgres.py) and encodes and decodes the reconcile event's payload (projection.py), refusing unknown fields and anything the encoder would refuse; the codec ties the payload to the envelope's allocation and version.application/ has the command and LlmAccess.apply_order, built on the foundation command port. It hands a durable reconcile intent on for each change. The rules, the open questions and the next steps are in the <a href="/source/services/metallo/docs/epics/shared-llm-access/3b-order-fulfilment.md.txt">design note</a>.
- [Member Credentials](/reference/services/metallo/contexts/member_credentials/README.md): Member Credentials is the Metallo context that holds a member's offline consent for native Site work and serves one fresh access token per native action. Each Slurm job runs with the member's personal credentials; nothing is impersonated. See ADR-0035.
- [Pipeline Coordination](/reference/services/metallo/contexts/pipeline_coordination/README.md): Pipeline Coordination is the Metallo context for executing ordered steps from saved Workload revisions. The Pipeline is its aggregate root and its steps are children, saved and changed together. The domain record identifies the member and Project, the ordered steps, each step's immutable Workload revision, k8s or slurm environment and advancement condition. The Workload selects repositories and branches; each step is pinned to their commits when it starts and fetches those when preparing. Its internal operations start, read and request cancellation of a Pipeline through explicit ports. Execution Core continues to own Runs and Attempts. Pipeline Coordination requests step launches through Execution Core's published interfaces and records its own progress and Run references. Workload lookup and Project permission checks use explicit owner ports. Required cross-aggregate work uses a durable hand-off, with each owner committing its own transaction.
- [Resource Provisioning (working name)](/reference/services/metallo/contexts/resource_provisioning/README.md): Proposed owner of the execution-facing facts about a Project's resources at a Site (#763): allocations, their lifecycle, configured and applied limits (#764), and later the provisioned native accounts and applied access (#765). The name, service and boundary await the ownership decision recorded in the context catalogue; the design is in <a href="/source/services/metallo/docs/resource-site/design.md.txt">the Resource and Site design</a>. It does not own ordering, approval or billing (Waldur), identity or Projects (Beto), or Site-local enforcement (Site Runtime). It never reads Waldur or a Site with privileged credentials as a fallback for a refused decision.
- [Site Coordination — ordered commands and results](/reference/services/metallo/contexts/site_coordination/README.md): Site Coordination owns the ordered instructions sent to Site Runtime. Execution Core owns Runs and Attempts. A caller supplies an already constructed instruction using the 1C contract. This context validates and stores it, allocates its sequence and records the result when the Runtime caller replies.
- [Site Runtime](/reference/services/metallo/contexts/site_runtime/README.md): The optional <a href="/source/gitops/manifests/pharos-site-runtime/serving/README.md.txt">Run serving listener</a> accepts the API's dedicated TLS transport credential. Each request resolves current viewer authority or explicit public access from the Run owner and validates the saved Attempt, native account, service, endpoint and Pod lineage. Public access uses the accepted workload's retained account without inventing a viewer identity. Open connections revalidate every five seconds and close on expiry, revocation, replacement or unavailable authority. Metallo owns Site-local command delivery, native recovery, private workspaces and execution access. Runtime retains private SQLite journals; it has no platform PostgreSQL credentials. Its domain and application remain provider-independent. The <a href="/source/services/metallo/docs/epics/instantiate-and-run/site-runtime-context.md.txt">context design</a> describes delivery, receipts and native recovery states. For accepted execution, Runtime queries Site Coordination's authenticated /v1/site/executions/authorise interface. The owner returns the saved Run/Attempt material, current Project authority and Resource Provisioning's confirmed member account binding. The material digest, canonical IDs and account reference must agree. Success is used for the immediate effect and is never retained as a reusable permission verdict. Repository grants use the separate Run-scoped Site API routes. The Slurm Runtime composes the same way: it passes no separate personal-access ports and takes the binding, the current authority, the Run-scoped repository grants and each per-action member token from those Site API routes over its own scoped bearer. Preparation and start supply the complete Run/Attempt identity and immutable material digest to the authority query. Its workspaces are member-owned directories with no filesystem ACLs. The Runtime writes credentials into its own read-only hand-off; each member staging job creates its temporary Git configuration beside the repository directory and removes it on exit. The configured REPOSITORY_CA_FILE, when present, supplies the repository provider's CA to both staging jobs. Before entering its service sandbox, the installer runs --initialise-workspaces as the Runtime user. This creates or validates only RUNTIME_WORKSPACE_ROOT/.runtime, owned by that user and pharos-users with mode 0711; it needs no Site credentials or network connection. The exact accepted member account is retained across personal membership removal. Each new dispatch or stage-in credential issue still needs accepted-work authority; Project closure and changed management generation refuse it. Existing native observation and UID-bound cancellation or cleanup continue. Personal-access provider ports remain for isolated legacy lanes. If endpoint creation is refused after a native service was created, Runtime deletes that exact native UID and confirms cleanup before reporting the refusal. Replaying the original command retains its recorded reason. A later command for the same Attempt returns native-start-compensated; Execution Core records a terminal rejection. A definite refusal before native creation can still be retried. Temporary authority or Kubernetes Secret refresh failures defer the refresh without asking Source Control to revoke the running Attempt's grant. Source Control owns any issued renewal until the same request recovers it. Expiry-driven rotation can still invalidate the previous token before the Secret update has completed. A definite authority denial and normal execution cleanup revoke the grant. Every ExecutionDeferred names a stable code from domain/deferrals.py, and the Runtime logs that code with its own message. The platform explains each code to members with fixed text. scripts/check_architecture.py refuses a deferral without a catalogued code. Any other error is logged as runtime-error, without its text. When the pending command defers, the worker also reports the code to POST /v1/site/commands/progress, once per command and code. The report is best effort: the command stays pending whether or not it arrives, and a failed report is sent again with the next deferral. When a repository grant fails and the cleanup revocation also has to wait, the grant's own code is the one reported; a definite denial still waits for the revocation before it is acknowledged. Observations of an unfinished workload can say why it has not run yet, with a code from domain/native_waits.py and never provider text: Kubernetes Pods that cannot be scheduled, whose image cannot be pulled, whose container cannot be created or keeps crashing, and pending Slurm jobs by their state_reason. A service's reason comes from the same Pods its health check below reads, so a Pod that cannot be trusted gives no reason, and private reads keep refusing it. A Deployment or InferenceService fails when its conditions say so, or when one of its own Pods (found by its labels and owner chain) has a container or init container that is not ready and has restarted more than SERVICE_RESTART_LIMIT times (default 3) since the service last stayed healthy for ten minutes (#1054). Kubernetes normally retains cumulative restart counts, so Runtime saves each container's count in the private journal once the service has been healthy, without a restart, for ten minutes, the period after which the kubelet resets its own restart back-off, and counts only later restarts. A node restart can reset a container's count. Runtime detects decreases against the last observed counts and saves a zero baseline for that container before judging its health, so the restart limit still applies after Runtime replacement. Other containers retain their baselines. A service that is ready for a while between crashes therefore still fails. Without that check a crash-looping service would wait for the rollout's progress deadline, and one that crashes after becoming healthy would never fail. The failed record keeps the container, its new restarts and its last exit code until the observation can carry a reason (#1053). Observing checks only that a Pod belongs to the service, not its exact template, so Pods a controller rolls after a KServe or serving-runtime upgrade still count. A Pod whose ownership or status cannot be confirmed is left out, and without Pods the observation uses the native object's conditions alone: a Pod never fails an observation, while private reads keep every check. A failed Deployment or InferenceService can leave native Pods retrying. Runtime records a failing cleanup state, deletes the exact service and endpoint UIDs, and confirms their absence before reporting terminal failure. Lost deletion responses and interrupted cleanup resume from the private journal. Private health reads do not trigger cleanup. Runs already reported as terminal by older code need separate operator recovery because Execution Core has stopped their supervision. Terminated and failed Kubernetes services report cleanupComplete only after their native service and endpoint have gone, temporary helper Jobs have finished deleting, repository grants are revoked and the ephemeral Attempt PVC has gone. Workspace disposal records its pending state before deletion and a completed tombstone after confirmation; retries and Runtime replacement cannot recreate that Attempt's workspace. Cancelled batch Jobs, abandoned preparations and permanently refused inputs dispose of their helper and workspace by the same path, once no native Job remains. A batch Job that finishes during cancellation retains its natural terminal outcome in the journal; Runtime reports it after native deletion and workspace disposal complete, including after restart. Native deletions use exact UIDs and independently enforced ownership policies. Execution Core can then publish the accepted-work release needed for account withdrawal. Completed batch Jobs retain their workspaces for authorised output and log reads. Their retention cleanup remains separate and can still consume Project storage quota. Existing terminal service Runs recorded without cleanup confirmation still need operator recovery because their supervisor has already stopped. Workspace PVC creation distinguishes a definite Kubernetes admission or quota refusal from an uncertain response. A refused create records native-request-denied through the normal command outcome, allowing subsequent commands, including cancellation, to cross the ordered Runtime queue. Conflicts, connection loss, authentication failures and server errors retain the pending command for recovery. The workspace intent remains journalled before creation and a recovered PVC must still match its saved identity. Kubernetes admission independently compares the submitted native material with the owner's saved payload and reads the current native account UID. The admission process has its own Site identity and read-only native grants. Restricted CEL policies continue to protect Pods and KServe controller work. Production composition lives under services/; installation details are in the <a href="/source/gitops/manifests/pharos-site-runtime/admission/README.md.txt">native admission profile</a>. Private reads use a separate owner-issued permit. read_output, read_logs, read_file and predict accept read_permit_id alongside the actual requesting Party and Project. Production checks the permit's exact Run, Attempt, viewer, read kind, path and byte limit before reading and before returning data. A completed batch Run can be read by another currently authorised Project member; the original initiator's personal account is unnecessary. Service files, logs and prediction require the native service to remain active. Its ephemeral workspace is discarded after termination, so no service output read is available afterwards. These reads use the current viewer's confirmed member identity and leave the accepted dispatch deadline unchanged. A viewer with no account in the Run's allocation is refused. This covers Kubernetes only. Slurm log reads use the separately activated <a href="/source/devops/hpc/site/RUN-LOGS.md.txt">dedicated spool</a>. Other Slurm output files remain owned by the initiator's POSIX account, so reading those files as the viewer still needs its own decision (#932). Private inference reads accept the installed KServe controller's INFERENCE_SERVICE_NAME environment addition only when its value is the exact saved InferenceService name. Duplicate, altered and unexpected additions remain invalid, alongside the existing controller UID and Pod template checks. GPU inference uses the class catalogue's vLLM runtime and its OpenAI routes: GET /v1/models and completed POST /v1/chat/completions, /v1/completions or /v1/embeddings requests. Prediction bodies must name the saved model and cannot request streaming. Duplicate JSON keys, non-finite values, other routes and CPU runtime argument additions are refused. Current permission, exact native identity and byte limits still apply before and after the request. This private Runtime method requires a separate authorised browser serving path; deployment qualification is described in the <a href="/source/acceptance/demo/gpu-staging.md.txt">GPU staging procedure</a>. Active service file reads execute the bounded, symlink-refusing reader in the verified service Pod, with its immutable image's Python interpreter. Native identity and current permission are checked before and after the read. This avoids mounting the live workspace in another Pod, which can change its SELinux label and revoke the service's file access. The reader selects the fixed /usr/local/bin/python or /usr/bin/python3 path for the managed and upstream KServe images; workload PATH and Python environment overrides cannot select the interpreter or its imports. The batch read-only workspace helper carries the opaque permit ID. Admission confirms it through /v1/site/executions/authorise-read and admits only the expected read-only helper without repository credentials. The viewer's own account creates the helper and admission requires that caller, so another member's account in the namespace cannot use the permit. Launch, preparation and mutable native objects cannot use a read permit. The owner rechecks current permission on every use; an expired permit requires a new authenticated request. See <a href="/source/services/metallo/contexts/execution_core/private-reads.md.txt">Execution Core's private read contract</a>. Repository grants carry separate stage-in and stage-out identities. Stage-in requires current continuation authority and the original deadline. Stage-out is limited to the exact saved repository of an outstanding submitted Attempt, so output recovery can cross that deadline and a policy cancellation. It cannot admit native compute. Repository ownership is checked on every issue; completed Attempts cannot renew. Kubernetes cleanup revokes both grant identities before acknowledging, and Slurm records each staging obligation in its private journal. GitLab's date-based expiry bounds a recovery token to the next UTC midnight; normal cleanup and Source Control reconciliation revoke it earlier. A Slurm submission whose outcome is uncertain is never repeated: the Attempt waits until its job is visible. A connection slurmrestd refused carried nothing, so that Attempt goes back to prepared and submits again once slurmrestd answers. slurmrestd answers a job that slurmctld refused with HTTP 500 and the controller's error_number, such as 2014 when no node has what the job asks for. Only an error saying the job does not fit the Site's nodes, partition, time limit or GRES (DEFINITE_REFUSALS in slurm_access.py) is a definite refusal, slurm-error-<number>. Execution Core ends the Run when Slurm refuses stage-in at start; when it refuses a later stage, the Attempt fails after its cleanup. Errors that turn the job away for now, before slurmctld creates it, such as 2050 while the jobs already submitted fill MaxSubmitJobs, send the stage back to prepared, and it is submitted again (slurm-submission-limited). Communication errors and any other 500 stay uncertain. A Slurm compute job asks for GPUs through the Site's class-to-GRES map, for example {"gh200": "gpu:nvidia_gh200_144g_hbm3e"}, as tres_per_node, the field Slurm fills from --gres. The class's GRES is saved when the Attempt is accepted, so a later change to the map never changes that Attempt's compute job. Stage jobs never get a GPU and never read the map. The Site's max_wall_seconds bounds the compute job only. A class the map lacks, the legacy gpu class included, is refused as unsupported-slurm-accelerator-class, and a longer request as slurm-wall-time-above-limit. Both refusals come before the workspace is touched, and Execution Core ends the Run with their reason. The executors take both settings as arguments; the production composition does not pass them yet. The member-token owner's four permanent consent refusals end the Attempt. Runtime saves the fixed refusal before cleanup, so a restart or a later command cannot submit another stage. Before stage-in, it returns a command rejection once cleanup succeeds. Between stages, it returns a failed observation with failureCode and cleanupComplete; already completed native jobs retain their recorded identities. Repository revocation and workspace cleanup remain retryable. Issuer text, tokens and sealed values never enter the refusal or journal. Temporary token-owner failures defer the same stage without producing a terminal outcome.
- [Source Control](/reference/services/metallo/contexts/source_control/README.md): Source Control owns repository/provider bindings and delegated source-control effects for canonical Beto Projects. It reconciles one private GitLab group per Project and its direct members, and it resolves Project repositories and reads their action files at exact commits. It also owns recoverable, repository-scoped credentials for accepted Run operations.
- [SSH Access](/reference/services/metallo/contexts/ssh_access/README.md): SSH Access is the Metallo context that owns members' SSH public keys for each Site and their #678 lifecycle. Waldur does not store these keys. The access agent on the Site applies them to LDAP through the Site API. See ADR-0055 and DD-0055.
- [Workload Portfolio](/reference/services/metallo/contexts/workload_portfolio/README.md): Workload Portfolio is the Metallo context that owns Workloads. The Workload is its aggregate root; its immutable, numbered revisions are children, saved and changed together. A revision holds the execution specification and required inputs that Execution Core plans from. Execution Core and Pipeline Coordination reference exact Workload revisions. They read saved Workloads through service-level owner adaptors in <a href="/source/services/metallo/adaptors/workload_evidence.py.txt">adaptors/workload_evidence.py</a> and check member access themselves.
- [AAI](/terms/aai.md)
- [Daedalus](/terms/daedalus.md)
- [Dataset](/terms/dataset.md)
- [eduGAIN](/terms/edugain.md)
- [eIDAS](/terms/eidas.md)
- [Gated Resource](/terms/gated-resource.md)
- [GitLab Root Actor](/terms/gitlab-root-actor.md)
- [GRNET](/terms/grnet.md)
- [Impersonation Token](/terms/impersonation-token.md)
- [K8s Root Actor](/terms/k8s-root-actor.md)
- [Katib](/terms/katib.md)
- [Kubeflow](/terms/kubeflow.md)
- [LFS](/terms/lfs.md)
- [MLflow](/terms/mlflow.md)
- [Model](/terms/model.md)
- [MyAccessID](/terms/myaccessid.md)
- [Node Hours](/terms/node-hours.md)
- [OIDC](/terms/oidc.md)
- [Open OnDemand](/terms/open-ondemand.md)
- [Pharos AAI](/terms/pharos-aai.md)
- [Pharos AI Factory](/terms/pharos-ai-factory.md)
- [RBAC](/terms/rbac.md)
- [RCIAM](/terms/rciam.md)
- [Recipe](/terms/recipe.md)
- [Resource](/terms/resource.md)
- [Slurm](/terms/slurm.md)
- [Space](/terms/space.md)
- [Themelio Catalogue](/terms/themelio-catalogue.md)
- [Themelio Explore](/terms/themelio-explore.md)
- [Themelio Project](/terms/themelio-project.md)
- [Themelio Web](/terms/themelio-web.md)
- [Themelio](/terms/themelio.md)
- [Upskilling Material](/terms/upskilling-material.md)
- [Valkey](/terms/valkey.md)
- [Waldur Site Agent](/terms/waldur-site-agent.md)
- [Waldur](/terms/waldur.md)