> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # Architecture The platform's five top-level service boundaries are Web, API, Metallo, Themelio and Beto. Metallo owns Execution Core for internal Run/Attempt identity and persistence, Site Coordination for ordered execution commands and recorded results, and Site Runtime for Site-local delivery, native recovery and member/workspace rules. Pipeline Coordination owns Pipeline records and their ordered child steps, with internal start, read and cancellation-request operations through published owner command/query interfaces and context-owned persistence ports. Its private PostgreSQL database commits each snapshot, receipt and pending coordinator work atomically. Its leased coordinator submits stable step launches through Execution Core and records Run progress through its published queries and completion events. Workload Portfolio owns Workloads and their immutable revisions. Execution Core and Pipeline Coordination read saved revisions through Workload Portfolio's published query interfaces. Source Control owns repository/provider bindings and delegated GitLab effects. Its first slice owns consumer projections of Beto Project/Party facts and reconciles Project groups and direct memberships. It never creates provider accounts. Its dedicated process owns repository requests, publication and recovery. Its [context guide](/reference/services/metallo/contexts/source_control/README.md) records owner bootstrap, activation requirements and qualification limits. Beto owns [Party Governance](/reference/services/beto/contexts/party_governance/README.md), the [Project context](/reference/services/beto/contexts/project/README.md) and the [Support context](/reference/services/beto/contexts/support/README.md), with separate logical PostgreSQL databases and credentials. Project owns its Waldur Project and membership acquisition ports. Support links Pharos users to Zammad customers and files and follows their tickets. Metallo consumes canonical queries/events when its receiving capabilities are enrolled; its other provider capabilities remain separate. Beto also owns [Marketplace](/reference/services/beto/contexts/marketplace/README.md), which has its Order domain, database and private OrderPlaced event, and serves PlaceOrder and ListProjectOrders over RabbitMQ and the API's routes with stand-in catalogue and buyer ports. [SSH Access](/reference/services/metallo/contexts/ssh_access/README.md) owns members' SSH public keys per Site, their lifecycle and the Site work that applies them. [Resource Provisioning](/reference/services/metallo/contexts/resource_provisioning/README.md) owns allocation and native-account evidence under its working boundary. [Member Credentials](/reference/services/metallo/contexts/member_credentials/README.md) owns sealed offline consent and native-action token refresh. Themelio has no populated bounded contexts. Process names do not pre-assign future domain authority. The [context catalogue](/reference/docs/context-catalogue.md) describes each registered context's responsibilities and owner, active proposals and unresolved boundaries. The Source Control context belongs to Metallo; its scope and remaining boundary decisions are recorded in the catalogue and its context README. LLM Access belongs to Metallo and owns its Project LLM allocation and provider fulfilment model; see [its context guide](/reference/services/metallo/contexts/llm_access/README.md). The shared rules below apply as capabilities are introduced. Context READMEs record active operations, adaptor choices, implementation limits and verification. Reusable foundations and empty registries alone do not establish a live capability. ## Repository map ```text services/ web/ src/{app,features,adaptors,contexts} Next.js api/ {services,application,adaptors,tests} Go module metallo/ {services,contexts,foundation,contracts,adaptors,integrations,tests} themelio/ {services,contexts,foundation,contracts,adaptors,integrations,tests} beto/ {services,contexts,foundation,contracts,adaptors,integrations,tests} contracts/ {events,realtime,persistence,http}, contexts.json scripts/ verification, contract generation and explicit replay dx/ personal tooling: common desktop/controller and service-specific tools gitops/ Kubernetes base and local/k3s overlays devops/ runtime bootstrap configuration and development-host Ansible docs/ decisions, onboarding and responsibility review legacy/ exact archived main tree, excluded from active builds and workspaces ``` Metallo and Themelio keep folders directly at their package roots. Themelio exposes one `themelio` Python package and an empty Dependency Injector composition; see its scaffold guide. Metallo's folders sit directly at its package root. Its wheel and editable installation expose the single `metallo` package; callers import `metallo.foundation`, `metallo.adaptors` and `metallo.services`. There are no copies of the same context in different language layouts. ## Authority and dependencies Domain and application code depend only on provider-independent ports and context-local values. Driving adaptors translate HTTP or broker delivery into those ports. Driven adaptors implement persistence and publication capabilities. Service-local `services` folders are composition roots. Shared root contracts contain wire/schema specifications, never cross-language business implementation. Composition roots bind concrete resources and own their lifecycle. Application handlers receive explicit constructor ports. Validate registrations and resolve dependencies before workers start; cancel and drain workers before closing their resources, including after partial startup failure. See [decision 0046](/reference/docs/decisions/0046-service-communication-and-composition.md). One command or incoming event changes at most one aggregate. Persist state, version, command/consumer receipt, recorded outcome and outgoing intents in one PostgreSQL transaction. Aggregate children have no separate write repository. Required reactions use a durable asynchronous hand-off, including within one process. Current-state storage is authoritative; an outbox does not imply event sourcing. Canonical IDs are independent of names and provider identifiers. Each context exclusively owns any data it persists, with dedicated credentials and enforced logical isolation. PostgreSQL contexts use separate logical databases, migration histories and restricted runtime roles. These databases may share a physical server; each role is denied access to other contexts' databases. Learn facts from another context through its published integration events or query interfaces. Maintain consumer-owned projections where local reads are needed. Between services, use the versioned Protobuf contracts and messaging conventions defined in [ADR 0046](/reference/docs/decisions/0046-service-communication-and-composition.md). Site Runtime declares SQLite persistence for private Site-local journals, with no platform PostgreSQL database, credentials or migration enrolment. A supporting context that wraps an authoritative third-party service may remain storage-free. Its domain model, provider mappings and published interfaces still belong to that context. Durable receipts, outboxes and local projections require context-owned storage even when the provider remains authoritative. Storage-free contexts declare `persistence: none` in `contracts/contexts.json`; administration omits their databases and migrations. The retained Project directory mock shares no authority with the context's canonical source-managed aggregate. See [context registration](/reference/docs/adding-a-context.md). A process hosting several contexts holds their credentials and forms a shared security boundary. Context-specific pools and ports preserve ownership inside that process. Isolation from compromised sibling code requires separate processes and separate secret delivery. Required external effects use durable asynchronous hand-offs outside aggregate transactions, with explicit idempotency and persisted recovery state. Read-only provider queries use application ports and owner adaptors. ## Delivery foundations Beto groups its shared boundary mechanisms in its [adaptor tree](/reference/services/beto/adaptors/README.md). Beto's plain command handlers use per-invocation write repositories inside a shared aggregate transaction. Durable command, event, reply and explicit realtime outboxes preserve receipts and exact delivery bytes. Its internal HTTP surface contains operational endpoints. Themelio has the corresponding independently packaged Python foundation, with separate persistence, messaging, realtime and HTTP adaptors. Its generated technical contracts and typed registries contain no business registrations. Its process exposes health and authenticated diagnostics without opening external connections. Future Themelio context databases receive its service-local, administrator-applied delivery extensions; the shared baseline and other services retain their schemas. Metallo has the corresponding typed Python ports, adaptors and managed worker lifecycle. Metallo opens separate Execution Core and Site Coordination database pools; broker and realtime delivery in the main HTTP process use explicit registries, currently empty. Its request transport receives the API's typed queries and commands on owner queues. Command replies commit with the deciding transaction in Metallo's service-local delivery extension and are relayed after commit. Execution Core and Workload Portfolio answer requests in the dedicated `metallo-requests` process, installed paused until their outside owners are connected. The dedicated Pipeline Coordinator composes its completion relay and subscription. The dedicated SSH Access process answers the API's HPC key requests from its own RabbitMQ queues and runs its propagation worker. A dedicated Metallo HTTPS composition exposes the Site pull/result protocol using its Site Coordination database and a Runtime identity-verification port. Kubernetes TokenReview authenticates projected ServiceAccount credentials; configured subjects select the environment. Native Runtimes and the SSH access agent present a native bearer instead: the listener keeps only digest-only grants that name the principal, Site, route scope, expiry and revocation, read afresh on every request. The same listener serves the SSH access pull, snapshot and acknowledgement routes; a grant for one route family is refused on the other. Runtime callers receive no database credentials. Every GitOps installation runs it as the `site-api` Deployment; its only cluster-scoped right, `create` on `tokenreviews`, comes from the dev box's bootstrap or the kubectl overlays, never from Argo. Metallo packages the `contexts/site_runtime` bounded context for the remote Runtime. Its provider-independent domain owns checkpoint, member-binding and repository/workspace rules; its application layer owns ordered processing, recovery, current-access workflows and the transport, codec, journal, native executor and provider ports. HTTPS, rotating identity, SQLite, Kubernetes and Git/workspace implementations belong to its adaptors. Process startup and wiring remain in `services/`. The shared Site wire codec lives in `adaptors/site_protocol` so context adaptors can use the same contracts without importing another context's implementation. The Runtime SQLite adaptor stores a pending command, exact result and acknowledgement cursor on persistent Site-local storage. Native executors must recover uncertain effects through stable command/Attempt identity; composition requires an executor explicitly. These journals are separate from the platform's authoritative PostgreSQL state. Execution Core still owns Run and Attempt state; Site Coordination owns the command stream and recorded results. The Site Runtime DD defines the boundary and the unchanged command and journal contracts. Its member-access component queries current bindings and permissions before each new native submission. The Kubernetes adaptor impersonates an approved member ServiceAccount using a separate restricted Runtime credential. Namespaced RBAC and administrator-installed admission policies enforce access independently of Python. The Kubernetes Job executor records preparation and native identity in a second private SQLite file, creates one deterministic Job per Attempt and recovers by checking its UID and specification. Reconciled services carry a required `serviceKind`: `application` creates a Deployment using the fixed platform image; `huggingface-inference` creates an InferenceService using KServe's installed Hugging Face runtime. Both fetch repositories into private Attempt PVCs and run pre-commands in an init container in the same Pod as the main application/server. The shared worker routes by family and saved kind. Its permanent Runtime Deployment retains `delivery.sqlite`, `jobs.sqlite` and `services.sqlite` on one PVC. Namespace-scoped reads and UID-bound cleanup survive member revocation. Private file/log/endpoint access checks current permissions and exact native identities. Live provisioning and repository credential providers remain explicit dependencies. Execution Core constructs lifecycle instructions from saved Runs. An outer adaptor uses Site Coordination's command/query interfaces to save commands and read recorded results; contexts do not access each other's tables. Applying a result changes one Run, its child Attempt, receipt and existing projection atomically. Observations carry workload state; command acknowledgement alone does not establish completion. The Run Supervisor polls Execution Core's durable work table. Tokens, generations and database-time leases fence progress and Run commits. Commands are saved before dispatch, and terminal state and a typed completion event commit together. Its composition requires explicit provider ports; the Kubernetes Job adaptor is qualified separately from the shared supervisor and the remaining native profiles. Beto composes Support with durable command reactions, integration through the Zammad adaptor and explicit realtime publication: ticket and message revisions go to each owner as principal-targeted messages. Support's Zammad webhook listener is a separate external-integration port. RabbitMQ publication uses durable messages, mandatory routing and publisher confirms. Command and event consumers manually acknowledge after their owner transaction commits state or projection changes, receipts, outcomes and outgoing intents as applicable. Query consumers acknowledge only after a routable reply receives a publisher confirm; uncertain reads may be repeated. See [decision 0046](/reference/docs/decisions/0046-service-communication-and-composition.md). Bounded retry and dead-letter transfer preserve message identity; explicit replay resets only the retry metadata. Context-private topics remain private. Each durable hand-off identifies who owns recovery. Until acceptance and required outgoing intent commit, the receiving consumer remains responsible for retry. After that commit, persisted dispatch work and its receiving consumer own the next step. Broker publication and database commit are separate operations; consumers tolerate duplicate publication and redelivery. A lost commit response leaves an uncertain outcome, resolved by retrying with the same command identity and material input. Business completion is established by the owner's recorded outcome; transport acknowledgement alone establishes only its delivery stage. Integration events and browser projections have separate versioned Protobuf envelopes. Opaque envelope payload bytes must contain registered context-specific typed Protobuf messages. Execution Core's completion contract lives under `services/metallo/contracts/events` and generates only Metallo Python bindings. The dedicated Metallo Pipeline Coordinator consumes completion through RabbitMQ, with consumer-owned receipts, progress and pending work committed together. Its service-readiness and recovery queries use Execution Core's published query interfaces. The coordinator and the execution worker build their owner adaptors in code and install paused until the Project provisioning connections exist; native pipeline qualification remains a separate integration task. Execution Core records a typed Run projection for the Run's canonical Project audience and enables its fenced delivery relay. Launch acceptance records its request and decision without a browser projection. Metallo's projection registry uses an explicit `None` entry for such internal aggregates; unregistered keys still fail closed. Enabled projections retain atomic bytes and delivery records. Unknown contracts fail closed rather than allowing arbitrary JSON through a generic business API. The API owns technical sessions in persistent Valkey. It holds no business database credentials; its broker credential may only send registered commands and queries and read its own replies. Its business routes come only from its [OpenAPI contract](/reference/contracts/http/README.md) and reach owning contexts over RabbitMQ. A separate realtime gateway receives the Centrifugo administration key. The experimental [MCP edge](/reference/contracts/mcp/README.md) remains within the API boundary. Its stateless Go gateway calls the generated HTTP client; a separate OAuth broker owns its private technical PostgreSQL state. Dedicated assertions bind each invocation to current Party authority and its declared operation. [ADR 0071](/reference/docs/decisions/0071-generated-mcp-api-edge.md) records the experiment. Context publishers send an exact Protobuf envelope with a Project, Party or principal target; the gateway verifies the owner and permitted target kinds, then derives the Centrifugo channel. Execution Core has a Project-only publisher credential; Support has a principal-only one. The API session credential can also publish typed Party-targeted readiness notifications, persisted atomically with its technical readiness observations in Valkey. The static Next.js web shell authenticates through the API-owned AAI flow and loads the Project directory through the API's generated OpenAPI client. The configured OIDC issuer may be local or remote; the browser uses the same same-origin start, callback and session protocol. The isolated integration stack retains its operator-code fallback. Each deployment must enrol the web's public origin as an API callback because login and session cookies are host-only. The dev-box installation enrols exact callbacks for retained Themelio and the Pharos web; the API fixes the selected callback in each login attempt. The shell opens the existing Centrifugo Protobuf WebSocket protocol. Its provider decodes registered messages and exposes typed hooks without a channel API. Ordinary authorised connections receive the principal channel. Limited API-owned sessions receive only their own Party readiness channel until AUP is current. An application outbound port supplies authorised Party and Project audiences. Its adaptor composes published owner queries: Party Governance supplies the Party authority available to the principal, and Project supplies current Project access for that Party. The API-owned authentication composition revalidates the canonical Party and supplies its own Party readiness audience for limited sessions. Sessions with current AUP retain the Project owner queries and authority expiry described in [the browser delivery policy](/reference/CLIENT-SUBSCRIPTIONS.md). Browser delivery uses server-authorised subscriptions without polling, except the Run log panel's polling of its logs route (decision). ## Runtime and administration The [workstation development loop](/reference/docs/development.md) borrows `pharos` and `themelio-compat` from Argo CD with a renewable deny sync window. Takeover owns technical session/lease facts only; it introduces no bounded context or business aggregate. Its commands select, replace and return workloads; queries report the session and logs. Applications receive their existing scoped configuration, and the separate controller retains administrator access. The last delivery remains the baseline; lease expiry lets Argo CD restore it. Personal development tooling lives in `dx/common`, with service-specific tools under `dx/metallo`, `dx/themelio` and `dx/beto`. It owns technical workstation session and lease facts; it adds no business bounded context. Commands take over, reload and hand back services, while queries inspect their state. Shared deployment libraries remain in `devops/platform`, Ansible in `devops/ansible` and canonical manifests in `gitops`. Application and deployment code cannot depend on DX implementation; the architecture check includes negative fixtures for that boundary. See the DX directory guide. Official [release publication and selection](/reference/docs/platform-releases.md) reuse the image catalogue and service manifests through `devops/platform`. Signed OCI packages and Git-selected digests retain service Application boundaries; the committed installation policy separately controls activation. Local deployment uses a dedicated Minikube profile and Kustomize overlay. Development k3s uses the same Kubernetes base and a separate image overlay. The independent Metallo dev-box controller in `dx/common` uses Make and Docker for its tools and takeover applications. Its dedicated namespace, private target profiles and journalled mode switches leave the original development workflow intact. It compares runtime inputs with the last ready deployment to select changed applications. A private browser router exposes the installed UIs through target-specific `localtest.me` names. The host HTTPS edge has a separate owner, `edge`, which resolves the committed platform service catalogue, retaining the existing issuer and CA. A laboratory brings its own routes file, which the edge serves on the laboratory's own hosts; the platform names no laboratory. A pinned Istio/CNI stage installs the shared loopback ingress gateway; catalogue routes and paired NetworkPolicies admit its exact backends, and the Ceph dashboard is restricted to management clients. The AAI's browser NodePort remains for rollback. Application namespaces keep injection disabled in this first phase ([decision 0038](/reference/docs/decisions/0038-platform-edge-catalogue.md)). Argo CD remains a subsequent stage. PostgreSQL, RabbitMQ and session Valkey use PVCs; realtime history uses `emptyDir`. Service accounts have no automatic Kubernetes API token, applications run as non-root, and NetworkPolicies restrict named traffic paths and DNS. Application images contain no administrator credentials. Beto's Zammad token is an agent token limited to the Pharos groups, not `admin.user`, which Zammad treats as full administration ([decision 0063](/reference/docs/decisions/0063-zammad-integration-token.md)). Schema migration and broker topology creation are explicit administration operations, separate from application startup. Credentials are generated into ignored private configuration and delivered through workload-specific Kubernetes Secret volumes. This is a development skeleton. Production identity, TLS rollout, high availability, backup/restore and runtime qualification need their own implementation and evidence. The [shared dev-box AAI](/reference/docs/decisions/0035-local-aai-laboratory.md) in `dx/common/local_aai` provides a development Keycloak with the clients Pharos registers with GRNET, without registering production contexts. The Icarus-hpc laboratory uses the shared AAI to exercise the DD-0055 Site on the HPC host: OpenLDAP, the Waldur Site agent, native Slurm, the Site Runtime and the mixed Slurm-to-Kubernetes Pipeline. It is installed by `make metallo-icarus` after the platform is deployed. Dev boxes take block, filesystem and S3 storage from Rook/Ceph on a dedicated disk, installed by the final deployment's storage role ([decision 0037](/reference/docs/decisions/0037-dev-box-storage.md)). The development platform now bootstraps pinned Argo CD and hands its installation and catalogue routes to exact-commit Applications. Ansible retains host, storage, mesh and public trust ownership; the personal controller retains applications. Versioned ownership records and admission policies prevent overlapping route writers. [Decision 0039](/reference/docs/decisions/0039-dev-box-argocd-ownership.md) and its companion DD describe the development implementation; wider review remains open. The accepted reset is recorded in [decision 0034](/reference/docs/decisions/0034-fresh-skeleton.md). API command bindings translate declared POST operations into owner Protobuf commands, preserving path identity, validated JSON, idempotency and expected version. The authenticated subject reaches the owner handler in plain command context and participates in receipt identity. See [decision 0046](/reference/docs/decisions/0046-service-communication-and-composition.md). ## Workstation desktop Pharos DX is a technical workstation-session interface under `dx/common/desktop`. Its queries describe local runtimes, the remote deployment and the existing controller's lease. Its commands invoke reviewed setup/reset, takeover, hand-back and private log capture through that controller. It introduces no business aggregate or service context. The Tauri Rust host runs the selected trusted checkout's controller and emits credential-free status through local IPC events. Its bundled React webview has no shell, filesystem or remote HTTP capability. Explicit credential access uses a separate, allowlisted SSH query: native Copy returns no secret to the webview; Reveal returns one password briefly to its requesting window. Neither path publishes credentials in status, Activity or logs. Existing controller ownership and recovery are authoritative; opening the app performs observation only. See [0045](/reference/docs/decisions/0045-development-loop.md) and the desktop guide. Personal development boxes share provider processes through `pharos-infra`, with consumer-owned databases, broker scopes and Valkey ACL prefixes. Administrator credentials remain confined to provider administration. See [decision 0048](/reference/docs/decisions/0048-shared-development-infrastructure.md). Inventory-driven distributed administration lives in `devops/platform`, with explicit environment/private paths and no dependency on personal tooling. Ansible owns selected foundation layers; Argo receives an exact source revision. Existing-site observation and fresh Kubespray installation have separate guards. See [decision 0051](/reference/docs/decisions/0051-distributed-inventory-setup.md) and the [operator guide](/reference/docs/distributed-setup.md).
Page source Source snapshot ยท Revision `f928834c8`.