#0067 — Kubeflow is one pinned Argo CD composition

Status: Proposed. Date: 29 October 2026.

The live Icarus staging cluster runs Kubeflow community distribution 26.03.1, installed by the legacy kubeflow-deploy Ansible role and then corrected by hand: NetworkPolicies, Pod Security labels and the Istio global deny-all policy were deleted, Dex was replaced by GRNET OIDC through oauth2-proxy, and the ingress gateway was patched to NodePorts. That installation works, but it is not the composition the platform now uses, and the post-apply corrections are not reviewable or repeatable.

#Decision

Kubeflow becomes five Argo CD Applications built from the same pinned upstream 26.03.1 source, with all platform differences expressed as Kustomize patches. The existing cert-manager, Istio control plane, GPU Operator and Ceph remain infrastructure and are not reinstalled by Kubeflow.

The composition installs kubeflow-core first, then kubeflow-notebooks, and only then Pipelines, Trainer and Katib. A closed-composition check accepts exactly this child set; an extra or missing Kubeflow Application fails delivery.

#Istio in Profile namespaces

Profile namespaces do not get the upstream istio-injection: enabled label. The Profile controller's namespace-label ConfigMap is patched to keep injection off, so Site Runtime Jobs and other execution workloads are never modified by Istio.

Notebooks still get a sidecar, but only Notebooks do. A PodDefault named istio-sidecar is created in every Profile namespace and adds the pod label sidecar.istio.io/inject: "true". The injector's object rule in an unlabelled namespace selects that label, so the Notebook controller's generated StatefulSet is injected and Site Runtime Jobs are not. The Jupyter web app selects the PodDefault by default. This gives the dashboard and Notebook traffic the same authentication path as upstream without changing Job templates or the Site Runtime admission policy.

Native sidecars are supported by the installed Istio 1.30 and Kubernetes 1.35 server, but they are not selected for Profile namespaces: the Site Runtime admission policy would still reject the volumes and init-container profile Istio adds. Namespace-wide injection is rejected for the same reason and because it keeps Jobs from finishing. The kubeflow and kubeflow-system namespaces keep the upstream injection labels for their own control-plane services.

The Profile controller only ever adds a namespace label, or deletes one that its ConfigMap names with an empty value; it never overwrites or removes a label it does not name. Site Runtime's labels therefore survive its reconcile, and the ConfigMap may not name any of them (DD §3).

#Pod security in Profile namespaces

Profile namespaces keep pod-security.kubernetes.io/enforce: restricted; the Site Agent writes it and the guard pins it. Istio therefore runs with its CNI plugin, so injection adds a non-root istio-validation container instead of the privileged istio-init that needs NET_ADMIN. Every Kubernetes pod template the composition ships declares the restricted profile itself. Pod Security admission refuses an incomplete pod before any ValidatingAdmissionPolicy sees it; the platform's policy exists for a namespace that loses the restricted label, and states PSA's restricted profile exactly, including a container's own runAsNonRoot and seccompProfile taking priority over the pod's. The platform never mutates a workload's securityContext and never sets runAsUser, so an image keeps the user it chooses; admission kinds stay on admissionregistration.k8s.io/v1 because a beta MutatingAdmissionPolicy is served by no API server the platform installs and would break its bootstrap. The Site Runtime Job policy is narrowed to its own callers so Katib and Trainer can create Jobs in the same namespace. No workload depends on a hand-deleted label, and only non-root images run there (DD §3).

#Identity

kubeflow-userid is the member's site_username at the Site (decided 29 October 2026): the Waldur offering username, which Metallo already resolves to the Party through the Party's site binding. No email, AAI subject or Party ID reaches the Site as an identity, and the k8s Site Agent needs no personal data it does not already hold.

GRNET returns site_username in the userinfo response only, never in the token. oauth2-proxy therefore takes it from the userinfo endpoint at login and keeps it in the session, writes it as kubeflow-userid, and the gateway copies that header upstream through the platform's oauth2-proxy ext_authz provider. Only that login sets the header: a value sent by a client is overwritten, and requests that skip the check are denied on the Kubeflow host (DD §4).

The oauth2-proxy runs in the oauth2-proxy namespace and exposes the Service oauth2-proxy on port 4180. The staging callback is https://kubeflow.staging.pharosfactory.ai/oauth2/callback; the devbox callback is https://kubeflow.localtest.me/oauth2/callback. The route, edge certificate and DNS entry are owned by the #800 edge work, not by Kubeflow. The client secret follows the platform's private-file pattern: a mode-0600 input on the administration machine, delivered as a Kubernetes Secret through standard input, never through Git or Argo values.

A member without a site_username has no Site account, so they have no Profile and no Kubeflow access. The value is a Site name, not a global identity: a changed username means the agent removes the old KFAM pair and writes the new one. Accepted work keeps its execution account (#765) but never the person's UI access.

#Profiles and the k8s Site Agent

kubeflow-core replaces the #742 design entry and keeps its two security corrections: no cluster-admin binding, and no Istio injection label on Profile namespaces. #742 is not deployed on Icarus. The complete legacy 26.03.1 installation is removed at cutover, and the k8s Site Agent recreates Profiles and namespaces for real Projects after the new composition is healthy.

The k8s Site Agent remains the writer of contributor access. A Profile's owner is the agent's own technical identity (profile_owner), never a person: it is fixed for the life of the Profile, because a different owner means a different Profile and a Profile is the namespace Site Runtime runs in. Dashboard access is granted only through the contributor pairs, so changing an administrator changes a pair and never moves the namespace, its volumes or its Jobs.

• each contributor is identified by their current site_username;
• a Project manager receives the admin pair and every other member the edit pair, with the per-user Istio AuthorizationPolicy, when membership sync sees the member;
• the same objects are removed when the member is no longer current;
• member-<id> ServiceAccounts remain Site Runtime execution identities and are not Kubeflow UI identities;
• retained accounts from #765/#766 may keep an execution account for accepted work, but they do not retain the person's Kubeflow UI access.
• until the agent forwards Waldur's project role to the pair, every member gets edit and nobody gets admin (DD §5.2).

The names, annotations and subjects of those contributor objects are KFAM's, not the agent's own convention: the dashboard lists what KFAM reads, and KFAM deletes by the name it derives. DD §5 states the rule and requires the agent to reproduce it with upstream's test vectors, so a binding written by the agent and one written from the dashboard are the same object.

Bootstrap rights move from the temporary site-agent scope to a dedicated Kubeflow scope. The Site Agent keeps only the Profile and namespace-scoped writes it needs; it no longer owns the Profile CRD or the Kubeflow role set.

#Storage, databases and quotas

Pipelines keep the upstream SeaweedFS artifact store and MySQL metadata database, with persistent volumes on the platform's Ceph storage class. Katib keeps its upstream MySQL database, also on Ceph. Trainer is stateless. GPU workloads continue to use the GPU Operator's extended resource names.

A Profile has one ResourceQuota, created by the Profile controller from spec.resourceQuotaSpec. The Site Agent is the source of that specification; Kubeflow does not add a second quota layer. Notebooks, Trainer and Katib consume the same per-Profile CPU, memory and GPU limits. Nothing else writes a quota in a Profile namespace: the controller replaces the quota's specification with the Profile's whenever they differ, and the Site Runtime provisions no ResourceQuota or LimitRange (DD §6).

#Network and authorization policy

The old role's post-apply deletions are replaced by reviewed manifests. The composition keeps the upstream NetworkPolicies and Istio AuthorizationPolicies for Kubeflow services and Profile namespaces. It does not install the upstream empty global-deny-all AuthorizationPolicy in the mesh's root namespace: that policy would deny every workload in the mesh, including Pharos, LLM, Site API and other platform services. Kubeflow instead installs namespace-scoped default-deny policies in kubeflow, kubeflow-system, oauth2-proxy and Profile namespaces, with explicit service and route allowances. check_platform_mesh rejects a root-namespace deny-all unless the reviewed inventory proves an explicit ALLOW for every platform service. No policy is deleted after a sync.

#Parity, cutover, devbox and lab

A single read-only live snapshot, captured on 29 October 2026, is the comparison baseline. Each component owner renders its manifests and performs a server-side diff against that snapshot without another live read. The expected-difference list records only deliberate changes: GRNET OIDC, the omitted components above, Profile-namespace injection, image pins, storage class, route integration and resource requests.

The legacy 26.03.1 installation is removed as a whole, after the #800 PVC snapshot. The captured live state contains one Profile namespace, kubeflow-user-example-com; it is the upstream example namespace, not a k8s-site-agent Project namespace. No real Project Profiles are present, so the agent recreates Profiles and namespaces for real Projects after the new composition is healthy. kubeflow-core replaces the #742 design entry; #742 is not deployed on Icarus. Devbox enables kubeflow-core and kubeflow-notebooks by default and makes Pipelines, Trainer and Katib explicit profile selections. The icaruscloudlab environment enables all five for qualification.

Deleting a Profile deletes its namespace, so its persistent volumes and any account kept for accepted work go with it. The Site Agent deletes one only after Metallo's retained list shows no accepted work for the Project and every volume has a ready snapshot, and an admission policy refuses a Profile the agent manages without its marker and caller (DD §5.6). The legacy example Profile is not the agent's and stays outside that fence.

#Consequences

• Kubeflow is reviewable and repeatable, and the live 26.03.1 installation can be compared without another live read.
• Notebook traffic keeps Istio authentication while Site Runtime Jobs remain sidecar-free, and Istio CNI becomes a hard requirement rather than an option.
• kubeflow-userid is a Site-local name, so a changed site_username must be reconciled by the Site Agent; the Profile owner never changes with it.
• Pipelines and Katib keep upstream storage choices, which minimises parity differences but leaves two small MySQL deployments in the composition.
• Profile namespaces stay restricted, so a Notebook, Pipeline, Katib or Trainer image that cannot run as a non-root user cannot run on the platform.
• KServe, LWS and any KServe UI belong to the separate KServe component and are not installed by Kubeflow.

Page source

Source snapshot · Revision f928834c8.