# 0067 — Kubeflow is one pinned Argo CD composition Status: Proposed. Date: 29 October 2026. The live Icarus staging cluster runs Kubeflow community distribution 26.03.1, installed by the legacy `kubeflow-deploy` Ansible role and then corrected by hand: NetworkPolicies, Pod Security labels and the Istio global deny-all policy were deleted, Dex was replaced by GRNET OIDC through oauth2-proxy, and the ingress gateway was patched to NodePorts. That installation works, but it is not the composition the platform now uses, and the post-apply corrections are not reviewable or repeatable. ## Decision **Kubeflow becomes five Argo CD Applications built from the same pinned upstream 26.03.1 source, with all platform differences expressed as Kustomize patches.** The existing cert-manager, Istio control plane, GPU Operator and Ceph remain infrastructure and are not reinstalled by Kubeflow. | Application | Namespace(s) | Installs | Upstream pieces omitted | | -------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | `kubeflow-core` | `kubeflow`, `oauth2-proxy`, selected `istio-system` objects | Kubeflow roles, Kubeflow Istio resources, Profile controller and KFAM, central dashboard, oauth2-proxy | cert-manager, Istio control plane, Dex, Knative, Spark, Hub, Workspaces, KServe | | `kubeflow-notebooks` | `kubeflow` | Notebook controller, Jupyter web app, Volumes web app, PVC viewer controller | Tensorboard controller and web app | | `kubeflow-pipelines` | `kubeflow` | Kubeflow Pipelines, SeaweedFS artifact store, MySQL metadata database | Knative, KServe | | `kubeflow-trainer` | `kubeflow-system` | Trainer v2 controller and JobSet | LWS and data-cache; LWS belongs to the separate KServe component | | `kubeflow-katib` | `kubeflow` | Katib controller, UI, DB manager and MySQL | — | The composition installs `kubeflow-core` first, then `kubeflow-notebooks`, and only then Pipelines, Trainer and Katib. A closed-composition check accepts exactly this child set; an extra or missing Kubeflow Application fails delivery. ## Istio in Profile namespaces Profile namespaces do **not** get the upstream `istio-injection: enabled` label. The Profile controller's namespace-label ConfigMap is patched to keep injection off, so Site Runtime Jobs and other execution workloads are never modified by Istio. Notebooks still get a sidecar, but only Notebooks do. A `PodDefault` named `istio-sidecar` is created in every Profile namespace and adds the pod label `sidecar.istio.io/inject: "true"`. The injector's object rule in an unlabelled namespace selects that label, so the Notebook controller's generated StatefulSet is injected and Site Runtime Jobs are not. The Jupyter web app selects the PodDefault by default. This gives the dashboard and Notebook traffic the same authentication path as upstream without changing Job templates or the Site Runtime admission policy. Native sidecars are supported by the installed Istio 1.30 and Kubernetes 1.35 server, but they are not selected for Profile namespaces: the Site Runtime admission policy would still reject the volumes and init-container profile Istio adds. Namespace-wide injection is rejected for the same reason and because it keeps Jobs from finishing. The `kubeflow` and `kubeflow-system` namespaces keep the upstream injection labels for their own control-plane services. The Profile controller only ever adds a namespace label, or deletes one that its ConfigMap names with an empty value; it never overwrites or removes a label it does not name. Site Runtime's labels therefore survive its reconcile, and the ConfigMap may not name any of them (DD §3). ## Pod security in Profile namespaces Profile namespaces keep `pod-security.kubernetes.io/enforce: restricted`; the Site Agent writes it and the guard pins it. Istio therefore runs with its CNI plugin, so injection adds a non-root `istio-validation` container instead of the privileged `istio-init` that needs `NET_ADMIN`. Every Kubernetes pod template the composition ships declares the restricted profile itself. Pod Security admission refuses an incomplete pod before any `ValidatingAdmissionPolicy` sees it; the platform's policy exists for a namespace that loses the `restricted` label, and states PSA's restricted profile exactly, including a container's own `runAsNonRoot` and `seccompProfile` taking priority over the pod's. The platform never mutates a workload's `securityContext` and never sets `runAsUser`, so an image keeps the user it chooses; admission kinds stay on `admissionregistration.k8s.io/v1` because a beta `MutatingAdmissionPolicy` is served by no API server the platform installs and would break its bootstrap. The Site Runtime Job policy is narrowed to its own callers so Katib and Trainer can create Jobs in the same namespace. No workload depends on a hand-deleted label, and only non-root images run there (DD §3). ## Identity **`kubeflow-userid` is the member's `site_username` at the Site** (decided 29 October 2026): the Waldur offering username, which Metallo already resolves to the Party through the Party's `site` binding. No email, AAI subject or Party ID reaches the Site as an identity, and the k8s Site Agent needs no personal data it does not already hold. GRNET returns `site_username` in the userinfo response only, never in the token. oauth2-proxy therefore takes it from the userinfo endpoint at login and keeps it in the session, writes it as `kubeflow-userid`, and the gateway copies that header upstream through the platform's `oauth2-proxy` ext_authz provider. Only that login sets the header: a value sent by a client is overwritten, and requests that skip the check are denied on the Kubeflow host (DD §4). The oauth2-proxy runs in the `oauth2-proxy` namespace and exposes the Service `oauth2-proxy` on port 4180. The staging callback is `https://kubeflow.staging.pharosfactory.ai/oauth2/callback`; the devbox callback is `https://kubeflow.localtest.me/oauth2/callback`. The route, edge certificate and DNS entry are owned by the #800 edge work, not by Kubeflow. The client secret follows the platform's private-file pattern: a mode-0600 input on the administration machine, delivered as a Kubernetes Secret through standard input, never through Git or Argo values. A member without a `site_username` has no Site account, so they have no Profile and no Kubeflow access. The value is a Site name, not a global identity: a changed username means the agent removes the old KFAM pair and writes the new one. Accepted work keeps its execution account (#765) but never the person's UI access. ## Profiles and the k8s Site Agent `kubeflow-core` replaces the #742 design entry and keeps its two security corrections: no cluster-admin binding, and no Istio injection label on Profile namespaces. #742 is not deployed on Icarus. The complete legacy 26.03.1 installation is removed at cutover, and the k8s Site Agent recreates Profiles and namespaces for real Projects after the new composition is healthy. The k8s Site Agent remains the writer of contributor access. A Profile's owner is the agent's own technical identity (`profile_owner`), never a person: it is fixed for the life of the Profile, because a different owner means a different Profile and a Profile is the namespace Site Runtime runs in. Dashboard access is granted only through the contributor pairs, so changing an administrator changes a pair and never moves the namespace, its volumes or its Jobs. - each contributor is identified by their current `site_username`; - a Project manager receives the `admin` pair and every other member the `edit` pair, with the per-user Istio AuthorizationPolicy, when membership sync sees the member; - the same objects are removed when the member is no longer current; - `member-` ServiceAccounts remain Site Runtime execution identities and are not Kubeflow UI identities; - retained accounts from #765/#766 may keep an execution account for accepted work, but they do not retain the person's Kubeflow UI access. - until the agent forwards Waldur's project role to the pair, every member gets `edit` and nobody gets `admin` (DD §5.2). The names, annotations and subjects of those contributor objects are KFAM's, not the agent's own convention: the dashboard lists what KFAM reads, and KFAM deletes by the name it derives. DD §5 states the rule and requires the agent to reproduce it with upstream's test vectors, so a binding written by the agent and one written from the dashboard are the same object. Bootstrap rights move from the temporary `site-agent` scope to a dedicated Kubeflow scope. The Site Agent keeps only the Profile and namespace-scoped writes it needs; it no longer owns the Profile CRD or the Kubeflow role set. ## Storage, databases and quotas Pipelines keep the upstream SeaweedFS artifact store and MySQL metadata database, with persistent volumes on the platform's Ceph storage class. Katib keeps its upstream MySQL database, also on Ceph. Trainer is stateless. GPU workloads continue to use the GPU Operator's extended resource names. A Profile has one ResourceQuota, created by the Profile controller from `spec.resourceQuotaSpec`. The Site Agent is the source of that specification; Kubeflow does not add a second quota layer. Notebooks, Trainer and Katib consume the same per-Profile CPU, memory and GPU limits. Nothing else writes a quota in a Profile namespace: the controller replaces the quota's specification with the Profile's whenever they differ, and the Site Runtime provisions no `ResourceQuota` or `LimitRange` (DD §6). ## Network and authorization policy The old role's post-apply deletions are replaced by reviewed manifests. The composition keeps the upstream NetworkPolicies and Istio AuthorizationPolicies for Kubeflow services and Profile namespaces. It does **not** install the upstream empty `global-deny-all` AuthorizationPolicy in the mesh's root namespace: that policy would deny every workload in the mesh, including Pharos, LLM, Site API and other platform services. Kubeflow instead installs namespace-scoped default-deny policies in `kubeflow`, `kubeflow-system`, `oauth2-proxy` and Profile namespaces, with explicit service and route allowances. `check_platform_mesh` rejects a root-namespace deny-all unless the reviewed inventory proves an explicit ALLOW for every platform service. No policy is deleted after a sync. ## Parity, cutover, devbox and lab A single read-only live snapshot, captured on 29 October 2026, is the comparison baseline. Each component owner renders its manifests and performs a server-side diff against that snapshot without another live read. The expected-difference list records only deliberate changes: GRNET OIDC, the omitted components above, Profile-namespace injection, image pins, storage class, route integration and resource requests. The legacy 26.03.1 installation is removed as a whole, after the #800 PVC snapshot. The captured live state contains one Profile namespace, `kubeflow-user-example-com`; it is the upstream example namespace, not a k8s-site-agent Project namespace. No real Project Profiles are present, so the agent recreates Profiles and namespaces for real Projects after the new composition is healthy. `kubeflow-core` replaces the #742 design entry; #742 is not deployed on Icarus. Devbox enables `kubeflow-core` and `kubeflow-notebooks` by default and makes Pipelines, Trainer and Katib explicit profile selections. The `icaruscloudlab` environment enables all five for qualification. Deleting a Profile deletes its namespace, so its persistent volumes and any account kept for accepted work go with it. The Site Agent deletes one only after Metallo's retained list shows no accepted work for the Project and every volume has a ready snapshot, and an admission policy refuses a Profile the agent manages without its marker and caller (DD §5.6). The legacy example Profile is not the agent's and stays outside that fence. ## Consequences - Kubeflow is reviewable and repeatable, and the live 26.03.1 installation can be compared without another live read. - Notebook traffic keeps Istio authentication while Site Runtime Jobs remain sidecar-free, and Istio CNI becomes a hard requirement rather than an option. - `kubeflow-userid` is a Site-local name, so a changed `site_username` must be reconciled by the Site Agent; the Profile owner never changes with it. - Pipelines and Katib keep upstream storage choices, which minimises parity differences but leaves two small MySQL deployments in the composition. - Profile namespaces stay `restricted`, so a Notebook, Pipeline, Katib or Trainer image that cannot run as a non-root user cannot run on the platform. - KServe, LWS and any KServe UI belong to the separate KServe component and are not installed by Kubeflow.