> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # DD-0025: Scoped Istio Authorization Policies — Implementation **Related ADR:** [ADR-0025: Scoped Istio Authorization Policies](/adr/0025-scoped-istio-authorization-policies.md) **Status:** Draft **Date:** 2026-07-20 **Authors:** TBD ## ADR-derived technical notes ADR-0025 selects scoped ingress host allowlists and scoped Istio sidecar injection. The first internal policies authorize Kubernetes ServiceAccount identities and workload destination ports only. Envoy sidecars can enforce L7 attributes, but HTTP methods, paths, headers, request principals, `CUSTOM`, and `AUDIT` conditions remain outside the initial backend policies. Repository inspection found these implementation constraints: - `istio_version` is the common pin for Helm charts, Kubeflow Istio images, and manually rendered gateway proxy images. - `istio-deploy` currently installs `istio-base`, `istiod`, and a gateway with Helm, but it only patches an Istio CNI DaemonSet if another deployment already created one. Kubeflow's rendered kustomization still includes an overlapping Istio installation overlay. - The Themelio web and Argo CD gateways are GitOps-owned. The GitLab, Waldur, image registry, MLflow, monitoring, Ceph dashboard, Kubeflow, and Themelio workload gateways are Ansible-owned. - The ingress `ALLOW` policies for web, Argo CD, GitLab, Waldur, the image registry, MLflow, Grafana, Prometheus, Ceph, and the workload gateway contain an empty rule and therefore match every request. - `web` and `gitlab-root-actor` currently run as the `default` ServiceAccount in `themelio`. `k8s-root-actor` already runs as `k8s-root-actor-sa` in every deployed overlay. - The Waldur order, membership-sync, report, and component-loading workloads all use `waldur-site-agent-cloud`. Trusting that identity would trust more than the two GitLab-sync processes named in ADR-0025. - The seeder still calls `gitlab-root-actor`. It disables ServiceAccount token automount but does not currently select a dedicated ServiceAccount. - Local and staging have been reachable during implementation. Earlier staging evidence was collected from the superseded Ambient pilot and is not accepted as sidecar evidence; every mesh-specific gate must be rerun after migration. Dev and production runtime evidence remain promotion criteria. The related ADR-0024 (historical reference unavailable) currently lives on a separate documentation branch. It must be merged before the published ADR-0025 and DD-0025 links are expected to resolve. Its NetworkPolicy controls are a rollout prerequisite, not duplicated ownership in this DD. ## Overview This document specifies how Themelio will consolidate Istio component ownership under Ansible and Helm, restrict every deployed ingress gateway to its configured HTTP host, give gateways and selected applications unambiguous workload identities, and incrementally protect the ADR-0025 internal flows with sidecar mTLS and workload authorization. Local and dev remain open by default; staging and production use enforced security after their canary gates pass. No application API, database schema, or user-facing action contract changes. ## Goals and Non-Goals ### Goals - Make Helm, driven by `istio-deploy`, the single owner of Istio base, istiod, CNI, and the shared Kubeflow ingress gateway. - Remove every empty gateway `ALLOW` rule and derive allowed hosts from the same value used by the gateway and its VirtualService. - Give every ingress gateway a unique ServiceAccount without changing its external listener, route, or authentication behavior. - Protect the selected web, root-actor, site-agent, gateway, and seeder flows with mTLS-authenticated ServiceAccount identity and destination workload port. - Inject individual pod templates rather than whole application namespaces. - Scope each tenant (Kubeflow profile) namespace so a Themelio deployment or inference service is reachable only through the service ingress gateway identity, with no namespace-wide `ALLOW` rule. - Preserve external authorization, application authentication, user RBAC, NetworkPolicy, and service-specific credentials as independent controls. - Produce repeatable validation and rollback procedures for every security profile. ### Non-Goals - Namespace-wide sidecar injection or automatic injection into tenant workloads. - A root-namespace deny-all AuthorizationPolicy. - HTTP method, path, header, JWT, or new traffic-routing policy at application sidecars. - Mesh enrollment of PostgreSQL, Valkey, GitLab, or arbitrary user deployments. - Replacing Calico, enabling its eBPF dataplane, or removing kube-proxy. - Changing REST, GraphQL, action, database, or user authorization contracts. - Claiming the ADR is accepted before the canary and promotion evidence exists. ## Design ### 1. Security profiles and ownership boundaries `security_profile` controls only ADR-0025 sidecar injection and native `ALLOW`/strict-mTLS enforcement: | Profile | Sidecar injection | STRICT mTLS / ALLOW policies | | ------------ | ----------------- | ---------------------------- | | `open` | No | No | | `permissive` | Yes | No | | `enforced` | Yes | Yes | The environment assignments are explicit: | Environment | `security_profile` | | ----------- | ---------------------------------- | | Local | `open` | | Dev | `open` | | Staging | `enforced` after canary validation | | Production | `enforced` after canary validation | `permissive` is a transitional rollout and rollback profile, not a permanent environment class. Dedicated ServiceAccounts, Istio CNI, sidecar-capable istiod, Gateway and VirtualService routing, application authentication, user RBAC, service credentials, and existing `CUSTOM` external authorization remain present in all profiles. Ansible reads `security_profile` from each environment's `inventory//group_vars/all.yml`. GitOps cannot consume Ansible inventory directly, so matching Kustomize components implement the same state: - no security component for `open`, - `security-enrollment` only for `permissive`, and - both `security-enrollment` and `security-enforcement` for `enforced`. Static validation must compare the inventory value with the rendered GitOps resources and fail if they disagree. Local/dev overlays omit both components; staging includes both. Any future production Themelio or Argo CD overlay must include both before it is registered by the production root application. Other environment differences remain data such as `dns.*`, node selectors, tolerations, replica counts, and secret values. Resource ownership after this change is: | Resource family | Owner | | ---------------------------------------------------- | -------------------------------------------------------- | | Istio CRDs and cluster RBAC | `istio-base` Helm release through `istio-deploy` | | Control plane and mesh config | `istiod` Helm release plus `istio-deploy` mesh ConfigMap | | Sidecar traffic capture | `istio-cni` Helm release through `istio-deploy` | | Shared Kubeflow ingress gateway | Helm gateway release through `istio-deploy` | | Web and Argo CD gateway workloads and policies | their GitOps/Kustomize applications | | Other standalone gateway workloads and policies | the Ansible role that owns the exposed service | | Themelio application identities and backend policies | Themelio GitOps/Kustomize | | Waldur process identities | Waldur site-agent GitOps/Kustomize | | NetworkPolicy and network zones | ADR-0024 platform automation | | Metrics, dashboards, and alerts | monitoring Ansible roles | Tenant roles must not be able to edit these ServiceAccounts, gateway or backend AuthorizationPolicies, PeerAuthentications, enrollment labels, or platform-owned NetworkPolicies. ### 2. Sidecar-capable Istio installation `istio-deploy` will expose and validate these variables: | Variable | Default | Purpose | | ----------------------------------- | ---------------------- | ------------------------------------------------ | | `istio_cni_release_name` | `istio-cni` | Helm ownership identity for the CNI node agent | | `istio_shared_gateway_release_name` | `istio-ingressgateway` | Helm ownership identity for the Kubeflow gateway | | `istio_cni_resources` | role defaults | CNI resource requests and limits | | `istio_wait_timeout` | existing value | Atomic Helm and rollout timeout | Every chart uses `istio_chart_version`, which defaults to `istio_version`. Standalone gateways continue using `docker.io/istio/proxyv2:{{ istio_version }}`. A render or deployment must fail when those versions differ. The Helm sequence is: 1. Upgrade or install `istio-base`. 2. Upgrade or install `istiod` with the chart's default sidecar values and the existing resource, placement, mesh config, trust domain, and extension-provider settings. 3. Upgrade or install `istio-cni` with Ambient disabled on every schedulable Kubernetes node. 4. Reconcile the shared Kubeflow gateway with its existing service ports, NodePorts, selectors, and dedicated ServiceAccount. 5. Reconcile standalone gateway proxy images only after the control plane is available. The Helm operations remain atomic. The role waits for the `istiod` Deployment and CNI DaemonSet to report all desired pods ready. CNI installation does not inject a proxy by itself; injection is selected at each pod template. Kubespray sets `fs.inotify.max_user_instances=1024` through `additional_sysctl`. The kernel default of `128` was exhausted on a sidecar-heavy staging node and prevented the Istio CNI agent from creating its CNI-configuration watcher during an upgrade. #### 2.1 Kubeflow ownership migration The Kubeflow kustomization stops including `../common/istio/istio-install/overlays/oauth2-proxy`. It also drops the image rewrites for pilot, proxyv2, and install-cni and the istiod Deployment/HPA replica patches. It retains: - `../common/istio/cluster-local-gateway/base`, - `../common/istio/kubeflow-istio-resources/base`, - Kubeflow Gateway and VirtualService resources, - the custom OAuth2 proxy resources and host-scoped `CUSTOM` policy, and - the existing Ansible-managed Istio `extensionProviders` mesh configuration. Migration order matters: 1. Render the future Kubeflow kustomization and verify it no longer declares Istio base, istiod, the main ingress gateway, or the CNI DaemonSet. 2. Let Helm adopt matching live resources with `--take-ownership` while the old Kubeflow apply path is still available for rollback. 3. Verify Helm owns all four releases and the current Kubeflow gateway remains ready and routable. 4. Apply the reduced Kubeflow kustomization. Server-side apply must not regain ownership of fields managed by Helm. 5. Remove the old `istio-ingress` Helm gateway only after queries show that no Istio Gateway selector, Kubernetes Service, or traffic path uses it. The operator must stop if Helm adoption would replace a Service ClusterIP, NodePort, selector, or ServiceAccount rather than preserving it. #### 2.2 Scoped sidecar injection Existing Envoy gateways remain standalone router proxies. Selected application Deployments and Jobs receive `sidecar.istio.io/inject: "true"` in `spec.template.metadata.labels`. No Themelio or Waldur namespace receives `istio-injection=enabled`, so unrelated pods remain outside the mesh. Istio CNI performs traffic redirection without a privileged init container, while the injected Envoy remains visible as `istio-proxy` in pod status and telemetry. istiod is installed with `pilot.env.ENABLE_NATIVE_SIDECARS=true` (the `istio_enable_native_sidecars` default in the `istio-deploy` role), so the Envoy proxy is injected as a Kubernetes native sidecar - an init container with `restartPolicy: Always` — supported by the cluster's Kubernetes 1.34. The proxy therefore starts before a pod's app init containers, so init-time egress is captured by a running Envoy instead of failing against traffic redirection that is active before a classic sidecar would start. This is what makes injecting the KServe predictor (section 9), whose `workspace-init` container clones the model over git/LFS, safe. ### 3. Ingress host authorization An ingress host policy is an L7 policy enforced by that gateway's Envoy proxy. It uses the HTTP `Host` or HTTP/2 `:authority` value and does not validate DNS resolution. Every deployed gateway receives one scoped `ALLOW` policy: ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: namespace: istio-system spec: selector: matchLabels: app: action: ALLOW rules: - to: - operation: hosts: - "" - ":*" ``` The second host form is required because clients and reverse proxies may include an explicit port in `Host`. No wildcard domain is introduced unless the Gateway and VirtualService already use that same wildcard. | Gateway | Owner | Host source | Target ServiceAccount | | ------------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | `istio-ingressgateway-web` | Themelio GitOps | web overlay Gateway/VirtualService host | `istio-ingressgateway-web-service-account` | | `istio-ingressgateway-argocd` | Argo CD GitOps | Argo CD overlay Gateway/VirtualService host | `istio-ingressgateway-argocd-service-account` | | `istio-ingressgateway-gitlab` | `deploy_gitlab` | `dns.gitlab` | `istio-ingressgateway-gitlab-service-account` | | `istio-ingressgateway-waldur` | `waldur-deploy` | `dns.waldur` | `istio-ingressgateway-waldur-service-account` | | `istio-ingressgateway-image-cache` | `image-registry` | `dns.registry` | `istio-ingressgateway-image-cache-service-account` | | `istio-ingressgateway-mlflow` | `mlflow-router` | `dns.mlflow` | `istio-ingressgateway-mlflow-service-account` | | `istio-ingressgateway-mlflow-router` | `mlflow-router` | `dns.mlflow` | `istio-ingressgateway-mlflow-router-service-account` | | `istio-ingressgateway-grafana` | monitoring role | `dns.grafana` | `istio-ingressgateway-grafana-service-account` | | `istio-ingressgateway-prometheus` | monitoring role | `dns.prometheus` | `istio-ingressgateway-prometheus-service-account` | | `istio-ingressgateway-ceph-dashboard` | storage role | `dns.ceph_dashboard` | `istio-ingressgateway-ceph-dashboard-service-account` | | `istio-ingressgateway-service` | Themelio Ansible role | `dns.service`; the matching GitOps overlay sets `SERVICE_PUBLIC_HOST` directly on `k8s-root-actor` | `istio-ingressgateway-themelio-service-account` | | `istio-ingressgateway` | `istio-deploy`/Kubeflow | `dns.kubeflow` | `istio-ingressgateway-service-account` | `istio-ingressgateway-service-account` is deliberately retained for Kubeflow. The Waldur Kubeflow client already writes that principal into per-user policies. The Themelio workload gateway must move away from that shared identity. Existing `CUSTOM` policies for Kubeflow and MLflow remain present. Istio evaluates `CUSTOM` before native `ALLOW`; therefore a request must pass external authorization when its `CUSTOM` rule applies and must also match the gateway host allowlist in the `enforced` profile. An unknown host then matches no `ALLOW` rule and returns `403`. In `open` and `permissive`, the ADR-0025 gateway `ALLOW` policy is absent; the existing `CUSTOM` and application controls still apply. Host rules are generated from the same input as the matching Gateway and VirtualService: - Kustomize overlays patch the Gateway, VirtualService, and AuthorizationPolicy with one environment host value. - Ansible templates reference the existing `dns.*` value in all three resources. - The Themelio service gateway spans ownership boundaries: Ansible renders the gateway host from `dns.service`, while each GitOps overlay sets the same environment-specific hostname directly as `SERVICE_PUBLIC_HOST` on `k8s-root-actor`. No routing ConfigMap is used. Render validation must compare these two values and fail on any mismatch before promotion. - A role does not render the policy when its gateway is disabled or its host is undefined; it fails validation if the gateway is enabled without a host. - Ansible reconciles the native host `ALLOW` policy to `absent` unless `security_profile` is `enforced`; merely skipping a task is prohibited because it could leave a stale policy active. Gateway readiness port `15021`, Envoy metrics, and NodePort behavior must be tested after each policy. They are not added as public host exceptions. ### 4. Workload identities The target identities and SPIFFE principals are: | Workload | ServiceAccount | Principal | | ----------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------- | | web | `web` | `cluster.local/ns/themelio/sa/web` | | k8s-root-actor | `k8s-root-actor-sa` | `cluster.local/ns/themelio/sa/k8s-root-actor-sa` | | gitlab-root-actor | `gitlab-root-actor` | `cluster.local/ns/themelio/sa/gitlab-root-actor` | | web gateway | `istio-ingressgateway-web-service-account` | `cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account` | | Waldur GitLab sync | `waldur-site-agent-cloud-gitlab-sync` | `cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync` | | Waldur report and component loading | `waldur-site-agent-cloud` | `cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud` | | seeder | `themelio-seeder` | `cluster.local/ns/themelio/sa/themelio-seeder` | ServiceAccounts are declared in the same desired-state package as their workloads. A Deployment changes identity by setting `serviceAccountName`; this causes a normal rollout. `automountServiceAccountToken: false` remains on the seeder because application access to the Kubernetes API is unnecessary. Workload identity comes from the certificate issued to its injected sidecar and does not require exposing a Kubernetes token inside the seeder container. The Waldur order and membership-sync Deployments move to `waldur-site-agent-cloud-gitlab-sync`. Its ClusterRoleBinding contains this new subject so existing Kubernetes reconciliation behavior continues. Report and component-loading retain `waldur-site-agent-cloud` and are not included in the GitLab root actor allowlist. This split is mandatory before trusting the sync principal. ### 5. Sidecar injection and backend policy In `permissive` and `enforced`, the enrollment component places `sidecar.istio.io/inject: "true"` only on these pod templates: - `web` - `k8s-root-actor` - `gitlab-root-actor` - Waldur order process - Waldur membership-sync process - `themelio-seeder` The web ingress gateway is already an Istio proxy and does not receive an application sidecar. In `open`, none of these pod templates requests injection. In `permissive`, they receive sidecars but the backend AuthorizationPolicies and workload-scoped strict PeerAuthentications are absent. In `enforced`, the same injection is combined with the policies below. Backend `AuthorizationPolicy` resources live beside the destination workload. The `ports` values are destination workload ports, not Kubernetes Service ports. The three Themelio applications listen on workload port `3000` even though their Services expose port `80`. #### 5.1 k8s-root-actor canary policy ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: k8s-root-actor-allow-web namespace: themelio spec: selector: matchLabels: app: k8s-root-actor action: ALLOW rules: - from: - source: principals: - cluster.local/ns/themelio/sa/web to: - operation: ports: ["3000"] --- apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: k8s-root-actor-strict-mtls namespace: themelio spec: selector: matchLabels: app: k8s-root-actor mtls: mode: STRICT ``` #### 5.2 web policy ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: web-allow-ingress-gateway namespace: themelio spec: selector: matchLabels: app: web action: ALLOW rules: - from: - source: principals: - cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account to: - operation: ports: ["3000"] --- apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: web-strict-mtls namespace: themelio spec: selector: matchLabels: app: web mtls: mode: STRICT ``` #### 5.3 gitlab-root-actor policy ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: gitlab-root-actor-allow-callers namespace: themelio spec: selector: matchLabels: app: gitlab-root-actor action: ALLOW rules: - from: - source: principals: - cluster.local/ns/themelio/sa/web - cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync - cluster.local/ns/themelio/sa/themelio-seeder to: - operation: ports: ["3000"] --- apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: gitlab-root-actor-strict-mtls namespace: themelio spec: selector: matchLabels: app: gitlab-root-actor mtls: mode: STRICT ``` The policies intentionally contain no L7 fields. The destination Envoy sidecar is the enforcement point and authenticates the original caller's sidecar certificate. Adding HTTP methods, paths, or headers is technically possible in sidecar mode but requires a new or revised DD defining stable L7 contracts. Application authorization is unchanged. For example, a pod with an allowed workload identity still needs the configured root-actor bearer token where the endpoint requires one. Workload identity does not grant a human permission. ### 6. NetworkPolicy contract with ADR-0024 Before any pod receives a sidecar, ADR-0024 policy must preserve: - existing application traffic at Kubernetes Service and workload ports, - DNS and control-plane traffic needed by the workload and Istio proxy, - kubelet health probes to the workload, including Istio probe rewriting, and - Prometheus access to the Envoy metrics port where sidecar scraping is enabled. No HBONE port `15008` exception is required for this design. Sidecar-to-sidecar mTLS uses the application's normal destination port. The pilot runs on the current standard Calico dataplane. If a later ADR-0024 rollout enables Calico eBPF, its connect-time load balancing must remain disabled before sidecars are promoted. The combined conformance suite must cover ClusterIP, NodePort, DNS, host-to-Service, pod-to-Service, Gateway/VirtualService routing, failure recovery, and rollback. ### 7. End-to-end request paths ```mermaid flowchart LR Client[Browser or API client] -->|TLS| Nginx[Bastion Nginx] Nginx -->|HTTP + preserved Host| GW[Dedicated Envoy gateway] GW -->|host ALLOW + mTLS| WebProxy[web Envoy sidecar] WebProxy --> Web[web :3000] Web --> WebProxy WebProxy -->|mTLS as themelio/web| KProxy[k8s-root-actor Envoy] KProxy -->|principal + port ALLOW| KRA[k8s-root-actor :3000] WebProxy -->|mTLS as themelio/web| GProxy[gitlab-root-actor Envoy] WaldurSync[Waldur sync + sidecar] -->|mTLS| GProxy Seeder[seeder + sidecar] -->|mTLS| GProxy GProxy -->|principal + port ALLOW| GRA[gitlab-root-actor :3000] ``` The bastion-to-gateway hop remains plain HTTP on the private network as recorded in ADR-0025. The gateway-to-backend and selected internal flows use Envoy sidecar mTLS. Calls from injected workloads to destinations outside the mesh, such as GitLab, PostgreSQL, or Valkey, retain their existing transport and application security behavior. ### 8. Observability Sidecar telemetry is collected separately from application telemetry: - Prometheus scrapes istiod, Envoy sidecars, and Envoy gateway metrics explicitly. - Mesh configuration selects the built-in Prometheus metrics provider through `defaultProviders.metrics`. - Application metrics continue to use their existing scrape configuration. Application/sidecar metric merging is not assumed. - The monitoring role adds an `istio-workload-sidecars` `PodMonitor` selected by `security.istio.io/tlsMode=istio`. - Existing Alloy pod-log collection forwards gateway and `istio-proxy` logs to Loki. - Grafana panels show sidecar scrape readiness, mTLS request/connection counts, TCP bytes, source and destination principals, and authorization denials where the exported labels support them. Alerts and runbooks distinguish: | Symptom | Primary evidence | | ------------------------------------- | --------------------------------------------------------------- | | Unknown or missing HTTP host | gateway Envoy access log with `403`; gateway RBAC metrics | | External authorization rejection | gateway/ext-authz log and response details | | Plaintext or certificate failure | destination Envoy log; PeerAuthentication state | | Unauthorized workload identity | destination Envoy RBAC log and metric with source principal | | NetworkPolicy denial | ADR-0024 Calico flow verdict; connection timeout | | Application authentication or failure | application response and application logs after mesh validation | The rollout checklist records links or captured output for each of these signals. ### 9. Tenant-namespace workload authorization Kubeflow profile namespaces host tenant workloads. The Themelio deployments and KServe inference services that `k8s-root-actor` creates must be reachable from the shared Themelio service ingress gateway, but the namespace must not carry a namespace-wide `ALLOW` rule (`rules: [{}]`) that would admit every source. Instead `k8s-root-actor` generates one workload-scoped `AuthorizationPolicy` per deployment and per inference service, and deletes it with the workload. The following Kubeflow-generated policies are retained unchanged and are not replaced by this design: - `ns-owner-access-istio` — the namespace owner through the Kubeflow gateway, `ml-pipeline-ui` traffic carrying the owner identity, same-namespace workload traffic, health and metrics endpoints, and notebook-controller access to notebook kernels. - every `user--clusterrole-edit` policy — each contributor through the Kubeflow gateway and `ml-pipeline-ui` only when the `kubeflow-userid` header matches that contributor. Generated with every Themelio `k8s-deployment` (`k8s-root-actor` template `k8s-deployment/authorizationpolicy.yaml`): ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: -service-gateway namespace: spec: selector: matchLabels: app: component: themelio-k8s-deployment-runner action: ALLOW rules: - from: - source: principals: - cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account to: - operation: ports: - "" ``` Generated with every `k8s-inferenceservice` (`k8s-root-actor` template `k8s-inferenceservice/authorizationpolicy.yaml`). The selector targets the KServe-generated predictor pod labels: ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: -service-gateway namespace: spec: selector: matchLabels: serving.kserve.io/inferenceservice: component: predictor action: ALLOW rules: - from: - source: principals: - cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account to: - operation: ports: - "" ``` - `` is the workload's configured port, defaulting to `8000` — the same default the Deployment container port, the predictor `--http_port`, and the Service `targetPort` already use. - The admitted principal is the Themelio service gateway ServiceAccount, read from `k8s-root-actor`'s `SERVICE_GATEWAY_PRINCIPAL` config (default `cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account`). - Each policy is applied in the same concurrent tier as the workload it selects and is deleted by the matching teardown path (`destroyK8sDeployment`, `destroyK8sInferenceService`), so it never outlives its workload. Enforcement prerequisite: Istio enforces an `AuthorizationPolicy` at the selected pod's sidecar. The inference predictor template therefore sets `sidecar.istio.io/inject: "true"` (with `holdApplicationUntilProxyStarts` on the model-server container), so the `-service-gateway` policy is enforced. The deployment-runner pod carries no explicit injection annotation, so its policy is enforced only where the tenant namespace already injects the pod; where it does not, the policy is inert (fail-open) but harmless until the runner joins the mesh. Injecting the predictor depends on the mesh running native sidecars: the predictor's `workspace-init` container performs `apt`/`curl`/`git`+LFS egress during the init phase, which a classic (non-native) sidecar would break because traffic redirection is active during init while Envoy is not yet running (`holdApplicationUntilProxyStarts` does not cover init containers). istiod is installed with `ENABLE_NATIVE_SIDECARS=true` (section 2.2), so the proxy starts before the init container and this egress succeeds. No tenant policy is generated for two workload classes: - `slurm-job` — its launcher and cancellation Jobs run in the fixed `themelio` namespace, not a tenant namespace. - the Waldur site agent — it mutates tenant resources through the Kubernetes API, so its access is governed by Kubernetes RBAC, not Istio. The result preserves every currently implemented tenant workflow — Kubeflow owner and contributor access, notebooks, pipelines, health checks, tenant-internal communication, Themelio deployment and inference reachability through the dedicated gateway, and Waldur resource management — while removing namespace-wide unrestricted access. No tenant namespace retains a `rules: [{}]` policy, and requests from unapproved external identities are denied. ## Data Model No persistent application data changes. The security configuration itself is a versioned contract: | Contract | Source of truth | Compatibility rule | | ----------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | Security profile | Ansible `security_profile` plus matching GitOps component selection | `open`: neither component; `permissive`: enrollment only; `enforced`: enrollment and enforcement | | Istio component version | `istio_version` | all charts and proxy images match | | Trust domain | mesh config `cluster.local` | principal strings change atomically if the trust domain changes | | Gateway allowed host | Gateway/VirtualService environment input | AuthorizationPolicy consumes the same input | | Workload identity | Kubernetes ServiceAccount name and namespace | policy update precedes any intentional rename | | Sidecar injection | pod-template label | namespace labels are prohibited for this rollout | | Backend listener | Themelio workload port `3000` | policy uses workload port, not Service port `80` | | Tenant workload scoping | `k8s-root-actor` `SERVICE_GATEWAY_PRINCIPAL` and the `-service-gateway` policy | policy is created and deleted with its deployment/inference workload; principal must match the service gateway ServiceAccount | ServiceAccount names and their SPIFFE principals are security-sensitive public deployment interfaces. Renaming one without updating and revalidating every policy that trusts it is a breaking change. ## API Design No Themelio REST, GraphQL, database, or action API changes. The Kubernetes API surface added or changed by this design consists of: - `ServiceAccount` resources and Deployment/Job `serviceAccountName` fields, - pod-template `sidecar.istio.io/inject: "true"` labels, - scoped `security.istio.io/v1` `AuthorizationPolicy` resources, - workload-scoped `security.istio.io/v1` `PeerAuthentication` resources, and - Helm releases and values for Istio base, istiod, CNI, and the shared gateway, - Kustomize `security-enrollment` and `security-enforcement` components selected by the environment overlay to match `security_profile`. All backend AuthorizationPolicies use workload selectors and are enforced by destination sidecars. Existing `RequestAuthentication` and `CUSTOM` AuthorizationPolicy contracts remain unchanged. ## Testing Strategy ### Static and rendered-manifest tests Run before connecting to a cluster: 1. Build every Themelio, Argo CD, and Waldur site-agent Kustomize overlay. 2. Render the affected Ansible templates for `open`, `permissive`, and `enforced`. 3. Run repository YAML checks, Ansible syntax checks, and Ansible lint. 4. Run `helm template` for base, default sidecar istiod, sidecar-only CNI, and the shared gateway at the pinned `istio_version`. 5. Run offline `istioctl analyze` over the composed Istio and application manifests. 6. Assert that no gateway `ALLOW` policy contains `rules: [{}]`, `rules: - {}`, or an omitted host operation. 7. Assert that no gateway Deployment uses `default`, `istiod`, `istio-ingress`, or another gateway's ServiceAccount. 8. Assert that local and dev render no sidecar injection labels, native ADR-0025 `ALLOW` policies, or strict PeerAuthentications. 9. Assert that `permissive` renders selected sidecar injection labels but no native ADR-0025 `ALLOW` policies or strict PeerAuthentications. 10. Assert that `enforced` renders enrollment plus all expected policies. 11. Assert that all backend AuthorizationPolicies contain only `principals` and `ports`; Themelio backends use `3000`. 12. Assert that only the selected application pod templates carry the explicit sidecar injection label. 13. Assert that the Waldur report and load-components workloads do not use the GitLab-sync ServiceAccount. 14. Assert that the rendered `k8s-root-actor` tenant `AuthorizationPolicy` templates select the workload (deployment-runner or KServe predictor) labels, admit only the service gateway principal on the workload port, and contain no `rules: [{}]` or namespace-wide rule. ### Component and ownership validation ```bash helm list -n istio-system kubectl -n istio-system get deploy,daemonset,service,serviceaccount kubectl -n istio-system rollout status deployment/istiod kubectl -n istio-system rollout status daemonset/istio-cni-node istioctl analyze --all-namespaces ``` Validation requires one ready CNI pod on every schedulable node, no ztunnel DaemonSet, healthy istiod replicas, unchanged gateway Service and NodePort values, and no ownership or analyzer warnings relevant to the rollout. ### Gateway policy matrix For each deployed gateway, send requests directly to its HTTP NodePort so DNS and the bastion do not mask the policy: ```bash curl -i -H 'Host: ' http://:/ curl -i -H 'Host: :' http://:/ curl -i -H 'Host: rejected.invalid' http://:/ ``` The first two requests must reach the normal application or authentication response. The last must return gateway `403`, not merely an application 404. Gateway readiness, metrics scraping, and the normal bastion URL must remain healthy. Repeat after each gateway policy rather than batching all gateways. ### End-to-end validation cases | Case | Expected result | | ------------------------------------------------------- | ------------------------------------------------------------------------------------ | | web → k8s-root-actor | succeeds with `themelio/web` principal | | arbitrary `themelio` pod → k8s-root-actor | denied | | web gateway → web | succeeds with dedicated gateway principal | | direct plaintext or un-enrolled pod → web | denied | | web → gitlab-root-actor | succeeds and still requires application credentials | | Waldur order → gitlab-root-actor | succeeds with GitLab-sync principal | | Waldur membership sync → gitlab-root-actor | succeeds with GitLab-sync principal | | Waldur report → gitlab-root-actor | denied | | component-loading Job → gitlab-root-actor | denied | | seeder → gitlab-root-actor | succeeds with `themelio-seeder` principal | | arbitrary seeder-like pod → gitlab-root-actor | denied | | service gateway → k8s-deployment runner | succeeds with service-gateway principal (once the runner pod is injected) | | same-namespace pod → k8s-deployment runner | allowed by retained `ns-owner-access-istio` (`source.namespace`) — not a denial case | | foreign-namespace pod → k8s-deployment runner | denied once the runner pod is injected; inert (allowed) while un-injected | | service gateway → KServe predictor | succeeds with service-gateway principal (predictor is sidecar-injected) | | same-namespace pod → KServe predictor | allowed by retained `ns-owner-access-istio` — not a denial case | | foreign-namespace pod → KServe predictor | denied at the predictor sidecar | | injected predictor `workspace-init` clone | apt/curl/git+LFS egress succeeds (requires native sidecars or exempted init egress) | | deleted deployment/inference service | its `-service-gateway` policy is removed and no orphan remains | | valid workload identity without valid application token | application rejects request | Regression validation also covers normal browser login and navigation, project APIs, Git clone/push/LFS through the web gateway, deployment actions, Waldur reconciliation, the seeding workflow, readiness probes, external authorization, Prometheus scraping, and Loki log discovery. The operator commands, expected results, evidence fields, and cleanup steps are maintained in the [ADR-0025 validation document](/product/operations/adr-0025-validation.md). ### Definition of done - All static and render checks pass for `open`, `permissive`, and `enforced`. - `open` and `permissive` contain no ADR-0025 native `ALLOW` or strict-mTLS policies; `open` also contains no selected application sidecars. - `enforced` contains no empty gateway allow rules in desired state or the live cluster. - Every gateway and trusted application uses its specified unique identity. - Positive and negative identity tests pass for every protected destination. - Strict mTLS rejects plaintext and non-mesh direct access. - Sidecar mTLS, expected principals, metrics, and logs are observable. - Application authentication and user RBAC regression tests pass. - Rollback from `enforced` through `permissive` has been exercised in staging at least once. - ADR-0025 stays Draft until the recorded canary evidence is reviewed. ## Deployment & Rollout ### Phase 0: Preconditions and backup 1. Merge ADR-0024 so the dependency and links resolve. 2. Confirm the current Calico dataplane; stop if eBPF connect-time load balancing is enabled. 3. Inventory every deployed gateway, attached Gateway/VirtualService, Service, NodePort, ServiceAccount, AuthorizationPolicy, PeerAuthentication, RequestAuthentication, mesh telemetry configuration, namespace label, and Helm release. 4. Export these resources and current Helm values to an operator-controlled backup outside the cluster. 5. Record baseline success rates, latency, gateway `4xx/5xx`, root-actor calls, and normal product smoke-test results. 6. Ensure ADR-0024 preserves application ports, health probes, DNS, control-plane access, and Envoy metrics scraping. ### Phase 1: Consolidate ownership and install the sidecar control plane 1. Render the reduced Kubeflow desired state. 2. Upgrade base and istiod with Helm ownership and the default profile. 3. Adopt or install the default-profile CNI release. 4. Adopt the shared Kubeflow gateway without changing its Service contract. 5. Apply the reduced Kubeflow desired state. 6. Run component, Kubeflow, gateway, external-authz, browser, and Git regression checks. No selected application receives a sidecar in this phase. ### Phase 2: Reconcile gateway profile Local and dev remain `open`: reconcile every ADR-0025 native gateway `ALLOW` policy to `absent` and verify their normal routing plus independent application and `CUSTOM` authorization behavior. For one gateway at a time in the staging `enforced` promotion: 1. Create its dedicated ServiceAccount and update the gateway Deployment. 2. Wait for the rollout and confirm its certificate principal. 3. Replace the empty `ALLOW` rule with the configured host and host-with-port. 4. Run the three-request gateway test and its full service smoke test. 5. Check Envoy RBAC logs and metrics before moving to the next gateway. The phase completes only when all gateways deployed in that environment have scoped policies. A gateway absent from an environment is recorded as not deployed, not treated as a failed test. ### Phase 3: Establish application identities 1. Create `web`, `gitlab-root-actor`, `themelio-seeder`, and `waldur-site-agent-cloud-gitlab-sync` ServiceAccounts. 2. Add the Waldur sync identity to the required RBAC binding. 3. Move Waldur order and membership-sync to the sync identity; leave report and component-loading on their existing identity. 4. Update `web` and `gitlab-root-actor`; retain `k8s-root-actor-sa`. 5. Roll out each environment's `k8s-root-actor` overlay with its direct `SERVICE_PUBLIC_HOST` value and verify that it exactly matches the Ansible inventory's `dns.service`. Only after the rollout is healthy, remove any legacy `themelio-service-routing` ConfigMap from that cluster. 6. Update the seeder Job template while keeping token automount disabled. 7. Roll and smoke-test all long-running workloads. Do not apply backend allow or strict-mTLS policies yet. ### Phase 4: Staging permissive canary 1. Set staging `security_profile: permissive` and include only the GitOps `security-enrollment` components. 2. Label the `web` and `k8s-root-actor` pod templates for sidecar injection and roll them. 3. Confirm both workloads contain synchronized sidecars with the expected SPIFFE principals. 4. Test allowed and unapproved calls while the destination is permissive. 5. Confirm no native ADR-0025 `ALLOW` or strict PeerAuthentication exists. 6. Repeat allowed, unknown-caller, plaintext, direct pod-IP, application-token, and normal deployment tests. 7. Hold the canary for an agreed observation window long enough to cover normal browser and automated action traffic. Any unexplained connectivity error, identity mismatch, CNI/sidecar restart, or missing telemetry fails the gate. ### Phase 5: Staging enforcement Set staging `security_profile: enforced`, add the GitOps `security-enforcement` components, and enforce one destination path at a time: 1. Web gateway → web. 2. Web → gitlab-root-actor. 3. Waldur order and membership-sync → gitlab-root-actor. 4. Seeder → gitlab-root-actor. Because the gitlab-root-actor policy contains all three allowed principals, all three caller classes must already be sidecar-injected and positively tested before that policy becomes enforcing. Report, component-loading, and arbitrary pod negative tests run immediately afterward. ### Rollback Rollback is scoped to the last successful phase and performed in reverse order. #### Backend policy or enrollment rollback 1. Capture sidecar, application, policy, pod, and NetworkPolicy evidence. 2. Change `security_profile` to `permissive` and remove the enforcement component, which removes the destination AuthorizationPolicy and strict PeerAuthentication before enrollment changes. 3. Verify plaintext/current traffic is restored. 4. If a complete rollback is required, change `security_profile` to `open`, remove the enrollment component, and roll the affected pod templates. 5. Revert ServiceAccount changes only after no active policy refers to the new principal. Removing sidecar injection before strict mTLS or allow policy would leave the caller without an authenticated principal and can prolong the outage. #### Gateway policy rollback Change the affected environment to `permissive` or `open`, reconcile the native gateway `ALLOW` policy to `absent`, and wait for propagation. Do not restore an empty `rules: [{}]` policy. If the ServiceAccount caused the issue, revert it only after no backend trusts the new gateway principal. Preserve the unexpected-host test result for diagnosis. #### Sidecar control-plane or ownership rollback 1. Remove backend strict-mTLS and `ALLOW` policies. 2. Remove explicit application sidecar labels and roll the selected workloads. 3. Verify application traffic is restored before changing CNI or istiod. 4. Revert the shared gateway and control-plane values. 5. Restore the Kubeflow Istio install overlay only if Helm ownership itself is being abandoned, then apply the backed-up Kubeflow and Istio state. 6. Re-run all pre-rollout service and authentication tests. Changing the sidecar injector or CNI can affect pod startup cluster-wide. Production rollback therefore uses the same maintenance and node-safety controls as rollout. ## Open Questions 1. ~~Injecting the KServe predictor (section 9) requires the predictor's `workspace-init` egress to succeed while the pod is in the mesh. With a classic sidecar under Istio CNI this fails, because redirection is active during the init phase before Envoy starts, and `holdApplicationUntilProxyStarts` does not cover init containers.~~ **Resolved:** istiod is installed with `ENABLE_NATIVE_SIDECARS=true` (section 2.2), so the Envoy proxy starts as a native sidecar before the pod's app init containers on the cluster's Kubernetes 1.34, and the predictor's init egress is captured by a running proxy. The native-sidecar switch is a mesh-wide change and must pass the standard sidecar-startup and regression gates for every already-injected workload before promotion. Otherwise there are no unresolved design questions in this DD. Runtime compatibility and canary outcomes are explicit promotion gates; a failed gate keeps ADR-0025 Draft and triggers rollback rather than an undocumented design variation. ## References - [ADR-0025: Scoped Istio Authorization Policies](/adr/0025-scoped-istio-authorization-policies.md) - ADR-0024: Baseline L3/L4 Network Segmentation (historical reference unavailable) - [Istio: Install with Helm](https://istio.io/latest/docs/setup/install/helm/) - [Istio: Sidecar injection](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/) - [Istio: Install the CNI node agent](https://istio.io/latest/docs/setup/additional-setup/cni/) - [Istio: AuthorizationPolicy reference](https://istio.io/latest/docs/reference/config/security/authorization-policy/) - [Istio: PeerAuthentication reference](https://istio.io/latest/docs/reference/config/security/peer_authentication/) - [Istio: Verify mutual TLS](https://istio.io/latest/docs/tasks/security/authentication/authn-policy/#verify-mutual-tls-configuration) - [Istio: Debugging Envoy and istiod](https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/) - [Istio: Prometheus integration](https://istio.io/latest/docs/ops/integrations/prometheus/) - [Calico: eBPF and service meshes](https://docs.tigera.io/calico/latest/operations/ebpf/use-cases-ebpf)
Page source Source snapshot · Revision `f928834c8`.