---
title: "DD-0025: Scoped Istio Authorization Policies — Implementation"
---

# DD-0025: Scoped Istio Authorization Policies — Implementation

**Related ADR:** [ADR-0025: Scoped Istio Authorization Policies](/adr/0025-scoped-istio-authorization-policies)

**Status:** Draft

**Date:** 2026-07-20

**Authors:** TBD

## ADR-derived technical notes

ADR-0025 selects scoped ingress host allowlists and scoped Istio sidecar
injection. The first internal policies authorize Kubernetes ServiceAccount
identities and workload destination ports only. Envoy sidecars can enforce L7
attributes, but HTTP methods, paths, headers, request principals, `CUSTOM`, and
`AUDIT` conditions remain outside the initial backend policies.

Repository inspection found these implementation constraints:

- `istio_version` is the common pin for Helm charts, Kubeflow Istio images, and
  manually rendered gateway proxy images.
- `istio-deploy` currently installs `istio-base`, `istiod`, and a gateway with
  Helm, but it only patches an Istio CNI DaemonSet if another deployment already
  created one. Kubeflow's rendered kustomization still includes an overlapping
  Istio installation overlay.
- The Themelio web and Argo CD gateways are GitOps-owned. The GitLab, Waldur,
  image registry, MLflow, monitoring, Ceph dashboard, Kubeflow, and Themelio
  workload gateways are Ansible-owned.
- The ingress `ALLOW` policies for web, Argo CD, GitLab, Waldur, the image
  registry, MLflow, Grafana, Prometheus, Ceph, and the workload gateway contain
  an empty rule and therefore match every request.
- `web` and `gitlab-root-actor` currently run as the `default` ServiceAccount in
  `themelio`. `k8s-root-actor` already runs as `k8s-root-actor-sa` in every
  deployed overlay.
- The Waldur order, membership-sync, report, and component-loading workloads all
  use `waldur-site-agent-cloud`. Trusting that identity would trust more than the
  two GitLab-sync processes named in ADR-0025.
- The seeder still calls `gitlab-root-actor`. It disables ServiceAccount token
  automount but does not currently select a dedicated ServiceAccount.
- Local and staging have been reachable during implementation. Earlier staging
  evidence was collected from the superseded Ambient pilot and is not accepted
  as sidecar evidence; every mesh-specific gate must be rerun after migration.
  Dev and production runtime evidence remain promotion criteria.

The related [ADR-0024](/adr/0024-baseline-l3-l4-network-segmentation) currently
lives on a separate documentation branch. It must be merged before the published
ADR-0025 and DD-0025 links are expected to resolve. Its NetworkPolicy controls
are a rollout prerequisite, not duplicated ownership in this DD.

## Overview

This document specifies how Themelio will consolidate Istio component ownership
under Ansible and Helm, restrict every deployed ingress gateway to its configured
HTTP host, give gateways and selected applications unambiguous workload
identities, and incrementally protect the ADR-0025 internal flows with sidecar
mTLS and workload authorization. Local and dev remain open by default; staging and
production use enforced security after their canary gates pass. No application
API, database schema, or user-facing action contract changes.

## Goals and Non-Goals

### Goals

- Make Helm, driven by `istio-deploy`, the single owner of Istio base, istiod,
  CNI, and the shared Kubeflow ingress gateway.
- Remove every empty gateway `ALLOW` rule and derive allowed hosts from the same
  value used by the gateway and its VirtualService.
- Give every ingress gateway a unique ServiceAccount without changing its
  external listener, route, or authentication behavior.
- Protect the selected web, root-actor, site-agent, gateway, and seeder flows
  with mTLS-authenticated ServiceAccount identity and destination workload port.
- Inject individual pod templates rather than whole application namespaces.
- Scope each tenant (Kubeflow profile) namespace so a Themelio deployment or
  inference service is reachable only through the service ingress gateway
  identity, with no namespace-wide `ALLOW` rule.
- Preserve external authorization, application authentication, user RBAC,
  NetworkPolicy, and service-specific credentials as independent controls.
- Produce repeatable validation and rollback procedures for every security
  profile.

### Non-Goals

- Namespace-wide sidecar injection or automatic injection into tenant workloads.
- A root-namespace deny-all AuthorizationPolicy.
- HTTP method, path, header, JWT, or new traffic-routing policy at application
  sidecars.
- Mesh enrollment of PostgreSQL, Valkey, GitLab, or arbitrary user deployments.
- Replacing Calico, enabling its eBPF dataplane, or removing kube-proxy.
- Changing REST, GraphQL, action, database, or user authorization contracts.
- Claiming the ADR is accepted before the canary and promotion evidence exists.

## Design

### 1. Security profiles and ownership boundaries

`security_profile` controls only ADR-0025 sidecar injection and native
`ALLOW`/strict-mTLS enforcement:

| Profile | Sidecar injection | STRICT mTLS / ALLOW policies |
|---|---|---|
| `open` | No | No |
| `permissive` | Yes | No |
| `enforced` | Yes | Yes |

The environment assignments are explicit:

| Environment | `security_profile` |
|---|---|
| Local | `open` |
| Dev | `open` |
| Staging | `enforced` after canary validation |
| Production | `enforced` after canary validation |

`permissive` is a transitional rollout and rollback profile, not a permanent
environment class. Dedicated ServiceAccounts, Istio CNI, sidecar-capable
istiod, Gateway and VirtualService routing, application authentication, user
RBAC, service credentials, and existing `CUSTOM` external authorization remain
present in all profiles.

Ansible reads `security_profile` from each environment's
`inventory/<environment>/group_vars/all.yml`. GitOps cannot consume Ansible
inventory directly, so matching Kustomize components implement the same state:

- no security component for `open`,
- `security-enrollment` only for `permissive`, and
- both `security-enrollment` and `security-enforcement` for `enforced`.

Static validation must compare the inventory value with the rendered GitOps
resources and fail if they disagree. Local/dev overlays omit both components;
staging includes both. Any future production Themelio or Argo CD overlay must
include both before it is registered by the production root application.
Other environment differences remain data such as `dns.*`, node selectors,
tolerations, replica counts, and secret values.

Resource ownership after this change is:

| Resource family | Owner |
|---|---|
| Istio CRDs and cluster RBAC | `istio-base` Helm release through `istio-deploy` |
| Control plane and mesh config | `istiod` Helm release plus `istio-deploy` mesh ConfigMap |
| Sidecar traffic capture | `istio-cni` Helm release through `istio-deploy` |
| Shared Kubeflow ingress gateway | Helm gateway release through `istio-deploy` |
| Web and Argo CD gateway workloads and policies | their GitOps/Kustomize applications |
| Other standalone gateway workloads and policies | the Ansible role that owns the exposed service |
| Themelio application identities and backend policies | Themelio GitOps/Kustomize |
| Waldur process identities | Waldur site-agent GitOps/Kustomize |
| NetworkPolicy and network zones | ADR-0024 platform automation |
| Metrics, dashboards, and alerts | monitoring Ansible roles |

Tenant roles must not be able to edit these ServiceAccounts, gateway or backend
AuthorizationPolicies, PeerAuthentications, enrollment labels, or platform-owned
NetworkPolicies.

### 2. Sidecar-capable Istio installation

`istio-deploy` will expose and validate these variables:

| Variable | Default | Purpose |
|---|---|---|
| `istio_cni_release_name` | `istio-cni` | Helm ownership identity for the CNI node agent |
| `istio_shared_gateway_release_name` | `istio-ingressgateway` | Helm ownership identity for the Kubeflow gateway |
| `istio_cni_resources` | role defaults | CNI resource requests and limits |
| `istio_wait_timeout` | existing value | Atomic Helm and rollout timeout |

Every chart uses `istio_chart_version`, which defaults to `istio_version`.
Standalone gateways continue using `docker.io/istio/proxyv2:{{ istio_version }}`.
A render or deployment must fail when those versions differ.

The Helm sequence is:

1. Upgrade or install `istio-base`.
2. Upgrade or install `istiod` with the chart's default sidecar values and the
   existing resource, placement, mesh config, trust domain, and
   extension-provider settings.
3. Upgrade or install `istio-cni` with Ambient disabled on every schedulable
   Kubernetes node.
4. Reconcile the shared Kubeflow gateway with its existing service ports,
   NodePorts, selectors, and dedicated ServiceAccount.
5. Reconcile standalone gateway proxy images only after the control plane is
   available.

The Helm operations remain atomic. The role waits for the `istiod` Deployment
and CNI DaemonSet to report all desired pods ready. CNI installation does not
inject a proxy by itself; injection is selected at each pod template.

Kubespray sets `fs.inotify.max_user_instances=1024` through
`additional_sysctl`. The kernel default of `128` was exhausted on a
sidecar-heavy staging node and prevented the Istio CNI agent from creating its
CNI-configuration watcher during an upgrade.

#### 2.1 Kubeflow ownership migration

The Kubeflow kustomization stops including
`../common/istio/istio-install/overlays/oauth2-proxy`. It also drops the image
rewrites for pilot, proxyv2, and install-cni and the istiod Deployment/HPA
replica patches. It retains:

- `../common/istio/cluster-local-gateway/base`,
- `../common/istio/kubeflow-istio-resources/base`,
- Kubeflow Gateway and VirtualService resources,
- the custom OAuth2 proxy resources and host-scoped `CUSTOM` policy, and
- the existing Ansible-managed Istio `extensionProviders` mesh configuration.

Migration order matters:

1. Render the future Kubeflow kustomization and verify it no longer declares
   Istio base, istiod, the main ingress gateway, or the CNI DaemonSet.
2. Let Helm adopt matching live resources with `--take-ownership` while the old
   Kubeflow apply path is still available for rollback.
3. Verify Helm owns all four releases and the current Kubeflow gateway remains
   ready and routable.
4. Apply the reduced Kubeflow kustomization. Server-side apply must not regain
   ownership of fields managed by Helm.
5. Remove the old `istio-ingress` Helm gateway only after queries show that no
   Istio Gateway selector, Kubernetes Service, or traffic path uses it.

The operator must stop if Helm adoption would replace a Service ClusterIP,
NodePort, selector, or ServiceAccount rather than preserving it.

#### 2.2 Scoped sidecar injection

Existing Envoy gateways remain standalone router proxies. Selected application
Deployments and Jobs receive `sidecar.istio.io/inject: "true"` in
`spec.template.metadata.labels`. No Themelio or Waldur namespace receives
`istio-injection=enabled`, so unrelated pods remain outside the mesh. Istio CNI
performs traffic redirection without a privileged init container, while the
injected Envoy remains visible as `istio-proxy` in pod status and telemetry.

istiod is installed with `pilot.env.ENABLE_NATIVE_SIDECARS=true` (the
`istio_enable_native_sidecars` default in the `istio-deploy` role), so the Envoy
proxy is injected as a Kubernetes native sidecar - an init container with
`restartPolicy: Always` — supported by the cluster's Kubernetes 1.34. The proxy
therefore starts before a pod's app init containers, so init-time egress is
captured by a running Envoy instead of failing against traffic redirection that
is active before a classic sidecar would start. This is what makes injecting the
KServe predictor (section 9), whose `workspace-init` container clones the model
over git/LFS, safe.

### 3. Ingress host authorization

An ingress host policy is an L7 policy enforced by that gateway's Envoy proxy.
It uses the HTTP `Host` or HTTP/2 `:authority` value and does not
validate DNS resolution.

Every deployed gateway receives one scoped `ALLOW` policy:

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <gateway-name>
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: <gateway-name>
  action: ALLOW
  rules:
    - to:
        - operation:
            hosts:
              - "<configured-host>"
              - "<configured-host>:*"
```

The second host form is required because clients and reverse proxies may include
an explicit port in `Host`. No wildcard domain is introduced unless the Gateway
and VirtualService already use that same wildcard.

| Gateway | Owner | Host source | Target ServiceAccount |
|---|---|---|---|
| `istio-ingressgateway-web` | Themelio GitOps | web overlay Gateway/VirtualService host | `istio-ingressgateway-web-service-account` |
| `istio-ingressgateway-argocd` | Argo CD GitOps | Argo CD overlay Gateway/VirtualService host | `istio-ingressgateway-argocd-service-account` |
| `istio-ingressgateway-gitlab` | `deploy_gitlab` | `dns.gitlab` | `istio-ingressgateway-gitlab-service-account` |
| `istio-ingressgateway-waldur` | `waldur-deploy` | `dns.waldur` | `istio-ingressgateway-waldur-service-account` |
| `istio-ingressgateway-image-cache` | `image-registry` | `dns.registry` | `istio-ingressgateway-image-cache-service-account` |
| `istio-ingressgateway-mlflow` | `mlflow-router` | `dns.mlflow` | `istio-ingressgateway-mlflow-service-account` |
| `istio-ingressgateway-mlflow-router` | `mlflow-router` | `dns.mlflow` | `istio-ingressgateway-mlflow-router-service-account` |
| `istio-ingressgateway-grafana` | monitoring role | `dns.grafana` | `istio-ingressgateway-grafana-service-account` |
| `istio-ingressgateway-prometheus` | monitoring role | `dns.prometheus` | `istio-ingressgateway-prometheus-service-account` |
| `istio-ingressgateway-ceph-dashboard` | storage role | `dns.ceph_dashboard` | `istio-ingressgateway-ceph-dashboard-service-account` |
| `istio-ingressgateway-service` | Themelio Ansible role | `dns.service`; the matching GitOps overlay sets `SERVICE_PUBLIC_HOST` directly on `k8s-root-actor` | `istio-ingressgateway-themelio-service-account` |
| `istio-ingressgateway` | `istio-deploy`/Kubeflow | `dns.kubeflow` | `istio-ingressgateway-service-account` |

`istio-ingressgateway-service-account` is deliberately retained for Kubeflow.
The Waldur Kubeflow client already writes that principal into per-user policies.
The Themelio workload gateway must move away from that shared identity.

Existing `CUSTOM` policies for Kubeflow and MLflow remain present. Istio evaluates
`CUSTOM` before native `ALLOW`; therefore a request must pass external
authorization when its `CUSTOM` rule applies and must also match the gateway host
allowlist in the `enforced` profile. An unknown host then matches no `ALLOW` rule
and returns `403`. In `open` and `permissive`, the ADR-0025 gateway `ALLOW`
policy is absent; the existing `CUSTOM` and application controls still apply.

Host rules are generated from the same input as the matching Gateway and
VirtualService:

- Kustomize overlays patch the Gateway, VirtualService, and AuthorizationPolicy
  with one environment host value.
- Ansible templates reference the existing `dns.*` value in all three resources.
- The Themelio service gateway spans ownership boundaries: Ansible renders the
  gateway host from `dns.service`, while each GitOps overlay sets the same
  environment-specific hostname directly as `SERVICE_PUBLIC_HOST` on
  `k8s-root-actor`. No routing ConfigMap is used. Render validation must compare
  these two values and fail on any mismatch before promotion.
- A role does not render the policy when its gateway is disabled or its host is
  undefined; it fails validation if the gateway is enabled without a host.
- Ansible reconciles the native host `ALLOW` policy to `absent` unless
  `security_profile` is `enforced`; merely skipping a task is prohibited because
  it could leave a stale policy active.

Gateway readiness port `15021`, Envoy metrics, and NodePort behavior must be
tested after each policy. They are not added as public host exceptions.

### 4. Workload identities

The target identities and SPIFFE principals are:

| Workload | ServiceAccount | Principal |
|---|---|---|
| web | `web` | `cluster.local/ns/themelio/sa/web` |
| k8s-root-actor | `k8s-root-actor-sa` | `cluster.local/ns/themelio/sa/k8s-root-actor-sa` |
| gitlab-root-actor | `gitlab-root-actor` | `cluster.local/ns/themelio/sa/gitlab-root-actor` |
| web gateway | `istio-ingressgateway-web-service-account` | `cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account` |
| Waldur GitLab sync | `waldur-site-agent-cloud-gitlab-sync` | `cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync` |
| Waldur report and component loading | `waldur-site-agent-cloud` | `cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud` |
| seeder | `themelio-seeder` | `cluster.local/ns/themelio/sa/themelio-seeder` |

ServiceAccounts are declared in the same desired-state package as their
workloads. A Deployment changes identity by setting `serviceAccountName`; this
causes a normal rollout. `automountServiceAccountToken: false` remains on the
seeder because application access to the Kubernetes API is unnecessary. Workload
identity comes from the certificate issued to its injected sidecar and does
not require exposing a Kubernetes token inside the seeder container.

The Waldur order and membership-sync Deployments move to
`waldur-site-agent-cloud-gitlab-sync`. Its ClusterRoleBinding contains this new
subject so existing Kubernetes reconciliation behavior continues. Report and
component-loading retain `waldur-site-agent-cloud` and are not included in the
GitLab root actor allowlist. This split is mandatory before trusting the sync
principal.

### 5. Sidecar injection and backend policy

In `permissive` and `enforced`, the enrollment component places
`sidecar.istio.io/inject: "true"` only on these pod templates:

- `web`
- `k8s-root-actor`
- `gitlab-root-actor`
- Waldur order process
- Waldur membership-sync process
- `themelio-seeder`

The web ingress gateway is already an Istio proxy and does not receive an
application sidecar.

In `open`, none of these pod templates requests injection. In `permissive`,
they receive sidecars but the backend AuthorizationPolicies and
workload-scoped strict PeerAuthentications are absent. In `enforced`, the same
injection is combined with the policies below.

Backend `AuthorizationPolicy` resources live beside the destination workload.
The `ports` values are destination workload ports, not Kubernetes Service ports.
The three Themelio applications listen on workload port `3000` even though
their Services expose port `80`.

#### 5.1 k8s-root-actor canary policy

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: k8s-root-actor-allow-web
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: k8s-root-actor
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/themelio/sa/web
      to:
        - operation:
            ports: ["3000"]
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: k8s-root-actor-strict-mtls
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: k8s-root-actor
  mtls:
    mode: STRICT
```

#### 5.2 web policy

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: web-allow-ingress-gateway
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: web
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account
      to:
        - operation:
            ports: ["3000"]
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: web-strict-mtls
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: web
  mtls:
    mode: STRICT
```

#### 5.3 gitlab-root-actor policy

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: gitlab-root-actor-allow-callers
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: gitlab-root-actor
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/themelio/sa/web
              - cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync
              - cluster.local/ns/themelio/sa/themelio-seeder
      to:
        - operation:
            ports: ["3000"]
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: gitlab-root-actor-strict-mtls
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: gitlab-root-actor
  mtls:
    mode: STRICT
```

The policies intentionally contain no L7 fields. The destination Envoy sidecar
is the enforcement point and authenticates the original caller's sidecar
certificate. Adding HTTP methods, paths, or headers is technically possible in
sidecar mode but requires a new or revised DD defining stable L7 contracts.

Application authorization is unchanged. For example, a pod with an allowed
workload identity still needs the configured root-actor bearer token where the
endpoint requires one. Workload identity does not grant a human permission.

### 6. NetworkPolicy contract with ADR-0024

Before any pod receives a sidecar, ADR-0024 policy must preserve:

- existing application traffic at Kubernetes Service and workload ports,
- DNS and control-plane traffic needed by the workload and Istio proxy,
- kubelet health probes to the workload, including Istio probe rewriting, and
- Prometheus access to the Envoy metrics port where sidecar scraping is enabled.

No HBONE port `15008` exception is required for this design. Sidecar-to-sidecar
mTLS uses the application's normal destination port.

The pilot runs on the current standard Calico dataplane. If a later ADR-0024
rollout enables Calico eBPF, its connect-time load balancing must remain disabled
before sidecars are promoted. The combined conformance suite must cover ClusterIP,
NodePort, DNS, host-to-Service, pod-to-Service, Gateway/VirtualService routing,
failure recovery, and rollback.

### 7. End-to-end request paths

```mermaid
flowchart LR
  Client[Browser or API client] -->|TLS| Nginx[Bastion Nginx]
  Nginx -->|HTTP + preserved Host| GW[Dedicated Envoy gateway]
  GW -->|host ALLOW + mTLS| WebProxy[web Envoy sidecar]
  WebProxy --> Web[web :3000]
  Web --> WebProxy
  WebProxy -->|mTLS as themelio/web| KProxy[k8s-root-actor Envoy]
  KProxy -->|principal + port ALLOW| KRA[k8s-root-actor :3000]
  WebProxy -->|mTLS as themelio/web| GProxy[gitlab-root-actor Envoy]
  WaldurSync[Waldur sync + sidecar] -->|mTLS| GProxy
  Seeder[seeder + sidecar] -->|mTLS| GProxy
  GProxy -->|principal + port ALLOW| GRA[gitlab-root-actor :3000]
```

The bastion-to-gateway hop remains plain HTTP on the private network as recorded
in ADR-0025. The gateway-to-backend and selected internal flows use Envoy
sidecar mTLS. Calls from injected workloads to destinations outside the mesh, such
as GitLab, PostgreSQL, or Valkey, retain their existing transport and application
security behavior.

### 8. Observability

Sidecar telemetry is collected separately from application telemetry:

- Prometheus scrapes istiod, Envoy sidecars, and Envoy gateway metrics explicitly.
- Mesh configuration selects the built-in Prometheus metrics provider through
  `defaultProviders.metrics`.
- Application metrics continue to use their existing scrape configuration.
  Application/sidecar metric merging is not assumed.
- The monitoring role adds an `istio-workload-sidecars` `PodMonitor` selected by
  `security.istio.io/tlsMode=istio`.
- Existing Alloy pod-log collection forwards gateway and `istio-proxy` logs
  to Loki.
- Grafana panels show sidecar scrape readiness, mTLS request/connection counts,
  TCP bytes, source and destination principals, and authorization denials where
  the exported labels support them.

Alerts and runbooks distinguish:

| Symptom | Primary evidence |
|---|---|
| Unknown or missing HTTP host | gateway Envoy access log with `403`; gateway RBAC metrics |
| External authorization rejection | gateway/ext-authz log and response details |
| Plaintext or certificate failure | destination Envoy log; PeerAuthentication state |
| Unauthorized workload identity | destination Envoy RBAC log and metric with source principal |
| NetworkPolicy denial | ADR-0024 Calico flow verdict; connection timeout |
| Application authentication or failure | application response and application logs after mesh validation |

The rollout checklist records links or captured output for each of these signals.

### 9. Tenant-namespace workload authorization

Kubeflow profile namespaces host tenant workloads. The Themelio deployments and
KServe inference services that `k8s-root-actor` creates must be reachable from
the shared Themelio service ingress gateway, but the namespace must not carry a
namespace-wide `ALLOW` rule (`rules: [{}]`) that would admit every source.
Instead `k8s-root-actor` generates one workload-scoped `AuthorizationPolicy` per
deployment and per inference service, and deletes it with the workload.

The following Kubeflow-generated policies are retained unchanged and are not
replaced by this design:

- `ns-owner-access-istio` — the namespace owner through the Kubeflow gateway,
  `ml-pipeline-ui` traffic carrying the owner identity, same-namespace workload
  traffic, health and metrics endpoints, and notebook-controller access to
  notebook kernels.
- every `user-<email>-clusterrole-edit` policy — each contributor through the
  Kubeflow gateway and `ml-pipeline-ui` only when the `kubeflow-userid` header
  matches that contributor.

Generated with every Themelio `k8s-deployment`
(`k8s-root-actor` template `k8s-deployment/authorizationpolicy.yaml`):

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <deployment-name>-service-gateway
  namespace: <tenant>
spec:
  selector:
    matchLabels:
      app: <deployment-name>
      component: themelio-k8s-deployment-runner
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account
      to:
        - operation:
            ports:
              - "<configured-port-or-8000>"
```

Generated with every `k8s-inferenceservice`
(`k8s-root-actor` template `k8s-inferenceservice/authorizationpolicy.yaml`). The
selector targets the KServe-generated predictor pod labels:

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <inferenceservice-name>-service-gateway
  namespace: <tenant>
spec:
  selector:
    matchLabels:
      serving.kserve.io/inferenceservice: <inferenceservice-name>
      component: predictor
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account
      to:
        - operation:
            ports:
              - "<configured-port-or-8000>"
```

- `<configured-port-or-8000>` is the workload's configured port, defaulting to
  `8000` — the same default the Deployment container port, the predictor
  `--http_port`, and the Service `targetPort` already use.
- The admitted principal is the Themelio service gateway ServiceAccount, read
  from `k8s-root-actor`'s `SERVICE_GATEWAY_PRINCIPAL` config (default
  `cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account`).
- Each policy is applied in the same concurrent tier as the workload it selects
  and is deleted by the matching teardown path (`destroyK8sDeployment`,
  `destroyK8sInferenceService`), so it never outlives its workload.

Enforcement prerequisite: Istio enforces an `AuthorizationPolicy` at the selected
pod's sidecar. The inference predictor template therefore sets
`sidecar.istio.io/inject: "true"` (with `holdApplicationUntilProxyStarts` on the
model-server container), so the `<name>-service-gateway` policy is enforced. The
deployment-runner pod carries no explicit injection annotation, so its policy is
enforced only where the tenant namespace already injects the pod; where it does
not, the policy is inert (fail-open) but harmless until the runner joins the
mesh.

Injecting the predictor depends on the mesh running native sidecars: the
predictor's `workspace-init` container performs `apt`/`curl`/`git`+LFS egress
during the init phase, which a classic (non-native) sidecar would break because
traffic redirection is active during init while Envoy is not yet running
(`holdApplicationUntilProxyStarts` does not cover init containers). istiod is
installed with `ENABLE_NATIVE_SIDECARS=true` (section 2.2), so the proxy starts
before the init container and this egress succeeds.

No tenant policy is generated for two workload classes:

- `slurm-job` — its launcher and cancellation Jobs run in the fixed `themelio`
  namespace, not a tenant namespace.
- the Waldur site agent — it mutates tenant resources through the Kubernetes
  API, so its access is governed by Kubernetes RBAC, not Istio.

The result preserves every currently implemented tenant workflow — Kubeflow
owner and contributor access, notebooks, pipelines, health checks,
tenant-internal communication, Themelio deployment and inference reachability
through the dedicated gateway, and Waldur resource management — while removing
namespace-wide unrestricted access. No tenant namespace retains a `rules: [{}]`
policy, and requests from unapproved external identities are denied.

## Data Model

No persistent application data changes. The security configuration itself is a
versioned contract:

| Contract | Source of truth | Compatibility rule |
|---|---|---|
| Security profile | Ansible `security_profile` plus matching GitOps component selection | `open`: neither component; `permissive`: enrollment only; `enforced`: enrollment and enforcement |
| Istio component version | `istio_version` | all charts and proxy images match |
| Trust domain | mesh config `cluster.local` | principal strings change atomically if the trust domain changes |
| Gateway allowed host | Gateway/VirtualService environment input | AuthorizationPolicy consumes the same input |
| Workload identity | Kubernetes ServiceAccount name and namespace | policy update precedes any intentional rename |
| Sidecar injection | pod-template label | namespace labels are prohibited for this rollout |
| Backend listener | Themelio workload port `3000` | policy uses workload port, not Service port `80` |
| Tenant workload scoping | `k8s-root-actor` `SERVICE_GATEWAY_PRINCIPAL` and the `<name>-service-gateway` policy | policy is created and deleted with its deployment/inference workload; principal must match the service gateway ServiceAccount |

ServiceAccount names and their SPIFFE principals are security-sensitive public
deployment interfaces. Renaming one without updating and revalidating every
policy that trusts it is a breaking change.

## API Design

No Themelio REST, GraphQL, database, or action API changes.

The Kubernetes API surface added or changed by this design consists of:

- `ServiceAccount` resources and Deployment/Job `serviceAccountName` fields,
- pod-template `sidecar.istio.io/inject: "true"` labels,
- scoped `security.istio.io/v1` `AuthorizationPolicy` resources,
- workload-scoped `security.istio.io/v1` `PeerAuthentication` resources, and
- Helm releases and values for Istio base, istiod, CNI, and the shared gateway,
- Kustomize `security-enrollment` and `security-enforcement` components selected
  by the environment overlay to match `security_profile`.

All backend AuthorizationPolicies use workload selectors and are enforced by
destination sidecars. Existing `RequestAuthentication` and `CUSTOM`
AuthorizationPolicy contracts remain unchanged.

## Testing Strategy

### Static and rendered-manifest tests

Run before connecting to a cluster:

1. Build every Themelio, Argo CD, and Waldur site-agent Kustomize overlay.
2. Render the affected Ansible templates for `open`, `permissive`, and
   `enforced`.
3. Run repository YAML checks, Ansible syntax checks, and Ansible lint.
4. Run `helm template` for base, default sidecar istiod, sidecar-only CNI, and
   the shared gateway at the pinned `istio_version`.
5. Run offline `istioctl analyze` over the composed Istio and application
   manifests.
6. Assert that no gateway `ALLOW` policy contains `rules: [{}]`, `rules: - {}`,
   or an omitted host operation.
7. Assert that no gateway Deployment uses `default`, `istiod`, `istio-ingress`,
   or another gateway's ServiceAccount.
8. Assert that local and dev render no sidecar injection labels, native ADR-0025
   `ALLOW` policies, or strict PeerAuthentications.
9. Assert that `permissive` renders selected sidecar injection labels but no
   native ADR-0025 `ALLOW` policies or strict PeerAuthentications.
10. Assert that `enforced` renders enrollment plus all expected policies.
11. Assert that all backend AuthorizationPolicies contain only `principals` and
   `ports`; Themelio backends use `3000`.
12. Assert that only the selected application pod templates carry the explicit
    sidecar injection label.
13. Assert that the Waldur report and load-components workloads do not use the
    GitLab-sync ServiceAccount.
14. Assert that the rendered `k8s-root-actor` tenant `AuthorizationPolicy`
    templates select the workload (deployment-runner or KServe predictor)
    labels, admit only the service gateway principal on the workload port, and
    contain no `rules: [{}]` or namespace-wide rule.

### Component and ownership validation

```bash
helm list -n istio-system
kubectl -n istio-system get deploy,daemonset,service,serviceaccount
kubectl -n istio-system rollout status deployment/istiod
kubectl -n istio-system rollout status daemonset/istio-cni-node
istioctl analyze --all-namespaces
```

Validation requires one ready CNI pod on every schedulable node, no ztunnel
DaemonSet, healthy istiod replicas, unchanged gateway Service and NodePort
values, and no ownership or analyzer warnings relevant to the rollout.

### Gateway policy matrix

For each deployed gateway, send requests directly to its HTTP NodePort so DNS
and the bastion do not mask the policy:

```bash
curl -i -H 'Host: <configured-host>' http://<node-ip>:<node-port>/
curl -i -H 'Host: <configured-host>:<node-port>' http://<node-ip>:<node-port>/
curl -i -H 'Host: rejected.invalid' http://<node-ip>:<node-port>/
```

The first two requests must reach the normal application or authentication
response. The last must return gateway `403`, not merely an application 404.
Gateway readiness, metrics scraping, and the normal bastion URL must remain
healthy. Repeat after each gateway policy rather than batching all gateways.

### End-to-end validation cases

| Case | Expected result |
|---|---|
| web → k8s-root-actor | succeeds with `themelio/web` principal |
| arbitrary `themelio` pod → k8s-root-actor | denied |
| web gateway → web | succeeds with dedicated gateway principal |
| direct plaintext or un-enrolled pod → web | denied |
| web → gitlab-root-actor | succeeds and still requires application credentials |
| Waldur order → gitlab-root-actor | succeeds with GitLab-sync principal |
| Waldur membership sync → gitlab-root-actor | succeeds with GitLab-sync principal |
| Waldur report → gitlab-root-actor | denied |
| component-loading Job → gitlab-root-actor | denied |
| seeder → gitlab-root-actor | succeeds with `themelio-seeder` principal |
| arbitrary seeder-like pod → gitlab-root-actor | denied |
| service gateway → k8s-deployment runner | succeeds with service-gateway principal (once the runner pod is injected) |
| same-namespace pod → k8s-deployment runner | allowed by retained `ns-owner-access-istio` (`source.namespace`) — not a denial case |
| foreign-namespace pod → k8s-deployment runner | denied once the runner pod is injected; inert (allowed) while un-injected |
| service gateway → KServe predictor | succeeds with service-gateway principal (predictor is sidecar-injected) |
| same-namespace pod → KServe predictor | allowed by retained `ns-owner-access-istio` — not a denial case |
| foreign-namespace pod → KServe predictor | denied at the predictor sidecar |
| injected predictor `workspace-init` clone | apt/curl/git+LFS egress succeeds (requires native sidecars or exempted init egress) |
| deleted deployment/inference service | its `<name>-service-gateway` policy is removed and no orphan remains |
| valid workload identity without valid application token | application rejects request |

Regression validation also covers normal browser login and navigation, project
APIs, Git clone/push/LFS through the web gateway, deployment actions, Waldur
reconciliation, the seeding workflow, readiness probes, external authorization,
Prometheus scraping, and Loki log discovery.

The operator commands, expected results, evidence fields, and cleanup steps are
maintained in the
[ADR-0025 validation document](/product/operations/adr-0025-validation).

### Definition of done

- All static and render checks pass for `open`, `permissive`, and `enforced`.
- `open` and `permissive` contain no ADR-0025 native `ALLOW` or strict-mTLS
  policies; `open` also contains no selected application sidecars.
- `enforced` contains no empty gateway allow rules in desired state or the live
  cluster.
- Every gateway and trusted application uses its specified unique identity.
- Positive and negative identity tests pass for every protected destination.
- Strict mTLS rejects plaintext and non-mesh direct access.
- Sidecar mTLS, expected principals, metrics, and logs are observable.
- Application authentication and user RBAC regression tests pass.
- Rollback from `enforced` through `permissive` has been exercised in staging at
  least once.
- ADR-0025 stays Draft until the recorded canary evidence is reviewed.

## Deployment & Rollout

### Phase 0: Preconditions and backup

1. Merge ADR-0024 so the dependency and links resolve.
2. Confirm the current Calico dataplane; stop if eBPF connect-time load balancing
   is enabled.
3. Inventory every deployed gateway, attached Gateway/VirtualService, Service,
   NodePort, ServiceAccount, AuthorizationPolicy, PeerAuthentication,
   RequestAuthentication, mesh telemetry configuration, namespace label, and
   Helm release.
4. Export these resources and current Helm values to an operator-controlled
   backup outside the cluster.
5. Record baseline success rates, latency, gateway `4xx/5xx`, root-actor calls,
   and normal product smoke-test results.
6. Ensure ADR-0024 preserves application ports, health probes, DNS,
   control-plane access, and Envoy metrics scraping.

### Phase 1: Consolidate ownership and install the sidecar control plane

1. Render the reduced Kubeflow desired state.
2. Upgrade base and istiod with Helm ownership and the default profile.
3. Adopt or install the default-profile CNI release.
4. Adopt the shared Kubeflow gateway without changing its Service contract.
5. Apply the reduced Kubeflow desired state.
6. Run component, Kubeflow, gateway, external-authz, browser, and Git regression
   checks. No selected application receives a sidecar in this phase.

### Phase 2: Reconcile gateway profile

Local and dev remain `open`: reconcile every ADR-0025 native gateway `ALLOW`
policy to `absent` and verify their normal routing plus independent application
and `CUSTOM` authorization behavior.

For one gateway at a time in the staging `enforced` promotion:

1. Create its dedicated ServiceAccount and update the gateway Deployment.
2. Wait for the rollout and confirm its certificate principal.
3. Replace the empty `ALLOW` rule with the configured host and host-with-port.
4. Run the three-request gateway test and its full service smoke test.
5. Check Envoy RBAC logs and metrics before moving to the next gateway.

The phase completes only when all gateways deployed in that environment have
scoped policies. A gateway absent from an environment is recorded as not
deployed, not treated as a failed test.

### Phase 3: Establish application identities

1. Create `web`, `gitlab-root-actor`, `themelio-seeder`, and
   `waldur-site-agent-cloud-gitlab-sync` ServiceAccounts.
2. Add the Waldur sync identity to the required RBAC binding.
3. Move Waldur order and membership-sync to the sync identity; leave report and
   component-loading on their existing identity.
4. Update `web` and `gitlab-root-actor`; retain `k8s-root-actor-sa`.
5. Roll out each environment's `k8s-root-actor` overlay with its direct
   `SERVICE_PUBLIC_HOST` value and verify that it exactly matches the Ansible
   inventory's `dns.service`. Only after the rollout is healthy, remove any
   legacy `themelio-service-routing` ConfigMap from that cluster.
6. Update the seeder Job template while keeping token automount disabled.
7. Roll and smoke-test all long-running workloads. Do not apply backend allow or
   strict-mTLS policies yet.

### Phase 4: Staging permissive canary

1. Set staging `security_profile: permissive` and include only the GitOps
   `security-enrollment` components.
2. Label the `web` and `k8s-root-actor` pod templates for sidecar injection and
   roll them.
3. Confirm both workloads contain synchronized sidecars with the expected
   SPIFFE principals.
4. Test allowed and unapproved calls while the destination is permissive.
5. Confirm no native ADR-0025 `ALLOW` or strict PeerAuthentication exists.
6. Repeat allowed, unknown-caller, plaintext, direct pod-IP, application-token,
   and normal deployment tests.
7. Hold the canary for an agreed observation window long enough to cover normal
   browser and automated action traffic. Any unexplained connectivity error,
   identity mismatch, CNI/sidecar restart, or missing telemetry fails the gate.

### Phase 5: Staging enforcement

Set staging `security_profile: enforced`, add the GitOps
`security-enforcement` components, and enforce one destination path at a time:

1. Web gateway → web.
2. Web → gitlab-root-actor.
3. Waldur order and membership-sync → gitlab-root-actor.
4. Seeder → gitlab-root-actor.

Because the gitlab-root-actor policy contains all three allowed principals, all
three caller classes must already be sidecar-injected and positively tested before that
policy becomes enforcing. Report, component-loading, and arbitrary pod negative
tests run immediately afterward.

### Rollback

Rollback is scoped to the last successful phase and performed in reverse order.

#### Backend policy or enrollment rollback

1. Capture sidecar, application, policy, pod, and NetworkPolicy evidence.
2. Change `security_profile` to `permissive` and remove the enforcement
   component, which removes the destination AuthorizationPolicy and strict
   PeerAuthentication before enrollment changes.
3. Verify plaintext/current traffic is restored.
4. If a complete rollback is required, change `security_profile` to `open`,
   remove the enrollment component, and roll the affected pod templates.
5. Revert ServiceAccount changes only after no active policy refers to the new
   principal.

Removing sidecar injection before strict mTLS or allow policy would leave the
caller without an authenticated principal and can prolong the outage.

#### Gateway policy rollback

Change the affected environment to `permissive` or `open`, reconcile the native
gateway `ALLOW` policy to `absent`, and wait for propagation. Do not restore an
empty `rules: [{}]` policy. If the ServiceAccount caused the issue, revert it
only after no backend trusts the new gateway principal. Preserve the
unexpected-host test result for diagnosis.

#### Sidecar control-plane or ownership rollback

1. Remove backend strict-mTLS and `ALLOW` policies.
2. Remove explicit application sidecar labels and roll the selected workloads.
3. Verify application traffic is restored before changing CNI or istiod.
4. Revert the shared gateway and control-plane values.
5. Restore the Kubeflow Istio install overlay only if Helm ownership itself is
   being abandoned, then apply the backed-up Kubeflow and Istio state.
6. Re-run all pre-rollout service and authentication tests.

Changing the sidecar injector or CNI can affect pod startup cluster-wide.
Production rollback therefore uses the same maintenance and node-safety controls
as rollout.

## Open Questions

1. ~~Injecting the KServe predictor (section 9) requires the predictor's
   `workspace-init` egress to succeed while the pod is in the mesh. With a
   classic sidecar under Istio CNI this fails, because redirection is active
   during the init phase before Envoy starts, and
   `holdApplicationUntilProxyStarts` does not cover init containers.~~
   **Resolved:** istiod is installed with `ENABLE_NATIVE_SIDECARS=true` (section
   2.2), so the Envoy proxy starts as a native sidecar before the pod's app init
   containers on the cluster's Kubernetes 1.34, and the predictor's init egress
   is captured by a running proxy. The native-sidecar switch is a mesh-wide
   change and must pass the standard sidecar-startup and regression gates for
   every already-injected workload before promotion.

Otherwise there are no unresolved design questions in this DD. Runtime
compatibility and canary outcomes are explicit promotion gates; a failed gate
keeps ADR-0025 Draft and triggers rollback rather than an undocumented design
variation.

## References

- [ADR-0025: Scoped Istio Authorization Policies](/adr/0025-scoped-istio-authorization-policies)
- [ADR-0024: Baseline L3/L4 Network Segmentation](/adr/0024-baseline-l3-l4-network-segmentation)
- [Istio: Install with Helm](https://istio.io/latest/docs/setup/install/helm/)
- [Istio: Sidecar injection](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/)
- [Istio: Install the CNI node agent](https://istio.io/latest/docs/setup/additional-setup/cni/)
- [Istio: AuthorizationPolicy reference](https://istio.io/latest/docs/reference/config/security/authorization-policy/)
- [Istio: PeerAuthentication reference](https://istio.io/latest/docs/reference/config/security/peer_authentication/)
- [Istio: Verify mutual TLS](https://istio.io/latest/docs/tasks/security/authentication/authn-policy/#verify-mutual-tls-configuration)
- [Istio: Debugging Envoy and istiod](https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/)
- [Istio: Prometheus integration](https://istio.io/latest/docs/ops/integrations/prometheus/)
- [Calico: eBPF and service meshes](https://docs.tigera.io/calico/latest/operations/ebpf/use-cases-ebpf)