Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0025: Scoped Istio Authorization Policies — Implementation

Related ADR: ADR-0025: Scoped Istio Authorization Policies

Status: Draft

Date: 2026-07-20

Authors: TBD

#ADR-derived technical notes

ADR-0025 selects scoped ingress host allowlists and scoped Istio sidecar injection. The first internal policies authorize Kubernetes ServiceAccount identities and workload destination ports only. Envoy sidecars can enforce L7 attributes, but HTTP methods, paths, headers, request principals, CUSTOM, and AUDIT conditions remain outside the initial backend policies.

Repository inspection found these implementation constraints:

• istio_version is the common pin for Helm charts, Kubeflow Istio images, and manually rendered gateway proxy images.
• istio-deploy currently installs istio-base, istiod, and a gateway with Helm, but it only patches an Istio CNI DaemonSet if another deployment already created one. Kubeflow's rendered kustomization still includes an overlapping Istio installation overlay.
• The Themelio web and Argo CD gateways are GitOps-owned. The GitLab, Waldur, image registry, MLflow, monitoring, Ceph dashboard, Kubeflow, and Themelio workload gateways are Ansible-owned.
• The ingress ALLOW policies for web, Argo CD, GitLab, Waldur, the image registry, MLflow, Grafana, Prometheus, Ceph, and the workload gateway contain an empty rule and therefore match every request.
• web and gitlab-root-actor currently run as the default ServiceAccount in themelio. k8s-root-actor already runs as k8s-root-actor-sa in every deployed overlay.
• The Waldur order, membership-sync, report, and component-loading workloads all use waldur-site-agent-cloud. Trusting that identity would trust more than the two GitLab-sync processes named in ADR-0025.
• The seeder still calls gitlab-root-actor. It disables ServiceAccount token automount but does not currently select a dedicated ServiceAccount.
• Local and staging have been reachable during implementation. Earlier staging evidence was collected from the superseded Ambient pilot and is not accepted as sidecar evidence; every mesh-specific gate must be rerun after migration. Dev and production runtime evidence remain promotion criteria.

The related ADR-0024 (historical reference unavailable) currently lives on a separate documentation branch. It must be merged before the published ADR-0025 and DD-0025 links are expected to resolve. Its NetworkPolicy controls are a rollout prerequisite, not duplicated ownership in this DD.

#Overview

This document specifies how Themelio will consolidate Istio component ownership under Ansible and Helm, restrict every deployed ingress gateway to its configured HTTP host, give gateways and selected applications unambiguous workload identities, and incrementally protect the ADR-0025 internal flows with sidecar mTLS and workload authorization. Local and dev remain open by default; staging and production use enforced security after their canary gates pass. No application API, database schema, or user-facing action contract changes.

#Goals and Non-Goals

#Goals

• Make Helm, driven by istio-deploy, the single owner of Istio base, istiod, CNI, and the shared Kubeflow ingress gateway.
• Remove every empty gateway ALLOW rule and derive allowed hosts from the same value used by the gateway and its VirtualService.
• Give every ingress gateway a unique ServiceAccount without changing its external listener, route, or authentication behavior.
• Protect the selected web, root-actor, site-agent, gateway, and seeder flows with mTLS-authenticated ServiceAccount identity and destination workload port.
• Inject individual pod templates rather than whole application namespaces.
• Scope each tenant (Kubeflow profile) namespace so a Themelio deployment or inference service is reachable only through the service ingress gateway identity, with no namespace-wide ALLOW rule.
• Preserve external authorization, application authentication, user RBAC, NetworkPolicy, and service-specific credentials as independent controls.
• Produce repeatable validation and rollback procedures for every security profile.

#Non-Goals

• Namespace-wide sidecar injection or automatic injection into tenant workloads.
• A root-namespace deny-all AuthorizationPolicy.
• HTTP method, path, header, JWT, or new traffic-routing policy at application sidecars.
• Mesh enrollment of PostgreSQL, Valkey, GitLab, or arbitrary user deployments.
• Replacing Calico, enabling its eBPF dataplane, or removing kube-proxy.
• Changing REST, GraphQL, action, database, or user authorization contracts.
• Claiming the ADR is accepted before the canary and promotion evidence exists.

#Design

#1. Security profiles and ownership boundaries

security_profile controls only ADR-0025 sidecar injection and native ALLOW/strict-mTLS enforcement:

The environment assignments are explicit:

permissive is a transitional rollout and rollback profile, not a permanent environment class. Dedicated ServiceAccounts, Istio CNI, sidecar-capable istiod, Gateway and VirtualService routing, application authentication, user RBAC, service credentials, and existing CUSTOM external authorization remain present in all profiles.

Ansible reads security_profile from each environment's inventory/<environment>/group_vars/all.yml. GitOps cannot consume Ansible inventory directly, so matching Kustomize components implement the same state:

• no security component for open,
• security-enrollment only for permissive, and
• both security-enrollment and security-enforcement for enforced.

Static validation must compare the inventory value with the rendered GitOps resources and fail if they disagree. Local/dev overlays omit both components; staging includes both. Any future production Themelio or Argo CD overlay must include both before it is registered by the production root application. Other environment differences remain data such as dns.*, node selectors, tolerations, replica counts, and secret values.

Resource ownership after this change is:

Tenant roles must not be able to edit these ServiceAccounts, gateway or backend AuthorizationPolicies, PeerAuthentications, enrollment labels, or platform-owned NetworkPolicies.

#2. Sidecar-capable Istio installation

istio-deploy will expose and validate these variables:

Every chart uses istio_chart_version, which defaults to istio_version. Standalone gateways continue using docker.io/istio/proxyv2:{{ istio_version }}. A render or deployment must fail when those versions differ.

The Helm sequence is:

1. Upgrade or install istio-base.
2. Upgrade or install istiod with the chart's default sidecar values and the existing resource, placement, mesh config, trust domain, and extension-provider settings.
3. Upgrade or install istio-cni with Ambient disabled on every schedulable Kubernetes node.
4. Reconcile the shared Kubeflow gateway with its existing service ports, NodePorts, selectors, and dedicated ServiceAccount.
5. Reconcile standalone gateway proxy images only after the control plane is available.

The Helm operations remain atomic. The role waits for the istiod Deployment and CNI DaemonSet to report all desired pods ready. CNI installation does not inject a proxy by itself; injection is selected at each pod template.

Kubespray sets fs.inotify.max_user_instances=1024 through additional_sysctl. The kernel default of 128 was exhausted on a sidecar-heavy staging node and prevented the Istio CNI agent from creating its CNI-configuration watcher during an upgrade.

#2.1 Kubeflow ownership migration

The Kubeflow kustomization stops including ../common/istio/istio-install/overlays/oauth2-proxy. It also drops the image rewrites for pilot, proxyv2, and install-cni and the istiod Deployment/HPA replica patches. It retains:

• ../common/istio/cluster-local-gateway/base,
• ../common/istio/kubeflow-istio-resources/base,
• Kubeflow Gateway and VirtualService resources,
• the custom OAuth2 proxy resources and host-scoped CUSTOM policy, and
• the existing Ansible-managed Istio extensionProviders mesh configuration.

Migration order matters:

1. Render the future Kubeflow kustomization and verify it no longer declares Istio base, istiod, the main ingress gateway, or the CNI DaemonSet.
2. Let Helm adopt matching live resources with --take-ownership while the old Kubeflow apply path is still available for rollback.
3. Verify Helm owns all four releases and the current Kubeflow gateway remains ready and routable.
4. Apply the reduced Kubeflow kustomization. Server-side apply must not regain ownership of fields managed by Helm.
5. Remove the old istio-ingress Helm gateway only after queries show that no Istio Gateway selector, Kubernetes Service, or traffic path uses it.

The operator must stop if Helm adoption would replace a Service ClusterIP, NodePort, selector, or ServiceAccount rather than preserving it.

#2.2 Scoped sidecar injection

Existing Envoy gateways remain standalone router proxies. Selected application Deployments and Jobs receive sidecar.istio.io/inject: "true" in spec.template.metadata.labels. No Themelio or Waldur namespace receives istio-injection=enabled, so unrelated pods remain outside the mesh. Istio CNI performs traffic redirection without a privileged init container, while the injected Envoy remains visible as istio-proxy in pod status and telemetry.

istiod is installed with pilot.env.ENABLE_NATIVE_SIDECARS=true (the istio_enable_native_sidecars default in the istio-deploy role), so the Envoy proxy is injected as a Kubernetes native sidecar - an init container with restartPolicy: Always — supported by the cluster's Kubernetes 1.34. The proxy therefore starts before a pod's app init containers, so init-time egress is captured by a running Envoy instead of failing against traffic redirection that is active before a classic sidecar would start. This is what makes injecting the KServe predictor (section 9), whose workspace-init container clones the model over git/LFS, safe.

#3. Ingress host authorization

An ingress host policy is an L7 policy enforced by that gateway's Envoy proxy. It uses the HTTP Host or HTTP/2 :authority value and does not validate DNS resolution.

Every deployed gateway receives one scoped ALLOW policy:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <gateway-name>
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: <gateway-name>
  action: ALLOW
  rules:
    - to:
        - operation:
            hosts:
              - "<configured-host>"
              - "<configured-host>:*"

The second host form is required because clients and reverse proxies may include an explicit port in Host. No wildcard domain is introduced unless the Gateway and VirtualService already use that same wildcard.

istio-ingressgateway-service-account is deliberately retained for Kubeflow. The Waldur Kubeflow client already writes that principal into per-user policies. The Themelio workload gateway must move away from that shared identity.

Existing CUSTOM policies for Kubeflow and MLflow remain present. Istio evaluates CUSTOM before native ALLOW; therefore a request must pass external authorization when its CUSTOM rule applies and must also match the gateway host allowlist in the enforced profile. An unknown host then matches no ALLOW rule and returns 403. In open and permissive, the ADR-0025 gateway ALLOW policy is absent; the existing CUSTOM and application controls still apply.

Host rules are generated from the same input as the matching Gateway and VirtualService:

• Kustomize overlays patch the Gateway, VirtualService, and AuthorizationPolicy with one environment host value.
• Ansible templates reference the existing dns.* value in all three resources.
• The Themelio service gateway spans ownership boundaries: Ansible renders the gateway host from dns.service, while each GitOps overlay sets the same environment-specific hostname directly as SERVICE_PUBLIC_HOST on k8s-root-actor. No routing ConfigMap is used. Render validation must compare these two values and fail on any mismatch before promotion.
• A role does not render the policy when its gateway is disabled or its host is undefined; it fails validation if the gateway is enabled without a host.
• Ansible reconciles the native host ALLOW policy to absent unless security_profile is enforced; merely skipping a task is prohibited because it could leave a stale policy active.

Gateway readiness port 15021, Envoy metrics, and NodePort behavior must be tested after each policy. They are not added as public host exceptions.

#4. Workload identities

The target identities and SPIFFE principals are:

ServiceAccounts are declared in the same desired-state package as their workloads. A Deployment changes identity by setting serviceAccountName; this causes a normal rollout. automountServiceAccountToken: false remains on the seeder because application access to the Kubernetes API is unnecessary. Workload identity comes from the certificate issued to its injected sidecar and does not require exposing a Kubernetes token inside the seeder container.

The Waldur order and membership-sync Deployments move to waldur-site-agent-cloud-gitlab-sync. Its ClusterRoleBinding contains this new subject so existing Kubernetes reconciliation behavior continues. Report and component-loading retain waldur-site-agent-cloud and are not included in the GitLab root actor allowlist. This split is mandatory before trusting the sync principal.

#5. Sidecar injection and backend policy

In permissive and enforced, the enrollment component places sidecar.istio.io/inject: "true" only on these pod templates:

• web
• k8s-root-actor
• gitlab-root-actor
• Waldur order process
• Waldur membership-sync process
• themelio-seeder

The web ingress gateway is already an Istio proxy and does not receive an application sidecar.

In open, none of these pod templates requests injection. In permissive, they receive sidecars but the backend AuthorizationPolicies and workload-scoped strict PeerAuthentications are absent. In enforced, the same injection is combined with the policies below.

Backend AuthorizationPolicy resources live beside the destination workload. The ports values are destination workload ports, not Kubernetes Service ports. The three Themelio applications listen on workload port 3000 even though their Services expose port 80.

#5.1 k8s-root-actor canary policy

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: k8s-root-actor-allow-web
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: k8s-root-actor
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/themelio/sa/web
      to:
        - operation:
            ports: ["3000"]
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: k8s-root-actor-strict-mtls
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: k8s-root-actor
  mtls:
    mode: STRICT

#5.2 web policy

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: web-allow-ingress-gateway
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: web
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account
      to:
        - operation:
            ports: ["3000"]
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: web-strict-mtls
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: web
  mtls:
    mode: STRICT

#5.3 gitlab-root-actor policy

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: gitlab-root-actor-allow-callers
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: gitlab-root-actor
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/themelio/sa/web
              - cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync
              - cluster.local/ns/themelio/sa/themelio-seeder
      to:
        - operation:
            ports: ["3000"]
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: gitlab-root-actor-strict-mtls
  namespace: themelio
spec:
  selector:
    matchLabels:
      app: gitlab-root-actor
  mtls:
    mode: STRICT

The policies intentionally contain no L7 fields. The destination Envoy sidecar is the enforcement point and authenticates the original caller's sidecar certificate. Adding HTTP methods, paths, or headers is technically possible in sidecar mode but requires a new or revised DD defining stable L7 contracts.

Application authorization is unchanged. For example, a pod with an allowed workload identity still needs the configured root-actor bearer token where the endpoint requires one. Workload identity does not grant a human permission.

#6. NetworkPolicy contract with ADR-0024

Before any pod receives a sidecar, ADR-0024 policy must preserve:

• existing application traffic at Kubernetes Service and workload ports,
• DNS and control-plane traffic needed by the workload and Istio proxy,
• kubelet health probes to the workload, including Istio probe rewriting, and
• Prometheus access to the Envoy metrics port where sidecar scraping is enabled.

No HBONE port 15008 exception is required for this design. Sidecar-to-sidecar mTLS uses the application's normal destination port.

The pilot runs on the current standard Calico dataplane. If a later ADR-0024 rollout enables Calico eBPF, its connect-time load balancing must remain disabled before sidecars are promoted. The combined conformance suite must cover ClusterIP, NodePort, DNS, host-to-Service, pod-to-Service, Gateway/VirtualService routing, failure recovery, and rollback.

#7. End-to-end request paths

The bastion-to-gateway hop remains plain HTTP on the private network as recorded in ADR-0025. The gateway-to-backend and selected internal flows use Envoy sidecar mTLS. Calls from injected workloads to destinations outside the mesh, such as GitLab, PostgreSQL, or Valkey, retain their existing transport and application security behavior.

#8. Observability

Sidecar telemetry is collected separately from application telemetry:

• Prometheus scrapes istiod, Envoy sidecars, and Envoy gateway metrics explicitly.
• Mesh configuration selects the built-in Prometheus metrics provider through defaultProviders.metrics.
• Application metrics continue to use their existing scrape configuration. Application/sidecar metric merging is not assumed.
• The monitoring role adds an istio-workload-sidecars PodMonitor selected by security.istio.io/tlsMode=istio.
• Existing Alloy pod-log collection forwards gateway and istio-proxy logs to Loki.
• Grafana panels show sidecar scrape readiness, mTLS request/connection counts, TCP bytes, source and destination principals, and authorization denials where the exported labels support them.

Alerts and runbooks distinguish:

The rollout checklist records links or captured output for each of these signals.

#9. Tenant-namespace workload authorization

Kubeflow profile namespaces host tenant workloads. The Themelio deployments and KServe inference services that k8s-root-actor creates must be reachable from the shared Themelio service ingress gateway, but the namespace must not carry a namespace-wide ALLOW rule (rules: [{}]) that would admit every source. Instead k8s-root-actor generates one workload-scoped AuthorizationPolicy per deployment and per inference service, and deletes it with the workload.

The following Kubeflow-generated policies are retained unchanged and are not replaced by this design:

• ns-owner-access-istio — the namespace owner through the Kubeflow gateway, ml-pipeline-ui traffic carrying the owner identity, same-namespace workload traffic, health and metrics endpoints, and notebook-controller access to notebook kernels.
• every user-<email>-clusterrole-edit policy — each contributor through the Kubeflow gateway and ml-pipeline-ui only when the kubeflow-userid header matches that contributor.

Generated with every Themelio k8s-deployment (k8s-root-actor template k8s-deployment/authorizationpolicy.yaml):

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <deployment-name>-service-gateway
  namespace: <tenant>
spec:
  selector:
    matchLabels:
      app: <deployment-name>
      component: themelio-k8s-deployment-runner
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account
      to:
        - operation:
            ports:
              - "<configured-port-or-8000>"

Generated with every k8s-inferenceservice (k8s-root-actor template k8s-inferenceservice/authorizationpolicy.yaml). The selector targets the KServe-generated predictor pod labels:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <inferenceservice-name>-service-gateway
  namespace: <tenant>
spec:
  selector:
    matchLabels:
      serving.kserve.io/inferenceservice: <inferenceservice-name>
      component: predictor
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account
      to:
        - operation:
            ports:
              - "<configured-port-or-8000>"

• <configured-port-or-8000> is the workload's configured port, defaulting to 8000 — the same default the Deployment container port, the predictor --http_port, and the Service targetPort already use.
• The admitted principal is the Themelio service gateway ServiceAccount, read from k8s-root-actor's SERVICE_GATEWAY_PRINCIPAL config (default cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account).
• Each policy is applied in the same concurrent tier as the workload it selects and is deleted by the matching teardown path (destroyK8sDeployment, destroyK8sInferenceService), so it never outlives its workload.

Enforcement prerequisite: Istio enforces an AuthorizationPolicy at the selected pod's sidecar. The inference predictor template therefore sets sidecar.istio.io/inject: "true" (with holdApplicationUntilProxyStarts on the model-server container), so the <name>-service-gateway policy is enforced. The deployment-runner pod carries no explicit injection annotation, so its policy is enforced only where the tenant namespace already injects the pod; where it does not, the policy is inert (fail-open) but harmless until the runner joins the mesh.

Injecting the predictor depends on the mesh running native sidecars: the predictor's workspace-init container performs apt/curl/git+LFS egress during the init phase, which a classic (non-native) sidecar would break because traffic redirection is active during init while Envoy is not yet running (holdApplicationUntilProxyStarts does not cover init containers). istiod is installed with ENABLE_NATIVE_SIDECARS=true (section 2.2), so the proxy starts before the init container and this egress succeeds.

No tenant policy is generated for two workload classes:

• slurm-job — its launcher and cancellation Jobs run in the fixed themelio namespace, not a tenant namespace.
• the Waldur site agent — it mutates tenant resources through the Kubernetes API, so its access is governed by Kubernetes RBAC, not Istio.

The result preserves every currently implemented tenant workflow — Kubeflow owner and contributor access, notebooks, pipelines, health checks, tenant-internal communication, Themelio deployment and inference reachability through the dedicated gateway, and Waldur resource management — while removing namespace-wide unrestricted access. No tenant namespace retains a rules: [{}] policy, and requests from unapproved external identities are denied.

#Data Model

No persistent application data changes. The security configuration itself is a versioned contract:

ServiceAccount names and their SPIFFE principals are security-sensitive public deployment interfaces. Renaming one without updating and revalidating every policy that trusts it is a breaking change.

#API Design

No Themelio REST, GraphQL, database, or action API changes.

The Kubernetes API surface added or changed by this design consists of:

• ServiceAccount resources and Deployment/Job serviceAccountName fields,
• pod-template sidecar.istio.io/inject: "true" labels,
• scoped security.istio.io/v1 AuthorizationPolicy resources,
• workload-scoped security.istio.io/v1 PeerAuthentication resources, and
• Helm releases and values for Istio base, istiod, CNI, and the shared gateway,
• Kustomize security-enrollment and security-enforcement components selected by the environment overlay to match security_profile.

All backend AuthorizationPolicies use workload selectors and are enforced by destination sidecars. Existing RequestAuthentication and CUSTOM AuthorizationPolicy contracts remain unchanged.

#Testing Strategy

#Static and rendered-manifest tests

Run before connecting to a cluster:

1. Build every Themelio, Argo CD, and Waldur site-agent Kustomize overlay.
2. Render the affected Ansible templates for open, permissive, and enforced.
3. Run repository YAML checks, Ansible syntax checks, and Ansible lint.
4. Run helm template for base, default sidecar istiod, sidecar-only CNI, and the shared gateway at the pinned istio_version.
5. Run offline istioctl analyze over the composed Istio and application manifests.
6. Assert that no gateway ALLOW policy contains rules: [{}], rules: - {}, or an omitted host operation.
7. Assert that no gateway Deployment uses default, istiod, istio-ingress, or another gateway's ServiceAccount.
8. Assert that local and dev render no sidecar injection labels, native ADR-0025 ALLOW policies, or strict PeerAuthentications.
9. Assert that permissive renders selected sidecar injection labels but no native ADR-0025 ALLOW policies or strict PeerAuthentications.
10. Assert that enforced renders enrollment plus all expected policies.
11. Assert that all backend AuthorizationPolicies contain only principals and ports; Themelio backends use 3000.
12. Assert that only the selected application pod templates carry the explicit sidecar injection label.
13. Assert that the Waldur report and load-components workloads do not use the GitLab-sync ServiceAccount.
14. Assert that the rendered k8s-root-actor tenant AuthorizationPolicy templates select the workload (deployment-runner or KServe predictor) labels, admit only the service gateway principal on the workload port, and contain no rules: [{}] or namespace-wide rule.

#Component and ownership validation

helm list -n istio-system
kubectl -n istio-system get deploy,daemonset,service,serviceaccount
kubectl -n istio-system rollout status deployment/istiod
kubectl -n istio-system rollout status daemonset/istio-cni-node
istioctl analyze --all-namespaces

Validation requires one ready CNI pod on every schedulable node, no ztunnel DaemonSet, healthy istiod replicas, unchanged gateway Service and NodePort values, and no ownership or analyzer warnings relevant to the rollout.

#Gateway policy matrix

For each deployed gateway, send requests directly to its HTTP NodePort so DNS and the bastion do not mask the policy:

curl -i -H 'Host: <configured-host>' http://<node-ip>:<node-port>/
curl -i -H 'Host: <configured-host>:<node-port>' http://<node-ip>:<node-port>/
curl -i -H 'Host: rejected.invalid' http://<node-ip>:<node-port>/

The first two requests must reach the normal application or authentication response. The last must return gateway 403, not merely an application 404. Gateway readiness, metrics scraping, and the normal bastion URL must remain healthy. Repeat after each gateway policy rather than batching all gateways.

#End-to-end validation cases

Regression validation also covers normal browser login and navigation, project APIs, Git clone/push/LFS through the web gateway, deployment actions, Waldur reconciliation, the seeding workflow, readiness probes, external authorization, Prometheus scraping, and Loki log discovery.

The operator commands, expected results, evidence fields, and cleanup steps are maintained in the ADR-0025 validation document.

#Definition of done

• All static and render checks pass for open, permissive, and enforced.
• open and permissive contain no ADR-0025 native ALLOW or strict-mTLS policies; open also contains no selected application sidecars.
• enforced contains no empty gateway allow rules in desired state or the live cluster.
• Every gateway and trusted application uses its specified unique identity.
• Positive and negative identity tests pass for every protected destination.
• Strict mTLS rejects plaintext and non-mesh direct access.
• Sidecar mTLS, expected principals, metrics, and logs are observable.
• Application authentication and user RBAC regression tests pass.
• Rollback from enforced through permissive has been exercised in staging at least once.
• ADR-0025 stays Draft until the recorded canary evidence is reviewed.

#Deployment & Rollout

#Phase 0: Preconditions and backup

1. Merge ADR-0024 so the dependency and links resolve.
2. Confirm the current Calico dataplane; stop if eBPF connect-time load balancing is enabled.
3. Inventory every deployed gateway, attached Gateway/VirtualService, Service, NodePort, ServiceAccount, AuthorizationPolicy, PeerAuthentication, RequestAuthentication, mesh telemetry configuration, namespace label, and Helm release.
4. Export these resources and current Helm values to an operator-controlled backup outside the cluster.
5. Record baseline success rates, latency, gateway 4xx/5xx, root-actor calls, and normal product smoke-test results.
6. Ensure ADR-0024 preserves application ports, health probes, DNS, control-plane access, and Envoy metrics scraping.

#Phase 1: Consolidate ownership and install the sidecar control plane

1. Render the reduced Kubeflow desired state.
2. Upgrade base and istiod with Helm ownership and the default profile.
3. Adopt or install the default-profile CNI release.
4. Adopt the shared Kubeflow gateway without changing its Service contract.
5. Apply the reduced Kubeflow desired state.
6. Run component, Kubeflow, gateway, external-authz, browser, and Git regression checks. No selected application receives a sidecar in this phase.

#Phase 2: Reconcile gateway profile

Local and dev remain open: reconcile every ADR-0025 native gateway ALLOW policy to absent and verify their normal routing plus independent application and CUSTOM authorization behavior.

For one gateway at a time in the staging enforced promotion:

1. Create its dedicated ServiceAccount and update the gateway Deployment.
2. Wait for the rollout and confirm its certificate principal.
3. Replace the empty ALLOW rule with the configured host and host-with-port.
4. Run the three-request gateway test and its full service smoke test.
5. Check Envoy RBAC logs and metrics before moving to the next gateway.

The phase completes only when all gateways deployed in that environment have scoped policies. A gateway absent from an environment is recorded as not deployed, not treated as a failed test.

#Phase 3: Establish application identities

1. Create web, gitlab-root-actor, themelio-seeder, and waldur-site-agent-cloud-gitlab-sync ServiceAccounts.
2. Add the Waldur sync identity to the required RBAC binding.
3. Move Waldur order and membership-sync to the sync identity; leave report and component-loading on their existing identity.
4. Update web and gitlab-root-actor; retain k8s-root-actor-sa.
5. Roll out each environment's k8s-root-actor overlay with its direct SERVICE_PUBLIC_HOST value and verify that it exactly matches the Ansible inventory's dns.service. Only after the rollout is healthy, remove any legacy themelio-service-routing ConfigMap from that cluster.
6. Update the seeder Job template while keeping token automount disabled.
7. Roll and smoke-test all long-running workloads. Do not apply backend allow or strict-mTLS policies yet.

#Phase 4: Staging permissive canary

1. Set staging security_profile: permissive and include only the GitOps security-enrollment components.
2. Label the web and k8s-root-actor pod templates for sidecar injection and roll them.
3. Confirm both workloads contain synchronized sidecars with the expected SPIFFE principals.
4. Test allowed and unapproved calls while the destination is permissive.
5. Confirm no native ADR-0025 ALLOW or strict PeerAuthentication exists.
6. Repeat allowed, unknown-caller, plaintext, direct pod-IP, application-token, and normal deployment tests.
7. Hold the canary for an agreed observation window long enough to cover normal browser and automated action traffic. Any unexplained connectivity error, identity mismatch, CNI/sidecar restart, or missing telemetry fails the gate.

#Phase 5: Staging enforcement

Set staging security_profile: enforced, add the GitOps security-enforcement components, and enforce one destination path at a time:

1. Web gateway → web.
2. Web → gitlab-root-actor.
3. Waldur order and membership-sync → gitlab-root-actor.
4. Seeder → gitlab-root-actor.

Because the gitlab-root-actor policy contains all three allowed principals, all three caller classes must already be sidecar-injected and positively tested before that policy becomes enforcing. Report, component-loading, and arbitrary pod negative tests run immediately afterward.

#Rollback

Rollback is scoped to the last successful phase and performed in reverse order.

#Backend policy or enrollment rollback

1. Capture sidecar, application, policy, pod, and NetworkPolicy evidence.
2. Change security_profile to permissive and remove the enforcement component, which removes the destination AuthorizationPolicy and strict PeerAuthentication before enrollment changes.
3. Verify plaintext/current traffic is restored.
4. If a complete rollback is required, change security_profile to open, remove the enrollment component, and roll the affected pod templates.
5. Revert ServiceAccount changes only after no active policy refers to the new principal.

Removing sidecar injection before strict mTLS or allow policy would leave the caller without an authenticated principal and can prolong the outage.

#Gateway policy rollback

Change the affected environment to permissive or open, reconcile the native gateway ALLOW policy to absent, and wait for propagation. Do not restore an empty rules: [{}] policy. If the ServiceAccount caused the issue, revert it only after no backend trusts the new gateway principal. Preserve the unexpected-host test result for diagnosis.

#Sidecar control-plane or ownership rollback

1. Remove backend strict-mTLS and ALLOW policies.
2. Remove explicit application sidecar labels and roll the selected workloads.
3. Verify application traffic is restored before changing CNI or istiod.
4. Revert the shared gateway and control-plane values.
5. Restore the Kubeflow Istio install overlay only if Helm ownership itself is being abandoned, then apply the backed-up Kubeflow and Istio state.
6. Re-run all pre-rollout service and authentication tests.

Changing the sidecar injector or CNI can affect pod startup cluster-wide. Production rollback therefore uses the same maintenance and node-safety controls as rollout.

#Open Questions

1. Injecting the KServe predictor (section 9) requires the predictor's workspace-init egress to succeed while the pod is in the mesh. With a classic sidecar under Istio CNI this fails, because redirection is active during the init phase before Envoy starts, and holdApplicationUntilProxyStarts does not cover init containers. Resolved: istiod is installed with ENABLE_NATIVE_SIDECARS=true (section 2.2), so the Envoy proxy starts as a native sidecar before the pod's app init containers on the cluster's Kubernetes 1.34, and the predictor's init egress is captured by a running proxy. The native-sidecar switch is a mesh-wide change and must pass the standard sidecar-startup and regression gates for every already-injected workload before promotion.

Otherwise there are no unresolved design questions in this DD. Runtime compatibility and canary outcomes are explicit promotion gates; a failed gate keeps ADR-0025 Draft and triggers rollback rather than an undocumented design variation.

#References

• ADR-0025: Scoped Istio Authorization Policies
• ADR-0024: Baseline L3/L4 Network Segmentation (historical reference unavailable)
• Istio: Install with Helm
• Istio: Sidecar injection
• Istio: Install the CNI node agent
• Istio: AuthorizationPolicy reference
• Istio: PeerAuthentication reference
• Istio: Verify mutual TLS
• Istio: Debugging Envoy and istiod
• Istio: Prometheus integration
• Calico: eBPF and service meshes