> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # ADR-0025: Scoped Istio Authorization Policies **Status:** Draft **Date:** 2026-07-16 **Deciders:** TBD **Related:** ADR-0024 Baseline L3/L4 Network Segmentation (referenced in the original record; absent from the archived tree) ## Context Themelio exposes platform applications through Envoy ingress gateways in `istio-system`. Several gateway AuthorizationPolicy resources use `action: ALLOW` with an empty rule. That rule matches every request and does not restrict access. Application workloads are not generally enrolled in the Istio mesh. The repository has no general sidecar injection or mesh-wide mTLS enforcement. The current ingress gateways can enforce request policy, but most backends cannot yet authenticate a calling workload through Istio. We want to protect both entry points and selected internal calls. Ingress policy will restrict configured HTTP hosts. Backend policy will use ServiceAccount-derived workload identity and mTLS so that an unapproved pod cannot call a protected service directly. An ingress host rule checks the HTTP Host or `:authority` value. It is not a DNS check and does not prove which DNS name the client resolved. ## Requirements 1. In `enforced`, ingress gateways must accept only their configured HTTP hosts. 2. In `enforced`, selected internal services must authenticate callers by workload identity over mTLS. 3. Every workload trusted by identity must use a dedicated Kubernetes ServiceAccount. 4. Authorization must start with caller identity and destination port. 5. HTTP method and path checks will be added only where they are stable and useful. 6. Platform automation must own mesh enrollment and security policy. 7. The design must avoid adding a sidecar to every workload by default. ## Considered Options ### Option A: Use Cilium service mesh Cilium provides service mesh and L7 policy capabilities. Its mutual authentication feature, which is required for authenticated workload identity in this ADR, is currently documented as beta and incomplete. Cilium is not selected for service-to-service authorization in this ADR. This does not mean that all Cilium networking or service mesh features are beta. The networking provider discussion remains separate in ADR-0024. ### Option B: Continue with Istio sidecars Inject an Istio sidecar into each selected caller and destination. Enforce strict mTLS at protected destinations and allow requests by ServiceAccount identity and destination port. A sidecar can also enforce HTTP methods and paths. This is mature and suitable for a small pilot. Each participating pod carries its own proxy, resource use, configuration, and upgrade work. The cost grows with every pod and becomes less attractive if many internal flows are added later. [Calico requires connect-time load balancing to be disabled so service mesh sidecars can see Kubernetes Service addresses](https://docs.tigera.io/calico/latest/operations/ebpf/use-cases-ebpf). This option can use the Calico eBPF dataplane, but not its connect-time load-balancing optimization. ### Option C: Use Istio ambient mode Install the Istio ambient components and enroll only the selected workloads. A ztunnel on each node provides transparent mTLS, ServiceAccount-derived identity, and L4 authorization without an application sidecar. Waypoints are not required for mTLS. They are added only where L7 policy or routing is needed, such as HTTP methods, paths, or headers. Existing ingress gateways can remain in place. This reduces per-pod proxy overhead and makes later enrollment easier. It adds ztunnel and optional waypoint operations. Ambient L7 AuthorizationPolicy is still beta, so the first internal policies should remain at identity and port level. This option is not selected. Kubeflow Profile and other HTTP authorization flows would require an additional waypoint and policy migration, while scoped sidecars preserve the existing enforcement model. ## Decision We propose Option B, scoped Istio sidecar injection. Only the callers and destinations named by this ADR receive a sidecar; namespace-wide injection is not introduced for Themelio or Waldur. The ADR remains Draft until the sidecar canary proves the current gateways, `VirtualService` resources, Calico networking, application startup, and operational tooling. Istio CNI remains installed so sidecars do not require a privileged `istio-init` container. Calico remains on its standard dataplane during the pilot. If Calico eBPF is enabled later, connect-time load balancing must remain disabled so the sidecars can observe Kubernetes Service addresses. **See also:** [DD-0025: Scoped Istio Authorization Policies — Implementation](/dd/0025-scoped-istio-authorization-policies.md) ### Security profiles ADR-0025 enrollment and enforcement are controlled by `security_profile`: | Profile | Sidecar injection | STRICT mTLS / ALLOW policies | | ------------ | ----------------- | ---------------------------- | | `open` | No | No | | `permissive` | Yes | No | | `enforced` | Yes | Yes | Local and dev use `open`; staging and production use `enforced` after their canary gates pass. `permissive` is the intermediate rollout and rollback state used to verify sidecar mTLS and workload identities before restrictive policies are applied. Dedicated ServiceAccounts and sidecar-capable Istio CNI infrastructure remain installed in every profile. Existing application authentication, user RBAC, service credentials, and `CUSTOM` external authorization are independent controls and are not disabled by `open`. ### Ingress policy In `enforced`, each ingress gateway will have a platform-owned `ALLOW` policy that selects only that gateway workload. It will allow the hosts configured on its Gateway and VirtualService, including the form with a port when clients may send one. Once an `ALLOW` policy selects a gateway, unmatched requests are denied at that gateway. The ADR-0025 native `ALLOW` policy is absent in `open` and `permissive`. Host policy is defense in depth. Gateway and VirtualService resources remain the source of truth for listeners and routing. Existing external authorization integrations will remain in place. In the current deployment, public HTTPS terminates at Nginx on the bastion. Nginx forwards plain HTTP over the private network to the gateway's HTTP NodePort and preserves the Host header. The gateway can therefore check the HTTP host, method, and path. For backend workloads added to the mesh, Istio mTLS protects traffic from the ingress gateway onward. We will not add one root namespace deny-all policy at first. A scoped `ALLOW` policy already denies unmatched requests at the selected gateway and limits the impact of a mistake. ### Internal workload policy The first scoped sidecar injection will cover these identities and flows: - the web ingress gateway to `web` - `web` to `k8s-root-actor` - `web` to `gitlab-root-actor` - the Waldur order process to `gitlab-root-actor` - the Waldur membership sync process to `gitlab-root-actor` - the seeder to `gitlab-root-actor` if that call remains The directional communication matrix for the selected application services is: | Services | Waldur | Web | k8s-root-actor | gitlab-root-actor | | ----------------- | ------ | --- | -------------- | ----------------- | | Waldur | | X | X | ✓ | | Web | X | | ✓ | ✓ | | k8s-root-actor | X | X | | X | | gitlab-root-actor | X | X | X | | Rows are callers and columns are destinations. A `✓` means ADR-0025 defines and tests that directional connection; `X` means it does not. Cells where the caller and destination are the same service are intentionally empty. A reverse call requires its own `✓`. The ingress-gateway → Web and conditional seeder → gitlab-root-actor infrastructure flows remain in the preceding list because they are not application services represented by this matrix. Application credentials and human authorization remain required after the workload-level connection is admitted. ```mermaid graph LR webgw([Web ingress gateway]):::gw -->|mTLS| web[web]:::dest web -->|mTLS| k8s[k8s-root-actor]:::dest web -->|mTLS| glra[gitlab-root-actor]:::dest order[Waldur order process]:::caller -->|mTLS| glra membership[Waldur membership sync]:::caller -->|mTLS| glra seeder[Seeder]:::caller -. mTLS conditional .-> glra report[Waldur report]:::deny -. denied .-> glra k8s -. denied .-> glra classDef gw fill:#e0e2fa,stroke:#4753c9,color:#142029 classDef dest fill:#d6ecec,stroke:#0c7a7a,color:#142029 classDef caller fill:#ffffff,stroke:#c2ccd4,color:#142029 classDef deny fill:#f5dfe3,stroke:#bd3f56,color:#142029 ``` Solid edges are allowed flows enforced by STRICT mTLS and ServiceAccount identity; dashed edges are denied or conditional. `web`, `k8s-root-actor`, and `gitlab-root-actor` are the protected destinations. `k8s-root-actor` additionally generates one scoped `-service-gateway` policy per tenant deployment and inference service, replacing the former namespace-wide `allow-all`: ```mermaid graph LR sgw([Themelio service gateway]):::gw -->|admitted by name-service-gateway| dep[Tenant deployment]:::wl sgw -->|admitted| pred[KServe predictor]:::wl same[Same-namespace caller]:::caller -->|ns-owner-access-istio| dep foreign[Foreign-namespace pod]:::deny -. 403 .-> dep foreign -. 403 .-> pred classDef gw fill:#e0e2fa,stroke:#4753c9,color:#142029 classDef wl fill:#d6ecec,stroke:#0c7a7a,color:#142029 classDef caller fill:#ffffff,stroke:#c2ccd4,color:#142029 classDef deny fill:#f5dfe3,stroke:#bd3f56,color:#142029 ``` Each generated policy admits only the Themelio service-gateway principal on the workload port, `ns-owner-access-istio` retains same-namespace access. Tenants cannot manage these policies ownership stays with platform automation. `k8s-root-actor` already has a dedicated ServiceAccount. `web`, `gitlab-root-actor`, the web ingress gateway, and the seeder will receive dedicated ServiceAccounts before their identities are trusted. The Waldur order and membership processes share the dedicated `waldur-site-agent-cloud-gitlab-sync` ServiceAccount because they have the same trust level. Report and component-loading retain their existing identity and remain denied from `gitlab-root-actor`. These identities will be split further if their allowed flows diverge. The web and ArgoCD gateways currently share the `istio-ingress` ServiceAccount. The GitLab, Waldur, image registry, Grafana, Prometheus, and Ceph dashboard gateways currently use the `istiod` ServiceAccount. Each gateway must receive its own ServiceAccount before a backend AuthorizationPolicy trusts that gateway identity. Every allowed caller and protected destination must be sidecar-injected before strict mTLS is enforced. Selected destinations will then enforce strict mTLS. Their initial `AuthorizationPolicy` rules allow only documented ServiceAccount identities on documented destination ports. Sidecars can enforce HTTP methods, paths, and headers, but this ADR intentionally starts with identity and port rules only. PostgreSQL and Valkey are not included in the first mesh enrollment. ADR-0024 NetworkPolicy, database or cache authentication, and application TLS will protect them. They can be reconsidered if there is a clear need for mesh identity or telemetry. Application authentication and user RBAC remain required. Istio workload identity answers which workload made a request, not which human user is authorized to perform an action. ### Ownership and rollout `AuthorizationPolicy`, `PeerAuthentication`, `RequestAuthentication`, and sidecar-injection labels will be owned through platform GitOps or Ansible. Tenant roles will not be able to manage them or change the labels and ServiceAccounts used for policy identity. Existing `NetworkPolicy` application ports remain required; no Ambient HBONE exception is introduced. Rollout will first replace empty ingress rules one gateway at a time. It will then assign the dedicated ServiceAccounts and run a sidecar canary for `web` to `k8s-root-actor`. The remaining internal flows will receive sidecars only after positive and negative tests pass. Each rollout must test an accepted request, an unknown host or caller, direct backend access from an unapproved pod, and the service's normal browser and API flows. `istioctl analyze`, Envoy sidecar logs/configuration, and policy metrics will be checked before promotion. ### Observability Istio monitoring will show where a request was rejected. Ingress Envoy access logs and metrics cover gateway and host policy decisions. Application Envoy sidecar logs and metrics cover mTLS identity and backend authorization failures. These signals will be sent to the existing Prometheus, Grafana, and Loki stack. Dashboards and alerts will distinguish gateway rejection, mTLS failure, workload identity denial, NetworkPolicy denial, and application failure where the available telemetry allows it. NetworkPolicy denials will use the flow visibility defined in ADR-0024. ## Validation status ADR-0025 is validated in staging one check at a time using the [ADR-0025 validation document](/product/operations/adr-0025-validation.md). A row may be changed to `✅ Passed` only after its command or manual procedure has been run and its non-secret evidence has been recorded. `⏳ Pending`, `⚠️ Blocked`, and `❌ Failed` keep this ADR in Draft. The conditional seeder row may be marked `➖ Not applicable` only if the seeder no longer calls `gitlab-root-actor`. | ID | ADR-0025 validation | Status | Evidence | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PRE-01 | Staging context and Istio/application components are healthy | ✅ Passed | Runtime is healthy and the three Argo CD applications reconciled to Synced after the sidecar rollback was committed and pushed | | PRE-02 | Enforced policies, sidecar injection, and dedicated identities match this ADR | ✅ Passed | staging:\~/adr-0025-validation/20260723T135550Z-sidecar | | GW-01 | Configured gateway hosts and host-with-port reach expected handling | ✅ Passed | All configured hosts reached normal handling, including the Space route on the Themelio service gateway | | GW-02 | Unknown gateway hosts return `403` | ✅ Passed | Every enforced gateway returned `403` for an unknown host | | FLOW-01 | Web ingress gateway → Web succeeds | ✅ Passed | Gateway-to-Web mTLS and successful handling recorded in sidecar metrics | | FLOW-02 | Web → k8s-root-actor succeeds | ✅ Passed | Real Web pod returned `200` | | FLOW-03 | Web → gitlab-root-actor succeeds | ✅ Passed | Real Web pod returned `200` | | FLOW-04 | Waldur order process → gitlab-root-actor succeeds | ✅ Passed | Real order-process pod returned `200` | | FLOW-05 | Waldur membership sync → gitlab-root-actor succeeds | ✅ Passed | Real membership-sync pod returned `200` | | DENY-01 | Waldur report → gitlab-root-actor is denied | ✅ Passed | Report connection was reset | | DENY-02 | Arbitrary Themelio identity → protected actors is denied | ✅ Passed | Both actor sidecars returned `403` | | DENY-03 | Plaintext and direct pod-IP access is rejected | ✅ Passed | All protected pod-IP calls were reset without an HTTP response | | FLOW-06 | Seeder → gitlab-root-actor succeeds if the call remains | ✅ Passed | Seeder sidecar returned `200` | | AUTH-01 | Application authentication remains required after mesh admission | ✅ Passed | Missing bearer token returned `401`; unauthenticated Web request redirected | | OBS-01 | Expected SPIFFE principals appear in Envoy sidecars | ✅ Passed | Proxies are subscribed and certificates are valid | | OBS-02 | Sidecar mutual TLS is confirmed | ✅ Passed | Prometheus contains the expected source and destination principals with `mutual_tls` | | OBS-03 | Authorization denials appear in sidecar logs and monitoring | ✅ Passed | Destination Envoy, Prometheus, and Loki contain the generated denials | | APP-01 | Browser and OIDC login still work | ✅ Passed | Operator-confirmed staging OIDC login | | APP-02 | Normal Themelio navigation still works | ✅ Passed | Operator-confirmed authenticated navigation | | APP-03 | Git clone, push, and Git LFS still work | ✅ Passed | Disposable repository and LFS lifecycle completed | | APP-04 | Kubernetes deployment actions still work | ✅ Passed | Space Deployment, Service, and VirtualService were created | | APP-05 | Waldur reconciliation still works | ✅ Passed | Reconciliation workloads are healthy and protected calls succeeded | | APP-06 | Kubeflow access and external authorization still work | ✅ Passed | Expected authentication handling and healthy Kubeflow sidecars observed | | APP-07 | Registry pull and push still work | ✅ Passed | Pull, disposable push, re-pull, and digest verification completed | | APP-08 | Grafana and Prometheus access still work | ✅ Passed | Operator confirmed both UIs and sidecar telemetry | | APP-09 | Readiness probes and ADR-0025 alerting remain healthy | ✅ Passed | Workloads and all CNI pods are ready; sidecar alerts are loaded and evaluating | | TENANT-01 | Per-workload scoped policies are generated with the correct selector, service-gateway principal, and port; no namespace-wide `allow-all` remains | ✅ Passed | Real Space policy carries the deployment-runner selector and service-gateway principal, `allow-all` is absent from the tenant namespace; inference selector confirmed via a GPU-free mock predictor | | TENANT-02 | Access from outside the tenant namespace is denied; same-namespace access remains allowed by `ns-owner-access-istio` | ✅ Passed | Foreign-namespace caller received `403` to the real Space Service and a mock predictor; same-namespace caller reached the app | | TENANT-03 | Authorized access through the Themelio service gateway reaches the tenant workload | ✅ Passed | Gateway route reached the real deployment and the mock predictor rather than returning an RBAC `403` | | TENANT-04 | Deleting a workload removes its `-service-gateway` policy with no orphan | ✅ Passed | Staging, 2026-07-24: deleting the Space from the UI removed its Deployment, Service, VirtualService, PVC, and `…-service-gateway` policy (`NotFound`); the tenant namespace retained only the Kubeflow `ns-owner-access-istio` and per-user policies | The application rows are limited to regression paths affected by ADR-0025 gateways, identities, mTLS, or observability. They are not a general platform release test suite. ## Consequences **Positive:** - Unknown HTTP hosts are rejected at the selected ingress gateway. - Existing OAuth and external authorization integrations remain in place. - Protected backends can reject direct calls from unapproved workloads. - Sidecars are injected only into the callers and destinations in scope. - Policy ownership stays with platform automation rather than tenant users. **Negative:** - A missing host or listener rule can return an unexpected 403 response. - The bastion-to-gateway hop remains plain HTTP on the private network. - Each selected pod carries an Envoy proxy with measurable CPU and memory cost. - Sidecar image/configuration upgrades roll every selected workload. **Neutral:** - These policies authenticate workloads, not human users. - NetworkPolicy remains required for L3 and L4 reachability. - Namespace-wide sidecar injection remains out of scope. ## References - [Istio Authorization Policy](https://istio.io/latest/docs/reference/config/security/authorization-policy/) - [Istio Sidecar Injection](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/) - [Istio CNI](https://istio.io/latest/docs/setup/additional-setup/cni/) - [Istio Envoy Access Logs](https://istio.io/latest/docs/tasks/observability/logs/access-log/) - [Istio Standard Metrics](https://istio.io/latest/docs/reference/config/metrics/) - [Istio Security Best Practices](https://istio.io/latest/docs/ops/best-practices/security/) - [Cilium Mutual Authentication](https://docs.cilium.io/en/stable/network/servicemesh/mutual-authentication/mutual-authentication/)
Page source Source snapshot · Revision `f928834c8`.