---
title: ADR-0025 Validation
---

# ADR-0025 validation

This document records the commands and manual checks used to validate
[ADR-0025](/adr/0025-scoped-istio-authorization-policies) in staging. Execute it
with two people or with an operator and an agent: run one numbered validation,
inspect its evidence, record the result, and only then continue. Do not execute
the whole document as an unattended script.

The validation covers only behavior required by ADR-0025: native Istio security
controls, proof that the protected application flows still work, and regression
checks for the gateways and applications affected by those controls. A failed
required validation keeps ADR-0025 in Draft.

## Safety rules

- Run against staging only. Confirm the context before every mutating test.
- Never paste PATs, passwords, cookies, or Kubernetes Secret values into this
  document or an evidence file.
- Commands marked **mutating** create disposable test resources. Stop and obtain
  operator confirmation before running them.
- Preserve relevant logs before deleting a failed test pod.
- Treat the commands in this document as the executable source of truth. Run
  them verbatim after replacing documented placeholders only. If a command is
  wrong or the environment requires a different command, stop, update this
  document first, and then run the corrected command.
- Treat an ingress-host rejection and a backend identity rejection differently:
  an unknown ingress host must return HTTP `403`; a sidecar authorization
  denial can appear as HTTP `403` or as a refused/reset connection depending on
  whether Envoy classified the destination listener as HTTP or TCP.

## Result values

Use one of these values in the result tables:

- `NOT RUN`
- `PASS`
- `FAIL`
- `BLOCKED`
- `NOT APPLICABLE`

## Validation checklist

| ID | Test | Required | Result | Evidence or notes |
| --- | --- | --- | --- | --- |
| PRE-01 | Staging context and component health | Yes | PASS | Cluster components are healthy and `staging-apps`, `themelio`, and `waldur-site-agent-cloud` reconciled to Synced/Healthy after the sidecar rollback was committed and pushed; `20260723T135550Z-sidecar/argocd-applications.txt` |
| PRE-02 | Enforced policy, sidecar, and identity inventory | Yes | PASS | Dedicated identities, selected sidecars, STRICT policies, and non-empty ingress rules verified; `20260723T135550Z-sidecar/{security-resources,themelio-identities,waldur-identities,no-empty-gateway-rules}.txt` |
| GW-01 | Configured gateway hosts and host-with-port reach expected handling | Yes | PASS | All configured hosts and host-with-port requests reached normal application/authentication handling, including the Space route on the Themelio service gateway; `20260723T135550Z-sidecar/gateway-host-results.tsv` |
| GW-02 | Unknown gateway hosts return `403` | Yes | PASS | Every enforced gateway returned `403` for `rejected.invalid`; `20260723T135550Z-sidecar/gateway-host-results.tsv` |
| FLOW-01 | Web gateway to Web | Yes | PASS | Gateway-to-Web mTLS principal and successful Web handling observed; `20260723T135550Z-sidecar/sidecar-mtls-requests.json` |
| FLOW-02 | Web to k8s-root-actor | Yes | PASS | Real Web sidecar returned `200`; `20260723T135550Z-sidecar/web-to-actors.txt` |
| FLOW-03 | Web to gitlab-root-actor | Yes | PASS | Real Web sidecar returned `200`; `20260723T135550Z-sidecar/web-to-actors.txt` |
| FLOW-04 | Waldur order process to gitlab-root-actor | Yes | PASS | Real order-process sidecar returned `200`; `20260723T135550Z-sidecar/waldur-sync-to-gitlab-root-actor.txt` |
| FLOW-05 | Waldur membership sync to gitlab-root-actor | Yes | PASS | Real membership-sync sidecar returned `200`; `20260723T135550Z-sidecar/waldur-sync-to-gitlab-root-actor.txt` |
| DENY-01 | Waldur report to gitlab-root-actor is denied | Yes | PASS | Non-injected report identity received a connection reset; `20260723T135550Z-sidecar/waldur-report-denial.txt` |
| DENY-02 | Arbitrary sidecar-injected Themelio identity is denied | Yes | PASS | Disposable identity received `403` from both actors; `20260723T135550Z-sidecar/arbitrary-identity-denials.txt` |
| DENY-03 | Plaintext/direct pod-IP access is rejected | Yes | PASS | All three direct calls were reset with no HTTP response; `20260723T135550Z-sidecar/plaintext-pod-ip-denials.txt` |
| FLOW-06 | Seeder to gitlab-root-actor | Conditional | PASS | Disposable seeder sidecar returned `200`; `20260723T135550Z-sidecar/seeder-to-gitlab-root-actor.txt` |
| AUTH-01 | Application authentication remains required | Yes | PASS | Actor returned `401` without a bearer token and Web redirected unauthenticated access; `20260723T135550Z-sidecar/{waldur-application-auth,web-session-auth}.txt` |
| OBS-01 | Expected SPIFFE identities appear in Envoy sidecars | Yes | PASS | All selected proxies are subscribed with valid certificates; `20260723T135550Z-sidecar/{sidecar-proxy-status,themelio-sidecar-state,waldur-sidecar-state}.txt` |
| OBS-02 | Sidecar mutual TLS is observed | Yes | PASS | Prometheus contains every expected mTLS principal pair; `20260723T135550Z-sidecar/sidecar-mtls-requests.json` |
| OBS-03 | Denials appear in sidecar logs and monitoring | Yes | PASS | Envoy, Prometheus, and Loki contain the generated `403` denials and principals; `20260723T135550Z-sidecar/sidecar-denials.txt`, `sidecar-denial-metrics.json`, and `loki-istio-denials.json` |
| APP-01 | Browser and OIDC login | Yes | PASS | Operator-confirmed staging OIDC login retained as valid regression evidence |
| APP-02 | Normal Themelio navigation | Yes | PASS | Operator-confirmed authenticated Themelio navigation retained as valid regression evidence |
| APP-03 | Git clone, push, and Git LFS | Yes | PASS | Disposable repository, push, LFS upload/download, and cleanup completed successfully |
| APP-04 | Kubernetes deployment action | Yes | PASS | The Space deployment created its Deployment, Service, and routed VirtualService successfully |
| APP-05 | Waldur order and membership reconciliation | Yes | PASS | Both live reconciliation processes are healthy and their protected actor calls returned `200` |
| APP-06 | Kubeflow and external authorization | Yes | PASS | Kubeflow gateway reached the expected authentication flow and its sidecar workloads remained healthy |
| APP-07 | Registry pull and push | Yes | PASS | Disposable registry pull/push/re-pull and digest verification completed successfully |
| APP-08 | Grafana and Prometheus access | Yes | PASS | Operator confirmed both UIs; sidecar, gateway, and istiod telemetry is available |
| APP-09 | Readiness probes and alert rules | Yes | PASS | Selected workloads and all 11 CNI pods are ready; sidecar alerting is loaded and evaluating; `20260723T135550Z-sidecar/istio-pods.txt`, `themelio-pods.txt`, `waldur-agent-pods.txt`, and `istio-alerts.json` |
| TENANT-01 | Per-workload scoped policies exist with the correct selector/principal/port and no namespace-wide `rules: [{}]` | Yes | PASS | Staging, 2026-07-24: real Space `spaces-…-service-gateway` generated with the deployment-runner selector, `istio-ingressgateway-themelio-service-account` principal, and the workload port; `allow-all` absent from `pharos`. Inference selector shape confirmed via a GPU-free mock predictor; generation on a live InferenceService is pending a GPU environment |
| TENANT-02 | Access from outside the tenant namespace to a tenant deployment/predictor is denied (predictor is native-sidecar injected so the policy is enforced); same-namespace access remains allowed by `ns-owner-access-istio` | Yes | PASS | Staging, 2026-07-24: foreign-namespace pod (`themelio`) denied `403` to the real Space Service and to a mock KServe predictor; same-namespace caller allowed (`404`, reached app) as expected via `ns-owner-access-istio` |
| TENANT-03 | Authorized access through the Themelio service gateway reaches the tenant deployment and predictor (policy admits the gateway identity) | Yes | PASS | Staging, 2026-07-24: through the service gateway the real deployment returned `404` (app response, not an RBAC `403`) and the mock predictor `200`; predictor confirmed native-sidecar injected so the policy is enforced |
| TENANT-04 | Deleting a workload from the UI removes its `<name>-service-gateway` policy with no orphan | Yes | PASS | Staging, 2026-07-24: deleting the Space from the UI removed its Deployment/Service/VirtualService/PVC and `…-service-gateway` policy (`NotFound`); no orphan; tenant namespace left with only the Kubeflow `ns-owner-access-istio` and per-user policies |

## 0. Connect and initialize evidence

Run the following from the repository workstation:

```bash
cd /home/dkats/themelio/devops/ansible
make show-env
```

Expected output: `staging`. Stop if any other environment is selected.

```bash
eval "$(ssh-agent -s)" \
  && ssh-add ~/.ssh/pharos-admin-key \
  && ssh -J admin@18.195.179.229 admin@10.0.100.44
```

Run the remaining cluster commands on the staging master unless a section says
otherwise. Create a directory for non-secret evidence:

```bash
export ADR0025_RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
export ADR0025_EVIDENCE="$HOME/adr-0025-validation/$ADR0025_RUN_ID"
mkdir -p "$ADR0025_EVIDENCE"
printf 'run_id=%s\nstarted_utc=%s\n' \
  "$ADR0025_RUN_ID" "$(date -u +%FT%TZ)" \
  | tee "$ADR0025_EVIDENCE/run.txt"
```

To resume an existing validation run in a new shell, replace the documented
placeholder with its run ID and execute:

```bash
export ADR0025_RUN_ID='<existing-run-id>'
export ADR0025_EVIDENCE="$HOME/adr-0025-validation/$ADR0025_RUN_ID"
test -d "$ADR0025_EVIDENCE"
```

## 1. Preconditions

### PRE-01: context and component health

```bash
kubectl config current-context | tee "$ADR0025_EVIDENCE/context.txt"
kubectl get nodes -o wide | tee "$ADR0025_EVIDENCE/nodes.txt"
kubectl get pods -n istio-system -o wide \
  | tee "$ADR0025_EVIDENCE/istio-pods.txt"
kubectl get pods -n themelio -o wide \
  | tee "$ADR0025_EVIDENCE/themelio-pods.txt"
kubectl get pods -n waldur-site-agent-cloud -o wide \
  | tee "$ADR0025_EVIDENCE/waldur-agent-pods.txt"
kubectl get applications.argoproj.io -n argocd \
  -o custom-columns='NAME:.metadata.name,REVISION:.spec.source.targetRevision,SYNC:.status.sync.status,HEALTH:.status.health.status' \
  | tee "$ADR0025_EVIDENCE/argocd-applications.txt"
```

Pass when:

- the context and node names are staging;
- every schedulable node has a ready Istio CNI pod and no ztunnel remains;
- istiod and all ingress gateways are ready;
- the selected Themelio and Waldur workloads have ready `istio-proxy` sidecars;
- the relevant Argo CD applications are healthy and at the intended revision.

### PRE-02: enforced policy and identity inventory

```bash
kubectl get authorizationpolicy,peerauthentication -A -o wide \
  | tee "$ADR0025_EVIDENCE/security-resources.txt"

kubectl get deploy -n themelio -o json \
  | jq -r '.items[] | [.metadata.name,
      (.spec.template.spec.serviceAccountName // "default"),
      (.spec.template.metadata.labels["sidecar.istio.io/inject"] // "false")]
      | @tsv' \
  | tee "$ADR0025_EVIDENCE/themelio-identities.txt"

kubectl get deploy -n waldur-site-agent-cloud -o json \
  | jq -r '.items[] | [.metadata.name,
      (.spec.template.spec.serviceAccountName // "default"),
      (.spec.template.metadata.labels["sidecar.istio.io/inject"] // "false")]
      | @tsv' \
  | tee "$ADR0025_EVIDENCE/waldur-identities.txt"

kubectl get authorizationpolicy -n istio-system -o json \
  | jq -e '[.items[]
      | select((.spec.selector.matchLabels.app // "")
          | test("ingressgateway"))
      | .spec.rules[]?
      | select(. == {})]
      | length == 0' \
  | tee "$ADR0025_EVIDENCE/no-empty-gateway-rules.txt"
```

Expected protected identities:

| Workload | ServiceAccount | Sidecar |
| --- | --- | --- |
| web | `web` | Yes |
| k8s-root-actor | `k8s-root-actor-sa` | Yes |
| gitlab-root-actor | `gitlab-root-actor` | Yes |
| Waldur order process | `waldur-site-agent-cloud-gitlab-sync` | Yes |
| Waldur membership sync | `waldur-site-agent-cloud-gitlab-sync` | Yes |
| Waldur report | `waldur-site-agent-cloud` | No |

Pass when the policies are present, selected workloads have the expected unique
ServiceAccounts, and no ingress `ALLOW` policy contains `rules: [{}]`.

## 2. Gateway host enforcement

### GW-01 and GW-02: test every rendered host policy

This command discovers the policies, gateway Services, configured hosts, and
ClusterIPs from the live cluster. It does not hard-code a second hostname
inventory. Calling the gateway Service directly avoids public DNS, the bastion,
and NodePort traffic-policy differences.

```bash
kubectl get authorizationpolicy -n istio-system -o json \
  | jq -r '.items[]
      | select([.spec.rules[]?.to[]?.operation.hosts[]?] | length > 0)
      | [.metadata.name,
         .spec.selector.matchLabels.app,
         .spec.rules[0].to[0].operation.hosts[0]]
      | @tsv' \
  | while IFS=$'\t' read -r policy app configured_host; do
      service="$(kubectl get service -n istio-system -l "app=$app" -o json \
        | jq -r '.items[] | select(.spec.type=="NodePort") | .metadata.name' \
        | head -n1)"
      service_ip="$(kubectl get service -n istio-system "$service" \
        -o jsonpath='{.spec.clusterIP}')"

      configured_code="$(curl -sS -o /dev/null -w '%{http_code}' \
        -H "Host: $configured_host" \
        --connect-timeout 5 --max-time 10 "http://$service_ip/" || true)"
      with_port_code="$(curl -sS -o /dev/null -w '%{http_code}' \
        -H "Host: $configured_host:80" \
        --connect-timeout 5 --max-time 10 "http://$service_ip/" || true)"
      rejected_code="$(curl -sS -o /dev/null -w '%{http_code}' \
        -H 'Host: rejected.invalid' \
        --connect-timeout 5 --max-time 10 "http://$service_ip/" || true)"

      printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
        "$policy" "$service@$service_ip" "$configured_host" \
        "$configured_code" "$with_port_code" "$rejected_code"
    done \
  | tee "$ADR0025_EVIDENCE/gateway-host-results.tsv"
```

Columns are policy, Service, configured host, configured-host response,
host-with-port response, and rejected-host response.

Pass when:

- each configured-host request reaches the normal application or
  authentication behavior;
- each host-with-port request behaves the same way;
- every `rejected.invalid` request returns `403`.

A configured host can legitimately produce an authentication redirect or an
application `401`/`403`. Record headers for any ambiguous response:

```bash
curl -si -H 'Host: <configured-host>' \
  --connect-timeout 5 --max-time 10 \
  "http://<gateway-service-cluster-ip>/" \
  | tee "$ADR0025_EVIDENCE/<gateway>-configured-response.txt"
```

### GW-DIAG: diagnose an unreachable gateway

Run this read-only block when GW-01 or GW-02 cannot establish a connection. It
records the Kubernetes traffic chain, attached Istio routing resources, Envoy
listeners, and relevant control-plane logs.

```bash
export ADR0025_FAILED_GATEWAY=istio-ingressgateway-service
export ADR0025_FAILED_GATEWAY_POD="$(kubectl get pod -n istio-system \
  -l "app=$ADR0025_FAILED_GATEWAY" \
  -o jsonpath='{.items[0].metadata.name}')"
export ADR0025_FAILED_GATEWAY_POD_IP="$(kubectl get pod -n istio-system \
  "$ADR0025_FAILED_GATEWAY_POD" \
  -o jsonpath='{.status.podIP}')"

{
  kubectl get service -n istio-system "$ADR0025_FAILED_GATEWAY" -o yaml
  kubectl get endpointslice -n istio-system \
    -l "kubernetes.io/service-name=$ADR0025_FAILED_GATEWAY" -o yaml
  kubectl get deployment -n istio-system "$ADR0025_FAILED_GATEWAY" -o yaml
  kubectl get pod -n istio-system "$ADR0025_FAILED_GATEWAY_POD" \
    -o jsonpath='{.metadata.labels}{"\n"}{.spec.serviceAccountName}{"\n"}{.status.conditions}{"\n"}'
  kubectl get authorizationpolicy -n istio-system \
    "$ADR0025_FAILED_GATEWAY" -o yaml
} | tee "$ADR0025_EVIDENCE/gateway-failure-kubernetes.txt"

kubectl get gateway.networking.istio.io -A -o json \
  | jq -c --arg app "$ADR0025_FAILED_GATEWAY" \
      '.items[]
       | select(.spec.selector.app == $app
           or .spec.selector.istio == "ingressgateway-service")
       | {namespace:.metadata.namespace,name:.metadata.name,
          selector:.spec.selector,servers:.spec.servers}' \
  | tee "$ADR0025_EVIDENCE/gateway-failure-attached-gateways.jsonl"

kubectl get virtualservice.networking.istio.io -A -o json \
  | jq -c '.items[]
      | select(any(.spec.gateways[]?;
          . == "service-gateway" or endswith("/service-gateway")))
      | {namespace:.metadata.namespace,name:.metadata.name,
         gateways:.spec.gateways,hosts:.spec.hosts}' \
  | tee "$ADR0025_EVIDENCE/gateway-failure-virtualservices.jsonl"

export ADR0025_ISTIO_VERSION="$(kubectl get deployment -n istio-system istiod \
  -o jsonpath='{.spec.template.spec.containers[0].image}' \
  | sed -E 's/.*:([0-9]+\.[0-9]+\.[0-9]+).*/\1/')"
export ADR0025_ISTIOCTL="$HOME/.local/bin/istioctl-$ADR0025_ISTIO_VERSION"
if [ ! -x "$ADR0025_ISTIOCTL" ]; then
  mkdir -p "$HOME/.local/bin"
  curl -fsSL --max-time 120 \
    "https://github.com/istio/istio/releases/download/$ADR0025_ISTIO_VERSION/istioctl-$ADR0025_ISTIO_VERSION-linux-amd64.tar.gz" \
    | tar -xz -C "$HOME/.local/bin"
  mv "$HOME/.local/bin/istioctl" "$ADR0025_ISTIOCTL"
fi

"$ADR0025_ISTIOCTL" proxy-status "$ADR0025_FAILED_GATEWAY_POD.istio-system" \
  | tee "$ADR0025_EVIDENCE/gateway-failure-proxy-status.txt"
"$ADR0025_ISTIOCTL" proxy-config listeners \
  "$ADR0025_FAILED_GATEWAY_POD.istio-system" \
  | tee "$ADR0025_EVIDENCE/gateway-failure-listeners.txt"
"$ADR0025_ISTIOCTL" proxy-config routes \
  "$ADR0025_FAILED_GATEWAY_POD.istio-system" \
  | tee "$ADR0025_EVIDENCE/gateway-failure-routes.txt"

{
  kubectl logs -n istio-system "$ADR0025_FAILED_GATEWAY_POD" --since=30m
  kubectl logs -n istio-system -l app=istiod --prefix --since=30m \
    | grep -Ei "$ADR0025_FAILED_GATEWAY|service-gateway|reject|error|warn" \
    || true
} | tee "$ADR0025_EVIDENCE/gateway-failure-control-plane-logs.txt"

set +e
{
  curl -sS -o /dev/null -w 'service_http_code=%{http_code}\n' \
    --connect-timeout 5 --max-time 10 \
    "http://$(kubectl get service -n istio-system \
      "$ADR0025_FAILED_GATEWAY" -o jsonpath='{.spec.clusterIP}')/"
  echo "service_curl_rc=$?"
  curl -sS -o /dev/null -w 'pod_http_code=%{http_code}\n' \
    --connect-timeout 5 --max-time 10 \
    "http://$ADR0025_FAILED_GATEWAY_POD_IP:8080/"
  echo "pod_curl_rc=$?"
} |& tee "$ADR0025_EVIDENCE/gateway-failure-connectivity.txt"
set -e
```

An empty listener/route result together with no matching `Gateway` means Envoy
is healthy but intentionally has no traffic listener. A matching `Gateway` with
no listener indicates an Istio attachment or configuration problem. A listener
with failed connectivity indicates a Service, endpoint, NetworkPolicy, or
process-level problem.

## 3. Positive protected flows

### FLOW-01: Web gateway to Web

```bash
export ADR0025_WEB_HOST="$(kubectl get gateway -n themelio web-gateway \
  -o jsonpath='{.spec.servers[0].hosts[0]}')"
export ADR0025_WEB_GATEWAY_IP="$(kubectl get service -n istio-system \
  -l app=istio-ingressgateway-web \
  -o jsonpath='{.items[0].spec.clusterIP}')"

curl -si -H "Host: $ADR0025_WEB_HOST" \
  --connect-timeout 5 --max-time 10 \
  "http://$ADR0025_WEB_GATEWAY_IP/" \
  | tee "$ADR0025_EVIDENCE/web-gateway-to-web.txt"
```

Pass when the response is the normal Web response or OIDC redirect, rather
than an Istio authorization denial. The expected source principal is:

```text
cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account
```

### FLOW-02 and FLOW-03: Web to both root actors

The Web image contains Node.js, so these calls originate from the real Web pod
and therefore use its sidecar-issued workload identity.

```bash
kubectl exec -i -n themelio deploy/web -- node <<'NODE' \
  | tee "$ADR0025_EVIDENCE/web-to-actors.txt"
const urls = [
  'http://k8s-root-actor.themelio.svc.cluster.local/-/health',
  'http://gitlab-root-actor.themelio.svc.cluster.local/-/health',
];
for (const url of urls) {
  const response = await fetch(url);
  console.log(url, response.status, await response.text());
  if (response.status !== 200) process.exitCode = 1;
}
NODE
```

Pass when both calls return `200`. The source principal must be:

```text
cluster.local/ns/themelio/sa/web
```

### FLOW-04 and FLOW-05: Waldur GitLab-sync workloads

```bash
for deployment in \
  waldur-agent-cloud-order-process \
  waldur-agent-cloud-membership-sync; do
  kubectl exec -n waldur-site-agent-cloud "deploy/$deployment" -- \
    python -c "import urllib.request; r=urllib.request.urlopen('http://gitlab-root-actor.themelio.svc.cluster.local/-/health', timeout=10); print('$deployment', r.status, r.read().decode())"
done | tee "$ADR0025_EVIDENCE/waldur-sync-to-gitlab-root-actor.txt"
```

Pass when both return `200`. Both must use this source principal:

```text
cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync
```

## 4. Negative protected flows

Run these calls one at a time. After each denial, immediately perform the
observability checks in section 7 so the evidence remains inside the metric and
log query windows.

### DENY-01: Waldur report is denied

```bash
set +e
kubectl exec -n waldur-site-agent-cloud deploy/waldur-agent-cloud-report -- \
  python -c "import urllib.request; urllib.request.urlopen('http://gitlab-root-actor.themelio.svc.cluster.local/-/health', timeout=10)" \
  >"$ADR0025_EVIDENCE/waldur-report-denial.txt" 2>&1
rc=$?
set -e
cat "$ADR0025_EVIDENCE/waldur-report-denial.txt"
test "$rc" -ne 0
```

Pass when the connection fails. An HTTP `200` is a failure. This workload uses
`waldur-site-agent-cloud`, which is intentionally absent from the actor's
allowed-principal list.

### DENY-02: arbitrary sidecar-injected Themelio identity is denied

**Mutating:** creates one ServiceAccount and one disposable pod.

```bash
kubectl apply -f - <<'YAML'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: adr0025-denied
  namespace: themelio
---
apiVersion: v1
kind: Pod
metadata:
  name: adr0025-denied
  namespace: themelio
  labels:
    app: adr0025-denied
    sidecar.istio.io/inject: "true"
spec:
  serviceAccountName: adr0025-denied
  automountServiceAccountToken: false
  restartPolicy: Never
  containers:
    - name: curl
      image: curlimages/curl:8.12.1
      command: ["sleep", "3600"]
YAML
kubectl wait -n themelio --for=condition=Ready pod/adr0025-denied --timeout=180s
```

```bash
for destination in k8s-root-actor gitlab-root-actor; do
  set +e
  kubectl exec -n themelio adr0025-denied -- \
    curl -fsS --connect-timeout 5 --max-time 10 \
    "http://$destination.themelio.svc.cluster.local/-/health"
  rc=$?
  set -e
  printf '%s rc=%s\n' "$destination" "$rc"
  test "$rc" -ne 0
done |& tee "$ADR0025_EVIDENCE/arbitrary-identity-denials.txt"
```

Preserve logs, then clean up:

```bash
kubectl delete pod,serviceaccount -n themelio adr0025-denied --wait=true
```

### DENY-03: direct plaintext pod-IP access is rejected

Obtain both protected destination pod IPs and call them directly from the
staging master, which has no workload SPIFFE identity:

```bash
for app in web k8s-root-actor gitlab-root-actor; do
  pod_ip="$(kubectl get pod -n themelio -l "app=$app" \
    -o jsonpath='{.items[0].status.podIP}')"
  path='/-/health'
  if [ "$app" = web ]; then path='/'; fi
  set +e
  code="$(curl -sS -o /dev/null -w '%{http_code}' \
    --connect-timeout 5 --max-time 10 \
    "http://$pod_ip:3000$path")"
  rc=$?
  set -e
  printf '%s pod_ip=%s http_code=%s rc=%s\n' \
    "$app" "$pod_ip" "$code" "$rc"
  test "$rc" -ne 0
  test "$code" = 000
done |& tee "$ADR0025_EVIDENCE/plaintext-pod-ip-denials.txt"
```

Pass only when no HTTP response is received (`http_code=000`). An application
`401`, `403`, or `404` proves that plaintext reached the application and is a
failure for this test.

### FLOW-06: conditional seeder identity

Run only while the seeder still calls gitlab-root-actor. This does not run the
full repository mirror.

**Mutating:** creates one disposable pod using the existing seeder
ServiceAccount.

```bash
kubectl run adr0025-seeder-call -n themelio \
  --image=curlimages/curl:8.12.1 \
  --restart=Never \
  --labels='app=adr0025-seeder-call,sidecar.istio.io/inject=true' \
  --overrides='{"spec":{"serviceAccountName":"themelio-seeder","automountServiceAccountToken":false}}' \
  --command -- sleep 3600
kubectl wait -n themelio --for=condition=Ready pod/adr0025-seeder-call --timeout=180s
kubectl exec -n themelio adr0025-seeder-call -- \
  curl -fsS --max-time 10 \
  http://gitlab-root-actor.themelio.svc.cluster.local/-/health \
  | tee "$ADR0025_EVIDENCE/seeder-to-gitlab-root-actor.txt"
kubectl delete pod -n themelio adr0025-seeder-call --wait=true
```

Pass when the call returns `200` with principal
`cluster.local/ns/themelio/sa/themelio-seeder`.

## 5. Application authentication remains required

### AUTH-01A: Waldur token is still required after mesh authorization

This request originates from an allowed sidecar identity but deliberately omits
the application bearer token. The empty body is safe because authentication is
checked before request validation and mutation.

```bash
kubectl exec -i -n waldur-site-agent-cloud \
  deploy/waldur-agent-cloud-order-process -- python - <<'PY' \
  | tee "$ADR0025_EVIDENCE/waldur-application-auth.txt"
import urllib.error
import urllib.request

request = urllib.request.Request(
    'http://gitlab-root-actor.themelio.svc.cluster.local/waldur/project_create',
    data=b'{}',
    headers={'Content-Type': 'application/json'},
    method='POST',
)
try:
    urllib.request.urlopen(request, timeout=10)
    raise SystemExit('FAIL: request without bearer token succeeded')
except urllib.error.HTTPError as error:
    print('status', error.code)
    if error.code != 401:
        raise
PY
```

Pass when the mesh permits the connection but the application returns `401`.

### AUTH-01B: Web session authentication is still required

From a workstation allowed to reach staging, run:

```bash
curl -skI https://staging.themelio.pharos.grnet.gr/settings/tokens \
  | sed -E 's/^(set-cookie:).*/\1 [REDACTED]/I' \
  | tee "$ADR0025_EVIDENCE/web-session-auth.txt"
```

Pass when an unauthenticated request redirects to the login flow or returns the
normal application authentication response. It must not expose the protected
page.

## 6. Sidecar identity and mutual TLS

### OBS-01: sidecar proxy and certificate state

```bash
export ADR0025_ISTIO_VERSION="$(kubectl get deployment -n istio-system istiod \
  -o jsonpath='{.spec.template.spec.containers[0].image}' \
  | sed -E 's/.*:([0-9]+\.[0-9]+\.[0-9]+).*/\1/')"
export ADR0025_ISTIOCTL="$HOME/.local/bin/istioctl-$ADR0025_ISTIO_VERSION"

if [ ! -x "$ADR0025_ISTIOCTL" ]; then
  mkdir -p "$HOME/.local/bin"
  curl -fsSL --max-time 120 \
    "https://github.com/istio/istio/releases/download/$ADR0025_ISTIO_VERSION/istioctl-$ADR0025_ISTIO_VERSION-linux-amd64.tar.gz" \
    | tar -xz -C "$HOME/.local/bin"
  mv "$HOME/.local/bin/istioctl" "$ADR0025_ISTIOCTL"
fi

"$ADR0025_ISTIOCTL" version --remote=false \
  | tee "$ADR0025_EVIDENCE/istioctl-version.txt"
"$ADR0025_ISTIOCTL" proxy-status \
  | tee "$ADR0025_EVIDENCE/sidecar-proxy-status.txt"

for workload in web k8s-root-actor gitlab-root-actor; do
  pod="$(kubectl get pod -n themelio -l "app=$workload" \
    -o jsonpath='{.items[0].metadata.name}')"
  printf '=== themelio/%s ===\n' "$pod"
  kubectl get pod -n themelio "$pod" \
    -o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'
  grep -F "$pod.themelio" "$ADR0025_EVIDENCE/sidecar-proxy-status.txt"
  "$ADR0025_ISTIOCTL" proxy-config secret "$pod.themelio"
done | tee "$ADR0025_EVIDENCE/themelio-sidecar-state.txt"

for workload in \
  waldur-agent-cloud-order-process \
  waldur-agent-cloud-membership-sync; do
  pod="$(kubectl get pod -n waldur-site-agent-cloud -l "app=$workload" \
    -o jsonpath='{.items[0].metadata.name}')"
  printf '=== waldur-site-agent-cloud/%s ===\n' "$pod"
  kubectl get pod -n waldur-site-agent-cloud "$pod" \
    -o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'
  grep -F "$pod.waldur-site-agent-cloud" \
    "$ADR0025_EVIDENCE/sidecar-proxy-status.txt"
  "$ADR0025_ISTIOCTL" proxy-config secret \
    "$pod.waldur-site-agent-cloud"
done | tee "$ADR0025_EVIDENCE/waldur-sidecar-state.txt"
```

Pass when every selected workload contains `istio-proxy`, is synchronized in
`proxy-status`, and its proxy has a valid `cluster.local` workload certificate.

### OBS-02: Prometheus confirms sidecar mTLS traffic

```bash
export ADR0025_PROM_IP="$(kubectl get service -n monitoring \
  kube-prometheus-stack-prometheus -o jsonpath='{.spec.clusterIP}')"

curl -fsSG "http://$ADR0025_PROM_IP:9090/api/v1/query" \
  --data-urlencode 'query=sum by (source_principal,destination_principal) (istio_requests_total{reporter="destination",connection_security_policy="mutual_tls"})' \
  | jq -c '.data.result' \
  | tee "$ADR0025_EVIDENCE/sidecar-mtls-requests.json"

curl -fsSG "http://$ADR0025_PROM_IP:9090/api/v1/query" \
  --data-urlencode 'query=sum by (destination_workload) (istio_requests_total{reporter="destination",connection_security_policy="mutual_tls"})' \
  | jq -c '.data.result' \
  | tee "$ADR0025_EVIDENCE/sidecar-mtls-workloads.json"
```

Pass when the positive tests appear with `connection_security_policy="mutual_tls"`
and the expected source/destination principals.

## 7. Denial observability

Run this section within 30 minutes of each negative test.

### OBS-03A: destination sidecar logs

```bash
for workload in web k8s-root-actor gitlab-root-actor; do
  kubectl logs -n themelio -l "app=$workload" \
    -c istio-proxy --prefix --since=30m
done | grep -Ei 'rbac|deny|denied|authorization|tls|identity| 403 ' \
  | tee "$ADR0025_EVIDENCE/sidecar-denials.txt"
```

### OBS-03B: Prometheus failure metrics and alerts

```bash
curl -fsSG "http://$ADR0025_PROM_IP:9090/api/v1/query" \
  --data-urlencode 'query=sum by (source_principal,destination_principal,response_code) (istio_requests_total{reporter="destination",response_code="403"})' \
  | jq -c '.data.result' \
  | tee "$ADR0025_EVIDENCE/sidecar-denial-metrics.json"

curl -fsS "http://$ADR0025_PROM_IP:9090/api/v1/alerts" \
  | jq '.data.alerts[] | select(.labels.alertname | startswith("Istio"))' \
  | tee "$ADR0025_EVIDENCE/istio-alerts.json"
```

### OBS-03C: Loki evidence

```bash
export ADR0025_LOKI_IP="$(kubectl get service -n monitoring loki-gateway \
  -o jsonpath='{.spec.clusterIP}')"
export ADR0025_LOKI_START="$(date -u -d '30 minutes ago' +%s)000000000"

curl -fsSG "http://$ADR0025_LOKI_IP/loki/api/v1/query_range" \
  --data-urlencode 'query={container="istio-proxy"} |~ "RBAC|DENY|denied|policy rejection|TLS_FAILURE| 403 "' \
  --data-urlencode "start=$ADR0025_LOKI_START" \
  --data-urlencode 'limit=500' \
  | jq -c '.data.result' \
  | tee "$ADR0025_EVIDENCE/loki-istio-denials.json"
```

Pass when the deliberately generated denials can be attributed to the expected
source and destination in sidecar logs or metrics, and the monitoring pipeline
contains corresponding evidence. Record any missing label or scrape problem as
a failure rather than inferring success from the client error alone.

## 8. Broader application smoke tests

### APP-01 and APP-02: browser/OIDC and normal navigation

From a browser with access to staging:

1. Open `https://staging.themelio.pharos.grnet.gr` in a private window.
2. Confirm redirect to the configured OIDC provider.
3. Log in with a staging test user.
4. Confirm the catalogue, project selector, model, dataset, repo, space,
   recipe, upskilling, settings, and token pages load without unexpected
   `401`, `403`, or `5xx` responses.
5. In browser developer tools, export a sanitized HAR with cookies and
   authorization headers removed.

Correlate browser actions with application health:

```bash
kubectl logs -n themelio deploy/web --since=15m \
  | tee "$ADR0025_EVIDENCE/web-navigation.log"
kubectl get pods -n themelio \
  | tee "$ADR0025_EVIDENCE/themelio-after-navigation.txt"
```

### APP-03: Git clone, push, and LFS

**Mutating:** creates a short-lived GitLab PAT for the existing `pharos-admin`
staging user, a small disposable repository, and a disposable sidecar-injected test pod.
The pod uses the existing `themelio-seeder` identity to request the token from
gitlab-root-actor. It revokes the PAT and impersonation token and deletes the
repository on exit. The PAT is never printed or written to evidence.

```bash
kubectl delete pod -n themelio adr0025-git-smoke \
  --ignore-not-found --wait=true

cat <<YAML | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: adr0025-git-smoke
  namespace: themelio
  labels:
    app: adr0025-git-smoke
    sidecar.istio.io/inject: "true"
spec:
  serviceAccountName: themelio-seeder
  automountServiceAccountToken: false
  restartPolicy: Never
  containers:
    - name: git-smoke
      image: alpine:3.22
      command: ["/bin/sh", "-c"]
      env:
        - name: ADR0025_RUN_ID
          value: "$ADR0025_RUN_ID"
      args:
        - |
          set -eu
          apk add --no-cache ca-certificates coreutils curl git git-lfs jq \
            >/dev/null

          gitlab=https://staging.gitlab.pharos.grnet.gr
          actor=http://gitlab-root-actor.themelio.svc.cluster.local
          username=pharos-admin
          project_id=''
          impersonation_id=''
          impersonation_token=''
          pat=''

          cleanup() {
            command_status=\$?
            set +e
            cleanup_failed=0
            if [ -n "\$project_id" ] && [ -n "\$pat" ]; then
              curl -fsS -X DELETE -H "PRIVATE-TOKEN: \$pat" \
                "\$gitlab/api/v4/projects/\$project_id" >/dev/null \
                && echo 'repository_cleanup=ok' \
                || cleanup_failed=1
            fi
            if [ -n "\$pat" ]; then
              curl -fsS -X DELETE -H "PRIVATE-TOKEN: \$pat" \
                "\$gitlab/api/v4/personal_access_tokens/self" >/dev/null \
                && echo 'temporary_pat_revoked=ok' \
                || cleanup_failed=1
            fi
            if [ -n "\$impersonation_id" ]; then
              curl -fsS -X DELETE \
                "\$actor/users/\$user_id/impersonation_tokens/\$impersonation_id" \
                >/dev/null \
                && echo 'impersonation_token_revoked=ok' \
                || cleanup_failed=1
            fi
            trap - EXIT
            if [ "\$command_status" -ne 0 ] || [ "\$cleanup_failed" -ne 0 ]; then
              exit 1
            fi
          }
          trap cleanup EXIT

          user_id="\$(curl -fsSG "\$gitlab/api/v4/users" \
            --data-urlencode "username=\$username" \
            | jq -er '.[0].id')"

          impersonation_response="\$(curl -fsS -X POST \
            "\$actor/users/\$user_id/impersonation_tokens")"
          impersonation_id="\$(printf '%s' "\$impersonation_response" \
            | jq -er '.id')"
          impersonation_token="\$(printf '%s' "\$impersonation_response" \
            | jq -er '.token')"

          expires_at="\$(date -u -d '+1 day' +%F)"
          pat_response="\$(jq -nc \
            --arg name "adr0025-\$ADR0025_RUN_ID" \
            --arg expires_at "\$expires_at" \
            '{name:\$name,expires_at:\$expires_at,scopes:["api","read_repository","write_repository"]}' \
            | curl -fsS -X POST \
                -H 'Content-Type: application/json' \
                -H "Authorization: Bearer \$impersonation_token" \
                --data-binary @- \
                "\$actor/api/v4/user/personal_access_tokens")"
          pat="\$(printf '%s' "\$pat_response" | jq -er '.token')"
          unset impersonation_token impersonation_response pat_response
          echo 'temporary_pat=created'

          project_name="adr0025-smoke-\$(printf '%s' "\$ADR0025_RUN_ID" \
            | tr '[:upper:]' '[:lower:]')"
          project_response="\$(curl -fsS -X POST \
            -H "PRIVATE-TOKEN: \$pat" \
            --data-urlencode "name=\$project_name" \
            --data-urlencode 'visibility=private' \
            --data-urlencode 'initialize_with_readme=true' \
            "\$gitlab/api/v4/projects")"
          project_id="\$(printf '%s' "\$project_response" | jq -er '.id')"
          project_path="\$(printf '%s' "\$project_response" \
            | jq -er '.path_with_namespace')"
          unset project_response
          echo "repository=created path=\$project_path"

          cat >/tmp/askpass <<'SH'
          #!/bin/sh
          case "\$1" in
            *Username*) printf '%s\n' oauth2 ;;
            *Password*) printf '%s\n' "\$ADR0025_GITLAB_PAT" ;;
          esac
          SH
          chmod 700 /tmp/askpass
          export ADR0025_GITLAB_PAT="\$pat"
          export GIT_ASKPASS=/tmp/askpass GIT_TERMINAL_PROMPT=0

          repository_url="\$gitlab/\$project_path.git"
          git clone "\$repository_url" /tmp/repository >/dev/null
          echo 'clone=ok'
          cd /tmp/repository
          git config user.name 'ADR-0025 validation'
          git config user.email 'adr-0025-validation@example.invalid'
          git switch -c adr0025-smoke >/dev/null
          git lfs install --local >/dev/null
          git lfs track '*.adr0025.bin' >/dev/null
          dd if=/dev/urandom of=smoke.adr0025.bin \
            bs=1024 count=64 status=none
          sha256sum smoke.adr0025.bin | cut -d' ' -f1 >/tmp/source.sha256
          git add .gitattributes smoke.adr0025.bin
          git commit -m 'test: ADR-0025 Git and LFS smoke test' >/dev/null
          git push origin adr0025-smoke >/dev/null
          echo 'push_and_lfs_upload=ok'

          GIT_LFS_SKIP_SMUDGE=1 git clone --branch adr0025-smoke \
            "\$repository_url" /tmp/verification >/dev/null
          git -C /tmp/verification lfs pull >/dev/null
          sha256sum /tmp/verification/smoke.adr0025.bin \
            | cut -d' ' -f1 >/tmp/verification.sha256
          cmp -s /tmp/source.sha256 /tmp/verification.sha256
          echo 'fresh_clone_and_lfs_download=ok'

          git push origin --delete adr0025-smoke >/dev/null
          echo 'branch_cleanup=ok'
YAML

kubectl wait -n themelio --for=condition=Ready \
  pod/adr0025-git-smoke --timeout=180s
kubectl wait -n themelio --for=jsonpath='{.status.phase}'=Succeeded \
  pod/adr0025-git-smoke --timeout=600s
```

While the completion wait is active, the following documented diagnostic may
be run from a second staging-master shell. It does not expose token values:

```bash
kubectl get pod -n themelio adr0025-git-smoke -o wide
kubectl logs -n themelio adr0025-git-smoke --tail=100
```

If the diagnostic reports `Error`, interrupt the completion wait with
`Ctrl-C`, preserve the failed log, and remove the failed pod before correcting
this document and retrying:

```bash
kubectl logs -n themelio adr0025-git-smoke \
  | tee "$ADR0025_EVIDENCE/git-clone-push-lfs-failed.txt"
kubectl delete pod -n themelio adr0025-git-smoke --wait=true
```

After the completion wait succeeds, continue in its original shell:

```bash
kubectl logs -n themelio adr0025-git-smoke \
  | tee "$ADR0025_EVIDENCE/git-clone-push-lfs.txt"
kubectl delete pod -n themelio adr0025-git-smoke --wait=true
```

Pass when the log records `temporary_pat=created`, `clone=ok`,
`push_and_lfs_upload=ok`, `fresh_clone_and_lfs_download=ok`, and
`branch_cleanup=ok`, followed by `repository_cleanup=ok`,
`temporary_pat_revoked=ok`, and `impersonation_token_revoked=ok`. Preserve pod
logs before deletion if the test fails, then verify that the PAT, impersonation
token, and disposable repository were removed before retrying.

### APP-04: Kubernetes deployment action

**Mutating:** create and then remove a disposable deployment/job through the
Themelio UI, not directly with kubectl. In parallel, observe:

```bash
export ADR0025_TENANT_NAMESPACE='<disposable-staging-project-namespace>'
kubectl get pods,jobs,deployments,inferenceservices -n "$ADR0025_TENANT_NAMESPACE" -w
```

In another terminal:

```bash
kubectl logs -n themelio deploy/k8s-root-actor --since=20m \
  | tee "$ADR0025_EVIDENCE/k8s-root-actor-action.log"
```

Pass when create, status updates, logs, and deletion work from the UI and the
actor remains ready.

### APP-05: Waldur reconciliation

**Mutating:** approve a disposable staging order and add/remove a disposable
member in Waldur. Observe both real processors:

```bash
kubectl logs -n waldur-site-agent-cloud \
  deploy/waldur-agent-cloud-order-process -f --since=5m
```

```bash
kubectl logs -n waldur-site-agent-cloud \
  deploy/waldur-agent-cloud-membership-sync -f --since=5m
```

After reconciliation:

```bash
kubectl get namespace "$ADR0025_TENANT_NAMESPACE" --show-labels
kubectl get resourcequota,serviceaccount,rolebinding \
  -n "$ADR0025_TENANT_NAMESPACE"
kubectl logs -n themelio deploy/gitlab-root-actor --since=20m \
  | tee "$ADR0025_EVIDENCE/waldur-gitlab-reconciliation.log"
```

Pass when the namespace/quota and GitLab groups/membership match the Waldur
state. Remove the disposable order/project using the supported workflow.

### APP-06: Kubeflow and external authorization

```bash
curl -skI https://staging.kubeflow.pharos.grnet.gr/ \
  | tee "$ADR0025_EVIDENCE/kubeflow-auth-response.txt"
kubectl logs -n oauth2-proxy -l app.kubernetes.io/name=oauth2-proxy \
  --all-containers --prefix --since=10m \
  | tee "$ADR0025_EVIDENCE/kubeflow-oauth2-proxy.log"
```

Then log in through a browser and open an authorized Kubeflow page. Pass when
unauthenticated access enters the external-authorization/login flow, authorized
access works, and the existing `CUSTOM` policy continues to operate alongside
the scoped host `ALLOW` policy.

### APP-07: registry pull and push

**Mutating:** creates one disposable in-cluster pod and pushes one disposable
tag. The pod reads the existing registry credentials through Secret references;
the commands never decode or print them. The pod and its temporary credential
file are deleted after the test. The pushed tag is left for the normal registry
retention workflow.

```bash
kubectl delete pod -n registry adr0025-registry-smoke \
  --ignore-not-found --wait=true

cat <<YAML | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: adr0025-registry-smoke
  namespace: registry
spec:
  restartPolicy: Never
  automountServiceAccountToken: false
  containers:
    - name: crane
      image: gcr.io/go-containerregistry/crane:debug
      command: ["/busybox/sh", "-c"]
      args:
        - |
          set -eu
          registry=staging.registry.pharos.grnet.gr
          source="\$registry/themelio/web:staging"
          target="\$registry/themelio/web:adr0025-$ADR0025_RUN_ID"
          crane auth login "\$registry" \
            -u "\$REGISTRY_USER" -p "\$REGISTRY_PASSWORD" >/dev/null
          echo "source_digest=\$(crane digest "\$source")"
          crane pull "\$source" /tmp/source.tar
          echo "pull_source=ok bytes=\$(wc -c </tmp/source.tar)"
          crane push /tmp/source.tar "\$target"
          echo "push_test=ok"
          crane pull "\$target" /tmp/test.tar
          echo "pull_test=ok bytes=\$(wc -c </tmp/test.tar)"
          test "\$(crane digest "\$source")" = \
            "\$(crane digest "\$target")"
          echo "digest_match=ok"
      env:
        - name: REGISTRY_USER
          valueFrom:
            secretKeyRef:
              name: registry-auth
              key: k8s_user
        - name: REGISTRY_PASSWORD
          valueFrom:
            secretKeyRef:
              name: registry-auth
              key: k8s_pass
YAML

kubectl wait -n registry --for=condition=Ready \
  pod/adr0025-registry-smoke --timeout=180s
kubectl wait -n registry --for=jsonpath='{.status.phase}'=Succeeded \
  pod/adr0025-registry-smoke --timeout=600s
kubectl logs -n registry adr0025-registry-smoke \
  | tee "$ADR0025_EVIDENCE/registry-pull-push.txt"
kubectl delete pod -n registry adr0025-registry-smoke --wait=true
```

Pass when the log records `pull_source=ok`, `push_test=ok`, `pull_test=ok`, and
`digest_match=ok`. Preserve logs before deleting the pod if it fails.

### APP-08: Grafana and Prometheus

The staging reverse proxy restricts both UIs by source CIDR. Test from an
allowed network:

```bash
curl -skI https://staging.grafana.pharos.grnet.gr/ \
  | tee "$ADR0025_EVIDENCE/grafana-public-response.txt"
curl -skI https://staging.prometheus.pharos.grnet.gr/ \
  | tee "$ADR0025_EVIDENCE/prometheus-public-response.txt"
```

Open both UIs in a browser. Confirm the ADR-0025 sidecar dashboard loads and
Prometheus has ready sidecar, istiod, and gateway targets. A public `403` from a
non-allowed source tests the Nginx CIDR restriction, not the Istio policy; use
the direct gateway Service ClusterIP test in section 2 for Istio evidence.

### APP-09: readiness probes and alert definitions

```bash
kubectl get deploy -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas' \
  | tee "$ADR0025_EVIDENCE/deployment-readiness.txt"

kubectl get prometheusrule -n monitoring istio-sidecar-security -o yaml \
  | tee "$ADR0025_EVIDENCE/istio-alert-rules.yaml"

kubectl get podmonitor -n monitoring \
  istio-workload-sidecars istio-control-plane istio-gateways -o wide \
  | tee "$ADR0025_EVIDENCE/istio-podmonitors.txt"
```

Pass when protected workloads and platform components remain ready, all three
Istio scrape definitions exist, and the ADR-0025 alert classes are loaded:
control-plane availability, gateway host rejection, workload-identity denial,
mTLS failure, and application authorization rejection.

## 9. Tenant workload authorization

Validates the scoped tenant-namespace policies that `k8s-root-actor` generates
per Themelio deployment and inference service (DD-0025 §9). The positive path —
the workload is reachable through the Themelio service gateway — is already
exercised by APP-04 and the Space route in GW-01; this section verifies the
*security* behavior that a happy-path UI test cannot: that the policy is scoped,
enforced, denies access from outside the tenant namespace, and is cleaned up.

```mermaid
graph LR
  sgw([Themelio service gateway]):::gw -->|admitted by name-service-gateway| dep[Tenant deployment]:::wl
  sgw -->|admitted| pred[KServe predictor]:::wl
  same[Same-namespace caller]:::caller -->|ns-owner-access-istio| dep
  foreign[Foreign-namespace pod]:::deny -. 403 .-> dep
  foreign -. 403 .-> pred

  classDef gw fill:#e0e2fa,stroke:#4753c9,color:#142029
  classDef wl fill:#d6ecec,stroke:#0c7a7a,color:#142029
  classDef caller fill:#ffffff,stroke:#c2ccd4,color:#142029
  classDef deny fill:#f5dfe3,stroke:#bd3f56,color:#142029
```

Each generated `<name>-service-gateway` policy admits only the Themelio
service-gateway principal on the workload port; `ns-owner-access-istio` retains
same-namespace access, and the former namespace-wide `allow-all` is removed.

**Prerequisites:** the mesh runs native sidecars
(`ENABLE_NATIVE_SIDECARS=true`, DD-0025 §2.2), and one disposable deployment and
one disposable inference service have been created from the UI (APP-04) in a
disposable tenant namespace. Set the placeholders:

```bash
export ADR0025_TENANT_NAMESPACE='<disposable-staging-project-namespace>'
export ADR0025_DEPLOYMENT='<deployment-name>'
export ADR0025_INFERENCE='<inferenceservice-name>'
export ADR0025_GATEWAY_PRINCIPAL='cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account'
```

### TENANT-01: scoped policies exist and carry no namespace-wide rule

```bash
kubectl get authorizationpolicy -n "$ADR0025_TENANT_NAMESPACE" \
  -o custom-columns=NAME:.metadata.name,ACTION:.spec.action \
  | tee "$ADR0025_EVIDENCE/tenant-authz-policies.txt"

# Full shape of the two generated policies: selector, admitted principal, port.
kubectl get authorizationpolicy \
  "${ADR0025_DEPLOYMENT}-service-gateway" \
  "${ADR0025_INFERENCE}-service-gateway" \
  -n "$ADR0025_TENANT_NAMESPACE" -o yaml \
  | tee "$ADR0025_EVIDENCE/tenant-scoped-policies.yaml"

# No ALLOW policy in the namespace contains an empty (match-all) rule.
kubectl get authorizationpolicy -n "$ADR0025_TENANT_NAMESPACE" -o json \
  | jq '[.items[]
         | select(.spec.action=="ALLOW")
         | select((.spec.rules // []) | index({}))
         | .metadata.name]' \
  | tee "$ADR0025_EVIDENCE/tenant-namespace-wide-rules.json"
```

Pass when each `<name>-service-gateway` policy selects its workload (the
deployment-runner labels, or the KServe `serving.kserve.io/inferenceservice` +
`component: predictor` labels), admits only `$ADR0025_GATEWAY_PRINCIPAL` on the
workload port, and the empty-rule array is `[]`.

### TENANT-02: enforcement denies an identity outside the tenant namespace

The retained `ns-owner-access-istio` policy allows all same-namespace traffic
(`source.namespace == <tenant>`), so an in-namespace caller is admitted by
design. The scoped policy's contribution is blocking callers from *outside* the
tenant namespace that are not the service gateway — the access `allow-all` used
to permit. The deny test therefore runs from a foreign namespace.

First confirm the predictor carries the Envoy proxy as a **native** sidecar —
without it the policy is silently inert:

```bash
predictor_pod="$(kubectl get pod -n "$ADR0025_TENANT_NAMESPACE" \
  -l "serving.kserve.io/inferenceservice=${ADR0025_INFERENCE},component=predictor" \
  -o jsonpath='{.items[0].metadata.name}')"
kubectl get pod -n "$ADR0025_TENANT_NAMESPACE" "$predictor_pod" \
  -o jsonpath='{.spec.initContainers[*].name}{"\n"}' \
  | tee "$ADR0025_EVIDENCE/tenant-predictor-initcontainers.txt"
```

istio-proxy must appear among the init containers.

**Mutating:** create one disposable pod in a foreign (non-tenant) namespace and
confirm it is denied on each workload's port:

```bash
export ADR0025_FOREIGN_NAMESPACE='themelio'
kubectl -n "$ADR0025_FOREIGN_NAMESPACE" run adr0025-foreign-denied \
  --restart=Never --image=curlimages/curl:8.12.1 --command -- sleep 3600
kubectl -n "$ADR0025_FOREIGN_NAMESPACE" \
  wait --for=condition=Ready pod/adr0025-foreign-denied --timeout=180s
```

```bash
for target in \
  "http://${ADR0025_DEPLOYMENT}.${ADR0025_TENANT_NAMESPACE}.svc.cluster.local/" \
  "http://${ADR0025_INFERENCE}-predictor.${ADR0025_TENANT_NAMESPACE}.svc.cluster.local/"; do
  set +e
  code="$(kubectl exec -n "$ADR0025_FOREIGN_NAMESPACE" adr0025-foreign-denied -- \
    curl -sS -o /dev/null -w '%{http_code}' \
    --connect-timeout 5 --max-time 10 "$target")"
  rc=$?
  set -e
  printf '%s http_code=%s rc=%s\n' "$target" "$code" "$rc"
  [ "$rc" -ne 0 ] || [ "$code" = 403 ]
done |& tee "$ADR0025_EVIDENCE/tenant-unauthorized-denials.txt"
```

Pass when both targets return `403` (or a refused/reset connection). A same-
namespace caller returning `200` is expected and is not a failure.

Pass only when both targets are denied — a refused/reset connection (non-zero
`rc`) or an RBAC `403`. A `200` proves the policy is not enforcing and is a
failure. Preserve logs, then clean up:

```bash
kubectl delete pod -n "$ADR0025_TENANT_NAMESPACE" adr0025-tenant-denied --wait=true
```

### TENANT-03: authorized access through the service gateway succeeds

The counterpart to TENANT-02: prove the policy admits the *right* identity, not
just that it denies the wrong one. Route to each workload through the Themelio
service gateway (whose ServiceAccount is the admitted principal) and confirm the
application answers rather than the destination Envoy rejecting with an RBAC
`403`. Discover each workload's host and route path from its VirtualService, and
reuse the Themelio service gateway Service ClusterIP that GW-01 already found for
the Space route:

```bash
export ADR0025_SERVICE_GATEWAY_IP='<Themelio service-gateway Service ClusterIP from GW-01>'

for entry in \
  "${ADR0025_DEPLOYMENT}-vs" \
  "${ADR0025_INFERENCE}-predictor-vs"; do
  host="$(kubectl get virtualservice "$entry" -n "$ADR0025_TENANT_NAMESPACE" \
    -o jsonpath='{.spec.hosts[0]}')"
  path="$(kubectl get virtualservice "$entry" -n "$ADR0025_TENANT_NAMESPACE" \
    -o jsonpath='{.spec.http[0].match[0].uri.exact}')"
  code="$(curl -sS -o /dev/null -w '%{http_code}' -H "Host: $host" \
    --connect-timeout 5 --max-time 15 \
    "http://${ADR0025_SERVICE_GATEWAY_IP}${path}" || true)"
  printf '%s\t%s%s\t%s\n' "$entry" "$host" "$path" "$code"
done |& tee "$ADR0025_EVIDENCE/tenant-authorized-through-gateway.txt"
```

Pass when each request reaches the normal application or authentication
behavior. A gateway/RBAC `403` is a failure — the policy is wrongly denying the
gateway identity. Distinguish the two: an Envoy RBAC denial returns the body
`RBAC: access denied` with no application content, so capture headers and body
for any `403`:

```bash
curl -si -H "Host: <workload-host>" \
  --connect-timeout 5 --max-time 15 \
  "http://${ADR0025_SERVICE_GATEWAY_IP}<workload-path>" \
  | tee "$ADR0025_EVIDENCE/tenant-authorized-response.txt"
```

### TENANT-04: teardown removes the scoped policy

Delete the disposable deployment and inference service **through the UI**, then
confirm no `<name>-service-gateway` policy is orphaned:

```bash
kubectl get authorizationpolicy -n "$ADR0025_TENANT_NAMESPACE" \
  | grep -E "${ADR0025_DEPLOYMENT}-service-gateway|${ADR0025_INFERENCE}-service-gateway" \
  | tee "$ADR0025_EVIDENCE/tenant-policy-teardown.txt" \
  || echo "no orphan service-gateway policies" \
  | tee "$ADR0025_EVIDENCE/tenant-policy-teardown.txt"
```

Pass when neither policy remains after the UI deletion completes.

## 10. Final review

Do not mark the run accepted solely because the positive calls passed. Review
every required row in the top-level checklist and record links or paths to the
evidence.

```bash
printf 'finished_utc=%s\n' "$(date -u +%FT%TZ)" \
  | tee -a "$ADR0025_EVIDENCE/run.txt"
find "$ADR0025_EVIDENCE" -maxdepth 1 -type f -printf '%f\n' | sort
```

Validation succeeds only when:

- every required row is `PASS`;
- conditional seeder coverage is either `PASS` or documented as
  `NOT APPLICABLE`;
- no unexplained regression, missing denial evidence, or unexpected principal
  remains;
- temporary pods, branches, and application resources have been removed;
- credentials used during testing have been unset, and any credential exposed
  outside an approved secret store has been rotated.