Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.
ADR-0025 validation
This document records the commands and manual checks used to validate
ADR-0025 in staging. Execute it
with two people or with an operator and an agent: run one numbered validation,
inspect its evidence, record the result, and only then continue. Do not execute
the whole document as an unattended script.
The validation covers only behavior required by ADR-0025: native Istio security
controls, proof that the protected application flows still work, and regression
checks for the gateways and applications affected by those controls. A failed
required validation keeps ADR-0025 in Draft.
Safety rules
- Run against staging only. Confirm the context before every mutating test.
- Never paste PATs, passwords, cookies, or Kubernetes Secret values into this
document or an evidence file.
- Commands marked mutating create disposable test resources. Stop and obtain
operator confirmation before running them.
- Preserve relevant logs before deleting a failed test pod.
- Treat the commands in this document as the executable source of truth. Run
them verbatim after replacing documented placeholders only. If a command is
wrong or the environment requires a different command, stop, update this
document first, and then run the corrected command.
- Treat an ingress-host rejection and a backend identity rejection differently:
an unknown ingress host must return HTTP
403; a sidecar authorization
denial can appear as HTTP 403 or as a refused/reset connection depending on
whether Envoy classified the destination listener as HTTP or TCP.
Result values
Use one of these values in the result tables:
NOT RUNPASSFAILBLOCKEDNOT APPLICABLE
Validation checklist
0. Connect and initialize evidence
Run the following from the repository workstation:
cd /home/dkats/themelio/devops/ansible
make show-env
Expected output: staging. Stop if any other environment is selected.
eval "$(ssh-agent -s)" \
&& ssh-add ~/.ssh/pharos-admin-key \
&& ssh -J admin@18.195.179.229 admin@10.0.100.44
Run the remaining cluster commands on the staging master unless a section says
otherwise. Create a directory for non-secret evidence:
export ADR0025_RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
export ADR0025_EVIDENCE="$HOME/adr-0025-validation/$ADR0025_RUN_ID"
mkdir -p "$ADR0025_EVIDENCE"
printf 'run_id=%s\nstarted_utc=%s\n' \
"$ADR0025_RUN_ID" "$(date -u +%FT%TZ)" \
| tee "$ADR0025_EVIDENCE/run.txt"
To resume an existing validation run in a new shell, replace the documented
placeholder with its run ID and execute:
export ADR0025_RUN_ID='<existing-run-id>'
export ADR0025_EVIDENCE="$HOME/adr-0025-validation/$ADR0025_RUN_ID"
test -d "$ADR0025_EVIDENCE"
1. Preconditions
PRE-01: context and component health
kubectl config current-context | tee "$ADR0025_EVIDENCE/context.txt"
kubectl get nodes -o wide | tee "$ADR0025_EVIDENCE/nodes.txt"
kubectl get pods -n istio-system -o wide \
| tee "$ADR0025_EVIDENCE/istio-pods.txt"
kubectl get pods -n themelio -o wide \
| tee "$ADR0025_EVIDENCE/themelio-pods.txt"
kubectl get pods -n waldur-site-agent-cloud -o wide \
| tee "$ADR0025_EVIDENCE/waldur-agent-pods.txt"
kubectl get applications.argoproj.io -n argocd \
-o custom-columns='NAME:.metadata.name,REVISION:.spec.source.targetRevision,SYNC:.status.sync.status,HEALTH:.status.health.status' \
| tee "$ADR0025_EVIDENCE/argocd-applications.txt"
Pass when:
- the context and node names are staging;
- every schedulable node has a ready Istio CNI pod and no ztunnel remains;
- istiod and all ingress gateways are ready;
- the selected Themelio and Waldur workloads have ready
istio-proxy sidecars;
- the relevant Argo CD applications are healthy and at the intended revision.
PRE-02: enforced policy and identity inventory
kubectl get authorizationpolicy,peerauthentication -A -o wide \
| tee "$ADR0025_EVIDENCE/security-resources.txt"
kubectl get deploy -n themelio -o json \
| jq -r '.items[] | [.metadata.name,
(.spec.template.spec.serviceAccountName // "default"),
(.spec.template.metadata.labels["sidecar.istio.io/inject"] // "false")]
| @tsv' \
| tee "$ADR0025_EVIDENCE/themelio-identities.txt"
kubectl get deploy -n waldur-site-agent-cloud -o json \
| jq -r '.items[] | [.metadata.name,
(.spec.template.spec.serviceAccountName // "default"),
(.spec.template.metadata.labels["sidecar.istio.io/inject"] // "false")]
| @tsv' \
| tee "$ADR0025_EVIDENCE/waldur-identities.txt"
kubectl get authorizationpolicy -n istio-system -o json \
| jq -e '[.items[]
| select((.spec.selector.matchLabels.app // "")
| test("ingressgateway"))
| .spec.rules[]?
| select(. == {})]
| length == 0' \
| tee "$ADR0025_EVIDENCE/no-empty-gateway-rules.txt"
Expected protected identities:
Pass when the policies are present, selected workloads have the expected unique
ServiceAccounts, and no ingress ALLOW policy contains rules: [{}].
2. Gateway host enforcement
GW-01 and GW-02: test every rendered host policy
This command discovers the policies, gateway Services, configured hosts, and
ClusterIPs from the live cluster. It does not hard-code a second hostname
inventory. Calling the gateway Service directly avoids public DNS, the bastion,
and NodePort traffic-policy differences.
kubectl get authorizationpolicy -n istio-system -o json \
| jq -r '.items[]
| select([.spec.rules[]?.to[]?.operation.hosts[]?] | length > 0)
| [.metadata.name,
.spec.selector.matchLabels.app,
.spec.rules[0].to[0].operation.hosts[0]]
| @tsv' \
| while IFS=$'\t' read -r policy app configured_host; do
service="$(kubectl get service -n istio-system -l "app=$app" -o json \
| jq -r '.items[] | select(.spec.type=="NodePort") | .metadata.name' \
| head -n1)"
service_ip="$(kubectl get service -n istio-system "$service" \
-o jsonpath='{.spec.clusterIP}')"
configured_code="$(curl -sS -o /dev/null -w '%{http_code}' \
-H "Host: $configured_host" \
--connect-timeout 5 --max-time 10 "http://$service_ip/" || true)"
with_port_code="$(curl -sS -o /dev/null -w '%{http_code}' \
-H "Host: $configured_host:80" \
--connect-timeout 5 --max-time 10 "http://$service_ip/" || true)"
rejected_code="$(curl -sS -o /dev/null -w '%{http_code}' \
-H 'Host: rejected.invalid' \
--connect-timeout 5 --max-time 10 "http://$service_ip/" || true)"
printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
"$policy" "$service@$service_ip" "$configured_host" \
"$configured_code" "$with_port_code" "$rejected_code"
done \
| tee "$ADR0025_EVIDENCE/gateway-host-results.tsv"
Columns are policy, Service, configured host, configured-host response,
host-with-port response, and rejected-host response.
Pass when:
- each configured-host request reaches the normal application or
authentication behavior;
- each host-with-port request behaves the same way;
- every
rejected.invalid request returns 403.
A configured host can legitimately produce an authentication redirect or an
application 401/403. Record headers for any ambiguous response:
curl -si -H 'Host: <configured-host>' \
--connect-timeout 5 --max-time 10 \
"http://<gateway-service-cluster-ip>/" \
| tee "$ADR0025_EVIDENCE/<gateway>-configured-response.txt"
GW-DIAG: diagnose an unreachable gateway
Run this read-only block when GW-01 or GW-02 cannot establish a connection. It
records the Kubernetes traffic chain, attached Istio routing resources, Envoy
listeners, and relevant control-plane logs.
export ADR0025_FAILED_GATEWAY=istio-ingressgateway-service
export ADR0025_FAILED_GATEWAY_POD="$(kubectl get pod -n istio-system \
-l "app=$ADR0025_FAILED_GATEWAY" \
-o jsonpath='{.items[0].metadata.name}')"
export ADR0025_FAILED_GATEWAY_POD_IP="$(kubectl get pod -n istio-system \
"$ADR0025_FAILED_GATEWAY_POD" \
-o jsonpath='{.status.podIP}')"
{
kubectl get service -n istio-system "$ADR0025_FAILED_GATEWAY" -o yaml
kubectl get endpointslice -n istio-system \
-l "kubernetes.io/service-name=$ADR0025_FAILED_GATEWAY" -o yaml
kubectl get deployment -n istio-system "$ADR0025_FAILED_GATEWAY" -o yaml
kubectl get pod -n istio-system "$ADR0025_FAILED_GATEWAY_POD" \
-o jsonpath='{.metadata.labels}{"\n"}{.spec.serviceAccountName}{"\n"}{.status.conditions}{"\n"}'
kubectl get authorizationpolicy -n istio-system \
"$ADR0025_FAILED_GATEWAY" -o yaml
} | tee "$ADR0025_EVIDENCE/gateway-failure-kubernetes.txt"
kubectl get gateway.networking.istio.io -A -o json \
| jq -c --arg app "$ADR0025_FAILED_GATEWAY" \
'.items[]
| select(.spec.selector.app == $app
or .spec.selector.istio == "ingressgateway-service")
| {namespace:.metadata.namespace,name:.metadata.name,
selector:.spec.selector,servers:.spec.servers}' \
| tee "$ADR0025_EVIDENCE/gateway-failure-attached-gateways.jsonl"
kubectl get virtualservice.networking.istio.io -A -o json \
| jq -c '.items[]
| select(any(.spec.gateways[]?;
. == "service-gateway" or endswith("/service-gateway")))
| {namespace:.metadata.namespace,name:.metadata.name,
gateways:.spec.gateways,hosts:.spec.hosts}' \
| tee "$ADR0025_EVIDENCE/gateway-failure-virtualservices.jsonl"
export ADR0025_ISTIO_VERSION="$(kubectl get deployment -n istio-system istiod \
-o jsonpath='{.spec.template.spec.containers[0].image}' \
| sed -E 's/.*:([0-9]+\.[0-9]+\.[0-9]+).*/\1/')"
export ADR0025_ISTIOCTL="$HOME/.local/bin/istioctl-$ADR0025_ISTIO_VERSION"
if [ ! -x "$ADR0025_ISTIOCTL" ]; then
mkdir -p "$HOME/.local/bin"
curl -fsSL --max-time 120 \
"https://github.com/istio/istio/releases/download/$ADR0025_ISTIO_VERSION/istioctl-$ADR0025_ISTIO_VERSION-linux-amd64.tar.gz" \
| tar -xz -C "$HOME/.local/bin"
mv "$HOME/.local/bin/istioctl" "$ADR0025_ISTIOCTL"
fi
"$ADR0025_ISTIOCTL" proxy-status "$ADR0025_FAILED_GATEWAY_POD.istio-system" \
| tee "$ADR0025_EVIDENCE/gateway-failure-proxy-status.txt"
"$ADR0025_ISTIOCTL" proxy-config listeners \
"$ADR0025_FAILED_GATEWAY_POD.istio-system" \
| tee "$ADR0025_EVIDENCE/gateway-failure-listeners.txt"
"$ADR0025_ISTIOCTL" proxy-config routes \
"$ADR0025_FAILED_GATEWAY_POD.istio-system" \
| tee "$ADR0025_EVIDENCE/gateway-failure-routes.txt"
{
kubectl logs -n istio-system "$ADR0025_FAILED_GATEWAY_POD" --since=30m
kubectl logs -n istio-system -l app=istiod --prefix --since=30m \
| grep -Ei "$ADR0025_FAILED_GATEWAY|service-gateway|reject|error|warn" \
|| true
} | tee "$ADR0025_EVIDENCE/gateway-failure-control-plane-logs.txt"
set +e
{
curl -sS -o /dev/null -w 'service_http_code=%{http_code}\n' \
--connect-timeout 5 --max-time 10 \
"http://$(kubectl get service -n istio-system \
"$ADR0025_FAILED_GATEWAY" -o jsonpath='{.spec.clusterIP}')/"
echo "service_curl_rc=$?"
curl -sS -o /dev/null -w 'pod_http_code=%{http_code}\n' \
--connect-timeout 5 --max-time 10 \
"http://$ADR0025_FAILED_GATEWAY_POD_IP:8080/"
echo "pod_curl_rc=$?"
} |& tee "$ADR0025_EVIDENCE/gateway-failure-connectivity.txt"
set -e
An empty listener/route result together with no matching Gateway means Envoy
is healthy but intentionally has no traffic listener. A matching Gateway with
no listener indicates an Istio attachment or configuration problem. A listener
with failed connectivity indicates a Service, endpoint, NetworkPolicy, or
process-level problem.
3. Positive protected flows
FLOW-01: Web gateway to Web
export ADR0025_WEB_HOST="$(kubectl get gateway -n themelio web-gateway \
-o jsonpath='{.spec.servers[0].hosts[0]}')"
export ADR0025_WEB_GATEWAY_IP="$(kubectl get service -n istio-system \
-l app=istio-ingressgateway-web \
-o jsonpath='{.items[0].spec.clusterIP}')"
curl -si -H "Host: $ADR0025_WEB_HOST" \
--connect-timeout 5 --max-time 10 \
"http://$ADR0025_WEB_GATEWAY_IP/" \
| tee "$ADR0025_EVIDENCE/web-gateway-to-web.txt"
Pass when the response is the normal Web response or OIDC redirect, rather
than an Istio authorization denial. The expected source principal is:
cluster.local/ns/istio-system/sa/istio-ingressgateway-web-service-account
FLOW-02 and FLOW-03: Web to both root actors
The Web image contains Node.js, so these calls originate from the real Web pod
and therefore use its sidecar-issued workload identity.
kubectl exec -i -n themelio deploy/web -- node <<'NODE' \
| tee "$ADR0025_EVIDENCE/web-to-actors.txt"
const urls = [
'http://k8s-root-actor.themelio.svc.cluster.local/-/health',
'http://gitlab-root-actor.themelio.svc.cluster.local/-/health',
];
for (const url of urls) {
const response = await fetch(url);
console.log(url, response.status, await response.text());
if (response.status !== 200) process.exitCode = 1;
}
NODE
Pass when both calls return 200. The source principal must be:
cluster.local/ns/themelio/sa/web
FLOW-04 and FLOW-05: Waldur GitLab-sync workloads
for deployment in \
waldur-agent-cloud-order-process \
waldur-agent-cloud-membership-sync; do
kubectl exec -n waldur-site-agent-cloud "deploy/$deployment" -- \
python -c "import urllib.request; r=urllib.request.urlopen('http://gitlab-root-actor.themelio.svc.cluster.local/-/health', timeout=10); print('$deployment', r.status, r.read().decode())"
done | tee "$ADR0025_EVIDENCE/waldur-sync-to-gitlab-root-actor.txt"
Pass when both return 200. Both must use this source principal:
cluster.local/ns/waldur-site-agent-cloud/sa/waldur-site-agent-cloud-gitlab-sync
4. Negative protected flows
Run these calls one at a time. After each denial, immediately perform the
observability checks in section 7 so the evidence remains inside the metric and
log query windows.
DENY-01: Waldur report is denied
set +e
kubectl exec -n waldur-site-agent-cloud deploy/waldur-agent-cloud-report -- \
python -c "import urllib.request; urllib.request.urlopen('http://gitlab-root-actor.themelio.svc.cluster.local/-/health', timeout=10)" \
>"$ADR0025_EVIDENCE/waldur-report-denial.txt" 2>&1
rc=$?
set -e
cat "$ADR0025_EVIDENCE/waldur-report-denial.txt"
test "$rc" -ne 0
Pass when the connection fails. An HTTP 200 is a failure. This workload uses
waldur-site-agent-cloud, which is intentionally absent from the actor's
allowed-principal list.
DENY-02: arbitrary sidecar-injected Themelio identity is denied
Mutating: creates one ServiceAccount and one disposable pod.
kubectl apply -f - <<'YAML'
apiVersion: v1
kind: ServiceAccount
metadata:
name: adr0025-denied
namespace: themelio
---
apiVersion: v1
kind: Pod
metadata:
name: adr0025-denied
namespace: themelio
labels:
app: adr0025-denied
sidecar.istio.io/inject: "true"
spec:
serviceAccountName: adr0025-denied
automountServiceAccountToken: false
restartPolicy: Never
containers:
- name: curl
image: curlimages/curl:8.12.1
command: ["sleep", "3600"]
YAML
kubectl wait -n themelio --for=condition=Ready pod/adr0025-denied --timeout=180s
for destination in k8s-root-actor gitlab-root-actor; do
set +e
kubectl exec -n themelio adr0025-denied -- \
curl -fsS --connect-timeout 5 --max-time 10 \
"http://$destination.themelio.svc.cluster.local/-/health"
rc=$?
set -e
printf '%s rc=%s\n' "$destination" "$rc"
test "$rc" -ne 0
done |& tee "$ADR0025_EVIDENCE/arbitrary-identity-denials.txt"
Preserve logs, then clean up:
kubectl delete pod,serviceaccount -n themelio adr0025-denied --wait=true
DENY-03: direct plaintext pod-IP access is rejected
Obtain both protected destination pod IPs and call them directly from the
staging master, which has no workload SPIFFE identity:
for app in web k8s-root-actor gitlab-root-actor; do
pod_ip="$(kubectl get pod -n themelio -l "app=$app" \
-o jsonpath='{.items[0].status.podIP}')"
path='/-/health'
if [ "$app" = web ]; then path='/'; fi
set +e
code="$(curl -sS -o /dev/null -w '%{http_code}' \
--connect-timeout 5 --max-time 10 \
"http://$pod_ip:3000$path")"
rc=$?
set -e
printf '%s pod_ip=%s http_code=%s rc=%s\n' \
"$app" "$pod_ip" "$code" "$rc"
test "$rc" -ne 0
test "$code" = 000
done |& tee "$ADR0025_EVIDENCE/plaintext-pod-ip-denials.txt"
Pass only when no HTTP response is received (http_code=000). An application
401, 403, or 404 proves that plaintext reached the application and is a
failure for this test.
FLOW-06: conditional seeder identity
Run only while the seeder still calls gitlab-root-actor. This does not run the
full repository mirror.
Mutating: creates one disposable pod using the existing seeder
ServiceAccount.
kubectl run adr0025-seeder-call -n themelio \
--image=curlimages/curl:8.12.1 \
--restart=Never \
--labels='app=adr0025-seeder-call,sidecar.istio.io/inject=true' \
--overrides='{"spec":{"serviceAccountName":"themelio-seeder","automountServiceAccountToken":false}}' \
--command -- sleep 3600
kubectl wait -n themelio --for=condition=Ready pod/adr0025-seeder-call --timeout=180s
kubectl exec -n themelio adr0025-seeder-call -- \
curl -fsS --max-time 10 \
http://gitlab-root-actor.themelio.svc.cluster.local/-/health \
| tee "$ADR0025_EVIDENCE/seeder-to-gitlab-root-actor.txt"
kubectl delete pod -n themelio adr0025-seeder-call --wait=true
Pass when the call returns 200 with principal
cluster.local/ns/themelio/sa/themelio-seeder.
5. Application authentication remains required
AUTH-01A: Waldur token is still required after mesh authorization
This request originates from an allowed sidecar identity but deliberately omits
the application bearer token. The empty body is safe because authentication is
checked before request validation and mutation.
kubectl exec -i -n waldur-site-agent-cloud \
deploy/waldur-agent-cloud-order-process -- python - <<'PY' \
| tee "$ADR0025_EVIDENCE/waldur-application-auth.txt"
import urllib.error
import urllib.request
request = urllib.request.Request(
'http://gitlab-root-actor.themelio.svc.cluster.local/waldur/project_create',
data=b'{}',
headers={'Content-Type': 'application/json'},
method='POST',
)
try:
urllib.request.urlopen(request, timeout=10)
raise SystemExit('FAIL: request without bearer token succeeded')
except urllib.error.HTTPError as error:
print('status', error.code)
if error.code != 401:
raise
PY
Pass when the mesh permits the connection but the application returns 401.
AUTH-01B: Web session authentication is still required
From a workstation allowed to reach staging, run:
curl -skI https://staging.themelio.pharos.grnet.gr/settings/tokens \
| sed -E 's/^(set-cookie:).*/\1 [REDACTED]/I' \
| tee "$ADR0025_EVIDENCE/web-session-auth.txt"
Pass when an unauthenticated request redirects to the login flow or returns the
normal application authentication response. It must not expose the protected
page.
6. Sidecar identity and mutual TLS
OBS-01: sidecar proxy and certificate state
export ADR0025_ISTIO_VERSION="$(kubectl get deployment -n istio-system istiod \
-o jsonpath='{.spec.template.spec.containers[0].image}' \
| sed -E 's/.*:([0-9]+\.[0-9]+\.[0-9]+).*/\1/')"
export ADR0025_ISTIOCTL="$HOME/.local/bin/istioctl-$ADR0025_ISTIO_VERSION"
if [ ! -x "$ADR0025_ISTIOCTL" ]; then
mkdir -p "$HOME/.local/bin"
curl -fsSL --max-time 120 \
"https://github.com/istio/istio/releases/download/$ADR0025_ISTIO_VERSION/istioctl-$ADR0025_ISTIO_VERSION-linux-amd64.tar.gz" \
| tar -xz -C "$HOME/.local/bin"
mv "$HOME/.local/bin/istioctl" "$ADR0025_ISTIOCTL"
fi
"$ADR0025_ISTIOCTL" version --remote=false \
| tee "$ADR0025_EVIDENCE/istioctl-version.txt"
"$ADR0025_ISTIOCTL" proxy-status \
| tee "$ADR0025_EVIDENCE/sidecar-proxy-status.txt"
for workload in web k8s-root-actor gitlab-root-actor; do
pod="$(kubectl get pod -n themelio -l "app=$workload" \
-o jsonpath='{.items[0].metadata.name}')"
printf '=== themelio/%s ===\n' "$pod"
kubectl get pod -n themelio "$pod" \
-o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'
grep -F "$pod.themelio" "$ADR0025_EVIDENCE/sidecar-proxy-status.txt"
"$ADR0025_ISTIOCTL" proxy-config secret "$pod.themelio"
done | tee "$ADR0025_EVIDENCE/themelio-sidecar-state.txt"
for workload in \
waldur-agent-cloud-order-process \
waldur-agent-cloud-membership-sync; do
pod="$(kubectl get pod -n waldur-site-agent-cloud -l "app=$workload" \
-o jsonpath='{.items[0].metadata.name}')"
printf '=== waldur-site-agent-cloud/%s ===\n' "$pod"
kubectl get pod -n waldur-site-agent-cloud "$pod" \
-o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'
grep -F "$pod.waldur-site-agent-cloud" \
"$ADR0025_EVIDENCE/sidecar-proxy-status.txt"
"$ADR0025_ISTIOCTL" proxy-config secret \
"$pod.waldur-site-agent-cloud"
done | tee "$ADR0025_EVIDENCE/waldur-sidecar-state.txt"
Pass when every selected workload contains istio-proxy, is synchronized in
proxy-status, and its proxy has a valid cluster.local workload certificate.
OBS-02: Prometheus confirms sidecar mTLS traffic
export ADR0025_PROM_IP="$(kubectl get service -n monitoring \
kube-prometheus-stack-prometheus -o jsonpath='{.spec.clusterIP}')"
curl -fsSG "http://$ADR0025_PROM_IP:9090/api/v1/query" \
--data-urlencode 'query=sum by (source_principal,destination_principal) (istio_requests_total{reporter="destination",connection_security_policy="mutual_tls"})' \
| jq -c '.data.result' \
| tee "$ADR0025_EVIDENCE/sidecar-mtls-requests.json"
curl -fsSG "http://$ADR0025_PROM_IP:9090/api/v1/query" \
--data-urlencode 'query=sum by (destination_workload) (istio_requests_total{reporter="destination",connection_security_policy="mutual_tls"})' \
| jq -c '.data.result' \
| tee "$ADR0025_EVIDENCE/sidecar-mtls-workloads.json"
Pass when the positive tests appear with connection_security_policy="mutual_tls"
and the expected source/destination principals.
7. Denial observability
Run this section within 30 minutes of each negative test.
OBS-03A: destination sidecar logs
for workload in web k8s-root-actor gitlab-root-actor; do
kubectl logs -n themelio -l "app=$workload" \
-c istio-proxy --prefix --since=30m
done | grep -Ei 'rbac|deny|denied|authorization|tls|identity| 403 ' \
| tee "$ADR0025_EVIDENCE/sidecar-denials.txt"
OBS-03B: Prometheus failure metrics and alerts
curl -fsSG "http://$ADR0025_PROM_IP:9090/api/v1/query" \
--data-urlencode 'query=sum by (source_principal,destination_principal,response_code) (istio_requests_total{reporter="destination",response_code="403"})' \
| jq -c '.data.result' \
| tee "$ADR0025_EVIDENCE/sidecar-denial-metrics.json"
curl -fsS "http://$ADR0025_PROM_IP:9090/api/v1/alerts" \
| jq '.data.alerts[] | select(.labels.alertname | startswith("Istio"))' \
| tee "$ADR0025_EVIDENCE/istio-alerts.json"
OBS-03C: Loki evidence
export ADR0025_LOKI_IP="$(kubectl get service -n monitoring loki-gateway \
-o jsonpath='{.spec.clusterIP}')"
export ADR0025_LOKI_START="$(date -u -d '30 minutes ago' +%s)000000000"
curl -fsSG "http://$ADR0025_LOKI_IP/loki/api/v1/query_range" \
--data-urlencode 'query={container="istio-proxy"} |~ "RBAC|DENY|denied|policy rejection|TLS_FAILURE| 403 "' \
--data-urlencode "start=$ADR0025_LOKI_START" \
--data-urlencode 'limit=500' \
| jq -c '.data.result' \
| tee "$ADR0025_EVIDENCE/loki-istio-denials.json"
Pass when the deliberately generated denials can be attributed to the expected
source and destination in sidecar logs or metrics, and the monitoring pipeline
contains corresponding evidence. Record any missing label or scrape problem as
a failure rather than inferring success from the client error alone.
8. Broader application smoke tests
APP-01 and APP-02: browser/OIDC and normal navigation
From a browser with access to staging:
- Open
https://staging.themelio.pharos.grnet.gr in a private window.
- Confirm redirect to the configured OIDC provider.
- Log in with a staging test user.
- Confirm the catalogue, project selector, model, dataset, repo, space,
recipe, upskilling, settings, and token pages load without unexpected
401, 403, or 5xx responses.
- In browser developer tools, export a sanitized HAR with cookies and
authorization headers removed.
Correlate browser actions with application health:
kubectl logs -n themelio deploy/web --since=15m \
| tee "$ADR0025_EVIDENCE/web-navigation.log"
kubectl get pods -n themelio \
| tee "$ADR0025_EVIDENCE/themelio-after-navigation.txt"
APP-03: Git clone, push, and LFS
Mutating: creates a short-lived GitLab PAT for the existing pharos-admin
staging user, a small disposable repository, and a disposable sidecar-injected test pod.
The pod uses the existing themelio-seeder identity to request the token from
gitlab-root-actor. It revokes the PAT and impersonation token and deletes the
repository on exit. The PAT is never printed or written to evidence.
kubectl delete pod -n themelio adr0025-git-smoke \
--ignore-not-found --wait=true
cat <<YAML | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: adr0025-git-smoke
namespace: themelio
labels:
app: adr0025-git-smoke
sidecar.istio.io/inject: "true"
spec:
serviceAccountName: themelio-seeder
automountServiceAccountToken: false
restartPolicy: Never
containers:
- name: git-smoke
image: alpine:3.22
command: ["/bin/sh", "-c"]
env:
- name: ADR0025_RUN_ID
value: "$ADR0025_RUN_ID"
args:
- |
set -eu
apk add --no-cache ca-certificates coreutils curl git git-lfs jq \
>/dev/null
gitlab=https://staging.gitlab.pharos.grnet.gr
actor=http://gitlab-root-actor.themelio.svc.cluster.local
username=pharos-admin
project_id=''
impersonation_id=''
impersonation_token=''
pat=''
cleanup() {
command_status=\$?
set +e
cleanup_failed=0
if [ -n "\$project_id" ] && [ -n "\$pat" ]; then
curl -fsS -X DELETE -H "PRIVATE-TOKEN: \$pat" \
"\$gitlab/api/v4/projects/\$project_id" >/dev/null \
&& echo 'repository_cleanup=ok' \
|| cleanup_failed=1
fi
if [ -n "\$pat" ]; then
curl -fsS -X DELETE -H "PRIVATE-TOKEN: \$pat" \
"\$gitlab/api/v4/personal_access_tokens/self" >/dev/null \
&& echo 'temporary_pat_revoked=ok' \
|| cleanup_failed=1
fi
if [ -n "\$impersonation_id" ]; then
curl -fsS -X DELETE \
"\$actor/users/\$user_id/impersonation_tokens/\$impersonation_id" \
>/dev/null \
&& echo 'impersonation_token_revoked=ok' \
|| cleanup_failed=1
fi
trap - EXIT
if [ "\$command_status" -ne 0 ] || [ "\$cleanup_failed" -ne 0 ]; then
exit 1
fi
}
trap cleanup EXIT
user_id="\$(curl -fsSG "\$gitlab/api/v4/users" \
--data-urlencode "username=\$username" \
| jq -er '.[0].id')"
impersonation_response="\$(curl -fsS -X POST \
"\$actor/users/\$user_id/impersonation_tokens")"
impersonation_id="\$(printf '%s' "\$impersonation_response" \
| jq -er '.id')"
impersonation_token="\$(printf '%s' "\$impersonation_response" \
| jq -er '.token')"
expires_at="\$(date -u -d '+1 day' +%F)"
pat_response="\$(jq -nc \
--arg name "adr0025-\$ADR0025_RUN_ID" \
--arg expires_at "\$expires_at" \
'{name:\$name,expires_at:\$expires_at,scopes:["api","read_repository","write_repository"]}' \
| curl -fsS -X POST \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer \$impersonation_token" \
--data-binary @- \
"\$actor/api/v4/user/personal_access_tokens")"
pat="\$(printf '%s' "\$pat_response" | jq -er '.token')"
unset impersonation_token impersonation_response pat_response
echo 'temporary_pat=created'
project_name="adr0025-smoke-\$(printf '%s' "\$ADR0025_RUN_ID" \
| tr '[:upper:]' '[:lower:]')"
project_response="\$(curl -fsS -X POST \
-H "PRIVATE-TOKEN: \$pat" \
--data-urlencode "name=\$project_name" \
--data-urlencode 'visibility=private' \
--data-urlencode 'initialize_with_readme=true' \
"\$gitlab/api/v4/projects")"
project_id="\$(printf '%s' "\$project_response" | jq -er '.id')"
project_path="\$(printf '%s' "\$project_response" \
| jq -er '.path_with_namespace')"
unset project_response
echo "repository=created path=\$project_path"
cat >/tmp/askpass <<'SH'
#!/bin/sh
case "\$1" in
*Username*) printf '%s\n' oauth2 ;;
*Password*) printf '%s\n' "\$ADR0025_GITLAB_PAT" ;;
esac
SH
chmod 700 /tmp/askpass
export ADR0025_GITLAB_PAT="\$pat"
export GIT_ASKPASS=/tmp/askpass GIT_TERMINAL_PROMPT=0
repository_url="\$gitlab/\$project_path.git"
git clone "\$repository_url" /tmp/repository >/dev/null
echo 'clone=ok'
cd /tmp/repository
git config user.name 'ADR-0025 validation'
git config user.email 'adr-0025-validation@example.invalid'
git switch -c adr0025-smoke >/dev/null
git lfs install --local >/dev/null
git lfs track '*.adr0025.bin' >/dev/null
dd if=/dev/urandom of=smoke.adr0025.bin \
bs=1024 count=64 status=none
sha256sum smoke.adr0025.bin | cut -d' ' -f1 >/tmp/source.sha256
git add .gitattributes smoke.adr0025.bin
git commit -m 'test: ADR-0025 Git and LFS smoke test' >/dev/null
git push origin adr0025-smoke >/dev/null
echo 'push_and_lfs_upload=ok'
GIT_LFS_SKIP_SMUDGE=1 git clone --branch adr0025-smoke \
"\$repository_url" /tmp/verification >/dev/null
git -C /tmp/verification lfs pull >/dev/null
sha256sum /tmp/verification/smoke.adr0025.bin \
| cut -d' ' -f1 >/tmp/verification.sha256
cmp -s /tmp/source.sha256 /tmp/verification.sha256
echo 'fresh_clone_and_lfs_download=ok'
git push origin --delete adr0025-smoke >/dev/null
echo 'branch_cleanup=ok'
YAML
kubectl wait -n themelio --for=condition=Ready \
pod/adr0025-git-smoke --timeout=180s
kubectl wait -n themelio --for=jsonpath='{.status.phase}'=Succeeded \
pod/adr0025-git-smoke --timeout=600s
While the completion wait is active, the following documented diagnostic may
be run from a second staging-master shell. It does not expose token values:
kubectl get pod -n themelio adr0025-git-smoke -o wide
kubectl logs -n themelio adr0025-git-smoke --tail=100
If the diagnostic reports Error, interrupt the completion wait with
Ctrl-C, preserve the failed log, and remove the failed pod before correcting
this document and retrying:
kubectl logs -n themelio adr0025-git-smoke \
| tee "$ADR0025_EVIDENCE/git-clone-push-lfs-failed.txt"
kubectl delete pod -n themelio adr0025-git-smoke --wait=true
After the completion wait succeeds, continue in its original shell:
kubectl logs -n themelio adr0025-git-smoke \
| tee "$ADR0025_EVIDENCE/git-clone-push-lfs.txt"
kubectl delete pod -n themelio adr0025-git-smoke --wait=true
Pass when the log records temporary_pat=created, clone=ok,
push_and_lfs_upload=ok, fresh_clone_and_lfs_download=ok, and
branch_cleanup=ok, followed by repository_cleanup=ok,
temporary_pat_revoked=ok, and impersonation_token_revoked=ok. Preserve pod
logs before deletion if the test fails, then verify that the PAT, impersonation
token, and disposable repository were removed before retrying.
APP-04: Kubernetes deployment action
Mutating: create and then remove a disposable deployment/job through the
Themelio UI, not directly with kubectl. In parallel, observe:
export ADR0025_TENANT_NAMESPACE='<disposable-staging-project-namespace>'
kubectl get pods,jobs,deployments,inferenceservices -n "$ADR0025_TENANT_NAMESPACE" -w
In another terminal:
kubectl logs -n themelio deploy/k8s-root-actor --since=20m \
| tee "$ADR0025_EVIDENCE/k8s-root-actor-action.log"
Pass when create, status updates, logs, and deletion work from the UI and the
actor remains ready.
APP-05: Waldur reconciliation
Mutating: approve a disposable staging order and add/remove a disposable
member in Waldur. Observe both real processors:
kubectl logs -n waldur-site-agent-cloud \
deploy/waldur-agent-cloud-order-process -f --since=5m
kubectl logs -n waldur-site-agent-cloud \
deploy/waldur-agent-cloud-membership-sync -f --since=5m
After reconciliation:
kubectl get namespace "$ADR0025_TENANT_NAMESPACE" --show-labels
kubectl get resourcequota,serviceaccount,rolebinding \
-n "$ADR0025_TENANT_NAMESPACE"
kubectl logs -n themelio deploy/gitlab-root-actor --since=20m \
| tee "$ADR0025_EVIDENCE/waldur-gitlab-reconciliation.log"
Pass when the namespace/quota and GitLab groups/membership match the Waldur
state. Remove the disposable order/project using the supported workflow.
APP-06: Kubeflow and external authorization
curl -skI https://staging.kubeflow.pharos.grnet.gr/ \
| tee "$ADR0025_EVIDENCE/kubeflow-auth-response.txt"
kubectl logs -n oauth2-proxy -l app.kubernetes.io/name=oauth2-proxy \
--all-containers --prefix --since=10m \
| tee "$ADR0025_EVIDENCE/kubeflow-oauth2-proxy.log"
Then log in through a browser and open an authorized Kubeflow page. Pass when
unauthenticated access enters the external-authorization/login flow, authorized
access works, and the existing CUSTOM policy continues to operate alongside
the scoped host ALLOW policy.
APP-07: registry pull and push
Mutating: creates one disposable in-cluster pod and pushes one disposable
tag. The pod reads the existing registry credentials through Secret references;
the commands never decode or print them. The pod and its temporary credential
file are deleted after the test. The pushed tag is left for the normal registry
retention workflow.
kubectl delete pod -n registry adr0025-registry-smoke \
--ignore-not-found --wait=true
cat <<YAML | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: adr0025-registry-smoke
namespace: registry
spec:
restartPolicy: Never
automountServiceAccountToken: false
containers:
- name: crane
image: gcr.io/go-containerregistry/crane:debug
command: ["/busybox/sh", "-c"]
args:
- |
set -eu
registry=staging.registry.pharos.grnet.gr
source="\$registry/themelio/web:staging"
target="\$registry/themelio/web:adr0025-$ADR0025_RUN_ID"
crane auth login "\$registry" \
-u "\$REGISTRY_USER" -p "\$REGISTRY_PASSWORD" >/dev/null
echo "source_digest=\$(crane digest "\$source")"
crane pull "\$source" /tmp/source.tar
echo "pull_source=ok bytes=\$(wc -c </tmp/source.tar)"
crane push /tmp/source.tar "\$target"
echo "push_test=ok"
crane pull "\$target" /tmp/test.tar
echo "pull_test=ok bytes=\$(wc -c </tmp/test.tar)"
test "\$(crane digest "\$source")" = \
"\$(crane digest "\$target")"
echo "digest_match=ok"
env:
- name: REGISTRY_USER
valueFrom:
secretKeyRef:
name: registry-auth
key: k8s_user
- name: REGISTRY_PASSWORD
valueFrom:
secretKeyRef:
name: registry-auth
key: k8s_pass
YAML
kubectl wait -n registry --for=condition=Ready \
pod/adr0025-registry-smoke --timeout=180s
kubectl wait -n registry --for=jsonpath='{.status.phase}'=Succeeded \
pod/adr0025-registry-smoke --timeout=600s
kubectl logs -n registry adr0025-registry-smoke \
| tee "$ADR0025_EVIDENCE/registry-pull-push.txt"
kubectl delete pod -n registry adr0025-registry-smoke --wait=true
Pass when the log records pull_source=ok, push_test=ok, pull_test=ok, and
digest_match=ok. Preserve logs before deleting the pod if it fails.
APP-08: Grafana and Prometheus
The staging reverse proxy restricts both UIs by source CIDR. Test from an
allowed network:
curl -skI https://staging.grafana.pharos.grnet.gr/ \
| tee "$ADR0025_EVIDENCE/grafana-public-response.txt"
curl -skI https://staging.prometheus.pharos.grnet.gr/ \
| tee "$ADR0025_EVIDENCE/prometheus-public-response.txt"
Open both UIs in a browser. Confirm the ADR-0025 sidecar dashboard loads and
Prometheus has ready sidecar, istiod, and gateway targets. A public 403 from a
non-allowed source tests the Nginx CIDR restriction, not the Istio policy; use
the direct gateway Service ClusterIP test in section 2 for Istio evidence.
APP-09: readiness probes and alert definitions
kubectl get deploy -A \
-o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas' \
| tee "$ADR0025_EVIDENCE/deployment-readiness.txt"
kubectl get prometheusrule -n monitoring istio-sidecar-security -o yaml \
| tee "$ADR0025_EVIDENCE/istio-alert-rules.yaml"
kubectl get podmonitor -n monitoring \
istio-workload-sidecars istio-control-plane istio-gateways -o wide \
| tee "$ADR0025_EVIDENCE/istio-podmonitors.txt"
Pass when protected workloads and platform components remain ready, all three
Istio scrape definitions exist, and the ADR-0025 alert classes are loaded:
control-plane availability, gateway host rejection, workload-identity denial,
mTLS failure, and application authorization rejection.
9. Tenant workload authorization
Validates the scoped tenant-namespace policies that k8s-root-actor generates
per Themelio deployment and inference service (DD-0025 §9). The positive path —
the workload is reachable through the Themelio service gateway — is already
exercised by APP-04 and the Space route in GW-01; this section verifies the
security behavior that a happy-path UI test cannot: that the policy is scoped,
enforced, denies access from outside the tenant namespace, and is cleaned up.
Each generated <name>-service-gateway policy admits only the Themelio
service-gateway principal on the workload port; ns-owner-access-istio retains
same-namespace access, and the former namespace-wide allow-all is removed.
Prerequisites: the mesh runs native sidecars
(ENABLE_NATIVE_SIDECARS=true, DD-0025 §2.2), and one disposable deployment and
one disposable inference service have been created from the UI (APP-04) in a
disposable tenant namespace. Set the placeholders:
export ADR0025_TENANT_NAMESPACE='<disposable-staging-project-namespace>'
export ADR0025_DEPLOYMENT='<deployment-name>'
export ADR0025_INFERENCE='<inferenceservice-name>'
export ADR0025_GATEWAY_PRINCIPAL='cluster.local/ns/istio-system/sa/istio-ingressgateway-themelio-service-account'
TENANT-01: scoped policies exist and carry no namespace-wide rule
kubectl get authorizationpolicy -n "$ADR0025_TENANT_NAMESPACE" \
-o custom-columns=NAME:.metadata.name,ACTION:.spec.action \
| tee "$ADR0025_EVIDENCE/tenant-authz-policies.txt"
# Full shape of the two generated policies: selector, admitted principal, port.
kubectl get authorizationpolicy \
"${ADR0025_DEPLOYMENT}-service-gateway" \
"${ADR0025_INFERENCE}-service-gateway" \
-n "$ADR0025_TENANT_NAMESPACE" -o yaml \
| tee "$ADR0025_EVIDENCE/tenant-scoped-policies.yaml"
# No ALLOW policy in the namespace contains an empty (match-all) rule.
kubectl get authorizationpolicy -n "$ADR0025_TENANT_NAMESPACE" -o json \
| jq '[.items[]
| select(.spec.action=="ALLOW")
| select((.spec.rules // []) | index({}))
| .metadata.name]' \
| tee "$ADR0025_EVIDENCE/tenant-namespace-wide-rules.json"
Pass when each <name>-service-gateway policy selects its workload (the
deployment-runner labels, or the KServe serving.kserve.io/inferenceservice +
component: predictor labels), admits only $ADR0025_GATEWAY_PRINCIPAL on the
workload port, and the empty-rule array is [].
TENANT-02: enforcement denies an identity outside the tenant namespace
The retained ns-owner-access-istio policy allows all same-namespace traffic
(source.namespace == <tenant>), so an in-namespace caller is admitted by
design. The scoped policy's contribution is blocking callers from outside the
tenant namespace that are not the service gateway — the access allow-all used
to permit. The deny test therefore runs from a foreign namespace.
First confirm the predictor carries the Envoy proxy as a native sidecar —
without it the policy is silently inert:
predictor_pod="$(kubectl get pod -n "$ADR0025_TENANT_NAMESPACE" \
-l "serving.kserve.io/inferenceservice=${ADR0025_INFERENCE},component=predictor" \
-o jsonpath='{.items[0].metadata.name}')"
kubectl get pod -n "$ADR0025_TENANT_NAMESPACE" "$predictor_pod" \
-o jsonpath='{.spec.initContainers[*].name}{"\n"}' \
| tee "$ADR0025_EVIDENCE/tenant-predictor-initcontainers.txt"
istio-proxy must appear among the init containers.
Mutating: create one disposable pod in a foreign (non-tenant) namespace and
confirm it is denied on each workload's port:
export ADR0025_FOREIGN_NAMESPACE='themelio'
kubectl -n "$ADR0025_FOREIGN_NAMESPACE" run adr0025-foreign-denied \
--restart=Never --image=curlimages/curl:8.12.1 --command -- sleep 3600
kubectl -n "$ADR0025_FOREIGN_NAMESPACE" \
wait --for=condition=Ready pod/adr0025-foreign-denied --timeout=180s
for target in \
"http://${ADR0025_DEPLOYMENT}.${ADR0025_TENANT_NAMESPACE}.svc.cluster.local/" \
"http://${ADR0025_INFERENCE}-predictor.${ADR0025_TENANT_NAMESPACE}.svc.cluster.local/"; do
set +e
code="$(kubectl exec -n "$ADR0025_FOREIGN_NAMESPACE" adr0025-foreign-denied -- \
curl -sS -o /dev/null -w '%{http_code}' \
--connect-timeout 5 --max-time 10 "$target")"
rc=$?
set -e
printf '%s http_code=%s rc=%s\n' "$target" "$code" "$rc"
[ "$rc" -ne 0 ] || [ "$code" = 403 ]
done |& tee "$ADR0025_EVIDENCE/tenant-unauthorized-denials.txt"
Pass when both targets return 403 (or a refused/reset connection). A same-
namespace caller returning 200 is expected and is not a failure.
Pass only when both targets are denied — a refused/reset connection (non-zero
rc) or an RBAC 403. A 200 proves the policy is not enforcing and is a
failure. Preserve logs, then clean up:
kubectl delete pod -n "$ADR0025_TENANT_NAMESPACE" adr0025-tenant-denied --wait=true
TENANT-03: authorized access through the service gateway succeeds
The counterpart to TENANT-02: prove the policy admits the right identity, not
just that it denies the wrong one. Route to each workload through the Themelio
service gateway (whose ServiceAccount is the admitted principal) and confirm the
application answers rather than the destination Envoy rejecting with an RBAC
403. Discover each workload's host and route path from its VirtualService, and
reuse the Themelio service gateway Service ClusterIP that GW-01 already found for
the Space route:
export ADR0025_SERVICE_GATEWAY_IP='<Themelio service-gateway Service ClusterIP from GW-01>'
for entry in \
"${ADR0025_DEPLOYMENT}-vs" \
"${ADR0025_INFERENCE}-predictor-vs"; do
host="$(kubectl get virtualservice "$entry" -n "$ADR0025_TENANT_NAMESPACE" \
-o jsonpath='{.spec.hosts[0]}')"
path="$(kubectl get virtualservice "$entry" -n "$ADR0025_TENANT_NAMESPACE" \
-o jsonpath='{.spec.http[0].match[0].uri.exact}')"
code="$(curl -sS -o /dev/null -w '%{http_code}' -H "Host: $host" \
--connect-timeout 5 --max-time 15 \
"http://${ADR0025_SERVICE_GATEWAY_IP}${path}" || true)"
printf '%s\t%s%s\t%s\n' "$entry" "$host" "$path" "$code"
done |& tee "$ADR0025_EVIDENCE/tenant-authorized-through-gateway.txt"
Pass when each request reaches the normal application or authentication
behavior. A gateway/RBAC 403 is a failure — the policy is wrongly denying the
gateway identity. Distinguish the two: an Envoy RBAC denial returns the body
RBAC: access denied with no application content, so capture headers and body
for any 403:
curl -si -H "Host: <workload-host>" \
--connect-timeout 5 --max-time 15 \
"http://${ADR0025_SERVICE_GATEWAY_IP}<workload-path>" \
| tee "$ADR0025_EVIDENCE/tenant-authorized-response.txt"
TENANT-04: teardown removes the scoped policy
Delete the disposable deployment and inference service through the UI, then
confirm no <name>-service-gateway policy is orphaned:
kubectl get authorizationpolicy -n "$ADR0025_TENANT_NAMESPACE" \
| grep -E "${ADR0025_DEPLOYMENT}-service-gateway|${ADR0025_INFERENCE}-service-gateway" \
| tee "$ADR0025_EVIDENCE/tenant-policy-teardown.txt" \
|| echo "no orphan service-gateway policies" \
| tee "$ADR0025_EVIDENCE/tenant-policy-teardown.txt"
Pass when neither policy remains after the UI deletion completes.
10. Final review
Do not mark the run accepted solely because the positive calls passed. Review
every required row in the top-level checklist and record links or paths to the
evidence.
printf 'finished_utc=%s\n' "$(date -u +%FT%TZ)" \
| tee -a "$ADR0025_EVIDENCE/run.txt"
find "$ADR0025_EVIDENCE" -maxdepth 1 -type f -printf '%f\n' | sort
Validation succeeds only when:
- every required row is
PASS;
- conditional seeder coverage is either
PASS or documented as
NOT APPLICABLE;
- no unexplained regression, missing denial evidence, or unexpected principal
remains;
- temporary pods, branches, and application resources have been removed;
- credentials used during testing have been unset, and any credential exposed
outside an approved secret store has been rotated.
Page source
Source snapshot · Revision f928834c8.
