# Run logs: from the Site to the member Status: partially implemented (#1056), updated 6 November 2026. The current-viewer request, upload and browser path is connected and disabled by default. Native Slurm storage requires the separate Site activation procedure. A failed service's final copy is kept for seven days. The original design decisions and proposals below retain their review status; implementation alone does not establish deployment or live qualification. Scope: how a member reads a Run's logs through the API. It covers how the text travels from the Site to the platform, what it contains, how long it is kept and who may read it. The Run page's log panel shows each line's recorded timestamp. Its logs come from a small hook that polls the logs route, and the URL it loads changes (UI work on #1012). Following logs live over realtime is left for later; for now the panel polls (points 9 and 13). ## What exists today | Fact | Where | | --- | --- | | A permit names one viewer, Run, Attempt, kind (`output`, `logs`, `endpoint`), path and byte limit. It lasts 60 seconds and allows at most 4 MiB. A `logs` permit has an empty path. | `contexts/execution_core/domain/private_read.py`, `application/private_reads.py`; [private-reads.md](../../../contexts/execution_core/private-reads.md) | | `POST /api/v1/runs/{runId}/read-permits` needs the Attempt's ID, which the Run detail carries since #1059. `GET /api/v1/run-read-permits/{permitId}` returns only the permit, never bytes. | [private-reads.md](../../../contexts/execution_core/private-reads.md) | | Kubernetes reads the owned Pods' init and main containers, including previous instances after restarts. The shared reader bounds the aggregate bytes and labels sources and truncation. | `site_runtime/adaptors/kubernetes_log_reader.py`, `kubernetes_workspaces.py`, `kubernetes_service_data.py` | | A service's logs are refused once it is not active (`service-not-active`). A failed service is terminated and cleaned up before its failure is reported. | `kubernetes_services.py` (`_finish_failure`, `_observe`) | | The Runtime's `LogCommandExecutor` calls the native readers and uploads their bounded result. It has no inbound server: it pulls Site Coordination commands and acknowledges each. | `site_runtime/adaptors/run_log_commands.py`, `services/site_runtime.py`, `application/worker.py` | | Commands form one ordered stream per environment. The Runtime handles them one at a time. An `ExecutionDeferred` keeps the command pending, so every later command waits. A `CommandRejected` is a definite failure that is recorded and acknowledged, and the stream moves on. | `application/worker.py`, `domain/model.py`, [ordered-commands.md](ordered-commands.md) | | Read commands carry the permit's `validUntil` in their payload. The reader shares a ten-second deadline across native, authority and upload calls. | `contracts/http/site-execution.v1.yaml`, `site_runtime/adaptors/log_deadline.py` | | Ordinary Site API requests and acknowledgements retain their 64 KiB limit. The dedicated `site-api-run-logs` route admits an encoded upload of at most 5 MiB, with at most 4 MiB of message text. | `site_coordination/adaptors/site_http.py`, `gitops/manifests/routes/base/services.yaml` | | Run realtime messages go to the Project audience, authorised with `runs.view`. Logs need `runs.output.view`. Browser delivery uses server-authorised Centrifugo, and polling and client-selected subscriptions are not allowed, with one exception for the log panel (see Browser polling for the log panel). | `CLIENT-SUBSCRIPTIONS.md`, `AGENTS.md` | | The panel renders the recorded timestamp, level, message and source status. `COMMAND_OUTPUT` has no status label and collapses carriage-return overwrites. An empty read shows "Waiting for Run logs"; a refused read clears previously displayed bytes. | `compat/themelio/services/web/src/components/logs/LogConsole.tsx`, `hooks/useRunLogs.ts`, `components/resources/ResourceDetailView.tsx` | | Run detail reads follow owner updates and explicit Refresh. Only the separate log hook polls, every five seconds, with one request in flight and cancellation on close. | `components/resources/ResourceDetailView.tsx`, `hooks/useRunLogs.ts`, `scripts/check_architecture.py` | | New Slurm Attempts may use the dedicated spool; legacy scratch files remain unavailable. | `site_runtime/adaptors/slurm_log_spool.py` | ## Decisions and open points | # | Point | Status | | --- | --- | --- | | 1 | Log text travels by a read command and a separate upload route that the Runtime calls outward | Decided | | 2 | The upload is bound to its read command and permit and is discarded when the permit expires | Decided | | 3 | A slow log read must not hold up launches or cancellations | Decided | | 4 | The upload route has its own route in the route catalogue and its own limit | Decided | | 5 | Limits, deadlines and the cap on unanswered reads | Implemented | | 6 | Where permit-scoped bytes wait, and when they are removed | Implemented; final copies have a table of their own | | 7 | What the logs contain, how each part is labelled, how they are truncated, and the shape of the answer | Implemented | | 8 | A bounded final copy of a failed service's logs | Implemented | | 9 | How the member receives the text: the panel polls a logs route every 5 seconds, for now | Decided | | 10 | Slurm: which identity reads, how the file is bound, how long logs are kept | Implemented with separate Site activation prerequisite | | 11 | How long a final copy is kept | Implemented: seven days, for review | | 12 | Missing and forbidden look the same at every step | Decided | | 13 | The polling has a narrow exception to the no-polling rule, by file | Decided | | 14 | The route answers at once from the latest copy and requests a read when that copy is old | Implemented | | 15 | One copy per Attempt or one per member | Open | | 16 | The read command: one kind for each executor | Implemented | ## Implemented request and upload flow 1. The panel makes one GET to the Run's logs route, every 5 seconds while it is open. It does not ask for a permit and it does not choose an Attempt. 2. A read-only owner query checks current `runs.view` and `runs.output.view`, resolves the saved Run's latest Attempt and returns that viewer's latest unexpired copy. 3. When no copy exists or its capture is at least ten seconds old, the API sends a separate `request_run_logs` command for the exact queried Attempt. The command identity is also the permit identity. Its sixty-second lifetime and native target are immutable on retry. The query itself creates no work. Site Coordination admits at most four pending log reads per environment, independently of ordinary execution commands. 4. Execution Core appends a read command (`queued-batch.read-logs` or `reconciled-service.read-logs`, see The read command) to the Site's stream. Its payload names the permit, Run, Attempt, native material digest, viewer, Project, byte limit and the permit's `validUntil`. The deadline is in the payload because the command envelope has none. 5. The Runtime pulls the command. If `validUntil` has passed it answers `CommandRejected("log-read-expired")` without reading anything. Otherwise it confirms the permit with the Site API (`authorise-read`, as the other reads do), reads within its bounds and uploads the result. 6. The upload route checks that the command is an unanswered read command of that Site, that the permit has not expired and that the body is within the limit. Execution Core keeps the bytes in its private log-copy table, bound to that permit. 7. The Runtime acknowledges the command. Its result is small: the byte count, whether it was truncated, the capture time and a digest. It never carries log text, because a result cannot exceed 64 KiB. The next poll finds the new copy. Every step that can refuse answers the member as `not_found`, whether the Run, the Attempt or the permit is missing or the member has no right to it (point 12). The Runtime's own failure codes (`log-read-expired`, `log-read-timeout`, `log-source-unavailable`, `log-read-too-large`) stay between the Site and Execution Core. ## The read command (point 16) **Proposal.** One command kind for each executor that can read a Run's logs, not one kind for all. - Each executor of the Runtime accepts one kind prefix and refuses the others: the Kubernetes Job executor `queued-batch.`, the Kubernetes service executor `reconciled-service.`, and the two Slurm executors `queued-batch.` in their own environment (`kubernetes_jobs.py`, `kubernetes_services.py`, `slurm_jobs.py`, `slurm_chain.py`). A single `run-logs.read` would match none of them. - The commands are `queued-batch.read-logs` for a Job (Kubernetes now, Slurm with M6) and `reconciled-service.read-logs` for a service. Each executor gets one more branch, which calls the `read_logs` it already has. - The payload and the failure codes are the ones in the Flow. Both kinds are added to the Site contracts (`contracts/http/site-coordination.v1.yaml`, `site-execution.v1.yaml`) the way the existing kinds are. ## The upload route (points 2, 4 and 5) **Decided.** The route is bound to the read command and its permit. A Site can upload only for a read it was asked for, and the upload is thrown away when the permit expires. The route needs its own entry in `gitops/manifests/routes/base/services.yaml` with a larger body limit, as well as a larger limit for it in `site_http.py`. Raising the limit in `site_http.py` alone would leave nginx refusing at 64k. **Proposal.** - Path `/v1/site/commands/run-logs/{commandId}`: on the same host, inside the `/v1/site/commands/` prefix that the Sites already use, with the same `pharos-site` profile and Runtime credential as the other Site routes. It has an entry of its own in the route catalogue (see below), because it needs a larger body limit than `site-api-commands`. - A JSON document of `LogEntry` lines (see What the logs contain). The limit applies to the encoded body and is 5 MiB, so a 4 MiB permit still fits after escaping. The Runtime cuts the oldest text until the encoded body fits. - One upload per command. A repeat with the same digest is a replay; a different body is refused. - Anything else is refused: another Site, an answered command, an expired permit, a body over the permit's byte limit. Checked against the code. `devops/ingress/default.conf` is the nginx of the platform's own web and API and does not serve `/v1/site/`. The Site routes are defined in the route catalogue, and the dev box renders each route's `body.limit` into its edge. The catalogue exposes only `/v1/site/commands/` and `/v1/site/ssh-access/`, and the Sites are told to append only those ([secure-site-communication.md](secure-site-communication.md)). So the upload route stays inside the first prefix, as an entry of its own with a larger body limit. The route chart puts the longest prefix first, as it does for the Kubeflow routes inside `/`, and the Site API checks a credential in its handlers and not by path. The entry goes in `devops/platform/scripts/platform_edge/catalogue.py` (`EXPOSURES`, `SURFACES`, `BACKEND_TLS`), `gitops/manifests/routes/base/services.yaml` and the dev-box values. The Site routes have a 60 second timeout, which a read of 10 seconds and its upload fit in. ## Not holding up launches and cancellations (points 3 and 5) **Decided.** A slow read must not delay launches or cancellations. It matters because the stream is ordered and the Runtime takes one command at a time. **Proposal.** - A log read never ends in `ExecutionDeferred`. Every failure is a `CommandRejected` with a code, so the command is acknowledged and the stream moves on. Two places defer today: the read confirmation with the Site API (`authorise_read`), when the Site API cannot answer, and the Kubernetes reader, when `kubectl` fails. The read command's handler turns both into a `CommandRejected`. If either escaped, a Site API or kubelet blip would hold the stream like any other deferral. - A read has one deadline of 10 seconds in total, including the upload. The Runtime does not start one when the permit has less than that left. Each native call gets the time that is left, and a source that would start after the deadline is skipped, as an unreadable source is. The current main container is still read. The work on the Kubernetes readers checks this. - Execution Core appends a new read only when at most 4 reads of that environment are unanswered. When 4 are already waiting, the route answers with the last copy it has, not with an error, and starts no read. In the worst case a launch waits for 4 reads of 10 seconds each. Both numbers are starting values to tune; a read normally takes well under a second. - If measurements show that this delay is not acceptable, log reads get a stream of their own. That is a larger change in Site Coordination, which has one stream per environment today. ## Where the bytes wait (point 6, proposal) Execution Core stores the uploaded bytes next to the permit, in its own aggregate store, with the permit's `validUntil`. A newer copy replaces the older one. The latest copy is kept until a newer one replaces it or its permit ends, and a sweep removes what is left. It is not deleted once it has been fetched, because the panel fetches it again every 5 seconds. The bytes are never written to a log, an event or a receipt. An expired or foreign permit finds nothing. ## What the logs contain (point 7, proposal) - **Sources.** The main container of each Pod; its init containers; and the previous, crashed instance of a container when it has restarted. For a service, each Pod separately. For a Job, each Pod it has had. - **The answer.** The existing `Log` list of the HTTP contract, which the panel already renders, and not a document of parts: `timestamp`, `level`, `message` and `source_status`. - **Program output** is one entry per line, with `source_status: COMMAND_OUTPUT`. It is the one value that the panel prints with no label and with carriage returns collapsed. `level` is `info`, because the Runtime cannot tell the level of a line. The Runtime asks Kubernetes for line timestamps and the entry takes the line's own. - **One entry per line** matters because the panel scrolls down only when the number of entries changes. - **Labels** are entries of their own. Where the source changes, one entry names the Pod, the container and whether it is the current or the previous instance (for example `source_status: LOG_SOURCE`, `message: pod web-0, container main, previous instance`). A part that was cut gets an entry that says how many bytes it left out (`source_status: LOG_CUT`, `level: warning`, `message: cut 123456 bytes`). A Slurm part names its stage and job instead (point 10 is open). - The `timestamp` of a label entry is the capture time, because the contract requires one. - **Capture time.** One time for the whole read, taken when the Runtime reads. - **Truncation.** The newest text is kept. The budget (the permit's byte limit) is shared equally between the parts, and a part that needs less gives the rest back. - **Where it is served.** Beside the Run detail routes, at `…/jobs/{id}/logs` and `…/deployments/{id}/logs`, so that the panel's change is the URL it calls and one function next to `getResource` in `resourceProxy.ts`. The `logs` list of the Run detail itself is not used by the panel any more; whether it stays is for the API route. To settle with the panel change: `LogConsole` keys each line by its timestamp, so lines that share one, such as several labels with the capture time, repeat a key. Either the route gives each entry a distinct timestamp or the panel keys by position. ## The numbers (points 5 and 14, proposal) The panel polls every 5 seconds. The two figures of the review (10 seconds and 20 seconds) are taken as starting values, with these reasons. - **A copy is old after about 10 seconds.** A read takes one to three seconds in the usual case, because the Runtime pulls once a second. Two polls (at 5 and 10 seconds) still find the copy young, and the third (at 15) finds it old and asks for a new read. So a read is asked for about every 15 seconds, a viewer costs about four reads a minute, and the member sees a copy that is up to about 15 seconds old. - **The next permit is issued with 30 seconds left, not 20.** The new copy has to arrive before the old one is discarded, or the panel receives an empty list and shows "No logs yet". With 20 seconds a single slow read still fits (up to 10 seconds), but not a read that waits behind other reads in the stream, which the cap of 4 allows to be up to 40 seconds. 30 seconds covers one slow read and some waiting. It does not remove the gap in the worst case; the cap and the answer from the last copy limit how often it can happen. ## A failed service's final logs (points 8 and 11) A failed service is cleaned up before its failure is reported, so its logs are gone when they are needed most. - **When.** The Runtime reads the copy in the observation that finds the Deployment or KServe service failed, from a crash loop or from its native conditions, just before its cleanup removes the Pods. It reads once for the Attempt and marks its record (`finalLogs`), so a cleanup that is still pending or a restarted Runtime does not read again. A service that is cancelled, a batch Job and a Slurm job have no final copy. - **What.** The same readers and Log lines as a read command: every Pod and container, current and previous instance, at most 1 MiB of text in total with the newest kept. The read and the upload share one deadline of 10 seconds. No member asked for this read, so it has no read permit: the execution's accepted authority is checked before and after it, as for a native effect, and every native identity check of a member's read stays. A member's read still needs its permit. - **Where it goes.** `POST /v1/site/commands/run-logs/final/{runId}/{attemptId}`, with the headers and the 5 MiB limit of the upload of a read command. It is under the same prefix, so the route catalogue and the edge need no entry of their own. No command asks for the copy. The Site API accepts it only for a started service of the Runtime's own environment whose Attempt has not ended yet, which holds because the Runtime reports the failure after the cleanup. The first copy is kept; the same body again is a replay and a later one is ignored. - **Where it is kept.** `pharos.final_run_log_copies`, one row per Attempt, apart from the copies kept for permits. - **How it is read.** The logs route checks the viewer's current permission first, as for any copy. For a service whose Attempt has ended it answers from the final copy and asks for no read, and without a final copy it asks for none either, because the service is gone and the read would only be refused. A batch Job that has ended is still read from the Site. - **When it fails.** Any failure of the read or the upload records `snapshot-unavailable` in the Runtime's record and the cleanup goes on: a failed capture never keeps a service alive, holds the commands behind it or stops the Runtime. The member then sees no logs for that Attempt. - **How long (point 11).** Seven days from the capture (`retain_until`), the same period as the CPU qualification setting of the Slurm spool. The code has no retention for a Run's accepted work to follow. The next final upload removes the copies that have ended, and a read ignores them. The period is for review. ## One copy per Attempt or per member (point 15, open) A permit names one viewer, so a copy kept next to the permit belongs to one member. If several members watch the same Run, each of them causes a read about every 15 seconds in the same ordered stream, and the cap of 4 unanswered reads is shared among them. One copy per Attempt, checked against each reader's current permission when it is served, would make the reads depend on the Attempt and not on the number of viewers. It changes where the bytes are kept and what a permit means for reading, so it needs a decision. Until then this note assumes one copy per permit. ## Slurm (point 10, scoped implementation) New Attempts can use a root-managed per-Run log spool. Runtime receives only the exclusive reader group; it gains no sudo, ACL, LDAP or member-scratch access. The root Prolog creates fixed `0640` output and identity receipts below `0751` root-owned directories. Runtime compares its saved submission with the exact scheduler job, actual output paths and receipt. Completed jobs require output sealed to root ownership after terminal scheduler evidence. Missing or legacy receipts fail closed. The Site must enforce numeric-identity retention while member-owned output exists; the current Waldur deletion path can recycle released UIDs. This is a fail-closed activation prerequisite, separate from the root finalisation hook. Retention is configured explicitly; seven days is the CPU qualification setting. The [Site procedure](../../../../../devops/hpc/site/RUN-LOGS.md) owns the precise permissions, installation and removal behaviour. ## Browser polling for the log panel Decided on 4 November 2026 with Vasilis Danias. For now the panel polls the logs route every 5 seconds while it is open. A realtime notification can replace the polling later: a message targeted at the requesting principal that says the content is ready, followed by one fetch. Project realtime cannot carry the text, because it is authorised with `runs.view` and the text needs `runs.output.view`, so that option needs a new principal-targeted message type. It is not needed yet. The rule against polling (`AGENTS.md`, `CLIENT-SUBSCRIPTIONS.md`, `ARCHITECTURE.md`) has one narrow exception for this panel. It is made in `scripts/check_architecture.py` by file, not by content, and the file is a new small hook of the retained web, `hooks/useRunLogs.ts`, that polls the logs route and nothing else. It is not the detail view, `ResourceDetailView.tsx`: that file still polls the Run detail itself today, #1012 replaces that polling with Centrifugo, and allowing the whole file would let it come back unnoticed. The detail view is in the negative test of `scripts/test_architecture.py`, with every other file, so any `setInterval` outside the hook still fails. A client-selected subscription stays forbidden in the hook too. Polling stays out of `lib/runs/` and `lib/resourceProxy.ts`, which have a second rule against it. `docs/compat-exceptions.json` does not fit: each of its entries is pinned to the file's exact content, so it would need updating on every edit of the panel. The exception is a separate small change that comes before the change to the panel. ## Tests The checks in #1056 apply: a service, a batch Job, a failed Job and service and a Slurm job, each read as a member allowed to read the output, as a member who is not and as someone outside the Project. They also cover a permit that expires during the read, a permission removed after issue, another viewer's permit, a Run and Attempt that do not match, output that is too large, a Runtime restart and a late delivery. Missing and forbidden must look the same when issuing a permit, reading a permit and reading the logs, and a slow read must not delay a launch or a cancellation. The polling adds these: a poll with a fresh copy puts no command in the stream; a poll with an old copy puts exactly one; polls while a read is running put none; with 4 reads waiting the route answers with the last copy; and the answer does not turn empty when the permit is renewed. ## Steps | Step | Content | Depends on | | --- | --- | --- | | M1 | The Attempt's ID in the Run detail (#1059). It stays as it is: the permit routes still need the Attempt's ID, but the panel does not use it | | | M2 | This note | | | Before the panel change | The polling exception, a separate small change | Point 13 | | M3 | The Kubernetes readers: sources, labels, truncation, the final copy | Points 7, 8, 11 | | M4a | The upload route: the Site contract, the Site API handler and its dev-box script, where the bytes wait (point 6) with its database migration, and the entry in the route catalogue | Points 2, 4, 5, 6, and #1076, which changes the same contract and worker | | M4b | The read commands in the Site contracts and in Execution Core: issuing them and binding them to the permit and the Attempt | Points 1, 3, 14, 16, and #1076 | | M4c | The Runtime: the executors read, upload and acknowledge | M3, M4a, M4b | | M5 | The API route that returns the logs as `LogEntry` lines | Points 5, 6, 7, 14 and 15 | | M6 | The Slurm reader | Point 10 | | M7 | The checks on a dev box | M3 to M6 | ## Source verification for the connected log path Run `pnpm verify` and `pnpm test:integration --group metallo` in the owned qualification checkout. The retained UI additionally uses `pnpm --dir compat/themelio/services/web type-check`, its existing `test` and `build` commands, and `pnpm exec playwright test --config acceptance/browser/run-logs.config.ts`. The browser fixture renders the actual Run panel and checks refresh, revoked-access clearing, route restrictions and cleanup. The root filesystem lane is documented in the Site procedure. These fixture results do not establish a live M18-23 pass.