chore(deps): update all minor updates

chore(paperless): keep deployment inactive
feat(homelab): isolate PR runner and add Paperless
2026-10-09 04:19:00 +00:00 · 2026-10-09 01:02:33 +02:00 · 2026-10-09 01:01:35 +02:00 · 2026-10-08 23:57:55 +02:00 · 2026-10-08 19:15:52 +00:00 · 2026-10-08 19:15:52 +00:00
65 changed files with 1466 additions and 155 deletions
@@ -1,42 +1,50 @@
 # EDU ownership handoff

-## Current status
+## Status

-EDU PR #1 merged at 2026-10-07 08:04:30 UTC. Main release `5094952464ce315130839303985fd04d721bc1f2` passed CI run 1585 and deploy run 1586. The workstation checkout `/srv/edu-master` is at that SHA. The release changed the application image digests:
+The EDU ownership handoff is complete. The homelab repository no longer owns
+EDU workloads, images, routes, alerts, or deployment selection. The EDU
+repository is the only deployment owner: [forust/edu-master](https://git.forust.xyz/forust/edu-master).

- Session keeper: `sha256:1e59473bd40fe4c22622017d808a8927a68788275fe073dc23d718c44b2fd5dd`
- Webinar checker: `sha256:987d9bf0770272766523ea5b94c7f3f849175d737551d46591ae55e058cf9f12`
+Homelab PRs #99 and #105 are merged. PR #105 removed the EDU subtree and its
+build, deploy, rollback, verification, route-probe, and registry references.
+It also added the serial image build matrix for the homelab services. This
+handoff record is the only remaining EDU-specific file in homelab Git.

-The live workloads remain healthy in context `Default`, namespace `edu-master`. Both health and live probes return 200. Redis AUTH passes, session TTL is 1178 seconds, the delivery backlog is zero, all nine EDU alert rules are healthy, and the scrape target is UP. The unauthorized-pod Redis check passed. The Redis PVC UID and Secret UID and values, including the Fernet key, match their pre-release state.
+The dedicated workstation checkout is `/srv/edu-master`, at release
+`4f2b2a0e37dc11ac2c75441a15076c178e219d37`. It contains `k8s/active`; root
+`active` is absent. The old untracked `/srv/homelab/edu_master` checkout was
+moved outside the homelab repository to
+`/srv/edu-master-legacy-archive-20261007/edu_master`. Its private files remain
+mode `0600` inside an archive directory with mode `0700`. The homelab deploy
+checkout has no EDU marker or tracked EDU application/deployment files.
+`AUTODEPLOY=false` remains in place for homelab deployment.

-The homelab EDU active marker was present after the EDU deployment. It was moved to the private snapshot as `homelab-k8s-active.marker` while holding `/tmp/homelab-apply.lock`. The homelab checkout at `/srv/homelab` is at `5f9354b` and has the tracked marker deletion. Its deploy preflight blocks a dirty checkout until this removal is reconciled. Preserve private ignored configuration when syncing that checkout.
+## Release evidence

-The remaining homelab change is PR #105, branch `feat/edu-handoff-matrix`, based on `codex/ci-visible-checks`. Its eight protected checks passed. Renovate runs 1587 and 1588 passed. Image publishing was skipped for the PR. The EDU runtime changes are in PR #3 from `fix/handoff-runtime` to `main`; CI run 1589 is in progress. Those runtime changes have not been released.
+EDU PR #4 merged after its review and CI checks. Main-push CI run 1652 passed
+all validation and both image builds. Deploy run 1653 passed for the exact main
+SHA above.

-## Approval gate and next steps
+The workstation rollout completed for both Deployments. The deployment
+verified `/health` and `/live` with HTTP 200, Redis AUTH, session TTL of 1058
+seconds, a delivery backlog of zero, and all nine EDU vmalert rules with
+matching expressions and healthy evaluation.

-PR #99 must merge before PR #105 can target `main`. A merge attempt for PR #99 returned HTTP 405 because it needs one approval; the protected branch has `required_approvals=1` and whitelist approval is enabled. This approval gate prevents the remaining transfer steps.
+The images now run by digest:

-After the required approval:
+- Session keeper: `sha256:998dea51aa3015fd9cabefb0f53b030157a650c3bef72e02fe84f17d5762613d`
+- Webinar checker: `sha256:92f3c1fa2bb7f9b4680a9fc76a5b33dfbea8ef3dd9c6490ebc45876fd4c54461`

-1. Merge PR #99.
-2. Retarget PR #105 to `main`. Complete CI and review, then approve and merge it.
-3. Under the homelab apply lock, sync `/srv/homelab` to the merged removal. Preserve private ignored configuration and keep the active marker removed. Confirm the deploy preflight is clean.
-4. Merge the EDU runtime PR after its CI and review pass. The main-push CI run must complete successfully before its exact SHA can deploy.
-5. Verify the new release SHA, image digests, workload health, Redis AUTH and TTL, backlog, PVC and Secret identity, and monitoring. Record the results in the EDU PR.
+Redis StatefulSet was unchanged. PVC `redis-data-pvc` remains bound to PV
+`pvc-a4f2a79a-363a-4c12-ae91-92cdfc2a0d2e` with capacity 1 GiB. The existing
+runtime Secret and Fernet key were preserved during the handoff. Notification
+delivery was verified before closeout, as confirmed by the operator. The
+deployment did not record downtime.

-`AUTODEPLOY=false` is explicitly configured. The EDU repository path and port secrets are confirmed, and `EDU_KUBE_CONTEXT=Default` is configured as a repository variable. Keep deployment and registry credentials outside Git. Never run both homelab and EDU deployment paths at the same time.
-
-## Change summary
-
-The homelab PR removes the EDU subtree, deployment and image selection, rollback and verification cases, route probes, Renovate references, and external-image exceptions. It adds a serial dynamic matrix for the three homelab images. Each job builds an image or reuses a matching immutable digest. The final job checks all image results and publishes full-SHA tags and the existing release artifact only after they pass. PRs do not publish images. The protected check names from PR #99 are preserved. PR #100's service-metrics work is independent of this handoff.
-
-The EDU runtime PR adds the Playwright service manifest, reconciles Redis storage and Secret reload annotations, and adds pre-apply Redis backup and identity checks. It verifies application endpoints, Redis AUTH, session TTL, metrics, and all nine vmalert rules. Rollback checks workload and application health and reports when manual recovery is needed. Its deployment guard rejects an unexpected or dirty checkout and refuses deployment while either legacy homelab EDU marker exists.
-
-## Rollback and limits
-
-The private snapshot is `/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z` on the workstation. It contains the pre-handoff Redis RDB and recovery data. RDB checksum verification confirmed twelve keys. Keep the snapshot outside Git. For an EDU release failure, restore the saved Kubernetes resources and inspect application health. The rollback does not automatically restore the Redis RDB; restore old Redis data only when recovery requires it.
-
-For an ownership rollback, stop EDU deployment triggers first, restore the reviewed homelab source and marker, then reapply recorded immutable image digests. Verify both workload and application health. Never delete or recreate the Redis PVC.
-
-The initial EDU release and the homelab marker move are complete. PR #99 approval and merge, PR #105 retarget and merge, homelab checkout reconciliation, EDU runtime PR merge, and release of those runtime changes remain pending. Synthetic Telegram delivery and Alertmanager-to-Telegram notification were not tested.
+The release rollback snapshot is
+`/home/forust/.local/state/edu-master-deploy/20261007T180541Z-4f2b2a0e37dc11ac2c75441a15076c178e219d37`.
+The handoff data snapshot remains at
+`/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z`.
+Both snapshots are outside Git. Do not restore old Redis data unless recovery
+requires it. Never delete or recreate the Redis PVC.
@@ -7,4 +7,5 @@ self-hosted-runner:
  labels:
    - arch
    - homelab
+    - homelab-pr
    - prod
@@ -1,17 +1,23 @@
 # Homelab CI/CD

-The native Gitea runner runs on **vps**; production runs on **workstation**.
-Compose, workflow, shell, Python, formatting, YAML, Dockerfile and Kubernetes
-checks appear as separate jobs. Jobs run on `homelab:host`, one at a time; the
-build waits for every check to pass. CI and deploy runs also show a summary with
+The native Gitea runners run on **vps**; production runs on **workstation**.
+Main-branch checks and image builds use `homelab:host`. Pull request checks use
+`homelab-pr` in a Docker job container. CI PR checks use `pull_request_target`,
+so Gitea loads the workflow from the trusted base branch. That event then runs
+untrusted PR code, so the workflow must select `homelab-pr` before checkout and
+must not expose secrets. The CI validation jobs grant only `contents: read` and
+checkout the explicit PR head SHA with `persist-credentials: false`. Register
+`homelab-pr` at repository scope so only this repository can schedule its jobs.
+Each runner accepts one job at a time; the build waits for every check to pass.
+CI and deploy runs also show a summary with
 the release SHA, image build or reuse results, deploy mode, selected services,
 and image digests. Failed runs keep a summary of completed image builds, stage
 results, apply results, and recorded Kubernetes recovery. The final deploy
 summary is in the smoke job; earlier jobs show the state observed at that time.
 Apply success is separate from health and recovery. Update the installed
 workstation controller with `setup-workstation.sh` when no deploy is running.
-No job images or Kubernetes credentials
-are needed on the VPS. Builds use one pinned BuildKit helper container. CI and deploy are separate workflows.
+No job images or Kubernetes credentials are needed on the VPS. Builds use one
+pinned BuildKit helper container. CI and deploy are separate workflows.

 ## Runner installation

@@ -37,6 +43,60 @@ pushes directly to the registry, and caps retained local cache at 1 GiB with a
 2 GiB free-space target. This is not a hard limit on peak build disk usage.
 Nothing runs `docker system prune`, removes unrelated images, or deletes volumes.

+### Pull request runner
+
+Install the PR container runner on the VPS:
+
+```sh
+sudo bash .gitea/runner/setup-pr-runner.sh
+```
+
+Create a runner registration token from this repository's Actions runner
+settings. Run the installer in a terminal. It asks for the token without
+echoing it and registers `homelab-pr` with label
+`homelab-pr:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.09.01`. Confirm that
+Gitea lists the runner at Repository scope. The service runs as
+`gitea-pr-runner`; systemd grants that service access to the Docker socket with
+`SupplementaryGroups=docker`. Keep the account itself out of the `docker`
+group. The work directory is `/var/lib/gitea-pr-runner`.
+
+The runner config disables privileged containers, forbids workflow volume
+mounts, and prevents the Docker socket from being mounted into job and action
+containers. Do not mount the runner home or its host-side tool cache into a job.
+The existing host-side cache is retained, but PR job containers cannot read it.
+An unmatched label can fall back to the default job image; check the registered
+label before enabling PR checks.
+
+The installer reuses `/var/lib/gitea-pr-runner/.runner` when it exists. That
+file keeps the registration scope assigned by Gitea. To move an existing
+User-scoped runner to Repository scope, stop the service, remove the old runner
+from Gitea, back up and remove that registration file, then run the installer
+with a token created in this repository's Actions runner settings. Confirm the
+new scope in Gitea before enabling PR checks.
+
+The CI and Renovate workflows use `pull_request_target`, which reads the
+workflow from the base branch. They select `homelab-pr` before checking out PR
+code. The explicit head SHA and `persist-credentials: false` are mandatory:
+without the latter, checkout can leave the job token in Git configuration.
+Keep PR validation read-only and do not add Actions secrets. In the checked-in
+workflows, only a push to `main` or a manual CI run on `main` can select the
+trusted `homelab` runner. Gitea schedules jobs by matching `runs-on` labels; the
+runner does not restrict jobs by event or branch. Keep Gitea's approval gate for
+fork PR workflows enabled. Verify the live Gitea version and approval setting
+before relying on this gate; the image tag in the repository does not prove the
+version currently running. Before approving a fork workflow run, review all new
+and changed workflow files: a PR-defined `pull_request` workflow can request
+the `homelab` label. Automatic CI and Renovate PR checks use the trusted base
+workflow and select only `homelab-pr`. The release and deploy jobs stay on the
+trusted runner.
+
+The runner service can access the host Docker daemon, but job and action
+containers do not receive its socket or arbitrary host mounts. The runner and
+job containers still share the VPS kernel and Docker daemon. A container escape
+can therefore affect the host and other workloads. This is container isolation,
+not VM isolation; use disposable VMs for PRs that require a separate kernel and
+Docker daemon.
+
 ## Workstation setup

 As the existing SSH deploy user on workstation:
@@ -128,3 +188,25 @@ run first. Restore the runner config/unit from `.before-<timestamp>` backups,
 reload systemd and restart the runner. Restore the prior workflows from Git.
 Production data and persistent volumes stay where they were. Do not remove run
 state or Compose recovery files until recovery is confirmed.
+
+### Compose configuration recovery
+
+Successful deploys save the complete resolved Compose configuration in
+`~/.local/state/homelab-deploy/compose-configs/`. These files can contain secrets.
+Keep them private and do not commit or upload them.
+The next deploy uses this configuration for its recovery file, including old
+commands, environment, mounts, ports, and removed services. The recovery command
+uses `--remove-orphans` to remove services added by the failed deploy. It does
+not restore volume data or reverse database migrations.
+
+On the first run after this update, the controller can use the Compose file
+from the previous successful run. If that file is absent, it reads the persistent
+checkout and checks its service configuration hashes against existing containers.
+A mismatch stops preflight. Restore the previous configuration before retrying.
+Update the installed controller with `bash .gitea/runner/setup-workstation.sh`
+from the reviewed checkout before using this change.
+
+New namespaces are checked during preflight. Server validation of their resources
+runs after namespace creation and before application resources are applied.
+Plan mode does not create namespaces. A failed deferred check can leave an empty
+namespace; inspect it before removing it.
@@ -0,0 +1,12 @@
+runner:
+  file: /var/lib/gitea-pr-runner/.runner
+  capacity: 1
+  timeout: 5h
+  labels:
+    - homelab-pr:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.09.01
+cache:
+  enabled: false
+container:
+  privileged: false
+  valid_volumes: []
+  docker_host: "-"
@@ -0,0 +1,28 @@
+[Unit]
+Description=Gitea Actions untrusted pull request runner
+After=network-online.target docker.service
+Wants=network-online.target docker.service
+
+[Service]
+User=gitea-pr-runner
+Group=gitea-pr-runner
+SupplementaryGroups=docker
+WorkingDirectory=/var/lib/gitea-pr-runner
+Environment=HOME=/var/lib/gitea-pr-runner
+Environment=PATH=/var/lib/gitea-pr-runner/.cache/homelab-ci/bin:/usr/local/bin:/usr/bin:/bin
+ExecStart=/usr/local/bin/gitea-runner daemon --config /etc/gitea-pr-runner/config.yaml
+Restart=on-failure
+RestartSec=5
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=full
+ProtectHome=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectControlGroups=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+UMask=0077
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Install the containerized runner service for untrusted PR jobs.
+set -euo pipefail
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+[ "$(id -u)" -eq 0 ] || { echo 'Run with sudo on the runner host' >&2; exit 1; }
+for tool in cp cut date docker getent id install runuser systemctl useradd; do
+  command -v "$tool" >/dev/null || { echo "Install missing prerequisite: $tool" >&2; exit 1; }
+done
+docker info >/dev/null || {
+  echo 'Start Docker Engine before installing the PR runner' >&2
+  exit 1
+}
+command -v /usr/local/bin/gitea-runner >/dev/null || {
+  echo 'Install gitea-runner 3.0.2 at /usr/local/bin/gitea-runner first' >&2
+  exit 1
+}
+runner_version_output="$(/usr/local/bin/gitea-runner --version 2>&1)" || {
+  echo 'Cannot read the installed gitea-runner version' >&2
+  exit 1
+}
+if [[ ! "$runner_version_output" =~ (^|[[:space:]])v?3\.0\.2($|[[:space:]]) ]]; then
+  printf 'Expected gitea-runner 3.0.2; found: %s\n' "$runner_version_output" >&2
+  exit 1
+fi
+
+getent group docker >/dev/null || {
+  echo 'Install Docker Engine first; the docker group is missing' >&2
+  exit 1
+}
+id gitea-pr-runner >/dev/null 2>&1 || \
+  useradd --system --create-home --home-dir /var/lib/gitea-pr-runner --shell /usr/bin/bash gitea-pr-runner
+runner_home="$(getent passwd gitea-pr-runner | cut -d: -f6)"
+[ "$runner_home" = /var/lib/gitea-pr-runner ] || {
+  echo 'Unexpected PR runner home; inspect the existing service first' >&2
+  exit 1
+}
+case " $(id -nG gitea-pr-runner) " in
+  *' docker '*)
+    echo 'Remove gitea-pr-runner from the docker group; only the systemd service gets Docker access' >&2
+    exit 1
+    ;;
+esac
+
+install -d -m 0755 /etc/gitea-pr-runner
+stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+for existing in /etc/gitea-pr-runner/config.yaml /etc/systemd/system/gitea-pr-runner.service; do
+  [ ! -f "$existing" ] || cp -p "$existing" "$existing.before-$stamp"
+done
+install -m 0644 "$here/pr-config.yaml" /etc/gitea-pr-runner/config.yaml
+install -m 0644 "$here/pr-runner.service" /etc/systemd/system/gitea-pr-runner.service
+
+if [ ! -f /var/lib/gitea-pr-runner/.runner ]; then
+  read -r -s -p 'Enter the Gitea repository runner registration token: ' runner_token
+  printf '\n'
+  [ -n "$runner_token" ] || { echo 'Runner token is required' >&2; exit 1; }
+  export GITEA_RUNNER_REGISTRATION_TOKEN="$runner_token"
+  unset runner_token
+  runuser --preserve-environment -u gitea-pr-runner -- \
+    /usr/local/bin/gitea-runner register \
+    --config /etc/gitea-pr-runner/config.yaml \
+    --instance https://gitea.forust.xyz \
+    --name homelab-pr \
+    --labels 'homelab-pr:docker://docker.gitea.com/runner-images:ubuntu-24.04-v26.09.01' \
+    --no-interactive
+  unset GITEA_RUNNER_REGISTRATION_TOKEN
+fi
+chmod 0600 /var/lib/gitea-pr-runner/.runner
+
+systemctl daemon-reload
+systemctl enable --now gitea-pr-runner.service
+systemctl restart gitea-pr-runner.service
+echo "PR runner ready. Configuration backups: *.before-$stamp"
@@ -91,4 +91,66 @@ if check_referenced_secrets >"$scratch/secrets.log"; then
  echo 'Secret check accepted a failed manifest render' >&2
  exit 1
 fi
+
+# New declared namespaces defer only their own resources during preflight.
+render_selected_resources() {
+  cat <<'JSON'
+{"apiVersion":"v1","kind":"List","items":[
+ {"apiVersion":"v1","kind":"Namespace","metadata":{"name":"new"}},
+ {"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"new-config","namespace":"new"}},
+ {"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"existing-config","namespace":"default"}}
+]}
+JSON
+}
+kubectl() {
+  case "$1" in
+    get) printf '%s\n' '{"items":[{"metadata":{"name":"default"}}]}' ;;
+    apply) cat >"$scratch/server-input.json" ;;
+    *) return 1 ;;
+  esac
+}
+validate_server_resources true
+jq -e '.items | length == 2 and all(.metadata.name != "new-config")' "$scratch/server-input.json" >/dev/null
+if validate_server_resources false 2>"$scratch/deferred.log"; then
+  echo 'Post-namespace validation accepted a missing namespace' >&2
+  exit 1
+fi
+kubectl() {
+  case "$1" in
+    get) printf '%s\n' '{"items":[{"metadata":{"name":"default"}},{"metadata":{"name":"new"}}]}' ;;
+    apply) cat >"$scratch/server-input.json" ;;
+    *) return 1 ;;
+  esac
+}
+validate_server_resources false
+jq -e '.items | length == 3' "$scratch/server-input.json" >/dev/null
+render_selected_resources() {
+  printf '%s\n' '{"items":[{"kind":"ConfigMap","metadata":{"name":"bad","namespace":"undeclared"}}]}'
+}
+if validate_server_resources true 2>"$scratch/undeclared.log"; then
+  echo 'Preflight accepted an undeclared missing namespace' >&2
+  exit 1
+fi
+# Count services, not characters in the newline-separated service names.
+compose() {
+  case "$*" in
+    *'config --format json') printf '%s\n' '{"services":{"headscale":{},"headplane":{},"web":{},"init":{"restart":"no"}}}' ;;
+    *'ps --status running --services') printf '%s\n' headscale headplane web ;;
+    *) return 1 ;;
+  esac
+}
+verify_compose_stack example.yaml >"$scratch/compose-count.log"
+grep -qF 'all 3 service(s) running' "$scratch/compose-count.log"
+compose() {
+  case "$*" in
+    *'config --format json') printf '%s\n' '{"services":{"headscale":{},"headplane":{},"web":{}}}' ;;
+    *'ps --status running --services') printf '%s\n' headscale headplane ;;
+    *) return 0 ;;
+  esac
+}
+if verify_compose_stack example.yaml >"$scratch/compose-missing.log"; then
+  echo 'Compose verification accepted a missing service' >&2
+  exit 1
+fi
+grep -qF 'NOT RUNNING: web' "$scratch/compose-missing.log"
 printf '%s\n' 'Deploy validation regressions passed.'
@@ -3,22 +3,30 @@ name: ci
  push:
    branches:
      - main
-  pull_request: null
+  # Use the base-branch workflow so PR changes cannot select trusted runners.
+  pull_request_target: null
  workflow_dispatch: null
 permissions:
  contents: read
  actions: read
 concurrency:
-  group: ci-${{ github.ref }}
-  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+  group: ci-${{ github.event_name == 'pull_request_target' && format('pr-{0}', github.event.pull_request.number) || github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request_target' || github.ref != 'refs/heads/main' }}
 jobs:
+  # pull_request_target uses the base ref (often main); check the event as well
+  # as the ref so every PR job stays on the isolated runner.
  compose:
    name: Compose
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Validate Compose files
        shell: bash
@@ -66,11 +74,16 @@ jobs:
          fi
  workflows:
    name: Workflows
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -102,11 +115,16 @@ jobs:
          fi
  shell:
    name: Shell
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -146,11 +164,16 @@ jobs:
          fi
  formatting:
    name: Formatting
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -194,11 +217,16 @@ jobs:
          fi
  python:
    name: Python and tests
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -232,11 +260,16 @@ jobs:
          fi
  yaml:
    name: YAML
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -280,11 +313,16 @@ jobs:
          fi
  dockerfiles:
    name: Dockerfiles
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -326,11 +364,16 @@ jobs:
          fi
  kubernetes:
    name: Kubernetes
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
        id: source
      - name: Prepare pinned tools
        shell: bash
@@ -377,7 +420,7 @@ jobs:
          fi
  image-plan:
    needs: [compose, workflows, shell, formatting, python, yaml, dockerfiles, kubernetes]
-    if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+    if: github.event_name != 'pull_request_target' && github.ref == 'refs/heads/main'
    runs-on: homelab
    timeout-minutes: 10
    outputs:
@@ -388,6 +431,7 @@ jobs:
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
        with:
          fetch-depth: 0
+          persist-credentials: false
      - name: Detect build inputs against successful CI
        id: plan
        env:
@@ -395,7 +439,7 @@ jobs:
        run: python3 .gitea/workflows/release.py prepare --output build-plan.json
      - name: Store the image plan
        id: artifact
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: build-plan
          path: build-plan.json
@@ -433,9 +477,11 @@ jobs:
      - name: Checkout repository
        id: source
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          persist-credentials: false
      - name: Download the checked image plan
        id: inputs
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
        with:
          name: build-plan
      - name: Build or reuse this image
@@ -447,7 +493,7 @@ jobs:
        run: python3 .gitea/workflows/release.py image --image "$IMAGE_NAME" --output image.json
      - name: Store the image result
        id: artifact
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: image-${{ matrix.name }}
          path: image.json
@@ -480,9 +526,11 @@ jobs:
      - name: Checkout repository
        id: source
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          persist-credentials: false
      - name: Download all image results
        id: inputs
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
        with:
          path: artifacts
      - name: Pin SHA tags and write the complete release
@@ -495,7 +543,7 @@ jobs:
          --plan artifacts/build-plan/build-plan.json
      - name: Store commit release
        id: artifact
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: release-${{ github.sha }}
          path: release.json
@@ -42,15 +42,50 @@ def prepare(source_file):
    images_file = directory / 'compose-images.json'
    locks = json.loads(images_file.read_text()) if images_file.exists() else previous.get('compose-images', {})
    release = json.loads((directory / 'release.json').read_text())
-    before = json.loads(json.dumps(config))
+    state = Path(os.environ.get('HOMELAB_STATE', Path.home() / '.local/state/homelab-deploy'))
+    baseline = state / 'compose-configs' / f'{relative.parent.name}.json'
+    if not baseline.exists() and re.fullmatch(r'[0-9]+-[0-9]+', previous.get('run_id', '')):
+        baseline = state / 'runs' / previous['run_id'] / 'compose' / baseline.name
+    bootstrap = not baseline.exists()
+    if not bootstrap:
+        before = json.loads(baseline.read_text())
+    else:
+        # Bootstrap from the persistent configuration, never from the new source.
+        persistent_file = config_repo / relative
+        if persistent_file.exists():
+            before = json.loads(
+                output(
+                    'docker',
+                    'compose',
+                    '--project-directory',
+                    str(project_dir),
+                    '-f',
+                    str(persistent_file),
+                    'config',
+                    '--format',
+                    'json',
+                    cwd=config_repo,
+                )
+            )
+        elif output('docker', 'ps', '-aq', '--filter', f'label=com.docker.compose.project={project}'):
+            raise ValueError(f'{project}: no previous Compose configuration; restore it before deploy')
+        else:
+            before = {'name': project, 'services': {}}
+    if before['name'] != project:
+        raise ValueError('Compose project name changed; manual migration is required')
    for service, settings in config['services'].items():
        reference = settings.get('image')
+        nextcloud_aio_master = project == 'nextcloud' and service == 'nextcloud-aio-mastercontainer'
        if not reference or settings.get('build'):
            raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
        image_repo = reference.split('@')[0].rsplit('/', 1)
        image_repo[-1] = image_repo[-1].split(':')[0]
        image_repo = '/'.join(image_repo)
-        if image_repo in release['images']:
+        # Nextcloud AIO validates the mastercontainer image reference and rejects
+        # a digest. Keep its configured tag so AIO can start and manage its stack.
+        if nextcloud_aio_master:
+            pinned = reference
+        elif image_repo in release['images']:
            pinned = image_repo + '@' + release['images'][image_repo]
        elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
            pinned = locks[reference]
@@ -58,6 +93,12 @@ def prepare(source_file):
            pinned = resolve(reference)
        settings['image'] = pinned
        locks[reference] = pinned
+    for service, settings in before['services'].items():
+        reference = settings['image']
+        image_repo = reference.split('@')[0].rsplit('/', 1)
+        image_repo[-1] = image_repo[-1].split(':')[0]
+        image_repo = '/'.join(image_repo)
+        nextcloud_aio_master = project == 'nextcloud' and service == 'nextcloud-aio-mastercontainer'
        # Capture what is running, not the current value of its mutable tag.
        ids = output(
            'docker',
@@ -69,13 +110,43 @@ def prepare(source_file):
            f'label=com.docker.compose.service={service}',
        ).splitlines()
        actual = set()
+        if bootstrap and ids:
+            expected_hash = output(
+                'docker',
+                'compose',
+                '--project-directory',
+                str(project_dir),
+                '-f',
+                str(persistent_file),
+                'config',
+                '--hash',
+                service,
+                cwd=config_repo,
+            ).split()[-1]
+            for container in ids:
+                running_hash = output(
+                    'docker',
+                    'inspect',
+                    container,
+                    '--format',
+                    '{{ index .Config.Labels "com.docker.compose.config-hash" }}',
+                )
+                if running_hash != expected_hash:
+                    raise ValueError(
+                        f'{project}/{service}: persistent config differs from running config; restore the previous config'
+                    )
        for container in ids:
            image_id = output('docker', 'inspect', container, '--format', '{{.Image}}')
            digests = json.loads(output('docker', 'image', 'inspect', image_id, '--format', '{{json .RepoDigests}}'))
            actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
        if len(actual) > 1:
            raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
-        before['services'][service]['image'] = next(iter(actual)) if actual else reference
+        # AIO also rejects a digest in its recovery config. Preserve its tag in
+        # both deploy and recovery files.
+        if nextcloud_aio_master:
+            before['services'][service]['image'] = reference
+        else:
+            before['services'][service]['image'] = next(iter(actual)) if actual else reference
    for name, data in (('compose', config), ('compose-before', before)):
        folder = directory / name
        folder.mkdir(mode=0o700, exist_ok=True)
@@ -85,7 +156,7 @@ def prepare(source_file):
    images_file.write_text(json.dumps(locks, indent=2) + '\n')
    print(f'Compose {project}: images pinned; local paths preserved')
    print(
-        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never'
+        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never --remove-orphans'
    )


@@ -141,7 +141,8 @@ def make_plan(directory):
    planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
    request = json.loads((directory / 'request.json').read_text())
    previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
-    helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
+    # Helm 4 lists every release status by default and removed the --all flag.
+    helm = json.loads(command('helm', 'list', '-A', '-o', 'json'))
    plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
    if request['refresh_images']:
        plan['selected']['compose'] = plan['active']['compose']
@@ -164,6 +165,10 @@ def finish_success(directory, plan):
        if previous.exists()
        else {}
    )
+    configs = STATE / 'compose-configs'
+    configs.mkdir(mode=0o700, exist_ok=True)
+    for config in (directory / 'compose').glob('*.json'):
+        atomic_json(configs / config.name, json.loads(config.read_text()))
    atomic_json(STATE / 'last-success.json', plan)
    status = json.loads((directory / 'status.json').read_text())
    status['state'] = 'success'
@@ -180,7 +180,8 @@ save_snapshot() {
          | select(any(.metadata.ownerReferences[]?; .uid == $w.metadata.uid))
          | select($w.kind != "StatefulSet" or .metadata.name == $w.status.currentRevision) | .revision] | max // 0) end)
    }]' "$dir/workloads.json" >"$dir/revisions.json" || return 1
-  releases="$(helm list --all -A -o json)" || return 1
+  # Helm 4 lists every release status by default and removed the --all flag.
+  releases="$(helm list -A -o json)" || return 1
  for entry in "${HELM_RELEASES[@]}"; do
    IFS='|' read -r release _ namespace _ _ _ <<<"$entry"
    if ! jq -e --arg r "$release" --arg n "$namespace" \
@@ -329,11 +330,11 @@ rollback_workloads() {
 # written straight into a `helm upgrade` command would never be updated: these
 # have to be declared as custom.regex managers in renovate/renovate.json.
 HELM_RELEASES=(
-  "prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.2.3|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
+  "prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.3.2|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
  "victoria-operator|victoriametrics/victoria-metrics-operator|prometheus|0.68.1|prometheus-stack/k8s/victoria-operator-values.yaml|prometheus-stack/k8s/active"
  "loki|grafana/loki|prometheus|7.3.0|loki/k8s/loki-values.yaml|loki/k8s/active"
  "alloy|grafana/alloy|prometheus|1.12.1|loki/k8s/alloy-values.yaml|loki/k8s/active"
-  "reloader|stakater/reloader|reloader|2.2.17|reloader/k8s/reloader-values.yaml|reloader/k8s/active"
+  "reloader|stakater/reloader|reloader|2.2.18|reloader/k8s/reloader-values.yaml|reloader/k8s/active"
 )

 # "name url" for the Helm repository hosting a chart, empty if unknown.
@@ -570,6 +571,41 @@ skip_uninstalled_vmagent_crd() {
  return 1
 }

+# Render one complete resource list so new namespaces can be identified across
+# files and Kustomize apps. A missing undeclared namespace remains an error.
+render_selected_resources() {
+  local m k
+  {
+    for m in "${K8S_MANIFESTS[@]}"; do
+      if skip_uninstalled_vmagent_crd "$m" >/dev/null; then continue; fi
+      kubectl create --dry-run=client --validate=false -f "$m" -o json || return 1
+    done
+    for k in "${KUSTOMIZE_APPS[@]}"; do
+      kubectl kustomize "$k" | kubectl create --dry-run=client --validate=false -f - -o json || return 1
+    done
+  } | jq -s '{apiVersion: "v1", kind: "List", items: [ .[] | if .kind == "List" then .items[] else . end ]}'
+}
+
+validate_server_resources() {
+  local defer_new="$1" resources existing filtered
+  resources="$(render_selected_resources)" || return 1
+  existing="$(kubectl get namespaces -o json)" || return 1
+  filtered="$(jq --argjson existing "$existing" --argjson defer "$defer_new" '
+    [.items[] | select(.kind == "Namespace") | .metadata.name] as $declared
+    | [$existing.items[].metadata.name] as $present
+    | .items |= map(
+        (.metadata.namespace // "default") as $ns
+        | if .kind == "Namespace" or ($present | index($ns)) != null then .
+          elif ($declared | index($ns)) == null then error("Undeclared missing namespace: " + $ns)
+          elif $defer then empty
+          else error("Namespace still missing after namespace apply: " + $ns)
+          end)
+  ' <<<"$resources")" || return 1
+  if [ "$(jq '.items | length' <<<"$filtered")" -gt 0 ]; then
+    kubectl apply --dry-run=server -f - <<<"$filtered" >/dev/null
+  fi
+}
+
 stage_validate() {
  check_prune_mode || return 1
  cd "$REPO"
@@ -596,15 +632,7 @@ stage_validate() {
    kubectl apply -k "$k" --dry-run=client >/dev/null
  done
  log "Validate k8s manifests (kubectl dry-run=server)"
-  for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
-    if skip_uninstalled_vmagent_crd "$m"; then
-      continue
-    fi
-    kubectl apply --dry-run=server -f "$m" >/dev/null
-  done
-  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
-    kubectl apply -k "$k" --dry-run=server >/dev/null
-  done
+  validate_server_resources true
  log "Checking referenced Secrets exist"
  echo "  (deploy never applies *secret*.yaml; create missing ones manually)"
  check_referenced_secrets
@@ -656,6 +684,14 @@ stage_apply_k8s() {
      record_apply kubectl "${m#"$REPO"/}" success
    done
  fi
+  # Kustomize may declare namespaces inside its rendered resources too.
+  local namespace_resources
+  namespace_resources="$(render_selected_resources | jq '.items |= map(select(.kind == "Namespace"))')" || return 1
+  if [ "$(jq '.items | length' <<<"$namespace_resources")" -gt 0 ]; then
+    kubectl apply -f - <<<"$namespace_resources" || return 1
+  fi
+  # Complete the deferred server checks before Helm or application resources change.
+  validate_server_resources false || return 1
  if selected_service k8s prometheus-stack && [ -f "$REPO/prometheus-stack/k8s/active" ]; then
    if [ ! -f "$CONFIG_REPO/prometheus-stack/k8s/grafana-values.yaml" ]; then
      echo "ERROR: prometheus-stack/k8s/grafana-values.yaml (gitignored) missing on workstation, restore it first."
@@ -791,12 +827,13 @@ stage_verify_k8s() {
 # actually be running.
 verify_compose_stack() {
  local cf="$1"
-  local expected running missing=()
+  local expected running svc missing=() service_count=0
  expected="$(compose "$cf" config --format json | jq -r ' .services | to_entries[] | select(.value.restart != "no") | .key' | sort)" || return 1
  running="$(compose "$cf" ps --status running --services | sort)" || return 1
  [ -n "$expected" ] || return 0
  while IFS= read -r svc; do
    [ -n "$svc" ] || continue
+    service_count=$((service_count + 1))
    # restart:"no" services are allowed to have exited.
    if ! printf '%s\n' "$running" | grep -qx "$svc"; then
      missing+=("$svc")
@@ -807,7 +844,7 @@ verify_compose_stack() {
    compose "$cf" ps --all 2>/dev/null | sed 's/^/    /' || true
    return 1
  fi
-  echo "  all ${#expected} service(s) running"
+  echo "  all $service_count service(s) running"
  return 0
 }

@@ -83,20 +83,20 @@ def make_plan(repo, config_repo, release, previous, mode, live_helm):
        removed = []
    else:
        paths = output('git', '-C', str(repo), 'diff', '--name-only', previous['sha'], release['sha']).splitlines()
-        changed = {path.split('/')[0] for path in paths}
+        changed = {service for service in all_services for path in paths if path.startswith(service + '/')}
        if any(path.startswith('.gitea/') for path in paths):
            changed |= all_services
        changed |= {s for s in all_services if previous.get('local_inputs', {}).get(s) != local_inputs[s]}
        for file in tracked(repo):
-            service = file.split('/')[0]
-            if service not in all_services or not file.endswith(('.yaml', '.yml')):
+            owners = {service for service in all_services if file.startswith(service + '/')}
+            if not owners or not file.endswith(('.yaml', '.yml')):
                continue
            text = (repo / file).read_text()
            if any(
                image in text and previous.get('images', {}).get(image) != digest
                for image, digest in release['images'].items()
            ):
-                changed.add(service)
+                changed |= owners
        removed = sorted(
            set(previous.get('active', {}).get('k8s', []) + previous.get('active', {}).get('compose', []))
            - all_services
@@ -80,7 +80,7 @@ jobs:
  apply:
    needs: [gate]
    runs-on: homelab
-    timeout-minutes: 100
+    timeout-minutes: 120
    steps:
      - name: Checkout checked commit
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -305,7 +305,6 @@ def build_images(output, report, name, plan):
        if exists:
            print(f'Reuse {name}: inputs unchanged')
            digest = old_digest
-            report['reused'].append(name)
        else:
            print(f'Build {name}', flush=True)
            metadata = Path(docker_config) / 'metadata.json'
@@ -332,14 +331,14 @@ def build_images(output, report, name, plan):
                env=env,
            )
            digest = json.loads(metadata.read_text())['containerimage.digest']
-            report['built'].append(name)
+        if not isinstance(digest, str) or not DIGEST.fullmatch(digest):
+            raise ValueError('Image job returned an invalid digest')
        release['images'][image] = digest
        release['inputs'][image] = inputs
-        if not DIGEST.fullmatch(digest):
-            raise ValueError('Image job returned an invalid digest')
+        report['reused' if exists else 'built'].append(name)
        output.write_text(json.dumps(release, indent=2) + '\n')
        report['current'] = None
-        report['phase'] = 'Release file saved'
+        report['phase'] = 'Image result file saved'
    finally:
        # Cleanup errors must neither leak credentials nor mask the original build error.
        try:
@@ -395,13 +394,17 @@ def build(output, name, plan):
        result = 'success'
    finally:
        lines = [
-            f'## Image release `{os.environ.get("GITHUB_SHA", "unknown")}`',
+            f'## Image build result `{name}`',
+            '',
+            f'- Commit: `{os.environ.get("GITHUB_SHA", "unknown")}`',
            '',
            f'- Result: **{result}**',
            f'- Last stage: {report["phase"]}',
        ]
        if result == 'failure':
-            lines.append('- No release from this build can be deployed. Open the failed step log.')
+            lines.append('- This image job failed. The complete release cannot be published. Open the failed step log.')
+        if result == 'success':
+            lines.append('- This is one image result. The final build job must publish the complete release.')
        if report['current']:
            lines.append(f'- Image at the failure: `{report["current"]}`')
        for title, key in (('Built', 'built'), ('Reused from successful CI', 'reused')):
@@ -1,7 +1,9 @@
 name: renovate-ci

 on:
-  pull_request:
+  # Read the workflow from the trusted base branch. PR code runs only on the
+  # unprivileged runner selected below.
+  pull_request_target:
    paths:
      - "renovate/**"
      - ".gitea/workflows/renovate-ci.yaml"
@@ -26,37 +28,50 @@ permissions:

 jobs:
  validate-renovate:
-    runs-on: homelab
+    permissions:
+      contents: read
+    runs-on: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 20
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false

-      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag,
-      # so the same version that runs in the cluster is the one validated here.
-      - name: Resolve the deployed Renovate image
+      # renovate/k8s/cronjob.yaml is the single source of truth for the version.
+      - name: Resolve the deployed Renovate version
        id: image
        shell: bash
        run: |
          set -euo pipefail
          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
            renovate/k8s/cronjob.yaml | head -1)"
-          if [ -z "$image" ]; then
-            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+          if [[ ! "$image" =~ ^renovate/renovate:([0-9]+\.[0-9]+\.[0-9]+)$ ]]; then
+            echo "::error::expected a pinned renovate/renovate semantic version in renovate/k8s/cronjob.yaml"
            exit 1
          fi
-          echo "using $image"
-          echo "image=$image" >> "$GITHUB_OUTPUT"
+          version="${BASH_REMATCH[1]}"
+          echo "using Renovate $version"
+          printf 'version=%s\n' "$version" >> "$GITHUB_OUTPUT"

-      - name: Validate Renovate repository config
+      - name: Prepare pinned validation tools
        shell: bash
        run: |
          set -euo pipefail
-          docker run --rm \
-            -v "$PWD/renovate:/opt/renovate:ro" \
-            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
-            "${{ steps.image.outputs.image }}" \
-            renovate-config-validator /opt/renovate/renovate.json
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform node)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+
+      - name: Validate Renovate repository config
+        shell: bash
+        env:
+          RENOVATE_VERSION: ${{ steps.image.outputs.version }}
+        run: |
+          set -euo pipefail
+          npm_cache="$(mktemp -d "${RUNNER_TEMP:-/tmp}/renovate-npm-cache.XXXXXXXX")"
+          trap 'rm -rf "$npm_cache"' EXIT
+          NPM_CONFIG_CACHE="$npm_cache" RENOVATE_CONFIG_FILE="$PWD/renovate/renovate.json" \
+            npm exec --yes --package="renovate@${RENOVATE_VERSION}" -- renovate-config-validator

      # The CronJob cannot read the repository, so renovate/k8s/configmap.yaml
      # carries an inlined copy of the config. Fail if it no longer matches.
@@ -70,8 +85,6 @@ jobs:
        shell: bash
        run: |
          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
-          export PATH="$tools_dir:$PATH"
          kubeconform \
            -strict \
            -ignore-missing-schemas \
@@ -32,11 +32,14 @@ concurrency:

 jobs:
  run-renovate:
+    if: github.ref == 'refs/heads/main'
    runs-on: homelab
    timeout-minutes: 60
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: refs/heads/main

      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag.
      # Reading it here means this workflow validates and runs the exact version
@@ -48,21 +51,23 @@ jobs:
          set -euo pipefail
          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
            renovate/k8s/cronjob.yaml | head -1)"
-          if [ -z "$image" ]; then
-            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+          if [[ ! "$image" =~ ^renovate/renovate:[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+            echo "::error::expected a pinned renovate/renovate semantic version in renovate/k8s/cronjob.yaml"
            exit 1
          fi
          echo "using $image"
-          echo "image=$image" >> "$GITHUB_OUTPUT"
+          printf 'image=%s\n' "$image" >> "$GITHUB_OUTPUT"

      - name: Validate Renovate config
        shell: bash
+        env:
+          RENOVATE_IMAGE: ${{ steps.image.outputs.image }}
        run: |
          set -euo pipefail
          docker run --rm \
            -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
-            "${{ steps.image.outputs.image }}" \
+            "$RENOVATE_IMAGE" \
            renovate-config-validator

      - name: Run Renovate
@@ -73,6 +78,7 @@ jobs:
          RENOVATE_REPOSITORIES: ${{ inputs.repositories }}
          RENOVATE_DRY_RUN: ${{ inputs.dry_run && 'full' || '' }}
          LOG_LEVEL: ${{ inputs.log_level }}
+          RENOVATE_IMAGE: ${{ steps.image.outputs.image }}
        run: |
          set -euo pipefail

@@ -89,4 +95,4 @@ jobs:
            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
            -e RENOVATE_BASE_DIR=/tmp/renovate \
            -e LOG_LEVEL="${LOG_LEVEL:-info}" \
-            "${{ steps.image.outputs.image }}"
+            "$RENOVATE_IMAGE"
@@ -13,8 +13,8 @@
 ACTIONLINT_VERSION="1.7.7"
 SHELLCHECK_VERSION="0.11.0"
 KUBECONFORM_VERSION="0.8.0"
-PRETTIER_VERSION="3.8.1"
-RUFF_VERSION="0.16.8"
+PRETTIER_VERSION="3.9.9"
+RUFF_VERSION="0.16.10"
 YAMLLINT_VERSION="1.38.0"
 HADOLINT_VERSION="2.14.0"
 # pip-audit reads the advisory database over the network, so a floating version
@@ -36,4 +36,4 @@ NODE_VERSION="22.23.3"
 JQ_VERSION="1.8.1"

 # BuildKit is the only auxiliary CI container; jobs themselves stay on the host.
-BUILDKIT_IMAGE="moby/buildkit:v0.33.1"
+BUILDKIT_IMAGE="moby/buildkit:v0.34.0"
@@ -115,3 +115,6 @@ prometheus-stack/k8s/grafana-values.yaml
 traefik/k8s/local-tls.yaml
 converters/k8s/config.yaml
 convertx/k8s/config.yaml
+
+# Graphify local index and generated reports
+graphify-out/
@@ -42,7 +42,7 @@ services:

  bentopdf:
    container_name: bentopdf
-    image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
+    image: bentopdfteam/bentopdf-simple:2.8.8
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
@@ -26,7 +26,7 @@ spec:
        app: bentopdf
    spec:
      containers:
-        - image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
+        - image: bentopdfteam/bentopdf-simple:2.8.8
          imagePullPolicy: Always
          name: bentopdf
          ports:
@@ -1,6 +1,6 @@
 services:
  server:
-    image: docker.gitea.com/gitea:28.0.0
+    image: docker.gitea.com/gitea:28.1.0
    container_name: gitea
    restart: always
    environment:
@@ -20,6 +20,8 @@ data:

  GITEA__mailer__ENABLED: "false"

+  GITEA__metrics__ENABLED: "true"
+
  # No code/issue search needed: bleve reindexes the whole issue index on
  # every pod restart (cron.rebuild_issue_indexer RUN_AT_START) and hammers
  # the rotational disk for an hour. "db" serves issue search from postgres.
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: gitea-service
  namespace: gitea
+  labels:
+    app: gitea
 spec:
  selector:
    app: gitea
@@ -35,7 +37,7 @@ spec:
    spec:
      containers:
        - name: gitea
-          image: gitea/gitea:28.0.0
+          image: gitea/gitea:28.1.0
          envFrom:
            - configMapRef:
                name: gitea-config
@@ -7,7 +7,8 @@ spec:
  entryPoints:
    - websecure
  routes:
-    - match: Host(`gitea.forust.xyz`) || Host(`git.forust.xyz`)
+    # Metrics are scraped directly through the cluster Service.
+    - match: (Host(`gitea.forust.xyz`) || Host(`git.forust.xyz`)) && !PathPrefix(`/metrics`)
      kind: Rule
      services:
        - name: gitea-service
@@ -0,0 +1,16 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: gitea
+  namespace: gitea
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: gitea
+  endpoints:
+    - port: http
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -7,7 +7,7 @@
 # # Dev server_url
 # server_url: https://hs.dev_internal_domain.internal
 listen_addr: 0.0.0.0:8080
-metrics_listen_addr: 127.0.0.1:9090
+metrics_listen_addr: 0.0.0.0:9090
 grpc_listen_addr: 127.0.0.1:50443
 grpc_allow_insecure: false
 noise:
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: headscale-server-external
  namespace: headscale
+  labels:
+    app: headscale
 spec:
  ports:
    - port: 8080
@@ -0,0 +1,18 @@
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMServiceScrape
+metadata:
+  name: headscale
+  namespace: headscale
+  labels:
+    release: prometheus-stack
+spec:
+  # The external Service has a manually managed EndpointSlice, not Endpoints.
+  discoveryRole: endpointslice
+  selector:
+    matchLabels:
+      app: headscale
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -1,7 +1,7 @@
 services:
  homarr:
    container_name: homarr
-    image: ghcr.io/homarr-labs/homarr:v2.2.0
+    image: ghcr.io/homarr-labs/homarr:v2.3.0
    restart: unless-stopped
    volumes:
      - ./appdata:/appdata
@@ -32,7 +32,7 @@ spec:
      serviceAccountName: homarr
      containers:
        - name: homarr
-          image: ghcr.io/homarr-labs/homarr:v2.2.0
+          image: ghcr.io/homarr-labs/homarr:v2.3.0
          envFrom:
            - configMapRef:
                name: homarr-config
@@ -6,6 +6,10 @@ metadata:
 data:
  TZ: "Europe/Bratislava"

+  IMMICH_TELEMETRY_INCLUDE: "all"
+  IMMICH_API_METRICS_PORT: "8081"
+  IMMICH_MICROSERVICES_METRICS_PORT: "8082"
+
  # The database in this namespace, not the shared one in the database
  # namespace: v3 needs VectorChord, and only the dedicated image carries it.
  DB_HOSTNAME: "immich-postgres"
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: immich-service
  namespace: immich
+  labels:
+    app: immich
 spec:
  selector:
    app: immich
@@ -10,6 +12,12 @@ spec:
    - name: http
      port: 2283
      targetPort: 2283
+    - name: api-metrics
+      port: 8081
+      targetPort: api-metrics
+    - name: worker-metrics
+      port: 8082
+      targetPort: worker-metrics
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -41,6 +49,10 @@ spec:
          ports:
            - name: http
              containerPort: 2283
+            - name: api-metrics
+              containerPort: 8081
+            - name: worker-metrics
+              containerPort: 8082
          volumeMounts:
            - name: immich-data
              mountPath: /data
@@ -0,0 +1,20 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: immich
+  namespace: immich
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: immich
+  endpoints:
+    - port: api-metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
+    - port: worker-metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -1,6 +1,6 @@
 services:
  kener:
-    image: rajnandan1/kener:4.1.5
+    image: rajnandan1/kener:v4.1.7
    container_name: kener
    restart: unless-stopped
    # ports:
@@ -31,7 +31,7 @@ spec:
    spec:
      containers:
        - name: kener
-          image: rajnandan1/kener:4.1.5
+          image: rajnandan1/kener:v4.1.7
          envFrom:
            - configMapRef:
                name: kener-config
@@ -1,6 +1,6 @@
 services:
  n8n:
-    image: docker.n8n.io/n8nio/n8n:2.43.0
+    image: docker.n8n.io/n8nio/n8n:2.43.2
    container_name: n8n
    restart: unless-stopped
    environment:
@@ -31,7 +31,7 @@ spec:
    spec:
      containers:
        - name: n8n
-          image: docker.n8n.io/n8nio/n8n:2.43.0
+          image: docker.n8n.io/n8nio/n8n:2.43.2
          envFrom:
            - configMapRef:
                name: n8n-config
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: netbird-server-service
  namespace: netbird
+  labels:
+    app: netbird-server
 spec:
  selector:
    app: netbird-server
@@ -11,6 +13,10 @@ spec:
      name: http
      targetPort: 80
      protocol: TCP
+    - port: 9090
+      name: metrics
+      targetPort: metrics
+      protocol: TCP
    - port: 3478
      name: stun
      targetPort: 3478
@@ -59,6 +65,9 @@ spec:
            - containerPort: 80
              name: http
              protocol: TCP
+            - containerPort: 9090
+              name: metrics
+              protocol: TCP
            - containerPort: 3478
              name: stun
              protocol: UDP
@@ -0,0 +1,16 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: netbird-server
+  namespace: netbird
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: netbird-server
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -0,0 +1,49 @@
+# Paperless-ngx
+
+Paperless-ngx runs in the `paperless` namespace. It uses the shared PostgreSQL
+service in the `database` namespace and Valkey for its task queue. The document
+library, exports, and consume folder are stored on the `local-path-retain`
+volume. The PVC size is fixed at 50 GiB because this storage class does not
+support volume expansion.
+
+The local route is `https://papers.workstation.internal`; the public route is
+`https://papers.forust.xyz`. Both use TLS. Paperless keeps its own login and
+password authentication. OCR is configured for Russian and English documents.
+
+## Prepare the secret
+
+Create `k8s/secrets.yaml` on the workstation from
+`k8s/secrets.yaml.example`. Set a unique random `PAPERLESS_SECRET_KEY`, a long
+`PAPERLESS_ADMIN_PASSWORD`, and `PAPERLESS_DB_PASSWORD`.
+
+Add the same `PAPERLESS_DB_PASSWORD` value to the local
+`postgres/k8s/secrets.yaml` file. Keep both secret files out of Git. The
+database bootstrap Job creates the `paperless` role and database from the
+shared PostgreSQL secret. The job runs in the `database` namespace and needs
+that namespace's existing `postgres-shared-secrets` Secret.
+
+For example, generate a key with:
+
+```sh
+python3 -c 'import secrets; print(secrets.token_urlsafe(64))'
+```
+
+Then apply the secret before enabling the service:
+
+```sh
+kubectl apply -f paperless/k8s/namespace.yaml
+kubectl apply -f postgres/k8s/secrets.yaml
+kubectl apply -f paperless/k8s/secrets.yaml
+```
+
+The normal deploy workflow applies the remaining manifests when
+`paperless/k8s/active` is present. Verify the rollout and ingress after deploy:
+
+```sh
+kubectl -n paperless rollout status deployment/paperless
+kubectl -n paperless get pods,pvc,services
+```
+
+Back up the `paperless-data` PVC and the shared PostgreSQL database. The PVC
+contains the originals, archived PDFs, and export/consume folders. Valkey has
+no persistent volume; queued tasks are recreated after a restart.
@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: paperless-prod-tls
+  namespace: paperless
+spec:
+  secretName: paperless-prod-tls
+  dnsNames:
+    - papers.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: paperless
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer
@@ -0,0 +1,58 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: paperless-database-init
+  namespace: database
+spec:
+  backoffLimit: 5
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless-database-init
+    spec:
+      restartPolicy: OnFailure
+      containers:
+        - name: create-database
+          image: postgres:17.11-alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              PGPASSWORD="$POSTGRES_ADMIN_PASSWORD" psql \
+                --host postgres \
+                --username postgres \
+                --dbname postgres \
+                --set ON_ERROR_STOP=1 \
+                --set paperless_password="$PAPERLESS_DB_PASSWORD" <<'SQL'
+              SELECT format(
+                'CREATE ROLE paperless LOGIN PASSWORD %L',
+                :'paperless_password'
+              )
+              WHERE NOT EXISTS (
+                SELECT FROM pg_roles WHERE rolname = 'paperless'
+              )
+              \gexec
+              SELECT format('CREATE DATABASE paperless OWNER paperless')
+              WHERE NOT EXISTS (
+                SELECT FROM pg_database WHERE datname = 'paperless'
+              )
+              \gexec
+              SQL
+          env:
+            - name: POSTGRES_ADMIN_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: postgres-shared-secrets
+                  key: POSTGRES_ADMIN_PASSWORD
+            - name: PAPERLESS_DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: postgres-shared-secrets
+                  key: PAPERLESS_DB_PASSWORD
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
@@ -0,0 +1,33 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: paperless-prod
+  namespace: paperless
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`papers.forust.xyz`)
+      kind: Rule
+      services:
+        - name: paperless
+          port: 8000
+  tls:
+    secretName: paperless-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: paperless-local
+  namespace: paperless
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`papers.workstation.internal`)
+      kind: Rule
+      services:
+        - name: paperless
+          port: 8000
+  tls:
+    secretName: internal-wildcard-tls
@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: paperless
@@ -0,0 +1,39 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: paperless-ingress
+  namespace: paperless
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperless
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: traefik
+      ports:
+        - protocol: TCP
+          port: 8000
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: paperless-valkey-ingress
+  namespace: paperless
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperless-valkey
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: paperless
+      ports:
+        - protocol: TCP
+          port: 6379
@@ -0,0 +1,210 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless
+spec:
+  selector:
+    app.kubernetes.io/name: paperless
+  ports:
+    - name: http
+      port: 8000
+      targetPort: http
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperless
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless
+    spec:
+      enableServiceLinks: false
+      containers:
+        - name: paperless
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.3.0
+          ports:
+            - name: http
+              containerPort: 8000
+          env:
+            - name: PAPERLESS_URL
+              value: https://papers.forust.xyz
+            - name: PAPERLESS_ALLOWED_HOSTS
+              value: papers.forust.xyz,papers.workstation.internal
+            - name: PAPERLESS_CSRF_TRUSTED_ORIGINS
+              value: https://papers.forust.xyz,https://papers.workstation.internal
+            - name: PAPERLESS_TIME_ZONE
+              value: Europe/Bratislava
+            - name: PAPERLESS_REDIS
+              value: redis://paperless-valkey:6379
+            - name: PAPERLESS_DBENGINE
+              value: postgresql
+            - name: PAPERLESS_DBHOST
+              value: postgres.database.svc.cluster.local
+            - name: PAPERLESS_DBNAME
+              value: paperless
+            - name: PAPERLESS_DBUSER
+              value: paperless
+            - name: PAPERLESS_DBPASS
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_DB_PASSWORD
+            - name: PAPERLESS_OCR_LANGUAGE
+              value: rus+eng
+            - name: PAPERLESS_OCR_LANGUAGES
+              value: rus
+            - name: PAPERLESS_TASK_WORKERS
+              value: "1"
+            - name: PAPERLESS_ADMIN_USER
+              value: admin
+            - name: PAPERLESS_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_SECRET_KEY
+            - name: PAPERLESS_ADMIN_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_ADMIN_PASSWORD
+          volumeMounts:
+            - name: documents
+              mountPath: /usr/src/paperless/data
+              subPath: data
+            - name: documents
+              mountPath: /usr/src/paperless/media
+              subPath: media
+            - name: documents
+              mountPath: /usr/src/paperless/export
+              subPath: export
+            - name: documents
+              mountPath: /usr/src/paperless/consume
+              subPath: consume
+          startupProbe:
+            httpGet:
+              path: /
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: papers.workstation.internal
+            failureThreshold: 60
+            periodSeconds: 10
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: papers.workstation.internal
+            periodSeconds: 10
+            timeoutSeconds: 5
+          livenessProbe:
+            httpGet:
+              path: /
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: papers.workstation.internal
+            periodSeconds: 30
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              cpu: "2"
+              memory: 2Gi
+      volumes:
+        - name: documents
+          persistentVolumeClaim:
+            claimName: paperless-data
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data
+  namespace: paperless
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: local-path-retain
+  resources:
+    requests:
+      storage: 50Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless-valkey
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless-valkey
+spec:
+  selector:
+    app.kubernetes.io/name: paperless-valkey
+  ports:
+    - name: redis
+      port: 6379
+      targetPort: redis
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless-valkey
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless-valkey
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperless-valkey
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless-valkey
+    spec:
+      containers:
+        - name: valkey
+          image: valkey/valkey:9.1.2-alpine
+          args:
+            - valkey-server
+            - --save
+            - ""
+            - --appendonly
+            - "no"
+          ports:
+            - name: redis
+              containerPort: 6379
+          readinessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            periodSeconds: 10
+          livenessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              cpu: 250m
+              memory: 256Mi
@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: paperless-secrets
+  namespace: paperless
+type: Opaque
+stringData:
+  PAPERLESS_SECRET_KEY: "<GENERATE_WITH_python3_-c_import_secrets;_print(secrets.token_urlsafe(64))>"
+  PAPERLESS_ADMIN_PASSWORD: "<SET_A_LONG_UNIQUE_PASSWORD>"
+  PAPERLESS_DB_PASSWORD: "<SET_THE_SAME_VALUE_AS_database_PAPERLESS_DB_PASSWORD>"
@@ -1,6 +1,6 @@
 services:
  portainer:
-    image: portainer/portainer-ce:2.45.1
+    image: portainer/portainer-ce:2.45.2
    container_name: portainer
    restart: always
    volumes:
@@ -29,7 +29,7 @@ spec:
    spec:
      containers:
        - name: portainer
-          image: portainer/portainer-ce:2.45.1
+          image: portainer/portainer-ce:2.45.2
          ports:
            - containerPort: 9000
          volumeMounts:
@@ -15,3 +15,29 @@ The VictoriaMetrics Operator chart and its CRDs are installed before the
 Kubernetes manifests by the normal deploy workflow. On a cluster where the
 operator CRDs are not installed yet, CI skips the server-side dry-run of the
 `VMAgent` resource; the deploy installs the chart before applying that resource.
+
+## Application metrics
+
+The application ServiceMonitors use a 30s interval and a 10s timeout:
+
+- Headscale: the external Service points to the Compose host on port 19090.
+  A VMServiceScrape uses EndpointSlice discovery for this manually managed target.
+  The Compose configuration must bind metrics to `0.0.0.0:9090`.
+- NetBird: the combined server exports `/metrics` on port 9090. The existing
+  `server.metricsPort` setting enables the listener.
+- Gitea: `GITEA__metrics__ENABLED` enables `/metrics` on the HTTP port. The public
+  ingress excludes this path. The monitor uses the internal Service directly.
+- Immich: `IMMICH_TELEMETRY_INCLUDE=all` enables API and worker metrics on ports
+  8081 and 8082. The monitor scrapes both ports on each server replica.
+
+Deploy through the existing CI and deploy workflow. Gitea and Immich reload their
+ConfigMap changes through Reloader. Check the VMAgent targets after deployment
+and query `up{scraper="victoria",namespace=~"netbird|gitea|immich|headscale"}` in
+VictoriaMetrics. All targets should report 1.
+
+For rollback, revert the application metrics changes, run CI, and deploy the
+revert. Remove the three application ServiceMonitors and the Headscale VMServiceScrape explicitly: the deployment
+workflow applies manifests and does not prune removed resources.
+
+For Headscale rollback, remove its VMServiceScrape and Service label, restore the
+previous Compose metrics bind address, and restart only the Headscale service.
@@ -19,7 +19,7 @@ spec:
          restartPolicy: Never
          containers:
            - name: renovate
-              image: renovate/renovate:44.140.0
+              image: renovate/renovate:44.147.0
              env:
                - name: RENOVATE_PLATFORM
                  value: gitea
@@ -2,7 +2,7 @@ services:
  renovate:
    # Kept in step with renovate/k8s/cronjob.yaml by the "renovate self-update"
    # package rule in renovate/renovate.json.
-    image: renovate/renovate:44.136.0
+    image: renovate/renovate:44.147.0
    container_name: renovate
    restart: "no"
    env_file:
@@ -87,7 +87,8 @@ services:
      - streaming

  jellyseerr:
-    image: fallenbagel/jellyseerr:latest
+    image: ghcr.io/seerr-team/seerr:v3.5.0
+    init: true
    container_name: jellyseerr
    restart: unless-stopped
    environment:
@@ -78,16 +78,29 @@ spec:
 #   issuerRef:
 #     name: letsencrypt-prod
 #     kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: jellyseerr-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: jellyseerr-prod-tls
-#   dnsNames:
-#     - jellyseerr.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: seerr-prod-tls
+  namespace: streaming
+spec:
+  secretName: seerr-prod-tls
+  dnsNames:
+    - seerr.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: jelly-prod-tls
+  namespace: streaming
+spec:
+  secretName: jelly-prod-tls
+  dnsNames:
+    - jelly.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
@@ -1,5 +1,39 @@
 apiVersion: traefik.io/v1alpha1
 kind: IngressRoute
+metadata:
+  name: seerr-prod
+  namespace: streaming
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`seerr.forust.xyz`)
+      kind: Rule
+      services:
+        - name: jellyseerr
+          port: 15055
+  tls:
+    secretName: seerr-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: jelly-prod
+  namespace: streaming
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`jelly.forust.xyz`)
+      kind: Rule
+      services:
+        - name: jellyfin
+          port: 18096
+  tls:
+    secretName: jelly-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
 metadata:
  name: jellyfin-local
  namespace: streaming
@@ -0,0 +1,24 @@
+"""Keep unit-test workflow commands out of the real CI job files."""
+
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+CI_COMMAND_FILES = ('GITHUB_STEP_SUMMARY', 'GITHUB_OUTPUT', 'GITHUB_ENV', 'GITHUB_PATH', 'GITHUB_STATE')
+
+
+class IsolatedCITestCase(unittest.TestCase):
+    def setUp(self):
+        super().setUp()
+        directory = tempfile.TemporaryDirectory(prefix='homelab-test-ci-')
+        self.addCleanup(directory.cleanup)
+        paths = {}
+        for variable in CI_COMMAND_FILES:
+            path = Path(directory.name) / variable
+            path.touch()
+            paths[variable] = str(path)
+        environment = patch.dict(os.environ, paths)
+        environment.start()
+        self.addCleanup(environment.stop)
@@ -0,0 +1,60 @@
+"""Run the real unit tests with external CI files and detect leaked writes."""
+
+import os
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+from unittest.mock import patch
+
+from ci_test_case import CI_COMMAND_FILES, IsolatedCITestCase
+
+
+class CIOutputIsolationTests(IsolatedCITestCase):
+    def test_all_command_files_are_private_and_environment_is_restored(self):
+        with tempfile.TemporaryDirectory() as scratch:
+            external = {variable: str(Path(scratch) / variable) for variable in CI_COMMAND_FILES}
+            for path in external.values():
+                Path(path).write_text('external CI file\n')
+            with patch.dict(os.environ, external):
+                probe = IsolatedCITestCase()
+                probe.setUp()
+                private = []
+                try:
+                    for variable in CI_COMMAND_FILES:
+                        self.assertNotEqual(os.environ[variable], external[variable])
+                        path = Path(os.environ[variable])
+                        private.append(path)
+                        path.write_text('test-only command\n')
+                finally:
+                    probe.doCleanups()
+                for variable in CI_COMMAND_FILES:
+                    self.assertEqual(os.environ[variable], external[variable])
+                    self.assertEqual(Path(external[variable]).read_text(), 'external CI file\n')
+                self.assertTrue(all(not path.exists() for path in private))
+
+    def test_unit_suite_preserves_external_ci_files(self):
+        tests = Path(__file__).resolve().parent
+        modules = sorted(p.stem for p in tests.glob('test_*.py') if p.name != Path(__file__).name)
+        with tempfile.TemporaryDirectory() as scratch:
+            environment = os.environ.copy()
+            environment['PYTHONPATH'] = str(tests) + os.pathsep + environment.get('PYTHONPATH', '')
+            expected = {}
+            for variable in CI_COMMAND_FILES:
+                path = Path(scratch) / variable
+                content = f'external {variable}\n'
+                path.write_text(content)
+                environment[variable] = str(path)
+                expected[path] = content
+            result = subprocess.run(  # noqa: S603 -- Run local test modules with the current Python interpreter.
+                [sys.executable, '-m', 'unittest', *modules, '-q'],
+                cwd=tests.parent,
+                env=environment,
+                capture_output=True,
+                text=True,
+                check=False,
+                timeout=60,
+            )
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            for path, content in expected.items():
+                self.assertEqual(path.read_text(), content, f'Unit tests wrote to external {path.name}')
@@ -9,6 +9,8 @@ import unittest
 from pathlib import Path
 from unittest.mock import patch

+from ci_test_case import IsolatedCITestCase
+
 ROOT = Path(__file__).resolve().parents[1]


@@ -34,7 +36,7 @@ def release(sha='a' * 40):
    }


-class ReleaseGateTests(unittest.TestCase):
+class ReleaseGateTests(IsolatedCITestCase):
    def test_release_rejects_wrong_sha_missing_images_and_mutable_tags(self):
        for mutation in ('sha', 'missing', 'tag'):
            data = release()
@@ -91,8 +93,9 @@ class ReleaseGateTests(unittest.TestCase):
                api.release({'id': 1, 'head_sha': 'a' * 40})


-class SelectionTests(unittest.TestCase):
+class SelectionTests(IsolatedCITestCase):
    def setUp(self):
+        super().setUp()
        self.scratch = tempfile.TemporaryDirectory()
        self.addCleanup(self.scratch.cleanup)
        self.repo = Path(self.scratch.name)
@@ -128,6 +131,23 @@ class SelectionTests(unittest.TestCase):
        self.assertEqual(result['selected']['k8s'], ['one'])
        self.assertEqual(result['helm'], [])

+    def test_nested_service_change_and_owned_image_are_selected(self):
+        directory = self.repo / 'vpn/xui/k8s'
+        directory.mkdir(parents=True)
+        (directory / 'active').touch()
+        image = next(iter(release()['images']))
+        (directory / 'app.yaml').write_text('image: ' + image + ':main\n')
+        baseline_sha = self.commit()
+        baseline = planner.make_plan(self.repo, self.repo, release(baseline_sha), None, 'full', [])
+        (directory / 'app.yaml').write_text('image: ' + image + ':prod\n')
+        result = planner.make_plan(self.repo, self.repo, release(self.commit()), baseline, 'changed', [])
+        self.assertEqual(result['selected']['k8s'], ['vpn/xui'])
+        baseline = result
+        updated = release(result['sha'])
+        updated['images'][image] = 'sha256:' + 'e' * 64
+        result = planner.make_plan(self.repo, self.repo, updated, baseline, 'changed', [])
+        self.assertEqual(result['selected']['k8s'], ['vpn/xui'])
+
    def test_failed_intermediate_deploy_does_not_lose_changes(self):
        (self.repo / 'one/k8s/app.yaml').write_text('kind: StatefulSet\n')
        self.commit()  # This commit failed deploy: baseline must remain initial.
@@ -154,7 +174,7 @@ class SelectionTests(unittest.TestCase):
        self.assertEqual(result['selected']['k8s'], ['one', 'postgres', 'two'])


-class ComposeConfigurationTests(unittest.TestCase):
+class ComposeConfigurationTests(IsolatedCITestCase):
    def test_pin_preserves_project_volumes_paths_and_previous_image(self):
        with tempfile.TemporaryDirectory() as scratch:
            root = Path(scratch)
@@ -162,7 +182,8 @@ class ComposeConfigurationTests(unittest.TestCase):
            source = run / 'source'
            config_repo = root / 'persistent'
            (source / 'headscale').mkdir(parents=True)
-            config_repo.mkdir()
+            (config_repo / 'headscale').mkdir(parents=True)
+            (config_repo / 'headscale/compose.yaml').touch()
            (run / 'release.json').write_text(json.dumps(release()))
            old = 'busybox@sha256:' + 'd' * 64
            new = 'busybox@sha256:' + 'e' * 64
@@ -180,19 +201,41 @@ class ComposeConfigurationTests(unittest.TestCase):
                'volumes': {'data': {'name': 'headscale_data'}},
            }

+            previous_config = json.loads(json.dumps(config))
+            previous_config['services']['app']['command'] = ['old-command']
+            previous_config['services']['app']['environment'] = {'VALUE': 'old'}
+            previous_config['services']['removed'] = {'image': 'busybox:latest'}
+            config['services']['app']['command'] = ['new-command']
+            config['services']['app']['environment'] = {'VALUE': 'new'}
+            config['services']['added'] = {'image': 'busybox:latest'}
+
            def fake_output(*args, **kwargs):
                if args[:2] == ('docker', 'compose'):
                    self.assertEqual(kwargs['cwd'], config_repo)
                    self.assertIn(str(config_repo / 'headscale'), args)
-                    return json.dumps(config)
+                    if '--hash' in args:
+                        return 'app matching-hash'
+                    return json.dumps(
+                        previous_config if str(config_repo / 'headscale/compose.yaml') in args else config
+                    )
                if args[:2] == ('docker', 'ps'):
                    return 'container'
                if args[:2] == ('docker', 'inspect'):
+                    if 'com.docker.compose.config-hash' in args[-1]:
+                        return 'matching-hash'
                    return 'sha256:' + 'f' * 64
                return json.dumps([old])

            with (
-                patch.dict(os.environ, {'CONFIG_REPO': str(config_repo), 'REPO': str(source), 'RUN_DIR': str(run)}),
+                patch.dict(
+                    os.environ,
+                    {
+                        'CONFIG_REPO': str(config_repo),
+                        'REPO': str(source),
+                        'RUN_DIR': str(run),
+                        'HOMELAB_STATE': str(root / 'state'),
+                    },
+                ),
                patch.object(compose_module, 'output', side_effect=fake_output),
                patch.object(compose_module, 'resolve', return_value=new),
            ):
@@ -204,6 +247,57 @@ class ComposeConfigurationTests(unittest.TestCase):
            self.assertEqual(pinned['services']['app']['volumes'], config['services']['app']['volumes'])
            self.assertEqual(pinned['services']['app']['image'], new)
            self.assertEqual(before['services']['app']['image'], old)
+            self.assertEqual(before['services']['app']['command'], ['old-command'])
+            self.assertEqual(before['services']['app']['environment'], {'VALUE': 'old'})
+            self.assertIn('removed', before['services'])
+            self.assertNotIn('added', before['services'])
+
+            def mismatched_output(*args, **kwargs):
+                if args[:2] == ('docker', 'inspect') and 'com.docker.compose.config-hash' in args[-1]:
+                    return 'different-hash'
+                return fake_output(*args, **kwargs)
+
+            with (
+                patch.dict(
+                    os.environ,
+                    {
+                        'CONFIG_REPO': str(config_repo),
+                        'REPO': str(source),
+                        'RUN_DIR': str(run),
+                        'HOMELAB_STATE': str(root / 'state'),
+                    },
+                ),
+                patch.object(compose_module, 'output', side_effect=mismatched_output),
+                patch.object(compose_module, 'resolve', return_value=new),
+                self.assertRaisesRegex(ValueError, 'differs from running config'),
+            ):
+                compose_module.prepare(source / 'headscale/compose.yaml')
+            state = root / 'state'
+            with patch.object(controller, 'STATE', state):
+                state.mkdir()
+                (run / 'status.json').write_text('{"state": "running", "stages": {}}')
+                with patch.object(controller, 'retain_completed'):
+                    controller.finish_success(run, {})
+            self.assertEqual(json.loads((state / 'compose-configs/headscale.json').read_text()), pinned)
+            # A stale persistent checkout must not replace the successful baseline.
+            with (
+                patch.dict(
+                    os.environ,
+                    {
+                        'CONFIG_REPO': str(config_repo),
+                        'REPO': str(source),
+                        'RUN_DIR': str(run),
+                        'HOMELAB_STATE': str(state),
+                    },
+                ),
+                patch.object(compose_module, 'output', side_effect=fake_output),
+                patch.object(compose_module, 'resolve', return_value=new),
+            ):
+                compose_module.prepare(source / 'headscale/compose.yaml')
+            before = json.loads((run / 'compose-before/headscale.json').read_text())
+            self.assertEqual(before['services']['app']['command'], ['new-command'])
+            self.assertIn('added', before['services'])
+            self.assertNotIn('removed', before['services'])
            self.assertEqual((run / 'compose/headscale.json').stat().st_mode & 0o777, 0o600)

    def test_registry_index_and_single_image_descriptors(self):
@@ -214,7 +308,7 @@ class ComposeConfigurationTests(unittest.TestCase):
                )


-class ControllerTests(unittest.TestCase):
+class ControllerTests(IsolatedCITestCase):
    def test_completed_stage_cannot_apply_again(self):
        with tempfile.TemporaryDirectory() as scratch:
            directory = Path(scratch)
@@ -10,10 +10,11 @@ import zipfile
 from pathlib import Path
 from unittest.mock import Mock, patch

+from ci_test_case import IsolatedCITestCase
 from test_cicd import ROOT, controller, release, release_module


-class ArtifactTests(unittest.TestCase):
+class ArtifactTests(IsolatedCITestCase):
    def test_archive_rejects_nested_or_extra_files(self):
        api = object.__new__(release_module.Gitea)
        api.base = 'https://example.test/api/v1/repos/a/b'
@@ -103,7 +104,7 @@ class ArtifactTests(unittest.TestCase):
            self.assertEqual(json.loads((root / 'error-pages.json').read_text())['sha'], 'e' * 40)


-class DurableRunTests(unittest.TestCase):
+class DurableRunTests(IsolatedCITestCase):
    def test_duplicate_start_only_reattaches(self):
        with tempfile.TemporaryDirectory() as scratch:
            state = Path(scratch)
@@ -203,7 +204,7 @@ class DurableRunTests(unittest.TestCase):
            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')


-class FailureSummaryTests(unittest.TestCase):
+class FailureSummaryTests(IsolatedCITestCase):
    def test_build_failure_keeps_progress_and_does_not_expose_exception_text(self):
        with tempfile.TemporaryDirectory() as scratch:
            summary = Path(scratch) / 'summary.md'
@@ -225,6 +226,77 @@ class FailureSummaryTests(unittest.TestCase):
            self.assertIn('xdfnx-homepage', content)
            self.assertNotIn('private value', content)

+    def test_invalid_digest_is_not_reported_as_a_completed_image(self):
+        for digest in ('invalid-private-metadata', None, ['invalid']):
+            with self.subTest(digest=digest), tempfile.TemporaryDirectory() as scratch:
+                root = Path(scratch)
+                summary = root / 'summary.md'
+                name = 'error-pages'
+                context, dockerfile = release_module.IMAGES[name]
+                plan = {
+                    'sha': 'a' * 40,
+                    'targets': [
+                        {
+                            'name': name,
+                            'context': context,
+                            'dockerfile': dockerfile,
+                            'inputs': 'c' * 64,
+                            'reuse_digest': None,
+                        }
+                    ],
+                }
+
+                def fake_command(*args, digest=digest, **_kwargs):
+                    if args[:3] == ('docker', 'buildx', 'build'):
+                        Path(args[args.index('--metadata-file') + 1]).write_text(
+                            json.dumps({'containerimage.digest': digest})
+                        )
+                    return ''
+
+                with (
+                    patch.dict(
+                        os.environ,
+                        {
+                            'GITHUB_STEP_SUMMARY': str(summary),
+                            'GITHUB_SHA': 'a' * 40,
+                            'REGISTRY_USERNAME': 'test',
+                            'REGISTRY_PASSWORD': 'placeholder',
+                        },
+                    ),
+                    patch.object(release_module, 'checked_plan', return_value=plan),
+                    patch.object(release_module.Path, 'home', return_value=root),
+                    patch.object(release_module, 'command', side_effect=fake_command),
+                    patch.object(subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)),
+                    self.assertRaisesRegex(ValueError, 'invalid digest'),
+                ):
+                    release_module.build(root / 'image.json', name, root / 'plan.json')
+                self.assertFalse((root / 'image.json').exists())
+                content = summary.read_text()
+                self.assertIn('**failure**', content)
+                self.assertIn('### Built\n- None', content)
+                self.assertIn('### Completed image digests\n- None', content)
+                self.assertNotIn('invalid-private-metadata', content)
+
+    def test_successful_image_result_does_not_claim_complete_release(self):
+        def complete_image(_output, report, _name, _plan):
+            report.update(phase='Image result file saved', built=['error-pages'])
+            report['images']['gcr.forust.xyz/forust/error-pages'] = 'sha256:' + 'b' * 64
+
+        with (
+            patch.dict(os.environ, {'GITHUB_SHA': 'a' * 40}),
+            patch.object(
+                release_module,
+                'build_images',
+                side_effect=complete_image,
+            ),
+        ):
+            release_module.build(Path('unused.json'), 'error-pages', Path('unused-plan.json'))
+        content = Path(os.environ['GITHUB_STEP_SUMMARY']).read_text()
+        self.assertIn('## Image build result `error-pages`', content)
+        self.assertIn('Commit: `' + 'a' * 40 + '`', content)
+        self.assertIn('final build job must publish the complete release', content)
+        self.assertNotIn('## Image release', content)
+
    def test_deploy_failure_reports_completed_apply_and_rollback_result(self):
        with tempfile.TemporaryDirectory() as scratch:
            state = Path(scratch)
@@ -261,7 +333,7 @@ class FailureSummaryTests(unittest.TestCase):
            self.assertIn('Compose requires manual recovery', content)


-class InstallerTests(unittest.TestCase):
+class InstallerTests(IsolatedCITestCase):
    def test_version_comparison_is_exact_without_network_or_host_packages(self):
        with tempfile.TemporaryDirectory() as scratch:
            root = Path(scratch)
@@ -3,10 +3,10 @@
 import json
 import os
 import tempfile
-import unittest
 from pathlib import Path
 from unittest.mock import Mock, call, patch

+from ci_test_case import IsolatedCITestCase
 from test_cicd import release, release_module


@@ -26,7 +26,7 @@ def plan_data(changed):
    return {'sha': 'a' * 40, 'targets': targets}


-class MatrixTests(unittest.TestCase):
+class MatrixTests(IsolatedCITestCase):
    def test_no_change_one_image_all_images_and_missing_baseline(self):
        for changed in (set(), {'error-pages'}, set(release_module.IMAGES)):
            with self.subTest(changed=changed), tempfile.TemporaryDirectory() as scratch:
@@ -7,11 +7,14 @@ import tempfile
 import unittest
 from pathlib import Path

+from ci_test_case import IsolatedCITestCase
+
 ROOT = Path(__file__).resolve().parents[1]


-class NetbirdRuntimeTests(unittest.TestCase):
+class NetbirdRuntimeTests(IsolatedCITestCase):
    def setUp(self):
+        super().setUp()
        self.temp = tempfile.TemporaryDirectory()
        self.addCleanup(self.temp.cleanup)
        self.root = Path(self.temp.name)
@@ -1,6 +1,6 @@
 services:
  traefik:
-    image: traefik:v3.7.13
+    image: traefik:v3.7.14
    container_name: traefik
    restart: unless-stopped
    command:
@@ -3,7 +3,7 @@ hostNetwork: false
 image:
  registry: docker.io/library
  repository: traefik
-  tag: v3.7.13
+  tag: v3.7.14

 securityContext:
  capabilities: