ci: deploy the image the commit built, not whatever the tag points at

Every service tracked the mutable `:prod` tag, so a deploy applied whatever
that tag happened to name at the time rather than the commit it was
deploying. A rollback had no way to state what it was rolling back to, and
two deploys of one commit could land different images.

CI now publishes an immutable `sha-<commit12>` tag beside `:prod` on main,
and re-tags it for every image a push did not rebuild. That re-tag copies
the manifest list, so no layer moves. The deploy resolves the immutable tag
to a digest and pins the workload to it, and only falls back to the moving
tag when the immutable one cannot be resolved -- which it says out loud,
because that fallback is the deploy quietly ceasing to be reproducible from
its own commit.

The image list comes out of the tree with git grep rather than being written
out a second time, so adding a service no longer means keeping two lists in
step.

build also gains the three jobs it was skipping -- scan-deps, test-backend,
test-frontend -- so a change that breaks them cannot be tagged at all. The
two run blocks where a mid-loop failure was survivable now run under
set -euo pipefail: the build loop and the service detector both carried on
past an error and could report a green build having produced nothing.

The registry password moves from run: substitution into an env: block. A
quote, a backtick or a $(...) in the password is parsed as shell before the
command ever runs, and a login that failed that way looked exactly like a
build that failed.

The apply and verify timeouts stay at 45 and 30 minutes. The comments now
record the arithmetic that says so rather than leaving the numbers to be
raised on the next scare: three no-op helm upgrades run 3-5 minutes, one
broken release is a single 10 minute rollback because the loop aborts on
the first failure, and the apply loop itself is about a minute. That is
roughly 15 minutes of work against a 45 minute budget. verify is 32
workloads at 8 wide -- four waves of 300 seconds, 20 minutes -- which
leaves room for two serial rollbacks, and only becomes derivable at 45 once
rollback_workloads is parallelised.

.gitea/workflows/deploy-lib.sh

@@ -242,11 +242,49 @@ registry_digest() {
     | head -1 || true
 }
 
+# The commit this deploy is for: what CI validated, or - on a manual dispatch,
+# whatever stage_preflight just checked out.
+deploy_commit() {
+  local c="${DEPLOY_SHA:-}"
+  [ -n "$c" ] || c="$(git -C "$REPO" rev-parse HEAD 2>/dev/null || true)"
+  printf '%.12s' "${c:-}"
+}
+
+# Resolves one of our image refs to the digest THIS commit's build produced.
+#
+# A manifest naming `:prod` names a pointer, not a version, and the deploy
+# resolves it when the apply runs - which is not when CI ran it. Deploy runs are
+# queued rather than cancelled (see deploy.yaml), so two pushes in a row leave
+# the first deploy resolving the second push's build: the right manifests with
+# the wrong code, and nothing anywhere reports it. ci therefore publishes every
+# image it ships under `sha-<commit12>`, a name that cannot move, and that is
+# the name resolved here.
+#
+# The fallback to the plain tag is for an image this pipeline never built. It
+# reports itself, because a fallback nobody sees is the failure this removes.
+pinned_digest() {
+  local ref="$1" commit pinned
+  commit="$(deploy_commit)"
+  if [ -n "$commit" ]; then
+    pinned="$(registry_digest "${ref%:*}:sha-$commit")"
+    if [ -n "$pinned" ]; then
+      printf '%s' "$pinned"
+      return 0
+    fi
+  fi
+  pinned="$(registry_digest "$ref")"
+  if [ -n "$pinned" ]; then
+    echo "WARNING: ${ref} carries no sha-${commit:-<unknown>} tag; resolved the moving tag instead" >&2
+  fi
+  printf '%s' "$pinned"
+}
+
 # Rewrites our own images to immutable digests on the way into the cluster.
 # Reads a manifest stream on stdin, writes the pinned stream to stdout.
 #
-# A digest is not knowable when a manifest is written, so it is resolved here, at
-# apply time, and never committed: git keeps a readable `:prod` tag. That is what
+# A digest is not knowable when a manifest is written, so it is never committed:
+# git keeps a readable `:prod` tag and the exact bytes are chosen here, at apply
+# time, from the tag ci published for the commit being deployed. That is what
 # makes rollback mean something. `kubectl rollout undo` restores the previous
 # ReplicaSet's pod template verbatim, and a template naming a digest restores the
 # exact bytes that were serving before. A template naming a moving tag does not —
@@ -271,7 +309,7 @@ render_pinned() {
 
   while read -r ref; do
     [ -n "$ref" ] || continue
-    digest="$(registry_digest "$ref")"
+    digest="$(pinned_digest "$ref")"
     if [ -z "$digest" ]; then
       echo "ERROR: cannot resolve ${ref} in the registry; applying nothing." >&2
       echo "       The build job has to push that tag before the deploy resolves it." >&2
@@ -339,7 +377,7 @@ restart_stale_images() {
   while read -r ns target image; do
     [ -n "${target:-}" ] || continue
     if [ -z "${digests[$image]:-}" ]; then
-      digests[$image]="$(registry_digest "$image")"
+      digests[$image]="$(pinned_digest "$image")"
     fi
     want="${digests[$image]}"
     if [ -z "$want" ]; then