fix(deploy): retry registry lookups with a timeout
ci / lint-compose (push) Successful in 3s
ci / lint-actionlint (push) Successful in 1s
ci / lint-shellcheck (push) Successful in 2s
ci / lint-prettier (push) Successful in 3s
ci / lint-ruff (push) Successful in 1s
ci / lint-yaml (push) Successful in 2s
ci / lint-dockerfiles (push) Successful in 1s
ci / scan-deps (push) Successful in 19s
ci / test-backend (push) Successful in 8s
ci / test-frontend (push) Successful in 10s
ci / validate (push) Successful in 2s
renovate-ci / validate-renovate (push) Successful in 14s
ci / build (push) Successful in 7s
ci / lint-compose (push) Successful in 3s
ci / lint-actionlint (push) Successful in 1s
ci / lint-shellcheck (push) Successful in 2s
ci / lint-prettier (push) Successful in 3s
ci / lint-ruff (push) Successful in 1s
ci / lint-yaml (push) Successful in 2s
ci / lint-dockerfiles (push) Successful in 1s
ci / scan-deps (push) Successful in 19s
ci / test-backend (push) Successful in 8s
ci / test-frontend (push) Successful in 10s
ci / validate (push) Successful in 2s
renovate-ci / validate-renovate (push) Successful in 14s
ci / build (push) Successful in 7s
A single blink of the registry failed render_pinned for the whole file and redded the apply stage. registry_digest now retries 3 times under a 25s timeout with a warning per attempt; empty still means unresolvable and callers report it by name as before.
This commit is contained in:
1 parent
a6af69dca0
commit
2cb06debc5
2 files changed
+28
-13
No files matched your search
@@ -233,13 +233,28 @@ registry_digest() {
|
||||
# pipefail reports the rightmost non-zero stage, so a ref the registry does not
|
||||
# have would abort the caller at the assignment instead of yielding an empty
|
||||
# string. The callers check for empty themselves and report it by name.
|
||||
docker manifest inspect "$1" 2>/dev/null \
|
||||
| jq -r --arg arch "$arch" '
|
||||
.manifests[]?
|
||||
| select(.platform.os == "linux" and .platform.architecture == $arch)
|
||||
| .digest
|
||||
' 2>/dev/null \
|
||||
| head -1 || true
|
||||
#
|
||||
# Retried with a hard timeout because the registry has a known hang mode (and
|
||||
# a known blink mode: a single failed lookup aborts the whole apply file in
|
||||
# render_pinned). A short sleep between attempts lets a restarting registry
|
||||
# come back instead of failing the deploy on one bad second.
|
||||
local attempt=0 digest=""
|
||||
while [ "$attempt" -lt 3 ]; do
|
||||
digest="$(timeout 25s docker manifest inspect "$1" 2>/dev/null \
|
||||
| jq -r --arg arch "$arch" '
|
||||
.manifests[]?
|
||||
| select(.platform.os == "linux" and .platform.architecture == $arch)
|
||||
| .digest
|
||||
' 2>/dev/null \
|
||||
| head -1 || true)"
|
||||
[ -n "$digest" ] && break
|
||||
attempt=$((attempt + 1))
|
||||
if [ "$attempt" -lt 3 ]; then
|
||||
echo "WARNING: registry lookup of $1 failed (attempt $attempt/3), retrying in 5s" >&2
|
||||
sleep 5
|
||||
fi
|
||||
done
|
||||
printf '%s' "$digest"
|
||||
}
|
||||
|
||||
# The commit this deploy is for: what CI validated, or - on a manual dispatch,
|
||||
|
||||
@@ -95,12 +95,12 @@ jobs:
|
||||
# The 45 minutes this was last raised to 45 were still not enough, and the
|
||||
# job logs for those runs no longer exist, so what actually consumed the
|
||||
# budget is not known - the two measurable candidates above account for
|
||||
# ~15 of it. The one unbounded thing left in this stage is
|
||||
# `docker manifest inspect` at deploy-lib.sh:236, which has no timeout
|
||||
# against a registry with a known hang mode. Bound it, and make the stage
|
||||
# announce what it is working on, before spending any of that on a larger
|
||||
# ceiling: a stage that is killed with a diagnosable last line is a bug
|
||||
# report, one that vanishes is not.
|
||||
# ~15 of it. The unbounded `docker manifest inspect` against the registry's
|
||||
# known hang mode is now bounded inside registry_digest (25s timeout, 3
|
||||
# attempts): a dead registry fails each owned image after ~85s instead of
|
||||
# hanging the stage, and a blinking one is retried instead of failing the
|
||||
# whole apply file. Still open: make the stage announce which manifest it
|
||||
# is working on, so a killed run leaves a diagnosable last line.
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
|
||||
Reference in new issue
Block a user