fix(immich): size postgres probes for HDD stalls
ci / lint-compose (push) Successful in 4s
ci / lint-actionlint (push) Successful in 1s
ci / lint-shellcheck (push) Successful in 2s
ci / lint-prettier (push) Successful in 3s
ci / lint-ruff (push) Successful in 2s
ci / lint-yaml (push) Successful in 2s
ci / lint-dockerfiles (push) Successful in 2s
ci / validate (push) Successful in 3s
renovate-ci / validate-renovate (push) Successful in 17s
ci / build (push) Successful in 13s

Postmaster was SIGKILLed in a loop: 70s fsync stalls on the loaded rotational disk outlasted the 5-minute startup budget and the 60s liveness tolerance, and every kill bought another full WAL replay. Startup budget 15min, liveness 5x60s. Already applied live with kubectl; this keeps git in sync.
This commit is contained in:
forust committed 2026-09-29 12:36:04 +02:00
1 parent a2af854867
commit b6e1dc0362
1 file changed
+10 -3
+10 -3
View File
@@ -90,10 +90,16 @@ spec:
# rotational disk on a loaded single node, and pg_isready can take
# seconds during WAL recovery. A 1s timeout kills the container
# mid-recovery and restarts the spiral.
#
# Budgets are sized for HDD stalls, not for a healthy disk: fsync of
# a single file was observed taking 70s under node IO pressure, so
# the startup budget is 15 minutes and liveness tolerates 5 minutes
# of unresponsiveness. Killing a stalled-but-healthy postmaster only
# buys another full WAL replay, which is more IO, not less.
startupProbe:
exec:
command: ["sh", "-c", "pg_isready -U immich -d immich"]
failureThreshold: 60
failureThreshold: 180
periodSeconds: 5
timeoutSeconds: 5
readinessProbe:
@@ -105,8 +111,9 @@ spec:
exec:
command: ["sh", "-c", "pg_isready -U immich -d immich"]
initialDelaySeconds: 30
periodSeconds: 20
timeoutSeconds: 5
periodSeconds: 60
timeoutSeconds: 10
failureThreshold: 5
# The image template sets shared_buffers to 512MB, and the vchord and
# vectors workers are Rust binaries with a real RSS footprint on top
# of postmaster, checkpointer and friends. 1Gi was enough to start