676 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
forust	872f64b887	fix(ci): silence intentional SC2029 in ssh-run.sh	2026-09-29 14:54:17 +02:00
forust	a90fb19ed4	fix(traefik): size probes for HDD stalls ... Single replica is the whole ingress; liveness kills at 2s timeouts took every public service down in a loop. Same stall-sized budgets as postgres/metallb. Applied live via helm (pinned 41.5.0, values from git).	2026-09-29 14:47:53 +02:00
forust	6f4cd03f4b	feat(deploy): serialize apply stages and wait for calm node ... apply-k8s and apply-compose share a workstation flock so host docker churn never overlaps cluster churn. Each helm upgrade and the apply loop wait up to 10m for load <28 first, so a deploy never piles onto an already-hot node (the load-40/netbird-death/pending-helm spiral).	2026-09-29 14:42:56 +02:00
forust	e56662194b	fix(ci): log in to registry for manifest-only main pushes ... The pin step writes manifest PUTs on every main push, but login was gated on services != ''. Manifest-only pushes skipped login and pushed anonymously (401); this only worked before via a stale persistent login on the old runner.	2026-09-29 14:21:57 +02:00
forust	4856348a6e	chore(searxng,glance): disable services ... Workloads, services, routes and certs removed from the cluster. Configs, manifests and PVCs kept for easy re-enable via k8s/active.	2026-09-29 13:59:59 +02:00
forust	49d0da1dd0	chore(termix): disable service ... Deployment was stuck in ImagePullBackOff and only generated log churn. Workload, service, routes and certs removed from the cluster; PVC, config and manifests kept so it can be re-enabled by restoring k8s/active.	2026-09-29 13:58:27 +02:00
forust	6bc938436f	Merge pull request 'chore(deps): update grafana/grafana docker tag to v13.2.3' (#67) from renovate/grafana-monorepo into main ... Reviewed-on: #67	2026-09-29 11:57:17 +00:00
renovate-bot	04b33d0736	chore(deps): update grafana/grafana docker tag to v13.2.3	2026-09-29 11:57:17 +00:00
forust	5dcad7eb38	fix(ci): resolve uv from BIN_DIR in install-ci-tools ... Callers prepend BIN_DIR to PATH only after the script exits, so the bare uv invocation in install_uv_tool died with 127 on clean runners. Export BIN_DIR to PATH inside the script and invoke the just-installed binary by absolute path.	2026-09-29 13:31:28 +02:00
forust	d0872bc918	feat(deploy): autodeploy off unless AUTODEPLOY=true ... Pushes no longer reach the cluster by default; set the AUTODEPLOY repo variable to 'true' to re-enable, or dispatch manually. Skips propagate through the existing needs chain.	2026-09-29 12:54:17 +02:00
forust	91dd749a29	feat(deploy): AUTODEPLOY kill-switch via repo variable ... Set AUTODEPLOY=false under Settings -> Actions -> Variables and pushes stop deploying with no commit; unset means on. Manual Run workflow always bypasses the switch. The existing needs/skipped chaining propagates the skip through verify and smoke untouched.	2026-09-29 12:52:35 +02:00
forust	b6e1dc0362	fix(immich): size postgres probes for HDD stalls ... Postmaster was SIGKILLed in a loop: 70s fsync stalls on the loaded rotational disk outlasted the 5-minute startup budget and the 60s liveness tolerance, and every kill bought another full WAL replay. Startup budget 15min, liveness 5x60s. Already applied live with kubectl; this keeps git in sync.	2026-09-29 12:36:04 +02:00
forust	a2af854867	fix(alerts): measure traefik latency per route via exported_service ... The service label the rule grouped by is the scrape target name, not the backend: with honorLabels=false the per-route value traefik emits is renamed to exported_service, so the old rule measured one global aggregate and both exclusions matched nothing. Group the latency and 5xx alerts by exported_service with exclusions in traefik's real namespace-routename-hash format. Already applied live with kubectl; this commit keeps git in sync.	2026-09-29 09:55:02 +02:00
forust	314ec0cda7	chore(prometheus): drop dead grafana routes and cert ... No grafana is deployed from this stack, so grafana-prod/grafana-local only ever served 503s. Live objects already deleted directly; this keeps git from resurrecting them on the next deploy.	2026-09-29 09:33:38 +02:00
forust	e45ae10204	feat(immich): migrate postgres 14 to 16 with dump/restore ... Reverts the v16 revert: data migrated via pg_dumpall from PG14 into a fresh PG16 data directory (same extension versions vectorchord 0.4.3/pgvectors 0.2.0). Verified 1944 assets, 2 users, 10 albums on 16.10. Note: the restore OOMed once at the 2Gi limit during index builds and recovered via WAL replay; worth watching under PG16 load.	2026-09-29 09:16:58 +02:00
forust	357ee26111	Revert "Merge pull request 'chore(deps): update ghcr.io/immich-app/postgres docker tag to v16' (#65) from renovate/ghcr.io-immich-app-postgres-16.x into main" ... This reverts commit 0c9743e241, reversing changes made to 2cb06debc5.	2026-09-29 08:25:43 +02:00
forust	436fd1ecae	chore: extract userbot subtree to its own repo ... userbot/ (bot plus panel) now lives at /home/forust/userbot as a clone of forust/userbot instead of a subtree in homelab. Cleans up the pipeline references that only existed for it: scan-deps, test-backend and test-frontend jobs, the userbot build matrix entries, the deploy panel hook, and the userbot-only pyrightconfig. Running cluster workloads are untouched; homelab just stops building and testing upstream's code.	2026-09-29 08:24:14 +02:00
forust	a564dd8b67	fix(alerts): exclude netbird streams from traefik latency alert ... SignalExchange ConnectStream holds 60s gRPC streams by design; at night they exceed 5% of samples and pin P95 to the 5.0s bucket ceiling, flapping the alert. Also fixes the stale xui exclusion pattern, which matched no real service label.	2026-09-29 08:12:07 +02:00
forust	c2c90c490d	Merge pull request 'chore(deps): update ghcr.io/autobrr/netronome docker tag to v0.15.0' (#62) from renovate/ghcr.io-autobrr-netronome-0.x into main ... Reviewed-on: #62	2026-09-28 21:05:42 +00:00
renovate-bot	da0f9e84c3	chore(deps): update ghcr.io/autobrr/netronome docker tag to v0.15.0	2026-09-28 21:05:42 +00:00
forust	3ecc12300a	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.28' (#61) from renovate/container-patch-updates into main ... Reviewed-on: #61	2026-09-28 21:05:33 +00:00
renovate-bot	19029fa012	chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.28	2026-09-28 21:05:33 +00:00
forust	2a5d690e34	Merge pull request 'chore(deps): update netbirdio/dashboard docker tag to v2.94.0' (#63) from renovate/netbirdio-dashboard-2.x into main ... Reviewed-on: #63	2026-09-28 21:05:25 +00:00
renovate-bot	b53ce36d89	chore(deps): update netbirdio/dashboard docker tag to v2.94.0	2026-09-28 21:05:25 +00:00
forust	a8c4e9bcbe	Merge pull request 'chore(deps): update renovate/renovate docker tag to v44.117.0' (#64) from renovate/renovate-self-update into main ... Reviewed-on: #64	2026-09-28 21:02:09 +00:00
renovate-bot	761f97ef8e	chore(deps): update renovate/renovate docker tag to v44.117.0	2026-09-28 21:02:09 +00:00
forust	0c9743e241	Merge pull request 'chore(deps): update ghcr.io/immich-app/postgres docker tag to v16' (#65) from renovate/ghcr.io-immich-app-postgres-16.x into main ... Reviewed-on: #65	2026-09-28 21:01:58 +00:00
renovate-bot	2b3c28a46b	chore(deps): update ghcr.io/immich-app/postgres docker tag to v16	2026-09-28 21:01:58 +00:00
forust	2cb06debc5	fix(deploy): retry registry lookups with a timeout ... A single blink of the registry failed render_pinned for the whole file and redded the apply stage. registry_digest now retries 3 times under a 25s timeout with a warning per attempt; empty still means unresolvable and callers report it by name as before.	2026-09-28 22:52:54 +02:00
forust	a6af69dca0	fix(k8s): Recreate singletons and trim requests for scheduler headroom ... RollingUpdate with default maxSurge needs a spare pod the single node does not have (99% CPU requested), so multi-workload restarts end Pending and verify times out. Recreate on all replicas:1 Deployments (immich-server and bentopdf keep RollingUpdate at replicas 2). Also trims CPU/memory requests toward measured use (adguard, authentik, gitea, netbox, uptime-kuma, netbird-server) and gives traefik requests/limits so it is no longer BestEffort.	2026-09-28 22:36:46 +02:00
forust	76f39da90c	fix(ci): skip heavy jobs on renovate branches, automerge digest and patch ... Renovate branches only carry version/digest bumps, so scan-deps, test-backend, test-frontend and build just burn runner time on the box that also serves prod. Static checks and validate still run. Digest and patch updates automerge (playwright, helm and major rules below still override to no-automerge). Also replaces deprecated helm --atomic with --wait --rollback-on-failure.	2026-09-28 22:19:57 +02:00
forust	86730ff0c4	Merge pull request 'chore(deps): update ghcr.io/c4illin/convertx docker tag to v0.19.0' (#55) from renovate/ghcr.io-c4illin-convertx-0.x into main ... Reviewed-on: #55	2026-09-28 19:57:17 +00:00
renovate-bot	af66d3e7fd	chore(deps): update ghcr.io/c4illin/convertx docker tag to v0.19.0	2026-09-28 19:57:17 +00:00
forust	deeaefa695	Merge pull request 'chore(deps): update ghcr.io/henriquesebastiao/downtify docker tag to v3.2.0' (#60) from renovate/ghcr.io-henriquesebastiao-downtify-3.x into main ... Reviewed-on: #60	2026-09-28 19:57:05 +00:00
renovate-bot	6999dd2728	chore(deps): update ghcr.io/henriquesebastiao/downtify docker tag to v3.2.0	2026-09-28 19:57:05 +00:00
forust	742d78944b	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.27' (#54) from renovate/container-patch-updates into main ... Reviewed-on: #54	2026-09-28 19:56:55 +00:00
renovate-bot	e451c97dfc	chore(deps): update container patch updates	2026-09-28 19:56:45 +00:00
forust	33c54ac830	Merge pull request 'chore(deps): update docker.io/valkey/valkey:9 docker digest to 418652c' (#59) from renovate/docker.io-valkey-valkey-9 into main ... Reviewed-on: #59	2026-09-28 19:56:27 +00:00
renovate-bot	b09d718310	chore(deps): update docker.io/valkey/valkey:9 docker digest to 418652c	2026-09-28 16:20:21 +00:00
forust	b4f76373bb	fix(gitea): serve issue search from postgres instead of reindexing on boot ... cron.rebuild_issue_indexer runs at start, so every gitea pod restart reindexed the whole issue index and read ~7MB/s off the rotational disk for an hour.	2026-09-28 17:01:36 +02:00
forust	74adf38d63	feat(alerts): cover OOM kills, restart loops and evictions ... The OOMKilled container behind the immich crash loop was invisible: PodCrashLooping only fires once kubelet has already given up and started the backoff.	2026-09-28 17:01:36 +02:00
forust	f5b2f89f38	style(immich): match repo prettier quoting in compose file	2026-09-28 17:00:41 +02:00
forust	8e63284240	feat(immich): add self-hosted photo backup with dedicated postgres ... Server x2, machine learning, valkey and VectorChord postgres on local storage, Traefik routes for external and internal access.	2026-09-28 16:42:05 +02:00
forust	1d81410cd8	fix(traefik): persist plugin storage on a PVC ... Mount traefik-plugins PVC at /plugins-storage instead of the chart default emptyDir, so the crowdsec-bouncer download survives node reboots. Without this Traefik starts before the network is ready, the download from plugins.traefik.io times out, plugins get disabled and every route behind the middleware returns 404/503 until a manual restart.	2026-09-28 13:36:46 +02:00
forust	2f891a5d31	fix(postgres): give probes room on an I/O-bound single node ... pg_isready with the 1s default times out under I/O stall and kubelet kills a healthy postgres mid-recovery; each kill restarts a multi-minute fsync from zero and loops forever. readiness/liveness timeout 5s, liveness threshold 5, startup budget 15min.	2026-09-28 12:35:16 +02:00
forust	cda0022d81	fix(renovate): reap finished job pods with ttlSecondsAfterFinished ... History limits never delete manual 'create job --from' runs, so Failed pods accumulated for a week. Keep a day for debugging, reap the rest.	2026-09-28 12:03:26 +02:00
forust	dde6b1c743	fix(deploy): recover helm releases from pending-* and skip helm-owned rollbacks ... An --atomic upgrade whose own rollback never finishes leaves the release in pending-*, blocking every future run until a human rolls back (loki rev 18/21). Recover automatically before and after each upgrade, and fail loud when recovery does not land on deployed. Also skip helm-managed workloads in rollback_workloads: rollout undo there would step back to the revision --atomic just escaped.	2026-09-28 12:03:26 +02:00
forust	7c4843c88c	fix(loki): unblock gateway rollout on a single node ... Chart default is required podAntiAffinity on hostname plus RollingUpdate 25%/25%, which is maxUnavailable=0 at replicas=1: the new pod stays Unschedulable while the old one lives, and the old one never leaves while the new one is not Ready. Null the affinity (an empty map deep-merges with the default and keeps the rule) and set maxSurge/maxUnavailable to 1.	2026-09-28 11:33:38 +02:00
forust	f9e4623ade	fix(k8s): size the remaining workloads against measured use ... Finishes the sizing pass over every workload the deploy actually manages. Each request is at or above the container's p95 over the last seven days, so nothing is sized below what it is known to use, and each limit is between 1.6x and 5x the observed max, which is the figure that decides whether a burst gets an OOMKill. Some of these go up, and that is the point. adguard was holding 975M against a 500Mi request and netbox 962M against 512Mi, so both sat permanently above their own request and were standing eviction candidates on a node that has about 300M of headroom. Raising a request costs scheduler room; leaving it low costs the pod its place in the queue when the node gets tight. Others come down. loki ran with a 2Gi limit on 224M, gitea 1.5Gi on 305M, the authentik worker 1Gi on 305M, and a tail of single-purpose pods -- redis, glance, the two homepages, cfddns, session-keeper, the netbird dashboard, the loki gateway -- each reserved 4x to 16x more than they have ever touched. prometheus gets the opposite treatment: 768Mi/2560Mi, above its p95, because it compacts its TSDB in place and that is a burst worth budgeting for rather than throttling. Two of these limits are close enough to the observed max to be worth watching rather than trusting: adguard at 1.5x, and its DNS cache grows monotonically, so the ceiling is a date, not a margin. That was true before this change too; the pod sizing does not fix it and the cache needs bounding. CPU limits are untouched throughout. Leaving postgres alone as well: it sits in an uncommitted file that belongs to other work in progress. Verified: every request is at or above p95 and every limit above the observed max across all 74 containers, and 16/16 local gates pass.	2026-09-28 10:38:19 +02:00
forust	2ad4fa1b82	chore(portainer): stop deploying a container manager nothing routes to ... Portainer had been running for 111 days with a 512Mi request and a 2Gi limit against 50M of measured use, on a node that is short of memory. It is a UI over the Docker socket; nothing in the repo or the cluster depends on it. The marker goes, not the manifests. `portainer/k8s/active` is what puts these files in the deploy's manifest set, so without it the next push leaves the namespace alone and the manifests stay on disk for a one-command return. This also matters for the smoke stage: that host list is built from the active directories, so `portainer.forust.xyz` leaves it and the new router check does not go looking for a route to a service we just retired. In the cluster the Deployment, the Service and both IngressRoutes are deleted. The routes go first: leaving an IngressRoute behind a deleted Service keeps a Traefik router pointing at nothing, which answers 502 while looking perfectly healthy to the stage that just started checking for routers. Deliberately kept, so this is reversible rather than destructive: the namespace, the 2Gi `portainer-data-pvc` and both Certificates stay. Re-enabling is `git checkout HEAD~1 -- portainer/k8s/active` plus an apply, and no Let's Encrypt quota is spent reissuing the production certificate. `glance` still links to `portainer.forust.xyz` and that tile will now be a 404. Left alone on purpose. The Cloudflare record is manual and cfddns only ever creates records, so `portainer.forust.xyz` keeps resolving until it is removed in the dashboard, same as `dockmon.forust.xyz`. Verified: no Traefik router matches portainer any more, all 19 hosts left in the smoke list still have a router, and 16/16 local gates pass.	2026-09-28 10:22:51 +02:00