Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 forust
|
5b3ef39102
|
No files matched your search
@@ -1,191 +0,0 @@
|
||||
# Инструкция: Анализ хранилища Kubernetes и настройка NFS
|
||||
|
||||
## Цель
|
||||
Проанализировать текущую конфигурацию хранилища Kubernetes и подготовить план внедрения NFS StorageClass для сохранения данных при удалении namespace.
|
||||
|
||||
## 1. Собрать информацию о кластере
|
||||
|
||||
### 1.1. Версия Kubernetes и тип дистрибутива
|
||||
```bash
|
||||
kubectl version --short
|
||||
# или
|
||||
kubectl version
|
||||
```
|
||||
|
||||
Определить, используется ли k3s, k8s, microk8s и т.д.:
|
||||
```bash
|
||||
# Проверить наличие k3s
|
||||
which k3s
|
||||
# Проверить процесс
|
||||
ps aux | grep -E 'kube|k3s'
|
||||
```
|
||||
|
||||
### 1.2. StorageClass
|
||||
```bash
|
||||
kubectl get storageclass -o wide
|
||||
```
|
||||
|
||||
Запомнить:
|
||||
- `PROVISIONER` — какой драйвер используется
|
||||
- `RECLAIMPOLICY` — Delete или Retain
|
||||
- Какой StorageClass помечен как `(default)`
|
||||
|
||||
### 1.3. Существующие PV и PVC
|
||||
```bash
|
||||
kubectl get pv -o wide
|
||||
kubectl get pvc --all-namespaces
|
||||
```
|
||||
|
||||
Посмотреть, какие PVC привязаны к каким PV, и какой reclaimPolicy у PV.
|
||||
|
||||
### 1.4. Нода и диски
|
||||
```bash
|
||||
# Список нод
|
||||
kubectl get nodes -o wide
|
||||
|
||||
# На каждой ноде (через ssh или локально):
|
||||
lsblk
|
||||
df -h
|
||||
cat /etc/fstab
|
||||
```
|
||||
|
||||
Определить:
|
||||
- Есть ли отдельный раздел/диск для данных
|
||||
- Куда смонтированы разделы
|
||||
- Сколько свободного места
|
||||
- Есть ли монтирование NTFS-разделов (как `/media/forust/Programs`)
|
||||
|
||||
### 1.5. Где local-path хранит данные (для k3s)
|
||||
```bash
|
||||
ls -la /var/lib/rancher/k3s/storage/ 2>/dev/null
|
||||
# или для microk8s
|
||||
ls -la /var/snap/microk8s/common/ 2>/dev/null
|
||||
```
|
||||
|
||||
## 2. Анализ: сохраняются ли данные при удалении namespace?
|
||||
|
||||
| Сценарий | Результат |
|
||||
|---|---|
|
||||
| `kubectl delete ns <ns>` | Все PVC в namespace удаляются |
|
||||
| PVC → PV c `reclaimPolicy: Delete` | PV и данные удалены |
|
||||
| PVC → PV c `reclaimPolicy: Retain` | PV остаётся (статус Released), данные целы |
|
||||
|
||||
**Вывод:** Если reclaimPolicy в StorageClass = `Delete`, то данные **пропадут**. Если `Retain` — сохранятся.
|
||||
|
||||
## 3. План внедрения NFS
|
||||
|
||||
### 3.1. Проверить, установлен ли NFS
|
||||
```bash
|
||||
which nfsstat exportfs mount.nfs
|
||||
systemctl status nfs-server 2>/dev/null || systemctl status nfs-kernel-server 2>/dev/null
|
||||
```
|
||||
|
||||
### 3.2. Выбрать директорию для NFS-экспорта
|
||||
|
||||
Варианты (выбрать подходящий):
|
||||
- `/var/lib/k8s-nfs/` — на корневом разделе
|
||||
- `<путь к отдельному разделу>/k8s-nfs/` — если есть отдельный диск/раздел
|
||||
- Не рекомендуется использовать NTFS-раздел (проблемы с правами и производительностью)
|
||||
|
||||
Требования:
|
||||
- Файловая система: ext4 или xfs (не ntfs!)
|
||||
- Достаточно свободного места
|
||||
- Права: `755`, владелец root
|
||||
|
||||
### 3.3. Установить NFS-сервер
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
apt update && apt install -y nfs-kernel-server
|
||||
|
||||
# RHEL/Fedora
|
||||
dnf install -y nfs-utils
|
||||
```
|
||||
|
||||
### 3.4. Настроить экспорт
|
||||
|
||||
Создать директорию:
|
||||
```bash
|
||||
mkdir -p /var/lib/k8s-nfs
|
||||
chmod 755 /var/lib/k8s-nfs
|
||||
```
|
||||
|
||||
Добавить в `/etc/exports`:
|
||||
```
|
||||
/var/lib/k8s-nfs *(rw,sync,no_subtree_check,no_root_squash)
|
||||
```
|
||||
|
||||
Применить:
|
||||
```bash
|
||||
exportfs -rav
|
||||
```
|
||||
|
||||
Проверить:
|
||||
```bash
|
||||
showmount -e localhost
|
||||
```
|
||||
|
||||
### 3.5. Выбрать способ интеграции с Kubernetes
|
||||
|
||||
#### Вариант A: nfs-subdir-external-provisioner (проще)
|
||||
```bash
|
||||
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
|
||||
helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
|
||||
--namespace kube-system \
|
||||
--set nfs.server=127.0.0.1 \
|
||||
--set nfs.path=/var/lib/k8s-nfs \
|
||||
--set storageClass.name=nfs \
|
||||
--set storageClass.defaultClass=false \
|
||||
--set storageClass.reclaimPolicy=Retain
|
||||
```
|
||||
|
||||
#### Вариант B: NFS CSI Driver
|
||||
```bash
|
||||
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
|
||||
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system
|
||||
```
|
||||
|
||||
После установки CSI драйвера создать StorageClass:
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: nfs
|
||||
provisioner: nfs.csi.k8s.io
|
||||
parameters:
|
||||
server: 127.0.0.1
|
||||
share: /var/lib/k8s-nfs
|
||||
reclaimPolicy: Retain
|
||||
volumeBindingMode: Immediate
|
||||
```
|
||||
|
||||
### 3.6. Проверить результат
|
||||
```bash
|
||||
kubectl get storageclass
|
||||
kubectl get pods -n kube-system | grep -E 'nfs|provisioner'
|
||||
```
|
||||
|
||||
## 4. Итоговая конфигурация
|
||||
|
||||
После внедрения в кластере будет два StorageClass:
|
||||
|
||||
| Имя | Provisioner | ReclaimPolicy | Назначение |
|
||||
|---|---|---|---|
|
||||
| `local-path` (default) | rancher.io/local-path | Delete | Временные данные, stateless |
|
||||
| `nfs` | nfs-subdir-external-provisioner или nfs.csi.k8s.io | Retain | Данные, которые нужно сохранять |
|
||||
|
||||
**Главное преимущество:** PVC c `storageClassName: nfs` при удалении namespace сохраняют данные на диске, так как NFS-провизор использует `reclaimPolicy: Retain` или файлы физически остаются в NFS-экспорте.
|
||||
|
||||
## 5. Ответы на частые вопросы
|
||||
|
||||
**В:** Не упадёт ли local-path при установке NFS?
|
||||
**О:** Нет, они независимы. local-path продолжает работать как обычно.
|
||||
|
||||
**В:** Данные NFS и local-path будут на одном диске?
|
||||
**О:** Да, можно настроить оба на одном разделе, в разных каталогах.
|
||||
|
||||
**В:** Что если у меня несколько нод?
|
||||
**О:** NFS сервер нужно поднять на одной ноде, а с других нод должна быть доступна шари. Для multi-node лучше использовать отдельный сервер или distributed storage (Longhorn, Rook/Ceph).
|
||||
|
||||
**В:** Можно ли использовать существующий NTFS-раздел для NFS?
|
||||
**О:** Не рекомендуется — NTFS не поддерживает права Linux (no_root_squash не сработает корректно), возможны проблемы с блокировками и производительностью.
|
||||
@@ -1,24 +0,0 @@
|
||||
root = true
|
||||
|
||||
[*]
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
end_of_line = lf
|
||||
charset = utf-8
|
||||
trim_trailing_whitespace = true
|
||||
insert_final_newline = true
|
||||
|
||||
[*.{yml,yaml}]
|
||||
indent_size = 2
|
||||
|
||||
[*.{json,jsonc}]
|
||||
indent_size = 2
|
||||
|
||||
[*.md]
|
||||
trim_trailing_whitespace = false
|
||||
|
||||
[*.py]
|
||||
indent_size = 4
|
||||
|
||||
[{Makefile,makefile}]
|
||||
indent_style = tab
|
||||
@@ -1,10 +0,0 @@
|
||||
# actionlint configuration. Passed explicitly from the ci workflow:
|
||||
# actionlint -config-file .gitea/actionlint.yaml .gitea/workflows/*.yaml
|
||||
#
|
||||
# The self-hosted act_runner registers custom labels that actionlint cannot know
|
||||
# about, so declare them here instead of silencing the whole runner-label check.
|
||||
self-hosted-runner:
|
||||
labels:
|
||||
- arch
|
||||
- homelab
|
||||
- prod
|
||||
@@ -1,698 +0,0 @@
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "**"
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
# Every job here is checkout plus local tools. The token needs to read the tree
|
||||
# and nothing else, and saying so keeps a future step that reaches for the API
|
||||
# from quietly holding a token that can write to the repository.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
||||
|
||||
env:
|
||||
REGISTRY: gcr.forust.xyz
|
||||
|
||||
jobs:
|
||||
lint-compose:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
# Structure check for every committed Compose file, active or not.
|
||||
# Interpolation, env-file and bind-mount resolution are all switched off,
|
||||
# because inactive stacks have no .env here and would only fail on their
|
||||
# ${VAR:?} guards. Active stacks get the full check with interpolation in
|
||||
# the deploy workflow, where the real .env files live.
|
||||
- name: Validate Compose files
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
source .gitea/workflows/compose-lint.sh
|
||||
|
||||
mapfile -t safe_flags < <(compose_safe_flags)
|
||||
echo "docker compose config ${safe_flags[*]-}"
|
||||
|
||||
mapfile -t files < <(compose_files)
|
||||
if [ "${#files[@]}" -eq 0 ]; then
|
||||
echo "No Compose files found."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
failed=0
|
||||
for f in "${files[@]}"; do
|
||||
if ! out="$(validate_compose_file "$f" ${safe_flags[@]+"${safe_flags[@]}"} 2>&1)"; then
|
||||
failed=1
|
||||
echo "::error file=${f}::$(printf '%s' "$out" | head -1)"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$failed" -ne 0 ]; then
|
||||
echo "Compose validation failed."
|
||||
exit 1
|
||||
fi
|
||||
echo "checked ${#files[@]} Compose file(s)"
|
||||
|
||||
lint-actionlint:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Lint Gitea Actions workflows with actionlint
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
|
||||
|
||||
lint-shellcheck:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Lint shell scripts with ShellCheck
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
# userbot/ is a git subtree synced from forust/userbot, so its shell
|
||||
# scripts are upstream's to maintain, not ours. Linting them would let a
|
||||
# routine subtree pull turn the deploy gate red on code we do not own.
|
||||
mapfile -t scripts < <(
|
||||
git ls-files '*.sh' ':(glob)**/*.bash' ':!userbot/**'
|
||||
)
|
||||
if [ "${#scripts[@]}" -eq 0 ]; then
|
||||
echo "No shell scripts found."
|
||||
exit 0
|
||||
fi
|
||||
shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
|
||||
|
||||
lint-prettier:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Check formatting with Prettier
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
|
||||
mapfile -t prettier_files < <(
|
||||
git ls-files \
|
||||
| grep -E '\.(md|json|ya?ml|html|css)$' \
|
||||
| grep -Ev '^(\.docs/|\.zed/|errorpages/html/|homepages/(forust_files|xdfnx_files)/)'
|
||||
)
|
||||
|
||||
if [ "${#prettier_files[@]}" -eq 0 ]; then
|
||||
echo "No Prettier-managed files found."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
prettier --check --ignore-unknown "${prettier_files[@]}"
|
||||
|
||||
lint-ruff:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Lint and format-check Python with Ruff
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
ruff check .
|
||||
ruff format --check .
|
||||
|
||||
lint-yaml:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Lint YAML syntax
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
|
||||
mapfile -t yaml_files < <(
|
||||
git ls-files '*.yaml' '*.yml' \
|
||||
':!node_modules/**' \
|
||||
':!**/.venv/**'
|
||||
)
|
||||
|
||||
if [ "${#yaml_files[@]}" -eq 0 ]; then
|
||||
echo "No YAML files found."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
yamllint -c .yamllint "${yaml_files[@]}"
|
||||
|
||||
lint-dockerfiles:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Lint Dockerfiles
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
|
||||
mapfile -t dockerfiles < <(
|
||||
git ls-files ':(glob)**/Dockerfile' ':(glob)**/Dockerfile.*'
|
||||
)
|
||||
|
||||
if [ "${#dockerfiles[@]}" -eq 0 ]; then
|
||||
echo "No Dockerfiles found."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
hadolint -c .hadolint.yaml "${dockerfiles[@]}"
|
||||
|
||||
# Known, accepted, and recorded. Each line is a real advisory against a
|
||||
# package we build into the panel image, kept in this workflow rather than in
|
||||
# the package manifest so that a subtree sync from forust/userbot cannot
|
||||
# silently widen the exemption.
|
||||
#
|
||||
# starlette is the reason this job is not simply "fail on everything":
|
||||
# fastapi 0.115.12 pins `starlette<0.47.0`, and the fixes for the last four
|
||||
# below need 0.49.1 through 1.3.1, so clearing them means a jump from fastapi
|
||||
# 0.115.12 to 0.141.x. That is upstream's call, not a drive-by in a lint
|
||||
# commit. Of the seven, four are reachable here in principle: 1942 is a
|
||||
# crafted Range header hitting FileResponse, and the panel serves its built
|
||||
# SPA through exactly that; 249 is request.form() ignoring max_fields for
|
||||
# x-www-form-urlencoded, which is the login form; 1941 is a large multipart
|
||||
# body blocking the event loop; 161 and 248 are unvalidated Host and request
|
||||
# path reaching request.url. 2280 needs HTTPEndpoint, which the panel does
|
||||
# not use, and 2281 is Windows-only, and this deploys on Linux.
|
||||
#
|
||||
# The panel answers on userbot.workstation.internal and has no public
|
||||
# forust.xyz route, which is what keeps the four reachable ones from being
|
||||
# an internet-facing DoS. It still manages Telegram credentials.
|
||||
#
|
||||
# Deleting an entry here is how you accept a new advisory, so the diff says
|
||||
# so out loud.
|
||||
scan-deps:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Audit the Python dependencies that ship in the image
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh pip-audit)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
# requirements.txt, not requirements-dev.txt: this is what the image
|
||||
# installs, and the test tooling is not a shipped attack surface.
|
||||
pip-audit -r userbot/panel/backend/requirements.txt --strict \
|
||||
--ignore-vuln CVE-2025-67720 \
|
||||
--ignore-vuln PYSEC-2026-161 \
|
||||
--ignore-vuln PYSEC-2026-1941 \
|
||||
--ignore-vuln PYSEC-2026-1942 \
|
||||
--ignore-vuln PYSEC-2026-2280 \
|
||||
--ignore-vuln PYSEC-2026-2281 \
|
||||
--ignore-vuln PYSEC-2026-248 \
|
||||
--ignore-vuln PYSEC-2026-249
|
||||
|
||||
# devDependencies are excluded on purpose. `npm audit` on the full tree
|
||||
# reports 7 findings, and every one of them is a build- or test-time
|
||||
# package: the esbuild CORS advisory needs a vite dev server serving to
|
||||
# the internet, and nanoid's infinite loop needs a custom generator
|
||||
# called with size 0, which postcss does not do. None of them are in the
|
||||
# 91 kB bundle the panel serves. The one production finding, devalue
|
||||
# via svelte, is moderate, which is where --audit-level draws the line;
|
||||
# this fails on the next high or critical one.
|
||||
- name: Audit the production npm dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The pinned node, not whatever the runner has. Its system node is a
|
||||
# rolling Arch package: during this very push its npm was missing
|
||||
# entirely, and an hour later it was npm 12 on node 26. Both are the
|
||||
# wrong major anyway — the panel image is node:22-alpine.
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh node)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
cd userbot/panel/frontend
|
||||
npm ci
|
||||
npm audit --omit=dev --audit-level=high
|
||||
|
||||
test-backend:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
# 25 tests over the panel's pydantic models, its auth flow, the SPA
|
||||
# fallback and the Kubernetes client it shells out with. They existed and
|
||||
# had never been executed by anything.
|
||||
#
|
||||
# Note that userbot/ is a subtree synced from forust/userbot, so a routine
|
||||
# sync can turn this red on upstream's code. Unlike the shellcheck job,
|
||||
# which skips that tree because style disagreements there are ours to
|
||||
# lose, a failing test here is a real defect in a service we deploy.
|
||||
- name: Run the panel backend test suite
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh uv)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
|
||||
# A venv in a temp dir rather than a checked-out one: the runner is
|
||||
# shared, and a leftover .venv would let a dependency the
|
||||
# requirements no longer pin still satisfy an import.
|
||||
#
|
||||
# --python is not optional. uv otherwise takes whatever interpreter it
|
||||
# finds first, and which one that is depends on the machine: this
|
||||
# runner runs jobs on the host, where the only interpreter is 3.14,
|
||||
# and pyrogram's sync.py calls the bare asyncio.get_event_loop() that
|
||||
# 3.14 no longer auto-creates, so three tests fail at collection. The
|
||||
# image is python:3.13-slim, so 3.13 is also the version worth
|
||||
# testing: uv fetches a managed build of it when the host has none,
|
||||
# which is what makes this job independent of the runner.
|
||||
venv="$(mktemp -d)/venv"
|
||||
uv venv --python 3.13 --quiet "$venv"
|
||||
uv pip install --quiet --python "$venv/bin/python" \
|
||||
-r userbot/panel/backend/requirements-dev.txt
|
||||
|
||||
# `python -m`, not bare `pytest`: the tests import `app.*` relative to
|
||||
# the backend directory, which only works if the cwd is on sys.path,
|
||||
# and only `python -m` puts it there.
|
||||
cd userbot/panel/backend
|
||||
"$venv/bin/python" -m pytest tests/ -q
|
||||
|
||||
test-frontend:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
# One `npm ci` for both checks below: it is by far the slowest part of
|
||||
# this job, and a second one would learn nothing the first did not.
|
||||
#
|
||||
# `npm ci`, not `npm install`, for the same reason the Dockerfile uses it:
|
||||
# the lockfile is what makes the tree that gets checked the tree that
|
||||
# gets shipped.
|
||||
- name: Type-check and test the panel frontend
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The pinned node, not whatever the runner has. Its system node is a
|
||||
# rolling Arch package: during this very push its npm was missing
|
||||
# entirely, and an hour later it was npm 12 on node 26. Both are the
|
||||
# wrong major anyway — the panel image is node:22-alpine.
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh node)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
cd userbot/panel/frontend
|
||||
npm ci
|
||||
|
||||
# svelte-check has been a devDependency all along with no script
|
||||
# pointing at it, so the type errors it reports had nowhere to
|
||||
# surface. It is clean today, which is the only reason it can be a
|
||||
# gate: it stops at whatever upstream introduces rather than
|
||||
# reporting a backlog we inherited.
|
||||
npm run check
|
||||
npm test
|
||||
|
||||
validate:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Validate Kubernetes manifests against JSON schemas
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
|
||||
mapfile -t manifests < <(
|
||||
git ls-files ':(glob)**/k8s/**/*.yaml' ':(glob)**/k8s/**/*.yml' \
|
||||
| grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
|
||||
)
|
||||
|
||||
if [ "${#manifests[@]}" -eq 0 ]; then
|
||||
echo "No Kubernetes manifests found."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
kubeconform \
|
||||
-strict \
|
||||
-ignore-missing-schemas \
|
||||
-summary \
|
||||
"${manifests[@]}"
|
||||
|
||||
# kubeconform has no schemas for CRDs, so every IngressRoute, Certificate,
|
||||
# PrometheusRule, Middleware, ServersTransport and ServiceMonitor is silently
|
||||
# skipped above. The live API server knows the real CRD schemas (and runs the
|
||||
# cert-manager / Traefik admission webhooks), so validate there too.
|
||||
#
|
||||
# Only services marked with a k8s/active marker are checked: server-side
|
||||
# dry-run needs the target namespace to exist, and inactive services are not
|
||||
# deployed. Services being enabled for the first time are still covered by
|
||||
# the JSON-schema pass above.
|
||||
#
|
||||
# Main pushes only. `--dry-run=server` persists nothing, but it does execute
|
||||
# the admission webhooks of the production API server, so anyone able to open
|
||||
# a pull request would be able to run arbitrary manifest content through
|
||||
# cert-manager and Traefik. A pull request has nothing to gain from it either:
|
||||
# only main is ever deployed, and this job runs to completion before the
|
||||
# deploy workflow is allowed to start, so a bad CRD is still caught before
|
||||
# anything reaches the cluster -- just on the push rather than on the PR.
|
||||
- name: Note the server-side check is not running here
|
||||
if: github.event_name == 'pull_request' || github.ref != 'refs/heads/main'
|
||||
shell: bash
|
||||
run: |
|
||||
echo "::notice::Skipping the server-side dry-run. It executes the cert-manager and" \
|
||||
"Traefik admission webhooks against the production API server, so it is limited" \
|
||||
"to pushes to main. CRDs are still schema-checked by kubeconform above, and the" \
|
||||
"server-side pass still runs on main before the deploy."
|
||||
|
||||
- name: Validate active manifests against the live API server
|
||||
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if ! kubectl get --raw='/readyz' --request-timeout=10s >/dev/null 2>&1; then
|
||||
echo "::warning::Cluster unreachable — skipped server-side validation of CRDs (IngressRoute, Certificate, PrometheusRule). Review manifest changes manually."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
mapfile -t k8s_dirs < <(
|
||||
git ls-files '*.yaml' '*.yml' \
|
||||
| grep -E '(^|/)k8s/' \
|
||||
| sed -E 's#((^|.*/)k8s)/.*#\1#' \
|
||||
| sort -u
|
||||
)
|
||||
|
||||
manifests=()
|
||||
kustomize_apps=()
|
||||
for dir in "${k8s_dirs[@]}"; do
|
||||
if [ ! -f "${dir}/active" ]; then
|
||||
echo "skip (no k8s/active): ${dir}"
|
||||
continue
|
||||
fi
|
||||
if [ -f "${dir}/overlays/prod/kustomization.yaml" ]; then
|
||||
kustomize_apps+=("${dir}/overlays/prod")
|
||||
elif [ -f "${dir}/base/kustomization.yaml" ]; then
|
||||
kustomize_apps+=("${dir}/base")
|
||||
else
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] && manifests+=("$f")
|
||||
done < <(
|
||||
git ls-files "${dir}/*.yaml" "${dir}/*.yml" \
|
||||
| grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
|
||||
)
|
||||
fi
|
||||
done
|
||||
|
||||
echo "server-side dry-run: ${#manifests[@]} manifests, ${#kustomize_apps[@]} kustomize apps"
|
||||
failed=0
|
||||
for m in ${manifests[@]+"${manifests[@]}"}; do
|
||||
if ! out="$(kubectl apply --dry-run=server -f "$m" 2>&1)"; then
|
||||
failed=1
|
||||
echo "::error file=${m}::$(printf '%s' "$out" | head -1)"
|
||||
fi
|
||||
done
|
||||
for k in ${kustomize_apps[@]+"${kustomize_apps[@]}"}; do
|
||||
if ! out="$(kubectl apply -k "$k" --dry-run=server 2>&1)"; then
|
||||
failed=1
|
||||
echo "::error file=${k}::$(printf '%s' "$out" | head -1)"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$failed" -ne 0 ]; then
|
||||
echo "Server-side validation failed. The API server (or an admission webhook) rejected these manifests."
|
||||
exit 1
|
||||
fi
|
||||
echo "server-side dry-run: all active manifests accepted by the API server"
|
||||
|
||||
build:
|
||||
needs:
|
||||
[lint-actionlint, lint-shellcheck, lint-compose, lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
|
||||
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 60
|
||||
outputs:
|
||||
services: ${{ steps.services.outputs.services }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Detect changed docker-built services
|
||||
id: services
|
||||
shell: bash
|
||||
run: |
|
||||
base="${{ github.event.before }}"
|
||||
if [ -z "$base" ] || [ "$base" = "0000000000000000000000000000000000000000" ]; then
|
||||
base="$(git rev-list --max-parents=0 HEAD)"
|
||||
fi
|
||||
|
||||
mapfile -t changed_files < <(git diff --name-only "$base" "${GITHUB_SHA}")
|
||||
|
||||
services=()
|
||||
|
||||
add_service() {
|
||||
local name="$1"
|
||||
local seen=0
|
||||
for existing in "${services[@]}"; do
|
||||
if [ "$existing" = "$name" ]; then
|
||||
seen=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
if [ "$seen" -eq 0 ]; then
|
||||
services+=("$name")
|
||||
fi
|
||||
}
|
||||
|
||||
for file in "${changed_files[@]}"; do
|
||||
case "$file" in
|
||||
dtek_notif/*)
|
||||
add_service dtek_notif
|
||||
;;
|
||||
errorpages/*)
|
||||
add_service errorpages
|
||||
;;
|
||||
userbot/*)
|
||||
add_service userbot
|
||||
;;
|
||||
homepages/*)
|
||||
add_service homepages
|
||||
;;
|
||||
edu_master/phpsessid-bot/*|edu_master/webinar-checker/*|edu_master/compose.yaml)
|
||||
add_service edu_master
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "${#services[@]}" -eq 0 ]; then
|
||||
echo "No docker-built services changed."
|
||||
echo "services=" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf '%s\n' "${services[@]}" | tee /tmp/services.txt
|
||||
echo "services=$(paste -sd, /tmp/services.txt)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Log in to registry
|
||||
if: steps.services.outputs.services != ''
|
||||
shell: bash
|
||||
run: |
|
||||
echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login "${REGISTRY}" \
|
||||
-u "${{ secrets.REGISTRY_USERNAME }}" \
|
||||
--password-stdin
|
||||
|
||||
- name: Build and push changed images
|
||||
if: steps.services.outputs.services != ''
|
||||
shell: bash
|
||||
run: |
|
||||
IFS=, read -r -a services <<< "${{ steps.services.outputs.services }}"
|
||||
|
||||
for service in "${services[@]}"; do
|
||||
case "$service" in
|
||||
dtek_notif)
|
||||
image="${REGISTRY}/forust/dtek-notif"
|
||||
tags=()
|
||||
case "${GITHUB_REF_NAME}" in
|
||||
main)
|
||||
tags+=("main" "prod")
|
||||
;;
|
||||
dev)
|
||||
tags+=("dev")
|
||||
;;
|
||||
esac
|
||||
build_args=()
|
||||
for tag in "${tags[@]}"; do
|
||||
build_args+=(-t "${image}:${tag}")
|
||||
done
|
||||
docker build \
|
||||
--cache-from "type=registry,ref=${image}:buildcache" \
|
||||
--cache-to "type=registry,ref=${image}:buildcache,mode=max" \
|
||||
"${build_args[@]}" dtek_notif
|
||||
for tag in "${tags[@]}"; do
|
||||
docker push "${image}:${tag}"
|
||||
done
|
||||
;;
|
||||
errorpages)
|
||||
image="${REGISTRY}/forust/error-pages"
|
||||
tags=()
|
||||
case "${GITHUB_REF_NAME}" in
|
||||
main)
|
||||
tags+=("main" "prod")
|
||||
;;
|
||||
dev)
|
||||
tags+=("dev")
|
||||
;;
|
||||
esac
|
||||
build_args=()
|
||||
for tag in "${tags[@]}"; do
|
||||
build_args+=(-t "${image}:${tag}")
|
||||
done
|
||||
docker build \
|
||||
--cache-from "type=registry,ref=${image}:buildcache" \
|
||||
--cache-to "type=registry,ref=${image}:buildcache,mode=max" \
|
||||
"${build_args[@]}" errorpages
|
||||
for tag in "${tags[@]}"; do
|
||||
docker push "${image}:${tag}"
|
||||
done
|
||||
;;
|
||||
userbot)
|
||||
tags=()
|
||||
case "${GITHUB_REF_NAME}" in
|
||||
main)
|
||||
tags+=("main" "prod")
|
||||
;;
|
||||
dev)
|
||||
tags+=("dev")
|
||||
;;
|
||||
esac
|
||||
for target in runtime panel; do
|
||||
case "$target" in
|
||||
runtime)
|
||||
context="userbot"
|
||||
image="${REGISTRY}/forust/userbot"
|
||||
;;
|
||||
panel)
|
||||
context="userbot/panel"
|
||||
image="${REGISTRY}/forust/userbot-panel"
|
||||
;;
|
||||
esac
|
||||
build_args=()
|
||||
for tag in "${tags[@]}"; do
|
||||
build_args+=(-t "${image}:${tag}")
|
||||
done
|
||||
docker build \
|
||||
--cache-from "type=registry,ref=${image}:buildcache" \
|
||||
--cache-to "type=registry,ref=${image}:buildcache,mode=max" \
|
||||
"${build_args[@]}" "$context"
|
||||
for tag in "${tags[@]}"; do
|
||||
docker push "${image}:${tag}"
|
||||
done
|
||||
done
|
||||
;;
|
||||
homepages)
|
||||
for variant in forust xdfnx; do
|
||||
case "$variant" in
|
||||
forust)
|
||||
image="${REGISTRY}/forust/forust-homepage"
|
||||
;;
|
||||
xdfnx)
|
||||
image="${REGISTRY}/forust/xdfnx-homepage"
|
||||
;;
|
||||
esac
|
||||
tags=()
|
||||
case "${GITHUB_REF_NAME}" in
|
||||
main)
|
||||
tags+=("main" "prod")
|
||||
;;
|
||||
dev)
|
||||
tags+=("dev")
|
||||
;;
|
||||
esac
|
||||
build_args=()
|
||||
for tag in "${tags[@]}"; do
|
||||
build_args+=(-t "${image}:${tag}")
|
||||
done
|
||||
docker build \
|
||||
--cache-from "type=registry,ref=${image}:buildcache" \
|
||||
--cache-to "type=registry,ref=${image}:buildcache,mode=max" \
|
||||
"${build_args[@]}" -f "homepages/Dockerfile.${variant}" homepages
|
||||
for tag in "${tags[@]}"; do
|
||||
docker push "${image}:${tag}"
|
||||
done
|
||||
done
|
||||
;;
|
||||
edu_master)
|
||||
for variant in session-keeper webinar-checker; do
|
||||
case "$variant" in
|
||||
session-keeper)
|
||||
context="edu_master/phpsessid-bot"
|
||||
image="${REGISTRY}/forust/session-keeper"
|
||||
;;
|
||||
webinar-checker)
|
||||
context="edu_master/webinar-checker"
|
||||
image="${REGISTRY}/forust/webinar-checker"
|
||||
;;
|
||||
esac
|
||||
tags=()
|
||||
case "${GITHUB_REF_NAME}" in
|
||||
main)
|
||||
tags+=("main" "prod")
|
||||
;;
|
||||
dev)
|
||||
tags+=("dev")
|
||||
;;
|
||||
esac
|
||||
build_args=()
|
||||
for tag in "${tags[@]}"; do
|
||||
build_args+=(-t "${image}:${tag}")
|
||||
done
|
||||
docker build \
|
||||
--cache-from "type=registry,ref=${image}:buildcache" \
|
||||
--cache-to "type=registry,ref=${image}:buildcache,mode=max" \
|
||||
"${build_args[@]}" "$context"
|
||||
for tag in "${tags[@]}"; do
|
||||
docker push "${image}:${tag}"
|
||||
done
|
||||
done
|
||||
;;
|
||||
esac
|
||||
done
|
||||
@@ -1,46 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for validating Compose files. Sourced both by steps in
|
||||
# .gitea/workflows/ci.yaml and by deploy-lib.sh on the workstation.
|
||||
#
|
||||
# Two levels of checking, matching how the repo is structured:
|
||||
#
|
||||
# general every committed Compose file, active or not. Pure structure check:
|
||||
# no ${VAR} interpolation, no .env lookup, no bind-mount path
|
||||
# resolution. Disabled stacks deliberately have no .env in the repo
|
||||
# and no values on the CI runner, so a full `config` run would fail on
|
||||
# their `${VAR:?}` guards for reasons that have nothing to do with the
|
||||
# change under review.
|
||||
#
|
||||
# full active stacks only, with interpolation and env-file resolution, so
|
||||
# required variables and referenced files are actually resolved. Needs
|
||||
# the gitignored .env files, so this only runs in the deploy workflow
|
||||
# on the workstation.
|
||||
#
|
||||
# This file is meant to be sourced, not executed.
|
||||
|
||||
# All committed Compose files, including the ones deploy never starts.
|
||||
compose_files() {
|
||||
git ls-files \
|
||||
'*/compose.yaml' '*/compose.yml' 'compose.yaml' 'compose.yml' \
|
||||
'*/docker-compose.yaml' '*/docker-compose.yml'
|
||||
}
|
||||
|
||||
# Prints the flags that turn `docker compose config` into the general check.
|
||||
# Probed rather than hardcoded so an older Compose without --no-env-resolution
|
||||
# still gets the flags it does support.
|
||||
compose_safe_flags() {
|
||||
local help flag
|
||||
help="$(docker compose config --help 2>/dev/null || true)"
|
||||
for flag in --no-interpolate --no-env-resolution --no-path-resolution; do
|
||||
if printf '%s' "$help" | grep -q -- "$flag"; then
|
||||
printf '%s\n' "$flag"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# validate_compose_file <file> [extra docker compose config flags...]
|
||||
validate_compose_file() {
|
||||
local file="$1"
|
||||
shift
|
||||
docker compose -f "$file" config --quiet "$@"
|
||||
}
|
||||
@@ -1,937 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared stages for the deploy workflow. Runs on the workstation, invoked as:
|
||||
# REPO=/srv/homelab APPLY_PRUNE=false bash -se <<'EOF'
|
||||
# source "$REPO/.gitea/workflows/deploy-lib.sh"
|
||||
# run_stage "$STAGE"
|
||||
# EOF
|
||||
set -euo pipefail
|
||||
|
||||
: "${REPO:?REPO must be set}"
|
||||
APPLY_PRUNE="${APPLY_PRUNE:-false}"
|
||||
# Commit CI validated. Empty for a manual workflow_dispatch, which falls back to
|
||||
# the current origin/main.
|
||||
DEPLOY_SHA="${DEPLOY_SHA:-}"
|
||||
# Handoff point between the apply stage (writes) and the verify stage (reads).
|
||||
# Under the deploy user's own XDG state directory rather than /var/backups: the
|
||||
# deploy is unprivileged, /var/backups does not exist on a minimal Arch host, and
|
||||
# creating it would need root — which is why the first real deploy died here with
|
||||
# "is not writable" before touching a single workload. $HOME comes from sshd.
|
||||
DEPLOY_SNAPSHOT_DIR="${DEPLOY_SNAPSHOT_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/homelab-deploy}"
|
||||
# Per-workload rollout budget and how many workloads to watch at once. The whole
|
||||
# apply job has its own timeout-minutes as a backstop.
|
||||
ROLLOUT_TIMEOUT="${ROLLOUT_TIMEOUT:-300}"
|
||||
ROLLOUT_PARALLELISM="${ROLLOUT_PARALLELISM:-8}"
|
||||
WORKLOAD_KINDS="deployments.apps,statefulsets.apps,daemonsets.apps"
|
||||
|
||||
log() {
|
||||
echo "== $* =="
|
||||
}
|
||||
|
||||
warn() {
|
||||
echo "WARNING: $*" >&2
|
||||
}
|
||||
|
||||
collect_k8s() {
|
||||
git -C "$REPO" ls-files -- "$1" \
|
||||
| grep -E '\.ya?ml$' \
|
||||
| grep -Ev '/overlays/' \
|
||||
| grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$' \
|
||||
| grep -Ev '(^|/)[^/]*secret[^/]*\.ya?ml$' \
|
||||
| sort
|
||||
}
|
||||
|
||||
kustomize_overlay() {
|
||||
if [ -f "$1/overlays/prod/kustomization.yaml" ]; then
|
||||
echo "$1/overlays/prod"
|
||||
elif [ -f "$1/base/kustomization.yaml" ]; then
|
||||
echo "$1/base"
|
||||
elif [ -f "$1/kustomization.yaml" ]; then
|
||||
echo "$1"
|
||||
fi
|
||||
}
|
||||
|
||||
select_manifests() {
|
||||
K8S_MANIFESTS=()
|
||||
KUSTOMIZE_APPS=()
|
||||
COMPOSE_STACKS=()
|
||||
local kd_rel kd overlay cf_rel cf f
|
||||
while IFS= read -r kd_rel; do
|
||||
kd="$REPO/$kd_rel"
|
||||
if [ ! -f "$kd/active" ]; then
|
||||
echo "skip (no k8s/active): $kd_rel"
|
||||
continue
|
||||
fi
|
||||
overlay="$(kustomize_overlay "$kd" || true)"
|
||||
if [ -n "${overlay:-}" ]; then
|
||||
echo "kustomize app: ${overlay#"$REPO"/}"
|
||||
KUSTOMIZE_APPS+=("$overlay")
|
||||
else
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] && K8S_MANIFESTS+=("$REPO/$f")
|
||||
done < <(collect_k8s "$kd_rel" || true)
|
||||
fi
|
||||
done < <(
|
||||
git -C "$REPO" ls-files '*.yaml' '*.yml' \
|
||||
| grep -E '(^|/)k8s/' \
|
||||
| sed -E 's#((^|.*/)k8s)/.*#\1#' \
|
||||
| sort -u
|
||||
)
|
||||
while IFS= read -r cf_rel; do
|
||||
cf="$REPO/$cf_rel"
|
||||
if [ -f "$(dirname "$cf")/active" ]; then
|
||||
echo "compose: $cf_rel"
|
||||
COMPOSE_STACKS+=("$cf")
|
||||
else
|
||||
echo "skip (no root active): $cf_rel"
|
||||
fi
|
||||
done < <(git -C "$REPO" ls-files '*/compose.yaml' '*/compose.yml' compose.yaml compose.yml | sort)
|
||||
}
|
||||
|
||||
# --- post-apply verification and rollback -------------------------------------
|
||||
#
|
||||
# A green `kubectl apply` says nothing about the cluster being healthy. These
|
||||
# helpers watch exactly the workloads whose spec changed during this apply, and
|
||||
# on failure roll them back to the revision that was running before, so a bad
|
||||
# push to main cannot leave a service crash-looping.
|
||||
#
|
||||
# Verification lives in its own workflow job, not at the end of the apply stage.
|
||||
# Inside a single process it is worthless exactly when it is needed most: a job
|
||||
# killed by timeout-minutes or cancelled mid-apply never reaches the rollback
|
||||
# code, and leaves a half-applied cluster behind. Split out, the apply job can
|
||||
# die in any way and the verify job still runs.
|
||||
#
|
||||
# That split needs a handoff point on the workstation, because the two stages are
|
||||
# separate processes on separate runner jobs: DEPLOY_SNAPSHOT_DIR/current, written
|
||||
# before anything is applied, read by the verify stage afterwards.
|
||||
|
||||
# Creates this run's snapshot directory and publishes it as the handoff point for
|
||||
# the verify stage. Fails hard by design: a deploy that cannot record what it is
|
||||
# about to change must not start, because then nothing can be rolled back for it
|
||||
# automatically. Publishing happens before the first apply, so an apply killed
|
||||
# mid-flight still leaves a usable baseline behind.
|
||||
snapshot_dir() {
|
||||
local stamp dir
|
||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)-${DEPLOY_SHA:-$(git -C "$REPO" rev-parse --short HEAD 2>/dev/null || echo unknown)}"
|
||||
dir="$DEPLOY_SNAPSHOT_DIR/$stamp"
|
||||
|
||||
if ! mkdir -p "$DEPLOY_SNAPSHOT_DIR" 2>/dev/null || [ ! -w "$DEPLOY_SNAPSHOT_DIR" ]; then
|
||||
echo "ERROR: $DEPLOY_SNAPSHOT_DIR is not writable." >&2
|
||||
echo "The verify job needs it to learn which workloads this deploy touches." >&2
|
||||
echo "Refusing to deploy without a way to roll back." >&2
|
||||
return 1
|
||||
fi
|
||||
if ! mkdir -p "$dir" 2>/dev/null || [ ! -w "$dir" ]; then
|
||||
echo "ERROR: cannot create snapshot dir $dir" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! printf '%s\n' "$dir" >"$DEPLOY_SNAPSHOT_DIR/current" 2>/dev/null; then
|
||||
echo "ERROR: cannot publish the snapshot pointer at $DEPLOY_SNAPSHOT_DIR/current" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
printf '%s\n' "$dir"
|
||||
}
|
||||
|
||||
save_snapshot() {
|
||||
local dir="$1"
|
||||
log "Saving pre-apply snapshot to $dir"
|
||||
workload_generations >"$dir/generations.before" 2>/dev/null \
|
||||
|| warn "could not snapshot workload generations"
|
||||
kubectl get "$WORKLOAD_KINDS" -A -o yaml >"$dir/workloads.yaml" 2>/dev/null \
|
||||
|| warn "could not snapshot workloads"
|
||||
for release in prometheus-stack loki alloy; do
|
||||
if helm status "$release" -n prometheus >/dev/null 2>&1; then
|
||||
{
|
||||
echo "revision: $(helm history "$release" -n prometheus -o json 2>/dev/null)"
|
||||
helm get values "$release" -n prometheus --all 2>/dev/null
|
||||
} >"$dir/helm-$release.txt"
|
||||
fi
|
||||
done
|
||||
# The verify stage compares this against the commit it is deploying, to refuse
|
||||
# rolling back against a baseline left by an earlier run. A snapshot we cannot
|
||||
# attribute to a commit is unusable for that, so fail before anything is applied.
|
||||
if ! git -C "$REPO" rev-parse HEAD >"$dir/commit" 2>/dev/null; then
|
||||
echo "ERROR: cannot record the deploy commit in $dir/commit" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Prints "<ns> <name> <kind> <generation>" for every workload in the cluster.
|
||||
workload_generations() {
|
||||
kubectl get "$WORKLOAD_KINDS" -A \
|
||||
-o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,KIND:.kind,GEN:.metadata.generation' \
|
||||
--no-headers 2>/dev/null \
|
||||
| awk 'NF >= 4 { printf "%s %s %s %s\n", $1, $2, tolower($3), $4 }'
|
||||
}
|
||||
|
||||
# Prints "<kind> <ns> <name>" for every workload that is new or whose generation
|
||||
# moved since the snapshot, i.e. the ones this apply actually touched.
|
||||
changed_workloads() {
|
||||
local before="$1"
|
||||
local ns name kind gen old
|
||||
while read -r ns name kind gen; do
|
||||
[ -n "${gen:-}" ] || continue
|
||||
old="$(awk -v want_ns="$ns" -v want_name="$name" \
|
||||
'$1 == want_ns && $2 == want_name { print $4; exit }' "$before" 2>/dev/null || true)"
|
||||
if [ "$old" != "$gen" ]; then
|
||||
printf '%s %s %s\n' "$kind" "$ns" "$name"
|
||||
fi
|
||||
done < <(workload_generations)
|
||||
}
|
||||
|
||||
# Prints "<ns> <kind>/<name> <image>" for every workload this repository owns that
|
||||
# runs an image from our own registry.
|
||||
#
|
||||
# The repository is the scope, deliberately. The cluster also holds workloads on
|
||||
# our registry that no manifest here declares (they are applied out of band), and
|
||||
# those are somebody else's to deploy. Walking the manifests rather than the
|
||||
# cluster means those can never be restarted by this pipeline, now or later.
|
||||
owned_registry_workloads() {
|
||||
local kd_rel f
|
||||
while IFS= read -r kd_rel; do
|
||||
[ -f "$REPO/$kd_rel/active" ] || continue
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
# A file that does not mention the registry cannot declare a workload on it,
|
||||
# and parsing costs ~2.5s per file against a millisecond for the grep. The
|
||||
# filter keeps this at a handful of parses instead of one per manifest.
|
||||
grep -q 'gcr\.forust\.xyz/forust/' "$REPO/$f" 2>/dev/null || continue
|
||||
# kubectl prints a bare object for a single-document file and a List for a
|
||||
# multi-document one, so normalise both shapes before filtering.
|
||||
kubectl apply --dry-run=client -f "$REPO/$f" -o json 2>/dev/null \
|
||||
| jq -r '
|
||||
(if .items then .items[] else . end)
|
||||
| select(.kind | test("^(Deployment|StatefulSet|DaemonSet)$"))
|
||||
| select(any((.spec.template.spec.containers // [])[]?;
|
||||
(.image // "") | test("^gcr\\.forust\\.xyz/forust/")))
|
||||
| (.metadata.namespace // "default") as $ns
|
||||
| ([.spec.template.spec.containers[].image
|
||||
| select(test("^gcr\\.forust\\.xyz/forust/"))][0]) as $img
|
||||
| "\($ns) \(.kind | ascii_downcase)/\(.metadata.name) \($img)"
|
||||
' 2>/dev/null || true
|
||||
done < <(collect_k8s "$kd_rel" || true)
|
||||
done < <(
|
||||
git -C "$REPO" ls-files '*.yaml' '*.yml' \
|
||||
| grep -E '(^|/)k8s/' \
|
||||
| sed -E 's#((^|.*/)k8s)/.*#\1#' \
|
||||
| sort -u
|
||||
)
|
||||
}
|
||||
|
||||
# Prints the digest an image tag resolves to for this cluster's architecture, or
|
||||
# nothing when it cannot be resolved.
|
||||
#
|
||||
# Only the manifest entry matching the node architecture counts. A multi-arch tag
|
||||
# also carries `unknown/unknown` entries for the build attestation, and a pod's
|
||||
# imageID is always the per-platform digest, so comparing the wrong entry would
|
||||
# mark every workload stale forever and restart the whole cluster on every deploy.
|
||||
registry_digest() {
|
||||
local arch
|
||||
arch="$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}' 2>/dev/null || true)"
|
||||
[ -n "$arch" ] || arch=amd64
|
||||
# The || true is load-bearing. Every caller runs under set -euo pipefail, and
|
||||
# pipefail reports the rightmost non-zero stage, so a ref the registry does not
|
||||
# have would abort the caller at the assignment instead of yielding an empty
|
||||
# string. The callers check for empty themselves and report it by name.
|
||||
docker manifest inspect "$1" 2>/dev/null \
|
||||
| jq -r --arg arch "$arch" '
|
||||
.manifests[]?
|
||||
| select(.platform.os == "linux" and .platform.architecture == $arch)
|
||||
| .digest
|
||||
' 2>/dev/null \
|
||||
| head -1 || true
|
||||
}
|
||||
|
||||
# Rewrites our own images to immutable digests on the way into the cluster.
|
||||
# Reads a manifest stream on stdin, writes the pinned stream to stdout.
|
||||
#
|
||||
# A digest is not knowable when a manifest is written, so it is resolved here, at
|
||||
# apply time, and never committed: git keeps a readable `:prod` tag. That is what
|
||||
# makes rollback mean something. `kubectl rollout undo` restores the previous
|
||||
# ReplicaSet's pod template verbatim, and a template naming a digest restores the
|
||||
# exact bytes that were serving before. A template naming a moving tag does not —
|
||||
# the tag has already moved by the time the rollback runs, so the "rollback"
|
||||
# re-pulls the very image that just failed and the cluster stays broken.
|
||||
#
|
||||
# imagePullPolicy is deliberately left alone. The manifests no longer set it, and a
|
||||
# reference that is not `:latest` defaults to IfNotPresent, which is what the
|
||||
# Kubernetes docs ask for alongside a digest: the bytes under a digest cannot
|
||||
# change, so pulling again buys nothing.
|
||||
#
|
||||
# An image that cannot be resolved is fatal. Carrying on would quietly apply a
|
||||
# mutable tag again, which is the exact failure this function exists to remove.
|
||||
render_pinned() {
|
||||
local src refs map ref digest missing=0
|
||||
src="$(mktemp)"
|
||||
refs="$(mktemp)"
|
||||
map="$(mktemp)"
|
||||
|
||||
cat >"$src"
|
||||
grep -oE 'gcr\.forust\.xyz/forust/[A-Za-z0-9._-]+:[A-Za-z0-9._-]+' "$src" | sort -u >"$refs" || true
|
||||
|
||||
while read -r ref; do
|
||||
[ -n "$ref" ] || continue
|
||||
digest="$(registry_digest "$ref")"
|
||||
if [ -z "$digest" ]; then
|
||||
echo "ERROR: cannot resolve ${ref} in the registry; applying nothing." >&2
|
||||
echo " The build job has to push that tag before the deploy resolves it." >&2
|
||||
missing=$((missing + 1))
|
||||
continue
|
||||
fi
|
||||
printf '%s\t%s\n' "$ref" "$digest" >>"$map"
|
||||
done <"$refs"
|
||||
if [ "$missing" -gt 0 ]; then
|
||||
rm -f "$src" "$refs" "$map"
|
||||
return 1
|
||||
fi
|
||||
|
||||
awk -v mapfile="$map" '
|
||||
BEGIN {
|
||||
while ((getline line < mapfile) > 0) {
|
||||
i = index(line, "\t")
|
||||
d[substr(line, 1, i - 1)] = substr(line, i + 1)
|
||||
}
|
||||
}
|
||||
{
|
||||
if (match($0, /^[[:space:]]*image:[[:space:]]*gcr\.forust\.xyz\/forust\/[A-Za-z0-9._-]+:[A-Za-z0-9._-]+[[:space:]]*$/)) {
|
||||
name = $0
|
||||
sub(/^[[:space:]]*image:[[:space:]]*/, "", name)
|
||||
sub(/[[:space:]]*$/, "", name)
|
||||
if (name in d) {
|
||||
pad = $0
|
||||
sub(/image:.*/, "", pad)
|
||||
# Drop the tag: the canonical form used in the docs is repo@sha256:...,
|
||||
# and leaving :prod next to the digest reads like it still matters.
|
||||
repo = name
|
||||
sub(/:[A-Za-z0-9._-]+$/, "", repo)
|
||||
print pad "image: " repo "@" d[name]
|
||||
next
|
||||
}
|
||||
}
|
||||
print
|
||||
}
|
||||
' "$src"
|
||||
rm -f "$src" "$refs" "$map"
|
||||
}
|
||||
|
||||
# Restarts every owned workload whose running image is not the one its tag
|
||||
# resolves to now.
|
||||
#
|
||||
# This used to be how a rebuild reached the cluster at all: the manifests pinned
|
||||
# `:latest`, so a rebuild left the pod template byte-identical, `kubectl apply`
|
||||
# decided there was nothing to do, and the cluster served the previous build
|
||||
# indefinitely. The apply now pins digests via render_pinned, so a rebuild moves
|
||||
# the pod template and rolls out on its own.
|
||||
#
|
||||
# What is left is the drift check: a hand-run `kubectl set image`, or anything
|
||||
# else that edits a live workload behind the deploy's back, is the only way to end
|
||||
# up serving a digest the tag has moved past. It stays idempotent, so a redeploy
|
||||
# that changed no image still does not bounce healthy services.
|
||||
#
|
||||
# The container is matched on its repository rather than on the exact reference:
|
||||
# once render_pinned has run, a pod's status reports `repo@sha256:...` while this
|
||||
# still reads the repository's `:prod` tag out of the manifest.
|
||||
restart_stale_images() {
|
||||
local ns target image want selector running entry one
|
||||
local unchecked=0
|
||||
local -A digests=()
|
||||
local -a stale=()
|
||||
while read -r ns target image; do
|
||||
[ -n "${target:-}" ] || continue
|
||||
if [ -z "${digests[$image]:-}" ]; then
|
||||
digests[$image]="$(registry_digest "$image")"
|
||||
fi
|
||||
want="${digests[$image]}"
|
||||
if [ -z "$want" ]; then
|
||||
warn "cannot resolve ${image##*/} in the registry, leaving $target alone"
|
||||
unchecked=$((unchecked + 1))
|
||||
continue
|
||||
fi
|
||||
selector="$(kubectl get "$target" -n "$ns" -o jsonpath='{.spec.selector.matchLabels}' 2>/dev/null \
|
||||
| jq -r 'to_entries | map("\(.key)=\(.value)") | join(",")' 2>/dev/null)"
|
||||
if [ -z "$selector" ]; then
|
||||
warn "cannot read the pod selector of $target, skipping"
|
||||
unchecked=$((unchecked + 1))
|
||||
continue
|
||||
fi
|
||||
running="$(kubectl get pods -n "$ns" -l "$selector" -o json 2>/dev/null \
|
||||
| jq -r --arg repo "${image%%:*}" '
|
||||
.items[] | .status.containerStatuses[]?
|
||||
| select(.image == $repo
|
||||
or (.image | startswith($repo + ":"))
|
||||
or (.image | startswith($repo + "@")))
|
||||
| .imageID
|
||||
' 2>/dev/null)"
|
||||
if [ -z "$running" ]; then
|
||||
# Scaled to zero. Nothing is serving stale code, and imagePullPolicy
|
||||
# resolves the tag when it is scaled back up.
|
||||
continue
|
||||
fi
|
||||
entry=""
|
||||
while IFS= read -r one; do
|
||||
[ -n "$one" ] || continue
|
||||
entry="${one##*@}"
|
||||
if [ "$entry" != "$want" ]; then
|
||||
stale+=("$ns $target")
|
||||
break
|
||||
fi
|
||||
done <<<"$running"
|
||||
done < <(owned_registry_workloads)
|
||||
if [ "${#stale[@]}" -eq 0 ]; then
|
||||
if [ "$unchecked" -gt 0 ]; then
|
||||
# Say so plainly. Reporting "everything is current" after checking nothing
|
||||
# would tell the operator the deploy is fine when it may not be.
|
||||
warn "No workload needed a restart, but $unchecked could not be checked"
|
||||
else
|
||||
log "All owned workloads already run the image their tag points at"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
log "Restarting ${#stale[@]} workload(s) running an image their tag has moved past"
|
||||
for ref in "${stale[@]}"; do
|
||||
log " $ref"
|
||||
done
|
||||
local failed=()
|
||||
for ref in "${stale[@]}"; do
|
||||
ns="${ref%% *}"
|
||||
target="${ref#* }"
|
||||
if ! kubectl rollout restart "$target" -n "$ns" >/dev/null 2>&1; then
|
||||
failed+=("$ref")
|
||||
fi
|
||||
done
|
||||
if [ "${#failed[@]}" -gt 0 ]; then
|
||||
warn "could not restart: ${failed[*]}"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# verify_workloads <failed-file> <kind> <ns> <name> ...
|
||||
# Watches every workload in parallel and records the ones that never became
|
||||
# healthy. Returns non-zero if any of them failed.
|
||||
verify_workloads() {
|
||||
local failed_file="$1"
|
||||
shift
|
||||
[ "$#" -gt 0 ] || return 0
|
||||
: >"$failed_file"
|
||||
local running=0 pid kind ns name
|
||||
local -a pids=()
|
||||
for entry in "$@"; do
|
||||
read -r kind ns name <<<"$entry"
|
||||
(
|
||||
if kubectl rollout status "${kind}/${name}" -n "$ns" --timeout="${ROLLOUT_TIMEOUT}s" >/dev/null 2>&1; then
|
||||
echo " ok: ${kind}/${ns}/${name}"
|
||||
else
|
||||
echo " FAILED: ${kind}/${ns}/${name}"
|
||||
printf '%s %s %s\n' "$kind" "$ns" "$name" >>"$failed_file"
|
||||
fi
|
||||
) &
|
||||
pids+=($!)
|
||||
running=$((running + 1))
|
||||
if [ "$running" -ge "$ROLLOUT_PARALLELISM" ]; then
|
||||
wait -n 2>/dev/null || true
|
||||
running=$((running - 1))
|
||||
fi
|
||||
done
|
||||
for pid in ${pids[@]+"${pids[@]}"}; do
|
||||
wait "$pid" || true
|
||||
done
|
||||
# Non-zero when the file holds at least one failure, i.e. a workload never
|
||||
# became healthy. `[ -s ]` alone is the opposite test and silently disabled
|
||||
# every rollback this stage is meant to perform.
|
||||
[ ! -s "$failed_file" ]
|
||||
}
|
||||
|
||||
# rollback_workloads <failed-file>
|
||||
# Restores the previous revision of every failed workload and waits for it to
|
||||
# settle. Prints a report and returns non-zero if any workload is still unhealthy,
|
||||
# so the operator knows manual recovery is required.
|
||||
rollback_workloads() {
|
||||
local failed_file="$1"
|
||||
local kind ns name unrecovered=()
|
||||
local -a recovered=()
|
||||
while read -r kind ns name; do
|
||||
[ -n "${kind:-}" ] || continue
|
||||
if kubectl rollout undo "${kind}/${name}" -n "$ns" >/dev/null 2>&1 \
|
||||
&& kubectl rollout status "${kind}/${name}" -n "$ns" --timeout="${ROLLOUT_TIMEOUT}s" >/dev/null 2>&1; then
|
||||
echo " rolled back: ${kind}/${ns}/${name}"
|
||||
recovered+=("${kind}/${ns}/${name}")
|
||||
else
|
||||
echo " NOT RECOVERED: ${kind}/${ns}/${name}"
|
||||
unrecovered+=("${kind}/${ns}/${name}")
|
||||
fi
|
||||
done <"$failed_file"
|
||||
echo "ROLLED_BACK=${#recovered[@]}" >>"$failed_file"
|
||||
echo "UNRECOVERED=${#unrecovered[@]}" >>"$failed_file"
|
||||
[ "${#unrecovered[@]}" -eq 0 ]
|
||||
}
|
||||
|
||||
# Helm releases owned by this stage, one line each:
|
||||
#
|
||||
# release|chart|namespace|chart version|values file (rel. to $REPO)|active marker
|
||||
#
|
||||
# The chart version is the field Renovate keeps current. The helmv3 manager only
|
||||
# understands Chart.yaml and the helm-values manager only values files, so a pin
|
||||
# written straight into a `helm upgrade` command would never be updated: these
|
||||
# have to be declared as custom.regex managers in renovate/renovate.json.
|
||||
HELM_RELEASES=(
|
||||
"prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.2.3|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
|
||||
"loki|grafana/loki|prometheus|7.3.0|loki/k8s/loki-values.yaml|loki/k8s/active"
|
||||
"alloy|grafana/alloy|prometheus|1.12.1|loki/k8s/alloy-values.yaml|loki/k8s/active"
|
||||
"reloader|stakater/reloader|reloader|2.2.17|reloader/k8s/reloader-values.yaml|reloader/k8s/active"
|
||||
)
|
||||
|
||||
# "name url" for the Helm repository hosting a chart, empty if unknown.
|
||||
helm_repo_for() {
|
||||
case "$1" in
|
||||
prometheus-community/*) echo "prometheus-community https://prometheus-community.github.io/helm-charts" ;;
|
||||
grafana/*) echo "grafana https://grafana.github.io/helm-charts" ;;
|
||||
stakater/*) echo "stakater https://stakater.github.io/stakater-charts" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
upgrade_helm_releases() {
|
||||
local entry release chart namespace version values marker repo
|
||||
for entry in ${HELM_RELEASES[@]+"${HELM_RELEASES[@]}"}; do
|
||||
IFS='|' read -r release chart namespace version values marker <<<"$entry"
|
||||
if [ ! -f "$REPO/$marker" ]; then
|
||||
echo "skip (no $marker): $release"
|
||||
continue
|
||||
fi
|
||||
if [ ! -f "$REPO/$values" ]; then
|
||||
echo "ERROR: $values is gitignored but missing on the workstation, restore it first."
|
||||
return 1
|
||||
fi
|
||||
repo="$(helm_repo_for "$chart")"
|
||||
if [ -z "$repo" ]; then
|
||||
echo "ERROR: no Helm repository configured for chart $chart"
|
||||
return 1
|
||||
fi
|
||||
helm repo add "${repo%% *}" "${repo#* }" >/dev/null 2>&1 || true
|
||||
helm repo update "${repo%% *}" >/dev/null 2>&1 || true
|
||||
log "Upgrading $release ($chart $version)"
|
||||
# --atomic rolls the release back when the upgrade times out or the workloads
|
||||
# it touches never become ready, so a bad chart bump is not left half applied.
|
||||
helm upgrade --install "$release" "$chart" \
|
||||
--namespace "$namespace" \
|
||||
--version "$version" \
|
||||
--values "$REPO/$values" \
|
||||
--atomic --cleanup-on-fail --timeout 10m
|
||||
done
|
||||
}
|
||||
|
||||
stage_preflight() {
|
||||
if [ ! -d "$REPO/.git" ]; then
|
||||
echo "Repository not found at $REPO"
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "$DEPLOY_SHA" ]; then
|
||||
log "Checking out the commit CI validated ($DEPLOY_SHA)"
|
||||
git -C "$REPO" fetch origin --quiet "$DEPLOY_SHA" 2>/dev/null \
|
||||
|| git -C "$REPO" fetch origin main
|
||||
else
|
||||
git -C "$REPO" fetch origin main
|
||||
fi
|
||||
target="${DEPLOY_SHA:-origin/main}"
|
||||
log "Workstation state"
|
||||
echo " local: $(git -C "$REPO" rev-parse --short HEAD)"
|
||||
echo " target: $(git -C "$REPO" rev-parse --short "$target")"
|
||||
if [ -n "$(git -C "$REPO" status --porcelain --untracked-files=no)" ]; then
|
||||
echo "ERROR: workstation has local tracked modifications, refusing reset:"
|
||||
git -C "$REPO" status --porcelain --untracked-files=no
|
||||
git -C "$REPO" diff --stat
|
||||
echo "Fix it on the workstation (commit, or 'git restore .'), then re-run the deploy."
|
||||
exit 1
|
||||
fi
|
||||
git -C "$REPO" reset --hard "$target"
|
||||
}
|
||||
|
||||
stage_validate() {
|
||||
cd "$REPO"
|
||||
select_manifests
|
||||
local m k cf
|
||||
# Compose .env files and secret files are gitignored by design, so the
|
||||
# workstation never has real values for the inactive stacks. This stage only
|
||||
# runs the full check on active stacks; the general structure check for every
|
||||
# committed Compose file (active or not) lives in the ci workflow, which has no
|
||||
# .env at all.
|
||||
#
|
||||
# Active stacks are still validated with interpolation and env-file resolution
|
||||
# off, so required-variable guards (:?) and missing local files do not fail the
|
||||
# deploy. Normalization and consistency checks stay enabled.
|
||||
# shellcheck source=compose-lint.sh
|
||||
source "$REPO/.gitea/workflows/compose-lint.sh"
|
||||
local compose_validate_flags=()
|
||||
mapfile -t compose_validate_flags < <(compose_safe_flags)
|
||||
log "Validate compose stacks"
|
||||
for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
|
||||
echo " config: $cf"
|
||||
validate_compose_file "$cf" ${compose_validate_flags[@]+"${compose_validate_flags[@]}"}
|
||||
done
|
||||
log "Validate k8s manifests (kubectl dry-run=client)"
|
||||
for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
|
||||
kubectl apply --dry-run=client -f "$m" >/dev/null
|
||||
done
|
||||
for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
|
||||
kubectl apply -k "$k" --dry-run=client >/dev/null
|
||||
done
|
||||
log "Validate k8s manifests (kubectl dry-run=server)"
|
||||
for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
|
||||
kubectl apply --dry-run=server -f "$m" >/dev/null
|
||||
done
|
||||
for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
|
||||
kubectl apply -k "$k" --dry-run=server >/dev/null
|
||||
done
|
||||
log "Checking referenced Secrets exist"
|
||||
echo " (deploy never applies *secret*.yaml; create missing ones manually)"
|
||||
local ref_secrets=() missing_secrets=() all_secrets s
|
||||
if [ "${#K8S_MANIFESTS[@]}" -gt 0 ]; then
|
||||
while IFS= read -r s; do
|
||||
[ -n "$s" ] && ref_secrets+=("$s")
|
||||
done < <(
|
||||
{
|
||||
grep -h -A1 -E 'secretRef:|secretKeyRef:' "${K8S_MANIFESTS[@]}" 2>/dev/null || true
|
||||
grep -h -E 'secretName:' "${K8S_MANIFESTS[@]}" 2>/dev/null || true
|
||||
} | grep -E 'name:' | sed -E 's/.*name:[[:space:]]*//' | tr -d '"'"'"' "'"'" | sed -E 's/[[:space:]]*#.*//' | awk 'NF' | sort -u || true
|
||||
)
|
||||
fi
|
||||
all_secrets="$(kubectl get secrets -A --no-headers -o custom-columns=:metadata.name 2>/dev/null || true)"
|
||||
for s in ${ref_secrets[@]+"${ref_secrets[@]}"}; do
|
||||
if printf '%s\n' "$all_secrets" | grep -qx "$s"; then
|
||||
echo " ok: $s"
|
||||
else
|
||||
echo " MISSING: $s"
|
||||
missing_secrets+=("$s")
|
||||
fi
|
||||
done
|
||||
if [ "${#missing_secrets[@]}" -gt 0 ]; then
|
||||
echo "ERROR: ${#missing_secrets[@]} referenced Secret(s) not found in the cluster:"
|
||||
printf ' - %s\n' "${missing_secrets[@]}"
|
||||
echo "Create them manually from the laptop, e.g.:"
|
||||
echo " kubectl apply -f SERVICE/k8s/secrets.yaml # see SERVICE/k8s/secrets.yaml.example"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
stage_apply_k8s() {
|
||||
cd "$REPO"
|
||||
select_manifests >/dev/null
|
||||
local ns_files=() other_files=() m k prune_opts=()
|
||||
for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
|
||||
case "$m" in
|
||||
*/namespace.y?ml) ns_files+=("$m") ;;
|
||||
*) other_files+=("$m") ;;
|
||||
esac
|
||||
done
|
||||
if [ "$APPLY_PRUNE" = "true" ]; then
|
||||
prune_opts=(--prune -l app.kubernetes.io/managed-by=homelab-deploy)
|
||||
fi
|
||||
|
||||
# Record what is about to change, and publish it for the verify job, before
|
||||
# the first apply. Both are fatal on failure: see snapshot_dir.
|
||||
local snapshot
|
||||
snapshot="$(snapshot_dir)" || return 1
|
||||
save_snapshot "$snapshot" || return 1
|
||||
|
||||
if [ "${#ns_files[@]}" -gt 0 ]; then
|
||||
log "Applying namespaces (${#ns_files[@]} files)"
|
||||
for m in "${ns_files[@]}"; do
|
||||
kubectl apply -f "$m"
|
||||
done
|
||||
fi
|
||||
if [ -f "$REPO/prometheus-stack/k8s/active" ]; then
|
||||
if [ ! -f "$REPO/prometheus-stack/k8s/grafana-values.yaml" ]; then
|
||||
echo "ERROR: prometheus-stack/k8s/grafana-values.yaml (gitignored) missing on workstation, restore it first."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
upgrade_helm_releases
|
||||
if [ "${#other_files[@]}" -gt 0 ]; then
|
||||
log "Applying resources (${#other_files[@]} files, our images pinned to digests)"
|
||||
for m in "${other_files[@]}"; do
|
||||
if ! render_pinned <"$m" | kubectl apply "${prune_opts[@]}" -f -; then
|
||||
echo "ERROR: apply failed for ${m#"$REPO"/}" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
|
||||
log "Applying kustomize app: ${k#"$REPO"/} (our images pinned to digests)"
|
||||
if ! kubectl kustomize "$k" | render_pinned | kubectl apply -f -; then
|
||||
echo "ERROR: apply failed for kustomize app ${k#"$REPO"/}" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if [ -f "$REPO/userbot/k8s/active" ]; then
|
||||
log "userbot panel hook"
|
||||
if kubectl get secret userbot-common-secrets -n userbot >/dev/null 2>&1; then
|
||||
echo " userbot-common-secrets already present in userbot ns, not touching"
|
||||
elif kubectl get secret userbot-common-secrets -n default >/dev/null 2>&1; then
|
||||
echo " bootstrapping userbot-common-secrets into userbot ns"
|
||||
kubectl get secret userbot-common-secrets -n default -o json \
|
||||
| jq 'del(.metadata.annotations,.metadata.creationTimestamp,.metadata.resourceVersion,.metadata.uid,.metadata.managedFields) | .metadata.namespace = "userbot"' \
|
||||
| kubectl apply -f -
|
||||
else
|
||||
echo " WARNING: userbot-common-secrets missing in both default and userbot ns; create it manually from the laptop"
|
||||
fi
|
||||
fi
|
||||
restart_stale_images
|
||||
|
||||
# No verification here on purpose. This stage may be killed at any point by
|
||||
# timeout-minutes, by the runner cancelling the job, or by a dropped SSH
|
||||
# connection, and any code below that line would simply not run. stage_verify_k8s
|
||||
# picks the work up from the snapshot instead.
|
||||
log "Applied. Verification and rollback are the verify job's job, not this one's."
|
||||
}
|
||||
|
||||
# Runs as its own workflow job, after apply-k8s (and apply-compose) are done —
|
||||
# including when they failed, timed out or were cancelled. Reads the baseline the
|
||||
# apply stage published and works out what it changed, watches those workloads,
|
||||
# and rolls back the ones that never became healthy.
|
||||
stage_verify_k8s() {
|
||||
local pointer="$DEPLOY_SNAPSHOT_DIR/current"
|
||||
local snapshot want have generations
|
||||
local -a touched=()
|
||||
|
||||
if [ ! -s "$pointer" ]; then
|
||||
echo "ERROR: no snapshot pointer at $pointer."
|
||||
echo "The apply stage died before publishing any state, so there is no baseline to"
|
||||
echo "tell which workloads it touched. Nothing can be rolled back automatically —"
|
||||
echo "inspect the cluster by hand."
|
||||
return 1
|
||||
fi
|
||||
snapshot="$(head -1 "$pointer")"
|
||||
if [ ! -d "$snapshot" ]; then
|
||||
echo "ERROR: snapshot pointer refers to a missing directory: $snapshot"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Never trust the pointer blindly. If the apply stage was killed before it
|
||||
# published its own snapshot, `current` still points at the previous deploy's
|
||||
# baseline. Verifying against that would watch the wrong workloads and the
|
||||
# rollback would revert the wrong revisions, so refuse instead.
|
||||
want="${DEPLOY_SHA:-}"
|
||||
if [ -z "$want" ]; then
|
||||
want="$(git -C "$REPO" rev-parse HEAD 2>/dev/null || true)"
|
||||
fi
|
||||
have="$(cat "$snapshot/commit" 2>/dev/null || true)"
|
||||
if [ -z "$want" ] || [ "$have" != "$want" ]; then
|
||||
echo "ERROR: refusing to verify or roll back against a stale snapshot."
|
||||
echo " snapshot: $snapshot"
|
||||
echo " snapshot commit: ${have:-<missing>}"
|
||||
echo " deploy commit: ${want:-<unknown>}"
|
||||
return 1
|
||||
fi
|
||||
echo " snapshot: $snapshot (commit ${have:0:12})"
|
||||
|
||||
generations="$snapshot/generations.before"
|
||||
if [ ! -s "$generations" ]; then
|
||||
# Without a baseline we cannot tell which workloads the apply touched, so
|
||||
# fall back to watching everything rather than silently skipping the check.
|
||||
warn "no pre-apply baseline, verifying every workload in the cluster"
|
||||
: >"$generations"
|
||||
fi
|
||||
|
||||
while read -r kind ns name; do
|
||||
[ -n "${kind:-}" ] && touched+=("$kind $ns $name")
|
||||
done < <(changed_workloads "$generations")
|
||||
|
||||
log "Verifying ${#touched[@]} changed workload(s) (timeout ${ROLLOUT_TIMEOUT}s each)"
|
||||
if [ "${#touched[@]}" -eq 0 ]; then
|
||||
echo " nothing to verify"
|
||||
return 0
|
||||
fi
|
||||
printf ' watching: %s\n' "${touched[@]/#/ }"
|
||||
|
||||
local failed_file="$snapshot/failed-workloads"
|
||||
if ! verify_workloads "$failed_file" ${touched[@]+"${touched[@]}"}; then
|
||||
echo
|
||||
echo "ERROR: ${#touched[@]} workload(s) changed by this deploy, and these never became healthy:"
|
||||
grep -v -E '^(ROLLED_BACK|UNRECOVERED)=' "$failed_file" | sed 's/^/ - /'
|
||||
echo
|
||||
log "Rolling back to the previous revision"
|
||||
if rollback_workloads "$failed_file"; then
|
||||
echo
|
||||
echo "Rolled back successfully. The cluster is back on the pre-deploy revision."
|
||||
echo "Nothing else was reverted: Git holds desired state only, so config changes, PVCs and"
|
||||
echo "externally created resources from this commit are still in place. Review the failed"
|
||||
echo "workload, then re-run the deploy (Actions -> deploy -> Run workflow)."
|
||||
else
|
||||
echo
|
||||
echo "Rollback did NOT fully recover the cluster. Manual intervention required:"
|
||||
grep -E '^(ROLLED_BACK|UNRECOVERED)=' "$failed_file" | sed 's/^/ /'
|
||||
echo "Pre-apply snapshot: $snapshot"
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# verify_compose_stack <compose-file>
|
||||
# `docker compose up -d` exits 0 as soon as containers are created, so a stack can
|
||||
# come back broken with a green pipeline. Require every long-running service to
|
||||
# actually be running.
|
||||
verify_compose_stack() {
|
||||
local cf="$1"
|
||||
local expected running missing=()
|
||||
expected="$(docker compose -f "$cf" config --services 2>/dev/null | sort || true)"
|
||||
running="$(docker compose -f "$cf" ps --status running --services 2>/dev/null | sort || true)"
|
||||
[ -n "$expected" ] || return 0
|
||||
while IFS= read -r svc; do
|
||||
[ -n "$svc" ] || continue
|
||||
# restart:"no" services are allowed to have exited.
|
||||
if ! printf '%s\n' "$running" | grep -qx "$svc" \
|
||||
&& ! docker compose -f "$cf" config 2>/dev/null \
|
||||
| grep -A5 "^ ${svc}:" | grep -qE 'restart:\s*"?no"?'; then
|
||||
missing+=("$svc")
|
||||
fi
|
||||
done <<<"$expected"
|
||||
if [ "${#missing[@]}" -gt 0 ]; then
|
||||
echo " NOT RUNNING: ${missing[*]}"
|
||||
docker compose -f "$cf" ps --all 2>/dev/null | sed 's/^/ /' || true
|
||||
return 1
|
||||
fi
|
||||
echo " all ${#expected} service(s) running"
|
||||
return 0
|
||||
}
|
||||
|
||||
# The public hostname of every active service, one per line.
|
||||
#
|
||||
# Comments are stripped first, and deliberately so: a route that someone
|
||||
# disabled by commenting it out is not a service to probe, and naio and xui are
|
||||
# both still in the tree that way. A `#` only starts a comment when it is at the
|
||||
# start of a line or after whitespace, so `s/#.*//` alone would also cut a
|
||||
# legitimate value in half.
|
||||
#
|
||||
# Only the public names. The *.internal names are the same Traefik and the same
|
||||
# Services, reached by a different label, so probing both would double the run
|
||||
# to learn the same thing. The public name is also the one a user types.
|
||||
smoke_hosts() {
|
||||
local m k
|
||||
# The backticks below are literal. They are Traefik's Host() delimiter, and the
|
||||
# single quotes are precisely what keeps the shell from reading them as a
|
||||
# command substitution, so the warning is the opposite of a real problem.
|
||||
# shellcheck disable=SC2016
|
||||
{
|
||||
for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
|
||||
[ -f "$m" ] && cat "$m"
|
||||
done
|
||||
for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
|
||||
kubectl kustomize "$k" 2>/dev/null || true
|
||||
done
|
||||
} | sed -E 's/(^|[[:space:]])#.*$//' \
|
||||
| grep -oE 'Host\(`[^`]+`\)' \
|
||||
| sed -E 's/^Host\(`//; s/`\)$//' \
|
||||
| grep -E '(^|\.)forust\.xyz$' \
|
||||
| grep -v '\${' \
|
||||
| sort -u
|
||||
}
|
||||
|
||||
# stage_verify_k8s watches the rollout, which reports that the pods converged.
|
||||
# It cannot tell a converged pod from a serving one: a route pointing at the
|
||||
# wrong port, a Service selector that matches nothing the app listens on, a 500
|
||||
# from the app itself, an OOMKill loop that still counts as Available for long
|
||||
# enough to pass. All of those are green at the rollout level.
|
||||
#
|
||||
# So ask the thing users ask. Any HTTP response proves Traefik matched the
|
||||
# host, the Service resolved to a pod and the pod answered -- a 302 to a login
|
||||
# or a 404 from a path the service does not serve still means the chain is
|
||||
# intact. Only a transport failure (no DNS, refused, timeout) or a 5xx means
|
||||
# the service is not serving, and only those fail the run.
|
||||
stage_smoke() {
|
||||
cd "$REPO"
|
||||
select_manifests >/dev/null
|
||||
local -a hosts=()
|
||||
# Not named failed: an array of that name already exists in restart_stale_images
|
||||
# above, and a scalar shadowing an array is a trap rather than a shadow.
|
||||
local h code rc bad=0
|
||||
while IFS= read -r h; do
|
||||
[ -n "$h" ] && hosts+=("$h")
|
||||
done < <(smoke_hosts)
|
||||
|
||||
if [ "${#hosts[@]}" -eq 0 ]; then
|
||||
# Nothing to probe means the extraction broke, not that the cluster is empty.
|
||||
echo "ERROR: no public hostnames found in active manifests, refusing to report success"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log "Probing ${#hosts[@]} public route(s)"
|
||||
for h in "${hosts[@]}"; do
|
||||
code="$(curl -sS -o /dev/null --max-time 20 -w '%{http_code}' "https://$h/" 2>/dev/null)" && rc=0 || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo " UNREACHABLE $h (curl exit $rc)"
|
||||
bad=1
|
||||
continue
|
||||
fi
|
||||
# A glob, not a string compare. `case` on the leading digit is the only one
|
||||
# of these that survives a three-digit code, and the obvious expansion to
|
||||
# try first -- ${code%%[0-9]*} -- is empty for every input, so it silently
|
||||
# reports a 500 as healthy.
|
||||
case "$code" in
|
||||
5*)
|
||||
echo " SERVER ERROR $h $code"
|
||||
bad=1
|
||||
;;
|
||||
000)
|
||||
# curl exited 0 and still no status, so nothing on the far end replied.
|
||||
# Not a pass, whatever the transport thought.
|
||||
echo " NO RESPONSE $h"
|
||||
bad=1
|
||||
;;
|
||||
*)
|
||||
echo " ok $h $code"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$bad" -ne 0 ]; then
|
||||
echo "ERROR: at least one active service is not serving over its public route"
|
||||
return 1
|
||||
fi
|
||||
echo "all ${#hosts[@]} route(s) answered"
|
||||
}
|
||||
|
||||
stage_apply_compose() {
|
||||
cd "$REPO"
|
||||
select_manifests >/dev/null
|
||||
local cf
|
||||
log "Redeploying docker compose stacks (${#COMPOSE_STACKS[@]} stacks)"
|
||||
for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
|
||||
echo " compose: $cf"
|
||||
if grep -Eq '^\s+pull_policy:\s*build\b' "$cf"; then
|
||||
docker compose -f "$cf" build
|
||||
docker compose -f "$cf" push
|
||||
fi
|
||||
docker compose -f "$cf" up -d --pull always --remove-orphans
|
||||
done
|
||||
|
||||
local -a broken=()
|
||||
for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
|
||||
echo " verifying: $cf"
|
||||
if ! verify_compose_stack "$cf"; then
|
||||
broken+=("$cf")
|
||||
fi
|
||||
done
|
||||
if [ "${#broken[@]}" -gt 0 ]; then
|
||||
echo
|
||||
echo "ERROR: ${#broken[@]} compose stack(s) did not come up:"
|
||||
printf ' - %s\n' "${broken[@]}"
|
||||
echo "Compose stacks are not rolled back automatically: their images use mutable"
|
||||
echo "':latest' tags, so there is no previous version to return to. Check the logs"
|
||||
echo "above, then re-run the deploy once the cause is fixed."
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_stage() {
|
||||
case "${1:?stage required}" in
|
||||
preflight) stage_preflight ;;
|
||||
validate) stage_validate ;;
|
||||
apply-k8s) stage_apply_k8s ;;
|
||||
verify-k8s) stage_verify_k8s ;;
|
||||
smoke) stage_smoke ;;
|
||||
apply-compose) stage_apply_compose ;;
|
||||
*)
|
||||
echo "ERROR: unknown stage: $1"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
@@ -1,151 +0,0 @@
|
||||
name: deploy
|
||||
|
||||
on:
|
||||
# Deploy only what CI already validated. workflow_run is used instead of
|
||||
# workflow_dispatch so a red lint/validate run can never reach the cluster.
|
||||
workflow_run:
|
||||
workflows: [ci]
|
||||
types: [completed]
|
||||
workflow_dispatch:
|
||||
|
||||
# The deploy jobs read the tree, then reach the cluster over SSH with the
|
||||
# deploy key. The Actions token itself is not part of that path, so it gets
|
||||
# read-only contents and no more.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: deploy-main
|
||||
# Queue instead of cancelling. Cancelling a run kills the apply job mid-loop and
|
||||
# takes the verify job down with it, so a superseded deploy would leave the
|
||||
# cluster half-applied and unchecked — the exact failure the verify job exists
|
||||
# to catch. kubectl apply and docker compose up are both idempotent, so letting
|
||||
# the older run finish and then deploying the newer commit costs little.
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
|
||||
DEPLOY_PORT: ${{ secrets.DEPLOY_PORT }}
|
||||
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
|
||||
DEPLOY_PATH: ${{ secrets.DEPLOY_PATH }}
|
||||
DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
|
||||
APPLY_PRUNE: ${{ vars.APPLY_PRUNE }}
|
||||
# workflow_run's own GITHUB_SHA points at the branch head, not at the commit the
|
||||
# finished ci run checked. Pin the exact validated commit instead, so a push
|
||||
# landing mid-deploy cannot make the workstation deploy something else. Also
|
||||
# what the verify job checks the snapshot against. Empty for workflow_dispatch,
|
||||
# which falls back to the current origin/main.
|
||||
DEPLOY_SHA: ${{ github.event.workflow_run.head_sha }}
|
||||
|
||||
jobs:
|
||||
preflight:
|
||||
if: >-
|
||||
github.event_name != 'workflow_run' ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.head_branch == 'main')
|
||||
runs-on: [self-hosted, linux, arch, homelab, prod]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Fetch and reset workstation
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/ssh-run.sh preflight
|
||||
|
||||
validate:
|
||||
needs: [preflight]
|
||||
runs-on: [self-hosted, linux, arch, homelab, prod]
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Dry-run manifests and check Secrets
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/ssh-run.sh validate
|
||||
|
||||
apply-k8s:
|
||||
needs: [validate]
|
||||
runs-on: [self-hosted, linux, arch, homelab, prod]
|
||||
# Apply only, no verification, so this is just the work itself: snapshot,
|
||||
# then up to three sequential `helm upgrade --atomic --timeout 10m`, then the
|
||||
# apply loop. Verification has its own job and its own budget.
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Apply Kubernetes manifests
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/ssh-run.sh apply-k8s
|
||||
|
||||
apply-compose:
|
||||
needs: [validate]
|
||||
runs-on: [self-hosted, linux, arch, homelab, prod]
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Redeploy docker compose stacks
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/ssh-run.sh apply-compose
|
||||
|
||||
# Watches the workloads this deploy changed and rolls back the ones that never
|
||||
# became healthy. Runs even when the apply jobs failed, timed out or were
|
||||
# cancelled — that is the whole point of splitting it out. `always()` is what
|
||||
# lets it start after a failed dependency; the needs on apply-compose are a
|
||||
# barrier, so verification begins only once both applies are done.
|
||||
verify-k8s:
|
||||
needs: [apply-k8s, apply-compose]
|
||||
if: >-
|
||||
always() &&
|
||||
needs.apply-k8s.result != 'skipped' &&
|
||||
needs.apply-compose.result != 'skipped'
|
||||
runs-on: [self-hosted, linux, arch, homelab, prod]
|
||||
# ceil(changed_workloads / 8) waves of ROLLOUT_TIMEOUT each, plus rollback.
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Verify workloads and roll back on failure
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/ssh-run.sh verify-k8s
|
||||
|
||||
# Asks the public route of every active service whether it is actually
|
||||
# serving, which the rollout check above structurally cannot: a pod can
|
||||
# converge and still be crash-looping, or be listening on a port no Service
|
||||
# points at, or answer 500.
|
||||
#
|
||||
# `always()` for the same reason verify-k8s has it, and it runs after that job
|
||||
# specifically because a rollback is when a route most needs re-checking. The
|
||||
# needs is a barrier, not a filter: whether verify-k8s passed, failed or was
|
||||
# cancelled, the probes are what say whether the cluster is serving, and
|
||||
# suppressing them on a rollback would hide the one run where the answer
|
||||
# matters most.
|
||||
smoke:
|
||||
needs: [verify-k8s]
|
||||
if: always() && needs.verify-k8s.result != 'skipped'
|
||||
runs-on: [self-hosted, linux, arch, homelab, prod]
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
- name: Probe the public route of every active service
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/ssh-run.sh smoke
|
||||
@@ -1,233 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Installs the pinned CI tools into "$TOOLS_DIR/bin" and echoes that directory
|
||||
# on stdout, so callers can do:
|
||||
#
|
||||
# export PATH="$(bash .gitea/workflows/install-ci-tools.sh kubeconform shellcheck):$PATH"
|
||||
#
|
||||
# Versions come from tool-versions.env next to this script and are kept fresh by
|
||||
# Renovate. Re-running is cheap: an already-installed tool at the pinned version
|
||||
# is left alone.
|
||||
set -euo pipefail
|
||||
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=tool-versions.env
|
||||
. "$here/tool-versions.env"
|
||||
|
||||
TOOLS_DIR="${TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/homelab-tools}"
|
||||
BIN_DIR="$TOOLS_DIR/bin"
|
||||
mkdir -p "$BIN_DIR"
|
||||
|
||||
arch="$(uname -m)"
|
||||
# Upstream projects disagree on arch spelling: kubeconform and actionlint use
|
||||
# Go names (amd64/arm64), shellcheck uses uname names (x86_64/aarch64), node
|
||||
# uses neither (x64/arm64), and hadolint mixes the two in a single release
|
||||
# (x86_64 but arm64).
|
||||
case "$arch" in
|
||||
x86_64 | amd64)
|
||||
goarch=amd64
|
||||
sharch=x86_64
|
||||
nodearch=x64
|
||||
hadolintarch=x86_64
|
||||
;;
|
||||
aarch64 | arm64)
|
||||
goarch=arm64
|
||||
sharch=aarch64
|
||||
nodearch=arm64
|
||||
hadolintarch=arm64
|
||||
;;
|
||||
*)
|
||||
echo "install-ci-tools: unsupported architecture: $arch" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
fetch() {
|
||||
# fetch <url> <dest>
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
curl -sSLf --retry 3 -o "$2" "$1"
|
||||
elif command -v wget >/dev/null 2>&1; then
|
||||
wget -q -O "$2" "$1"
|
||||
else
|
||||
echo "install-ci-tools: neither curl nor wget is available" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# installed_version <command>
|
||||
# Prints the version of an already-installed tool, or nothing. Each tool spells
|
||||
# its version flag differently, hence the case.
|
||||
installed_version() {
|
||||
local out
|
||||
case "$1" in
|
||||
kubeconform) out="$("$1" -v 2>/dev/null | head -1 || true)" ;;
|
||||
*) out="$("$1" --version 2>/dev/null | head -1 || true)" ;;
|
||||
esac
|
||||
printf '%s' "$out"
|
||||
}
|
||||
|
||||
# at_version <command> <expected>
|
||||
at_version() {
|
||||
case "$(installed_version "$1")" in
|
||||
*"$2"*) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
install_kubeconform() {
|
||||
if at_version kubeconform "v${KUBECONFORM_VERSION}"; then
|
||||
return 0
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -d)"
|
||||
fetch "https://github.com/yannh/kubeconform/releases/download/v${KUBECONFORM_VERSION}/kubeconform-linux-${goarch}.tar.gz" \
|
||||
"$tmp/kubeconform.tar.gz"
|
||||
tar -xzf "$tmp/kubeconform.tar.gz" -C "$tmp" kubeconform
|
||||
install -m 0755 "$tmp/kubeconform" "$BIN_DIR/kubeconform"
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
install_shellcheck() {
|
||||
if at_version shellcheck "${SHELLCHECK_VERSION}"; then
|
||||
return 0
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -d)"
|
||||
fetch "https://github.com/koalaman/shellcheck/releases/download/v${SHELLCHECK_VERSION}/shellcheck-v${SHELLCHECK_VERSION}.linux.${sharch}.tar.xz" \
|
||||
"$tmp/shellcheck.tar.xz"
|
||||
tar -xJf "$tmp/shellcheck.tar.xz" -C "$tmp" --strip-components=1 "shellcheck-v${SHELLCHECK_VERSION}/shellcheck"
|
||||
install -m 0755 "$tmp/shellcheck" "$BIN_DIR/shellcheck"
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
install_uv() {
|
||||
if at_version uv "${UV_VERSION}"; then
|
||||
return 0
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -d)"
|
||||
# uv release tags carry no leading v, unlike every other tool installed here.
|
||||
fetch "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${sharch}-unknown-linux-gnu.tar.gz" \
|
||||
"$tmp/uv.tar.gz"
|
||||
tar -xzf "$tmp/uv.tar.gz" -C "$tmp" --strip-components=1 "uv-${sharch}-unknown-linux-gnu/uv"
|
||||
install -m 0755 "$tmp/uv" "$BIN_DIR/uv"
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
install_hadolint() {
|
||||
if at_version hadolint "${HADOLINT_VERSION}"; then
|
||||
return 0
|
||||
fi
|
||||
# A bare binary, no archive: hadolint ships one file per platform.
|
||||
fetch "https://github.com/hadolint/hadolint/releases/download/v${HADOLINT_VERSION}/hadolint-linux-${hadolintarch}" \
|
||||
"$BIN_DIR/hadolint"
|
||||
chmod 0755 "$BIN_DIR/hadolint"
|
||||
}
|
||||
|
||||
# ruff and yamllint both come from PyPI as wheels, which uv unpacks for us.
|
||||
install_uv_tool() {
|
||||
# <package> <pinned version>
|
||||
if at_version "$1" "$2"; then
|
||||
return 0
|
||||
fi
|
||||
install_uv
|
||||
UV_TOOL_BIN_DIR="$BIN_DIR" uv tool install --force "$1==$2" >/dev/null
|
||||
}
|
||||
|
||||
install_ruff() {
|
||||
install_uv_tool ruff "${RUFF_VERSION}"
|
||||
}
|
||||
|
||||
install_yamllint() {
|
||||
install_uv_tool yamllint "${YAMLLINT_VERSION}"
|
||||
}
|
||||
|
||||
install_pip_audit() {
|
||||
install_uv_tool pip-audit "${PIP_AUDIT_VERSION}"
|
||||
}
|
||||
|
||||
install_prettier() {
|
||||
if at_version prettier "${PRETTIER_VERSION}"; then
|
||||
return 0
|
||||
fi
|
||||
# Not a standalone binary: prettier's entry point requires ../package.json
|
||||
# relative to its own real path, so the package directory has to survive
|
||||
# next to it. Hence a versioned directory plus a relative symlink, rather
|
||||
# than copying the one file out as the other installers do.
|
||||
local dir="$BIN_DIR/prettier-${PRETTIER_VERSION}"
|
||||
if [ ! -f "$dir/package/package.json" ]; then
|
||||
rm -rf "$dir"
|
||||
mkdir -p "$dir"
|
||||
fetch "https://registry.npmjs.org/prettier/-/prettier-${PRETTIER_VERSION}.tgz" "$dir/prettier.tgz"
|
||||
tar -xzf "$dir/prettier.tgz" -C "$dir"
|
||||
rm -f "$dir/prettier.tgz"
|
||||
# npm strips the exec bit from bin/ on the way into the tarball.
|
||||
chmod 0755 "$dir/package/bin/prettier.cjs"
|
||||
fi
|
||||
# Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
|
||||
ln -sfn "prettier-${PRETTIER_VERSION}/package/bin/prettier.cjs" "$BIN_DIR/prettier"
|
||||
}
|
||||
|
||||
install_node() {
|
||||
# npm gets checked by running it, not by looking it up: what matters is that
|
||||
# it answers, so a stub, a half-removed Arch package or a name that resolves
|
||||
# to something broken all have to read as "not installed". The runner's npm
|
||||
# is a symlink into /usr/lib/node_modules/npm, which is exactly the kind of
|
||||
# thing that disappears between runs.
|
||||
if at_version node "v${NODE_VERSION}" && [ -n "$(installed_version npm)" ]; then
|
||||
return 0
|
||||
fi
|
||||
# Same shape as prettier above: the tarball's bin/npm and bin/npx are links
|
||||
# into lib/node_modules, so the whole tree has to survive next to them.
|
||||
local dir="$BIN_DIR/node-${NODE_VERSION}"
|
||||
if [ ! -x "$dir/bin/node" ]; then
|
||||
rm -rf "$dir"
|
||||
mkdir -p "$dir"
|
||||
fetch "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${nodearch}.tar.xz" \
|
||||
"$dir/node.tar.xz"
|
||||
tar -xJf "$dir/node.tar.xz" -C "$dir" --strip-components=1 "node-v${NODE_VERSION}-linux-${nodearch}"
|
||||
rm -f "$dir/node.tar.xz"
|
||||
fi
|
||||
# Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
|
||||
for bin in node npm npx; do
|
||||
ln -sfn "node-${NODE_VERSION}/bin/${bin}" "$BIN_DIR/${bin}"
|
||||
done
|
||||
}
|
||||
|
||||
install_actionlint() {
|
||||
if at_version actionlint "${ACTIONLINT_VERSION}"; then
|
||||
return 0
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -d)"
|
||||
fetch "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/actionlint_${ACTIONLINT_VERSION}_linux_${goarch}.tar.gz" \
|
||||
"$tmp/actionlint.tar.gz"
|
||||
tar -xzf "$tmp/actionlint.tar.gz" -C "$tmp" actionlint
|
||||
install -m 0755 "$tmp/actionlint" "$BIN_DIR/actionlint"
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
wanted=("$@")
|
||||
if [ "${#wanted[@]}" -eq 0 ]; then
|
||||
wanted=(kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
|
||||
fi
|
||||
|
||||
for tool in "${wanted[@]}"; do
|
||||
case "$tool" in
|
||||
kubeconform) install_kubeconform ;;
|
||||
shellcheck) install_shellcheck ;;
|
||||
actionlint) install_actionlint ;;
|
||||
prettier) install_prettier ;;
|
||||
ruff) install_ruff ;;
|
||||
yamllint) install_yamllint ;;
|
||||
pip-audit) install_pip_audit ;;
|
||||
hadolint) install_hadolint ;;
|
||||
node) install_node ;;
|
||||
uv) install_uv ;;
|
||||
*)
|
||||
echo "install-ci-tools: unknown tool: $tool" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
printf '%s\n' "$BIN_DIR"
|
||||
@@ -1,76 +0,0 @@
|
||||
name: renovate-ci
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
validate-renovate:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
# renovate/k8s/cronjob.yaml is the single source of truth for the image tag,
|
||||
# so the same version that runs in the cluster is the one validated here.
|
||||
- name: Resolve the deployed Renovate image
|
||||
id: image
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
|
||||
renovate/k8s/cronjob.yaml | head -1)"
|
||||
if [ -z "$image" ]; then
|
||||
echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
|
||||
exit 1
|
||||
fi
|
||||
echo "using $image"
|
||||
echo "image=$image" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Validate Renovate repository config
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
docker run --rm \
|
||||
-v "$PWD/renovate:/opt/renovate:ro" \
|
||||
-e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
|
||||
"${{ steps.image.outputs.image }}" \
|
||||
renovate-config-validator /opt/renovate/renovate.json
|
||||
|
||||
# The CronJob cannot read the repository, so renovate/k8s/configmap.yaml
|
||||
# carries an inlined copy of the config. Fail if it no longer matches.
|
||||
- name: Check the generated Renovate ConfigMap
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
./.gitea/workflows/sync-renovate-configmap.sh --check
|
||||
|
||||
- name: Validate Renovate Kubernetes manifests
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
|
||||
export PATH="$tools_dir:$PATH"
|
||||
kubeconform \
|
||||
-strict \
|
||||
-ignore-missing-schemas \
|
||||
-summary \
|
||||
renovate/k8s/namespace.yaml \
|
||||
renovate/k8s/configmap.yaml \
|
||||
renovate/k8s/cronjob.yaml
|
||||
|
||||
- name: Validate Renovate Compose file
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
source .gitea/workflows/compose-lint.sh
|
||||
mapfile -t safe_flags < <(compose_safe_flags)
|
||||
validate_compose_file renovate/renovate-compose.yaml \
|
||||
${safe_flags[@]+"${safe_flags[@]}"}
|
||||
@@ -1,92 +0,0 @@
|
||||
name: renovate-run
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
repositories:
|
||||
description: "Repositories to scan (comma-separated)"
|
||||
required: false
|
||||
default: "forust/homelab"
|
||||
log_level:
|
||||
description: "Renovate log level"
|
||||
required: false
|
||||
default: "info"
|
||||
type: choice
|
||||
options:
|
||||
- info
|
||||
- debug
|
||||
dry_run:
|
||||
description: "Plan only, do not open or update PRs"
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
# Renovate writes through its own bot PAT, passed in as RENOVATE_TOKEN, so the
|
||||
# Actions token is only ever used to read the checkout.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: renovate-run
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
run-renovate:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
timeout-minutes: 60
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
|
||||
|
||||
# renovate/k8s/cronjob.yaml is the single source of truth for the image tag.
|
||||
# Reading it here means this workflow validates and runs the exact version
|
||||
# that is deployed, instead of a copy that silently goes stale.
|
||||
- name: Resolve the deployed Renovate image
|
||||
id: image
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
|
||||
renovate/k8s/cronjob.yaml | head -1)"
|
||||
if [ -z "$image" ]; then
|
||||
echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
|
||||
exit 1
|
||||
fi
|
||||
echo "using $image"
|
||||
echo "image=$image" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Validate Renovate config
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
docker run --rm \
|
||||
-v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
|
||||
-e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
|
||||
"${{ steps.image.outputs.image }}" \
|
||||
renovate-config-validator
|
||||
|
||||
- name: Run Renovate
|
||||
shell: bash
|
||||
env:
|
||||
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
|
||||
RENOVATE_GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
|
||||
RENOVATE_REPOSITORIES: ${{ inputs.repositories }}
|
||||
RENOVATE_DRY_RUN: ${{ inputs.dry_run && 'full' || '' }}
|
||||
LOG_LEVEL: ${{ inputs.log_level }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
: "${RENOVATE_TOKEN:?missing RENOVATE_TOKEN secret — add a renovate-bot PAT in repo/org Actions secrets}"
|
||||
|
||||
docker run --rm \
|
||||
-v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
|
||||
-e RENOVATE_PLATFORM=gitea \
|
||||
-e RENOVATE_ENDPOINT=https://gitea.forust.xyz/api/v1 \
|
||||
-e RENOVATE_TOKEN="$RENOVATE_TOKEN" \
|
||||
-e RENOVATE_GITHUB_COM_TOKEN="${RENOVATE_GITHUB_COM_TOKEN:-}" \
|
||||
-e RENOVATE_REPOSITORIES="${RENOVATE_REPOSITORIES:-forust/homelab}" \
|
||||
-e RENOVATE_DRY_RUN="${RENOVATE_DRY_RUN:-}" \
|
||||
-e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
|
||||
-e RENOVATE_BASE_DIR=/tmp/renovate \
|
||||
-e LOG_LEVEL="${LOG_LEVEL:-info}" \
|
||||
"${{ steps.image.outputs.image }}"
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# usage: ssh-run.sh <stage>
|
||||
# Runs one deploy-lib.sh stage on the workstation over SSH.
|
||||
set -euo pipefail
|
||||
|
||||
: "${DEPLOY_HOST:?missing DEPLOY_HOST}"
|
||||
: "${DEPLOY_USER:?missing DEPLOY_USER}"
|
||||
: "${DEPLOY_KEY:?missing DEPLOY_SSH_KEY}"
|
||||
|
||||
deploy_port="${DEPLOY_PORT:-22}"
|
||||
deploy_path="${DEPLOY_PATH:-/srv/homelab}"
|
||||
deploy_path="$(printf '%s' "$deploy_path" | tr -d '\"' | tr -d '\r' | xargs)"
|
||||
|
||||
# The private key is written to a per-run directory that is removed on exit, so a
|
||||
# failed or cancelled job cannot leave deploy credentials in the runner's temp
|
||||
# directory. Do not use a fixed path: apply-k8s and apply-compose run in parallel.
|
||||
key_dir="$(mktemp -d "${RUNNER_TEMP:-/tmp}/homelab-deploy-key.XXXXXXXX")"
|
||||
trap 'rm -rf "$key_dir"' EXIT INT TERM
|
||||
|
||||
ssh_key="$key_dir/deploy_key"
|
||||
printf '%s\n' "$DEPLOY_KEY" > "$ssh_key"
|
||||
chmod 600 "$ssh_key"
|
||||
|
||||
ssh -i "$ssh_key" -p "$deploy_port" \
|
||||
-o BatchMode=yes -o StrictHostKeyChecking=accept-new \
|
||||
"${DEPLOY_USER}@${DEPLOY_HOST}" \
|
||||
"REPO=$deploy_path APPLY_PRUNE=${APPLY_PRUNE:-false} DEPLOY_SHA=${DEPLOY_SHA:-} DEPLOY_SNAPSHOT_DIR=${DEPLOY_SNAPSHOT_DIR:-} STAGE=$1 bash -se" <<'EOF'
|
||||
source "$REPO/.gitea/workflows/deploy-lib.sh"
|
||||
run_stage "$STAGE"
|
||||
EOF
|
||||
@@ -1,55 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerates renovate/k8s/configmap.yaml from renovate/renovate.json.
|
||||
#
|
||||
# renovate/renovate.json is the single source of truth: the CronJob, the Compose
|
||||
# file and the renovate-run workflow all mount that exact file. A ConfigMap cannot
|
||||
# read a file from the repository, so the same bytes are inlined here as a literal
|
||||
# block. This script keeps the copy honest:
|
||||
#
|
||||
# .gitea/workflows/sync-renovate-configmap.sh # rewrite in place
|
||||
# .gitea/workflows/sync-renovate-configmap.sh --check # fail if out of date
|
||||
#
|
||||
# renovate-ci runs the --check form on every PR and push, so a config change that
|
||||
# forgets to regenerate the ConfigMap cannot be merged.
|
||||
set -euo pipefail
|
||||
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo="$(git -C "$here" rev-parse --show-toplevel)"
|
||||
|
||||
src="$repo/renovate/renovate.json"
|
||||
dst="$repo/renovate/k8s/configmap.yaml"
|
||||
[ -f "$src" ] || {
|
||||
echo "missing $src" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
render() {
|
||||
cat <<'HEADER'
|
||||
# GENERATED FILE - do not edit by hand.
|
||||
# Source: renovate/renovate.json
|
||||
# Regenerate: .gitea/workflows/sync-renovate-configmap.sh
|
||||
# Verify: .gitea/workflows/sync-renovate-configmap.sh --check
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: renovate-config
|
||||
namespace: renovate
|
||||
data:
|
||||
renovate.json: |
|
||||
HEADER
|
||||
sed 's/^/ /' "$src"
|
||||
}
|
||||
|
||||
if [ "${1:-}" = "--check" ]; then
|
||||
if ! diff -u "$dst" <(render) >/dev/null 2>&1; then
|
||||
echo "ERROR: $dst is out of sync with renovate/renovate.json"
|
||||
echo "Run: .gitea/workflows/sync-renovate-configmap.sh"
|
||||
diff -u "$dst" <(render) || true
|
||||
exit 1
|
||||
fi
|
||||
echo "renovate/k8s/configmap.yaml is in sync with renovate/renovate.json"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
render >"$dst"
|
||||
echo "wrote $dst"
|
||||
@@ -1,33 +0,0 @@
|
||||
# Pinned versions of the CI tools installed by install-ci-tools.sh.
|
||||
# Renovate keeps these up to date (see customManagers in renovate/renovate.json).
|
||||
#
|
||||
# Every version here except NODE_VERSION matches what was already installed on
|
||||
# the runner, so pinning them changes what CI does not at all. It changes what
|
||||
# CI does when the runner is rebuilt with something else: today
|
||||
# install-ci-tools.sh finds the pinned version already on PATH and installs
|
||||
# nothing, and a runner that drifts gets the pinned one installed over it.
|
||||
#
|
||||
# The renovate image version is NOT pinned here: renovate/k8s/cronjob.yaml is the
|
||||
# single source of truth and the workflows read the tag from it, so there is
|
||||
# nothing to drift.
|
||||
ACTIONLINT_VERSION="1.7.7"
|
||||
SHELLCHECK_VERSION="0.11.0"
|
||||
KUBECONFORM_VERSION="0.8.0"
|
||||
PRETTIER_VERSION="3.8.1"
|
||||
RUFF_VERSION="0.16.8"
|
||||
YAMLLINT_VERSION="1.38.0"
|
||||
HADOLINT_VERSION="2.14.0"
|
||||
# pip-audit reads the advisory database over the network, so a floating version
|
||||
# would make the same commit report different things on different days. Pin it
|
||||
# like the rest: the advisories themselves are the moving part, not the tool.
|
||||
PIP_AUDIT_VERSION="2.10.1"
|
||||
# uv builds the throwaway venv the pytest job runs in, and unpacks the PyPI
|
||||
# wheels for ruff, yamllint and pip-audit.
|
||||
UV_VERSION="0.12.17"
|
||||
# node runs `npm ci` for the frontend tests and the npm audit, and it is the one
|
||||
# pin here that does NOT come from the runner: the runner's system node is a
|
||||
# rolling Arch package (it was node 26 with no npm at all when this was pinned),
|
||||
# and the panel image is node:22-alpine. Pinned to the image's major on purpose,
|
||||
# so the tree that gets tested is the tree that gets built. Renovate keeps this
|
||||
# in step with the Dockerfile's node: tag via the "node runtime" group.
|
||||
NODE_VERSION="22.23.3"
|
||||
+7
-31
@@ -5,12 +5,12 @@ sync.ffs_lock
|
||||
# Copyparty
|
||||
*.hist/
|
||||
|
||||
# Volumes, configs and data directories
|
||||
# Volumes and data directories
|
||||
gitea/gitea-db/
|
||||
gitea/gitea-data/*
|
||||
n8n/n8n-data/*
|
||||
n8n/n8n-node-data/*
|
||||
adguardhome/conf/*
|
||||
adguardhome/data/*
|
||||
dockmon/data/*
|
||||
portainer/portainer_data/*
|
||||
metube/MeTube_downloads
|
||||
@@ -21,10 +21,6 @@ checkmk/checkmk/*
|
||||
downtify/Downtify_downloads
|
||||
headscale/config/*
|
||||
headscale/data/*
|
||||
# NetBird local hostnames and generated secrets
|
||||
netbird/.env
|
||||
netbird/secrets/
|
||||
searxng/core-config/*
|
||||
|
||||
# Steaming services files
|
||||
streaming/jellyfin/*
|
||||
@@ -35,28 +31,23 @@ streaming/data/*
|
||||
streaming/qbittorrent/*
|
||||
streaming/prowlarr/*
|
||||
|
||||
# Homepage
|
||||
# Homepage
|
||||
homepages/forust_files/assets/images/team/*
|
||||
homepages/forust_files/.well-known/*
|
||||
|
||||
# Traefik files
|
||||
traefik/letsencrypt/acme.json
|
||||
traefik/dynamic/fileservers.yml
|
||||
traefik/dynamic/*.local.y*ml.*
|
||||
traefik/dynamic/*.external.y*ml
|
||||
traefik/k8s/fileservers.y*ml
|
||||
traefik/k8s/aliasHeadersStrategy.md
|
||||
|
||||
traefik/logs/*
|
||||
|
||||
# SSL Certificates
|
||||
adguardhome/certs/*
|
||||
traefik/certs/*
|
||||
certs/
|
||||
|
||||
# Monitoring
|
||||
monitoring/prometheus.yml
|
||||
|
||||
# Python
|
||||
# Python
|
||||
.python-version
|
||||
venv/
|
||||
pyc
|
||||
@@ -86,8 +77,6 @@ replacements.txt
|
||||
|
||||
# Git
|
||||
.gitattributes
|
||||
# Gitea/github Runners
|
||||
.runner
|
||||
|
||||
# Misc
|
||||
.DS_Store
|
||||
@@ -96,23 +85,10 @@ replacements.txt
|
||||
# Temp files
|
||||
edu_master/temp/
|
||||
temp/*
|
||||
# Local-only tooling scratch space (pinned CI tools, verification scripts)
|
||||
tmp/
|
||||
|
||||
# Environment
|
||||
# Environment
|
||||
.env
|
||||
.env.anna
|
||||
.env.forust
|
||||
.env.*
|
||||
!*example
|
||||
|
||||
# kubernetes
|
||||
*/k8s/*secret*
|
||||
!*/k8s/*secret*.example
|
||||
**/k8s/*secret*
|
||||
!**/k8s/*secret*.example
|
||||
# Local-only tweaks, not for upstream
|
||||
prometheus-stack/k8s/grafana-values.yaml
|
||||
traefik/k8s/local-tls.yaml
|
||||
converters/k8s/config.yaml
|
||||
convertx/k8s/config.yaml
|
||||
!*example
|
||||
@@ -1,10 +0,0 @@
|
||||
ignored:
|
||||
- DL3008
|
||||
- DL3042
|
||||
- DL3018
|
||||
- DL3059
|
||||
trustedRegistries:
|
||||
- docker.io
|
||||
- ghcr.io
|
||||
- quay.io
|
||||
- gcr.forust.xyz
|
||||
@@ -1,8 +0,0 @@
|
||||
{
|
||||
"default": true,
|
||||
"MD013": false,
|
||||
"MD024": false,
|
||||
"MD033": false,
|
||||
"MD041": false,
|
||||
"MD046": false
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
bracketSameLine: true
|
||||
htmlWhitespaceSensitivity: css
|
||||
printWidth: 120
|
||||
tabWidth: 2
|
||||
trailingComma: all
|
||||
proseWrap: preserve
|
||||
endOfLine: lf
|
||||
@@ -1,22 +0,0 @@
|
||||
extends: default
|
||||
|
||||
rules:
|
||||
comments:
|
||||
min-spaces-from-content: 1
|
||||
comments-indentation: false
|
||||
document-start: disable
|
||||
line-length: disable
|
||||
braces:
|
||||
min-spaces-inside: 0
|
||||
max-spaces-inside: 1
|
||||
brackets:
|
||||
min-spaces-inside: 0
|
||||
max-spaces-inside: 1
|
||||
indentation:
|
||||
spaces: 2
|
||||
indent-sequences: consistent
|
||||
truthy:
|
||||
allowed-values:
|
||||
- "true"
|
||||
- "false"
|
||||
- "on"
|
||||
@@ -1,40 +0,0 @@
|
||||
{
|
||||
"tab_size": 2,
|
||||
"soft_wrap": "prefer_line",
|
||||
"preferred_line_length": 120,
|
||||
"format_on_save": "on",
|
||||
"languages": {
|
||||
"YAML": {
|
||||
"tab_size": 2,
|
||||
"hard_tabs": false,
|
||||
"format_on_save": "on",
|
||||
"formatter": {
|
||||
"language_server": { "name": "yaml-language-server" },
|
||||
},
|
||||
},
|
||||
"Python": {
|
||||
"tab_size": 4,
|
||||
"format_on_save": "on",
|
||||
"language_servers": ["pyright", "ruff"],
|
||||
"formatter": {
|
||||
"language_server": { "name": "ruff" },
|
||||
},
|
||||
},
|
||||
},
|
||||
"lsp": {
|
||||
"yaml-language-server": {
|
||||
"settings": {
|
||||
"yaml": {
|
||||
"schemas": {
|
||||
"kubernetes": ["**/k8s/*.yaml", "**/k8s/*.yml"],
|
||||
},
|
||||
"validate": true,
|
||||
"completion": true,
|
||||
"format": {
|
||||
"enable": true,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
+14
-12
@@ -1,6 +1,6 @@
|
||||
services:
|
||||
adguard:
|
||||
image: adguard/adguardhome:v0.107.79
|
||||
image: adguard/adguardhome:latest
|
||||
container_name: adguardhome
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
@@ -11,38 +11,40 @@ services:
|
||||
# - "68:68/tcp" # DHCP
|
||||
# - "3000:3000/tcp"
|
||||
volumes:
|
||||
- data:/opt/adguardhome/work
|
||||
- ./conf:/opt/adguardhome/conf
|
||||
- ./data/work:/opt/adguardhome/work
|
||||
- ./data/conf:/opt/adguardhome/conf
|
||||
- ./certs:/certs:ro
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.docker.network=proxy"
|
||||
- "traefik.http.services.adguard.loadbalancer.server.port=3000"
|
||||
|
||||
|
||||
# Prod Router
|
||||
- "traefik.http.routers.adguard.rule=Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)"
|
||||
- "traefik.http.routers.adguard.entrypoints=websecure"
|
||||
- "traefik.http.routers.adguard.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.adguard.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.adguard.tls=true"
|
||||
# Local Router
|
||||
- "traefik.http.routers.adguard-local.rule=Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`)"
|
||||
- "traefik.http.routers.adguard-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.adguard-local.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.adguard-local.tls=true"
|
||||
# Dev Router
|
||||
- "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)"
|
||||
- "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`) Host(`dns.gigaforust.internal`)"
|
||||
- "traefik.http.routers.adguard-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.adguard-dev.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.adguard-dev.tls=true"
|
||||
# DoH Router
|
||||
- "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz` || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`))"
|
||||
- "traefik.http.routers.dns-over-https.entrypoints=websecure"
|
||||
- "traefik.http.routers.dns-over-https.tls.certresolver=letsencrypt"
|
||||
|
||||
- "traefik.http.routers.dns.rule=(Host(`dns.forust.xyz`) && PathPrefix(`/dns-query`))"
|
||||
- "traefik.http.routers.dns.entrypoints=websecure"
|
||||
- "traefik.http.routers.dns.tls.certresolver=letsencrypt"
|
||||
|
||||
# Glance Metadata
|
||||
- glance.name=adguard
|
||||
- glance.url=https://adguard.forust.xyz/
|
||||
- glance.description=AdGuard Home is a network-wide software for blocking ads.
|
||||
networks:
|
||||
- proxy
|
||||
volumes:
|
||||
data:
|
||||
networks:
|
||||
proxy:
|
||||
external: true
|
||||
Whitespace-only changes.
@@ -1,118 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: adguard-lb-service
|
||||
namespace: adguard
|
||||
annotations:
|
||||
metallb.io/loadBalancerIPs: "192.168.80.3"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
selector:
|
||||
app: adguard
|
||||
ports:
|
||||
- name: dns-udp
|
||||
port: 53
|
||||
targetPort: 53
|
||||
protocol: UDP
|
||||
- name: dns-tcp
|
||||
port: 53
|
||||
targetPort: 53
|
||||
protocol: TCP
|
||||
- name: dot
|
||||
port: 853
|
||||
targetPort: 853
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: adguard-service
|
||||
namespace: adguard
|
||||
spec:
|
||||
selector:
|
||||
app: adguard
|
||||
ports:
|
||||
- port: 3000
|
||||
name: webui
|
||||
targetPort: 3000
|
||||
- port: 53
|
||||
name: dns
|
||||
targetPort: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
name: dns-tcp
|
||||
targetPort: 53
|
||||
protocol: TCP
|
||||
- port: 853
|
||||
name: dot
|
||||
targetPort: 853
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: adguard-deployment
|
||||
namespace: adguard
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: adguard
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: adguard
|
||||
annotations:
|
||||
reloader.stakater.com/auto: "true"
|
||||
spec:
|
||||
containers:
|
||||
- name: adguard
|
||||
image: adguard/adguardhome:v0.107.79
|
||||
resources:
|
||||
limits:
|
||||
memory: "1.5Gi"
|
||||
cpu: "300m"
|
||||
requests:
|
||||
memory: "500Mi"
|
||||
cpu: "50m"
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
name: webui
|
||||
- containerPort: 53
|
||||
name: dns
|
||||
- containerPort: 853
|
||||
name: dot
|
||||
volumeMounts:
|
||||
- name: adguard-data
|
||||
mountPath: /opt/adguardhome/work
|
||||
subPath: work
|
||||
- name: adguard-data
|
||||
mountPath: /opt/adguardhome/conf
|
||||
subPath: conf
|
||||
- name: adguard-certs
|
||||
mountPath: /certs
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: adguard-data
|
||||
persistentVolumeClaim:
|
||||
claimName: adguard-pvc
|
||||
- name: adguard-certs
|
||||
secret:
|
||||
secretName: adguard-certs
|
||||
items:
|
||||
- key: tls.crt
|
||||
path: fullchain.pem
|
||||
- key: tls.key
|
||||
path: privkey.pem
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: adguard-pvc
|
||||
namespace: adguard
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
@@ -1,12 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: adguard-certs
|
||||
namespace: adguard
|
||||
spec:
|
||||
secretName: adguard-certs
|
||||
dnsNames:
|
||||
- dns.forust.xyz
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
@@ -1,46 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: adguard-prod
|
||||
namespace: adguard
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`dns.forust.xyz`)
|
||||
kind: Rule
|
||||
middlewares:
|
||||
- name: crowdsec-bouncer
|
||||
namespace: crowdsec
|
||||
services:
|
||||
- name: adguard-service
|
||||
port: 3000
|
||||
- match: (Host(`dns.forust.xyz`)) && PathPrefix(`/dns-query`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: adguard-service
|
||||
port: 3000
|
||||
tls:
|
||||
secretName: adguard-certs
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: adguard-local
|
||||
namespace: adguard
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: adguard-service
|
||||
port: 3000
|
||||
- match: (Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)) && PathPrefix(`/dns-query`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: adguard-service
|
||||
port: 3000
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
@@ -1,15 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: internal-wildcard-tls
|
||||
namespace: adguard
|
||||
spec:
|
||||
secretName: internal-wildcard-tls
|
||||
dnsNames:
|
||||
- "*.workstation.internal"
|
||||
- "*.gigaforust.internal"
|
||||
- workstation.internal
|
||||
- gigaforust.internal
|
||||
issuerRef:
|
||||
name: internal-ca
|
||||
kind: ClusterIssuer
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: adguard
|
||||
@@ -1,10 +0,0 @@
|
||||
kubectl apply -f k8s/namespace.yaml && \
|
||||
kubectl create secret tls adguard-certs -n adguard \
|
||||
--cert=certs/fullchain.pem \
|
||||
--key=certs/privkey.pem --dry-run=client -o yaml > \
|
||||
k8s/secrets.yaml
|
||||
|
||||
# OR WITH NO FILE CREATION:
|
||||
kubectl create secret tls adguard-certs -n adguard \
|
||||
--cert=certs/fullchain.pem --key=certs/privkey.pem \
|
||||
--save-config
|
||||
@@ -1,6 +1,6 @@
|
||||
services:
|
||||
postgresql:
|
||||
image: docker.io/library/postgres:15.19-alpine
|
||||
image: docker.io/library/postgres:15-alpine
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
@@ -42,20 +42,26 @@ services:
|
||||
- ./custom-templates:/templates
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.docker.network=proxy"
|
||||
- "traefik.http.services.authentik-server.loadbalancer.server.port=9000"
|
||||
|
||||
# Prod Router
|
||||
- "traefik.http.routers.authentik-server.rule=Host(`auth.forust.xyz`)"
|
||||
- "traefik.http.routers.authentik-server.entrypoints=websecure"
|
||||
- "traefik.http.routers.authentik-server.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.authentik-server.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.authentik-server.service=authentik-server"
|
||||
- "traefik.http.routers.authentik-server.tls=true"
|
||||
# Local Router
|
||||
- "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`)"
|
||||
- "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`) || Host(`auth-dashboard.internal`)"
|
||||
- "traefik.http.routers.authentik-server-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.authentik-server-local.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.authentik-server-local.service=authentik-server"
|
||||
- "traefik.http.routers.authentik-server-local.tls=true"
|
||||
# Dev Router
|
||||
- "traefik.http.routers.authentik-server-dev.rule=Host(`auth.gigaforust.internal`)"
|
||||
- "traefik.http.routers.authentik-server-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.authentik-server-dev.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.authentik-server-dev.service=authentik-server"
|
||||
- "traefik.http.routers.authentik-server-dev.tls=true"
|
||||
networks:
|
||||
- proxy
|
||||
|
||||
Whitespace-only changes.
@@ -1,93 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: authentik-server-service
|
||||
namespace: authentik
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: authentik-server
|
||||
ports:
|
||||
- port: 9000
|
||||
targetPort: 9000
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: authentik-worker-service
|
||||
namespace: authentik
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: authentik-worker
|
||||
ports:
|
||||
- port: 9000
|
||||
targetPort: 9000
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: authentik-server-deployment
|
||||
namespace: authentik
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: authentik-server
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: authentik-server
|
||||
spec:
|
||||
containers:
|
||||
- name: authentik-server
|
||||
image: ghcr.io/goauthentik/server:2026.8.3
|
||||
args: ["server"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: authentik-config
|
||||
- secretRef:
|
||||
name: authentik-secrets
|
||||
ports:
|
||||
- containerPort: 9000
|
||||
resources:
|
||||
requests:
|
||||
memory: "700Mi"
|
||||
cpu: "300m"
|
||||
limits:
|
||||
memory: "1.5Gi"
|
||||
cpu: "1000m"
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: authentik-worker-deployment
|
||||
namespace: authentik
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: authentik-worker
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: authentik-worker
|
||||
spec:
|
||||
containers:
|
||||
- name: authentik-worker
|
||||
image: ghcr.io/goauthentik/server:2026.8.3
|
||||
args: ["worker"]
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: authentik-config
|
||||
- secretRef:
|
||||
name: authentik-secrets
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "300m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "700m"
|
||||
@@ -1,28 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: authentik-prod-tls
|
||||
namespace: authentik
|
||||
spec:
|
||||
secretName: authentik-prod-tls
|
||||
dnsNames:
|
||||
- auth.forust.xyz
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: internal-wildcard-tls
|
||||
namespace: authentik
|
||||
spec:
|
||||
secretName: internal-wildcard-tls
|
||||
dnsNames:
|
||||
- "*.workstation.internal"
|
||||
- "*.gigaforust.internal"
|
||||
- workstation.internal
|
||||
- gigaforust.internal
|
||||
issuerRef:
|
||||
name: internal-ca
|
||||
kind: ClusterIssuer
|
||||
@@ -1,11 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: authentik-config
|
||||
namespace: authentik
|
||||
data:
|
||||
AUTHENTIK_IMAGE: ghcr.io/goauthentik/server
|
||||
AUTHENTIK_TAG: "2025.10.2"
|
||||
AUTHENTIK_POSTGRESQL__HOST: postgres.database.svc.cluster.local
|
||||
AUTHENTIK_POSTGRESQL__NAME: authentik
|
||||
AUTHENTIK_ERROR_REPORTING__ENABLED: "true"
|
||||
@@ -1,36 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: authentik-prod
|
||||
namespace: authentik
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`auth.forust.xyz`)
|
||||
kind: Rule
|
||||
middlewares:
|
||||
- name: crowdsec-bouncer
|
||||
namespace: crowdsec
|
||||
services:
|
||||
- name: authentik-server-service
|
||||
port: 9000
|
||||
tls:
|
||||
secretName: authentik-prod-tls
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: authentik-local
|
||||
namespace: authentik
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`auth.workstation.internal`) || Host(`auth.gigaforust.internal`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: authentik-server-service
|
||||
port: 9000
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: authentik
|
||||
@@ -1,11 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: authentik-secrets
|
||||
namespace: authentik
|
||||
type: Opaque
|
||||
stringData:
|
||||
AUTHENTIK_SECRET_KEY: ""
|
||||
AUTHENTIK_POSTGRESQL__PASSWORD: ""
|
||||
AUTHENTIK_POSTGRESQL__USER: authentik
|
||||
AUTHENTIK_BOOTSTRAP_PASSWORD: authentik
|
||||
@@ -1,9 +0,0 @@
|
||||
crds:
|
||||
enabled: true
|
||||
prometheus:
|
||||
servicemonitor:
|
||||
enabled: true
|
||||
interval: 60s
|
||||
scrapeTimeout: 30s
|
||||
labels:
|
||||
release: prometheus-stack
|
||||
@@ -1,29 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-staging
|
||||
spec:
|
||||
acme:
|
||||
email: bobrovod@national.shitposting.agency
|
||||
server: https://acme-staging-v02.api.letsencrypt.org/directory
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-staging-account-key
|
||||
solvers:
|
||||
- http01:
|
||||
ingress:
|
||||
class: traefik
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-prod
|
||||
spec:
|
||||
acme:
|
||||
email: bobrovod@national.shitposting.agency
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-prod-account-key
|
||||
solvers:
|
||||
- http01:
|
||||
ingress:
|
||||
class: traefik
|
||||
@@ -1,30 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIFFjCCAv6gAwIBAgIUetKpTfEDOn2985FFMu6G26itT+wwDQYJKoZIhvcNAQEN
|
||||
BQAwIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRlcm5hbCByb290IENBMB4XDTI2MDky
|
||||
MzEyNDA0N1oXDTM2MDkyMDEyNDA0N1owIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRl
|
||||
cm5hbCByb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmNP
|
||||
ZCOoD8NtNuYJKVXBlTPjX7D7sJCSK5neH7ZbYV5+lmUlEErY8Mik7j37V5k5NfpF
|
||||
Ig85pOjP7RckTPz5V6ek3yaN40s4AL053sN5ZPauDVYjalaEHTgj5sEMqlLACQWI
|
||||
yZmJOZspZykae8dIpQnqCoFpRT4FurJ78v4a0ylnFVLMQn/lyCHedwTjkEdtYWYr
|
||||
ccJy8vQwqkzs/rWvEH1lDqZhennLOrmcCjfonG7D/pruMn4z+6E28p4+ejkRrI6x
|
||||
luak3KnpT1XMeHtgU21hiRGaMDBchHMFgAhnY1qosymKenXvfTZItwgjZbwa1hJI
|
||||
GAiDm+jQDKMjzRZ3rH6Xfc0auUcykNz73PpNu1NGm78nndXwCXcXn1LFKNQJ+r1U
|
||||
sJiyAmUZmXVn4aM4OMf2F38k7wTYIKg7nRGaUkNeKDlNkjA4HvgWw+jwO1KmdHQ/
|
||||
mOem1rosDWHRK01wg+Gga9mQCnhNhxglg3t/UeSic6uOaRsvaz4qkzHq8MbCujVz
|
||||
DpKQjqdikYOAXZOs4KlBLWrS7NaK4NzfSD02pBUErh54ruJfY/bWz9KyXzBD/lQZ
|
||||
VUTKyvUVB0bkVHEdf1jJmX3H4IZRQSF5JPqOBotW6bJI5fEGNBvj9Zxy4nm2WWGz
|
||||
yyP3uWsQz8U/Wdx9nXZLHInTZBsvgLYtKUAWA30CAwEAAaNCMEAwDgYDVR0PAQH/
|
||||
BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEkKm2rxPaK6+O9WD80z
|
||||
BLC6F9QsMA0GCSqGSIb3DQEBDQUAA4ICAQAnFyHz97Umf5VIu+dKTJid7C73VugJ
|
||||
TIar/xJBs/4CxP+znBxhJjXygRoyIfzoVGWcB2ZSL//vL78Qlts79K/Imc9a4RFF
|
||||
wMvCxsRXAEQ4TpeWi3ophPNcs4rhsP+gQKQFtnyKP9519bqpfxp0bTqwOV2o18fn
|
||||
za7rlQViiEnNV58j7CVoM9+mJvVVfBEX1Km+GyJL9GadzbIQ7FxClVJZefCbft93
|
||||
zHVk9gDOw8ys1XGSR2OUCyCLinXO6mqS16CmBb2MAKXq/YyH7E0N8iotAPGtfA8V
|
||||
M/0ddy947rY0xCrtECfWwvGQpJS7NRv/Z9b2jCfXrI5LXmL2nfQRg0y9GE4Vjwr+
|
||||
WxtGU5jOeFt0jQ+xRzcgG0Op+qK3x55l5LSo2hOcOVYbiHxcHEJFgwNi1ADeBFwb
|
||||
q/HdysfURSOghqjIpMMAUabBp+DBUg2EUF7pIaUqbdqExFYcr9EYisEMiNsmKmN+
|
||||
8ZbcOeerFKDQj+t/R0bFXa7UBn2UWsjI8zlR74aa2kLDXwtyz/XlO/FlYm66eBFo
|
||||
2/eYUSeU+S4ej+wUAs/dvjF7f190/DUQGuwOTlLTahqWDztmhCk7qzbECu56CwKT
|
||||
E5Ect2P72UleYwdblkVOVd352AmiwEzdOaziIRrPh8uenEknH6JBYPO3mjk7cCg+
|
||||
GPGcNIBctjdXhg==
|
||||
-----END CERTIFICATE-----
|
||||
@@ -1,33 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: selfsigned
|
||||
spec:
|
||||
selfSigned: {}
|
||||
---
|
||||
# Homelab internal root CA (10y). Install the .crt on clients (see below).
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: internal-ca-root
|
||||
namespace: cert-manager
|
||||
spec:
|
||||
isCA: true
|
||||
commonName: homelab internal root CA
|
||||
duration: 87600h
|
||||
renewBefore: 7200h
|
||||
secretName: internal-ca-root
|
||||
privateKey:
|
||||
algorithm: RSA
|
||||
size: 4096
|
||||
issuerRef:
|
||||
name: selfsigned
|
||||
kind: ClusterIssuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: internal-ca
|
||||
spec:
|
||||
ca:
|
||||
secretName: internal-ca-root
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: cert-manager
|
||||
@@ -1,15 +0,0 @@
|
||||
CLOUDFLARE_API_TOKEN=YOUR_CLOUDFLARE_API_TOKEN
|
||||
DOMAINS=example.com,dns.example.com,mc.example.com,auth.example.com,ssh.example.com
|
||||
IP4_DOMAINS=
|
||||
IP6_DOMAINS=
|
||||
IP4_PROVIDER=cloudflare.trace
|
||||
IP6_PROVIDER=none # change if you want to update AAAA
|
||||
UPDATE_CRON=@every 5m
|
||||
UPDATE_ON_START=true
|
||||
DELETE_ON_STOP=false
|
||||
DELETE_ON_FAILURE=true
|
||||
TTL=1
|
||||
PROXIED=!is(dns.example.com) && !is(mc.example.com) && !is(ssh.example.com)
|
||||
EMOJI=true
|
||||
UPTIMEKUMA=https://uptime-kuma.example.com/api/push/AsaSDFGFkfklaFALSKffkfFKfkfkfkFK?status=up&msg=OK&ping=
|
||||
REJECT_CLOUDFLARE_IPS=true
|
||||
+6
-23
@@ -1,30 +1,13 @@
|
||||
services:
|
||||
cloudflare-ddns:
|
||||
image: timothyjmiller/cloudflare-ddns:2.2.0
|
||||
image: timothyjmiller/cloudflare-ddns:latest
|
||||
container_name: cloudflare-ddns
|
||||
restart: unless-stopped
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
network_mode: "host"
|
||||
# https://github.com/timothymiller/cloudflare-ddns#-quick-start
|
||||
network_mode: 'host'
|
||||
environment:
|
||||
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN:?Cloudflare API token is required}
|
||||
- DOMAINS=${DOMAINS:-}
|
||||
- IP4_DOMAINS=${IP4_DOMAINS:-}
|
||||
- IP6_DOMAINS=${IP6_DOMAINS:-}
|
||||
- IP4_PROVIDER=${IP4_PROVIDER:-cloudflare.trace}
|
||||
- IP6_PROVIDER=${IP6_PROVIDER:-none}
|
||||
- UPDATE_CRON=${UPDATE_CRON:-@every 5m}
|
||||
- UPDATE_ON_START=${UPDATE_ON_START:-true}
|
||||
- DELETE_ON_STOP=${DELETE_ON_STOP:-false}
|
||||
- DELETE_ON_FAILURE=${DELETE_ON_FAILURE:-true}
|
||||
- TTL=${TTL:-1} # 1=auto
|
||||
# to proxy only "dns.example.com" and "wfs.example.com" use "!is(dns.domain.com) && !is (wfs.domain.com)"
|
||||
- PROXIED=${PROXIED:-true}
|
||||
- EMOJI=${EMOJI:-true}
|
||||
- UPTIMEKUMA=${UPTIMEKUMA:-}
|
||||
- HEALTHCHECKS=${HEALTHCHECKS:-}
|
||||
- REJECT_CLOUDFLARE_IPS=${REJECT_CLOUDFLARE_IPS:-true}
|
||||
# volumes:
|
||||
# Prefer using environment variables for configuration, config.json legacy support
|
||||
# - ./config.json:/config.json
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
volumes:
|
||||
- ./config.json:/config.json
|
||||
@@ -7,7 +7,7 @@
|
||||
"api_key": {
|
||||
"api_key": "api_key_here",
|
||||
"account_email": "your_email_here"
|
||||
},
|
||||
}
|
||||
"zone_id": "your_zone-id",
|
||||
"subdomains": [
|
||||
{ "name": "", "proxied": true },
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
secret.yaml
|
||||
Whitespace-only changes.
@@ -1,32 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cfddns
|
||||
labels:
|
||||
app: cfddns
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: cfddns
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cfddns
|
||||
spec:
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
containers:
|
||||
- name: cloudflare-ddns
|
||||
image: timothyjmiller/cloudflare-ddns:2.2.0
|
||||
imagePullPolicy: Always
|
||||
resources:
|
||||
requests:
|
||||
memory: "20Mi"
|
||||
cpu: "30m"
|
||||
limits:
|
||||
memory: "64Mi"
|
||||
cpu: "50m"
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: cfddns-secrets
|
||||
@@ -1,18 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cfddns-secrets
|
||||
type: Opaque
|
||||
stringData:
|
||||
CLOUDFLARE_API_TOKEN: your_token
|
||||
DOMAINS: "example.com,www.example.com"
|
||||
IP4_PROVIDER: cloudflare.trace
|
||||
IP6_PROVIDER: none
|
||||
UPDATE_CRON: "@every 5m"
|
||||
UPDATE_ON_START: "true"
|
||||
DELETE_ON_STOP: "false"
|
||||
DELETE_ON_FAILURE: "true"
|
||||
TTL: "1"
|
||||
PROXIED: "true"
|
||||
EMOJI: "true"
|
||||
REJECT_CLOUDFLARE_IPS: "true"
|
||||
@@ -1,6 +1,6 @@
|
||||
services:
|
||||
checkmk:
|
||||
image: "checkmk/check-mk-raw:2.4.0-2026.09.14"
|
||||
image: "checkmk/check-mk-raw:2.4.0-latest"
|
||||
container_name: "checkmk"
|
||||
restart: unless-stopped
|
||||
# ports:
|
||||
@@ -13,21 +13,24 @@ services:
|
||||
environment:
|
||||
- CMK_PASSWORD=${CMK_PASSWORD:-password}
|
||||
- CMK_SITE_ID=cmk
|
||||
- TZ=${TZ:-Etc/UTC}
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.docker.network=proxy"
|
||||
- "traefik.http.services.checkmk.loadbalancer.server.port=5000"
|
||||
|
||||
# Prod Router
|
||||
- "traefik.http.routers.checkmk.rule=Host(`cmk.forust.xyz`)"
|
||||
- "traefik.http.routers.checkmk.entrypoints=websecure"
|
||||
- "traefik.http.routers.checkmk.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.checkmk.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.checkmk.tls=true"
|
||||
# Local Router
|
||||
- "traefik.http.routers.checkmk-local.rule=Host(`cmk.workstation.internal`)"
|
||||
- "traefik.http.routers.checkmk-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.checkmk-local.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.checkmk-local.tls=true"
|
||||
# Dev Router
|
||||
- "traefik.http.routers.checkmk-dev.rule=Host(`cmk.gigaforust.internal`)"
|
||||
- "traefik.http.routers.checkmk-dev.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.checkmk-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.checkmk-dev.tls=true"
|
||||
networks:
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: checkmk-prod-tls
|
||||
namespace: checkmk
|
||||
spec:
|
||||
secretName: checkmk-prod-tls
|
||||
dnsNames:
|
||||
- cmk.forust.xyz
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: internal-wildcard-tls
|
||||
namespace: checkmk
|
||||
spec:
|
||||
secretName: internal-wildcard-tls
|
||||
dnsNames:
|
||||
- "*.workstation.internal"
|
||||
- "*.gigaforust.internal"
|
||||
- workstation.internal
|
||||
- gigaforust.internal
|
||||
issuerRef:
|
||||
name: internal-ca
|
||||
kind: ClusterIssuer
|
||||
@@ -1,75 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: checkmk-service
|
||||
namespace: checkmk
|
||||
spec:
|
||||
selector:
|
||||
app: checkmk
|
||||
ports:
|
||||
- name: web
|
||||
port: 5000
|
||||
targetPort: 5000
|
||||
- name: agent-receiver
|
||||
port: 8000
|
||||
targetPort: 8000
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: checkmk-deployment
|
||||
namespace: checkmk
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: checkmk
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: checkmk
|
||||
spec:
|
||||
containers:
|
||||
- name: checkmk
|
||||
image: checkmk/check-mk-raw:2.4.0-2026.09.14
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: checkmk-secrets
|
||||
- configMapRef:
|
||||
name: checkmk-config
|
||||
ports:
|
||||
- name: web
|
||||
containerPort: 5000
|
||||
- name: agent-receiver
|
||||
containerPort: 8000
|
||||
volumeMounts:
|
||||
- name: sites
|
||||
mountPath: /omd/sites
|
||||
- name: tmp
|
||||
mountPath: /opt/omd/sites/cmk/tmp
|
||||
resources:
|
||||
requests:
|
||||
memory: "2Gi"
|
||||
cpu: "600m"
|
||||
limits:
|
||||
memory: "5Gi"
|
||||
cpu: "4"
|
||||
volumes:
|
||||
- name: sites
|
||||
persistentVolumeClaim:
|
||||
claimName: checkmk-sites-pvc
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: checkmk-sites-pvc
|
||||
namespace: checkmk
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
@@ -1,8 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: checkmk-config
|
||||
namespace: checkmk
|
||||
data:
|
||||
TZ: Europe/Bratislava
|
||||
CMK_SITE_ID: cmk
|
||||
@@ -1,52 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: checkmk-prod
|
||||
namespace: checkmk
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`cmk.forust.xyz`)
|
||||
kind: Rule
|
||||
middlewares:
|
||||
- name: crowdsec-bouncer
|
||||
namespace: crowdsec
|
||||
services:
|
||||
- name: checkmk-service
|
||||
port: 5000
|
||||
tls:
|
||||
secretName: checkmk-prod-tls
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRouteTCP
|
||||
metadata:
|
||||
name: checkmk-agent-receiver
|
||||
namespace: checkmk
|
||||
spec:
|
||||
entryPoints:
|
||||
- checkmk-agent
|
||||
routes:
|
||||
- match: HostSNI(`*`)
|
||||
services:
|
||||
- name: checkmk-service
|
||||
port: 8000
|
||||
tls:
|
||||
passthrough: true
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: checkmk-local
|
||||
namespace: checkmk
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`cmk.workstation.internal`) || Host(`cmk.gigaforust.internal`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: checkmk-service
|
||||
port: 5000
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: checkmk
|
||||
@@ -1,8 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: checkmk-secrets
|
||||
namespace: checkmk
|
||||
type: Opaque
|
||||
stringData:
|
||||
CMK_PASSWORD: "password"
|
||||
@@ -1 +0,0 @@
|
||||
secret.yaml
|
||||
Whitespace-only changes.
@@ -1,37 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cloudflared
|
||||
labels:
|
||||
app: cloudflared
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: cloudflared
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cloudflared
|
||||
spec:
|
||||
containers:
|
||||
- name: cloudflared
|
||||
image: cloudflare/cloudflared:2026.9.3
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- tunnel
|
||||
- --no-autoupdate
|
||||
- run
|
||||
env:
|
||||
- name: TUNNEL_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: cloudflared-secrets
|
||||
key: TUNNEL_TOKEN
|
||||
resources:
|
||||
requests:
|
||||
memory: "32Mi"
|
||||
cpu: "30m"
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
cpu: "200m"
|
||||
@@ -1,7 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cloudflared-secrets
|
||||
type: Opaque
|
||||
stringData:
|
||||
TUNNEL_TOKEN: your_tunnel_token_here
|
||||
@@ -1,9 +0,0 @@
|
||||
ACCOUNT_REGISTRATION=false
|
||||
HTTP_ALLOWED=false
|
||||
ALLOW_UNAUTHENTICAED=false
|
||||
AUTO_DELETE_EVERY_N_HOURS=24
|
||||
WEBROOT=/convert
|
||||
HIDE_HISTORY=false
|
||||
LANGUAGE=en
|
||||
UNAUTHED_USER_SHARING=false
|
||||
MAX_CONVERT_PROCESS=0
|
||||
@@ -1,70 +0,0 @@
|
||||
services:
|
||||
convertx:
|
||||
container_name: convertx
|
||||
image: ghcr.io/c4illin/convertx:v0.18.0
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9992:3000"
|
||||
# https://github.com/C4illin/ConvertX#environment-variables
|
||||
environment:
|
||||
- JWT_SECRET=$(JWT_SECRET)
|
||||
- ACCOUNT_REGISTRATION=$(ACCOUNT_REGISTRATION:-false)
|
||||
- HTTP_ALLOWED=$(HTTP_ALLOWED:-false)
|
||||
- ALLOW_UNAUTHENTICATED=$(ALLOW_UNAUTHENTICATED:-false)
|
||||
- AUTO_DELETE_EVERY_N_HOURS=$(AUTO_DELETE_EVERY_N_HOURS:-24)
|
||||
- WEBROOT=$(WEBROOT)
|
||||
- HIDE_HISTORY=$(HIDE_HISTORY:-false)
|
||||
- LANGUAGE=$(LANGUAGE:-en)
|
||||
- UNAUTHENTICATED_USER_SHARING=$(UNAUTHENTICATED_USER_SHARING:-false)
|
||||
- MAX_CONVERT_PROCESS=$(MAX_CONVERT_PROCESS:-0)
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.services.convertx.loadbalancer.server.port=3000"
|
||||
# Prod Router
|
||||
- "traefik.http.routers.convertx.rule=(Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)"
|
||||
- "traefik.http.routers.convertx.entrypoints=websecure"
|
||||
- "traefik.http.routers.convertx.priority=50"
|
||||
- "traefik.http.routers.convertx.tls.certresolver=letsencrypt"
|
||||
# Local Router
|
||||
- "traefik.http.routers.convertx-local.rule=Host(`workstation.internal`) && PathPrefix(`/convert`)"
|
||||
- "traefik.http.routers.convertx-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.convertx-local.priority=50"
|
||||
- "traefik.http.routers.convertx-local.tls=true"
|
||||
# Dev Router
|
||||
- "traefik.http.routers.convertx-dev.rule=Host(`gigaforust.internal`) && PathPrefix(`/convert`)"
|
||||
- "traefik.http.routers.convertx-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.convertx-dev.priority=50"
|
||||
- "traefik.http.routers.convertx-dev.tls=true"
|
||||
networks:
|
||||
- proxy
|
||||
volumes:
|
||||
- data:/app/data
|
||||
|
||||
bentopdf:
|
||||
container_name: bentopdf
|
||||
image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
|
||||
restart: unless-stopped
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.services.bentopdf.loadbalancer.server.port=8080"
|
||||
|
||||
# Prod router
|
||||
- "traefik.http.routers.bentopdf.rule=Host(`pdf.forust.xyz`)"
|
||||
- "traefik.http.routers.bentopdf.entrypoints=websecure"
|
||||
- "traefik.http.routers.bentopdf.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.routers.bentopdf.tls=true"
|
||||
# Local router
|
||||
- "traefik.http.routers.bentopdf-local.rule=Host(`pdf.workstation.internal`)"
|
||||
- "traefik.http.routers.bentopdf-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.bentopdf-local.tls=true"
|
||||
# Dev router
|
||||
- "traefik.http.routers.bentopdf-dev.rule=Host(`pdf.gigaforust.internal`)"
|
||||
- "traefik.http.routers.bentopdf-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.bentopdf-dev.tls=true"
|
||||
networks:
|
||||
- proxy
|
||||
networks:
|
||||
proxy:
|
||||
external: true
|
||||
volumes:
|
||||
data:
|
||||
Whitespace-only changes.
@@ -1,42 +0,0 @@
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: bentopdf-service
|
||||
namespace: converters
|
||||
spec:
|
||||
selector:
|
||||
app: bentopdf
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: 8080
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: bentopdf-deployment
|
||||
namespace: converters
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: bentopdf
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: bentopdf
|
||||
spec:
|
||||
containers:
|
||||
- image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
|
||||
imagePullPolicy: Always
|
||||
name: bentopdf
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
resources:
|
||||
requests:
|
||||
memory: "50Mi"
|
||||
cpu: "50m"
|
||||
ephemeral-storage: "100Mi"
|
||||
limits:
|
||||
memory: "700Mi"
|
||||
cpu: "700m"
|
||||
ephemeral-storage: "5Gi"
|
||||
@@ -1,42 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: convertx-prod-tls
|
||||
namespace: converters
|
||||
spec:
|
||||
secretName: convertx-prod-tls
|
||||
dnsNames:
|
||||
- forust.xyz
|
||||
- www.forust.xyz
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: bentopdf-prod-tls
|
||||
namespace: converters
|
||||
spec:
|
||||
secretName: bentopdf-prod-tls
|
||||
dnsNames:
|
||||
- pdf.forust.xyz
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: internal-wildcard-tls
|
||||
namespace: converters
|
||||
spec:
|
||||
secretName: internal-wildcard-tls
|
||||
dnsNames:
|
||||
- "*.workstation.internal"
|
||||
- "*.gigaforust.internal"
|
||||
- workstation.internal
|
||||
- gigaforust.internal
|
||||
issuerRef:
|
||||
name: internal-ca
|
||||
kind: ClusterIssuer
|
||||
@@ -1,16 +0,0 @@
|
||||
# test manifest with docker and k8s config keys mismatch
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: convertx-config
|
||||
namespace: converters
|
||||
data:
|
||||
ACCOUNT_REGISTRATION: "false"
|
||||
HTTP_ALLOWED: "false"
|
||||
ALLOW_UNAUTHENTICAED: "false"
|
||||
AUTO_DELETE_EVERY_N_HOURS: "24"
|
||||
WEBROOT: "/convert"
|
||||
HIDE_HISTORY: "false"
|
||||
LANGUAGE: "en"
|
||||
UNAUTHED_USER_SHARING: "false"
|
||||
MAX_CONVERT_PROCESS: "0"
|
||||
@@ -1,63 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: convertx-service
|
||||
namespace: converters
|
||||
spec:
|
||||
selector:
|
||||
app: convertx
|
||||
ports:
|
||||
- port: 3000
|
||||
targetPort: 3000
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: convertx-deployment
|
||||
namespace: converters
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: convertx
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: convertx
|
||||
spec:
|
||||
containers:
|
||||
- image: ghcr.io/c4illin/convertx:v0.18.0
|
||||
name: convertx
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: convertx-config
|
||||
- secretRef:
|
||||
name: convertx-secrets
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
resources:
|
||||
requests:
|
||||
memory: "250Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
cpu: "1500m"
|
||||
memory: "1.5Gi"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: convertx-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: convertx-pvc
|
||||
namespace: converters
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
@@ -1,70 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: convertx-prod
|
||||
namespace: converters
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: (Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)
|
||||
kind: Rule
|
||||
priority: 50
|
||||
services:
|
||||
- name: convertx-service
|
||||
port: 3000
|
||||
tls:
|
||||
secretName: convertx-prod-tls
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: convertx-local
|
||||
namespace: converters
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: (Host(`workstation.internal`) || Host(`gigaforust.internal`)) && PathPrefix(`/convert`)
|
||||
kind: Rule
|
||||
priority: 50
|
||||
services:
|
||||
- name: convertx-service
|
||||
port: 3000
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: bentopdf-prod
|
||||
namespace: converters
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`pdf.forust.xyz`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: bentopdf-service
|
||||
port: 8080
|
||||
tls:
|
||||
secretName: bentopdf-prod-tls
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: bentopdf-local
|
||||
namespace: converters
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`pdf.workstation.internal`) || Host(`pdf.gigaforust.internal`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: bentopdf-service
|
||||
port: 8080
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: converters
|
||||
@@ -1,8 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: convertx-secrets
|
||||
namespace: converters
|
||||
type: Opaque
|
||||
stringData:
|
||||
jwt-secret: ""
|
||||
@@ -1,24 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: Middleware
|
||||
metadata:
|
||||
name: crowdsec-bouncer
|
||||
namespace: crowdsec
|
||||
spec:
|
||||
plugin:
|
||||
crowdsec-bouncer:
|
||||
enabled: true
|
||||
LogLevel: INFO
|
||||
CrowdsecMode: live
|
||||
CrowdsecLapiScheme: http
|
||||
CrowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080
|
||||
CrowdsecLapiKeyFile: "/etc/traefik/secrets/traefik-api-key"
|
||||
# LAPI lookup is SYNCHRONOUS and per-request: the plugin blocks on
|
||||
# `GET /v1/decisions?ip=...&banned=true` before the request reaches
|
||||
# the backend, and fails CLOSED (403) if the lookup exceeds the
|
||||
# timeout. Unset, the fork defaults to 10s, which is an eternity for
|
||||
# a request path: a single slow LAPI (idle 1.3-7.4s here) turned
|
||||
# every request into a 10s hang and then a self-inflicted 403.
|
||||
# 2s keeps the fail-closed path fast and bounded; with the LAPI
|
||||
# resourced properly (see crowdsec-values.yaml) the lookup is
|
||||
# sub-100ms and this budget is never hit.
|
||||
CrowdsecLapiTimeout: "2s"
|
||||
@@ -1,126 +0,0 @@
|
||||
container_runtime: containerd
|
||||
|
||||
agent:
|
||||
acquisition: []
|
||||
additionalAcquisition:
|
||||
- labels:
|
||||
type: traefik
|
||||
limit: 1000
|
||||
query: |
|
||||
{namespace="traefik"}
|
||||
source: loki
|
||||
url: http://loki.prometheus.svc.cluster.local:3100/
|
||||
wait_for_ready: 30s
|
||||
env:
|
||||
- name: COLLECTIONS
|
||||
value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
|
||||
- name: DISABLE_COLLECTIONS
|
||||
value: crowdsecurity/sshd
|
||||
metrics:
|
||||
enabled: true
|
||||
serviceMonitor:
|
||||
additionalLabels:
|
||||
release: prometheus-stack
|
||||
enabled: true
|
||||
# Static machine identity: agent pods mount pre-created LAPI credentials
|
||||
# (Secret crowdsec-agent-credentials, key local_api_credentials.yaml)
|
||||
# at the exact path the agent entrypoint expects. Together with the
|
||||
# patched register-init (enforced by janitor-cronjob.yaml) the agent
|
||||
# never calls `cscli lapi register` in steady state, so pod names,
|
||||
# restarts and reboots can no longer break it.
|
||||
extraVolumes:
|
||||
- name: static-creds
|
||||
secret:
|
||||
secretName: crowdsec-agent-credentials
|
||||
items:
|
||||
- key: local_api_credentials.yaml
|
||||
path: local_api_credentials.yaml
|
||||
extraVolumeMounts:
|
||||
- name: static-creds
|
||||
mountPath: /tmp_config/local_api_credentials.yaml
|
||||
subPath: local_api_credentials.yaml
|
||||
readOnly: true
|
||||
resources:
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 500Mi
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 100Mi
|
||||
|
||||
config:
|
||||
parsers:
|
||||
s02-enrich:
|
||||
mobile-whitelist.yaml: |
|
||||
name: forust/mobile-whitelist
|
||||
description: "Whitelist SWAN/4ka mobile network"
|
||||
whitelist:
|
||||
reason: "Mobile IP whitelist"
|
||||
cidr:
|
||||
- "84.245.64.0/18"
|
||||
|
||||
postoverflows:
|
||||
s01-whitelist:
|
||||
home-dynamic-ip.yaml: |
|
||||
name: forust/home-dynamic-ip
|
||||
description: "Whitelist home dynamic IP"
|
||||
whitelist:
|
||||
reason: "Home dynamic IP"
|
||||
expression:
|
||||
- evt.Overflow.Alert.Source.IP in LookupHost("ddns.forust.xyz")
|
||||
# The hairpin-NAT address of the router (192.168.88.1) is what the
|
||||
# Gitea Actions runner presents to Traefik - it is NOT the home
|
||||
# dynamic IP, so the whitelist above did not cover it. During a
|
||||
# deploy the runner POSTs to the Actions API many times a second;
|
||||
# a single 403 storm was enough to earn it a 4h ban and break every
|
||||
# later job. Whitelisting the whole LAN also covers phones and
|
||||
# tablets browsing over 192.168.88.0/24.
|
||||
lan.yaml: |
|
||||
name: forust/lan
|
||||
description: "Whitelist local network"
|
||||
whitelist:
|
||||
reason: "Local network"
|
||||
cidr:
|
||||
- "127.0.0.0/8"
|
||||
- "10.0.0.0/8"
|
||||
- "172.16.0.0/12"
|
||||
- "192.168.0.0/16"
|
||||
|
||||
lapi:
|
||||
env:
|
||||
- name: COLLECTIONS
|
||||
value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
|
||||
- name: DISABLE_COLLECTIONS
|
||||
value: crowdsecurity/linux crowdsecurity/sshd
|
||||
metrics:
|
||||
enabled: true
|
||||
serviceMonitor:
|
||||
additionalLabels:
|
||||
release: prometheus-stack
|
||||
enabled: true
|
||||
persistentVolume:
|
||||
config:
|
||||
enabled: true
|
||||
size: 100Mi
|
||||
storageClassName: local-path-retain
|
||||
data:
|
||||
enabled: true
|
||||
size: 1Gi
|
||||
storageClassName: local-path-retain
|
||||
# LAPI answers a blocking /v1/decisions lookup for EVERY bouncer-protected
|
||||
# request (whole Traefik front door), so it is the hot path of the proxy.
|
||||
# At 400m/500Mi it went CPU-throttled and idle lookups measured 1.3-7.4s,
|
||||
# which pushed requests into the bouncer's fail-closed 403.
|
||||
# Single replica on purpose: LAPI is stateful (BoltDB on the `data` PVC,
|
||||
# credentials on the `config` PVC) - two replicas sharing those RWO
|
||||
# volumes would corrupt the decision store. Scale up CPU, not replicas.
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1500m
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 500Mi
|
||||
service:
|
||||
type: ClusterIP
|
||||
storeLAPICscliCredentialsInSecret: true
|
||||
@@ -1,32 +0,0 @@
|
||||
apiVersion: v1
|
||||
data:
|
||||
crowdsec-overview.json: "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"8.1.2\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"graph\",\n \"name\": \"Graph (old)\",\n \"version\": \"\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"stat\",\n \"name\": \"Stat\",\n \"version\": \"\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"timeseries\",\n \"name\": \"Time series\",\n \"version\": \"\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"links\": [],\n \"panels\": [\n {\n \"collapsed\": false,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 24,\n \"panels\": [],\n \"title\": \"Summary\",\n \"type\": \"row\"\n },\n {\n \"cacheTimeout\": null,\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"#E02F44\",\n \"value\": null\n },\n {\n \"color\": \"#E02F44\",\n \"value\": 10\n },\n {\n \"color\": \"#299c46\",\n \"value\": 10\n }\n ]\n },\n \"unit\": \"none\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 6,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 2,\n \"interval\": null,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"text\": {},\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"8.1.2\",\n \"targets\": [\n {\n \"exemplar\": true,\n \"expr\": \"count(cs_info)\",\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Running Crowdsec\",\n \"transparent\": true,\n \"type\": \"stat\"\n },\n {\n \"aliasColors\": {},\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"decimals\": 1,\n \"fieldConfig\": {\n \"defaults\": {\n \"links\": []\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 8,\n \"w\": 18,\n \"x\": 6,\n \"y\": 1\n },\n \"hiddenSeries\": false,\n \"id\": 8,\n \"legend\": {\n \"alignAsTable\": true,\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"rightSide\": true,\n \"show\": true,\n \"sort\": \"total\",\n \"sortDesc\": true,\n \"total\": true,\n \"values\": true\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"8.1.2\",\n \"pointradius\": 2,\n \"points\": false,\n \"renLine truncated
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/managed-by: manual
|
||||
grafana_dashboard: "1"
|
||||
name: crowdsec-crowdsec-overview
|
||||
namespace: prometheus
|
||||
---
|
||||
apiVersion: v1
|
||||
data:
|
||||
crowdsec-lapi-metrics.json: "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"panel\",\n \"id\": \"bargauge\",\n \"name\": \"Bar gauge\",\n \"version\": \"\"\n },\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"8.1.2\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"iteration\": 1655915193937,\n \"links\": [],\n \"panels\": [\n {\n \"collapsed\": false,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 10,\n \"panels\": [],\n \"title\": \"Agents\",\n \"type\": \"row\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 2,\n \"options\": {\n \"displayMode\": \"gradient\",\n \"orientation\": \"vertical\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showUnfilled\": false,\n \"text\": {}\n },\n \"pluginVersion\": \"8.1.2\",\n \"repeat\": \"query0\",\n \"repeatDirection\": \"h\",\n \"targets\": [\n {\n \"exemplar\": false,\n \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\", instance=\\\"$lapi\\\"}[$__rate_interval])) by (le)\",\n \"format\": \"heatmap\",\n \"interval\": \"\",\n \"legendFormat\": \"{{le}}\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Agents Login\",\n \"type\": \"heatmap\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n },\n \"unit\": \"none\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 1\n },\n \"id\": 6,\n \"options\": {\n \"displayMode\": \"gradient\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showUnfilled\": false,\n \"text\": {}\n },\n \"pluginVersion\": \"8.1.2\",\n \"targets\": [\n {\n \"exemplar\": true,\n \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\"}[$__rate_interval])) by (le)\",\n \"format\": \"heatmap\",\n \"interval\": \"\",\n \"legendFormat\": \"{{le}}\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Heartbeat\",\n \"type\": \"heatmap\"\n },\n {\n \"collapsed\": false,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 9\n },\n \"id\": 12,\n \"panels\": [],\n \"title\": \"Decisions\",\n \"type\": \"row\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n Line truncated
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/managed-by: manual
|
||||
grafana_dashboard: "1"
|
||||
name: crowdsec-crowdsec-lapi-metrics
|
||||
namespace: prometheus
|
||||
---
|
||||
apiVersion: v1
|
||||
data:
|
||||
crowdsec-insight.json: "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"panel\",\n \"id\": \"bargauge\",\n \"name\": \"Bar gauge\",\n \"version\": \"\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"gauge\",\n \"name\": \"Gauge\",\n \"version\": \"\"\n },\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"8.1.2\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"stat\",\n \"name\": \"Stat\",\n \"version\": \"\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"iteration\": 1655915159751,\n \"links\": [],\n \"panels\": [\n {\n \"collapsed\": true,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 22,\n \"panels\": [\n {\n \"cacheTimeout\": null,\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"dateTimeAsIso\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 9,\n \"w\": 5,\n \"x\": 2,\n \"y\": 1\n },\n \"id\": 2,\n \"interval\": null,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"none\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"text\": {},\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"8.1.2\",\n \"targets\": [\n {\n \"exemplar\": true,\n \"expr\": \"(process_start_time_seconds{instance=\\\"$instance\\\"})*1000\",\n \"interval\": \"\",\n \"legendFormat\": \"{{instance}}\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Up since\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"displayName\": \"\",\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n },\n \"unit\": \"decbytes\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 9,\n \"w\": 5,\n \"x\": 7,\n \"y\": 1\n },\n \"id\": 4,\n \"options\": {\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\Line truncated
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/managed-by: manual
|
||||
grafana_dashboard: "1"
|
||||
name: crowdsec-crowdsec-insight
|
||||
namespace: prometheus
|
||||
@@ -1,214 +0,0 @@
|
||||
# CrowdSec self-healing: static machine identity + enforcement loops.
|
||||
#
|
||||
# Problem it fixes: the chart's agent init container runs
|
||||
# `cscli lapi register --machine "$POD_NAME" ...`
|
||||
# unconditionally. Credentials live in an emptyDir, the machine row lives
|
||||
# in LAPI's persistent DB. Any init re-run for an already-known pod name
|
||||
# (kubelet restart, node reboot) dies with
|
||||
# 403 Forbidden: user '<pod>' already exist
|
||||
# and the DaemonSet pod sticks in Init forever. Every DS restart also
|
||||
# leaves an orphan machine row that is never cleaned.
|
||||
#
|
||||
# Design (name-independent):
|
||||
# * Agent identity is a STATIC machine `crowdsec-agent-workstation`
|
||||
# whose password lives in Secret `crowdsec-agent-credentials`
|
||||
# (created once, manually - like all other secrets in this repo).
|
||||
# The secret is mounted into agent pods at
|
||||
# /tmp_config/local_api_credentials.yaml (see extraVolumeMounts in
|
||||
# crowdsec-values.yaml), which is exactly the path the agent's main
|
||||
# container copies into place at startup.
|
||||
# * The DS init command is patched (strategic merge, by container name)
|
||||
# to SKIP registration when that file exists, keeping the legacy
|
||||
# register path only as fallback. Detection marker in the patched
|
||||
# command: `[ -s /tmp_config`.
|
||||
# * This CronJob enforces the desired state hourly, so recovery is
|
||||
# automatic even after `helm upgrade` reverts the DS patch or the
|
||||
# LAPI database is wiped:
|
||||
# 1. patch DS init if it still has the unconditional register
|
||||
# (no-op otherwise - no restart churn);
|
||||
# 2. prune machines with no heartbeat for 2h (orphan hygiene);
|
||||
# 3. ensure the static machine exists, recreating it with the
|
||||
# Secret password if missing (agent retry loops reconnect
|
||||
# on their own - same name + same password);
|
||||
# 4. prune bouncer entries idle for 30d;
|
||||
# 5. delete decisions from LePresidente/http-generic-403-bf, a hub
|
||||
# scenario that bans an IP for 4h after 5 POST-403s in 10s and
|
||||
# therefore bans us for our own bouncer's fail-closed 403s.
|
||||
#
|
||||
# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
|
||||
# kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
|
||||
# Force a run:
|
||||
# kubectl create job -n crowdsec --from=cronjob/crowdsec-janitor janitor-now
|
||||
#
|
||||
# Helm upgrades: the janitor's strategic patch puts the DS field under
|
||||
# the `kubectl-patch` field manager, so a plain `helm upgrade` FAILS
|
||||
# with an SSA conflict on initContainers[].command. Procedure:
|
||||
# 1. revert init to chart state (kills the conflict):
|
||||
# helm template crowdsec crowdsec/crowdsec --version <ver> \
|
||||
# -n crowdsec -f crowdsec/k8s/crowdsec-values.yaml > /tmp/r.yaml
|
||||
# python3 -c "import yaml,json; ..." # build revert patch from
|
||||
# the rendered DaemonSet init command, then
|
||||
# kubectl patch ds crowdsec-agent -n crowdsec \
|
||||
# --type strategic -p "\$(cat /tmp/revert_patch.json)"
|
||||
# 2. helm upgrade --install crowdsec ... (no --force needed)
|
||||
# 3. janitor-now right away (upgrade reverts init; new pods would
|
||||
# sit in Init until the next hourly run otherwise).
|
||||
#
|
||||
# One-time bootstrap (order matters):
|
||||
# 1. Create Secret + static machine (see commands in chat).
|
||||
# 2. Apply this file, trigger janitor-now, wait for agent 1/1.
|
||||
# 3. One-time orphan cleanup:
|
||||
# kubectl exec -n crowdsec deploy/crowdsec-lapi -- \
|
||||
# cscli machines prune --duration 1h --force
|
||||
# 4. Only then `helm upgrade` crowdsec with the extraVolumes values.
|
||||
# Upgrade reverts the DS patch; trigger janitor-now right after it
|
||||
# (otherwise new pods sit in Init until the next hourly run, then
|
||||
# self-heal anyway).
|
||||
#
|
||||
# Password rotation: update the Secret, delete the machine
|
||||
# (`cscli machines delete crowdsec-agent-workstation`), trigger
|
||||
# janitor-now (recreates it), then `kubectl rollout restart
|
||||
# ds/crowdsec-agent -n crowdsec` (agent reads the file at startup only).
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: crowdsec-janitor
|
||||
namespace: crowdsec
|
||||
labels:
|
||||
app.kubernetes.io/part-of: crowdsec
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: crowdsec-janitor
|
||||
namespace: crowdsec
|
||||
labels:
|
||||
app.kubernetes.io/part-of: crowdsec
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/exec"]
|
||||
verbs: ["create"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["daemonsets"]
|
||||
verbs: ["get", "patch"]
|
||||
# `kubectl exec deploy/<name>` resolves deploy -> replicaset -> pod,
|
||||
# which needs read access to these (exec itself is pods/exec above).
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments", "replicasets"]
|
||||
verbs: ["get", "list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: crowdsec-janitor
|
||||
namespace: crowdsec
|
||||
labels:
|
||||
app.kubernetes.io/part-of: crowdsec
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: crowdsec-janitor
|
||||
namespace: crowdsec
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: crowdsec-janitor
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: crowdsec-janitor
|
||||
namespace: crowdsec
|
||||
labels:
|
||||
app.kubernetes.io/part-of: crowdsec
|
||||
spec:
|
||||
schedule: "17 * * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
activeDeadlineSeconds: 300
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: crowdsec
|
||||
spec:
|
||||
serviceAccountName: crowdsec-janitor
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: janitor
|
||||
# Same image the chart itself uses for registration jobs;
|
||||
# IfNotPresent so it works while the node is offline
|
||||
# (layer cached from the chart install).
|
||||
image: alpine/kubectl:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: AGENT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: crowdsec-agent-credentials
|
||||
key: password
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
LAPI_EXEC="kubectl exec -n crowdsec deploy/crowdsec-lapi --"
|
||||
echo "== 1. enforce patched agent init =="
|
||||
CUR=$(kubectl get ds crowdsec-agent -n crowdsec \
|
||||
-o jsonpath='{.spec.template.spec.initContainers[0].command[2]}')
|
||||
case "$CUR" in
|
||||
*'-s /tmp_config'*)
|
||||
echo "init already patched"
|
||||
;;
|
||||
*)
|
||||
echo "patching init"
|
||||
WAIT='until nc "$LAPI_HOST" "$LAPI_PORT" -z'
|
||||
WAIT="$WAIT; do echo waiting for lapi to start; sleep 5; done"
|
||||
LINK='ln -s /staging/etc/crowdsec /etc/crowdsec'
|
||||
REG='cscli lapi register --machine "$USERNAME"'
|
||||
REG="$REG -u \"\$LAPI_URL\" --token \"\$REGISTRATION_TOKEN\""
|
||||
CREDS=/tmp_config/local_api_credentials.yaml
|
||||
CMD="$WAIT; $LINK; [ -s $CREDS ] || {"
|
||||
CMD="$CMD $REG && cp"
|
||||
CMD="$CMD /etc/crowdsec/local_api_credentials.yaml $CREDS; }"
|
||||
ESC=$(printf '%s' "$CMD" | sed 's/"/\\"/g')
|
||||
PATCH='{"spec":{"template":{"spec":{"initContainers":'
|
||||
PATCH=$PATCH'[{"name":"wait-for-lapi-and-register",'
|
||||
PATCH=$PATCH'"command":["sh","-c","'$ESC'"]}]}}}}'
|
||||
kubectl patch ds crowdsec-agent -n crowdsec \
|
||||
--type strategic -p "$PATCH"
|
||||
;;
|
||||
esac
|
||||
echo "== 2. prune orphan machines (no heartbeat for 2h) =="
|
||||
$LAPI_EXEC cscli machines prune --duration 2h --force
|
||||
echo "== 3. ensure static machine exists =="
|
||||
if $LAPI_EXEC cscli machines inspect \
|
||||
crowdsec-agent-workstation >/dev/null 2>&1; then
|
||||
echo "static machine present"
|
||||
else
|
||||
echo "recreating static machine"
|
||||
$LAPI_EXEC cscli machines add crowdsec-agent-workstation \
|
||||
--password "$AGENT_PASSWORD" --force
|
||||
fi
|
||||
echo "== 4. prune stale bouncers (no pull for 30d) =="
|
||||
$LAPI_EXEC cscli bouncers prune -d 720h --force
|
||||
echo "== 5. drop http-403-bf decisions (4h self-bans) =="
|
||||
# `LePresidente/http-generic-403-bf` (hub item
|
||||
# crowdsecurity/http-generic-bf v0.9) bans any source IP
|
||||
# after 5 POSTs answered 403 within 10s, for 4h. That
|
||||
# includes 403s this homelab generates ITSELF (any
|
||||
# bouncer fail-closed, any app CSRF/rate-limit 403), and a
|
||||
# 4h ban on the runner/home IP silently breaks deploys and
|
||||
# browsing. The scenario cannot be removed per-scenario -
|
||||
# it is baked into a hub item, and disabling the whole
|
||||
# base-http-scenarios collection would drop ~40 useful
|
||||
# detections. Instead we keep the detection and drop its
|
||||
# decisions hourly; the LAN/home whitelists in
|
||||
# crowdsec-values.yaml handle the legit sources, so this
|
||||
# only ever hits real scanners (who are re-banned anyway).
|
||||
$LAPI_EXEC cscli decisions delete \
|
||||
--scenario LePresidente/http-generic-403-bf --all || true
|
||||
@@ -1,6 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: crowdsec
|
||||
labels:
|
||||
app.kubernetes.io/part-of: crowdsec
|
||||
@@ -1,34 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: crowdsec-lapi
|
||||
namespace: crowdsec
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: crowdsec
|
||||
type: lapi
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: traefik
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
k8s-app: crowdsec
|
||||
type: agent
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6060
|
||||
+45
-41
@@ -1,46 +1,50 @@
|
||||
services:
|
||||
dockmon:
|
||||
image: darthnorse/dockmon:2.5.0
|
||||
container_name: dockmon
|
||||
restart: unless-stopped
|
||||
# ports:
|
||||
# - 8000:443
|
||||
volumes:
|
||||
- data:/app/data
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-k", "-f", "https://localhost:443/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.services.dockmon.loadbalancer.server.port=443"
|
||||
- "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
|
||||
- "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
|
||||
dockmon:
|
||||
image: darthnorse/dockmon:latest
|
||||
container_name: dockmon
|
||||
restart: unless-stopped
|
||||
# ports:
|
||||
# - 8000:443
|
||||
volumes:
|
||||
- data:/app/data
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
healthcheck:
|
||||
test: [ "CMD", "curl", "-k", "-f", "https://localhost:443/health" ]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.docker.network=proxy"
|
||||
- "traefik.http.services.dockmon.loadbalancer.server.port=443"
|
||||
- "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
|
||||
- "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
|
||||
|
||||
# Prod Router
|
||||
- "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
|
||||
- "traefik.http.routers.dockmon.entrypoints=websecure"
|
||||
- "traefik.http.routers.dockmon.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.dockmon.tls.certresolver=letsencrypt"
|
||||
# Local Router
|
||||
- "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`)"
|
||||
- "traefik.http.routers.dockmon-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.dockmon-local.tls=true"
|
||||
# Dev Router
|
||||
- "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
|
||||
- "traefik.http.routers.dockmon-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.dockmon-dev.tls=true"
|
||||
# Prod Router
|
||||
- "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
|
||||
- "traefik.http.routers.dockmon.entrypoints=websecure"
|
||||
- "traefik.http.routers.dockmon.middlewares=security-chain@file"
|
||||
- "traefik.http.routers.dockmon.service=dockmon"
|
||||
- "traefik.http.routers.dockmon.tls=true"
|
||||
# Local Router
|
||||
- "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`)"
|
||||
- "traefik.http.routers.dockmon-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.dockmon-local.middlewares=security-headers@file"
|
||||
- "traefik.http.routers.dockmon-local.tls=true"
|
||||
# Dev Router
|
||||
- "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
|
||||
- "traefik.http.routers.dockmon-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.dockmon-dev.middlewares=security-chain@file"
|
||||
- "traefik.http.routers.dockmon-dev.tls=true"
|
||||
|
||||
# Glance Metadata
|
||||
- glance.name=dockmon
|
||||
- glance.url=https://dockmon.forust.xyz/
|
||||
- glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
|
||||
networks:
|
||||
- proxy
|
||||
# Glance Metadata
|
||||
- glance.name=dockmon
|
||||
- glance.url=https://dockmon.forust.xyz/
|
||||
- glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
|
||||
networks:
|
||||
- proxy
|
||||
volumes:
|
||||
data:
|
||||
data:
|
||||
networks:
|
||||
proxy:
|
||||
external: true
|
||||
proxy:
|
||||
external: true
|
||||
@@ -1,28 +0,0 @@
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: dockmon-prod-tls
|
||||
namespace: dockmon
|
||||
spec:
|
||||
secretName: dockmon-prod-tls
|
||||
dnsNames:
|
||||
- dockmon.forust.xyz
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: internal-wildcard-tls
|
||||
namespace: dockmon
|
||||
spec:
|
||||
secretName: internal-wildcard-tls
|
||||
dnsNames:
|
||||
- "*.workstation.internal"
|
||||
- "*.gigaforust.internal"
|
||||
- workstation.internal
|
||||
- gigaforust.internal
|
||||
issuerRef:
|
||||
name: internal-ca
|
||||
kind: ClusterIssuer
|
||||
@@ -1,68 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: dockmon-service
|
||||
namespace: dockmon
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: dockmon
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 443
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: dockmon-statefulset
|
||||
namespace: dockmon
|
||||
spec:
|
||||
serviceName: dockmon-service
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: dockmon
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: dockmon
|
||||
spec:
|
||||
containers:
|
||||
- name: dockmon
|
||||
image: darthnorse/dockmon:2.5.0
|
||||
ports:
|
||||
- containerPort: 443
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /app/data
|
||||
- name: docker-sock
|
||||
mountPath: /var/run/docker.sock
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 443
|
||||
scheme: HTTPS
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 10
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "1.5Gi"
|
||||
cpu: "700m "
|
||||
volumes:
|
||||
- name: docker-sock
|
||||
hostPath:
|
||||
path: /var/run/docker.sock
|
||||
type: Socket
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
@@ -1,47 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: ServersTransport
|
||||
metadata:
|
||||
name: dockmon-transport
|
||||
namespace: dockmon
|
||||
spec:
|
||||
insecureSkipVerify: true
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: dockmon-prod
|
||||
namespace: dockmon
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`dockmon.forust.xyz`)
|
||||
kind: Rule
|
||||
middlewares:
|
||||
- name: crowdsec-bouncer
|
||||
namespace: crowdsec
|
||||
- name: security-headers@file
|
||||
services:
|
||||
- name: dockmon-service
|
||||
port: 443
|
||||
serversTransport: dockmon-transport
|
||||
tls:
|
||||
secretName: dockmon-prod-tls
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: dockmon-local
|
||||
namespace: dockmon
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`dockmon.workstation.internal`) || Host(`dockmon.gigaforust.internal`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: dockmon-service
|
||||
port: 443
|
||||
serversTransport: dockmon-transport
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: dockmon
|
||||
+17
-15
@@ -1,29 +1,31 @@
|
||||
services:
|
||||
downtify:
|
||||
container_name: downtify
|
||||
image: ghcr.io/henriquesebastiao/downtify:3.1.0
|
||||
restart: unless-stopped
|
||||
image: ghcr.io/henriquesebastiao/downtify:latest
|
||||
# ports:
|
||||
# - '7077:8000'
|
||||
volumes:
|
||||
- ./Downtify_downloads:/downloads
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.services.downtify.loadbalancer.server.port=8000"
|
||||
- traefik.enable=true
|
||||
- traefik.docker.network=proxy
|
||||
- traefik.http.services.downtify.loadbalancer.server.port=8000
|
||||
|
||||
# Prod Router
|
||||
- "traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)"
|
||||
- "traefik.http.routers.downtify.entrypoints=websecure"
|
||||
- "traefik.http.routers.downtify.middlewares=security-chain@file"
|
||||
- "traefik.http.routers.downtify.tls.certresolver=letsencrypt"
|
||||
# Prod Router
|
||||
- traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)
|
||||
- traefik.http.routers.downtify.entrypoints=websecure
|
||||
- traefik.http.routers.downtify.middlewares=security-chain@file
|
||||
- traefik.http.routers.downtify.tls=true
|
||||
# Local Router
|
||||
- "traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`)"
|
||||
- "traefik.http.routers.downtify-local.entrypoints=websecure"
|
||||
- "traefik.http.routers.downtify-local.tls=true"
|
||||
- traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`) || Host(`downtify.internal`)
|
||||
- traefik.http.routers.downtify-local.entrypoints=websecure
|
||||
- traefik.http.routers.downtify-local.middlewares=security-headers@file
|
||||
- traefik.http.routers.downtify-local.tls=true
|
||||
# Dev Router
|
||||
- "traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)"
|
||||
- "traefik.http.routers.downtify-dev.entrypoints=websecure"
|
||||
- "traefik.http.routers.downtify-dev.tls=true"
|
||||
- traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)
|
||||
- traefik.http.routers.downtify-dev.entrypoints=websecure
|
||||
- traefik.http.routers.downtify-dev.middlewares=security-chain@file
|
||||
- traefik.http.routers.downtify-dev.tls=true
|
||||
networks:
|
||||
- proxy
|
||||
networks:
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: downtify-service
|
||||
namespace: downtify
|
||||
spec:
|
||||
selector:
|
||||
app: downtify
|
||||
ports:
|
||||
- port: 8000
|
||||
targetPort: 8000
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: downtify-deployment
|
||||
namespace: downtify
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: downtify
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: downtify
|
||||
spec:
|
||||
containers:
|
||||
- name: downtify
|
||||
image: ghcr.io/henriquesebastiao/downtify:3.1.0
|
||||
ports:
|
||||
- containerPort: 8000
|
||||
volumeMounts:
|
||||
- name: downloads
|
||||
mountPath: /downloads
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "1"
|
||||
volumes:
|
||||
- name: downloads
|
||||
persistentVolumeClaim:
|
||||
claimName: downtify-downloads-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: downtify-downloads-pvc
|
||||
namespace: downtify
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -1,37 +0,0 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: downtify-prod
|
||||
namespace: downtify
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`downtify.forust.xyz`)
|
||||
kind: Rule
|
||||
middlewares:
|
||||
- name: crowdsec-bouncer
|
||||
namespace: crowdsec
|
||||
- name: security-chain@file
|
||||
services:
|
||||
- name: downtify-service
|
||||
port: 8000
|
||||
tls:
|
||||
secretName: downtify-prod-tls
|
||||
---
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: IngressRoute
|
||||
metadata:
|
||||
name: downtify-local
|
||||
namespace: downtify
|
||||
spec:
|
||||
entryPoints:
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`downtify.workstation.internal`) || Host(`downtify.gigaforust.internal`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: downtify-service
|
||||
port: 8000
|
||||
tls:
|
||||
secretName: internal-wildcard-tls
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: downtify
|
||||
@@ -3,13 +3,12 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: gcr.forust.xyz/forust/dtek-notif:latest
|
||||
pull_policy: build
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- TZ=Europe/Kyiv
|
||||
|
||||
dns:
|
||||
- 1.1.1.1
|
||||
- 8.8.8.8
|
||||
networks:
|
||||
- default
|
||||
- default
|
||||
+417
-387
File diff suppressed because it is too large.
Load diff
@@ -9,6 +9,5 @@ WEBINAR_CHECK_INTERVAL=60
|
||||
REDIS_HOST=redis
|
||||
REDIS_PORT=6379
|
||||
PLAYWRIGHT_WS=ws://playwright-service:3000/ws
|
||||
TZ=Europe/Kyiv
|
||||
WEBINAR_TELEGRAM_TOKEN=your_telegram_bot_token_here
|
||||
WEBINAR_ADMIN_ID=123456789
|
||||
@@ -1 +0,0 @@
|
||||
1.56.0
|
||||
@@ -1,11 +1,11 @@
|
||||
services:
|
||||
redis:
|
||||
image: redis:8.10.2-alpine
|
||||
image: redis:alpine
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
test: [ "CMD", "redis-cli", "ping" ]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
@@ -17,15 +17,13 @@ services:
|
||||
|
||||
session-keeper:
|
||||
build: ./phpsessid-bot
|
||||
image: gcr.forust.xyz/forust/session-keeper:latest
|
||||
pull_policy: build
|
||||
env_file: .env
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
|
||||
test: [ "CMD-SHELL", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1" ]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
@@ -33,8 +31,6 @@ services:
|
||||
|
||||
webinar-checker:
|
||||
build: ./webinar-checker
|
||||
image: gcr.forust.xyz/forust/webinar-checker:latest
|
||||
pull_policy: build
|
||||
env_file: .env
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
|
||||
Whitespace-only changes.
@@ -1,96 +0,0 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: edu-master-webinar
|
||||
namespace: edu-master
|
||||
labels:
|
||||
release: prometheus-stack
|
||||
spec:
|
||||
groups:
|
||||
- name: edu_master.webinar
|
||||
rules:
|
||||
# No successful webinar check for 5m (~2-3 missed 2-min checks).
|
||||
# Catches: playwright hangs/timeouts, version skew, site changes, hung job.
|
||||
# The last_success > 0 guard is mandatory: checker.py initialises
|
||||
# last_success to 0, so without it `time() - 0` equals the current epoch
|
||||
# and humanizeDuration renders ~20722d on every pod restart. Keep the
|
||||
# duration expression on the left so $value stays the real gap.
|
||||
- alert: WebinarCheckerNoSuccessfulCheck
|
||||
expr: |
|
||||
((time() - webinar_check_last_success_timestamp_seconds) > 300)
|
||||
and (webinar_check_last_success_timestamp_seconds > 0)
|
||||
and (webinar_check_last_run_timestamp_seconds > 0)
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Webinar checker has no successful check for 5m"
|
||||
description: "edu-master/webinar-checker: last successful webinar check was {{ $value | humanizeDuration }} ago. Checks are failing or hanging (see consecutive failures alert). Notifications about new webinars are NOT being sent."
|
||||
|
||||
# Checks are running but none has ever succeeded since pod start.
|
||||
# Split out from the rule above so a zeroed gauge never feeds
|
||||
# humanizeDuration.
|
||||
- alert: WebinarCheckerNeverSucceeded
|
||||
expr: |
|
||||
(webinar_check_last_success_timestamp_seconds == 0)
|
||||
and (webinar_check_last_run_timestamp_seconds > 0)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Webinar checker has never completed a successful check"
|
||||
description: 'edu-master/webinar-checker: checks have been running for 10m but not one has ever succeeded since the pod started, so every check is failing. Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
|
||||
|
||||
# Fast path: 3 consecutive failures (~6+ min at 2-min interval).
|
||||
- alert: WebinarCheckerConsecutiveFailures
|
||||
expr: |
|
||||
webinar_check_consecutive_failures >= 3
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Webinar checker failing consecutively"
|
||||
description: 'edu-master/webinar-checker: {{ $value }} consecutive webinar check failures (timeout / playwright error / page error). Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
|
||||
|
||||
# Metrics endpoint not scraped for 10m: pod down, metrics server dead, or ServiceMonitor broken.
|
||||
- alert: WebinarCheckerScrapeDown
|
||||
expr: |
|
||||
absent(webinar_check_last_run_timestamp_seconds) == 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Webinar checker metrics missing"
|
||||
description: "edu-master/webinar-checker: no metrics series for 10m. Pod may be down, metrics server dead, or ServiceMonitor/Service broken. Webinar checks are unobserved."
|
||||
|
||||
# EDU session lost: session-keeper down or credentials expired. Without PHPSESSID every check is skipped.
|
||||
- alert: EduPhpsessidMissing
|
||||
expr: |
|
||||
edu_phpsessid_present == 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "EDU_PHPSESSID missing"
|
||||
description: "edu-master: EDU_PHPSESSID absent from redis for 10m. Webinar/diari/schedule checks are all skipped. Check session-keeper logs and EDU credentials."
|
||||
|
||||
# Hard deps: checker and playwright deployments unavailable.
|
||||
- alert: WebinarCheckerDeploymentDown
|
||||
expr: |
|
||||
kube_deployment_status_replicas_unavailable{deployment="webinar-checker", namespace="edu-master"} > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Webinar checker deployment unavailable"
|
||||
description: "edu-master/webinar-checker deployment has {{ $value }} unavailable replica(s) for 10m."
|
||||
|
||||
- alert: PlaywrightServiceDown
|
||||
expr: |
|
||||
kube_deployment_status_replicas_unavailable{deployment="playwright-service", namespace="edu-master"} > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Playwright service unavailable"
|
||||
description: "edu-master/playwright-service deployment has {{ $value }} unavailable replica(s) for 10m. All webinar/diari/schedule checks fail without it."
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: edu-master
|
||||
@@ -1,58 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: playwright-service
|
||||
namespace: edu-master
|
||||
labels:
|
||||
app: edu-master-playwright
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: edu-master-playwright
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: edu-master-playwright
|
||||
spec:
|
||||
containers:
|
||||
- name: playwright
|
||||
# renovate: datasource=docker depName=mcr.microsoft.com/playwright versioning=docker
|
||||
image: mcr.microsoft.com/playwright:v1.56.0-jammy
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- npx
|
||||
- -y
|
||||
- playwright@1.56.0
|
||||
- run-server
|
||||
- --port
|
||||
- "3000"
|
||||
- --path
|
||||
- /ws
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
readinessProbe:
|
||||
tcpSocket:
|
||||
port: 3000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
livenessProbe:
|
||||
tcpSocket:
|
||||
port: 3000
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 3
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: playwright-service
|
||||
namespace: edu-master
|
||||
spec:
|
||||
selector:
|
||||
app: edu-master-playwright
|
||||
ports:
|
||||
- name: ws
|
||||
port: 3000
|
||||
targetPort: 3000
|
||||
@@ -1,75 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: edu-master
|
||||
labels:
|
||||
app: edu-master-redis
|
||||
spec:
|
||||
serviceName: redis
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: edu-master-redis
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: edu-master-redis
|
||||
spec:
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:8.10.2-alpine
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
volumeMounts:
|
||||
- name: redis-data
|
||||
mountPath: /data
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["redis-cli", "ping"]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["redis-cli", "ping"]
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
volumes:
|
||||
- name: redis-data
|
||||
persistentVolumeClaim:
|
||||
claimName: redis-data-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: redis-data-pvc
|
||||
namespace: edu-master
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: edu-master
|
||||
spec:
|
||||
selector:
|
||||
app: edu-master-redis
|
||||
ports:
|
||||
- name: redis
|
||||
port: 6379
|
||||
targetPort: 6379
|
||||
@@ -1,50 +0,0 @@
|
||||
# One-time Job to migrate redis state from docker compose to k8s (maintenance window).
|
||||
# The .example file is not applied by the deploy pipeline (mask *.example.yaml).
|
||||
#
|
||||
# Runbook:
|
||||
# 1. docker compose -f <repo>/edu_master/compose.yaml stop # SIGTERM -> redis will flush dump.rdb
|
||||
# 2. docker run --rm -v edu_master_redis-data:/data \
|
||||
# -v /tmp/edu-master-backup:/backup \
|
||||
# redis:alpine sh -c "cp /data/dump.rdb /backup/ && ls -la /backup"
|
||||
# 3. kubectl apply -f edu_master/k8s/namespace.yaml
|
||||
# 4. kubectl apply -f <only the PVC from redis.yaml> # seed must come BEFORE redis pod starts
|
||||
# 5. kubectl apply -f edu_master/k8s/restore-seed-job.yaml.example
|
||||
# kubectl wait --for=condition=complete job/redis-restore-seed -n edu-master --timeout=120s
|
||||
# 6. kubectl delete job redis-restore-seed -n edu-master
|
||||
# 7. kubectl apply -f edu_master/k8s/ -R # apply remaining manifests
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: redis-restore-seed
|
||||
namespace: edu-master
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: seed
|
||||
image: redis:alpine
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
ls -la /backup
|
||||
cp /backup/dump.rdb /data/dump.rdb
|
||||
chmod 644 /data/dump.rdb
|
||||
ls -la /data
|
||||
volumeMounts:
|
||||
- name: redis-data
|
||||
mountPath: /data
|
||||
- name: backup
|
||||
mountPath: /backup
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: redis-data
|
||||
persistentVolumeClaim:
|
||||
claimName: redis-data-pvc
|
||||
- name: backup
|
||||
hostPath:
|
||||
path: /tmp/edu-master-backup
|
||||
type: DirectoryOrCreate
|
||||
@@ -1,29 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: edu-master-secrets
|
||||
namespace: edu-master
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Session keeper credentials
|
||||
KEEPER_LOGIN: ""
|
||||
KEEPER_PASSWORD: ""
|
||||
KEEPER_INTERVAL: "10"
|
||||
# EDU links
|
||||
EDU_URL_BASE: "https://edu.edu.vn.ua"
|
||||
EDU_URL_LOGIN: "/user/login"
|
||||
EDU_URL_COURSES: "/course/userlist"
|
||||
EDU_URL_WEBINAR: "/webinar/useractive"
|
||||
# Playwright
|
||||
USER_AGENT: ""
|
||||
PLAYWRIGHT_WS: "ws://playwright-service:3000/ws"
|
||||
# Webinar-checker
|
||||
WEBINAR_TELEGRAM_TOKEN: ""
|
||||
WEBINAR_ADMIN_ID: ""
|
||||
WEBINAR_CHECK_INTERVAL: "60"
|
||||
# Prometheus metrics endpoint (scraped via ServiceMonitor, alerts in k8s/alerts.yaml)
|
||||
METRICS_PORT: "8000"
|
||||
# Database
|
||||
REDIS_HOST: "redis"
|
||||
REDIS_PORT: "6379"
|
||||
TZ: "Europe/Kyiv"
|
||||
@@ -1,15 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: webinar-checker
|
||||
namespace: edu-master
|
||||
labels:
|
||||
app: edu-master-webinar-checker
|
||||
spec:
|
||||
selector:
|
||||
app: edu-master-webinar-checker
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 8000
|
||||
targetPort: metrics
|
||||
protocol: TCP
|
||||
@@ -1,16 +0,0 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: webinar-checker
|
||||
namespace: edu-master
|
||||
labels:
|
||||
release: prometheus-stack
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: edu-master-webinar-checker
|
||||
endpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
@@ -1,51 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: session-keeper
|
||||
namespace: edu-master
|
||||
labels:
|
||||
app: edu-master-session-keeper
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: edu-master-session-keeper
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: edu-master-session-keeper
|
||||
spec:
|
||||
initContainers:
|
||||
- name: wait-redis
|
||||
image: redis:8.10.2-alpine
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
i=0
|
||||
until redis-cli -h redis ping | grep -q PONG; do
|
||||
i=$((i+1))
|
||||
[ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
|
||||
sleep 2
|
||||
done
|
||||
echo "redis is ready"
|
||||
containers:
|
||||
- name: session-keeper
|
||||
image: gcr.forust.xyz/forust/session-keeper:prod
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: edu-master-secrets
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 96Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["/bin/sh", "-ec", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 10
|
||||
@@ -1,73 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: webinar-checker
|
||||
namespace: edu-master
|
||||
labels:
|
||||
app: edu-master-webinar-checker
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: edu-master-webinar-checker
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: edu-master-webinar-checker
|
||||
spec:
|
||||
# Enforces dependency order like compose depends_on:
|
||||
# redis healthy -> session-keeper healthy (EXISTS EDU_PHPSESSID) -> playwright started
|
||||
initContainers:
|
||||
- name: wait-deps
|
||||
image: redis:8.10.2-alpine
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
i=0
|
||||
until redis-cli -h redis ping | grep -q PONG; do
|
||||
i=$((i+1))
|
||||
[ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
|
||||
sleep 2
|
||||
done
|
||||
echo "redis ok"
|
||||
until [ "$(redis-cli -h redis EXISTS EDU_PHPSESSID)" = "1" ]; do
|
||||
i=$((i+1))
|
||||
[ "$i" -ge 300 ] && echo "TIMEOUT: no PHPSESSID (session-keeper down?)" && exit 1
|
||||
sleep 2
|
||||
done
|
||||
echo "PHPSESSID ok"
|
||||
until nc -z playwright-service 3000; do
|
||||
i=$((i+1))
|
||||
[ "$i" -ge 300 ] && echo "TIMEOUT: playwright-service not reachable" && exit 1
|
||||
sleep 2
|
||||
done
|
||||
echo "playwright ok"
|
||||
containers:
|
||||
- name: webinar-checker
|
||||
image: gcr.forust.xyz/forust/webinar-checker:prod
|
||||
ports:
|
||||
- name: metrics
|
||||
containerPort: 8000
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 12
|
||||
initialDelaySeconds: 10
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: edu-master-secrets
|
||||
env:
|
||||
- name: TZ
|
||||
value: "Europe/Kyiv"
|
||||
resources:
|
||||
requests:
|
||||
cpu: "50m"
|
||||
memory: "128Mi"
|
||||
limits:
|
||||
cpu: "600m"
|
||||
memory: "512Mi"
|
||||
@@ -3,10 +3,10 @@ FROM python:3.11-slim
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends redis-tools && rm -rf /var/lib/apt/lists/*
|
||||
RUN apt-get update && apt-get install -y redis-tools && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install dependencies
|
||||
RUN pip install --no-cache-dir requests==2.32.3 redis==5.2.1
|
||||
RUN pip install requests redis
|
||||
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
Loaded 100 of 483 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user