homelab/prometheus-stack/k8s/README.md

VictoriaMetrics

The victoria-operator Helm release converts Prometheus Operator ServiceMonitor resources into owned VMServiceScrape resources. The VMAgent selects converted scrapes labeled release: prometheus-stack in all namespaces and writes them to the existing single-node VictoriaMetrics instance. Changes to selected ServiceMonitor resources are reconciled automatically; there is no copied Prometheus scrape-config blob to regenerate.

The agent drops targets for the Prometheus server service to avoid duplicating its self-scrape. scraper: victoria identifies the samples ingested by this VMAgent.

The VictoriaMetrics Operator chart and its CRDs are installed before the Kubernetes manifests by the normal deploy workflow. On a cluster where the operator CRDs are not installed yet, CI skips the server-side dry-run of the VMAgent resource; the deploy installs the chart before applying that resource.

Application metrics

The application ServiceMonitors use a 30s interval and a 10s timeout:

• Headscale: the external Service points to the Compose host on port 19090. A VMServiceScrape uses EndpointSlice discovery for this manually managed target. The Compose configuration must bind metrics to 0.0.0.0:9090.
• NetBird: the combined server exports /metrics on port 9090. The existing server.metricsPort setting enables the listener.
• Gitea: GITEA__metrics__ENABLED enables /metrics on the HTTP port. The public ingress excludes this path. The monitor uses the internal Service directly.
• Immich: IMMICH_TELEMETRY_INCLUDE=all enables API and worker metrics on ports 8081 and 8082. The monitor scrapes both ports on each server replica.

Deploy through the existing CI and deploy workflow. Gitea and Immich reload their ConfigMap changes through Reloader. Check the VMAgent targets after deployment and query up{scraper="victoria",namespace=~"netbird|gitea|immich|headscale"} in VictoriaMetrics. All targets should report 1.

For rollback, revert the application metrics changes, run CI, and deploy the revert. Remove the three application ServiceMonitors and the Headscale VMServiceScrape explicitly: the deployment workflow applies manifests and does not prune removed resources.

For Headscale rollback, remove its VMServiceScrape and Service label, restore the previous Compose metrics bind address, and restart only the Headscale service.