# VictoriaMetrics

The `victoria-operator` Helm release converts Prometheus Operator
`ServiceMonitor` resources into owned `VMServiceScrape` resources. The
`VMAgent` selects converted scrapes labeled `release: prometheus-stack` in all
namespaces and writes them to the existing single-node VictoriaMetrics
instance. Changes to selected `ServiceMonitor` resources are reconciled
automatically; there is no copied Prometheus scrape-config blob to regenerate.

The agent drops targets for the Prometheus server service to avoid duplicating
its self-scrape. `scraper: victoria` identifies the samples ingested by this
VMAgent.

The VictoriaMetrics Operator chart and its CRDs are installed before the
Kubernetes manifests by the normal deploy workflow. On a cluster where the
operator CRDs are not installed yet, CI skips the server-side dry-run of the
`VMAgent` resource; the deploy installs the chart before applying that resource.

## Application metrics

The application ServiceMonitors use a 30s interval and a 10s timeout:

- Headscale: the external Service points to the Compose host on port 19090.
  A VMServiceScrape uses EndpointSlice discovery for this manually managed target.
  The Compose configuration must bind metrics to `0.0.0.0:9090`.
- NetBird: the combined server exports `/metrics` on port 9090. The existing
  `server.metricsPort` setting enables the listener.
- Gitea: `GITEA__metrics__ENABLED` enables `/metrics` on the HTTP port. The public
  ingress excludes this path. The monitor uses the internal Service directly.
- Immich: `IMMICH_TELEMETRY_INCLUDE=all` enables API and worker metrics on ports
  8081 and 8082. The monitor scrapes both ports on each server replica.

Deploy through the existing CI and deploy workflow. Gitea and Immich reload their
ConfigMap changes through Reloader. Check the VMAgent targets after deployment
and query `up{scraper="victoria",namespace=~"netbird|gitea|immich|headscale"}` in
VictoriaMetrics. All targets should report 1.

For rollback, revert the application metrics changes, run CI, and deploy the
revert. Remove the three application ServiceMonitors and the Headscale VMServiceScrape explicitly: the deployment
workflow applies manifests and does not prune removed resources.

For Headscale rollback, remove its VMServiceScrape and Service label, restore the
previous Compose metrics bind address, and restart only the Headscale service.