md: archive overlayfs incident

incident-archive/Overlay FS failure.md

@@ -0,0 +1,54 @@
+# Resolved: Overlay FS failure (and so containers)
+
+15-12-2025 03:02 AM EET: Degraded control panels' performances, following by full cascade docker failure
+
+15-12-2025 04:36 AM EET: Identified: Services are terminated due to server software (Overlay FS) + hardware issues (HDD).
+
+15-12-2025 08:45 PM EET: Restored NextCloud service with few tweaks to lower I/O
+
+---
+
+16-12-2025 08:34 PM EET: Ordered new HDD, ETA 22nd of December - 2nd of January
+
+---
+
+17-12-2025 02:47 AM EET: To avoid additional I/O into kuma's database, disabled uptime monitoring for non-critical services, such as:
+
+- Game servers
+- Gitea (no public projects being hosted yet)
+- Landings
+- Cloud services
+- PenPot
+- Auth provider
+- Secondary management tools
+- Chernuha's non-important infrastructure
+
+These can be identified by seeing ">2m ago" under monitor's heartbeats.
+
+---
+
+09-01-2026 02:32 PM EET: NextCloud's frontend files are corrupted due to the unknown issue. All user data is integrity-verified. To prevent user data corruption, NextCloud service will be restored after new hardware will be available.
+
+---
+
+14-01-2026 08:42 PM EET: After planned updating and restarting server, critical firmware software were corrupted because of physical degradation of the disk. Server inaccessible in any way
+
+---
+
+15-01-2026 11:30 AM EET: A new NAS-Grade HDD (Seagate IronWolf Pro) was ordered. ETA 16-01-2026 EET Before 12:00 PM
+
+---
+
+16-01-2026 12:47 AM EET: New server system is installed, data backed up. Experiencing docker memory leak.
+
+---
+
+17-01-2026 01:27 PM EET: All services except NextCloud and Satisfactory server are online.
+
+17-01-2026 03:48 PM EET: Nextcloud is online. Satisfactory will be provided on-demand. Monitoring status
+
+---
+
+##### Status: All services are online
+
+**Solution: moving all infrastructure onto new NAS-Grade HDD with fresh OS install**