diff --git a/incident-archive/Overlay FS failure.md b/incident-archive/Overlay FS failure.md new file mode 100644 index 0000000..726a548 --- /dev/null +++ b/incident-archive/Overlay FS failure.md @@ -0,0 +1,54 @@ +# Resolved: Overlay FS failure (and so containers) + +15-12-2025 03:02 AM EET: Degraded control panels' performances, following by full cascade docker failure + +15-12-2025 04:36 AM EET: Identified: Services are terminated due to server software (Overlay FS) + hardware issues (HDD). + +15-12-2025 08:45 PM EET: Restored NextCloud service with few tweaks to lower I/O + +--- + +16-12-2025 08:34 PM EET: Ordered new HDD, ETA 22nd of December - 2nd of January + +--- + +17-12-2025 02:47 AM EET: To avoid additional I/O into kuma's database, disabled uptime monitoring for non-critical services, such as: + +- Game servers +- Gitea (no public projects being hosted yet) +- Landings +- Cloud services +- PenPot +- Auth provider +- Secondary management tools +- Chernuha's non-important infrastructure + +These can be identified by seeing ">2m ago" under monitor's heartbeats. + +--- + +09-01-2026 02:32 PM EET: NextCloud's frontend files are corrupted due to the unknown issue. All user data is integrity-verified. To prevent user data corruption, NextCloud service will be restored after new hardware will be available. + +--- + +14-01-2026 08:42 PM EET: After planned updating and restarting server, critical firmware software were corrupted because of physical degradation of the disk. Server inaccessible in any way + +--- + +15-01-2026 11:30 AM EET: A new NAS-Grade HDD (Seagate IronWolf Pro) was ordered. ETA 16-01-2026 EET Before 12:00 PM + +--- + +16-01-2026 12:47 AM EET: New server system is installed, data backed up. Experiencing docker memory leak. + +--- + +17-01-2026 01:27 PM EET: All services except NextCloud and Satisfactory server are online. + +17-01-2026 03:48 PM EET: Nextcloud is online. Satisfactory will be provided on-demand. Monitoring status + +--- + +##### Status: All services are online + +**Solution: moving all infrastructure onto new NAS-Grade HDD with fresh OS install**