So, I moved my nextcloud directory from a local SATA drive to a NFS mount from a nvme array on a 10G network
"I just need to change /docker/nextcloud
to /mnt/nfs/nextcloud
in the docker-compose.yml
, what's the issue, i do it live" - i tell myself
So i stop the container, copy /docker/nextcloud
to /mnt/nfs/nextcloud
, then edit the docker-compose.yml
... and.... because I'm doing it during a phone call without paying too much attention i change the main directory to /docker
I rebuild the container and I immediately hear a flood of telegram notifications from my uptime-kuma bot.... oh oh...
Looks like the nextcloud docker image has an initialization script that if it doesn't find the files in the directory, it will delete everything and install a fresh copy of nextcloud... so it deleted everything on my server
Luckily i had a very recent full borg backup and i'm restoring it (i kinda love-hate borg, i always forget the restore commands when in panic and the docs are a bit cryptic for me)
Lessons learned:
-
always double check everything
-
offsite backups are a must (if i accidentally wrote /
as path, i would have lost also the borg backups!)
-
offsite backups should not be permanently mounted, otherwise they would have been wiped as well
-
learn how to use and schedule filesystem snapshots, so the recovery wouldn't take ages like it's taking right now (2+ hours and i'm not even half way...)