Proxmox is good as a host OS, you'll set up a VM for docker and run your stuff in that.
It has a built in backup system to image your VMs and Containers, you can combine it with Proxmox Backup Server either in a VM or on another system for incremental backups and deduplication.
As far as Nextcloud goes I'm not surprised you had issues, their setup is weird, non-standard and very unstable in my experience. I switched to Syncthing long ago and it's so much better.
I don't use it on Android because I don't need sync, I need backups.
Photo backups are handled by Immich, and a general backup is done by the FolderSync app on a daily schedule over WebDAV to my server.