I noticed that I wasn't getting many mails (I need better monitoring), and discovered that my iredmail server was poorly.
I have spent far too much time and energy on getting it back and working these past few days, but I've finally got it back up and stable.
Some background: I've had iredmail running for probably going on 6 years now and have had very few issues at all. It runs on an Ubuntu VM on Proxmox and originally was running in the same VM on ESXi (I migrated it over). I haven't changed anything to do with the VM for years other than the Ubuntu LTS updates every 2-3 years, it's always been there and stable. I occasionally will update the Ubuntu OS and iredmail itself, no problems.
Back to the problem... I noticed that Postfix was running OK, but was showing a bunch of errors about clamav not being able to connect. Odd. I then noticed that amavis was not running and had seemed to just die. I couldn't find any reason in any log file. Very strange. Bunch of hunting, checking config file history in the git repo. Nothing significant for years.
Find that restarting the server got everything back up and running. Great, lets go to bed.... Wake up next morning to find that amavis was dead again - it only lasted about 40 mins and then just closed for no reason. Right, ok, time to turn off clamAV as that seemed be be coming up a bit wheilst looking, follow the guide, all is well. Hmm, this seems to be working, but I don't really want clamav off. A whole bunch of duck duck going and I still couldn't figure out a root cause.
And then it clicked, the thing that was causing amavis to close was that it was running out of memory and it was being killed. Bump the memory up to 4GB and re-enable everything as it originally was and.... it seems to have worked. Been going strong for over a day now.
I don't know what it was that's changed recently which has meant the memory requirements have gone up a bit, but at least it's now fixed and it took all of 2 minutes to adjust.
The joys of selfhosting!