blackstrat

I previously asked here about moving to ZFS. So a week on I'm here with an update. TL;DR: Surprisingly simple upgrade.

I decided to buy another HBA that came pre-flashed in IT mode and without an onboard BIOS (so that server bootups would be quicker - I'm not using the HBA attached disks as boot disks). For £30 it seems worth the cost to avoid the hassle of flashing it, plus if it all goes wrong I can revert back.

I read a whole load about Proxmox PCIE passthrough, most of it out of date it would seem. I am running an AMD system and there are many sugestions online to set grub parameters to amd_iommu=on, which when you read in to the kernel parameters for the 6.x version proxmox uses, isn't a valid value. I think I also read that there's no need to set iommu=pt on AMD systems. But it's all very confusing as most wikis that should know better are very Intel specific.

I eventually saw a youtube video of someone running proxmox 8 on AMD wanting to do the same as I was and they showed that if IOMMU isn't setup, then you get a warning in the web GUI when adding a device. Well that's interesting - I don't get that warning. I am also lucky that the old HBA is in its own IOMMU group, so it should pass through easy without breaking anything. I hope the new one will be the same.

Worth noting that there are a lot of bad Youtube videos with people giving bad advise on how to configure a VM for ZFS/TrueNAS use - you need them passed through properly so the VM's OS has full control of them. Which is why an IT HBA is required over an IR one, but just that alone doesn't mean you can't set the config up wrong.

I also discovered along the way that my existing file server VM was not setup to be able to handle PCIe passthrough. The default Machine Type that Proxmox suggests - i440fx - doesn't support it. So that needs changing to q35, also it has to be setup with UEFI. Well that's more of a problem as my VM is using BIOS. A this point it became easier to spin up a new VM with the correct setting and re-do the configuration of it. Other options to be aware of: Memory ballooning needs to be off and the CPU set to host.

At this point I haven't installed the new HBA yet.

Install a fresh version of Ubuntu Server 24.04 LTS and it all feels very snappy. Makes me wonder about my old VM, I think it might be an original install of 16.04 that I have upgraded every 2 years and was migrated over from my old ESXi R710 server a few years ago. Fair play to it, I have had zero issues with it in all that time. Ubuntu server is just absolutely rock solid.

Not too much to configure on this VM - SSH, NFS exports, etckeeper, a couple of users and groups. I use etckeeper, so I have a record of the /etc of all my VMs that I can look back to, which has come in handy on several occasions.

Now almost ready to swap the HBA after I run the final restic backup, which only takes 5 mins (I bloody love restic!). Also update the fstabs of VMS so they don't try mount the file server and stop a few from auto starting on boot, just temporarily.

Turn the server off and get inside to swap the cards over. Quite straightforward other than the SAS ports being in a worse place for ease of access. Power back on. Amazingly it all came up - last time I tried to add an NVME on a PCIe card it killed the system.

Set the PICe passthrough for the HBA on the new VM. Luckily the new HBA is on it's own IOMMU group (maybe that's somehow tied to the PCIE slot?) Make sure to tick the PCIE flag so it's not treated as PCI - remember PCI cards?!

Now the real deal. Boot the VM, SSH in. fdisk -l lists all the disks attached. Well this is good news! Try create the zpool zpool create storage raidz2 /dev/disk/by-id/XXXXXXX ...... Hmmm, can't do that as it knows it's a raid disk and mdadm has tried to mount it so they're in use. Quite a bit of investigation later with a combination of wipefs -af /dev/sdX, umount /dev/md126, mdadm --stop /dev/sd126 and shutdown -r now and the RAIDynes of the disks is gone and I can re-run the zpool command. It that worked! Note: I forgot to add in ashift=12 to my zpool creation command, I have only just noticed this as I write, but thankfully it was clever enough to pick the correct one.

$ zpool get all | grep ashift
storage  ashift                         0                              default

Hmmm, what's 0?

$ sudo zdb -l /dev/sdb1 | grep ashift
ashift: 12

Phew!!!

I also have passed through the USB backup disks I have, mounted them and started the restic backup restore. So far it's 1.503TB in after precisely 5 hours, which seems OK.

I'll setup monthly scrub cron jobs tomorrow.

P.S. I tried TrueNAS out in a VM with no disks to see what it's all about. It looks very nice, but I don't need any of that fancyness. I've always managed my VM's over SSH which I've felt is lighter weight and less open to attack.

Thanks for stopping by my Ted Talk.

At the moment I have my NAS setup as a Proxmox VM with a hardware RAID card handling 6 2TB disks. My VMs are running on NVMEs with the NAS VM handling the data storage with the RAIDed volume passed through to the VM direct in Proxmox. I am running it as a large ext4 partition. Mostly photos, personal docs and a few films. Only I really use it. My desktop and laptop mount it over NFS. I have restic backups running weekly to two external HDDs. It all works pretty well and has for years.

I am now getting ZFS curious. I know I'll need to IT flash the HBA, or get another. I'm guessing it's best to create the zpool in Proxmox and pass that through to the NAS VM? Or would it be better to pass the individual disks through to the VM and manage the zpool from there?

CDs are in every way better than vinyl records. They are smaller, much higher quality audio, lower noise floor and don't wear out by being played. The fact that CD sales are behind vinyl is a sign that the world has gone mad. The fact you can rip and stream your own CD media is fantastic because generally remasters are not good and streaming services typically only have remastered versions, not originals. You have no control on streaming services about what version of an album you're served or whether it'll still be there tomorrow. Not an issue with physical media.

The vast majority of people listen to music using equipment that produces audio of poor quality, especially those that stream using ear buds. It makes me very sad when people don't care that what they're listening to could sound so much better, especially if played through a hifi from a CD player, or using half decent (not beats) headphones.

There's plenty of good sounding and well produced music out there, but it's typically played back through the equivalent of two cans and some string. I'm not sure people remember how good good music can sound when played back through good kit.

I've run my own email server for a few years now without too many troubles. I also pay for a ProtonMail account that's been very good. But I've always struggled with PGP keys for encrypting messages to non-Proton users - basically everyone. The PGP key distribution setup just seemed half baked and a bit broken relying on central key servers.

Then I noticed that email I set from my personal email to my company provided email were being encrypted even though I wasn't doing anything to achieve this. This got me curious as to why that was happening which lead me to WKD (Web Key Directory). It's such a simple idea for providing discoverable downloads for public keys and it works really well having set it up for my own emails now.

It's basically a way of discovering the public key of someone's email by making it available over HTTPS at an address that can be calculated based on the email address itself. So if your email is name@example.com, then the public key can be hosted at (in this case) https://openpgpkey.example.com/.well-known/openpgpkey/example.com/hu/pmw31ijkbwshwfgsfaihtp5r4p55dzmc?l=name this is derived using a command like gpg-wks-client --print-wkd-url name@example.com. You just need an email client that can do this and find the key for you automatically. And when setting up your own server you generate the content using the keys in your gpg key ring using env GNUPGHOME=$(mktemp -d) gpg --locate-keys --auto-key-locate clear,wkd,nodefault name@example.com. Move this generated folder structure to your webserver and you're basically good to go.

I have this working with Thunderbird, which now prompts me to do the discoverability step when I enter an email that doesn't have an associated key. On Android, I've found OpenKeyChain can also do a search based just on the email address that apps like K9-Mail (to be Thunderbird mail) can then use.

Anyway, I thought this was pretty cool and was excited to see such an improvement in seamless encryption integration. It'd be nicer if on Thunderbird and K9 it all happened as soon as you enter an email address rather than a few extra steps to jump through to perform the search and confirm the keys. But it's a major improvement.

Does your email provider have WKD setup and working or do you use it already?

I noticed that I wasn't getting many mails (I need better monitoring), and discovered that my iredmail server was poorly.

I have spent far too much time and energy on getting it back and working these past few days, but I've finally got it back up and stable.

Some background: I've had iredmail running for probably going on 6 years now and have had very few issues at all. It runs on an Ubuntu VM on Proxmox and originally was running in the same VM on ESXi (I migrated it over). I haven't changed anything to do with the VM for years other than the Ubuntu LTS updates every 2-3 years, it's always been there and stable. I occasionally will update the Ubuntu OS and iredmail itself, no problems.

Back to the problem... I noticed that Postfix was running OK, but was showing a bunch of errors about clamav not being able to connect. Odd. I then noticed that amavis was not running and had seemed to just die. I couldn't find any reason in any log file. Very strange. Bunch of hunting, checking config file history in the git repo. Nothing significant for years.

Find that restarting the server got everything back up and running. Great, lets go to bed.... Wake up next morning to find that amavis was dead again - it only lasted about 40 mins and then just closed for no reason. Right, ok, time to turn off clamAV as that seemed be be coming up a bit wheilst looking, follow the guide, all is well. Hmm, this seems to be working, but I don't really want clamav off. A whole bunch of duck duck going and I still couldn't figure out a root cause.

And then it clicked, the thing that was causing amavis to close was that it was running out of memory and it was being killed. Bump the memory up to 4GB and re-enable everything as it originally was and.... it seems to have worked. Been going strong for over a day now.

I don't know what it was that's changed recently which has meant the memory requirements have gone up a bit, but at least it's now fixed and it took all of 2 minutes to adjust.

The joys of selfhosting!