Thanks for the detailed reply. I found a QSFP+ DAC that says it supports IB and Ethernet.
I don't have enough computers to set up a fabric, only the 2 I would be direct attaching have PCIE slots.
I've never used infiniband before so my reason for wanting to try it is just to learn what it is, and how it works. That said, some of those use-cases look very interesting, especially transporting NVMe namespaces, I didn't know that was possible.
I know a there are a lot of issues with self-hosting email, but I just don't thing this is one of them. First, it probably won't affect a self-hosted servers anyway unless you send a lot of emails, this requirement is only for servers sending 5,000 messages daily to Gmail. And even if you are, the requirements are not that harsh, it's a couple DNS records and a DKIM signing daemon, and if you are using a pre-build email package like mailcow it's probably already doing it.