Linux

would I just need to add a RX580 as a PCI host device and run the script after shutting down the vm?

Indeed, although don't just copy-paste the snippet I wrote: I just wrote it on the spot without testing it, you have to tweak it to run the function for the PCI device(s) you have in the IOMMU group of the GPU you want to pass through. In my case it's just 0000:03:00.0 and 0000:03:00.1, perhaps you will also only need two since the GPUs are the same.

You can procrastinate on doing all that, I'm fairly certain nothing will blow up.
Unfortunately my setup is very complex, I hacked together a framework of Zsh scripts that use libvirt hooks - otherwise I would just copy them here.

I didn't mean to say that you must use 0000:05:00.0 specifically, only to follow the rest of the guide without having the script - I'm not sure about identifying the correct device, I did that a long time ago, but I am pretty sure the AL Wiki guide has a way to list GPUs.

The error you get is self-explainatory: along with 0000:05:00.0 (or whatever device), you must also list the ones in the same IOMMU group, which should also be identified along with the PCI device(s) you want to pass through.

EDIT: I skimmed through the guide, apparently it's extremely un-straightforward (gayforward? idk), I'll try to make a director's cut.

The following script should allow you to see how your various PCI devices are mapped to IOMMU groups. If it does not return anything, you either have not enabled IOMMU support properly or your hardware does not support it.

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Example output:

IOMMU Group 1:
	00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU Group 13:
	06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together.

As to identifying which of the two is which GPU, your only safe bet is trying to determine which monitor is connected to which PCI device somehow, which I have no idea how to do - I went with trial and error, and hard resets.