Csdrhrt

A while ago, I have build a new Ryzen setup. I am currently experimenting with Unraid 6.6.6 and trying to setup my daily Ubuntu VM with a passed through GTX 1060 from Inno3D. Unfortunately, I get error messages related to the GPU.

-- Setup --

M/B: ASRock - B450M Pro4

CPU: AMD Ryzen 5 2600 Six-Core @ 3400

GPU: GTX 1060 6Gb from Inno3d

HVM and IOMMU: Enabled (According to the info panel of Unraid)

I have tried passing through the GPU with many different Graphic ROM Bioses. One was straight from Techpowerup, one was the first but modified according to these instructions from SpaceinvaderOne and one was taken from Spaceinvaders post which features a list of community dumps (I took the one available for the 1060 from EVGA). All of them failed with the following error message:

2018-12-16T22:17:35.335767Z qemu-system-x86_64: -device vfio-pci,host=23:00.0,id=hostdev0,bus=pci.4,addr=0x0,romfile=/mnt/disk1/isos/vbios/GTX1060.Inno3D.rom: Failed to mmap 0000:23:00.0 BAR 3. Performance may be slow

2018-12-16T22:17:35.410501Z qemu-system-x86_64: vfio: Cannot reset device 0000:23:00.0, depends on group 22 which is not owned.

2018-12-16T22:17:36.818822Z qemu-system-x86_64: vfio: Cannot reset device 0000:23:00.0, depends on group 22 which is not owned.

The PCI device with the number 0000:23:00 is the mentioned GPU. After shutting down the VM I see the following error in the logs, again indicating that somethign is not working out with the GPU (copied just once but it is many many times in the logs):

2018-12-16T22:18:08.354094Z qemu-system-x86_64: vfio_region_write(0000:23:00.0:region3+0x1098, 0x0,8) failed: Device or resource busy

So I suppose the issue is that my graphics card is used by some other part of the system. Looking for answers online, I found this post about the problem. I understand what the author explained but the fix did not work for me.

My configuration for the VM is the following. Since I am quite new to Unraid, I will rather post the full configuration and strip some parts off when some more experienced community members tell which parts are the most important.

<?xml version='1.0' encoding='UTF-8'?>

<domain type='kvm' id='5'>

  <name>Ubuntu New</name>

  <uuid>f6167326-d81b-2eb2-c0f4-4c5d33c40eee</uuid>

  <metadata>

    <vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu"/>

  </metadata>

  <memory unit='KiB'>8388608</memory>

  <currentMemory unit='KiB'>8388608</currentMemory>

  <memoryBacking>

    <nosharepages/>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='4'/>

    <vcpupin vcpu='1' cpuset='5'/>

    <vcpupin vcpu='2' cpuset='8'/>

    <vcpupin vcpu='3' cpuset='9'/>

  </cputune>

  <resource>

    <partition>/machine</partition>

  </resource>

  <os>

    <type arch='x86_64' machine='pc-q35-3.0'>hvm</type>

    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>

    <nvram>/etc/libvirt/qemu/nvram/f6167326-d81b-2eb2-c0f4-4c5d33c40eee_VARS-pure-efi.fd</nvram>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='host-passthrough' check='none'>

    <topology sockets='1' cores='4' threads='1'/>

  </cpu>

  <clock offset='utc'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/local/sbin/qemu</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/user/domains/Ubuntu New/vdisk1.img'/>

      <backingStore/>

      <target dev='hdc' bus='virtio'/>

      <boot order='1'/>

      <alias name='virtio-disk2'/>

      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/mnt/user/isos/ubuntu-18.04.1-desktop-amd64.iso'/>

      <backingStore/>

      <target dev='hda' bus='sata'/>

      <readonly/>

      <boot order='2'/>

      <alias name='sata0-0-0'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <alias name='usb'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <alias name='usb'/>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci2'>

      <alias name='usb'/>

      <master startport='2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci3'>

      <alias name='usb'/>

      <master startport='4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>

    </controller>

    <controller type='sata' index='0'>

      <alias name='ide'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pcie-root'>

      <alias name='pcie.0'/>

    </controller>

    <controller type='pci' index='1' model='pcie-root-port'>

      <model name='pcie-root-port'/>

      <target chassis='1' port='0x8'/>

      <alias name='pci.1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>

    </controller>

    <controller type='pci' index='2' model='pcie-root-port'>

      <model name='pcie-root-port'/>

      <target chassis='2' port='0x9'/>

      <alias name='pci.2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='pci' index='3' model='pcie-root-port'>

      <model name='pcie-root-port'/>

      <target chassis='3' port='0xa'/>

      <alias name='pci.3'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='4' model='pcie-root-port'>

      <model name='pcie-root-port'/>

      <target chassis='4' port='0xb'/>

      <alias name='pci.4'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>

    </controller>

    <controller type='pci' index='5' model='pcie-root-port'>

      <model name='pcie-root-port'/>

      <target chassis='5' port='0xc'/>

      <alias name='pci.5'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>

    </controller>

    <controller type='pci' index='6' model='pcie-root-port'>

      <model name='pcie-root-port'/>

      <target chassis='6' port='0xd'/>

      <alias name='pci.6'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <alias name='virtio-serial0'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:47:af:f1'/>

      <source bridge='br0'/>

      <target dev='vnet0'/>

      <model type='virtio'/>

      <alias name='net0'/>

      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>

    </interface>

    <serial type='pty'>

      <source path='/dev/pts/0'/>

      <target type='isa-serial' port='0'>

        <model name='isa-serial'/>

      </target>

      <alias name='serial0'/>

    </serial>

    <console type='pty' tty='/dev/pts/0'>

      <source path='/dev/pts/0'/>

      <target type='serial' port='0'/>

      <alias name='serial0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-Ubuntu New/org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>

      <alias name='channel0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <input type='mouse' bus='ps2'>

      <alias name='input0'/>

    </input>

    <input type='keyboard' bus='ps2'>

      <alias name='input1'/>

    </input>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x23' slot='0x00' function='0x0'/>

      </source>

      <alias name='hostdev0'/>

      <rom file='/mnt/user/isos/vbios/GTX1060.Inno3D.rom'/>

      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='usb' managed='no'>

      <source>

        <vendor id='0x046d'/>

        <product id='0xc52b'/>

        <address bus='1' device='2'/>

      </source>

      <alias name='hostdev1'/>

      <address type='usb' bus='0' port='1'/>

    </hostdev>

    <hostdev mode='subsystem' type='usb' managed='no'>

      <source>

        <vendor id='0x06a3'/>

        <product id='0x0cfa'/>

        <address bus='3' device='2'/>

      </source>

      <alias name='hostdev2'/>

      <address type='usb' bus='0' port='2'/>

    </hostdev>

    <memballoon model='none'/>

  </devices>

  <seclabel type='dynamic' model='dac' relabel='yes'>

    <label>+0:+100</label>

    <imagelabel>+0:+100</imagelabel>

  </seclabel>

</domain>

I also looked up the compatibility of my components but did not find any hard evidence that there is a hardware limitation in any way.

The corresponging question on the Unraid forum is this.

I am a bit desparate and open for any ideas...

PCIe passthrough works not on a device bases, but on a group bases. Your system tells you, that your GPU is in the same group as another device, that you don't pass through but use from the host.

The boot kernel log will tell you about those groups, if you post the relevant section, we might be able to help you further.