GPU Passthrough Crashes on UnRaid

Fixing RTX 3090 VFIO Passthrough Crashes on Unraid: QEMU, IOMMU and Physical Address Width

I recently ran into a particularly awkward GPU passthrough problem on my Unraid server. The machine is based on an AMD Threadripper 3960X with 128 GB RAM and an NVIDIA RTX 3090, with the 3090 passed through to a Windows 11 VM.

GPU passthrough initially appeared to work. Windows booted, the NVIDIA driver loaded, the desktop was stable, and even 4K video playback worked normally. However, as soon as the GPU was placed under substantial 3D load, it disappeared from the host and the VM failed.

Symptoms

Running Unigine Superposition at 1080p Medium caused the display to go black almost immediately. The Unraid/QEMU logs then contained errors along these lines:

vfio: Unable to power on device, stuck in D3 vfio_container_dma_map(..., 0x383000000000, 0x10000000, ...) = -14 (Bad address) 0000:4d:00.0: PCI peer-to-peer transactions on BARs are not supported. error: kvm run failed Bad address

Afterwards the GPU would often be inaccessible to VFIO, with failed D0/D3 power transitions, failed resets and PCIe link recovery attempts. A complete host power cycle was sometimes required before the card could be passed through again.

Mapping additional and network drives inside an unRaid Windows VM

On the unRaid forums recently, I was responding to a user who was asking why his VM gcow2 image did not shrink when files are removed (thread here). I advised that auto-shrink is not standard behaviour, and in the discussion, it became clear that the use case was security camera footage. The user figured out that perhaps saving video files to a network share would be better than to the VM boot / system drive, and I mentioned that I do this with my BlueIris setup. I promised a quick guide, so here it is....

Problems passing through GPU to an unRaid VM?

One of the most common problems I see people seeking help for on unRaid forums, reddit and discord is the thorny issue of GPU pass-through. People with the same problems tend to ask the same questions over and over, and, somewhat tired of providing the same troubleshooting steps over and over, I thought I'd write up the main pitfalls, and possible solutions.

Here are some steps to check before reaching out for help.

1. Have you done your research?

As noted, there's a ton of threads on unRaid forums covering this, many successfully resolved. Do conduct a search there. For example, searching 'problem with gpu passthrough' today produces 40 pages of results. Chances are, your issue has been raised and resolved several times before.

Another oft-posted remark is something along the lines of 'I followed spaceinvaderone's video, and it still doesn't work'.

In most cases where I've seen this, the user has incorrectly followed the video - missing a key step or subtle point. This is NOT a straightforward process. There are many variables involved and SpaceInvaderOne's videos are packed with information. He does an amazing job of explaining the complex process step by step, so make sure to rewatch these videos several times to ensure you haven't missed anything.

If you haven't been following one of his videos for GPU pas-through, why not? Get very familiar with the process and follow along before posting queries;

Here are some links to his channels;

https://forums.unraid.net/profile/67288-spaceinvaderone/

https://www.youtube.com/channel/UCZDfnUn74N0WeAPvMqTOrtA

https://www.reddit.com/user/spaceinvaderone/

Research done? Let's look at some of the common missteps....

Solving the -43 issue for GPU passthrough

I've usually built my VMs with AMD GPUs when passthrough was needed. I happened to have an old GeForce GTX770 lying around, but found it frustratingly difficult to get it working in a Windows 10 VM. nVidia drivers apparently really, really don't like running in a VM and cause the display device to show a -43 error.

For me, after a significant amount of tweaking and trying, I finally found a combination that worked;

Machine Type: Q35 4.2

Bios: OVMF

Graphic ROM Bios (techpowerup download, hexedited to remove header)

And the following edits to the </features> section of the XML. ˜The highlighted lines are added in addition to the standard unRAID config;

  <features>

    <acpi/>

    <apic/>

    <hyperv>

      <relaxed state='on'/>

      <vapic state='on'/>

      <spinlocks state='on' retries='8191'/>

      <vendor_id state='on' value='1234567890ab'/>

    </hyperv>

    <kvm>

      <hidden state='on'/>

    </kvm>

    <ioapic driver='kvm'/>

  </features>

unRAID discrete USB Passthrough to multiple VMs

For a while, I've been working towards consolidation of machines into VMs on my unRAID system.  unRAID is great for this as it allows for hardware passthrough of devices such as GPUs, USB adapters and other items. Passthrough of discrete USB controllers is especially important. unRAID does allow direct device passthrough, but I've found this unsatisfactory for a number of reasons. Mainly, it's not a great solution for high-bandwidth devices such as audio cards. Also, it's not really plug'n'play, with hot swapping being somewhat patchy.

Passthrough of the full controller solves these problems, One of the challenges of this approach is however, physical capacity for discrete devices. With each VM requiring a PCIe GPU and a USB adapter, it's possible to run out of slots fairly quickly. Yes, it's possible to use on-board controllers, but that's not always a full solution and PCIe cards are often required in addition.

Another approach is to use multi-controller PCIe USB adapters. These are cards the occupy a single slot, provide multiple ports and. crucially, have multiple on-board USB controllers that would, in theory, permit each controller to be passed through to a separate VM.

One such card is the Sonnet Allegro USB3-PRO-4PM-E, sadly now discontinued. There are alternatives, but as users in this thread are finding out, broad compatibility is patchy.

As an alternative, I've been using a PCIe expansion kit to pass through discrete controllers to VMs. These devices were all the rage during the mining boom as they allowed users attach multiple GPUs to a single PCIe slot in order to leverage compute power. They work by essentially switching multiple PCIe ports through a single 1x motherboard port. Although the device sports four 16x mechanical slots, each is in itself only a 1x slot - but that's all that's needed for a USB controller.

Splashtop Latency Demo

A user over at unRaid Forums asked me about latency when using SpalshTop Personal to access an OSX VM running on unRaid. I promised a quick video to show it in action. Sorry about the delay!

Enabling HEVC on Rx570 in MacOS VM

In following the MacOS Bare Metal thread on unRaid forums, it was bugging me that I wasn't getting full performance from my VM GPU. I have a Sapphire RX570 4GB passed through to a Mojave VM, and while it was showing Metal and H.264 encoding support, HVEC remained resolutely elusive.

It should be simple, enable Lilu and WhateverGreen kexts and go. It turned out to be anything but.

VM Audio Solution

When setting up VM access over ethernet using the AV Access device, I noted that one issue was the failure of my Harman Kardon Soundsticks to work with the system. Plugging them in to the receiver USB port would result in intermittent but regular screen blanking.

A few readers suggested additional power and I purchased a USB power/data cable, but that didn't work.

Next, I tried a cheap and cheerful external USB soundcard. I plugged this into the receiver and attached some powered speakers. This worked, in that my OSX VM recognised the device immediately and could play audio, but it was annoyingly distorted - just enough crackle & pop to distract.

Next, I tried attaching the USB soundcard directly to the (passed through) USB adapter on the unRaid box. I then passed the audio out to the AV Access transmitter audio in and attached the speakers to the receiver audio out.

Hey Presto, perfect audio*. No crackles, pops, noise or distortion. The lesson here is that with this method of video/IO/audio distribution, the soundcard needs to stay upstream. My SoundSticks are now moved to another workstation where they can be attached to that VMs USB adapter directly.

My final working audio chain is;

PCIe USB Adapter -> USB Audio Card -> AV Access TX Audio In -> AV Access RX Audio Out -> Speakers

*When I say 'perfect', I mean adequate for needs in that system sounds and youtube audio plays just fine. I wouldn't be using this set up for anything close to critical listening.

MediaServer 8.2

It’s been over 5 years since I built MediaServer8, an unRaid based system that looks after not only storage duties, but also hosts my main workstation, manages whole-house audio systems, whole house LiveTV and PVR, home automation and a bunch of other stuff through dockers, VMs and plugins.

The AMD FX8320 based system has been rock solid in all of that time. Indeed, the only problem I’ve ever had occurred in the past week when my dual 1TB spinners assigned to a cache pool started showing millions of unrecoverable BTRFS errors. Fortunately, I rescued my appdata and domains folders in time. My docker image was not so lucky :-(

A few months ago, the itch emerged and, try as I might, I found it difficult to resist the urge to scratch. There was nothing wrong with the system per se, but with ‘only’ 8 cores and 32GB RAM, it was a bit restrictive in the number of VMs I could run simultaneously, and no real scope for growth. I had a particular hankering to virtualise the dual boot Hackintosh the family use daily, as well as a few other utility systems. Threadripper was calling. Here’s how I answered....

Routing VMs anywhere

I'll start my saying that these things are freaking amazing!

I've been running a few VMs on my unRaid box for a while now. Specifically my main workstation (Win10 /OSX switchable), as well as some utility boxes.

One of the challenges has been physical location and signal transfer - my unRaid server resides in my attic and any VMs that required an interface (display & USB), would need to be located within the reach of a HDMI run.

This ruled out virtualising some systems, including a Win 10 / OSX dual boot hackintosh that's the main system the family uses. It's miles away from the server with no hope of running cables.

I'd toyed with HDMI over ethernet before, with little success. Recently, however, I came across the Av Access range of products...

Blog post #100!

So this week I mined my first cryptocurrency on my unRAID setup and the bug has well and truly bitten!

I thought I'd missed the bitcoin boat as I never got into mining and now bitcoin requires specialised hardware for mining. However, browsing AVForums I came across a thread from a member inviting participation in an OCUK forums mining project.

Looking into it a bit more, it turns out there's a whole altcoin market out there and there are still coins that can be mined on lowly PC hardware. Who knew! Given that I have an unRAID server sitting running 24/7, I figured I'd give it a go to see how it worked.

It's a bit of a learning curve but I got up to speed in a few evenings.

Lost Weekend

Now that Xen is slated for total removal from unRaid, it was time to seriously look at migrating to KVM. The easiest solution would probably have been to start from scratch and re-create all my VMs under KVM but that would require reactivating windows VMs, downloading all the updates, re-installing and re-configuring software etc.

The more attractive route was finding a way to migrate existing VMs from Xen to KVM. Thanks to jonp from LimeTech who published a how-to guide, this become fairly straightforward. I even recorded myself following the guide. Having done my Windows 7 VM, I then set about two Windows 8.1 systems and a Netrunner (ArchLinux / Manjaro) VM.

The Windows 8.1 migrations were as straightforward as the Win7 one had been. However, the Linux one just wouldn’t work for me. Once I’d set it up in KVM, pointed to the disk image and booted, I was greeted with a Grub error indicating that the root disk UUID could not be found. A quick check indicated that the UUID was in fact the same and no amount of Google Fu revealed a solution. I ended up re-installing the OS and dealing with a world of hurt in the updates (lot’s of conflicting dependencies).

So, with 4x VMs migrated, it was time to start playing with my shiny new KVM flavour of UnRaid. With a full weekend of playing around under my belt, I can now report that some things worked well, some didn’t. Some things I liked, some not so much;

Converting Xen VM to KVM

In recent days, it's become clear that LimeTech plan to support only KVM virtualisation in unRAID 6 (as well as Docker). When they started out with version 6, Xen was the only virtualisation available and that's the one I went with.

I've been anxious about the learning curve required to migrate to KVM but since Xen is likely to disappear sooner rather than later, I figured it was time to bite the bullet.

LimeTech have published a 'how to' guide on migrating a Win7 VM from Xen to KVM so I decided to record myself following it;

Overall, it's a breeze with just a couple of small hiccups, one relating to not being clear on how to set up KVM support in unRAID and the other not figuring out how to ad an additional virtual disk in the UI. Apart from that, all good - though the new VM UI doesn't seem to support all PCI/PCIe devices for passthrough so I need to figure out how to do that now...