One Box To Rule Them All: GPU Passthrough Crashes on UnRaid

Fixing RTX 3090 VFIO Passthrough Crashes on Unraid: QEMU, IOMMU and Physical Address Width

I recently ran into a particularly awkward GPU passthrough problem on my Unraid server. The machine is based on an AMD Threadripper 3960X with 128 GB RAM and an NVIDIA RTX 3090, with the 3090 passed through to a Windows 11 VM.

GPU passthrough initially appeared to work. Windows booted, the NVIDIA driver loaded, the desktop was stable, and even 4K video playback worked normally. However, as soon as the GPU was placed under substantial 3D load, it disappeared from the host and the VM failed.

Symptoms

Running Unigine Superposition at 1080p Medium caused the display to go black almost immediately. The Unraid/QEMU logs then contained errors along these lines:

vfio: Unable to power on device, stuck in D3
vfio_container_dma_map(..., 0x383000000000, 0x10000000, ...) = -14 (Bad address)
0000:4d:00.0: PCI peer-to-peer transactions on BARs are not supported.
error: kvm run failed Bad address

Afterwards the GPU would often be inaccessible to VFIO, with failed D0/D3 power transitions, failed resets and PCIe link recovery attempts. A complete host power cycle was sometimes required before the card could be passed through again.

Eliminating the Obvious Causes

This initially looked like a GPU reset, power-management or PCIe stability problem. I therefore worked through the likely causes one at a time.

Reduced the VM from 64 GB to 32 GB RAM.
Removed hugepages from the test VM.
Removed additional passed-through/virtual disks.
Created an entirely new Windows 11 VM with fresh OVMF/NVRAM.
Installed Windows and NVIDIA drivers from scratch.
Reseated the RTX 3090 and checked its power connections.
Forced its PCIe slot from Gen4 to Gen3.

None of these fixed the problem.

The particularly useful test was booting Windows 10 directly on the server hardware, without Unraid, KVM or VFIO involved. The same RTX 3090 then completed the same Superposition benchmark at 100% GPU utilisation without difficulty.

That strongly suggested that the GPU, motherboard slot, PSU and basic PCIe signalling were sound, and that the fault was somewhere in the VFIO/QEMU path.

The Important Clue: BAR1

Inspecting QEMU's live PCI map finally exposed the interesting detail. The RTX 3090's 256 MB BAR1 had been assigned this guest address:

BAR1: 64 bit prefetchable memory at
0x383000000000 [0x38300fffffff]

That address was particularly significant because it exactly matched the address in the VFIO failure:

vfio_container_dma_map(...,
    0x383000000000,
    0x10000000,
    ...) = -14 (Bad address)

The size is significant too: 0x10000000 is 256 MB, exactly the size of BAR1.

The Threadripper 3960X host reports:

Address sizes: 43 bits physical, 48 bits virtual

In other words, QEMU was placing the GPU's 64-bit MMIO BAR at an extremely high guest address, and VFIO was then failing when it attempted to map that BAR through the host IOMMU.

The Fix

Libvirt/QEMU allows the physical address width exposed to the guest to be constrained. For this machine I added the following line to the VM's CPU definition:

<maxphysaddr mode='passthrough' limit='43'/>

The resulting CPU section looked like this:

<cpu mode='host-passthrough' check='none' migratable='on'>
  <topology sockets='1' dies='1' clusters='1' cores='8' threads='2'/>
  <maxphysaddr mode='passthrough' limit='43'/>
  <feature policy='require' name='topoext'/>
</cpu>

Libvirt translated this into the corresponding QEMU CPU options:

host-phys-bits=on,host-phys-bits-limit=43

What Changed?

After rebooting the VM, the RTX 3090's BAR1 had moved dramatically:

Before:
BAR1 = 0x383000000000

After limiting physical addressing to 43 bits:
BAR1 = 0x70600000000

The new address is below the 43-bit physical-address ceiling.

Result

I reran exactly the workload which had previously caused the GPU to disappear almost immediately: Unigine Superposition, DirectX, 1080p Medium.

This time the RTX 3090 reached 100% utilisation, completed the entire benchmark and remained fully operational:

Superposition 1080p Medium

Score:            2372
Average FPS:      17.75
Maximum GPU load: 100%
Maximum GPU temp: 77 C

Result: PASS

The Unraid kernel log remained clean: no vfio_container_dma_map failure, no Bad address, no failed D3 power transition and no PCIe link recovery.

I also shut the VM down normally and restarted it. The RTX 3090 reset, detached and was successfully reassigned to the VM again.

Conclusion

In this case the apparent NVIDIA/VFIO reset problem was actually an address-space problem. The D3 errors and PCIe recovery messages were consequences of the failed mapping rather than the original fault.

The giveaway was correlating the failing VFIO DMA-map address with QEMU's live PCI map. The failed address and size matched the RTX 3090's BAR1 exactly.

Constraining the guest physical-address width to the host's 43-bit physical addressing caused QEMU/OVMF to place the GPU BAR inside a usable address range and eliminated the failure.

Important: 43 bits is appropriate to this particular system because that is the physical address width reported by its CPU and proved successful in testing. It should not simply be copied to unrelated hardware. Determine the appropriate CPU/IOMMU address width for the host in question.

One Box To Rule Them All

Wednesday, 16 September 2026

GPU Passthrough Crashes on UnRaid