unRAID discrete USB Passthrough to multiple VMs

For a while, I've been working towards consolidation of machines into VMs on my unRAID system.  unRAID is great for this as it allows for hardware passthrough of devices such as GPUs, USB adapters and other items. Passthrough of discrete USB controllers is especially important. unRAID does allow direct device passthrough, but I've found this unsatisfactory for a number of reasons. Mainly, it's not a great solution for high-bandwidth devices such as audio cards. Also, it's not really plug'n'play, with hot swapping being somewhat patchy.

Passthrough of the full controller solves these problems, One of the challenges of this approach is however, physical capacity for discrete devices. With each VM requiring a PCIe GPU and a USB adapter, it's possible to run out of slots fairly quickly. Yes, it's possible to use on-board controllers, but that's not always a full solution and PCIe cards are often required in addition.

Another approach is to use multi-controller PCIe USB adapters. These are cards the occupy a single slot, provide multiple ports and. crucially, have multiple on-board USB controllers that would, in theory, permit each controller to be passed through to a separate VM.

One such card is the Sonnet Allegro USB3-PRO-4PM-E, sadly now discontinued. There are alternatives, but as users in this thread are finding out, broad compatibility is patchy.

As an alternative, I've been using a PCIe expansion kit to pass through discrete controllers to VMs. These devices were all the rage during the mining boom as they allowed users attach multiple GPUs to a single PCIe slot in order to leverage compute power. They work by essentially switching multiple PCIe ports through a single 1x motherboard port. Although the device sports four 16x mechanical slots, each is in itself only a 1x slot - but that's all that's needed for a USB controller.


My setup was living outside my case for about 9 months but, having successfully used it with two Inateck USB 3.0 2 port cards. I use these particular models as they are known compatible with MacOS, my primary operating system on 2x Workstation VMs I run in unRAID. They also have an internal USB 3.0 header that provides good placement flexibility.

It was time to perform some case surgery and get it set up properly inside my cavernous Thermaltake Core X9. I'd identified the highlighted area of the case as a good candidate as it is largely empty.

I ordered an A4 sheet of 3mm perspex and set about measuring and trimming to find the best place to mount the expansion card. A few holes drilled and I was able to mount the card using some M3 nuts and bolts.

Then I thought 'look at all that extra space on the right - something could be mounted there!'. I thought about some 2.5" drives, but figured that, as I'd be essentially adding a new vertical partition in my case, maybe some ventilation would be good. Out came the hole saw;

And with the hardware mounted;

And adapter cards in place;

Time to test-fit the expansion card in it's new home;

Adapters in place;

And all wired up;

The really nice thing about this solution is that it's possible to provide 4x VMs with multiple ports on a discrete controller, while occupying only one 'system' PCIe slot. USB devices can be wired directly to the card inside the case or, as I've done, I use USB 3.0 internal adapter cables to expose the ports directly on the rear of the case. An additional benefit here is I can use the 'spare' back-panel mounts between dual-spaced PCIe slots so I'm not even blocking any other slot.

And finally, if needed, I'm not restricted to implementing four USB adapters. I could add in any other 1x PCIe device, though for now USB is fine.

Here are the 4x controllers neatly arranged in their IOMMU groups;

And available for passthrough;

Now I just need to find out how to accommodate 4x GPUs!