MediaServer 8.3 - Bifurcation Edition

Some hardware updates, some minor mods and lots of cabling later, I've managed to get my unRAID server hosting eleven (13?) usable PCIe/PCI expansion cards which I'm using to manage storage devices and populate VMs with passthrough GPU, USB and Utility adapters.

All with 2x slots to spare on my 5x slot mainboard.

Here's how...


Having upgraded to a ThreadRipper 2950x based system,  and been tenacious about consolidating physical machines to VMs, I've been constantly hitting the physical limits of my system when it comes to the quantity of devices available for passthrough.

I have plenty of cores, just not enough expansion slots.

I've recently written about using a PCIe mining expansion system to solve the problem of exposing discrete USB adapters for VMs, but the challenge of GPUs and other devices remains. For example, I run a whole-house a/v infrastructure based around Emby, SqueezeServer & Kodi. The VM driving this needs a pair of PCI audio cards and a bunch of DVB-S2 tuners passed through, where can I fit those?

The X399 architecture typically provides up to 5 physical PCIe slots. My Taichi X399 provides a pair of x16 slots, a pair of x8 slots (x16 physical), and a x1 (which is actually IOMMU bundled with an set of other onboard devices, so is pretty useless for passthrough.)

I've had really good luck using M.2 NGFF SSD to PCI-e Express 4X Converter Adapter Cards to add additional PCIe cards, but those M.2 ports are useful for other things, like storage, so I was always keen to find another solution.

I've been following the Bifurcation thread over at Hard Forum with interest for a while. Bifurcation is the process by which a single 16x or x8 PCIe slot can be split into smaller channel groups for the purpose of driving multiple cards from the same slot.

For example, a single 16x slot can be split into two (8x8x) or four (4x4x4x4x). To achieve this, the motherboard needs to support bifurcation. It's usually set in the Bios. My MB allows me split either of the 16x PCIe slots.

That's not the whole story though. To make use of a bifurcated slot, you need a PCIe adapter or riser cable that physically divides the single slot into two or four slots as appropriate.

The cool thing about Bifurcation is that it actually creates 2x or 4x genuine split slots. Unlike the mining risers and expanders that use PLX technology which works more like a switch, toggling the split devices on and off in sequence so that really only one is connected at a time, a bifurcated slot is, for all intents & purposes, a truly split slot with all connected devices having full access to the lanes they are allocated.

A challenge is that, while a motherboard may support bifurcation, finding a compatible riser can be difficult as they are pretty rare. There's a guy on the Hard Forum thread that has an online store selling his custom designs. However, I found another site almost by accident and I really liked the look of the design and a lot of the features, so I invested in a MaxCloudOn.com 16x -> 4x4x4x4 riser.

NOTE: The vendor webshop hasn't opened yet, but you can order on eBay.

I really like this approach for a number of reasons;

• The child PCIe cards are separate from each other, which allows for more flexible placement within the server.
• The system uses SFF 8087 cables, similar to those used by SAS expanders, so good re-usability potential.
• They are white!

On receipt, I popped it right in to one of my PCIe 16x slots which I'd bifurcated as 4x4x4x in the bios and started adding in various cards, including GPUs. Everything worked perfectly, so much so that it was time to start planning my layout.

The Core X9 I have is a generous case, but even so, with all the hardware I need to fit, it was going to be a tight squeeze. I though about reverting to a set up I had a few years ago, where I'd externalised my StartTech PCIe expander and some hard drives to an external case. The concept worked well at the time, but I had a lot of trailing wires between the two boxes. I wanted to look at this again, but do it right.

I again turned to my Silverstone Grandia GD07 as the external chassis, but this time, I gave it  a dedicated PSU and eliminated power tethering by using a PSU Jumper Bridge that permits the secondary system to power up without motherboard connections.

The systems are tethered using a couple of Dual Port Mini SAS SFF-8088 to SAS 36Pin SFF-8087 PCBA Adapters and a DVI cable connecting the offboard innards of my trusty Startech  Expansion system with it's onboard host adapter (re-located to a bifurcated slot).

This quick 'n' dirty diagram shows the general set up of the system across my main system and secondary chassis, with a focus on the PCIe slot usage. (click to expand to full size);

You'll see that connected to my Bifurcation adapter in motherboard PCIe slot 1 is

1. An old HD5450 GPU.
Since this is connected to the first breakout slot, it's essentially the first GPU enumerated by the system and therefore becomes my unRAID boot display device. Prior to this, I'd got an RX 570 in this slot. My MB insists on booting with the first GPU it finds, and so it used that card, which I'd later need to passthrough to a VM, thus losing unRAID console. Problem now solved.

2. Startech Expansion Adapter
This is the 1x PCIe card that connects to my (now externalised) Startech Expansion board. This provides legacy PCI slots for my M-Audio cards which drive whole house audio and a 1X PCIe Digital Connection DVB host adapter. All of these are passed through to a 'homecontrol' Win10 VM (headless)

3. AMD RX 570 GPU
This is passed through to an OSX Workstation VM, (with occasional Win10 substitution). Yes, in this configuration, it's only running in an effective PCIe 3.0 x4 slot, but as the VM is not used for graphics-intensive purpose or gaming beyond Cities Skylines, that's not a significant problem.

4. 1x PCIe expansion adapter
Connects via a USB 3.0 cable to a PLX expansion board populated with 3x discrete USB 3.0 adapters that are each passed through to different VMs. (detailed in my previous post).  The 4th slot on the expander is now populated by an Intel res2sv240 SAS Expander. This doesn't actually use the PCIe bus, but is simply piggybacking on the slot for power. It's connected to an LSI 9211-8i Raid Controller that lives in the MB PCIe2 slot and between them,  the Dual Port Mini SAS and the MB SATA ports provide support for up to 28 SATA drives across both chassis.


There's a further RX 570 GPU in motherboard PCIe4 running at 16x. PCIe5 (8x) is free for future expansion!

Altogether, there are 14 expansion cards in this system, 16 if you count the two DuoFlex DVB-S2 daughter-cards associated with the Octopus. 3 of these are expanders of one kind or another, so in total there are 11 (or 13) functional expansion cards occupying 3 of the 5 available motherboard slots.

And so far, so stable.

In my research, I've seen occasional reference to a limit of 7 PCIe devices in the Threadripper 2xxx architecture. I haven't seen this limit in any practical way yet. I believe  this is because 2 of the expanders are essentially PLX devices, so the cards behind those only count as one in this limit calculation. So, in my current configuration, I've only used 6 of my available 7 devices, (LSI 9211-8i, RX570(1), HD5450, Startech Expansion Card, RX570(2), PCIe Expander)

Overall, I'm delighted with this setup. The MaxCloudOn bifurcation riser is an excellent addition, and unlocks the full potential of the rest of my hardware.

The only fault I have with it is that the downstream PCIe cards, while having one physical slot, will occupy two spaces on a PCI mounting bracket. This is fine if using dual width GPUs, but when populating them with single-width cards, it's a little less optimal. A lesser pitch when configuring them side by side in a case would be beneficial.

I've had some fun with the placement of the downstream bifurcated PCIe slots that involves woodwork and case modification. Too much for this article though. Stay tuned, I'll document it in the coming days. Here's a teaser;

Here's a breakdown of my IOMMU groups courtesy of VFIO-PCI Config plugin for anyone interested. (ACS override disabled).