Enabling HEVC on Rx570 in MacOS VM

In following the MacOS Bare Metal thread on unRaid forums, it was bugging me that I wasn't getting full performance from my VM GPU. I have a Sapphire RX570 4GB passed through to a Mojave VM, and while it was showing Metal and H.264 encoding support, HVEC remained resolutely elusive.

It should be simple, enable Lilu and WhateverGreen kexts and go. It turned out to be anything but.


My first challenge was that I couldn't even get Lilu and WhateverGreen (WEG) to load from clover. My Mojave VM has been around for a while. I configured it over a year ago and it's been rock solid, even surviving a few OS updates.

I run three displays off it, a couple of Dell U3014 at 2560x1600 and a Samsung 1200x1920 (portrait).  I did manage to get WEG to load when I loaded into the OS Library/Extensions folder using HackinTool, but that wasn't as flexible as dropping into and maintaining in in Clover/extensions/other.

There is a debate online as to whether its better to inject kexts via Clover, or load them from Library/Extensions. The later seems to be preferred as the kexts run in OS protected memory space, and can be more stable. However, as I'm experimenting with various kext versions and configurations, I find it much faster to use Clover injection. Once stabilised, I may transfer to Library/Extensions later.

As a note to my future self, here's how to rebuild kext cache once kexts are dragged out of Library/Extensions; 

$ sudo -s 

touch /System/Library/Extensions

touch /Library/Extensions

kextcache -i /

I decided to try updating Clover from v4xxx to v5xxx, but this failed as the system just sat at the 'scanning devices' screen. It turns out, there's a compatibility issue between the version of QEMU included in unRaid and newer versions of clover.

In the end, I bit the bullet and went for a full system reinstall.

I run this VM OS on a 240GB SSD passed through. So first task was to back this up. I shut down the VM and use the following command in unRaid consolce to copy the SSD to an image file;

qemu-img convert -p -O raw /dev/sdX /mnt/user/backups/vdisk1.img

(where sdX is the id of my SSD drive, and the image path is a valid share on my system)

With the backup completed, I then attached the image to my Mojave VM as the boot volume and did a test boot to ensure the copy was OK.

I backed up my working VM XML for reference and deleted the VM. I then used the excellent Macinabox docker to set up a new VM. Before booting, I edited the XML to point to my SSD rather than the created image file and added in my CPU configuration. I left video on VNC for the moment and booted up my fresh, shiny new VM.

I elected to stay on Mojave for a few reasons. Mostly, this is my main day to day machine, and it's been very reliable on Mojave for some time. I do have another family system that I've updated to Catalina so can use that for cutting edge stuff.

With the VM up and running, I added SpalshTop for remote access and ran through a few system updates to get 10 10.14.6 and all security patches etc. With everything updated, I edited the VM XML to remove the Mojave installer images, and added back my previously backed up old system image.

Booting into the VM, I then used the Migration Assistant utility to restore all my previous user account, apps and files.

So now I was back where I started, except Clover was updated to v 5xxx on the boot EFI and I could finally get Lilu/WEG to load via clover.

I used IORegistryExplorer to check if WEG had loaded. There are other ways via command line, but I found this tool convenient, as I needed to use it to check something else as well (more later). To check, simply search 'What' and, if it's there, it will appear something like this;

So, with that in place, there was still no dice with HEVC encoding support. I was using the demo version of VideopProc to check this. (Just go to Setting -> Hardware Accelerate -> Options and click on the refresh button next to the HVEC panel)

After a bit of research, it looked like I had two key problems. One was that, apparently, support for HEVC in a MacOS VM on unRaid requires the machine type to be set to iMacPro 1,1 in Clover SMBIOS. Apparently, the new MacPro 7,1 SMBIOS works as well, but not for me.

The challenge I had here was that, when I set my SMBIOS to iMacPro 1,1, only one of my GPU ports would work, and I had to unplug 2x of my screens to even have that display work.

My other problem was that, apparently, for HEVC to work, my GPU needed to be configured under GFX0 in the system registry. WEG is supposed to look after this but, for me, nothing I could do would force it to change from 'S28@5'

Ultimately, here are magic combinations of settings I found to resiolve these issues....


1. Set SMBIOS to iMacPro 1,1

2. Use Clover Acpi -> Rename devices to force GFX0

3. Add agdpmod boot argument

4. Add HEVCEnabler.kext

Et Voila;

I'm unsure if the last 2 are actually required, but I've spent so long tinkering with this, I'm really not inclined to remove anything. Even as it stands, boot seems a bit delicate in that after TianoCore splash, nothing gets displayed on any screen. They all flick on and off a bit. The problem seems to be the display connected to DisplayPort.

My card only has 3x ports, DP, HDMI and DVIU. I have the two 30" connected to DP and HDMI, with the smaller screen on DVI via a HDMI adapter. When I connect all three, the DP screen seems to interfere with things. I find if I power it off close to boot completion, then power it on again, I gain access to the desktop across all three. Otherwise, they all stay black :-(

I'm not too concerned about this manual intervention, as I leave this VM on all the time and just power off the displays when not in use. Would like to solve it though.

UPDATE: Replaced my RX570 with an XFX RX580 8GB. HEVC is enabled with no further intervention.