You use Linux because you’re a good person. But when it comes to video games, you’re a little more pragmatic. So you bought an Nvidia card and set up a Windows VM. Then Valve announced their new Steam Play feature, and you’ve considered gaming on Linux. Not on your host, of course. Moving your GPU up a slot would be too difficult, so how about you make a second VM for Ubuntu?
Nvidia doesn’t get along with virtualization or Linux, so this guide should help you corral everyone until they get along. This setup was performed on a Fedora 28 workstation running on 1st gen Ryzen hardware, with an Nvidia GTX 1080 passed through to a guest running Ubuntu 18.04.
Virtual Machine Setup
First up, download an ISO file from Ubuntu’s website.
In virt-manager, create a new machine. Select the previously downloaded ISO file as your install method. Customize the system as you like until you hit step 5, then check the box to customize your configuration.
For the firmware, select OVMF. Assuming you have set up Windows before, this should already be an option. If this is your first VFIO setup, why are you reading this guide? Also, the arch wiki should help. And if you have Ryzen hardware, Level1Techs have a great guide as well. You may have to skip this step, but the install won’t boot without OVMF anyways.
Set the chipset to q35. This is, strictly speaking, optional. But I’ve found the i440FX chipset causes problems with Nvidia’s drivers, especially the proprietary ones.
Under CDROM, set the disk bus to SATA.
Select add hardware and add your graphics card. Typically it will be listed as two devices: a VGA component and an audio component. If you have a dedicated hard drive for this VM, be sure to add that here as well.
Remove DisplaySpice and Video QXL.
One of these should have been setup by default when you create the VM. For the bridge, set the network source to your host device (in my case, eno1). For the virtual network, set it to virtual network default.
Now add some peripherals and boot up. You should see a grub prompt in the console. Select “Install Now” and the Nvidia monitor should display shortly.
You know how to install Ubuntu. We’re not covering this.
After install, power off the VM and disconnect the iso file from the CDROM device.
Post Install VFIO Configuration
At this point, Ubuntu will launch, but some VFIO optimization is required before switching to Nvidia’s proprietary drivers.
Edit the XML configuration and implement the error 43 fix. Otherwise, that’s going to cause a kernel panic if you decide to use the proprietary drivers. Find the features tag and add hyperv and kvm tags to it, like so:
<features> <acpi/> <apic/> <hyperv> <vendor_id state='on' value='SUCKMYDICKNV'/> </hyperv> <kvm> <hidden state='on'/> </kvm> <vmport state='off'/> </features>
<hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x1e' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x1e' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </hostdev>
Your buses and slots may not match. Change the bus and slot of the 2nd device to match the first, and assign it function 1, and enable multifunction on the first device.
<hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x1e' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x1e' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/> </hostdev>
You can also take this opportunity to change the CPU topology. I’m lazy though, and I’ve never seen the need.
Steam, Nvidia, and CUDA
Steam has an official installer in Ubuntu’s software store. Install that and perform a software upgrade. To install Nvidia drivers, you’ll need to boot in commandline, to do this, enter sudo systemctl set-default multi-user.target and reboot. After login, you can follow the instructions to install Nvidia 410 and CUDA in this forum post. Run the following command and reboot to restore your graphical settings sudo systemctl set-default graphical.target
Ubuntu now natively supports the proprietary Nvidia drivers, just open the Software and Updates app, select the “Additional Drivers” tab, and select the recommended proprietary driver. Apply changes and reboot.
I encountered a lot of audio problems with Ubuntu when virtualized. The traditional passthrough via ich9 was completely useless. Nvidia 390 had no audio when using HDMI. Nvidia 410 had HDMI audio, but with stuttering pops. In the end, I just used an external hardware solution, but if you’re insistent on a software fix, you can pick up where I left off:
EDIT: AMD Issues with Fedora 29
After updating to Qemu 3.1, the topoext id flag was disabled by default, which disables hyperthreading on AMD CPUs. You can read about it here on the Arch Wiki. To rectify this, edit your virsh xml to include:
<cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='4' threads='2'/> <feature policy='require' name='topoext'/> </cpu>
EDIT: X370 BIOS Updates break VFIO. Read if you have an ASRock X370 Taichi
Note that more recent (June 2019) X370 BIOSes are starting to include AGESA. What is that? Fuck if I know. But AGESA 0.0.7.2 breaks VFIO. Anyone who updates their BIOS and can no longer get a screen on a VM should roll their BIOS version back to a 2018 version.
I’ve been told that on the ASRock X370 Taichi, updates to 5.5+ are irreversible. So consider not updating in the first place.
EDIT: VFIO Fails after Win10 update to 1903
I know this post is about Linux VMs, but no one reads my blog, so it’s really just for my reference. Plus, if the reason you’re switching to Linux is that your Windows 10 VM bricked itself after updating to 1903, run the following command:
echo 1 > /sys/module/kvm/parameters/ignore_msrs
Try booting. If Windows magically works, create a new conf file in modprobe with the content:
options kvm ignore_msrs=1
Props to tinfoil over on the Level1Techs forum for finding this fix