You use Linux because you’re a good person. But when it comes to video games, you’re a little more pragmatic. So you bought an Nvidia card and set up a Windows VM. Then Valve announced their new Steam Play feature, and you’ve considered gaming on Linux. Not on your host, of course. Moving your GPU up a slot would be too difficult, so how about you make a second VM for Ubuntu?

Nvidia doesn’t get along with virtualization or Linux, so this guide should help you corral everyone until they get along. This setup was performed on a Fedora 28 workstation running on 1st gen Ryzen hardware, with an Nvidia GTX 1080 passed through to a guest running Ubuntu 18.04.

Virtual Machine Setup

First up, download an ISO file from Ubuntu’s website.

In virt-manager, create a new machine. Select the previously downloaded ISO file as your install method. Customize the system as you like until you hit step 5, then check the box to customize your configuration.

For the firmware, select OVMF. Assuming you have set up Windows before, this should already be an option. If this is your first VFIO setup, why are you reading this guide? Also, the arch wiki should help. And if you have Ryzen hardware, Level1Techs have a great guide as well. You may have to skip this step, but the install won’t boot without OVMF anyways.

Set the chipset to q35. This is, strictly speaking, optional. But I’ve found the i440FX chipset causes problems with Nvidia’s drivers, especially the proprietary ones.

Under CDROM, set the disk bus to SATA.

Select add hardware and add your graphics card. Typically it will be listed as two devices: a VGA component and an audio component. If you have a dedicated hard drive for this VM, be sure to add that here as well.

Remove DisplaySpice and Video QXL.

Ideally, you want 2 networks: bridge network and a virtual network. The former allows the guest to connect to the internet. The latter allows rapid connection from host to guest, which is useful for Steam’s In-Home streaming. It eliminates the unneccesary middleman role your router would normally play.

One of these should have been setup by default when you create the VM. For the bridge, set the network source to your host device (in my case, eno1). For the virtual network, set it to virtual network default.

Now add some peripherals and boot up. You should see a grub prompt in the console. Select “Install Now” and the Nvidia monitor should display shortly.

Ubuntu Installation

You know how to install Ubuntu. We’re not covering this.

After install, power off the VM and disconnect the iso file from the CDROM device.

Post Install VFIO Configuration

At this point, Ubuntu will launch, but some VFIO optimization is required before switching to Nvidia’s proprietary drivers.

Edit the XML configuration and implement the error 43 fix. Otherwise, that’s going to cause a kernel panic if you decide to use the proprietary drivers. Find the features tag and add hyperv and kvm tags to it, like so:

    <vendor_id state='on' value='SUCKMYDICKNV'/>
    <hidden state='on'/>
  <vmport state='off'/>

Linux has been known to be picky about it’s PCIe topology, so if your GPU has separate VGA and audio devices, you can opt to put them on the same slot. I’ve encountered audio issues in this VM, and although I haven’t fixed them (in software), any hypothetical fix likely involves this modification. Find the lines below:

<hostdev mode='subsystem' type='pci' managed='yes'>
    <address domain='0x0000' bus='0x1e' slot='0x00' function='0x0'/>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
    <address domain='0x0000' bus='0x1e' slot='0x00' function='0x1'/>
  <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>

Your buses and slots may not match. Change the bus and slot of the 2nd device to match the first, and assign it function 1, and enable multifunction on the first device.

<hostdev mode='subsystem' type='pci' managed='yes'>
    <address domain='0x0000' bus='0x1e' slot='0x00' function='0x0'/>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
    <address domain='0x0000' bus='0x1e' slot='0x00' function='0x1'/>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>

You can also take this opportunity to change the CPU topology. I’m lazy though, and I’ve never seen the need.

Steam, Nvidia, and CUDA

Steam has an official installer in Ubuntu’s software store. Install that and perform a software upgrade.

To install Nvidia drivers, you’ll need to boot in commandline, to do this, enter

sudo systemctl set-default

and reboot. After login, you can follow the instructions to install Nvidia 410 and CUDA in this forum post. Run the following command and reboot to restore your graphical settings

sudo systemctl set-default

Ubuntu now natively supports the proprietary Nvidia drivers, just open the Software and Updates app, select the “Additional Drivers” tab, and select the recommended proprietary driver. Apply changes and reboot.

Audio Issues

I encountered a lot of audio problems with Ubuntu when virtualized. The traditional passthrough via ich9 was completely useless. Nvidia 390 had no audio when using HDMI. Nvidia 410 had HDMI audio, but with stuttering pops. In the end, I just used an external hardware solution, but if you’re insistent on a software fix, you can pick up where I left off:

If your card is using Intel audio drivers for some reason

Pulseaudio passthrough

EDIT: AMD Issues with Fedora 29

After updating to Qemu 3.1, the topoext id flag was disabled by default, which disables hyperthreading on AMD CPUs. You can read about it here on the Arch Wiki. To rectify this, edit your virsh xml to include:

 <cpu mode='host-passthrough' check='none'>
 <topology sockets='1' cores='4' threads='2'/>
 <feature policy='require' name='topoext'/>

EDIT: X370 BIOS Updates break VFIO. Read if you have an ASRock X370 Taichi

Note that more recent (June 2019) X370 BIOSes are starting to include AGESA. What is that? Fuck if I know. But AGESA breaks VFIO. Anyone who updates their BIOS and can no longer get a screen on a VM should roll their BIOS version back to a 2018 version.

I’ve been told that on the ASRock X370 Taichi, updates to 5.5+ are irreversible. So consider not updating in the first place.

EDIT: VFIO Fails after Win10 update to 1903

I know this post is about Linux VMs, but no one reads my blog, so it’s really just for my reference. Plus, if the reason you’re switching to Linux is that your Windows 10 VM bricked itself after updating to 1903, run the following command:

echo 1 > /sys/module/kvm/parameters/ignore_msrs

Try booting. If Windows magically works, create a new conf file in modprobe with the content:

options kvm ignore_msrs=1

Props to tinfoil over on the Level1Techs forum for finding this fix