Before you start
This may look like a long post at first, but in reality, it is but a few commands, the rest is output and small simple explanations, so don’t be discouraged by the length, it is really neither complicated nor lengthy.
Yet, you do need to check the hardware requirements before you get your hands dirty, you will find them in the “Minimum hardware requirements” section of this post
UPDATE 2024-07-31: 8 months down the line, Still using this, and it works like a charm, No issues at all
Why yet another tutorial
There are two reasons for this tutorial, Although there are many resources about every part of this hands on tutorial, they are all over the place, if you are new to Linux, you probably don’t know what to look for, The other reason is that all comprehensive tutorials assume you installed Linux with a GUI, this tutorial covers both with and without a GUI, for people who don’t need the Linux GUI, going without it saves them some precious ram and disk space.
In this tutorial, I assume you do not have a GUI on your Linux machine, but I also account for those who do, So some sections of this are for those who are running GUI, for example, how to disable hibernation, which is usually already disabled if you don’t use a GUI.
The problem
I use Linux for my software development and day to day work, this means I am out of luck if i wanted to use Adobe products, or play windows games, virtual machines are nice and all, but the graphics (VNC and RDP) are prohibitively weak for those applications. And I don’t really want another computer sitting on my desk, and last but not least dual boot is a hassle and has it’s limitations (Switching between my work machine and the adobe machine for the same task for example).
I also haven’t played any games in ages (Maybe last game I can say I played regularly was starcraft !! Yes the one developed in 1996), So i am excited about finding out what games look like at this stage.
The solution
Virtual machines can still be the solution, but with dedicated GPUs and USB ! passing the whole PCIe cards to the KVM guest. this is called PCI-PASSTHROUGH. I want to pass both my Amazing top of the line GPU, the NVIDIA GTX 1650 (I’m just kidding, I know it is old and entry level)… to the Windows guest OS, as well as my “VIA VL805/806 xHCI USB 3.0 Controller” and I then should be able to run all the games my heart desires ! I basically shouldn’t be able to tell that I am working on a virtual machine.
Because this tutorial should cover people who do not have a GUI installed, I will be setting up KVM machines with XML files, fixing networking with /network/interfaces, and so forth…
My Hardware
- A very old ASUS P9X79 motherboard with an I7 CPU (i7-4930K) and no internal GPU, You on the other hand might have an Internal GPU with your CPU which should spare you the need to install 2 GPUs on your motherboard’s PCIe slots.
- A 4K LG monitor (Relevant for the testing phase)
- Two graphics cards, NVIDIA GT 210 and NVIDIA GT 1650, you might only need one extrnal GPU if you have an internal GPU
- VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller PCIe card (Reports in windows having hardware IDs “PCI\VEN_1106&DEV_3483&SUBSYS_34831106&REV_01”
- IRRELEVANT: Water cooling (I will regret this soon) because I don’t want to hear loud fans, this way 3 radiators, 2 of them passively cooled, and one with a fan should cut down the noise considerably.
Minimum hardware requirements
- If you are trying to pass the second GPU of a laptop rather than a desktop, your laptop needs to have a Multiplexer for the HDMI port (Allowing you to pick the GPU connected to the physical HDMI connector), this is usually only available in high end and gaming laptops, so you will have to do your own research about the laptop you have.
- A motherboard and CPU that both support IOMMU (Called VT-d on intel, AMD-Vi: AMD IOMMUv2 on AMD), It should also be enabled in your bios (UEFI)
- A gpu that supports UEFI/IOMMU (most modern GPUs down to GT 710 do)
- A USB PCI-e card (If you want, just a preference of mine, but you can simply redirect ports if you so chose)
So, How I intend to go about that is simple, I have a very old NVIDIA GT 210 (Which will become my host operating system’s graphics card), and pass my 1650 graphics card to a Windows Guest machine to play games and use Adobe premier. I will also be delegating a USB3 PCI-e card to the guest system (Keyboard, mouse and god knows what else)
I will assume you know nothing about virtualization and very little about Linux and take you there step by step
1- go into bios/EFI and make sure VT-d and IOMMU are turned on.
2- Don’t install the other GPU and the USB card in yet, this step is not important, but it will help you avoid having linux install unnecessary drivers for them, so let us install the OS with one GPU *(Probably built in in most cases)
3- Install Debian 12, with or without GNOME (or pick whatever GUI you like, this tutorial is GUI agnostic).
4- Physically install the GPU and USB card in the computer case
5- Let us install a few packages, I use two types of virtualization, LXC and KVM, if you don’t want LXC, simply don’t install it
apt-get install lxc ntp ntpdate debootstrap logrotate net-tools apt-get install bridge-utils (No need, it is installed in the command above) apt-get install libvirt-daemon-system libvirt-clients apt-get install qemu-kvm virtinst nmap resolvconf
6- IF YOU HAVE GUI: Disable system suspend and hibernation
If you have a GUI, odds are it is also set to suspend or hibernate after some time, so if that is the case
6.1- Disable the following systemd targets
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
6.2- Now, if you want to make sure all is good, you need to reboot, then execute the following command
systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target
All 3 should now read “Active: inactive (dead)”
7- IF YOU HAVE GUI: Optional: Disable network manager, and enable /etc/network/interfaces
I am adding this step because this tutorial covers people with no GUI, so to make things persistent, This step makes both groups of people use the same tools, needless to say, it serves no functional purpose as network manager should do the job just fine.
7.1 – Compose a network/interfaces file
source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback auto br0 iface br0 inet static bridge_ports eno1 bridge_fd 0 address 192.168.7.120 netmask 255.255.255.0 gateway 192.168.7.1 bridge_stp off bridge_maxwait 0 dns-nameservers 8.8.8.8 dns-nameservers 8.8.4.4
7.2- disable network manager
systemctl stop NetworkManager
systemctl disable NetworkManager
8- Check if your system supports IOMMU (Intel VT-d, and AMD IOMMU)
On the linux terminal, Let us check if our CPU supports the things we need
grep --color -E "vmx|svm" /proc/cpuinfo
If, you see vmx (Intel), or svm (AMD) you are good to go, the command above displays them in color for your convenience, In my case, because I am using an Intel CPU, I can see VMX in red which is good news
9- Update GRUB
Now, we need to modify grub ! Grub is the boot loader for my Debian machine, so need to fix it as follows, I commented the existing and added a new line for clarity, In your case, you may just want to add the missing few characters ” intel_iommu=on”
Edit /etc/default/grub
# GRUB_CMDLINE_LINUX_DEFAULT="quiet" GRUD_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
Now run
update-grub
Or
grub-mkconfig -o /boot/grub/grub.cfg
Now, reboot your machine, after the reboot, use the following command to make sure it ends with intel_iommu=on
cat /proc/cmdline
10- Inspecting our PCI cards
Now, let us take a quick look at the PCI devices i have that have the word nvidia in them
lspci -nn | grep -i nvidia
And the results of the command above were
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2)
01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
As you can clearly see, every GPU comes with it’s own Audio Device ! in a minute you should be able to see how they are grouped together and need to be passed to the Guest OS together
11- Change the driver of the GPU to be passed to guests to vfio
This is all nice and dandy, we know plenty about our setup from the command above, but now the problem is that as soon as this machine starts up, those cards are grabbed by the drivers (nouveau by default), and that is not good, to overcome that, the graphics card that is meant to be passed to guests needs to be grabbed by the vfio driver. So let us tell that driver what we want it to grab
The card I want to assign to the Guest KVM machine is the 1650, according to the above, this card has devices, a VGA card and a sound card bearing the IDs 10de:1f82 and 10de:10fa ! to assign those a different driver, we have a few steps that need to be done
11.1- Edit the file /etc/modprobe.d/vfio.conf, and add a line similar to the following, the values in this line correspond to my own 1650 card from above, so you need to change those to match your card
options vfio-pci ids=10de:1f82,10de:10fa
11.2- We also want vfio to capture the card before any other driver, so we create the file “/etc/modprobe.d/nvidia.conf” and fill it with the following content
softdep nouveau pre: vfio-pci softdep nvidia pre: vfio-pci softdep nvidia* pre: vfio-pci
11.3- Now that vfio knows what card to pick, let us enable vfio with the following line
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
11.4- Now, edit the file /etc/modules, since we need a few modules for what is to come, add the following lines to it
# /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. vfio vfio_iommu_type1 vfio_pci vfio_virqfd
If your kernel was not compiled with vfio, you would have needed to run the command “modprobe vfio_pci”, but it is generally compiled in the kernel
11.6- and finally, you might want to restart the computer before the next step,
11.7- Now, in my case, When i checked (See 12), only the audio had been assigned the vfio driver ! I was forced to regenerate initramfs with the following command and reboot !
update-initramfs -u
12- Check that everything is good
let us explore our IOMMU groups and PCI devices etc… again, we want to see what devices have been moved to VFIO, the following command should show you what driver is being used in the “Kernel driver in use:” section, it also gives you more information such as the board manufacturer and IOMMU group the device belongs to (Every VGA card shares a group with it’s own audio adapter for example, and they must be passed together to the guest)
So, to recap, after running the following command, the VGA card you intend to pass to the guest and it’s audio device should have vfio_pci in its “Kernel driver in use:” section
lspci -vnn
In my case, the Geforece 210 GPU and it’s audio device were group 21, while the NVIDIA GEFORCE 1650 belonged to group 22, So the lines of interest would be
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. GT218 [GeForce 210] [1043:852d]
Flags: bus master, fast devsel, latency 0, IRQ 51, IOMMU group 21
Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel driver in use: nouveau
Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
Subsystem: ASUSTeK Computer Inc. High Definition Audio Controller [1043:852d]
Flags: bus master, fast devsel, latency 0, IRQ 54, IOMMU group 21
Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd TU117 [GeForce GTX 1650] [1458:3fcb]
Flags: fast devsel, IRQ 11, IOMMU group 22
Memory at f8000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at a0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at b0000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at d000 [disabled] [size=128]
Expansion ROM at f9000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Capabilities: [bb0] Physical Resizable BAR
Kernel driver in use: vfio-pci
Kernel modules: nouveau
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fcb]
Flags: fast devsel, IRQ 10, IOMMU group 22
Memory at f9080000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
13- Create the virtual machine, pass the PCI-e cards, and install windows.
In this example, I am making a 200GB hard drive in the location provided, assigning 12 gigabytes of ram, and starting it with a Windows installation DVD that i downloaded from microsoft’s website, and just because I can, I am leaving the VNC graphics adapter in, it simplifies my life.
virt-install –name gpu_win_adobe –os-variant win10 –ram 12288 –disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=200,sparse=yes –vcpu 6 –network bridge=br0 –hvm –graphics vnc,listen=0.0.0.0 –host-device 02:00.0 –noautoconsole –features kvm_hidden=on –cdrom /hds/12tb/Windows10_2023_11_21.iso –boot cdrom,hd
The machine should fire up the windows installer, and you can VNC to it via any VNC client of your choice, Now if you are doing this remotely from another let’s say windows machine, you can tunnel to your own machine and then VNC to it, I have written a post about doing this here
Now once the install is done, SHUT DOWN THE VIRTUAL MACHINE, and let us pass my GPU and USB cards to that machine, we do that by editing the file /etc/libvirt/quemu/gpu_win_adobe.xml , I have included the complete file that I use here for you so that you can compare and make changes, the parts that are noteworthy and relevant are the THREE <hostdev….> sections right before <memballoon model=’virtio’>, the first of which is my VGA card, the second is the sound card that comes with that VGA card, and the third being the USB controller
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit gpu_win_adobe
or other application using the libvirt API.
-->
<domain type='kvm'>
<name>gpu_win_adobe</name>
<uuid>968e01b8-8f35-4cd1-86b9-19839304912c</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode='custom'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='123456789123'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
<ioapic driver='kvm'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='on'>
<topology sockets='1' dies='1' cores='6' threads='2'/>
<feature policy='disable' name='hypervisor'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' discard='unmap'/>
<source file='/hds/12tb/virts_2024/kvm/gpu_win_adobe/main.qcow2'/>
<target dev='sda' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:7c:22:50'/>
<source bridge='br0'/>
<model type='e1000e'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<audio id='1' type='none'/>
<video>
<model type='vga' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>
Now, reflect the changes you made to the card with “virsh define /etc/libvirt/qemu/gpu_win_adobe.xml” then fire the mchine back up with “virsh start gpu_win_adobe“. sw
14- MSI interrupts
Within Windows, make sure you switch your video card and any other MSI interrupts capable card to MSI interrupts, some people have seen problems with regular interrupts where the host machine hangs as you switch off the guest which has the passthrough cards.
The easiest way to do this is with (MSI_util_v3.zip) which will edit the registry for you
15- Keyboard and mouse support
Even though I do have USB on the virtual machine (Guest VM), having more than 1 keyboard and 1 mouse on my desk is a problem from both an inconvinience and a space prespective, there are many solutions out there, one of them is a cheap hardware KVM for a couple of dollars, where we only connect the keyboard and mouse, and the other is software which is much more convinient. YET, if you are using wayland, those solutions will not work, hence, I am stuck with a hardware switch.
The popular software solution is synergy, but there is an alternative solution based on synergy that is already available for linux in the debian reporitories, you can find it here (Barrier) and later (Input Leap)
16- A useful script
The folowing script looks in the IOMMU groups folder, and displays what devices are in what group, so it will come in handy at one point, so i thought i’d add it here, you can skip this section, but there is a chance you will need this functionality once you want to do other things on your own
so create a new file named check_iommu_groups.sh and add the following script to it
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
Then run it like so (./check_iommu_groups.sh)
Once run you should be able to see the devices grouped together
IOMMU Group 22:
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
dmesg | grep -E "DMAR|IOMMU" dmesg | grep -i vfio dmesg |grep AMD-Vi (For amd) update-initramfs -u update-initramfs -u -k all
If you get the error [ 318.990400] DMAR: [DMA Read NO_PASID] Request device [02:00.0] fault addr 0x9e4b2000 [fault reason 0x06] PTE Read access is not set running dmesg | grep -E “DMAR|IOMMU”, odds are your vga card was captured by a different driver, otherwise If everything is okay, and we have not had any error or anything of the sort, we are ready to make KVM virtual machines
A virtual machine with no Passthrough first, this is my sandbox windows installation
virt-install --name windows_yazeed --os-variant win10 --ram 12288 --vcpu 6 --disk path=/hds/12tb/virts_2024/kvm/windows_yazeed/maint.qcow2,size=2000,sparse=yes --graphics vnc,listen=0.0.0.0 --noautoconsole --hvm --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd
Running it and connecting to it via VNC, done with windows installation, now another one, but this one is for the GPU passthrough
virt-install --name gpu_win_adobe --os-variant win10 --ram 12288 --disk path=/hds/12tb/virts_2024/kvm/gpu_win_adobe/dummy.qcow2,size=2,sparse=yes --vcpu 6 --network bridge=br0 --hvm --graphics vnc,listen=0.0.0.0 --host-device 02:00.0 --noautoconsole --features kvm_hidden=on --cdrom /hds/12tb/Windows10_2023_11_21.iso --boot cdrom,hd
But before we run this one, we will need to fix some stuff in the XML file ! /etc/libvirt/quemu/gpu_win_adobe.xml, so right before the <memballoon model=’virtio’> near the end, I added the needed parts based on my GPU’s position on the PCIe buss , I have included the full file so that you can compare my changes to the file generated by the command above
Whenever you change the xml file, you will need to tell libvirt about it, you can do that with the command
virsh define /etc/libvirt/qemu/gpu_win_adobe.xml
One useful thing to add to your virtual machine is an internal network that can not access the internet or the local network, something between the virtual machines, here are the instructions to that (Internal network for LXC and KVM)
A final note about power draw: VFIO does not manage power, it is more or less a dummy driver/placeholder, so your card will run hot until you assign it to a virtual machine which will do the power management and lower the power consumption, and consequently the heat and fan noise produced by your GPU