With its latest reales qemu added the Venus patches so that virtio-gpu now support venus encapsulation for vulkan. This is one more piece to the puzzle towards full Vulkan support.
An outdated blog post on clollabora described in 2021 how to enable 3D acceleration of Vulkan applications in QEMU through the Venus experimental Vulkan driver for VirtIO-GPU with a local development environment. Following up on the outdated write up, this is how its done today.
Let's start with the brief description of the projects mentioned in the post & extend them:
- QEMU is a machine emulator
- VirGL is an OpenGL driver for VirtIO-GPU, available in Mesa.
- Venus is an experimental Vulkan driver for VirtIO-GPU, also available in Mesa.
- Virglrenderer is a library that enables hardware acceleration to VM guests, effectively translating commands from the two drivers just mentioned to either OpenGL or Vulkan.
- libvirt is an API for managing platform virtualization
- virt-manager is a desktop user interface for managing virtual machines through libvirt
Merged Patches:
- 2024-08-14 venus: make cross-device optional merged in mesa 24.2
- 2024-11-25 KVM: Stop grabbing references to PFNMAP'd pages merged in linux 6.13
- 2024-11-12 Support blob memory and venus on qemu merged in qemu 9.2.0
Work in progress:
- libvirt Add support for more virtio-vga-gl arguments #638
- virt-manager Add support for Venus / Vulkan VirtIO-GPU driver #362
Make sure you have the proper version installed on the host:
- linux kernel >= 6.13 built with CONFIG_UDMABUF
- working Vulkan and kvm setup
- qemu >= 9.2.0
- virglrenderer with enabled venus support
- mesa >= 24.2.0
You can verify this like so:
$ uname -r
6.13.0
$ ls /dev/udmabuf
/dev/udmabuf
$ ls /dev/kvm
/dev/kvm
$ qemu-system-x86_64 --version
QEMU emulator version 9.2.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
Check your distros package sources how they build virglrenderer.
For Vulkan to work you need the proper drivers to be installed for your graphics card.
To verfiy your setup, install vulkan-tools. Make sure mesa >= 24.2.0 and test vkcube:
$ vulkaninfo --summary | grep driverInfo
driverInfo = Mesa 24.2.3-1ubuntu1
driverInfo = Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0)
...
$ vkcube
Selected GPU x: ..., type: ...
If your distro doesn't (yet) ship and updated version of qemu, you can build it yourself from source:
wget https://download.qemu.org/qemu-9.2.0.tar.xz
tar xvJf qemu-9.2.0.tar.xz
cd qemu-9.2.0
mkdir build && cd build
../configure --target-list=x86_64-softmmu \
--enable-kvm \
--enable-opengl \
--enable-virglrenderer \
--enable-gtk \
--enable-sdl
make -j4
The configuration step will throgh errors if packages are missing. Check the qemu wiki for further info what to install: https://wiki.qemu.org/Hosts/Linux
Create an image & fetch the distro of your choice:
ISO=ubuntu-24.10-desktop-amd64.iso
wget https://releases.ubuntu.com/oracular/ubuntu-24.10-desktop-amd64.iso
IMG=ubuntu-24-10.qcow2
qemu-img create -f qcow2 $IMG 16GRun a live version or install the distro
qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-smp 4 \
-m 4G \
-cpu host \
-net nic,model=virtio \
-net user,hostfwd=tcp::2222-:22 \
-device virtio-vga-gl,hostmem=4G,blob=true,venus=true \
-vga none \
-display gtk,gl=on,show-cursor=on \
-usb -device usb-tablet \
-object memory-backend-memfd,id=mem1,size=4G \
-machine memory-backend=mem1 \
-hda $IMG \
-cdrom $ISO
Adjust the parameters accordingly:
- smp: number of cpu cores
- m: RAM
- hostmem,size: VRAM
Install mesa-utilites and vulkan-tools to test the setup:
$ glxinfo -B
$ vkcube
Selected GPU x: ..., type: ...
If the deive is llvmpipe somehting is wrong. The device should be virgl (...).
- (host) add
-d guest_errorsto show error messages from the guest - (guest) try installing vulkan virtio drivers and mesa
- check the original blog post
This is how you do it on Ubuntu
Install mainline: https://github.com/bkw777/mainline
sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline
find the latest kernel (>= 6.13), at the time of writing 6.13 is a release candidate, so include those:
$ mainline check --include-rc
Install kernel:
$ sudo mainline install 6.13-rc1
Verfify installed kernels:
$ mainline list-installed
mainline 1.4.10
Installed Kernels:
linux-image-6.11.0-13-generic
linux-image-generic-hwe-24.04
linux-image-unsigned-6.13.0-061300rc1-generic
mainline: done
reboot into new kernel
verify running kernel
$ uname -r
6.13.0-061300rc1-generic
the ubuntu package is not compiled with the proper flags.
If installed remove it: $ sudo apt-get remove libvirglrenderer-dev
download, build & install from source with venus enabled
wget https://gitlab.freedesktop.org/virgl/virglrenderer/-/archive/1.1.0/virglrenderer-1.1.0.tar.gz
sudo apt-get install python3-full ninja-build libvulkan-dev libva-dev
python3 -m venv venv
venv/bin/pip install meson
venv/bin/meson build -Dvideo=true -Dvenus=true
ninja -C build
ninja install
install qemu >= 9.2.0, at the time of writing ubuntu has not yet packaged it
Install build depdencies: https://wiki.qemu.org/Hosts/Linux
sudo apt-get install build-essential pip libslirp-dev slirp
sudo apt-get install git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev ninja-build
sudo apt-get install git-email
sudo apt-get install libaio-dev libbluetooth-dev libcapstone-dev libbrlapi-dev libbz2-dev
sudo apt-get install libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev
sudo apt-get install libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev
sudo apt-get install librbd-dev librdmacm-dev
sudo apt-get install libsasl2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh-dev
sudo apt-get install libvde-dev libvdeplug-dev libvte-2.91-dev libxen-dev liblzo2-dev
sudo apt-get install valgrind xfslibs-dev
sudo apt-get install libnfs-dev libiscsi-dev
build and run as described
-- work in progress --
Currently this is work in progress, so there is no option to add vulkan support in virt-manager. There are no fields to configure this. Also xml doesnt work, because libvirt doesn't know about these options either, so xml validation fails. There is however an option for QEMU command-line passthrough which bypasses the validation.
If you setup a default machine with 4G of memory, you can do this:
<qemu:commandline>
<qemu:arg value="-device"/>
<qemu:arg value="virtio-vga-gl,hostmem=4G,blob=true,venus=true"/>
<qemu:arg value="-object"/>
<qemu:arg value="memory-backend-memfd,id=mem1,size=4G"/>
<qemu:arg value="-machine"/>
<qemu:arg value="memory-backend=mem1"/>
<qemu:arg value="-vga"/>
<qemu:arg value="none"/>
</qemu:commandline>Which gives this error:
qemu-system-x86_64: virgl could not be initialized: -1
Changing the number from 4G to 4194304k (same as memory) leds to this error:
qemu-system-x86_64: Spice: ../spice-0.15.2/server/red-qxl.cpp:435:spice_qxl_gl_scanout: condition `qxl_state->gl_draw_cookie == GL_DRAW_COOKIE_INVALID' failed
to be further investigated.
Try setting
venus_implicit_fencing=truefor a sizeable performance boost.I'm getting close to native for some games with that env (and after a bunch of mesa/virglrenderer/qemu updates). My bad, something on the host must have been playing up as the real native performance is still about twice as fast as the best case. Implicit fencing does help, but it doesn't close the gap on its own.Of course, there is some VM overhead too so that could impact things in CPU limited scenarios. It might not work on all systems from memory, either. And some games like Nier: Replicant still run horribly (single digit fps during loading screens, and it's not due to shader compilation AFAICT).
Also, I hope to see Vulkan Video support so I can one day set this up as a Sunshine server without it taking over my host. I currently do it with GPU passthrough but would like to retire that one day. I understand virglrenderer supports VAAPI at least, but QEMU doesn't seem to use it.
EDIT: I suspect VRAM issues with some of the games that perform poorly. Presumably since it has to share with the host, games that were borderline in terms of VRAM usage become unplayable. Which makes sense, since I'm testing with a GPU that only has 8GB of VRAM.
EDIT 2: Okay I'm not entirely sure how to interpret VRAM usage in this context, but it seems like the VM is consuming more and more VRAM after launching a bunch of games, and not releasing it. Since it's sharing with the host, it's causing performance issues with both the host and the guest, and giving unpredictable performance results.
If it's by design, it might be better to allocate some amount less than the physical VRAM so the host and guest don't interfere with each other as much. However, qemu/venus seems to only support power of 2 sizes which limits the options. Interestingly, GPU memory usage of QEMU keeps climbing even when I set
hostmem=4G, so assuming I can trustnvtopthe card will still use more GPU ram even with a value lower than the physical VRAM.tl;dr implicit fencing helps performance and there also seems to be a presumably unrelated VRAM leak with QEMU which eventually tanks performance on both host and guest
LAST EDIT: And after testing with a faster GPU, it's clear that Venus is heavily CPU limited and that the performance differential is much greater than I thought since the faster card is less likely to be GPU limited. The VRAM leak still occurs with 16GB of VRAM as well, but takes longer to fill up as you'd expect (or rather, more games need to be loaded as re-running the same game doesn't use additional VRAM).