Custom obs-backgroundremoval build with support for NVIDIA cuda 12 on linux

README.md

The below CMakeUserPreset.json can be used to successfully build and install obs-backgroundremoval with a version of the ONNXRUNTIME that supports current NVIDIA CUDA 12.

It makes use of changes introduced in my fork with commit 2c658d8a. EDIT: The PR has now been merged into master

Due to changes upstream with ONNXRUNTIME, the plugin now depends on libcudnn for CUDA which is provided by nvidia-cudnn.

To make use of this:

1. Clone from my fork on the feature/custom-onnxruntime branch. Clone locaal-ai/obs-backgroundremoval
2. Place the below CMakeUserPresets.json file at the projects root
3. Run the following:

   rm -rf build_x86_64 release \
     && cmake --preset cuda12 \
     && cmake --build --preset cuda12 \
     && cmake --build build_x86_64 -t package \
     && sudo apt install ./release/obs-backgroundremoval-1.2.0-x86_64-linux-gnu.deb

4. Profit

CMakeUserPresets.json

{
  "version": 3,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 28,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "cuda12",
      "displayName": "Linux x86_64 w/ cuda12",
      "description": "Build for Linux x86_64 against CUDA 12",
      "inherits": ["template"],
      "binaryDir": "${sourceDir}/build_x86_64",
      "condition": {
        "type": "equals",
        "lhs": "${hostSystemName}",
        "rhs": "Linux"
      },
      "generator": "Ninja",
      "warnings": {"dev": true, "deprecated": true},
      "cacheVariables": {
        "QT_VERSION": "6",
        "CMAKE_INSTALL_PREFIX": "/usr",
        "CMAKE_BUILD_TYPE": "Release",
        "CPACK_DEBIAN_PACKAGE_DEPENDS": "obs-studio, nvidia-driver, nvidia-cuda-toolkit | cuda, nvidia-cudnn | cudnn",
        "CUSTOM_ONNXRUNTIME_VERSION": "1.21.0",
        "CUSTOM_ONNXRUNTIME_URL": "https://github.com/microsoft/onnxruntime/releases/download/v1.21.0/onnxruntime-linux-x64-gpu-1.21.0.tgz",
        "CUSTOM_ONNXRUNTIME_HASH": "SHA256=ef37a33ba75e457aebfd0d7b342ab20424aa6126bc5f565d247f1201b66996cf"
      }
    }
  ],
  "buildPresets": [
    {
      "name": "cuda12",
      "configurePreset": "cuda12",
      "displayName": "Linux x86_64 w/ cuda12",
      "description": "Linux build for x86_64 against CUDA 12",
      "configuration": "Release"
    }
  ]
}

Dangit... I can't get this to work. I'm on Fedora, but I basically followed all the directions (except not building deb.) I installed CUDA, CUDnn, cloned the obs-backgroundremoval, downloaded the CMakeUserPresets.json and built with --preset cuda12. Everything built and installed no problem, I can select GPU - CUDA, but no worky.

cmake --preset cuda12 -B build_x86_64 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DENABLE_FRONTEND_API=ON -DENABLE_QT=OFF -DUSE_SYSTEM_OPENCV=ON

cmake --build build_x86_64 --preset cuda12

@mercster If I had to guess, it may be due to your CMAKE_INSTALL_PREFIX. Here it is set to match the path of the precompiled debs. You also may need a newer ONNX runtime, which can be manually set now that the PR was merged.

My CUDA stuff is in /usr/local, the plugin is in /usr (so OBS can see it), but it all linked, and that's the only CUDA stuff I have installed, so it's not like there should be a mismatch there. Also I thought the ONNX runtime was merged quite awhile ago, and was in the main branch? I don't know where the ONNX runtime comes from otherwise...

merc@fedora:/usr/local$ ls
bin   cuda-12    etc    include  lib64    man   sdrconnect  src
cuda  cuda-12.9  games  lib      libexec  sbin  share

merc@fedora:/usr/lib64/obs-plugins/obs-backgroundremoval$ ldd libonnxruntime_providers_cuda.so
ldd: warning: you do not have execution permission for `./libonnxruntime_providers_cuda.so'
        linux-vdso.so.1 (0x00007f62a4500000)
        libcublasLt.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.12 (0x00007f625d800000)
        libcublas.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.12 (0x00007f6257000000)
        libcurand.so.10 => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so.10 (0x00007f624c800000)
        libcufft.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so.11 (0x00007f623aa00000)
        libcudart.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12 (0x00007f623a600000)
        libcudnn.so.9 => /usr/local/cuda/targets/x86_64-linux/lib/libcudnn.so.9 (0x00007f623a200000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f62a44f4000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f62a44f0000)
        libnvrtc.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so.12 (0x00007f6233800000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f62a44ec000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6233400000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f628fae9000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f628fabd000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f623320e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f62a4502000)

@mercster The CMAKE_INSTALL_PREFIX guess was just an off the cuff guess that is a common gotcha when mixing compiled plugin debs with packaged obs. Admittedly I was on the go while responding and missed that you weren't creating a deb. I'm not that familiar with the rpm side of things if you're creating one or are you just installing directly to the dirs?

Regardless, your OBS log file might be helpful in chasing down what's happening since it appears to be loading into OBS.

Also I thought the ONNX runtime was merged quite awhile ago

This CMakeUserPresets.json preset is primarily to ensure required CUDA libraries are installed and override the default ONNX with a newer version that works with CUDA 12. Back in April my PR was accepted to have the flags for custom ONNX to work, but the actual runtime versions in the repo are still the older ones that don't work with CUDA 12.

Happy to try and troubleshoot it further with you!