Getting ollama to work in nixos

nixos-ollama.md

• NOTE:
  • baseline refers to the following: uses acceleration = "rocm", iGPU was disabled in bios, amd 7800xt(gfx1101- this will vary based on your gpu and rocminfo output nix-shell -p "rocmPackages.rocminfo" --run "rocminfo" | grep "gfx" and thus the versions used below("11.0.1"))
  • an overlay is used to patch the version and set in service.ollama.package
    • unstable refers to this commit 5df43628fdf08d642be8ba5b3625a6c70731c19c
    • stable refers to 24.11
  • (debugging) you can see a feed of ollama service log with journalctl -u ollama -ef
• Tests
  • rocmOverrideGfx = "11.0.1"; only
    • unstable
      • msg="Dynamic LLM libraries" runners="[cpu]"
      • used cpu
    • stable
      • msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm]"
      • used gpu
  • environmentVariables.HSA_OVERRIDE_GFX_VERSION = "11.0.1"; only
    • unstable
      • msg="Dynamic LLM libraries" runners="[cpu]"
      • used cpu
    • stable
      • msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm]"
      • used gpu
  • baseline
    • unstable
      • msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm]"
      • used cpu
    • stable
      • msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm]"
      • used cpu
  • environmentVariables.OLLAMA_LLM_LIBRARY = "rocm" only
    • unstable
      • msg="Dynamic LLM libraries" runners="[cpu]"
      • used cpu
    • stable
      • msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 rocm]"
      • used cpu
  • NOTE:
    • Using stable by itself didn't work. When used with rocmOverrideGfx or environmentVariables.HSA_OVERRIDE_GFX_VERSION and acceleration = "rocm";(baseline) it detected and used the gpu.
    • Using (HCC/HIP/ROCR)_AMDGPU_TARGET didn't solve the issue, but got detection to be logged(detection by the script and actual gpu usage are separate things. the log can say gpu is detected and ollama ps can also say "100% GPU", but not actually use the gpu)

• Nix Config(where pkgs.ollama is overlayed to use the version from nixpkgs 24.11)

  services.ollama = {
      enable = true;
      package = pkgs.ollama;
      acceleration = "rocm";
      loadModels = ["deepseek-r1:14b"];
      rocmOverrideGfx = "11.0.1";
    };