ROCm build notes - gfx1151 (Strix Halo) in Proxmox LXC

ROCm build notes.md

ROCm build notes - gfx1151 (Strix Halo) in Proxmox LXC

What was installed

• cmake, build-essential, libcurl4-openssl-dev, gdb, strace via apt.
• ROCm 7.9 (therock-dist-linux-gfx1151-7.9.0rc20251008.tar.gz) unpacked to /opt/rocm79. Debian 13's own ROCm is 5.5-5.7, which predates gfx1151 and cannot work.

Configure line

cmake -S . -B build-rocm -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_RPC=ON \
  -DAMDGPU_TARGETS=gfx1151 -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
  -DHIP_PLATFORM=amd \
  -DCMAKE_C_COMPILER=/opt/rocm79/lib/llvm/bin/amdclang \
  -DCMAKE_CXX_COMPILER=/opt/rocm79/lib/llvm/bin/amdclang++ \
  -DCMAKE_PREFIX_PATH=/opt/rocm79 \
  -DCMAKE_HIP_FLAGS="--rocm-path=/opt/rocm79 --rocm-device-lib-path=/opt/rocm79/lib/llvm/amdgcn/bitcode"

-DHIP_PLATFORM=amd and the explicit --rocm-device-lib-path are needed because TheRock's layout is not what hip-config.cmake expects.

Patch applied to libhsa-runtime64.so.1

Without this, every GPU queue creation fails and any HIP program segfaults.

AMDKFD_IOC_CREATE_QUEUE returned EINVAL, then ROCr dereferenced a half-built ScratchInfo in ReleaseQueueMainScratch() and crashed.

Cause: the kernel and ROCm disagree on the VGPR file size of gfx1151.

• kernel kfd_get_vgpr_size_per_cu() lists gfx1151 (110501) -> 0x60000
• ROCm hsakmt_get_vgpr_size_per_cu() does not list it -> falls back to 0x40000

So ROCm asked for a context-save area of 13,942,784 bytes while kfd_queue_acquire_buffers() required at least 19,185,664. The gap is exactly 40 CUs x 0x20000. gfx1151 is absent from the ROCm side even on current amd-staging, so no released ROCm version has this fixed.

debian@nau-lemonade:~$ cat /home/debian/llama.cpp/build-rocm/ROCM-SETUP-NOTES.md

ROCm build notes - gfx1151 (Strix Halo) in Proxmox LXC

What was installed

• cmake, build-essential, libcurl4-openssl-dev, gdb, strace via apt.
• ROCm 7.9 (therock-dist-linux-gfx1151-7.9.0rc20251008.tar.gz) unpacked to /opt/rocm79. Debian 13's own ROCm is 5.5-5.7, which predates gfx1151 and cannot work.

Configure line

cmake -S . -B build-rocm -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_RPC=ON \
  -DAMDGPU_TARGETS=gfx1151 -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
  -DHIP_PLATFORM=amd \
  -DCMAKE_C_COMPILER=/opt/rocm79/lib/llvm/bin/amdclang \
  -DCMAKE_CXX_COMPILER=/opt/rocm79/lib/llvm/bin/amdclang++ \
  -DCMAKE_PREFIX_PATH=/opt/rocm79 \
  -DCMAKE_HIP_FLAGS="--rocm-path=/opt/rocm79 --rocm-device-lib-path=/opt/rocm79/lib/llvm/amdgcn/bitcode"

-DHIP_PLATFORM=amd and the explicit --rocm-device-lib-path are needed because TheRock's layout is not what hip-config.cmake expects.

Patch applied to libhsa-runtime64.so.1

Without this, every GPU queue creation fails and any HIP program segfaults.

AMDKFD_IOC_CREATE_QUEUE returned EINVAL, then ROCr dereferenced a half-built ScratchInfo in ReleaseQueueMainScratch() and crashed.

Cause: the kernel and ROCm disagree on the VGPR file size of gfx1151.

• kernel kfd_get_vgpr_size_per_cu() lists gfx1151 (110501) -> 0x60000
• ROCm hsakmt_get_vgpr_size_per_cu() does not list it -> falls back to 0x40000

So ROCm asked for a context-save area of 13,942,784 bytes while kfd_queue_acquire_buffers() required at least 19,185,664. The gap is exactly 40 CUs x 0x20000. gfx1151 is absent from the ROCm side even on current amd-staging, so no released ROCm version has this fixed.

Two bytes were changed in /opt/rocm79/lib/libhsa-runtime64.so.1.18.0 (which libhsa-runtime64.so.1 symlinks to). The pristine original is restored at /opt/rocm79/lib/libhsa-runtime64.so.1.18.0.orig.

file offset 0x192ae3:  0x02 -> 0x03
file offset 0x193402:  0x04 -> 0x06
sha256 (first 32) original cd27dd36c95f837647490098a6ca3258
sha256 (first 32) patched  608be60ad392bd453e8014e55bdc8e0f

To revert: sudo cp /opt/rocm79/lib/libhsa-runtime64.so.1.18.0{.orig,}

vaddr	before	after	where
0x193ae1	or $0x2,%eax	or $0x3,%eax	hsakmt_get_vgpr_size_per_cu
0x1943fd	or $0x45000,%r8d	or $0x65000,%r8d	inlined copy in hsaKmtCreateQueueExt

Both compile to vgpr_size = 0x60000. The second one is the one that matters - it feeds the size sent to the kernel. This over-allocates by 128 KiB per CU on GPUs that only need 0x40000, which is harmless (the kernel checks >=) and moot here since gfx1151 is the only device.

Re-applying after a ROCm upgrade: find hsakmt_get_vgpr_size_per_cu, then the or $0x45000 inside hsaKmtCreateQueueExt, and bump both to the 0x60000 form.

Runtime requirement

GGML_CUDA_DISABLE_GRAPHS=1 is required. With HIP graphs on, prompt processing works but token generation aborts:

HSA_STATUS_ERROR_INVALID_PACKET_FORMAT: The AQL packet is malformed. code: 0x1009

Use . build-rocm/rocm-env.sh to set this and the library paths.

Measured

Qwen3.6-35B-A3B UD-Q4_K_XL, 21.27 GiB, -ngl 99:

test	t/s
pp512	1009.04 +/- 14.25
tg128	51.11 +/- 0.13

Note on the tree

This build includes the still-open PR 26610 (RPC: add -sm tensor) applied as uncommitted working-tree changes, plus its two stacked parent commits.