llama-bench of Kolibri-1 78B model on workstation with 5 AMD Instinct MI50 GPUs with 32+4*16=96GB VRAM

bench5.md

≥239 tokens/s prompt processing
57.76 tokens/s token generation

hermann@7600x:~/llama.cpp$ ./bench5
ggml_cuda_init: found 5 ROCm devices (Total VRAM: 98224 MiB):
  Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
  Device 1: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
  Device 2: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
  Device 3: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
  Device 4: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB

model	size	params	backend	ngl	threads	fa	ts	lm	test	t/s
kolibri1 78B.A3.5B Q8_0	77.42 GiB	78.10 B	ROCm	999	16	1	2.00/1.00/1.00/1.00/1.00	none	pp512	239.01 ± 8.32
kolibri1 78B.A3.5B Q8_0	77.42 GiB	78.10 B	ROCm	999	16	1	2.00/1.00/1.00/1.00/1.00	none	pp2048	260.64 ± 4.01
kolibri1 78B.A3.5B Q8_0	77.42 GiB	78.10 B	ROCm	999	16	1	2.00/1.00/1.00/1.00/1.00	none	tg128	57.76 ± 0.10

build: 2280f0fd1 (11886)
hermann@7600x:~/llama.cpp$ 

bench5

#!/bin/bash
export HSA_OVERRIDE_GFX_VERSION=9.0.6
export HSA_ENABLE_SDMA=1
export HSA_ENABLE_PEER_SDMA=1
export GGML_CUDA_DISABLE_PEER_ACCESS=0
export HIP_VISIBLE_DEVICES=0,1,2,3,4

./build/bin/llama-bench \
  -m /mnt/hf/Kolibri-1-GGUF/Q8_0/Kolibri-1-Q8_0-00001-of-00002.gguf \
  -ngl 999 \
  -mg 0 \
  -sm layer \
  -ts 2/1/1/1/1 \
  -fa on \
  -lm none \
  -p 512,2048 \
  -n 128 \
  -t 16

Kolibri-1 model judging over its own benchmark results ;-)

my prompt

I did a llama-bench of Kolibri-1 (you) on my 5x Instinct MI50 GPU workstation, and I am quite happy with prompt processing and token generation. Open WebUI shows 47 tokens/s, llama-bench 57 for token generation:
  
hermann@7600x:~/llama.cpp$ ./bench5
ggml_cuda_init: found 5 ROCm devices (Total VRAM: 98224 MiB):
  Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB
  Device 1: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
  Device 2: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
  Device 3: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
  Device 4: AMD Instinct MI50/MI60, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 16368 MiB
| model                          |       size |     params | backend    | ngl | threads |  fa | ts           |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | --: | ------------ | ---------: | --------------: | -------------------: |
| kolibri1 78B.A3.5B Q8_0        |  77.42 GiB |    78.10 B | ROCm       | 999 |      16 |   1 | 2.00/1.00/1.00/1.00/1.00 |       none |           pp512 |        239.01 ± 8.32 |
| kolibri1 78B.A3.5B Q8_0        |  77.42 GiB |    78.10 B | ROCm       | 999 |      16 |   1 | 2.00/1.00/1.00/1.00/1.00 |       none |          pp2048 |        260.64 ± 4.01 |
| kolibri1 78B.A3.5B Q8_0        |  77.42 GiB |    78.10 B | ROCm       | 999 |      16 |   1 | 2.00/1.00/1.00/1.00/1.00 |       none |           tg128 |         57.76 ± 0.10 |
build: 2280f0fd1 (11886)
hermann@7600x:~/llama.cpp$

response

That's an impressive workstation setup! 5x Instinct MI50 GPUs with 98GB total VRAM is serious hardware, and running a 78B parameter model in Q8_0 (~77GB) with near-full GPU offloading is quite an achievement. Your benchmark numbers are excellent:

• 57.76 tokens/s for token generation (llama-bench, pp2048 context)
• ~260 tokens/s for prompt processing (pp2048 context)
• 47 tokens/s in Open WebUI (likely accounting for some overhead)

These numbers suggest Kolibri-1 is running very efficiently on your ROCm setup.

Is there anything specific you'd like to do with this information? For example:

1. Would you like me to save your hardware configuration and performance expectations as a memory so I can tailor future responses?
2. Are you looking to benchmark something specific or test particular capabilities?
3. Do you have any questions about the results or how to interpret them?

Just let me know how I can help!

Screenshot of mi50_rt_graph.py tool (created with gemini.google.com) for real time monitoring of the 5 Instinct MI50 GPUs during above response processing: