Qwen3.8-Flash-Next on Strix Halo (AMD Ryzen AI MAX+ 395, 128GB unified RAM)

Following this reddit post: https://www.reddit.com/r/StrixHalo/comments/1w12gs2/qwen38flashnext_125ba6b_finally_running_on_my/

Step-by-step guide to run Qwen3.8-Flash-Next (125B MoE, ~6B active) locally on a Strix Halo machine (gfx1151) via ROCm and the EngramHalo llama.cpp fork, with MTP speculative decoding and 131K context.

Verified on: Framework Desktop, Ryzen AI MAX+ 395 (Radeon 8060S / gfx1151), 128GB LPDDR5X, Ubuntu 26.04 LTS, kernel 7.0.0-30-generic, ROCm 7.13. Based on the community guide by patrickmfurbert (r/LocalLLaMA), the EngramHalo.cpp fork by Aristo94, the MTP sidecar GGUF by EasiiX, and quants by unsloth.

0. Memory: BIOS UMA vs. GTT

The original guide sets BIOS → GFX Configuration → iGPU: UMA_SPECIFIED, UMA Frame Buffer Size: 96G. This is not required on Ubuntu 26.04 / ROCm 7.13. With the default 512MB UMA, the kernel exposes ~100GB as GTT and ROCm allocates GPU memory from it transparently. Check yours:

rocm-smi --showmeminfo gtt
# GTT Total Memory (B): 107374182400   <- ~100GB: you're fine, skip the BIOS change

If GTT is small on your machine, either set the BIOS UMA buffer to 96G (as in the original guide) or raise GTT via amdgpu/ttm kernel parameters.

1. Install ROCm dev packages

The runtime alone isn't enough to build llama.cpp — you need the HIP dev headers (cmake configs) and BLAS dev libraries. On Ubuntu 26.04 the ROCm 7.13 repo ships per-architecture packages:

sudo apt-get install -y \
  amdrocm-core-dev7.13-gfx1151 \
  amdrocm-blas-dev7.13-gfx1151 \
  amdrocm-hipblas-common-dev7.13-gfx1151

Verify: ls /opt/rocm/lib/cmake/hip should exist, and rocminfo | grep gfx should show gfx1151.

2. Add a 64GB swapfile (mandatory for 131K context)

At full context the model + KV cache can exceed physical RAM. Without swap the OOM killer will take out random system services mid-generation (the original poster lost tailscaled and their SSH session this way).

sudo fallocate -l 64G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

3. Build the EngramHalo llama.cpp fork

Mainline llama.cpp has qwen4exp arch support (PR #27742) but lacks the ROCm-specific kernels that fix long-context decode collapse (~6 tok/s at 100K+ context instead of 20+). Use the fork:

git clone --depth 1 https://github.com/Aristo94/EngramHalo.cpp ~/llama-engramhalo
cd ~/llama-engramhalo
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DGGML_HIP_NO_VMM=ON \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DCMAKE_C_COMPILER=/opt/rocm/bin/hipcc \
  -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc
cmake --build build --target llama-server llama-bench -j$(nproc)

Sanity check — should list the GPU with ~100GB:

~/llama-engramhalo/build/bin/llama-server --list-devices
# ROCm0: Radeon 8060S Graphics (102400 MiB, ...)

4. Download the models (~92GB total)

Requires the hf CLI (pip install -U huggingface_hub). Both repos are public.

Main model — unsloth IQ4_XS quant (~88GB, 3-part split; part 1 is a small metadata shard, that's normal):

hf download unsloth/Qwen3.8-Flash-Next-GGUF \
  "UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf" \
  "UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00002-of-00003.gguf" \
  "UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00003-of-00003.gguf" \
  --local-dir ~/models/qwen38-flash-next/

MTP sidecar for speculative decoding (~4.1GB):

hf download EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF \
  mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --local-dir ~/models/qwen38-flash-next/

5. Install the patched chat template

The official template rejects mid-conversation system messages, which agentic tools (Claude Code, Qwen Code CLI, ...) send constantly — you get a wall of 500 errors without the fix. The community-patched qwen3.8-froggeric-v22.4 template lives in patrickmfurbert's ai-tools repo:

git clone --depth 1 https://github.com/patrickmfurbert/ai-tools /tmp/ai-tools
mkdir -p ~/scripts/templates/qwen-fixed/3_8-27B
cp /tmp/ai-tools/docs/templates/qwen-fixed/3_8-27B/chat_template.jinja \
   ~/scripts/templates/qwen-fixed/3_8-27B/

6. Launch

Save as ~/scripts/start-qwen38-flash-next.sh and chmod +x it:

#!/usr/bin/env bash
HSA_ENABLE_SDMA=0 HSA_XNACK=1 ROCBLAS_USE_HIPBLASLT=1 \
"$HOME/llama-engramhalo/build/bin/llama-server" \
  -m "$HOME/models/qwen38-flash-next/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf" \
  -md "$HOME/models/qwen38-flash-next/mtp-Qwen3.8-Flash-Next-Q8_0.gguf" \
  -ngl 999 -fa on -ctk q8_0 -ctv q8_0 \
  -c 262144 -ub 2048 -t 4 --parallel 1 \
  --jinja \
  --chat-template-file "$HOME/scripts/templates/qwen-fixed/3_8-27B/chat_template.jinja" \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
  --host 0.0.0.0 --port 8082 "$@"

Only the first split file is passed with -m; llama.cpp finds the other parts automatically. Load time is under a minute.

Context size: 128K vs 256K

The GGUF metadata shows the model was trained at 262,144 tokens (256K) natively (qwen4exp.context_length = 262144), so you can run -c 262144 with no RoPE-scaling tricks. The KV cache is small for a model this size — grouped-query attention with only 2 KV heads across 48 layers, quantized to q8_0 — so going from 128K to 256K only adds ~5GB of GPU memory (measured: ~69.5GB → ~74.3GB GTT used at startup).

Caveats at 256K:

Community benchmarks (including the decode-collapse fix) were measured at ~100K+; nobody has published Strix Halo numbers at 250K depth. Expect slower decode as context actually fills, and real memory pressure only shows up deep into a session — if the OOM killer stirs, drop back to -c 131072.
Web UI: http://localhost:8082/ (add --no-ui to disable it for API-only/headless use)
OpenAI-compatible API: http://localhost:8082/v1
--host 0.0.0.0 exposes the server to your LAN; use --host 127.0.0.1 for local-only.

Quick test:

curl -s http://127.0.0.1:8082/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is 17*23?"}],"max_tokens":64}'

7. Using it for coding (agentic harnesses)

The launch script passes -a qwen3.8-flash-next, so the model is addressable by that short alias instead of its full GGUF path (/v1/models returns the alias).

This build exposes three API surfaces, all verified working:

OpenAI chat completions at /v1/chat/completions, with tool calling — a request carrying a tools array returns proper tool_calls and finish_reason: "tool_calls". This is what most harnesses use.
Anthropic messages at /v1/messages — returns real Anthropic-format responses (thinking + text blocks, usage). Claude Code can therefore point straight at this server via ANTHROPIC_BASE_URL=http://localhost:8082 with a dummy ANTHROPIC_AUTH_TOKEN; no translation proxy or router needed.
OpenAI responses at /v1/responses.

opencode

Add a provider to ~/.config/opencode/opencode.json (this sits alongside any existing providers):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llamacpp": {
      "name": "llama.cpp (Strix Halo)",
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://127.0.0.1:8082/v1" },
      "models": {
        "qwen3.8-flash-next": {
          "name": "Qwen3.8-Flash-Next (local)",
          "limit": { "context": 262144, "output": 32768 },
          "tool_call": true
        }
      }
    }
  }
}

Then select the model as llamacpp/qwen3.8-flash-next, e.g.:

opencode run --model llamacpp/qwen3.8-flash-next "fix the bug in calc.py"

Verified: a full agentic loop (glob → read → edit → report) completed in ~63s.

What to expect in real use

One request at a time. With --parallel 1, a harness's background requests (titles, summarization, subagents) queue behind the main one instead of running alongside it. Occasional stalls are queuing, not slowness.
Prefill dominates the first turn. At 327–720 tok/s, a harness's system prompt plus tool definitions (~10–15K tokens) costs roughly 20–45s before the first token on a cold cache. Later turns in the same conversation reuse the KV cache and are much faster, so harnesses with a stable prompt prefix feel far better than ones that reshuffle context every turn.
Give it a large max_tokens — see the thinking-model note below.

Troubleshooting: opencode fails with `no such column: name`

An opencode database created by a different version than the installed binary makes every command fail, including opencode models. This is an opencode issue, unrelated to the model server. opencode upgrade migrates the schema and keeps session history (fixed it here, 1.18.11 → 1.18.25). As a last resort, move ~/.local/share/opencode/opencode.db aside so it is recreated — that loses stored sessions. To confirm the server is not at fault, run opencode with a throwaway data dir: XDG_DATA_HOME=/tmp/ocdata opencode models.

8. Things that will bite you

Never use -ctk bf16 — crashes on gfx1151. Use q8_0.
--parallel 1 only — multi-slot is not validated on HIP with the QSA gather path.
Skip --tensor-read-lazy on — hangs indefinitely on ROCm.
This is a thinking model — reasoning tokens count against max_tokens. A request with a small budget (e.g. 32) returns an empty content because the whole budget goes to the reasoning phase. Give completions a generous max_tokens, especially from agentic clients.
Watch memory while it runs: watch -n 2 'free -h && rocm-smi --showmeminfo gtt' (with the GTT setup, GPU memory shows under GTT, not VRAM).

Expected performance (matches our verification)

Metric	Value
Decode (sustained)	17–28 tok/s
Decode peak (MTP on predictable output)	120+ tok/s
MTP acceptance rate	74–91% (we measured 79%)
Prefill	327–720 tok/s
GPU (GTT) memory used	~70GB at 128K ctx, ~74GB at 256K ctx
Load time	~45 seconds

Credits

patrickmfurbert — original r/LocalLLaMA writeup and the ai-tools repo (templates, scripts, router)
Aristo94 — EngramHalo.cpp fork with the ROCm HIP patches
EasiiX — prebuilt MTP sidecar GGUF
unsloth — quantized GGUFs
The Qwen team and the llama.cpp maintainers (qwen4exp support, PR #27742)

chm123/qwen3.8-flash-next-setup.md

Select an option

No results found

Select an option

No results found

Qwen3.8-Flash-Next on Strix Halo (AMD Ryzen AI MAX+ 395, 128GB unified RAM)

0. Memory: BIOS UMA vs. GTT

1. Install ROCm dev packages

2. Add a 64GB swapfile (mandatory for 131K context)

3. Build the EngramHalo llama.cpp fork

4. Download the models (~92GB total)

5. Install the patched chat template

6. Launch

Context size: 128K vs 256K

7. Using it for coding (agentic harnesses)

opencode

What to expect in real use

Troubleshooting: opencode fails with `no such column: name`

8. Things that will bite you

Expected performance (matches our verification)

Credits

chm123/qwen3.8-flash-next-setup.md

Qwen3.8-Flash-Next on Strix Halo (AMD Ryzen AI MAX+ 395, 128GB unified RAM)

0. Memory: BIOS UMA vs. GTT

1. Install ROCm dev packages

2. Add a 64GB swapfile (mandatory for 131K context)

3. Build the EngramHalo llama.cpp fork

4. Download the models (~92GB total)

5. Install the patched chat template

6. Launch

Context size: 128K vs 256K

7. Using it for coding (agentic harnesses)

opencode

What to expect in real use

Troubleshooting: opencode fails with no such column: name

8. Things that will bite you

Expected performance (matches our verification)

Credits

Troubleshooting: opencode fails with `no such column: name`