SGLang + DFlash on DGX Spark (Qwen3-Coder-Next NVFP4) — 150 tok/s

gist-gptoss-results.md

gpt-oss-120b + DFlash Results (2026-04-15)

Also tested openai/gpt-oss-120b (116.8B total, 5.1B active, MXFP4) with DFlash on the same Spark.

Setup

• Model: openai/gpt-oss-120b (~80 GB MXFP4)
• Drafter: z-lab/gpt-oss-120b-DFlash (block_size=10)
• Attention: triton (required — gpt-oss doesn't support flashinfer)
• Memory: 75 GiB (no room issues)
• No patches needed (gpt-oss is a first-class citizen in SGLang)

Launch

docker run -d --name sglang_gptoss \
  --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --restart no \
  -v /data/huggingface:/root/.cache/huggingface \
  -p 30000:30000 \
  -e HF_HUB_DISABLE_XET=1 \
  lmsysorg/sglang:nightly-dev-cu13-20260415-2c9e76d3 \
  python3 -m sglang.launch_server \
    --model-path openai/gpt-oss-120b \
    --served-model-name gpt-oss-120b-DFlash \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/gpt-oss-120b-DFlash \
    --attention-backend triton \
    --mem-fraction-static 0.55 \
    --max-running-requests 4 \
    --disable-cuda-graph \
    --trust-remote-code \
    --host 0.0.0.0 --port 30000

Benchmark (3 runs each)

Test	Tokens	tok/s	vs baseline (~38)
BST class	504	50	+31%
B-tree	1755	51	+34%
Redis store	1024	46	+21%
Math	256	46	+21%
JSON parser	1970	45	+18%
HTTP server	1255	44	+16%
Explanation	512	33	-13%
Average		44.8	+18%

DFlash adds +18-34% for code generation. Natural language reasoning is slower (lower acceptance rate). Memory: 75 GiB settled — plenty of headroom on 128 GB Spark.

gist-README.md

SGLang + DFlash on DGX Spark (Qwen3-Coder-Next NVFP4)

Running Qwen3-Coder-Next-NVFP4-GB10 with DFlash speculative decoding on SGLang. Tested on a Lenovo ThinkStation PGX (NVIDIA GB10 Grace Blackwell, 128 GB unified memory).

What you get

Test	SGLang DFlash	vLLM DFlash+Marlin	Delta
Short code (307 tok)	150 tok/s	108 tok/s	+38%
Medium code (917 tok)	81 tok/s	84 tok/s	-4%
Long code (1434 tok)	99 tok/s	—	—

Short-to-medium code generation is significantly faster. Long sequences are comparable. Tool calling works (--tool-call-parser qwen3_coder).

Why SGLang instead of vLLM

DFlash was merged into SGLang on April 7 2026 (PR #22077). SGLang uses FlashInfer which has native tree attention mask support — this makes the DFlash verification step faster than vLLM's FlashAttention backend, especially for short sequences.

Requirements

• DGX Spark / ThinkStation PGX (GB10, aarch64)
• Docker with NVIDIA runtime
• Model: saricles/Qwen3-Coder-Next-NVFP4-GB10 (~43 GB)
• Drafter: z-lab/Qwen3-Coder-Next-DFlash (~900 MB)
• Image: lmsysorg/sglang:nightly-dev-cu13-20260415-2c9e76d3 (36 GB, arm64)

Setup

1. Pull the image

docker pull lmsysorg/sglang:nightly-dev-cu13-20260415-2c9e76d3

Any SGLang nightly after 2026-04-07 with cu13 tag should work. The NVIDIA container (nvcr.io/nvidia/sglang:26.02-py3) ships 0.5.8 which does not have DFlash.

2. Download the models

huggingface-cli download saricles/Qwen3-Coder-Next-NVFP4-GB10
huggingface-cli download z-lab/Qwen3-Coder-Next-DFlash

3. Extract and patch two files

Two bugs in the SGLang nightly prevent Qwen3-Coder-Next NVFP4 from loading:

Bug 1: qwen3_next.py — GDN layers crash with compressed-tensors (NVFP4) quantization. MergedColumnParallelLinear doesn't have a .weight attribute during init.

Bug 2: expert_location.py — EPLB module can't resolve Qwen3NextForCausalLM and crashes.

Extract originals, apply patches, mount them into the container:

mkdir -p ~/sglang-patches

# Extract originals from the image
docker create --name sglang_tmp lmsysorg/sglang:nightly-dev-cu13-20260415-2c9e76d3 true
docker cp sglang_tmp:/sgl-workspace/sglang/python/sglang/srt/models/qwen3_next.py ~/sglang-patches/qwen3_next.py
docker cp sglang_tmp:/sgl-workspace/sglang/python/sglang/srt/eplb/expert_location.py ~/sglang-patches/expert_location.py
docker rm sglang_tmp

Apply patch 1 — qwen3_next.py (around line 136):

-        # Override weight_loader for packed checkpoint format.
-        # Must capture original_loader BEFORE overwriting.
-        self._override_weight_loader(
-            self.in_proj_qkvz, self._make_packed_weight_loader(self.in_proj_qkvz)
-        )
-        self._override_weight_loader(
-            self.in_proj_ba, self._make_packed_weight_loader(self.in_proj_ba)
-        )
+        # Override weight_loader for packed checkpoint format.
+        # Must capture original_loader BEFORE overwriting.
+        # Skip for quantized modules (e.g. compressed-tensors) where
+        # .weight is replaced by the quantization scheme.
+        if hasattr(self.in_proj_qkvz, 'weight') and hasattr(self.in_proj_qkvz.weight, 'weight_loader'):
+            self._override_weight_loader(
+                self.in_proj_qkvz, self._make_packed_weight_loader(self.in_proj_qkvz)
+            )
+        if hasattr(self.in_proj_ba, 'weight') and hasattr(self.in_proj_ba.weight, 'weight_loader'):
+            self._override_weight_loader(
+                self.in_proj_ba, self._make_packed_weight_loader(self.in_proj_ba)
+            )

Apply patch 2 — expert_location.py (in ModelConfigForExpertLocation.from_model_config):

-        model_class, _ = get_model_architecture(model_config)
-        if hasattr(model_class, "get_model_config_for_expert_location"):
+        try:
+            model_class, _ = get_model_architecture(model_config)
+        except (ValueError, KeyError):
+            return None
+        if hasattr(model_class, "get_model_config_for_expert_location"):

4. Launch

docker run -d --name sglang_production \
  --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  --restart unless-stopped \
  -v /data/huggingface:/root/.cache/huggingface \
  -v ~/sglang-patches/qwen3_next.py:/sgl-workspace/sglang/python/sglang/srt/models/qwen3_next.py \
  -v ~/sglang-patches/expert_location.py:/sgl-workspace/sglang/python/sglang/srt/eplb/expert_location.py \
  -p 8000:30000 \
  -e HF_HUB_DISABLE_XET=1 \
  -e SGLANG_ENABLE_JIT_DEEPGEMM=0 \
  -e SGLANG_ENABLE_DEEP_GEMM=0 \
  lmsysorg/sglang:nightly-dev-cu13-20260415-2c9e76d3 \
  python3 -m sglang.launch_server \
    --model-path saricles/Qwen3-Coder-Next-NVFP4-GB10 \
    --served-model-name Qwen3-Coder-Next-DFlash \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3-Coder-Next-DFlash \
    --attention-backend flashinfer \
    --mem-fraction-static 0.55 \
    --max-running-requests 4 \
    --disable-cuda-graph \
    --mamba-scheduler-strategy extra_buffer \
    --tool-call-parser qwen3_coder \
    --trust-remote-code \
    --host 0.0.0.0 --port 30000

Startup takes about 5 minutes. Watch logs with docker logs -f sglang_production.

5. Test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-Coder-Next-DFlash",
    "messages": [{"role": "user", "content": "Write a binary search tree in Python."}],
    "max_tokens": 1024,
    "temperature": 0.0
  }'

Notes

• --mamba-scheduler-strategy extra_buffer is required — Qwen3-Next has hybrid GDN (recurrent) layers that need this scheduler mode for speculative decoding
• DeepGEMM disabled — the scale format of this checkpoint doesn't match what DeepGEMM expects on Blackwell, causes accuracy degradation
• FP4 backend auto-selects flashinfer_cudnn which is fastest on SM120. cutlass is ~15% slower
• CUDA graphs: help long generations (+6%) but hurt short ones (-14%). I skip them for code generation workloads
• mem-fraction-static 0.55 is conservative. You might go to 0.60 once stable, but I wouldn't go higher — unified memory OOM crashes require a physical power cycle
• The API is OpenAI-compatible on port 8000 (mapped from internal 30000)