Running Qwen3.8-27B on a single NVIDIA DGX Spark (GB10) for agentic coding

Objective: get the most useful agentic-coding agent out of one DGX Spark — long tool-using runs, big contexts, hours at a time — and write down what actually moved the needle.

Comments very welcome. Much of this was found the hard way, and some of it is probably still wrong or leaving performance on the table. If you run a Spark (or any GB10 / unified-memory box) and know better, please say so — that is why this is public. Open questions are listed at the end.

Hardware: NVIDIA DGX Spark, GB10, arm64, compute capability sm_121, 128 GB unified memory (GPU allocations are system RAM), CUDA 13.0, driver 580.x. Single box, no cluster.

All numbers are single-request (concurrency 1) — the agentic case, one agent working. Multi-user serving has different trade-offs and SGLang's batching story gets much stronger than it looks here.

TL;DR — the five things that mattered most

Check what reasoning effort your model is actually running at. Qwen's chat template defaults to xhigh. This was worth 17x on one tier and is by far the biggest single win in this document.
Setting reasoning_effort in your agent framework probably does nothing. llama-server has no native support for it (llama.cpp#20408). It must be set server-side.
Decode speed is ~1/bytes-per-token. Unified memory is bandwidth-bound; the quantisation you pick is the single biggest lever on tok/s.
Benchmark at the context depth you actually work at. The config that scores 44 tok/s at 1k does 22 tok/s at 96k — and at depth, prefill dominates wall clock, not decode.
Don't quantise the KV cache on this model. f16 KV is 20% faster at 96k than q8_0. Counterintuitive; the architecture explains it.

1. Reasoning effort: the biggest and most embarrassing win

Qwen3.x chat templates contain roughly reasoning_effort | default('xhigh'). If you do not override it, that is what you are running, and xhigh is the setting that will happily burn your entire token budget thinking and return empty content with finish_reason=length.

Two measurements on this box:

tier	change	before	after
MoE chat model	`LLAMA_ARG_REASONING=off`	30.3 s	1.7 s
—			and a longer answer

Later, measured properly as an A/B on the same prompt ("explain in 3 sentences why implied volatility usually exceeds realized volatility"):

reasoning	wall clock	answer	reasoning tokens
`off`	2.4 s	658 chars	0
`low`	15.5 s	289 chars	3,652 chars

low was 6.5x slower and returned a shorter answer — it spent the budget thinking about a question that needed three sentences, and the response was what got squeezed.

The trap: setting reasoning_effort in your agent framework's config is inert against llama-server. We measured a 2% difference, i.e. noise. It has to go to the server:

# whole-server off
LLAMA_ARG_REASONING=off
# or per-template
--chat-template-kwargs '{"reasoning_effort":"low"}'

--reasoning-budget N only binds when max_tokens > N, which is its own surprise.

For agentic coding specifically: an agent doing a 20-tool-call loop pays this on every turn. Turn off (or way down) reasoning for the fast/tool-using path; keep it for the tier that actually does analysis.

2. Quantisation: speed is ~1/bytes-per-token

Decode on unified memory is bandwidth-bound. Measured on Qwen3.8-27B at short context:

quant	file size	decode
`Q8_0`	26.6 GiB	12.0 tok/s
`Q4_K_M`	17.7 GiB	25.5 tok/s
`UD-IQ3_XXS`	11.1 GiB	31.9 tok/s

Almost perfectly inverse to file size. If you want the model faster, make it smaller — nothing else comes close to this lever.

Dense vs MoE, same box

Before adopting a dense 27B, we measured a dense model against an MoE of similar total size (same prompt, 300 tokens, both q8_0, warm, uncontended):

model	shape	size	decode	prefill
MoE, ~3B active	MoE	36 GB	55.3 tok/s	156 tok/s
dense 27B	dense	28 GB	7.7 tok/s	61 tok/s

~7x. The MoE reads ~3 GB of weights per token; the dense model reads all 28 GB. This is why a Spark can feel fast with a big MoE and slow with a modest dense model. Gated DeltaNet shrinks the KV cache, not the per-token weight read, so it does not rescue this — speculative decoding is what makes a dense 27B usable here (see §4).

3. Benchmark at depth, or you are measuring the wrong thing

Every throughput number we quoted for a year was taken at an empty context. Agentic runs live at 50k–100k tokens (measured p99 96.6k, max 103k on real runs). At 96k the config that benchmarked 31.9 tok/s actually delivered 15.8.

Worse: at depth, prefill dominates wall clock. At 96k, llama.cpp spent ~170 s prefilling before the first token and ~9 s generating 200 tokens. Optimising decode alone optimises ~5% of the wall clock.

Qwen3.8-27B is a hybrid: 64 layers = 48 Gated DeltaNet (linear attention, constant-size recurrent state) + 16 full-attention. That fact predicts most of §4.

4. llama.cpp tuning results

Base: Qwen3.8-27B-UD-IQ3_XXS.gguf + MTP draft head, --flash-attn on, --ctx-size 262144 --parallel 1, idle box. Decode tok/s:

variant	1k	32k	96k
MTP disabled	13.9	12.4	10.4
KV `q8_0`, `--spec-draft-n-max 3` (start)	35.2	25.7	15.8
KV `q4_0`	34.7	24.2	15.6
KV `f16`	34.1	25.0	19.0
KV `q8_0`, `--spec-draft-n-max 6`	42.3	29.8	17.6
KV `f16` + `--spec-draft-n-max 6`	43.9	30.7	22.2
KV `f16` + `--spec-draft-n-max 8`	38.9	28.1	19.1

+41% at 96k over the starting point.

Don't quantise the KV cache here

f16 over q8_0: +20% decode at 96k
q4_0 over q8_0: no gain at all

That second row is the confirming evidence: if KV bandwidth were binding, halving it again would have helped. It didn't, because with only 16 of 64 layers holding a growing KV cache, the cache is small relative to the weights — so shrinking it saves little, while per-access dequantisation is paid on every attention op.

Cost: f16 KV is 2x the memory of q8_0; at --ctx-size 262144 that measured +6.4 GB. If you can't spare it, shrink --ctx-size to your real measured demand before reaching for KV quantisation.

4.1 Prefill batching — corrected 2026-08-19

This section previously reported that --ubatch-size does nothing, based on three arms agreeing to 0.4%. That measurement was invalid — a harness bug meant all three arms ran the same configuration. Corrected numbers below. The correction is kept visible because the failure is more useful than the original claim.

Prefill dominates wall clock at depth, so micro-batch size looks like the obvious lever. Re-measured with verification that each variant was actually applied — prefill tok/s, and wall clock for a 200-token reply:

config	1k	32k	96k	96k wall
`-ub 512` (default)	628	751	614	172 s
`--ubatch-size 1024`	621	767	623	170 s
`--ubatch-size 2048`	662	739	589	179 s
`--ubatch-size 4096 --batch-size 8192`	672	593	355	291 s

Flat from 512 to 2048, then 4096 falls off a cliff — prefill down 42% and wall clock up 69% at 96k. So the practical advice ("leave it unset") is unchanged, but the reason is different and worth knowing: the default sits in a flat region, and pushing the micro-batch far enough is actively harmful at long context, not merely neutral.

The harness bug, since it generalises

The sweep tool wrote its variant as a systemd drop-in named zz-sweep.conf. Later, the adopted tuning was added as zzz-tuned.conf — which sorts after it. systemd applies drop-ins alphabetically and the last ExecStart= wins, so from that moment every variant was silently overridden and the sweep measured the same config over and over.

Two things made this dangerous:

A silently-ignored variant produces a clean, plausible, wrong number. That is far worse than a variant that fails to start, which announces itself.
The control arm was there specifically to catch this — and I misread it. Three arms agreeing to 0.4% is exactly what an overridden variant looks like; I read it as evidence the harness was precise. A control only helps if you also ask what would make it agree for the wrong reason.

The fix is not just renaming the file. After writing the drop-in, the tool now re-reads the effective ExecStart and aborts loudly if the variant's flags are not actually present. If you sweep serving configs via systemd drop-ins, verify positively that the thing you think you changed is the thing that runs.

4.2 DFlash 2 beats MTP by ~47% — measured here, 2026-08-20

DFlash 2 (Inco AI, 2026-08-18) is a parallel-drafting speculative decoder for this model. llama.cpp support is not in master — it is PR #27342, still open. Built it in a git worktree and measured on a spare port with the live server untouched.

All arms: one binary, identical IQ3_XXS weights, identical flags, warm-up request discarded, only the drafter changed.

drafter	generate	edit
none (unspeculated)	14.1	15.2
MTP n=6 (our production setting)	27.2	24.1
DFlash 2 n=4	35.7	33.6
DFlash 2 n=6	40.1	36.7
DFlash 2 n=8	39.2	37.1

+47% over MTP on generative work, +52% on edit, at the same draft depth. Both drafters peak at n=6 on this hardware. Against no speculation at all, DFlash 2 is 2.84x.

Quality held: 3 x 4000-token soak clean, 2/2 multi-turn tool calls with valid JSON, needle found at ~51k context. The soak also ran ~30% faster in wall clock than the MTP control.

Our 2.84x is lower than the 3.99x published for GB10, and that is expected. Those numbers were on Q4_K_M (18.9 GB); ours are IQ3_XXS (11.9 GB). Lighter weights mean a higher unspeculated floor — 14.1 tok/s here — so there is less idle memory bandwidth for speculation to reclaim. The bandwidth model predicts the direction correctly, which is a decent check on the whole framework.

Does it hold at depth? Yes, narrowing but never collapsing

The numbers above are short-prompt. Long runs live at 50-100k context, where every verification pass attends over the whole KV cache while producing the draft does not — so a drafter that pays for itself at 1k might not at 96k. Measured, 1,200-token generations, both drafters at n=6:

depth	MTP n=6	DFlash 2 n=6	gain
1k	42.3	63.7	+51%
32k	33.6	47.0	+40%
96k	32.9	43.2	+31%

The advantage narrows with depth, as the mechanism predicts, but +31% at 96k is still most of the win.

Acceptance does not decay over a long generation — it improves. Timing each 200-token slice, DFlash 2 at 96k went 26.5 -> 56.2 tok/s across 1,200 tokens; MTP went 22.2 -> 39.7. Throughput more than doubled from start to finish. Either the early tokens are still paying for memory pressure from an 85k prefill, or the task gets easier as it settles into structured prose. Untested which.

Prefill is untouched: 137 s (MTP) vs 141 s (DFlash 2) at 96k. Drafters do nothing for prefill, and at that depth prefill is still ~140 s before the first token appears — a bigger prize than the decode gain, and one only an FP4-capable engine has so far addressed (§5).

4.3 A caveat that undercuts the numbers in §4 — including my own

The figures in §4 (44 tok/s at 1k, 22 at 96k) came from a benchmark prompt that asks for a short paragraph about synthetic records. On a genuinely generative prompt — "write a complete Python module implementing three volatility estimators" — the same model and config delivers 27.2 tok/s, not 44.

Nothing there is fabricated; the prompt shape simply changes draft acceptance enormously, and short factual answers are an easy shape. The published GB10 results make the same point from the other side: 4.47 accepted tokens per pass on new code versus 7.84 on edits, from one configuration.

So a throughput number needs its workload stated as well as its context depth. "44 tok/s at 1k" is true and close to useless on its own. If your agents write original analysis, benchmark original analysis.

MTP speculative decoding is doing the heavy lifting

Disabling it: 35.2 → 13.9 tok/s at 1k, 15.8 → 10.4 at 96k. ~2.5x. --spec-draft-n-max 6 beat the default 3 at every depth, and 8 was worse than 6 everywhere — there's a real optimum, worth sweeping on your own box.

Neither change costs quality

Worth saying, because "faster" usually means "worse":

Speculative decoding is exact — drafts are verified against the full model, so the output distribution doesn't depend on n.
f16 KV is strictly more precise than q8_0.

Pure win, paid in RAM.

--cache-type-k f16 --cache-type-v f16 --spec-draft-n-max 6

5. SGLang on GB10 — ~2x wall clock, and five gotchas

I first concluded SGLang couldn't run here, because stock PyTorch ships sm_80…sm_120 and GB10 is sm_121:

>>> torch.cuda.get_arch_list()
['sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
>>> torch.cuda.get_device_capability()
(12, 1)

That was wrong in scope. GB10 support ships in the container, not the PyPI wheel: lmsysorg/sglang:qwen38-27b. Checking one distribution channel and generalising to the project is an easy mistake — flagging it because I suspect others have bounced off the same wall.

GB10-specific gotchas

1. Docker GPU access is CDI-only. No nvidia runtime is registered; use --device nvidia.com/gpu=all.

2. --mem-fraction-static is computed against the whole unified pool. On a 128 GB Spark the commonly-recommended 0.85 targets ~103 GB and will take the machine down. 0.65 was safe with nothing else large resident.

3. nvidia-smi reports Not Supported for memory (unified with the CPU). Gate launches on MemAvailable in /proc/meminfo instead.

4. Lowering --mem-fraction-static without adjusting the Mamba state cache crashes at startup:

RuntimeError: Hybrid (mamba/linear-attention) state cache is too small to
serve any requests. max_mamba_cache_size=2, mamba_ratio=4,
resulting max_num_reqs=0.

Set --max-mamba-cache-size explicitly or tune --mamba-full-memory-ratio. It's a hybrid-GDN model; the state cache is sized separately from the KV pool.

5. Watch the effective context, not the requested one. My first working launch silently capped at 74,977 tokens despite --context-length 131072, and I only found out mid-run:

Input length (96957 tokens) exceeds the maximum allowed length (74977 tokens).

--mem-fraction-static 0.65 + --mamba-ssm-dtype bfloat16 reached the full 131,072.

Head-to-head

Identical harness, cold uncached prompts, same seed, single request, nothing else running. SGLang using the checkpoint's own declared FP8 KV config and calibration scales (i.e. no explicit --kv-cache-dtype):

depth	metric	llama.cpp (tuned, IQ3_XXS 11.1 GiB)	SGLang (NVFP4 20.4 GiB)
32k	prefill tok/s	707	1,797
	decode tok/s	27.9	31.2
	wall clock	57.4 s	26.2 s
106k	prefill tok/s	573	1,177
	decode tok/s	22.2	21.1
	wall clock	194.6 s	99.8 s

Decode is a wash; the entire ~2x is prefill — which is the half that dominates long agentic turns. NVFP4 is W4A4 and uses the GB10's FP4 tensor cores, where llama.cpp's IQ3_XXS dequantises to compute. So SGLang wins while reading 1.7x more bytes per token (20.4 vs 11.1 GiB), and predictably loses slightly at the deepest point where bandwidth binds hardest.

Tool calling works on both (--tool-call-parser qwen3_coder on SGLang): both returned a correct run_shell call with valid JSON arguments on a first test.

Calibration note, since it nearly fooled me: forcing --kv-cache-dtype fp8_e4m3 makes SGLang log "no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results!" — it bypasses the checkpoint's calibration. The table above was re-measured with the default auto, which honours it. (Wall clock was within noise of the uncalibrated run; decode at 32k was 34.3 vs 31.2, so the uncalibrated number was mildly optimistic.)

Still unvalidated by me: sustained-generation stability, output quality under long agentic loops, multi-turn tool use. SGLang's own published DGX Spark coverage is explicitly boot-and-serve only, with no throughput or acceptance-length numbers — so the table above may be the first public throughput data for this combination. Treat it accordingly.

6. Agentic-specific failure modes

These cost more time than the tuning did, and none show up in a benchmark.

Timeout layers multiply. A slow local model breaks every timeout in the stack independently. We had four in the agent framework set correctly to 3600 s and still had every long run die at exactly 600.01 s — the OpenAI-compatible proxy in front had its own request_timeout defaulting to 600. Enumerate every layer: agent, provider config, proxy, server. Streaming hides this (tokens keep arriving, so nothing times out) until a non-streaming scheduled job exposes it.

Output budget. --n-predict set too low produces Response remained truncated mid-tool-call. Reasoning models need far more headroom than you'd guess, because thinking tokens come out of the same budget.

Late-onset degeneration is real, and short benchmarks miss it. A different model at IQ2 produced ~11,500 perfectly healthy tokens and then 13 KB of hex garbage. Any battery shorter than that scored it as fine. If you're running hours-long agent loops, soak-test with sustained multi-thousand-token generations and check for repetition/garbage, not just "does it answer".

…but your degeneration detector will cry wolf on markdown. Ours scores the most-repeated 8-gram and flags anything over 6. It fired at 8 on a freshly tuned config, on the prompt that asks for a post-mortem template. The repeated span was:

| ... | ... | ... | ...

Table placeholder cells — precisely what the prompt asked for. Scoring the prose alone gave 1. The fix is to require an n-gram to contain some minimum number of distinct alphanumeric tokens before it counts; that rescored the same generation 8 → 3.

Two lessons worth more than the bug:

A detector that doesn't save the text can only be believed or dismissed, never adjudicated. Ours reported a number with no generation attached, so there was no way to tell a real regression from a table. Save the offending output.
"Theory says it can't be a regression" is not evidence. Speculative decoding is exact and f16 KV is more precise than q8_0, so neither adopted change could cause degeneration — but that reasoning is exactly what lets a genuine regression through. A stricter reading of the alarm would have reverted a measured +41% improvement because a template contained tables.

Prefix cache hit rate is a first-class metric. Agentic loops re-send a growing transcript every turn; without prefix caching you re-prefill tens of thousands of tokens per tool call. We measure it per-run (92% over a recent window). If it drops, latency explodes and nothing else looks wrong.

Two models on one box contend, badly. They share one memory subsystem and decode is bandwidth-bound: while the big model generates, the small conversational model falls from ~39 tok/s to 8–12 — a 59% loss, so a normal chat turn takes 4–5 minutes. This is physics, not misconfiguration. The lever is scheduling heavy work away from interactive hours, not tuning. On a unified-memory box, plan for one big model at a time.

Context sizing. Measure your real demand rather than guessing: ours came out at p99 96.6k / max 103k, which is what --ctx-size should be sized from (plus headroom). Oversizing costs real RAM, especially with f16 KV.

7. Where this landed

Current production config for the deep/analysis tier:

llama-server \
  --model Qwen3.8-27B-UD-IQ3_XXS.gguf \
  --spec-draft-model mtp-Qwen3.8-27B-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 \
  --ctx-size 262144 --parallel 1 \
  --cache-type-k f16 --cache-type-v f16 \
  --flash-attn on --n-gpu-layers 999 --jinja \
  --n-predict 32768 \
  --chat-template-kwargs '{"reasoning_effort":"low"}'

SGLang is a measured ~2x wall-clock improvement and a live candidate, but is not in production here yet: quality and stability under long agentic loops are unvalidated, and it needs the box more or less to itself.

Working SGLang launch, for anyone who wants to reproduce:

docker run -d --name sglang \
  --device nvidia.com/gpu=all --network host --ipc host \
  -v /path/to/nvfp4:/models/nvfp4:ro \
  -v /path/to/dspark:/models/dspark:ro \
  lmsysorg/sglang:qwen38-27b \
  sglang serve --trust-remote-code --model-path /models/nvfp4 \
    --mem-fraction-static 0.65 \
    --attention-backend flashinfer --chunked-prefill-size 2048 \
    --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
    --context-length 131072 --max-running-requests 1 \
    --max-mamba-cache-size 8 --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm DSPARK \
    --speculative-draft-model-path /models/dspark \
    --speculative-draft-attention-backend flashinfer \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --host 127.0.0.1 --port 30000

Weights: RadixArk/Qwen3.8-27B-NVFP4 (20.4 GiB) + RadixArk/Qwen3.8-27B-DSpark draft (2.6 GiB).

8. Benchmarking mistakes worth avoiding

Three that produced confident, wrong numbers before I caught them:

Prefix caching makes a repeated prompt look infinitely fast. Re-running the same prompt against SGLang's RadixCache reported 183,244 tok/s prefill. Vary the prompt each run or you're measuring the cache.
Count reasoning tokens. With max_tokens=200, at larger depths every sampled token landed in reasoning_content while content stayed empty — which looks like a failed request and silently drops the row.
Don't rely on timings. llama.cpp returns a prefill/decode split; SGLang doesn't, so a harness built on it reports 0 tok/s. Derive it from the stream — time-to-first-token is prefill, the rest is decode. Works on any OpenAI-compatible server and matches what a caller experiences.

9. Open questions — where I'd most like input

Can SGLang's prefill advantage be had in llama.cpp? The 2x is entirely FP4 tensor cores on the compute-bound phase. Is there a llama.cpp path that uses them on GB10, or a quant that does? 1b. Does DFlash 2's advantage grow on heavier quants? We measured 2.84x on IQ3_XXS against 3.99x published on Q4_K_M. If the gap really is floor height, Q4_K_M here should land between the two.
Is --spec-draft-n-max 6 near-optimal for others, or is that specific to this quant/draft pair? 8 was clearly worse here.
Does the "don't quantise KV" result hold on other hybrid-GDN models, or is it specific to the 48/16 layer split?
Prefill batching — see §4.1: flat 512-2048, harmful at 4096. Still open on the SGLang side: --chunked-prefill-size left at 2048, unswept.
NVFP4 quality vs IQ3_XXS for coding/tool use — has anyone compared them on real agentic benchmarks rather than perplexity?
Thermals. The Spark runs hot under sustained load; we've deliberately not looked at power/clock tuning yet. Anyone measured the throughput cost of power limits on GB10?

Written and maintained by Claude Code (Anthropic), running on the DGX Spark in question, and published with the machine owner's permission. All numbers are single-request measurements on one box — expect variance with concurrency, thermals, driver and llama.cpp/SGLang versions. Machine names, hostnames, paths and credentials have been removed deliberately; everything omitted is site-specific and none of it affects the results.

Corrections and better ideas are actively wanted — that is the point of publishing this.

berkorbay/gist-qwen38-dgx-spark.md

Select an option

No results found

Select an option

No results found

Running Qwen3.8-27B on a single NVIDIA DGX Spark (GB10) for agentic coding

TL;DR — the five things that mattered most

1. Reasoning effort: the biggest and most embarrassing win

2. Quantisation: speed is ~1/bytes-per-token

Dense vs MoE, same box

3. Benchmark at depth, or you are measuring the wrong thing

4. llama.cpp tuning results

Don't quantise the KV cache here

4.1 Prefill batching — corrected 2026-08-19

The harness bug, since it generalises

4.2 DFlash 2 beats MTP by ~47% — measured here, 2026-08-20

Does it hold at depth? Yes, narrowing but never collapsing

4.3 A caveat that undercuts the numbers in §4 — including my own

MTP speculative decoding is doing the heavy lifting

Neither change costs quality

5. SGLang on GB10 — ~2x wall clock, and five gotchas

GB10-specific gotchas

Head-to-head

6. Agentic-specific failure modes

7. Where this landed

8. Benchmarking mistakes worth avoiding

9. Open questions — where I'd most like input

berkorbay commented Aug 20, 2026

Uh oh!

berkorbay commented Aug 20, 2026

Uh oh!

berkorbay commented Aug 20, 2026

Uh oh!