RTX PRO 6000 Blackwell: vLLM vs llama.cpp spin-up + launch commands (Qwen3.6 27B)

README.md

RTX PRO 6000 Blackwell: vLLM vs llama.cpp spin-up (Qwen3.6 27B)

Exact setup and launch commands behind this thread: https://x.com/superalesha/status/2078481384805732369

Single card, local agents. Two 4-bit artifacts of the same model family (matched class, not bit-identical): NVFP4 for vLLM, UD-Q4_K_XL GGUF for llama.cpp.

Machine fingerprint

• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB, ECC on, 600 W limit
• VBIOS 98.02.81.00.01, compute cap sm_120
• Driver 610.43.02, CUDA 13.3 (nvcc 13.3.33)
• OS: Debian 13.5 (trixie), kernel 6.12.90-cloud-amd64, KVM guest
• vLLM 0.25.0 (venv), llama.cpp built from source for CUDA sm_120a
• Models:
  • unsloth/Qwen3.6-27B-NVFP4 (vLLM, loads as compressed-tensors)
  • unsloth/Qwen3.6-27B-MTP-GGUF -> Qwen3.6-27B-UD-Q4_K_XL.gguf (llama.cpp)

Two things that save you hours

1. On a fresh box, vLLM's first start is ~10 min of compilation, not serving. Warm restart was 48 s. Do not judge it by the cold start. llama.cpp is ~83 s cold, ~6.5 s warm.
2. Pin the vLLM KV pool so capacity does not drift between restarts. This run used --gpu-memory-utilization 0.90; for byte-stable capacity across boots prefer --kv-cache-memory-bytes <N> (newer vLLM) instead of trusting the boot-time profiler.

vLLM baseline

vllm serve /path/to/unsloth-Qwen3.6-27B-NVFP4 \
  --served-model-name bench \
  --host 0.0.0.0 --port 18000 \
  --max-model-len 65536 \
  --quantization compressed-tensors \
  --language-model-only \
  --generation-config vllm \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 32768 \
  --enable-prefix-caching \
  --no-enable-log-requests

Speculative (MTP): append the checkpoint's built-in multi-token-prediction head, no separate draft model needed.

  # ... same as above, plus:
  --spec-method mtp \
  --spec-tokens 3        # k=2 or k=3; k=3 was faster despite lower acceptance

llama.cpp baseline

llama-server \
  --model /path/to/Qwen3.6-27B-UD-Q4_K_XL.gguf \
  --alias bench \
  --host 0.0.0.0 --port 18000 \
  --ctx-size 270336 \
  --parallel 8 \
  --device CUDA0 \
  --n-gpu-layers all \
  --flash-attn on \
  --cache-type-k f16 --cache-type-v f16 \
  --spec-type none

--parallel 8 gives 8 slots, so ctx is split per slot (270336 / 8 ~= 33k each). KV for all slots is pre-allocated at startup, so peak VRAM barely moves with load.

Speculative (MTP draft head from the same GGUF):

  # ... same as above, but replace --spec-type none with:
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-type-k f16 --spec-draft-type-v f16

Measurement notes

• Client wall-clock only. Server-internal timings kept in separate columns.
• 5 warmup + 10 measured per short slice; 2+5 on the heavy 32K slice.
• Numbers, full matrix, raw logs and telemetry are in a packaged archive with a per-file manifest and checksums. Ask if you want the whole bundle.