Skip to content

Instantly share code, notes, and snippets.

@cguldogan

cguldogan/vllm serve Qwen3.8-27B-FP8

Created August 28, 2026 02:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save cguldogan/641a38b20a6309a68a1aa1a62a590c22 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save cguldogan/641a38b20a6309a68a1aa1a62a590c22 to your computer and use it in GitHub Desktop.
Download ZIP
vllm serve Qwen/Qwen3.8-27B-FP8
Raw
vllm serve Qwen3.8-27B-FP8
vllm serve Qwen/Qwen3.8-27B-FP8 \
--tensor-parallel-size 1 \
--data-parallel-size 2 \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 8 \
--enable-prefix-caching \
--language-model-only \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment