Skip to content

Instantly share code, notes, and snippets.

@Whamp

Whamp/README.md

Created July 27, 2026 15:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Whamp/1e584027cc99035068063e95b9d99b89 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Whamp/1e584027cc99035068063e95b9d99b89 to your computer and use it in GitHub Desktop.
Download ZIP
club-3090 TP=4 FP8/e4m3 vs INT8-PTH KV same-session A/B, vLLM 0.25.1, 4x RTX 3090
Raw
README.md

club-3090 TP=4 KV-cache A/B — 2026-07-27

Question

Does Whamp's old short-context throughput gain from int8_per_token_head KV survive on current vLLM, and how does it behave as context depth grows?

Fixed conditions

  • Repo commit: 86699d21b34f53c41ceb00357cb599f57523caa3
  • Slug: vllm/qwen-27b-multi-fast
  • Image / engine: vllm/vllm-openai:v0.25.1
  • Weights: Lorbus/Qwen3.6-27B-int4-AutoRound, SHA-verified by scripts/setup.sh
  • Model recipe: AutoRound INT4, MTP n=3, TP=4, max context 262,144, prefix caching on
  • Hardware: 4× RTX 3090, PCIe-only, no NVLink
  • Power cap: 230 W on every card before and after both arms
  • Driver/module: NVIDIA 595.71.05, patched open module, Linux 7.0.0-28-generic
  • P2P: enabled at launch with NCCL_P2P_LEVEL=PHB; nvidia-smi topo -p2p rw reported all 12 directed pairs OK
  • All-reduce: PYNCCL only; vLLM engine-gated custom all-reduce off at TP=4 in both arms
  • Run order: FP8/e4m3, then INT8-PTH

The only intended serving-variable change was KV_CACHE_DTYPE.

Resolved backend

Arm KV dtype Attention backend Candidate list
FP8 fp8_e4m3 FLASHINFER FLASHINFER, TRITON_ATTN
INT8-PTH int8_per_token_head TRITON_ATTN TRITON_ATTN only

Canonical short-context benchmark

Three warmups followed by five measured requests per prompt.

Metric FP8/e4m3 INT8-PTH INT8 delta
Narrative wall TPS 58.72 103.14 +75.6%
Narrative decode TPS 59.38 104.81 +76.5%
Code wall TPS 74.59 133.66 +79.2%
Code decode TPS 76.34 138.13 +80.9%
Narrative TTFT 188 ms 153 ms −18.6%
Code TTFT 184 ms 156 ms −15.2%

The old short-context result survives and is larger on this v0.25.1 run than the original v0.22.0 +57%/+65% wall-TPS result.

Fresh-prefill probes

One warmup and three measured, salted/cache-busted requests per depth.

Metric FP8/e4m3 INT8-PTH INT8 delta
Prefill @ ~10K 1,102.30 tok/s 1,061.42 tok/s −3.7%
TTFT @ ~10K 8.751 s 9.088 s +3.9%
Prefill @ ~90K 988.72 tok/s 735.06 tok/s −25.7%
TTFT @ ~90K 93.276 s 130.333 s +39.7%

Agentic decode-at-depth

Two independent 15-turn fixture sessions per arm. Prompt depth reached ~58K tokens. Values are mean decode TPS across the two sessions.

Prompt tokens FP8/e4m3 INT8-PTH INT8 delta
~1.5K warm baseline 104.8 163.6 +56.1%
~5.0K 94.2 120.9 +28.3%
~7.7K 102.8 102.1 −0.7%
~9.1K 86.5 76.6 −11.4%
~12.4K 119.8 103.0 −14.0%
~21.5K 107.2 64.4 −39.9%
~27.7K 103.9 52.6 −49.4%
~35.4K 99.3 45.5 −54.2%
~43.3K 113.2 40.5 −64.2%
~47.5K 121.5 40.6 −66.6%
~58.1K 119.2 33.5 −71.9%

At the deepest point, INT8-PTH TTFT was 19.689 s versus 13.973 s for FP8 (+40.9%). No tool-call misses were reported in either arm.

VRAM

Observed after canonical benchmark:

  • FP8: approximately 22,960–23,000 MiB/card
  • INT8-PTH: approximately 22,162–22,202 MiB/card
  • INT8-PTH saves roughly 798 MiB/card (~3.1 GiB aggregate)

Verdict

int8_per_token_head remains a very strong short-context accelerator on this TP=4 Ampere rig, but it is not a generally faster KV format. Its advantage is gone by roughly 8K prompt tokens; thereafter the Triton-only path degrades rapidly. At ~58K, FP8/FlashInfer is 3.56× faster in decode (119.2 vs 33.5 TPS) and has materially lower TTFT.

Practical split:

  • Short, interactive requests below roughly 5–8K: INT8-PTH wins decisively and saves VRAM.
  • Long-running agents / deep context: FP8/e4m3 is the clear choice.

Limitations

  • Arm order was not counterbalanced: FP8 ran first, INT8-PTH second.
  • Agentic completions are sampled, so generated token counts and exact prompt-token depths differ slightly between arms; the fixed fixture and two-session means keep the depth curve comparable, but this is not a token-identical decode microbenchmark.
  • One rig, one power cap, one model, one engine version.
  • P2P engagement was launcher/log-resolved and driver-reported; this run did not independently generate a new vLLM transfer-cache artifact.

Raw evidence

  • Rig metadata: rig.txt
  • FP8 arm: fp8/switch.log, fp8/resolved.txt, fp8/bench.log, fp8/bench-agentic.log, fp8/docker.log
  • INT8-PTH arm: int8-pth/switch.log, int8-pth/resolved.txt, int8-pth/bench.log, int8-pth/bench-agentic.log, int8-pth/docker-full.log
Raw
fp8-agentic-depth.log
========================================================================
SESSION 1/2 — 15 turns, context grows to ~45,889 tokens
========================================================================
Turn Prompt tok TTFT ms Decode TPS Result chars
----- ---------- --------- ----------- -------------
1 1,314 2720 99.0 307
2 1,501 1484 107.4 249
3 1,686 1645 95.2 278
4 1,896 1833 107.1 8,353
5 4,947 4577 105.4 8,912
6 7,676 4212 100.1 3,106
7 9,026 4020 88.0 6,495
8 10,966 4366 107.3 2,576
9 12,308 4144 128.0 25,250
10 21,472 11251 111.2 17,397
11 27,621 8220 106.3 21,289
12 35,317 9627 96.1 21,873
13 43,222 9743 109.8 11,279
14 47,395 6134 117.2 31,817
15 58,076 13964 125.1 24,305
========================================================================
SESSION 2/2 — 15 turns, context grows to ~45,889 tokens
========================================================================
Turn Prompt tok TTFT ms Decode TPS Result chars
----- ---------- --------- ----------- -------------
1 1,314 1370 81.4 307
2 1,535 1567 102.2 249
3 1,730 1700 91.3 278
4 1,952 1884 108.5 8,353
5 5,005 4640 83.0 8,912
6 7,739 4289 105.6 3,106
7 9,087 4119 85.1 6,495
8 11,033 4424 101.6 2,576
9 12,379 4206 111.5 25,250
10 21,545 11329 103.2 17,397
11 27,696 8379 101.5 21,289
12 35,394 9761 102.4 21,873
13 43,301 9831 116.5 11,279
14 47,476 6216 125.9 31,817
15 58,159 13983 113.3 24,305
========================================================================
SUMMARY — multi-turn prefill stress (2 session(s) × 15 turns)
========================================================================
Turn Prompt tok TTFT ms σ ms Decode TPS Notes
----- ---------- --------- ------ ----------- ───────────────────────────────────
1 1,314 2045 954 90.2 cold-start (compile/warmup — excluded from growth)
2 1,518 1526 59 104.8 warm baseline
3 1,708 1672 39 93.3
4 1,924 1859 36 107.8
5 4,976 4609 45 94.2 ↑ TTFT 3.0× warm-baseline
6 7,708 4251 54 102.8 ↑ TTFT 2.8× warm-baseline
7 9,056 4070 70 86.5 ↑ TTFT 2.7× warm-baseline
8 11,000 4395 41 104.4 ↑ TTFT 2.9× warm-baseline
9 12,344 4175 44 119.8 ↑ TTFT 2.7× warm-baseline
10 21,508 11290 55 107.2 ⚠ TTFT 7.4× warm-baseline (O(n)-like growth for this arch_class)
11 27,658 8299 112 103.9 ⚠ TTFT 5.4× warm-baseline (O(n)-like growth for this arch_class)
12 35,356 9694 95 99.3 ⚠ TTFT 6.4× warm-baseline (O(n)-like growth for this arch_class)
13 43,262 9787 62 113.2 ⚠ TTFT 6.4× warm-baseline (O(n)-like growth for this arch_class)
14 47,436 6175 58 121.5 ⚠ TTFT 4.0× warm-baseline (O(n)-like growth for this arch_class)
15 58,118 13973 13 119.2 ⚠ TTFT 9.2× warm-baseline (O(n)-like growth for this arch_class)
────────────────────────────────────────────────────────────────────────
TTFT growth by accumulated context (15 turns, 2 sessions):
Turn 1 (cold): 2045 ms TTFT — compile/warmup, excluded from growth
Turn 2 (warm base): 1526 ms TTFT @ 1,518 prompt tokens
Turn 15: 13973 ms TTFT @ 58,118 prompt tokens
Context grew 38.3×, TTFT grew 9.2× (warm baseline → last turn)
~ TTFT sub-linear for this cell (9.2× vs 38.3× context).
(Full-context O(n) growth would approach 38.3× with context)
Note — DeltaNet/SSM state is NOT prefix-cacheable on vLLM Qwen3-Next cells.
Attention KV caching can still work, but recurrent-state recomputation scales
O(n) with sequence length. Prior single-card 24 GB vLLM Qwen3-Next observations
saw degradation above ~35K tokens and timeouts around ~74K; treat those as
informational per-arch_class guideposts. llama.cpp is not affected.
=== GPU state ===
0, 60 %, 22992 MiB, 24576 MiB, 148.40 W, 61
1, 75 %, 23000 MiB, 24576 MiB, 155.56 W, 61
2, 74 %, 23000 MiB, 24576 MiB, 162.65 W, 69
3, 56 %, 22960 MiB, 24576 MiB, 162.25 W, 58
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 07-27 15:29:47 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 3.60 tokens/s, Drafted throughput: 3.40 tokens/s, Accepted: 36 tokens, Drafted: 34 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 105.9%
(APIServer pid=1) INFO 07-27 15:29:57 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 2.40 tokens/s, Drafted throughput: 2.40 tokens/s, Accepted: 24 tokens, Drafted: 24 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 07-27 15:30:07 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 2.10 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 21 tokens, Drafted: 21 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
Raw
fp8-bench.log
[autodetect] served model='qwen3.6-27b' (from http://localhost:8014/v1/models; set MODEL= to override)
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
warm-1 wall= 21.36s ttft= 2898ms toks=1000 wall_TPS= 46.83 decode_TPS= 54.18
warm-2 wall= 16.86s ttft= 138ms toks=1000 wall_TPS= 59.32 decode_TPS= 59.81
warm-3 wall= 16.71s ttft= 149ms toks= 976 wall_TPS= 58.40 decode_TPS= 58.93
=== measured (5) ===
run-1 wall= 17.12s ttft= 191ms toks=1000 wall_TPS= 58.41 decode_TPS= 59.07
run-2 wall= 17.31s ttft= 193ms toks=1000 wall_TPS= 57.78 decode_TPS= 58.43
run-3 wall= 16.85s ttft= 187ms toks=1000 wall_TPS= 59.35 decode_TPS= 60.02
run-4 wall= 16.83s ttft= 186ms toks=1000 wall_TPS= 59.42 decode_TPS= 60.09
run-5 wall= 17.05s ttft= 186ms toks=1000 wall_TPS= 58.64 decode_TPS= 59.29
=== summary [narrative] (n=5) ===
wall_TPS mean= 58.72 std= 0.69 CV= 1.2% min=57.78 max=59.42
decode_TPS mean= 59.38 std= 0.69 CV= 1.2% min=58.43 max=60.09
TTFT mean= 188ms std= 3ms min=186ms max=193ms
PP tok/s mean= 1.50 std= 1.37 CV=91.3% min=0.00 max=2.50
========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
warm-1 wall= 8.90s ttft= 154ms toks= 687 wall_TPS= 77.21 decode_TPS= 78.57
warm-2 wall= 5.85s ttft= 183ms toks= 453 wall_TPS= 77.37 decode_TPS= 79.86
warm-3 wall= 10.62s ttft= 184ms toks= 800 wall_TPS= 75.30 decode_TPS= 76.63
=== measured (5) ===
run-1 wall= 9.30s ttft= 148ms toks= 706 wall_TPS= 75.95 decode_TPS= 77.18
run-2 wall= 9.31s ttft= 184ms toks= 704 wall_TPS= 75.59 decode_TPS= 77.11
run-3 wall= 8.81s ttft= 197ms toks= 661 wall_TPS= 75.05 decode_TPS= 76.77
run-4 wall= 7.49s ttft= 198ms toks= 548 wall_TPS= 73.14 decode_TPS= 75.12
run-5 wall= 6.34s ttft= 194ms toks= 464 wall_TPS= 73.21 decode_TPS= 75.52
=== summary [code] (n=5) ===
wall_TPS mean= 74.59 std= 1.33 CV= 1.8% min=73.14 max=75.95
decode_TPS mean= 76.34 std= 0.96 CV= 1.3% min=75.12 max=77.18
TTFT mean= 184ms std= 21ms min=148ms max=198ms
PP tok/s mean= 3.00 std= 1.12 CV=37.3% min=2.50 max=5.00
========== PREFILL-10K (target=10000 prompt tokens, max_tokens=16, cache-busted: fresh haystack per run) ==========
=== warmups (1) ===
warm-1 wall= 12.51s ttft= 12476ms prompt_toks= 13363 PP_tok/s=1071.09
[calibrated: 13363 tok at request=10000 → measured runs request 7483]
=== measured (3) ===
run-1 wall= 8.80s ttft= 8764ms prompt_toks= 9646 PP_tok/s=1100.60
run-2 wall= 8.78s ttft= 8744ms prompt_toks= 9646 PP_tok/s=1103.11
run-3 wall= 8.78s ttft= 8744ms prompt_toks= 9646 PP_tok/s=1103.19
=== summary [prefill-10k] (n=3) ===
prefill tok/s mean=1102.30 std= 1.48 CV= 0.1% min=1100.60 max=1103.19
TTFT mean= 8751ms std= 12ms min=8744ms max=8764ms
PP tok/s (engine log, windowed) mean=1089.30 std=986.76 CV=90.6% min=2.50 max=1929.10
========== PREFILL-90K (target=90000 prompt tokens, max_tokens=16, cache-busted: fresh haystack per run) ==========
=== warmups (1) ===
warm-1 wall=120.63s ttft=120623ms prompt_toks=115729 PP_tok/s= 959.43
[calibrated: 115729 tok at request=90000 → measured runs request 69991]
=== measured (3) ===
run-1 wall= 94.54s ttft= 94535ms prompt_toks= 93331 PP_tok/s= 987.27
run-2 wall= 94.53s ttft= 94529ms prompt_toks= 93331 PP_tok/s= 987.33
run-3 wall= 90.77s ttft= 90764ms prompt_toks= 89998 PP_tok/s= 991.56
=== summary [prefill-90k] (n=3) ===
prefill tok/s mean= 988.72 std= 2.46 CV= 0.2% min=987.27 max=991.56
TTFT mean= 93276ms std= 2175ms min=90764ms max=94535ms
PP tok/s (engine log, windowed) mean=10079.37 std=1292.92 CV=12.8% min=9332.60 max=11572.30
=== GPU state ===
0, 27 %, 22992 MiB, 24576 MiB, 151.65 W, 60
1, 26 %, 23000 MiB, 24576 MiB, 154.85 W, 60
2, 22 %, 23000 MiB, 24576 MiB, 162.36 W, 69
3, 0 %, 22960 MiB, 24576 MiB, 160.97 W, 59
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 07-27 15:22:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.02 tokens/s, Drafted throughput: 0.02 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 07-27 15:24:07 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.03 tokens/s, Drafted throughput: 0.03 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 07-27 15:25:37 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.03 tokens/s, Drafted throughput: 0.03 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
Raw
fp8-resolved.log
--- resolved evidence ---
[nvlink] 4 GPUs — no NVLink, but nvidia-smi reports P2P=OK (patched driver / P2P-capable layout) — auto-enabling PCIe P2P (NCCL_P2P_LEVEL=PHB, custom all-reduce engine-gated (vLLM disables its custom kernel at >2 PCIe-only GPUs — P2P runs via NCCL; #786))
[nvlink] P2P ENABLED — NCCL_P2P_LEVEL=PHB, custom all-reduce engine-gated (vLLM disables its custom kernel at >2 PCIe-only GPUs — P2P runs via NCCL; #786), expandable_segments stripped (PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512)
(APIServer pid=1) INFO 07-27 15:10:22 [api_utils.py:273] non-default args: {'model_tag': '/root/.cache/huggingface/qwen3.6-27b-autoround-int4', 'chat_template': '/etc/qwen-froggeric-chat-template.jinja', 'default_chat_template_kwargs': {'enable_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '/root/.cache/huggingface/qwen3.6-27b-autoround-int4', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 262144, 'quantization': 'auto_round', 'served_model_name': ['qwen3.6-27b', 'qwen3.6-27b-autoround'], 'override_generation_config': {'temperature': 0.6, 'top_p': 0.95, 'top_k': 20, 'min_p': 0.0, 'repetition_penalty': 1.0}, 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 4, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 2, 'long_prefill_token_threshold': 2048, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(EngineCore pid=204) INFO 07-27 15:11:18 [core.py:114] Initializing a V1 LLM engine (v0.25.1) with config: model='/root/.cache/huggingface/qwen3.6-27b-autoround-int4', speculative_config=SpeculativeConfig(method='mtp', model='/root/.cache/huggingface/qwen3.6-27b-autoround-int4', num_spec_tokens=3), tokenizer='/root/.cache/huggingface/qwen3.6-27b-autoround-int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=inc, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=qwen3.6-27b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
(Worker pid=217) WARNING 07-27 15:11:33 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=216) WARNING 07-27 15:11:33 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=215) WARNING 07-27 15:11:33 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=214) WARNING 07-27 15:11:33 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=214) INFO 07-27 15:11:33 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker_TP0 pid=214) INFO 07-27 15:11:44 [cuda.py:476] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(Worker_TP3 pid=217) WARNING 07-27 15:15:08 [compilation.py:1409] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(Worker_TP2 pid=216) WARNING 07-27 15:15:08 [compilation.py:1409] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(Worker_TP1 pid=215) WARNING 07-27 15:15:08 [compilation.py:1409] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(Worker_TP0 pid=214) WARNING 07-27 15:15:08 [compilation.py:1409] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE); setting cudagraph_mode=PIECEWISE
(Worker_TP0 pid=214) INFO 07-27 15:15:08 [flashinfer.py:780] FlashInfer resolved query dtypes: prefill=torch.float16, decode=torch.float16, decode_backend=flashinfer-native, kv_cache_dtype=torch.float8_e4m3fn, arch=sm86
--- container env ---
NVIDIA_VISIBLE_DEVICES=all
NVLINK_MODE=auto
--- power caps before ---
0, 230.00 W
1, 230.00 W
2, 230.00 W
3, 230.00 W
Raw
int8-pth-agentic-depth.log
========================================================================
SESSION 1/2 — 15 turns, context grows to ~45,889 tokens
========================================================================
Turn Prompt tok TTFT ms Decode TPS Result chars
----- ---------- --------- ----------- -------------
1 1,314 2631 135.5 307
2 1,535 1468 168.5 249
3 1,730 1644 147.0 278
4 1,952 1841 172.5 8,353
5 5,005 4684 113.4 8,912
6 7,751 4448 101.6 3,106
7 9,111 4319 85.7 6,495
8 11,057 4753 90.8 2,576
9 12,415 4608 97.1 25,250
10 21,581 12792 62.4 17,397
11 27,732 10043 51.0 21,289
12 35,430 12309 46.0 21,873
13 43,321 13137 41.7 11,279
14 47,496 8768 42.0 31,817
15 58,179 19765 31.6 24,305
========================================================================
SESSION 2/2 — 15 turns, context grows to ~45,889 tokens
========================================================================
Turn Prompt tok TTFT ms Decode TPS Result chars
----- ---------- --------- ----------- -------------
1 1,314 1433 171.9 307
2 1,505 1441 158.8 249
3 1,696 1608 150.3 278
4 1,918 1810 196.8 8,353
5 4,969 4646 128.3 8,912
6 7,710 4407 102.5 3,106
7 9,066 4281 67.5 6,495
8 11,017 4720 92.0 2,576
9 12,371 4563 109.0 25,250
10 21,535 12738 66.5 17,397
11 27,684 9938 54.3 21,289
12 35,380 12262 45.0 21,873
13 43,285 13094 39.2 11,279
14 47,458 8734 39.2 31,817
15 58,139 19612 35.5 24,305
========================================================================
SUMMARY — multi-turn prefill stress (2 session(s) × 15 turns)
========================================================================
Turn Prompt tok TTFT ms σ ms Decode TPS Notes
----- ---------- --------- ------ ----------- ───────────────────────────────────
1 1,314 2032 847 153.7 cold-start (compile/warmup — excluded from growth)
2 1,520 1455 19 163.6 warm baseline
3 1,713 1626 25 148.7
4 1,935 1825 22 184.7
5 4,987 4665 27 120.9 ↑ TTFT 3.2× warm-baseline
6 7,730 4427 29 102.1 ↑ TTFT 3.0× warm-baseline
7 9,088 4300 27 76.6 ↑ TTFT 3.0× warm-baseline
8 11,037 4737 24 91.4 ↑ TTFT 3.3× warm-baseline
9 12,393 4586 32 103.0 ↑ TTFT 3.2× warm-baseline
10 21,558 12765 38 64.4 ⚠ TTFT 8.8× warm-baseline (O(n)-like growth for this arch_class)
11 27,708 9990 74 52.6 ⚠ TTFT 6.9× warm-baseline (O(n)-like growth for this arch_class)
12 35,405 12285 33 45.5 ⚠ TTFT 8.4× warm-baseline (O(n)-like growth for this arch_class)
13 43,303 13116 31 40.5 ⚠ TTFT 9.0× warm-baseline (O(n)-like growth for this arch_class)
14 47,477 8751 24 40.6 ⚠ TTFT 6.0× warm-baseline (O(n)-like growth for this arch_class)
15 58,159 19689 108 33.5 ⚠ TTFT 13.5× warm-baseline (O(n)-like growth for this arch_class)
────────────────────────────────────────────────────────────────────────
TTFT growth by accumulated context (15 turns, 2 sessions):
Turn 1 (cold): 2032 ms TTFT — compile/warmup, excluded from growth
Turn 2 (warm base): 1455 ms TTFT @ 1,520 prompt tokens
Turn 15: 19689 ms TTFT @ 58,159 prompt tokens
Context grew 38.3×, TTFT grew 13.5× (warm baseline → last turn)
~ TTFT sub-linear for this cell (13.5× vs 38.3× context).
(Full-context O(n) growth would approach 38.3× with context)
Note — DeltaNet/SSM state is NOT prefix-cacheable on vLLM Qwen3-Next cells.
Attention KV caching can still work, but recurrent-state recomputation scales
O(n) with sequence length. Prior single-card 24 GB vLLM Qwen3-Next observations
saw degradation above ~35K tokens and timeouts around ~74K; treat those as
informational per-arch_class guideposts. llama.cpp is not affected.
=== GPU state ===
0, 100 %, 22194 MiB, 24576 MiB, 125.24 W, 60
1, 100 %, 22202 MiB, 24576 MiB, 131.17 W, 60
2, 100 %, 22202 MiB, 24576 MiB, 136.04 W, 69
3, 100 %, 22162 MiB, 24576 MiB, 137.45 W, 59
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 07-27 15:52:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 1.65 tokens/s, Drafted throughput: 1.65 tokens/s, Accepted: 33 tokens, Drafted: 33 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 07-27 15:52:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 2.40 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 24 tokens, Drafted: 22 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 109.1%
(APIServer pid=1) INFO 07-27 15:52:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 2.10 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 21 tokens, Drafted: 20 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 105.0%
Raw
int8-pth-bench.log
[autodetect] served model='qwen3.6-27b' (from http://localhost:8014/v1/models; set MODEL= to override)
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
warm-1 wall= 14.50s ttft= 3813ms toks=1000 wall_TPS= 68.97 decode_TPS= 93.57
warm-2 wall= 9.24s ttft= 141ms toks=1000 wall_TPS=108.26 decode_TPS=109.93
warm-3 wall= 9.30s ttft= 158ms toks= 967 wall_TPS=103.99 decode_TPS=105.79
=== measured (5) ===
run-1 wall= 9.89s ttft= 155ms toks= 986 wall_TPS= 99.68 decode_TPS=101.26
run-2 wall= 9.62s ttft= 160ms toks=1000 wall_TPS=103.98 decode_TPS=105.73
run-3 wall= 9.70s ttft= 136ms toks=1000 wall_TPS=103.07 decode_TPS=104.53
run-4 wall= 9.44s ttft= 159ms toks= 994 wall_TPS=105.26 decode_TPS=107.05
run-5 wall= 9.64s ttft= 157ms toks=1000 wall_TPS=103.74 decode_TPS=105.46
=== summary [narrative] (n=5) ===
wall_TPS mean= 103.14 std= 2.09 CV= 2.0% min=99.68 max=105.26
decode_TPS mean= 104.81 std= 2.18 CV= 2.1% min=101.26 max=107.05
TTFT mean= 153ms std= 10ms min=136ms max=160ms
PP tok/s mean= 3.00 std= 1.12 CV=37.3% min=2.50 max=5.00
========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
warm-1 wall= 5.40s ttft= 140ms toks= 727 wall_TPS=134.66 decode_TPS=138.24
warm-2 wall= 5.84s ttft= 153ms toks= 800 wall_TPS=137.01 decode_TPS=140.70
warm-3 wall= 5.56s ttft= 159ms toks= 761 wall_TPS=136.93 decode_TPS=140.96
=== measured (5) ===
run-1 wall= 5.90s ttft= 155ms toks= 759 wall_TPS=128.65 decode_TPS=132.11
run-2 wall= 3.42s ttft= 153ms toks= 467 wall_TPS=136.54 decode_TPS=142.91
run-3 wall= 4.62s ttft= 153ms toks= 632 wall_TPS=136.73 decode_TPS=141.42
run-4 wall= 5.34s ttft= 158ms toks= 705 wall_TPS=132.14 decode_TPS=136.18
run-5 wall= 5.94s ttft= 162ms toks= 798 wall_TPS=134.24 decode_TPS=138.00
=== summary [code] (n=5) ===
wall_TPS mean= 133.66 std= 3.37 CV= 2.5% min=128.65 max=136.73
decode_TPS mean= 138.13 std= 4.29 CV= 3.1% min=132.11 max=142.91
TTFT mean= 156ms std= 4ms min=153ms max=162ms
PP tok/s mean= 4.00 std= 2.24 CV=55.9% min=2.50 max=7.50
========== PREFILL-10K (target=10000 prompt tokens, max_tokens=16, cache-busted: fresh haystack per run) ==========
=== warmups (1) ===
warm-1 wall= 13.15s ttft= 13110ms prompt_toks= 13363 PP_tok/s=1019.32
[calibrated: 13363 tok at request=10000 → measured runs request 7483]
=== measured (3) ===
run-1 wall= 9.13s ttft= 9096ms prompt_toks= 9646 PP_tok/s=1060.49
run-2 wall= 9.12s ttft= 9084ms prompt_toks= 9646 PP_tok/s=1061.92
run-3 wall= 9.12s ttft= 9084ms prompt_toks= 9646 PP_tok/s=1061.85
=== summary [prefill-10k] (n=3) ===
prefill tok/s mean=1061.42 std= 0.81 CV= 0.1% min=1060.49 max=1061.92
TTFT mean= 9088ms std= 7ms min=9084ms max=9096ms
PP tok/s (engine log, windowed) mean=1088.40 std=214.51 CV=19.7% min=964.50 max=1336.10
========== PREFILL-90K (target=90000 prompt tokens, max_tokens=16, cache-busted: fresh haystack per run) ==========
=== warmups (1) ===
warm-1 wall=160.79s ttft=160592ms prompt_toks=111443 PP_tok/s= 693.95
[calibrated: 111443 tok at request=90000 → measured runs request 72682]
=== measured (3) ===
run-1 wall=132.64s ttft=132462ms prompt_toks= 96943 PP_tok/s= 731.85
run-2 wall=126.26s ttft=126085ms prompt_toks= 93481 PP_tok/s= 741.41
run-3 wall=132.63s ttft=132452ms prompt_toks= 96943 PP_tok/s= 731.91
=== summary [prefill-90k] (n=3) ===
prefill tok/s mean= 735.06 std= 5.50 CV= 0.7% min=731.85 max=741.41
TTFT mean=130333ms std= 3679ms min=126085ms max=132462ms
PP tok/s (engine log, windowed) mean=10062.10 std=953.22 CV= 9.5% min=9347.90 max=11144.50
=== GPU state ===
0, 100 %, 22194 MiB, 24576 MiB, 169.68 W, 59
1, 100 %, 22202 MiB, 24576 MiB, 173.42 W, 62
2, 100 %, 22202 MiB, 24576 MiB, 181.34 W, 71
3, 100 %, 22162 MiB, 24576 MiB, 180.06 W, 61
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 07-27 15:42:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.02 tokens/s, Drafted throughput: 0.02 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 07-27 15:44:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.02 tokens/s, Drafted throughput: 0.02 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 07-27 15:46:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.02 tokens/s, Drafted throughput: 0.02 tokens/s, Accepted: 3 tokens, Drafted: 3 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
Raw
int8-pth-resolved.log
--- resolved evidence ---
[nvlink] 4 GPUs — no NVLink, but nvidia-smi reports P2P=OK (patched driver / P2P-capable layout) — auto-enabling PCIe P2P (NCCL_P2P_LEVEL=PHB, custom all-reduce engine-gated (vLLM disables its custom kernel at >2 PCIe-only GPUs — P2P runs via NCCL; #786))
[nvlink] P2P ENABLED — NCCL_P2P_LEVEL=PHB, custom all-reduce engine-gated (vLLM disables its custom kernel at >2 PCIe-only GPUs — P2P runs via NCCL; #786), expandable_segments stripped (PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512)
(APIServer pid=1) INFO 07-27 15:30:55 [api_utils.py:273] non-default args: {'model_tag': '/root/.cache/huggingface/qwen3.6-27b-autoround-int4', 'chat_template': '/etc/qwen-froggeric-chat-template.jinja', 'default_chat_template_kwargs': {'enable_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '/root/.cache/huggingface/qwen3.6-27b-autoround-int4', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 262144, 'quantization': 'auto_round', 'served_model_name': ['qwen3.6-27b', 'qwen3.6-27b-autoround'], 'override_generation_config': {'temperature': 0.6, 'top_p': 0.95, 'top_k': 20, 'min_p': 0.0, 'repetition_penalty': 1.0}, 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 4, 'kv_cache_dtype': 'int8_per_token_head', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 2, 'long_prefill_token_threshold': 2048, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(EngineCore pid=205) INFO 07-27 15:31:51 [core.py:114] Initializing a V1 LLM engine (v0.25.1) with config: model='/root/.cache/huggingface/qwen3.6-27b-autoround-int4', speculative_config=SpeculativeConfig(method='mtp', model='/root/.cache/huggingface/qwen3.6-27b-autoround-int4', num_spec_tokens=3), tokenizer='/root/.cache/huggingface/qwen3.6-27b-autoround-int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=inc, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=int8_per_token_head, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=qwen3.6-27b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
(Worker pid=216) WARNING 07-27 15:32:06 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=217) WARNING 07-27 15:32:06 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=218) WARNING 07-27 15:32:06 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=215) WARNING 07-27 15:32:06 [custom_all_reduce.py:151] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=215) INFO 07-27 15:32:06 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker_TP0 pid=215) INFO 07-27 15:32:18 [cuda.py:476] Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN'].
--- container env ---
NVIDIA_VISIBLE_DEVICES=all
NVLINK_MODE=auto
--- power caps before ---
0, 230.00 W
1, 230.00 W
2, 230.00 W
3, 230.00 W
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment