Generated: 2026-06-19 16:15:21 UTC
Redacted output (paths, host, user, tokens). Re-run with --no-redact for full data.
- OS: Ubuntu 26.04 LTS
- Kernel: 7.0.0-22-generic
- Environment: bare metal
- Locale: en_US.UTF-8
- Timezone: UTC
- Uptime: up 6 days, 17 hours, 53 minutes
- CPU: AMD Ryzen Threadripper 2950X 16-Core Processor (32 threads)
- RAM: 60Gi total, 40Gi available
- Swap: 8.0Gi
- <MODEL_DIR>: 235G available, ext4 filesystem
- ~/inference/serving/club-3090/models-cache: 178G available, ext4 filesystem
- /huggingface: 235G available, ext4 filesystem
- /var/lib/docker: 178G available, ext4 filesystem
- GPU 0: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 595.71.05 | VBIOS 94.02.42.C0.05 | persistence=Enabled
- Power: limit=230.00 W (default=420.00 W, max=450.00 W) | current_draw=34.27 W ⚠ user-capped below default
- PCIe: x4 lanes negotiated (GPU max x16, Gen up to 3) | bus 00000000:08:00.0 ⚠ slot is narrower than GPU capability — affects load + all-reduce bandwidth
- GPU 1: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 595.71.05 | VBIOS 94.02.42.C0.05 | persistence=Enabled
- Power: limit=230.00 W (default=420.00 W, max=450.00 W) | current_draw=40.80 W ⚠ user-capped below default
- PCIe: x16 lanes negotiated (GPU max x16, Gen up to 3) | bus 00000000:09:00.0
- GPU 2: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 595.71.05 | VBIOS 94.02.59.00.F2 | persistence=Enabled
- Power: limit=230.00 W (default=350.00 W, max=350.00 W) | current_draw=37.13 W ⚠ user-capped below default
- PCIe: x8 lanes negotiated (GPU max x16, Gen up to 3) | bus 00000000:41:00.0 ⚠ slot is narrower than GPU capability — affects load + all-reduce bandwidth
- GPU 3: NVIDIA GeForce RTX 3090 | 24576 MiB | driver 595.71.05 | VBIOS 94.02.42.40.B7 | persistence=Enabled
- Power: limit=230.00 W (default=370.00 W, max=390.00 W) | current_draw=35.89 W ⚠ user-capped below default
- PCIe: x16 lanes negotiated (GPU max x16, Gen up to 3) | bus 00000000:42:00.0
- CUDA Runtime (per driver): 13.2
- ECC mode: [N/A] (3090s don't have ECC; expect N/A)
No NVLink detected (PCIe-only)
PCIe / GPU topology matrix
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB NODE NODE 0-31 0 N/A
GPU1 PHB X NODE NODE 0-31 0 N/A
GPU2 NODE NODE X PHB 0-31 0 N/A
GPU3 NODE NODE PHB X 0-31 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
lspci PCIe/P2P detail (LnkSta / ACS / topology)
# lspci -t (PCIe topology tree)
-+-[0000:00]-+-00.0
| +-00.2
| +-01.0
| +-01.1-[01-06]--+-00.0
| | +-00.1
| | \-00.2-[02-06]--+-02.0-[03]----00.0
| | +-03.0-[04]--
| | +-04.0-[05]--
| | \-09.0-[06]----00.0
| +-01.2-[07]----00.0
| +-01.3-[08]--+-00.0
| | \-00.1
| +-02.0
| +-03.0
| +-03.1-[09]--+-00.0
| | \-00.1
| +-04.0
| +-07.0
| +-07.1-[0a]--+-00.0
| | +-00.2
| | \-00.3
| +-08.0
| +-08.1-[0b]--+-00.0
| | \-00.2
| +-14.0
| +-14.3
| +-18.0
| +-18.1
| +-18.2
| +-18.3
| +-18.4
| +-18.5
| +-18.6
| +-18.7
| +-19.0
| +-19.1
| +-19.2
| +-19.3
| +-19.4
| +-19.5
| +-19.6
| \-19.7
\-[0000:40]-+-00.0
+-00.2
+-01.0
+-01.3-[41]--+-00.0
| \-00.1
+-02.0
+-03.0
+-03.1-[42]--+-00.0
| \-00.1
+-04.0
+-07.0
+-07.1-[43]--+-00.0
| +-00.2
| \-00.3
+-08.0
\-08.1-[44]--+-00.0
\-00.2
# lspci -vvv -s 0000:08:00.0 (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded)
# lspci -vvv -s 0000:00:01.3 (upstream bridge of 0000:08:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #2, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x4
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
# lspci -vvv -s 0000:09:00.0 (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
LnkSta: Speed 2.5GT/s (downgraded), Width x16
# lspci -vvv -s 0000:00:03.1 (upstream bridge of 0000:09:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x16
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
# lspci -vvv -s 0000:41:00.0 (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded)
# lspci -vvv -s 0000:40:01.3 (upstream bridge of 0000:41:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x8
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
# lspci -vvv -s 0000:42:00.0 (GPU function: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
LnkSta: Speed 2.5GT/s (downgraded), Width x16
# lspci -vvv -s 0000:40:03.1 (upstream bridge of 0000:42:00.0: LnkCap/LnkSta/ACSCap/ACSCtl)
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x16
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
# lspci -nnk | grep -A3 -i nvidia (driver binding + device IDs)
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
Subsystem: EVGA Corporation Device [3842:3982]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nova_core, nvidia_drm, nvidia
08:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
Subsystem: EVGA Corporation Device [3842:3982]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
Subsystem: EVGA Corporation Device [3842:3982]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nova_core, nvidia_drm, nvidia
09:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
Subsystem: EVGA Corporation Device [3842:3982]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
--
41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3881]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nova_core, nvidia_drm, nvidia
41:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3881]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
42:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:4043]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nova_core, nvidia_drm, nvidia
42:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:4043]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
Full nvidia-smi output
Fri Jun 19 16:15:23 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:08:00.0 Off | N/A |
| 0% 37C P8 34W / 230W | 22380MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:09:00.0 Off | N/A |
| 0% 38C P8 40W / 230W | 22380MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GeForce RTX 3090 On | 00000000:41:00.0 Off | N/A |
| 0% 41C P8 37W / 230W | 22380MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GeForce RTX 3090 On | 00000000:42:00.0 Off | N/A |
| 0% 41C P8 35W / 230W | 22380MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1149580 C VLLM::Worker_TP0 22370MiB |
| 1 N/A N/A 1149675 C VLLM::Worker_TP1 22370MiB |
| 2 N/A N/A 1149743 C VLLM::Worker_TP2 22370MiB |
| 3 N/A N/A 1149845 C VLLM::Worker_TP3 22370MiB |
+-----------------------------------------------------------------------------------------+
- $DISPLAY: unset (headless)
- Display processes running: none detected
- GPU 0 idle VRAM: 22380 MiB (held by running
vllm-qwen36-27b-multi4-max) - GPU 1 idle VRAM: 22380 MiB (held by running
vllm-qwen36-27b-multi4-max) - GPU 2 idle VRAM: 22380 MiB (held by running
vllm-qwen36-27b-multi4-max) - GPU 3 idle VRAM: 22380 MiB (held by running
vllm-qwen36-27b-multi4-max)
- Docker: 29.6.0
- docker compose (v2): 5.1.4
- NVIDIA Container Toolkit: 1.19.1
- club-3090:
v0.9.0-46-ge81cd58(branch:master, SHAe81cd58) - GENESIS_PIN default:
7b9fd319(per scripts/setup.sh) - Cached vLLM images:
- tag
nightlydigestsha256:779772129ce2cbd64329e370aed9dd8f27ffea9b8eb69038e9a2d5ee5791202d(6 days ago) - tag
v0.22.0digestsha256:0fec7ec5f3e6bc168e54899935fb0557da908a4832a1dbc88e2debcf2f889416(3 weeks ago)
- tag
- Profile schema version: 1
- Profile counts: 9 hardware, 8 models, 5 workloads, 13 engines, 11 drafters
- Compose registry: 53 entries
- Canonical scenarios: 9
- Calibration:
- gemma-4-12b: 1 rows
- gemma-4-26b-a4b: 0 rows
- gemma-4-31b: 3 rows
- qwen3.6-27b: 2 rows
- qwen3.6-35b-a3b: 1 rows
- Active estate: none (
~/.club3090/estate.ymlnot found)
- Scoped to the running model
qwen3.6-27b— pass--full-calibrationfor all calibrated models. - Overall: 7/7 (100%)
- No FAIL rows. kv-calc projections should agree with measured VRAM within the ±1.5 GB error band.
Full kv-calc --calibration output
========================================================================================
Calibration — predicted per-card VRAM vs measured BENCHMARKS rows
========================================================================================
Predicted = weights + activation + overhead + drafter + (KV capped at available).
Budget = mem_util × VRAM. Measured = nvidia-smi peak during bench (target ≈ budget).
Verdict ✓ iff PASS/TIGHT and measured < VRAM (boot OK).
== qwen3.6-27b ==
compose predicted budget measured verdict
───────────────────────── ───────── ──────── ───────── ───────
dual 19.91 GB 22.80 GB 23.60 GB PASS ✓
minimal@64K 21.60 GB 21.60 GB 22.40 GB TIGHT ✓
Verdict accuracy: 2/2 (100%)
Overall: 7/7 (100%)
- Name:
vllm-qwen36-27b-multi4-max - Engine:
vllm - Status: Up 9 minutes
- Ports: 0.0.0.0:8015->8000/tcp
- Image:
vllm/vllm-openai:v0.22.0 - Image digest:
sha256:0fec7ec5f3e6bc168e54899935fb0557da908a4832a1dbc88e2debcf2f889416 - Build tag (OCI version):
vllm/vllm-openai:v0.22.0 - Upstream commit (OCI revision):
799c3afa5d5b17b676d04e0b58a5628943bb4003 - Upstream source: https://github.com/vllm-project/vllm
- PyTorch:
torch=2.11.0+cu130 torch_cuda_build=13.0 cudnn=91900 - vLLM:
0.22.0 - nvidia-smi inside container:
0, NVIDIA GeForce RTX 3090, 595.71.05 1, NVIDIA GeForce RTX 3090, 595.71.05 2, NVIDIA GeForce RTX 3090, 595.71.05 3, NVIDIA GeForce RTX 3090, 595.71.05
KV pool sizing:
(Worker_TP0 pid=169) INFO 06-19 16:11:05 [gpu_worker.py:466] Available KV cache memory: 12.89 GiB
(EngineCore pid=159) INFO 06-19 16:11:05 [kv_cache_utils.py:1733] GPU KV cache size: 1,430,929 tokens
(EngineCore pid=159) INFO 06-19 16:11:05 [kv_cache_utils.py:1734] Maximum concurrency for 262,144 tokens per request: 5.46x
Engine config (CLI flags + engine init):
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:278] non-default args: {'model_tag': '~/.cache/huggingface/qwen3.6-27b-fp8', 'chat_template': '/etc/qwen-froggeric-chat-template.jinja', 'default_chat_template_kwargs': {'enable_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '~/.cache/huggingface/qwen3.6-27b-fp8', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 262144, 'quantization': 'fp8', 'served_model_name': ['qwen3.6-27b-fp8'], 'override_generation_config': {'temperature': 0.6, 'top_p': 0.95, 'top_k': 20, 'min_p': 0.0, 'repetition_penalty': 1.0}, 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 4, 'disable_custom_all_reduce': True, 'kv_cache_dtype': 'int8_per_token_head', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 2, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(EngineCore pid=159) INFO 06-19 16:06:37 [core.py:112] Initializing a V1 LLM engine (v0.22.0) with config: model='~/.cache/huggingface/qwen3.6-27b-fp8', speculative_config=SpeculativeConfig(method='mtp', model='~/.cache/huggingface/qwen3.6-27b-fp8', num_spec_tokens=3), tokenizer='~/.cache/huggingface/qwen3.6-27b-fp8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=int8_per_token_head, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.6-27b-fp8, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
Recent warnings/errors (last 5):
WARNING 06-19 16:05:45 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option <USER> be removed in a future version.
First 200 lines of docker logs
[nvlink] 4 GPUs detected — no NVLink found, using PCIe mode
[nvlink] P2P DISABLED — NCCL_P2P_DISABLE=1, custom all-reduce OFF, expandable_segments ON
WARNING 06-19 16:05:45 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option <USER> be removed in a future version.
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:344]
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:344] █ █ █▄ ▄█
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:344] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.22.0
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:344] █▄█▀ █ █ █ █ model ~/.cache/huggingface/qwen3.6-27b-fp8
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:344] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:344]
(APIServer pid=1) INFO 06-19 16:05:45 [utils.py:278] non-default args: {'model_tag': '~/.cache/huggingface/qwen3.6-27b-fp8', 'chat_template': '/etc/qwen-froggeric-chat-template.jinja', 'default_chat_template_kwargs': {'enable_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '~/.cache/huggingface/qwen3.6-27b-fp8', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 262144, 'quantization': 'fp8', 'served_model_name': ['qwen3.6-27b-fp8'], 'override_generation_config': {'temperature': 0.6, 'top_p': 0.95, 'top_k': 20, 'min_p': 0.0, 'repetition_penalty': 1.0}, 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 4, 'disable_custom_all_reduce': True, 'kv_cache_dtype': 'int8_per_token_head', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 2, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(APIServer pid=1) WARNING 06-19 16:05:45 [envs.py:2057] Unknown vLLM environment variable detected: VLLM_BUILD_URL
(APIServer pid=1) WARNING 06-19 16:05:45 [envs.py:2057] Unknown vLLM environment variable detected: VLLM_IMAGE_TAG
(APIServer pid=1) WARNING 06-19 16:05:45 [envs.py:2057] Unknown vLLM environment variable detected: VLLM_BUILD_PIPELINE
(APIServer pid=1) WARNING 06-19 16:05:45 [envs.py:2057] Unknown vLLM environment variable detected: VLLM_BUILD_COMMIT
(APIServer pid=1) INFO 06-19 16:06:04 [model.py:617] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1) INFO 06-19 16:06:04 [model.py:1752] Using max model len 262144
(APIServer pid=1) INFO 06-19 16:06:04 [cache.py:254] Using int8_per_token_head data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Dynamic per-token-head scales <USER> be computed at runtime.
(APIServer pid=1) INFO 06-19 16:06:15 [model.py:617] Resolved architecture: Qwen3_5MTP
(APIServer pid=1) INFO 06-19 16:06:15 [model.py:1752] Using max model len 262144
(APIServer pid=1) WARNING 06-19 16:06:15 [speculative.py:709] Enabling num_speculative_tokens > 1 <USER> run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=1) INFO 06-19 16:06:15 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1) WARNING 06-19 16:06:15 [config.py:355] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 06-19 16:06:15 [config.py:375] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=1) INFO 06-19 16:06:15 [vllm.py:977] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 06-19 16:06:15 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) INFO 06-19 16:06:16 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and <USER> be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=159) INFO 06-19 16:06:37 [core.py:112] Initializing a V1 LLM engine (v0.22.0) with config: model='~/.cache/huggingface/qwen3.6-27b-fp8', speculative_config=SpeculativeConfig(method='mtp', model='~/.cache/huggingface/qwen3.6-27b-fp8', num_spec_tokens=3), tokenizer='~/.cache/huggingface/qwen3.6-27b-fp8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=int8_per_token_head, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.6-27b-fp8, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=159) INFO 06-19 16:06:37 [multiproc_executor.py:139] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.19.0.2 (local), world_size=4, local_world_size=4
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(Worker pid=169) INFO 06-19 16:06:48 [parallel_state.py:1422] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41253 backend=nccl
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(Worker pid=174) INFO 06-19 16:06:54 [parallel_state.py:1422] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41253 backend=nccl
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(Worker pid=183) INFO 06-19 16:07:01 [parallel_state.py:1422] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:41253 backend=nccl
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(Worker pid=199) INFO 06-19 16:07:08 [parallel_state.py:1422] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:41253 backend=nccl
(Worker pid=169) INFO 06-19 16:07:08 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=169) WARNING 06-19 16:07:08 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=169) INFO 06-19 16:07:08 [cuda_communicator.py:232] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=199) WARNING 06-19 16:07:08 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=183) WARNING 06-19 16:07:08 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=174) WARNING 06-19 16:07:08 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=169) INFO 06-19 16:07:08 [parallel_state.py:1735] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(Worker pid=169) INFO 06-19 16:07:09 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(Worker pid=199) WARNING 06-19 16:07:09 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=169) WARNING 06-19 16:07:09 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=174) WARNING 06-19 16:07:09 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=183) WARNING 06-19 16:07:09 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=174) [transformers] The `use_fast` parameter is deprecated and <USER> be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker pid=169) [transformers] The `use_fast` parameter is deprecated and <USER> be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker pid=199) [transformers] The `use_fast` parameter is deprecated and <USER> be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker pid=183) [transformers] The `use_fast` parameter is deprecated and <USER> be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker_TP0 pid=169) INFO 06-19 16:07:16 [gpu_model_runner.py:5037] Starting to load model ~/.cache/huggingface/qwen3.6-27b-fp8...
(Worker_TP0 pid=169) INFO 06-19 16:07:16 [cuda.py:433] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0 pid=169) INFO 06-19 16:07:16 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=169) INFO 06-19 16:07:16 [__init__.py:526] Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod
(Worker_TP0 pid=169) INFO 06-19 16:07:16 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=None).
(Worker_TP0 pid=169) INFO 06-19 16:07:16 [cuda.py:378] Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN'].
(Worker_TP0 pid=169) INFO 06-19 16:07:17 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 28.75 GiB. Available RAM: 46.47 GiB.
(Worker_TP0 pid=169) INFO 06-19 16:07:17 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 0% Completed | 0/66 [00:00<?, ?it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 3% Completed | 2/66 [00:00<00:03, 19.98it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 8% Completed | 5/66 [00:00<00:03, 17.17it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 11% Completed | 7/66 [00:00<00:03, 18.16it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 15% Completed | 10/66 [00:00<00:02, 19.65it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 18% Completed | 12/66 [00:00<00:02, 19.58it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 21% Completed | 14/66 [00:00<00:02, 19.21it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 24% Completed | 16/66 [00:00<00:02, 19.35it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 27% Completed | 18/66 [00:00<00:02, 19.30it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 30% Completed | 20/66 [00:01<00:02, 19.19it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 33% Completed | 22/66 [00:01<00:02, 19.30it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 36% Completed | 24/66 [00:01<00:02, 19.16it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 39% Completed | 26/66 [00:01<00:02, 19.30it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 44% Completed | 29/66 [00:01<00:01, 19.64it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 47% Completed | 31/66 [00:01<00:01, 19.01it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 50% Completed | 33/66 [00:01<00:01, 17.61it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 53% Completed | 35/66 [00:01<00:01, 16.88it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 56% Completed | 37/66 [00:02<00:01, 15.75it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 59% Completed | 39/66 [00:02<00:01, 13.80it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 62% Completed | 41/66 [00:02<00:02, 12.12it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 65% Completed | 43/66 [00:02<00:02, 9.65it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 68% Completed | 45/66 [00:03<00:02, 8.11it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 70% Completed | 46/66 [00:03<00:02, 7.57it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 71% Completed | 47/66 [00:03<00:02, 7.71it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 73% Completed | 48/66 [00:03<00:02, 7.14it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 74% Completed | 49/66 [00:03<00:02, 6.63it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 76% Completed | 50/66 [00:03<00:02, 6.24it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 77% Completed | 51/66 [00:04<00:02, 5.96it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 79% Completed | 52/66 [00:04<00:02, 6.12it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 80% Completed | 53/66 [00:04<00:02, 5.90it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 82% Completed | 54/66 [00:04<00:02, 5.94it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 83% Completed | 55/66 [00:04<00:01, 6.06it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 85% Completed | 56/66 [00:04<00:01, 6.16it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 86% Completed | 57/66 [00:05<00:01, 5.95it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 88% Completed | 58/66 [00:05<00:01, 5.82it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 89% Completed | 59/66 [00:05<00:01, 5.72it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 91% Completed | 60/66 [00:05<00:01, 5.99it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 92% Completed | 61/66 [00:05<00:00, 6.40it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 94% Completed | 62/66 [00:05<00:00, 6.22it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 95% Completed | 63/66 [00:06<00:00, 6.09it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 97% Completed | 64/66 [00:06<00:00, 6.17it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 100% Completed | 66/66 [00:08<00:00, 1.85it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 100% Completed | 66/66 [00:08<00:00, 8.03it/s]
(Worker_TP0 pid=169)
(Worker_TP0 pid=169) INFO 06-19 16:07:25 [default_loader.py:397] Loading weights took 8.24 seconds
(Worker_TP0 pid=169) WARNING 06-19 16:07:25 [marlin_utils_fp8.py:97] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression <USER> be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(Worker_TP2 pid=183) INFO 06-19 16:07:26 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP3 pid=199) INFO 06-19 16:07:26 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP1 pid=174) INFO 06-19 16:07:26 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=169) INFO 06-19 16:07:26 [gpu_model_runner.py:5061] Loading drafter model...
(Worker_TP0 pid=169) INFO 06-19 16:07:26 [vllm.py:977] Asynchronous scheduling is enabled.
(Worker_TP0 pid=169) INFO 06-19 16:07:26 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=169) INFO 06-19 16:07:26 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(Worker_TP0 pid=169) INFO 06-19 16:07:26 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 28.75 GiB. Available RAM: 46.35 GiB.
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 0% Completed | 0/66 [00:00<?, ?it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 98% Completed | 65/66 [00:00<00:00, 300.25it/s]
(Worker_TP0 pid=169)
Loading safetensors checkpoint shards: 100% Completed | 66/66 [00:00<00:00, 105.04it/s]
(Worker_TP0 pid=169)
(Worker_TP0 pid=169) INFO 06-19 16:07:26 [default_loader.py:397] Loading weights took 0.64 seconds
(Worker_TP3 pid=199) INFO 06-19 16:07:28 [llm_base_proposer.py:1327] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP3 pid=199) INFO 06-19 16:07:28 [llm_base_proposer.py:1383] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP0 pid=169) INFO 06-19 16:07:28 [llm_base_proposer.py:1327] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP0 pid=169) INFO 06-19 16:07:28 [llm_base_proposer.py:1383] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP2 pid=183) INFO 06-19 16:07:28 [llm_base_proposer.py:1327] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP2 pid=183) INFO 06-19 16:07:28 [llm_base_proposer.py:1383] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP1 pid=174) INFO 06-19 16:07:28 [llm_base_proposer.py:1327] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP1 pid=174) INFO 06-19 16:07:28 [llm_base_proposer.py:1383] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP3 pid=199) INFO 06-19 16:07:29 [interface.py:649] Setting attention block size to 1584 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP3 pid=199) INFO 06-19 16:07:29 [interface.py:673] Padding mamba page size by 0.80% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=169) INFO 06-19 16:07:29 [gpu_model_runner.py:5132] Model loading took 7.41 GiB memory and 11.897761 seconds
(Worker_TP2 pid=183) INFO 06-19 16:07:29 [interface.py:649] Setting attention block size to 1584 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP2 pid=183) INFO 06-19 16:07:29 [interface.py:673] Padding mamba page size by 0.80% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=169) INFO 06-19 16:07:29 [interface.py:649] Setting attention block size to 1584 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP0 pid=169) INFO 06-19 16:07:29 [interface.py:673] Padding mamba page size by 0.80% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP1 pid=174) INFO 06-19 16:07:29 [interface.py:649] Setting attention block size to 1584 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP1 pid=174) INFO 06-19 16:07:29 [interface.py:673] Padding mamba page size by 0.80% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=169) INFO 06-19 16:07:30 [gpu_model_runner.py:6136] Encoder cache <USER> be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP0 pid=169) INFO 06-19 16:07:48 [backends.py:1089] Using cache directory: ~/.cache/vllm/torch_compile_cache/7ec3e83971/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=169) INFO 06-19 16:07:48 [backends.py:1148] Dynamo bytecode transform time: 15.56 s
(Worker_TP0 pid=169) INFO 06-19 16:07:53 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(EngineCore pid=159) INFO 06-19 16:08:30 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=169) INFO 06-19 16:08:32 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 41.59 s
(Worker_TP0 pid=169) INFO 06-19 16:09:06 [decorators.py:708] saved AOT compiled function to ~/.cache/vllm/torch_compile_cache/torch_aot_compile/8980bb5cdaf360b9ce9d6cd96064f0bb18fcf0958a875a9be2ff7ed84f7bd357/rank_0_0/model
(Worker_TP0 pid=169) INFO 06-19 16:09:06 [monitor.py:53] torch.compile took 93.03 s in total
(EngineCore pid=159) INFO 06-19 16:09:30 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=159) INFO 06-19 16:10:30 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=169) INFO 06-19 16:10:39 [monitor.py:81] Initial profiling/warmup run took 92.90 s
(Worker_TP0 pid=169) INFO 06-19 16:10:40 [backends.py:1089] Using cache directory: ~/.cache/vllm/torch_compile_cache/7ec3e83971/rank_0_0/eagle_head for vLLM's torch.compile
(Worker_TP0 pid=169) INFO 06-19 16:10:40 [backends.py:1148] Dynamo bytecode transform time: 0.90 s
(Worker_TP0 pid=169) INFO 06-19 16:10:51 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 10.58 s
(Worker_TP0 pid=169) INFO 06-19 16:10:52 [decorators.py:708] saved AOT compiled function to ~/.cache/vllm/torch_compile_cache/torch_aot_compile/89b261797c2530c780e9e6d8028802f149f6f955519ebae78e01b1cac9ed5b88/rank_0_0/model
(Worker_TP0 pid=169) INFO 06-19 16:10:52 [monitor.py:53] torch.compile took 13.23 s in total
(Worker_TP0 pid=169) INFO 06-19 16:10:53 [monitor.py:81] Initial profiling/warmup run took 1.18 s
(Worker_TP1 pid=174) WARNING 06-19 16:10:56 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker_TP0 pid=169) WARNING 06-19 16:10:56 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker_TP2 pid=183) WARNING 06-19 16:10:56 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker_TP1 pid=174) INFO 06-19 16:10:56 [gpu_model_runner.py:6279] Profiling CUDA graph memory: PIECEWISE=3 (largest=16), FULL=2 (largest=8)
(Worker_TP2 pid=183) INFO 06-19 16:10:56 [gpu_model_runner.py:6279] Profiling CUDA graph memory: PIECEWISE=3 (largest=16), FULL=2 (largest=8)
(Worker_TP0 pid=169) INFO 06-19 16:10:56 [gpu_model_runner.py:6279] Profiling CUDA graph memory: PIECEWISE=3 (largest=16), FULL=2 (largest=8)
(Worker_TP3 pid=199) WARNING 06-19 16:10:56 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(Worker_TP3 pid=199) INFO 06-19 16:10:56 [gpu_model_runner.py:6279] Profiling CUDA graph memory: PIECEWISE=3 (largest=16), FULL=2 (largest=8)
(Worker_TP0 pid=169) INFO 06-19 16:11:04 [gpu_model_runner.py:6365] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP3 pid=199) INFO 06-19 16:11:04 [gpu_model_runner.py:6365] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP2 pid=183) INFO 06-19 16:11:04 [gpu_model_runner.py:6365] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP1 pid=174) INFO 06-19 16:11:04 [gpu_model_runner.py:6365] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP3 pid=199) INFO 06-19 16:11:05 [gpu_worker.py:481] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9155 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9245. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP2 pid=183) INFO 06-19 16:11:05 [gpu_worker.py:481] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9155 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9245. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=169) INFO 06-19 16:11:05 [gpu_worker.py:466] Available KV cache memory: 12.89 GiB
(Worker_TP0 pid=169) INFO 06-19 16:11:05 [gpu_worker.py:481] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9155 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9245. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=174) INFO 06-19 16:11:05 [gpu_worker.py:481] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9155 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9245. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=159) WARNING 06-19 16:11:05 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 6.25% KV cache memory
(EngineCore pid=159) INFO 06-19 16:11:05 [kv_cache_utils.py:1733] GPU KV cache size: 1,430,929 tokens
(EngineCore pid=159) INFO 06-19 16:11:05 [kv_cache_utils.py:1734] Maximum concurrency for 262,144 tokens per request: 5.46x
(Worker_TP0 pid=169)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/3 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 33%|███▎ | 1/3 [00:00<00:00, 6.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 67%|██████▋ | 2/3 [00:00<00:00, 6.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 3/3 [00:00<00:00, 6.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 3/3 [00:00<00:00, 6.88it/s]
(Worker_TP0 pid=169)
Capturing CUDA graphs (decode, FULL): 0%| | 0/2 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 50%|█████ | 1/2 [00:00<00:00, 1.17it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 2/2 [00:01<00:00, 1.57it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 2/2 [00:01<00:00, 1.49it/s]
(Worker_TP3 pid=199) INFO 06-19 16:11:09 [gpu_worker.py:619] CUDA graph pool memory: 0.06 GiB (actual), 0.11 GiB (estimated), difference: 0.04 GiB (74.2%).
(Worker_TP2 pid=183) INFO 06-19 16:11:09 [gpu_worker.py:619] CUDA graph pool memory: 0.06 GiB (actual), 0.11 GiB (estimated), difference: 0.04 GiB (74.2%).
(Worker_TP1 pid=174) INFO 06-19 16:11:09 [gpu_worker.py:619] CUDA graph pool memory: 0.06 GiB (actual), 0.11 GiB (estimated), difference: 0.04 GiB (74.2%).
(Worker_TP0 pid=169) INFO 06-19 16:11:09 [gpu_model_runner.py:6456] Graph capturing finished in 3 secs, took 0.06 GiB
(Worker_TP0 pid=169) INFO 06-19 16:11:09 [gpu_worker.py:619] CUDA graph pool memory: 0.06 GiB (actual), 0.11 GiB (estimated), difference: 0.04 GiB (74.2%).
(Worker_TP0 pid=169) INFO 06-19 16:11:09 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference <USER> be logged as warnings.
(Worker_TP1 pid=174) INFO 06-19 16:11:09 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference <USER> be logged as warnings.
(Worker_TP3 pid=199) INFO 06-19 16:11:09 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference <USER> be logged as warnings.
(Worker_TP2 pid=183) INFO 06-19 16:11:09 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference <USER> be logged as warnings.
(EngineCore pid=159) INFO 06-19 16:11:09 [core.py:302] init engine (profile, create kv cache, warmup model) took 219.53 s (compilation: 106.97 s)
(EngineCore pid=159) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=159) [transformers] The `use_fast` parameter is deprecated and <USER> be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=159) INFO 06-19 16:11:17 [vllm.py:977] Asynchronous scheduling is enabled.
(EngineCore pid=159) INFO 06-19 16:11:17 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=159) INFO 06-19 16:11:17 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1) INFO 06-19 16:11:17 [api_server.py:592] Supported tasks: ['generate']
(APIServer pid=1) INFO 06-19 16:11:18 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) WARNING 06-19 16:11:18 [model.py:1509] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95, 'min_p': 0.0}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 06-19 16:11:18 [hf.py:488] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) INFO 06-19 16:11:40 [base.py:224] Multi-modal warmup completed in 21.831s
(APIServer pid=1) INFO 06-19 16:11:42 [base.py:224] Readonly multi-modal warmup completed in 2.607s
(APIServer pid=1) INFO 06-19 16:11:42 [api_server.py:596] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1) INFO 06-19 16:11:42 [launcher.py:37] Available routes are:
No recently-exited vLLM or llama.cpp containers found.
verify-full output
[autodetect] using running container=vllm-qwen36-27b-multi4-max url=http://localhost:8015 (skip: PREFLIGHT_NO_AUTODETECT=1)
[autodetect] served model='qwen3.6-27b-fp8' (from http://localhost:8015/v1/models; set MODEL= to override)
Running FULL functional test against http://localhost:8015
model=qwen3.6-27b-fp8 container=vllm-qwen36-27b-multi4-max engine=vllm
[1/9] Server reachable on /v1/models ...
✓ server is serving
[2/9] Genesis patches applied ...
⊘ no Genesis marker in logs (container restarted, or Genesis not loaded) (skipped)
[warmup] priming engine (cold cudagraph/JIT, up to 180s, not scored) ...
[warmup] engine warm
[3/9] Basic completion — capital of France ...
✓ reply contains 'Paris'
[4/9] Tool calling ...
✓ tool_calls[] populated with get_weather
[5/9] Streaming (SSE) ...
✓ streamed 8 chunks, 64 chars: Staring at the screen, One missing semicolon, Code finally runs. ...
[6/9] Streaming tool-calls (thinking-on) ...
✓ streamed delta.tool_calls (get_weather) + finish_reason=tool_calls, no <tool_call> leak
[7/9] Thinking / reasoning mode ...
✓ reasoning 594 chars, content 3 chars (finish=stop)
reasoning: Here's a thinking process: 1. **Analyze User Input:** -...
content: 4...
[8/9] Output quality / cascade detection (2K-token completion) ...
✓ output OK — 9887 chars, variety=0.695, max_line_repeat=0, finish=stop
[9/9] MTP acceptance length threshold ...
✓ MTP acceptance length = 2.70 (>=2.0 — spec-decode contributing)
All checks passed. Stack is ready for full-functionality use.
verify-stress output (7 boundary checks incl. Cliff 2 needle recall)
[autodetect] using running container=vllm-qwen36-27b-multi4-max url=http://localhost:8015 (skip: PREFLIGHT_NO_AUTODETECT=1)
[autodetect] served model='qwen3.6-27b-fp8' (from http://localhost:8015/v1/models; set MODEL= to override)
Running STRESS / boundary test against http://localhost:8015
model=qwen3.6-27b-fp8 container=vllm-qwen36-27b-multi4-max engine=vllm
This script does the heavy stuff (longctx needle ladder + ~25K-token tool prefill).
For the fast functional smoke (~2 min), use verify-full.sh instead.
[1/8] Long-context needle small rungs (10K / 30K) ...
✓ 9821 tokens: recalled 'sapphire axolotl 89' (got: sapphire axolotl 89 ) prefill=637.2 t/s (15s)
✓ 29320 tokens: recalled 'emerald chinchilla 73' (got: emerald chinchilla 73 ) prefill=648.4 t/s (45s)
✓ all long-ctx depths recalled secret correctly
[2/8] Tool response prefill OOM (~25K-token mock tool response) ...
✓ tool prefill OK — text response (725 chars, finish=stop)
[3/8] IDE-agent one-shot prompt (sys + tool schemas + user request) ...
✓ IDE-agent one-shot OK — 49 completion tokens (207 chars), finish=stop
[4/8] Multi-turn agent prompt (sys + tools + 4-turn history) ...
✓ multi-turn agent OK
[5/8] LCB-coding shape (LeetCode-style problem + structured plan) ...
✓ LCB-coding shape OK
[6/8] Reasoning-heavy (math problem + max_tokens=8192) ...
✓ reasoning-heavy OK — 4488 completion tokens
[7/8] Long-context needle large rungs (60K / 90K — Cliff 2 territory) ...
✓ 58569 tokens: recalled 'golden narwhal 84' (got: golden narwhal 84 ) prefill=674.3 t/s (87s)
✓ 91069 tokens: recalled 'sapphire otter 74' (got: sapphire otter 74 ) prefill=695.3 t/s (131s)
✓ all long-ctx depths recalled secret correctly
[8/8] Context ceiling ladder (staggered NIAH from ~95000 → ~0.92 × n_ctx) ...
n_ctx=262144 ladder: 95000 → 125000 → 155000 → 185000 → 215000 → 241172 (6 rungs)
calibrated: scale=100 → 6515 tokens (tok/scale_unit=65.15)
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
VRAM free (ladder start): 4167 MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ rung 1/6: target=95K actual=94K tok (36%) recalled 'golden narwhal 60' prefill=841.9 t/s (113s) VRAM_free=4167MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ rung 2/6: target=125K actual=124K tok (47%) recalled 'amber narwhal 37' prefill=671.8 t/s (186s) VRAM_free=4167MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ rung 3/6: target=155K actual=154K tok (59%) recalled 'amber capybara 96' prefill=649.1 t/s (238s) VRAM_free=4167MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ rung 4/6: target=185K actual=184K tok (70%) recalled 'crimson chinchilla 54' prefill=622.4 t/s (297s) VRAM_free=4167MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ rung 5/6: target=215K actual=214K tok (81%) recalled 'silver capybara 95' prefill=592.9 t/s (362s) VRAM_free=4167MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ rung 6/6: target=241K actual=240K tok (91%) recalled 'crimson capybara 75' prefill=574.3 t/s (419s) VRAM_free=4167MB
[vram] WARN: could not determine model GPU(s) on 4-GPU host — summing all (margin may be inflated)
✓ ceiling ladder: all 6 rungs passed — fillable to 240635 tok (91% of n_ctx=262144)
VRAM: 4167 → 4167 MB (Δ -0 MB across ladder, margin threshold=1024 MB)
All stress / boundary checks passed. KV-cache and prefill paths are sound for the deployed config.
soak-test stdout (5-session × 5-turn ramping conversation, ~25 min)
[soak] running soak test against http://localhost:8015 (model=qwen3.6-27b-fp8, container=vllm-qwen36-27b-multi4-max)
[soak] mode=continuous sessions=5 turns=5 max_growth=200MiB timeout=1800s
[soak] output=results/report-soak-20260619-164949
[soak] session 1/5
[soak] turn 1/5: status=200 wall=1291ms ttft=1128ms decode_tps=177.806 vram=92342MiB
[soak] turn 2/5: status=200 wall=8769ms ttft=8543ms decode_tps=123.979 vram=92342MiB
[soak] turn 3/5: status=200 wall=13760ms ttft=13278ms decode_tps=80.935 vram=92342MiB
[soak] turn 4/5: status=200 wall=16675ms ttft=15887ms decode_tps=57.066 vram=92342MiB
[soak] turn 5/5: status=200 wall=17079ms ttft=15648ms decode_tps=37.725 vram=92342MiB
[soak] warm baseline after session 1: 92342 MiB
[soak] session 2/5
[soak] turn 1/5: status=200 wall=1254ms ttft=1089ms decode_tps=175.221 vram=92342MiB
[soak] turn 2/5: status=200 wall=4272ms ttft=4046ms decode_tps=123.894 vram=92342MiB
[soak] turn 3/5: status=200 wall=4526ms ttft=4044ms decode_tps=80.893 vram=92342MiB
[soak] turn 4/5: status=200 wall=4634ms ttft=3845ms decode_tps=57.074 vram=92342MiB
[soak] turn 5/5: status=200 wall=4311ms ttft=2976ms decode_tps=38.205 vram=92342MiB
[soak] session 3/5
[soak] turn 1/5: status=200 wall=1252ms ttft=1088ms decode_tps=176.803 vram=92342MiB
[soak] turn 2/5: status=200 wall=4268ms ttft=4041ms decode_tps=123.512 vram=92342MiB
[soak] turn 3/5: status=200 wall=4519ms ttft=4037ms decode_tps=80.922 vram=92342MiB
[soak] turn 4/5: status=200 wall=4636ms ttft=3847ms decode_tps=57.079 vram=92342MiB
[soak] turn 5/5: status=200 wall=4311ms ttft=2973ms decode_tps=40.367 vram=92342MiB
[soak] session 4/5
[soak] turn 1/5: status=200 wall=1253ms ttft=1087ms decode_tps=175.251 vram=92342MiB
[soak] turn 2/5: status=200 wall=4266ms ttft=4041ms decode_tps=123.962 vram=92342MiB
[soak] turn 3/5: status=200 wall=4504ms ttft=4022ms decode_tps=80.919 vram=92342MiB
[soak] turn 4/5: status=200 wall=4638ms ttft=3849ms decode_tps=57.084 vram=92342MiB
[soak] turn 5/5: status=200 wall=4507ms ttft=2978ms decode_tps=39.26 vram=92342MiB
[soak] session 5/5
[soak] turn 1/5: status=200 wall=1253ms ttft=1088ms decode_tps=175.548 vram=92342MiB
[soak] turn 2/5: status=200 wall=4266ms ttft=4040ms decode_tps=124.005 vram=92342MiB
[soak] turn 3/5: status=200 wall=4517ms ttft=4035ms decode_tps=80.892 vram=92342MiB
[soak] turn 4/5: status=200 wall=4613ms ttft=3825ms decode_tps=57.091 vram=92342MiB
[soak] turn 5/5: status=200 wall=4912ms ttft=2996ms decode_tps=35.493 vram=92342MiB
[soak] summary
[soak] verdict PASS
[soak] boot_vram_mib 92342
[soak] max_vram_mib 92342
[soak] max_growth_mib 0 / 200
[soak] errors 0
[soak] silent_empty 0 / 25 (0.0%)
[soak] p50_decode_tps 80.92
[soak] p95_ttft_ms 15174
[soak] tps_retention 100.0%
[soak] note PASS = no failure signal on this sample;
[soak] not patch validation (topology alone can
[soak] sidestep what overlays target). See
[soak] scripts/soak-test.sh --help and docs/CLIFFS.md.
[soak] artifacts: results/report-soak-20260619-164949
Soak summary (results/report-soak-20260619-164949/summary.md):
- Verdict: PASS
- Boot VRAM baseline: 92342 MiB
- Max VRAM observed: 92342 MiB
- Max growth observed: 0 MiB
- Sessions completed: 5
- Request errors: 0
- Silent-empty turns (HTTP 200 + 0 completion tokens): 0 / 25 (0.0%)
| Metric | Value |
|---|---|
| p50 decode TPS | 80.92 |
| p95 decode TPS | 176.55 |
| first-5 median TPS | 80.92 |
| last-5 median TPS | 80.92 |
| TPS retention | 100.0% |
| p50 TTFT | 3849 ms |
| p95 TTFT | 15174 ms |
| TTFT first/last ratio | 1.00x |
| VRAM oscillation | 0 MiB |
- Runtime VRAM growth and throughput retention stayed within v1 soak thresholds.
bench output (3 warmups + 5 measured per prompt)
[autodetect] using running container=vllm-qwen36-27b-multi4-max url=http://localhost:8015 (skip: PREFLIGHT_NO_AUTODETECT=1)
[autodetect] served model='qwen3.6-27b-fp8' (from http://localhost:8015/v1/models; set MODEL= to override)
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
warm-1 wall= 11.75s ttft= 171ms toks=1000 wall_TPS= 85.11 decode_TPS= 86.36
warm-2 wall= 12.48s ttft= 167ms toks=1000 wall_TPS= 80.10 decode_TPS= 81.19
warm-3 wall= 12.66s ttft= 163ms toks=1000 wall_TPS= 78.97 decode_TPS= 80.00
=== measured (5) ===
run-1 wall= 11.81s ttft= 172ms toks=1000 wall_TPS= 84.71 decode_TPS= 85.96
run-2 wall= 12.08s ttft= 169ms toks=1000 wall_TPS= 82.80 decode_TPS= 83.97
run-3 wall= 11.54s ttft= 162ms toks=1000 wall_TPS= 86.69 decode_TPS= 87.92
run-4 wall= 11.82s ttft= 186ms toks=1000 wall_TPS= 84.63 decode_TPS= 85.98
run-5 wall= 11.55s ttft= 186ms toks=1000 wall_TPS= 86.61 decode_TPS= 88.03
=== summary [narrative] (n=5) ===
wall_TPS mean= 85.09 std= 1.62 CV= 1.9% min=82.80 max=86.69
decode_TPS mean= 86.37 std= 1.68 CV= 1.9% min=83.97 max=88.03
TTFT mean= 175ms std= 11ms min=162ms max=186ms
PP tok/s mean= 2.00 std= 1.12 CV=55.9% min=0.00 max=2.50
========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
warm-1 wall= 3.77s ttft= 167ms toks= 395 wall_TPS=104.89 decode_TPS=109.76
warm-2 wall= 3.28s ttft= 192ms toks= 338 wall_TPS=103.19 decode_TPS=109.62
warm-3 wall= 7.44s ttft= 197ms toks= 718 wall_TPS= 96.49 decode_TPS= 99.12
=== measured (5) ===
run-1 wall= 3.54s ttft= 200ms toks= 361 wall_TPS=101.94 decode_TPS=108.02
run-2 wall= 3.70s ttft= 199ms toks= 386 wall_TPS=104.40 decode_TPS=110.32
run-3 wall= 6.46s ttft= 192ms toks= 681 wall_TPS=105.43 decode_TPS=108.66
run-4 wall= 5.98s ttft= 202ms toks= 592 wall_TPS= 98.95 decode_TPS=102.40
run-5 wall= 4.62s ttft= 201ms toks= 469 wall_TPS=101.45 decode_TPS=106.05
=== summary [code] (n=5) ===
wall_TPS mean= 102.43 std= 2.56 CV= 2.5% min=98.95 max=105.43
decode_TPS mean= 107.09 std= 3.04 CV= 2.8% min=102.40 max=110.32
TTFT mean= 198ms std= 4ms min=192ms max=202ms
PP tok/s mean= 4.50 std= 1.12 CV=24.8% min=2.50 max=5.00
=== GPU state ===
0, 99 %, 23086 MiB, 24576 MiB, 167.52 W, 61
1, 18 %, 23084 MiB, 24576 MiB, 171.72 W, 60
2, 40 %, 23086 MiB, 24576 MiB, 178.03 W, 63
3, 53 %, 23086 MiB, 24576 MiB, 180.62 W, 54
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 06-19 16:54:13 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.19, Accepted throughput: 67.60 tokens/s, Drafted throughput: 92.40 tokens/s, Accepted: 676 tokens, Drafted: 924 tokens, Per-position acceptance rate: 0.896, 0.744, 0.555, Avg Draft acceptance rate: 73.2%
(APIServer pid=1) INFO 06-19 16:54:23 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.48, Accepted throughput: 75.20 tokens/s, Drafted throughput: 90.90 tokens/s, Accepted: 752 tokens, Drafted: 909 tokens, Per-position acceptance rate: 0.924, 0.832, 0.726, Avg Draft acceptance rate: 82.7%
(APIServer pid=1) INFO 06-19 16:54:33 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.31, Accepted throughput: 70.10 tokens/s, Drafted throughput: 90.89 tokens/s, Accepted: 701 tokens, Drafted: 909 tokens, Per-position acceptance rate: 0.888, 0.789, 0.637, Avg Draft acceptance rate: 77.1%
bench-agentic output (1 session x 12 default turns, curve-shape estimate; ~8 min estimate)
[autodetect] using running container=vllm-qwen36-27b-multi4-max url=http://localhost:8015 (skip: PREFLIGHT_NO_AUTODETECT=1)
========================================================================
SESSION 1/1 — 12 turns, context grows to ~29,033 tokens
========================================================================
Turn Prompt tok TTFT ms Decode TPS Result chars
----- ---------- --------- ----------- -------------
1 1,256 3266 140.9 307
2 1,439 2258 156.1 249
3 1,620 2521 143.5 278
4 1,830 2833 244.0 8,373
5 4,887 7308 101.4 8,912
6 7,621 6813 101.2 3,106
7 8,967 6557 64.2 6,495
8 10,921 7133 87.0 2,576
9 12,266 6930 156.5 25,250
10 21,430 18893 68.2 17,407
11 27,582 14391 56.1 21,299
12 35,281 17399 42.8 21,883
========================================================================
SUMMARY — multi-turn prefill stress (1 session(s) × 12 turns)
========================================================================
Turn Prompt tok TTFT ms σ ms Decode TPS Notes
----- ---------- --------- ------ ----------- ───────────────────────────────────
1 1,256 3266 0 140.9 cold-start (compile/warmup — excluded from growth)
2 1,439 2258 0 156.1 warm baseline
3 1,620 2521 0 143.5
4 1,830 2833 0 244.0
5 4,887 7308 0 101.4 ↑ TTFT 3.2× warm-baseline
6 7,621 6813 0 101.2 ↑ TTFT 3.0× warm-baseline
7 8,967 6557 0 64.2 ↑ TTFT 2.9× warm-baseline
8 10,921 7133 0 87.0 ↑ TTFT 3.2× warm-baseline
9 12,266 6930 0 156.5 ↑ TTFT 3.1× warm-baseline
10 21,430 18893 0 68.2 ⚠ TTFT 8.4× warm-baseline (O(n)-like growth for this arch_class)
11 27,582 14391 0 56.1 ⚠ TTFT 6.4× warm-baseline (O(n)-like growth for this arch_class)
12 35,281 17399 0 42.8 ⚠ TTFT 7.7× warm-baseline (O(n)-like growth for this arch_class)
────────────────────────────────────────────────────────────────────────
TTFT growth by accumulated context (12 turns, 1 sessions):
Turn 1 (cold): 3266 ms TTFT — compile/warmup, excluded from growth
Turn 2 (warm base): 2258 ms TTFT @ 1,439 prompt tokens
Turn 12: 17399 ms TTFT @ 35,281 prompt tokens
Context grew 24.5×, TTFT grew 7.7× (warm baseline → last turn)
~ TTFT sub-linear for this cell (7.7× vs 24.5× context).
(Full-context O(n) growth would approach 24.5× with context)
Note — DeltaNet/SSM state is NOT prefix-cacheable on vLLM Qwen3-Next cells.
Attention KV caching can still work, but recurrent-state recomputation scales
O(n) with sequence length. Prior single-card 24 GB vLLM Qwen3-Next observations
saw degradation above ~35K tokens and timeouts around ~74K; treat those as
informational per-arch_class guideposts. llama.cpp is not affected.
=== GPU state ===
0, 100 %, 23086 MiB, 24576 MiB, 132.21 W, 61
1, 100 %, 23084 MiB, 24576 MiB, 137.66 W, 61
2, 100 %, 23086 MiB, 24576 MiB, 136.61 W, 61
3, 100 %, 23086 MiB, 24576 MiB, 138.94 W, 52
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 06-19 16:55:43 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.00, Accepted throughput: 1.40 tokens/s, Drafted throughput: 1.70 tokens/s, Accepted: 14 tokens, Drafted: 17 tokens, Per-position acceptance rate: 0.857, 0.714, 0.429, Avg Draft acceptance rate: 82.4%
(APIServer pid=1) INFO 06-19 16:55:53 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 0.90 tokens/s, Drafted throughput: 0.90 tokens/s, Accepted: 9 tokens, Drafted: 9 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 06-19 16:56:03 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.20, Accepted throughput: 2.20 tokens/s, Drafted throughput: 2.50 tokens/s, Accepted: 22 tokens, Drafted: 25 tokens, Per-position acceptance rate: 0.900, 0.800, 0.500, Avg Draft acceptance rate: 88.0%
Generated by bash scripts/report.sh. Flags: --verify (verify-full), --stress (verify-stress 7/7 incl. Cliff 2 needles), --soak (SOAK_MODE=continuous, catches Cliff 2b), --bench (canonical TPS), --agentic (multi-turn TTFT/decode curve-shape, ~8 min estimate), --full (all five, ~43 min estimate). Use --no-redact to disable redaction (internal sharing only).