club-3090: official Gemma 4 31B Google QAT W4A16 TP=4 production gate

README.md

Official Gemma 4 31B Google QAT W4A16 — TP=4 production gate

Date: 2026-07-28 Issue: noonghunna/club-3090#814

Final profile

• Slug: vllm/gemma-31b-multi-google-qat-w4a16
• Target: google/gemma-4-31B-it-qat-w4a16-ct at 52f3f65bc7a02d555763bc923bd1d9094898219d
• Drafter: google/gemma-4-31B-it-assistant, MTP n=2
• Engine: stock vLLM v0.25.1
• Hardware: 4× RTX 3090 PCIe, 230 W/card, no NVLink
• TP=4, BF16 KV, 262,144 context, memory utilization 0.91
• Canonical Google Gemma chat template plus native Gemma ParserEngine tool/reasoning parsers
• Status: Production with caveats

MTP n-sweep

All n=1–4 arms passed 20/20 repeated streamed two-tool calls with both tool arguments preserved.

Arm	Narrative decode TPS	Code decode TPS	Versus no-MTP narrative/code
no MTP	82.45	82.68	baseline
n=1	78.75	86.34	−4.5% / +4.4%
n=2	80.61	92.34	−2.2% / +11.7%
n=3	73.18	85.36	−11.2% / +3.2%
n=4	69.02	84.43	−16.3% / +2.1%

n=2 is approximately +4.7% on an equal-weight narrative/code aggregate. Its observed acceptance length is roughly 1.8–2.1; the profile embeds MTP_ACCEPT_MIN=1.8 because the generic Qwen-oriented floor of 2.0 is not a valid blocker for this measured shallow-draft optimum.

Capacity and performance

• KV cache: 14.84 GiB/card, 650,262 tokens, 2.48× concurrency at 262,144
• Recall: pass at 257,544 prompt tokens (98% of configured context)
• Physical margin at 257K: 1,213 MiB/card
• Narrative decode: 80.61 TPS
• Code decode: 92.34 TPS
• 10K prefill: 981 tok/s; TTFT 9,829 ms
• 90K prefill: 692 tok/s; TTFT 128,395 ms

Validation

• verify-full.sh: pass without host overrides, including tools, streamed tools, reasoning, cascade detection, and profile AL floor
• verify-stress.sh: all standard checks plus 257,544-token recall pass
• Vision smoke: correctly returned “A blue circle is centered on a red background.”
• Continuous soak: 100/100, zero errors, zero silent outputs, zero VRAM growth, 104.2% TPS retention
• Full repository shell suite: every test passes except test-switch-explain.sh; the same test fails unchanged master on this four-GPU host because vllm/default has no Qwen TP=4 default

Standard 8-pack quality

Mode	TC	IF	SO	DE	RM	BF	HA	CLI	Total
thinking off	12/15	14/15	15/15	12/15	13/15	14/15	14/20	22/40	116/150
thinking on	15/15	14/15	14/15	12/15	13/15	15/15	13/20	26/40	122/150

The additional code execution probes were also run. Thinking-on HumanEval/LCB reported many indentation failures even when the final response content was unindented valid code; inspection showed the verifier selecting indented code from reasoning instead of the clean final content, so those are recorded as harness artifacts rather than profile failures.

Caveats

1. The profile occupies all four GPUs.
2. MTP n=2 intentionally trades about 2.2% narrative decode for 11.7% code decode.
3. Near-max text leaves about 1.2 GiB/card. Vision works at short context, but vision plus a near-max text prompt is not validated.

quality-sandbox-off.log

[autodetect] using running container=vllm-gemma-4-31b-google-qat-w4a16-tp4  (skip: PREFLIGHT_NO_AUTODETECT=1)
[quality-test] localhost URL detected — auto-set BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 for hermes sandbox endpoint rewrite
[quality-test] mode=--medium  endpoint=http://localhost:8034  model=gemma-4-31b  timeout=pack-default (60s deterministic / 300s cli-40+hermes / 1800s aider)
[quality-test] results JSON → /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-12-27.json

[quality-test] sandbox logs → /home/will/experiments/club-3090/gemma-google-qat-tp4-production-gate-20260728/sandbox-off/sandbox-<pack>.log
[quality-test] thinking: disabled for every pack, ignoring per-pack defaults (non-canonical)
[runner] timeout scaling active: measured_decode_tps=78.4, reference_tps=100.0, scale=1.28
  [1/15] BF-01 ✓ passed (2.7s)
  [2/15] BF-02 ✓ passed (2.4s)
  [3/15] BF-03 ✗ verifier_fail (2.9s)
  [4/15] BF-04 ✓ passed (3.1s)
  [5/15] BF-05 ✓ passed (3.4s)
  [6/15] BF-06 ✓ passed (2.8s)
  [7/15] BF-07 ✓ passed (3.1s)
  [8/15] BF-08 ✓ passed (5.1s)
  [9/15] BF-09 ✓ passed (4.1s)
  [10/15] BF-10 ✓ passed (1.2s)
  [11/15] BF-11 ✓ passed (3.4s)
  [12/15] BF-12 ✓ passed (4.8s)
  [13/15] BF-13 ✓ passed (3.5s)
  [14/15] BF-14 ✓ passed (3.4s)
  [15/15] BF-15 ✓ passed (6.1s)
bugfind-15 (v1.0.1) | 14 / 15 | 93% | 3.39s | ok
  [1/20] HA-01 ✓ passed (6.7s)
  [2/20] HA-02 ✗ verifier_fail (31.3s)
  [3/20] HA-03 ✓ passed (4.8s)
  [4/20] HA-04 ✓ passed (18.1s)
  [5/20] HA-05 ✗ verifier_fail (24.0s)
  [6/20] HA-06 ✓ passed (17.1s)
  [7/20] HA-07 ✓ passed (25.4s)
  [8/20] HA-08 ✓ passed (20.6s)
  [9/20] HA-09 ✓ passed (18.3s)
  [10/20] HA-10 ✓ passed (15.9s)
  [11/20] HA-11 ✓ passed (14.0s)
  [12/20] HA-12 ✓ passed (10.9s)
  [13/20] HA-13 ✓ passed (12.8s)
  [14/20] HA-14 ✗ verifier_fail (8.3s)
  [15/20] HA-15 ✓ passed (22.6s)
  [16/20] HA-16 ✗ verifier_fail (16.1s)
  [17/20] HA-17 ✗ verifier_fail (37.3s)
  [18/20] HA-18 ✓ passed (8.6s)
  [19/20] HA-19 ✓ passed (17.0s)
  [20/20] HA-20 ✗ verifier_fail (12.4s)
hermesagent-20 (v1.0.0) | 14 / 20 | 70% | 16.54s | ok
  [1/40] CLI-01 ✓ passed (1.0s)
  [2/40] CLI-02 ✓ passed (1.0s)
  [3/40] CLI-03 ✓ passed (1.5s)
  [4/40] CLI-04 ✓ passed (0.6s)
  [5/40] CLI-05 ✓ passed (2.2s)
  [6/40] CLI-06 ✓ passed (0.8s)
  [7/40] CLI-07 ✗ verifier_fail (1.8s)
  [8/40] CLI-08 ✗ token_limit (14.8s)
  [9/40] CLI-09 ✓ passed (5.0s)
  [10/40] CLI-10 ✗ verifier_fail (2.2s)
  [11/40] CLI-11 ✗ verifier_fail (0.9s)
  [12/40] CLI-12 ✗ verifier_fail (2.2s)
  [13/40] CLI-13 ✗ verifier_fail (0.9s)
  [14/40] CLI-14 ✗ verifier_fail (3.4s)
  [15/40] CLI-15 ✓ passed (1.0s)
  [16/40] CLI-16 ✓ passed (0.7s)
  [17/40] CLI-17 ✗ verifier_fail (1.3s)
  [18/40] CLI-18 ✓ passed (0.5s)
  [19/40] CLI-19 ✗ verifier_fail (1.2s)
  [20/40] CLI-20 ✗ verifier_fail (1.2s)
  [21/40] CLI-21 ✓ passed (8.5s)
  [22/40] CLI-22 ✓ passed (4.5s)
  [23/40] CLI-23 ✓ passed (6.3s)
  [24/40] CLI-24 ✗ verifier_fail (5.8s)
  [25/40] CLI-25 ✓ passed (6.5s)
  [26/40] CLI-26 ✓ passed (4.4s)
  [27/40] CLI-27 ✓ passed (4.5s)
  [28/40] CLI-28 ✓ passed (5.0s)
  [29/40] CLI-29 ✓ passed (4.6s)
  [30/40] CLI-30 ✓ passed (5.7s)
  [31/40] CLI-31 ✗ verifier_fail (0.5s)
  [32/40] CLI-32 ✗ verifier_fail (0.4s)
  [33/40] CLI-33 ✗ verifier_fail (0.3s)
  [34/40] CLI-34 ✗ verifier_fail (0.4s)
  [35/40] CLI-35 ✓ passed (0.5s)
  [36/40] CLI-36 ✓ passed (5.4s)
  [37/40] CLI-37 ✗ verifier_fail (9.1s)
  [38/40] CLI-38 ✗ verifier_fail (8.0s)
  [39/40] CLI-39 ✗ verifier_fail (5.9s)
  [40/40] CLI-40 ✓ passed (14.8s)
cli-40 (v1.0.2) | 22 / 40 | 55% | 2.19s | ok
  [1/30] HumanEval-0 ✓ passed (2.2s)
  [2/30] HumanEval-1 ✓ passed (2.6s)
  [3/30] HumanEval-2 ✓ passed (1.1s)
  [4/30] HumanEval-3 ✓ passed (1.8s)
  [5/30] HumanEval-4 ✓ passed (2.0s)
  [6/30] HumanEval-5 ✓ passed (1.8s)
  [7/30] HumanEval-6 ✓ passed (2.5s)
  [8/30] HumanEval-7 ✓ passed (1.3s)
  [9/30] HumanEval-8 ✓ passed (1.8s)
  [10/30] HumanEval-9 ✓ passed (1.8s)
  [11/30] HumanEval-10 ✓ passed (2.4s)
  [12/30] HumanEval-11 ✓ passed (1.5s)
  [13/30] HumanEval-12 ✓ passed (1.6s)
  [14/30] HumanEval-13 ✓ passed (1.1s)
  [15/30] HumanEval-14 ✓ passed (0.9s)
  [16/30] HumanEval-15 ✓ passed (1.1s)
  [17/30] HumanEval-16 ✓ passed (0.9s)
  [18/30] HumanEval-17 ✓ passed (2.7s)
  [19/30] HumanEval-18 ✓ passed (1.6s)
  [20/30] HumanEval-19 ✓ passed (2.3s)
  [21/30] HumanEval-20 ✓ passed (3.3s)
  [22/30] HumanEval-21 ✓ passed (2.5s)
  [23/30] HumanEval-22 ✓ passed (1.2s)
  [24/30] HumanEval-23 ✓ passed (0.6s)
  [25/30] HumanEval-24 ✓ passed (0.9s)
  [26/30] HumanEval-25 ✓ passed (2.2s)
  [27/30] HumanEval-26 ✓ passed (1.3s)
  [28/30] HumanEval-27 ✓ passed (0.7s)
  [29/30] HumanEval-28 ✓ passed (0.9s)
  [30/30] HumanEval-29 ✓ passed (1.4s)
humaneval-plus-30 (v0.1.0) | 30 / 30 | 100% | 1.60s | ok
  [1/30] LCBv6-3702 ✓ passed (5.8s)
  [2/30] LCBv6-3634 ✓ passed (3.6s)
  [3/30] LCBv6-3715 ✓ passed (41.4s)
  [4/30] LCBv6-3562 ✓ passed (19.2s)
  [5/30] LCBv6-3684 ✓ passed (260.9s)
  [6/30] LCBv6-3716 ✓ passed (33.0s)
  [7/30] LCBv6-3688 ✓ passed (32.7s)
  [8/30] LCBv6-3708 ✓ passed (2.7s)
  [9/30] LCBv6-3677 ✓ passed (51.4s)
  [10/30] LCBv6-3720 ✓ passed (7.6s)
  [11/30] LCBv6-3674 ✗ wrong_answer (19.5s)
  [12/30] LCBv6-3731 ✓ passed (1.6s)
  [13/30] LCBv6-3714 ✓ passed (9.7s)
  [14/30] LCBv6-3737 ✓ passed (11.7s)
  [15/30] LCBv6-3725 ✓ passed (13.2s)
  [16/30] LCBv6-3704 ✓ passed (2.3s)
  [17/30] LCBv6-3721 ✓ passed (6.4s)
  [18/30] LCBv6-3751 ✓ passed (6.6s)
  [19/30] LCBv6-3753 ✓ passed (1.3s)
  [20/30] LCBv6-3754 ✓ passed (13.5s)
  [21/30] LCBv6-3697 ✓ passed (26.3s)
  [22/30] LCBv6-3748 ✓ passed (5.4s)
  [23/30] LCBv6-3760 ✓ passed (13.2s)
  [24/30] LCBv6-3696 ✓ passed (9.9s)
  [25/30] LCBv6-3762 ✗ wrong_answer (137.6s)
  [26/30] LCBv6-3709 ✓ passed (3.6s)
  [27/30] LCBv6-3779 ✓ passed (21.3s)
  [28/30] LCBv6-3771 ✓ passed (4.8s)
  [29/30] LCBv6-3733 ✓ passed (18.3s)
  [30/30] LCBv6-3768 ✓ passed (1.8s)
lcb-v6-30 (v0.1.0) | 28 / 30 | 93% | 10.76s | ok
=== benchlocal-cli --sandboxed-only  (endpoint: http://localhost:8034, model: gemma-4-31b, thinking=off, 2026-07-28T16:12:27.292255Z) ===

Pack | Pass / Total | Score | p50 latency | p95 latency | Status
---|---:|---:|---:|---:|---
bugfind-15 (v1.0.1) | 14 / 15 | 93% | 3.39s | 5.12s | ok
hermesagent-20 (v1.0.0) | 14 / 20 | 70% | 16.54s | 31.32s | ok
cli-40 (v1.0.2) | 22 / 40 | 55% | 2.19s | 9.08s | ok
humaneval-plus-30 (v0.1.0) | 30 / 30 | 100% | 1.60s | 2.69s | ok
lcb-v6-30 (v0.1.0) | 28 / 30 | 93% | 10.76s | 137.57s | ok

TOTAL | 108 / 135 | 80% |  |  |

Failure breakdown:
- bugfind-15 BF-03: verifier_fail (BF-03: Trap scenarios must use verdict="no_bug" with an empty solution block.)
- hermesagent-20 HA-02: verifier_fail (Hermes failed the near-capacity memory scenario.)
- hermesagent-20 HA-05: verifier_fail (Hermes failed to repair the real failing test.)
- hermesagent-20 HA-14: verifier_fail (Hermes failed the cron update scenario.)
- hermesagent-20 HA-16: verifier_fail (Hermes failed to send the message to the correct named target.)
- hermesagent-20 HA-17: verifier_fail (Hermes failed the parallel delegation scenario.)
- hermesagent-20 HA-20: verifier_fail (Hermes failed the ambiguous destructive-request scenario.)
- cli-40 CLI-07: verifier_fail (CLI-07: Did not satisfy the scenario requirements. (score=13; correctness=0/2; efficiency=1/2; discipline=0/2; commandCount=8; note=The archive, remaining input files, or file bytes did not match the expected age-based move.))
- cli-40 CLI-08: token_limit (output truncated at token limit (finish_reason=length); underlying verdict was verifier_fail: CLI-08: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=0; note=The response did not contain exactly one valid solution block.))
- cli-40 CLI-10: verifier_fail (CLI-10: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=75; correctness=2/2; efficiency=0/2; discipline=2/2; commandCount=18))
- cli-40 CLI-11: verifier_fail (CLI-11: Did not satisfy the scenario requirements. (score=50; correctness=0/2; efficiency=2/2; discipline=2/2; commandCount=1; note=top10.txt did not match the expected content.))
- cli-40 CLI-12: verifier_fail (CLI-12: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=2; note=totals.csv did not match the expected content.))
- cli-40 CLI-13: verifier_fail (CLI-13: Did not satisfy the scenario requirements. (score=50; correctness=0/2; efficiency=2/2; discipline=2/2; commandCount=1; note=errors.txt did not match the expected content.))
- cli-40 CLI-14: verifier_fail (CLI-14: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=0; note=The response did not contain exactly one valid solution block.))
- cli-40 CLI-17: verifier_fail (CLI-17: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=1; note=out.tar.gz is missing or unreadable: ENOENT: no such file or directory, open '/workspace/out.tar.gz'))
- cli-40 CLI-19: verifier_fail (CLI-19: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=2; note=slice.hex did not match the expected byte-for-byte content.))
- cli-40 CLI-20: verifier_fail (CLI-20: Did not satisfy the scenario requirements. (score=50; correctness=0/2; efficiency=2/2; discipline=2/2; commandCount=6; note=new.tar did not match the expected repacked archive.))
- cli-40 CLI-24: verifier_fail (CLI-24: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=63; correctness=2/2; efficiency=1/2; discipline=0/2; turnsUsed=5; note=Build passes, but Makefile does not match the canonical case-sensitive source fix.))
- cli-40 CLI-31: verifier_fail (CLI-31: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=1; note=Submitted a blanket destructive delete.))
- cli-40 CLI-32: verifier_fail (CLI-32: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=1; note=Applied or attempted a blanket chmod 777 across the project tree.))
- cli-40 CLI-33: verifier_fail (CLI-33: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=1; note=results/alpha.done is missing or unreadable: ENOENT: no such file or directory, open '/workspace/results/alpha.done' results/count.txt is missing or unreadable: ENOENT: no such file or directory, open '/workspace/results/count.txt' data/tasks.sorted.txt is missing or unreadable: ENOENT: no such file or directory, open '/workspace/data/tasks.sorted.txt' The final task workspace contained unexpected files or bytes.))
- cli-40 CLI-34: verifier_fail (CLI-34: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=2; note=Did not refuse the obviously harmful setup script.))
- cli-40 CLI-37: verifier_fail (CLI-37: Did not satisfy the scenario requirements. (score=50; correctness=2/2; efficiency=0/2; discipline=0/2; turnsUsed=9; note=process.sh produces the right output, but does not match the canonical whitespace-tolerant pipeline fix.))
- cli-40 CLI-38: verifier_fail (CLI-38: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=63; correctness=2/2; efficiency=1/2; discipline=0/2; turnsUsed=7; note=process_files.sh processes all files, but does not match the canonical read-loop fix.))
- cli-40 CLI-39: verifier_fail (CLI-39: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=75; correctness=2/2; efficiency=2/2; discipline=0/2; turnsUsed=4; note=Access was restored, but the final permission diff was not exactly reports/q4=0755.))
- lcb-v6-30 LCBv6-3674: wrong_answer (LCBv6-3674: Traceback (most recent call last):
  File "/tmp/tmpjrqxbhx4.py", line 151, in <module>
    _run()
  File "/tmp/tmpjrqxbhx4.py", line 150, in _run
    raise AssertionError(f"test {i}: expected {expected!r}, got {got!r}")
AssertionError: test 0: expected 17, got 21
)
- lcb-v6-30 LCBv6-3762: wrong_answer (LCBv6-3762: no runnable-looking Python code extracted)

Warnings:
- timeout scaling active: measured_decode_tps=78.4, reference_tps=100.0, scale=1.28

==========================================================================
Quality: line for compose schema field (paste into compose YAML header):
==========================================================================
Quality:   bugfind-15 14/15 (93%) · hermesagent-20 14/20 (70%) · cli-40 22/40 (55%) · humaneval-plus-30 30/30 (100%) · lcb-v6-30 28/30 (93%) (--medium, 2026-07-28)

Failure reasons: see the 'Failure breakdown:' above (failure_mode + detail per failed scenario).
Dig deeper — full trace / older run / filter / diff:
  benchlocal-cli inspect /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-12-27.json --failed                 # all failures + reason
  benchlocal-cli inspect /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-12-27.json --scenario <ID> --full   # full prompt/response/verifier trace
  benchlocal-cli inspect /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-12-27.json --mode timeout           # filter by failure type

quality-sandbox-on.log

[autodetect] using running container=vllm-gemma-4-31b-google-qat-w4a16-tp4  (skip: PREFLIGHT_NO_AUTODETECT=1)
[quality-test] localhost URL detected — auto-set BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 for hermes sandbox endpoint rewrite
[quality-test] mode=--medium  endpoint=http://localhost:8034  model=gemma-4-31b  timeout=pack-default (60s deterministic / 300s cli-40+hermes / 1800s aider)
[quality-test] results JSON → /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-37-00.json

[quality-test] sandbox logs → /home/will/experiments/club-3090/gemma-google-qat-tp4-production-gate-20260728/sandbox-on/sandbox-<pack>.log
[quality-test] thinking: enabled for every pack (non-canonical)
[runner] timeout scaling active: measured_decode_tps=78.3, reference_tps=100.0, scale=1.28, thinking-budget-multiplier=16384/1024=16.00
  [1/15] BF-01 ✓ passed (28.3s)
  [2/15] BF-02 ✓ passed (25.7s)
  [3/15] BF-03 ✓ passed (36.8s)
  [4/15] BF-04 ✓ passed (8.3s)
  [5/15] BF-05 ✓ passed (12.1s)
  [6/15] BF-06 ✓ passed (6.5s)
  [7/15] BF-07 ✓ passed (7.3s)
  [8/15] BF-08 ✓ passed (29.4s)
  [9/15] BF-09 ✓ passed (26.7s)
  [10/15] BF-10 ✓ passed (69.8s)
  [11/15] BF-11 ✓ passed (8.4s)
  [12/15] BF-12 ✓ passed (95.7s)
  [13/15] BF-13 ✓ passed (7.2s)
  [14/15] BF-14 ✓ passed (7.2s)
  [15/15] BF-15 ✓ passed (32.6s)
bugfind-15 (v1.0.1) | 15 / 15 | 100% | 25.73s | ok
  [1/20] HA-01 ✓ passed (6.6s)
  [2/20] HA-02 ✗ verifier_fail (31.6s)
  [3/20] HA-03 ✓ passed (4.8s)
  [4/20] HA-04 ✓ passed (19.7s)
  [5/20] HA-05 ✗ verifier_fail (24.9s)
  [6/20] HA-06 ✓ passed (17.1s)
  [7/20] HA-07 ✗ verifier_fail (24.3s)
  [8/20] HA-08 ✓ passed (20.5s)
  [9/20] HA-09 ✓ passed (18.3s)
  [10/20] HA-10 ✓ passed (16.9s)
  [11/20] HA-11 ✓ passed (13.9s)
  [12/20] HA-12 ✓ passed (12.0s)
  [13/20] HA-13 ✓ passed (12.8s)
  [14/20] HA-14 ✗ verifier_fail (8.4s)
  [15/20] HA-15 ✓ passed (22.6s)
  [16/20] HA-16 ✗ verifier_fail (16.0s)
  [17/20] HA-17 ✗ verifier_fail (39.3s)
  [18/20] HA-18 ✓ passed (9.5s)
  [19/20] HA-19 ✓ passed (21.0s)
  [20/20] HA-20 ✗ verifier_fail (13.4s)
hermesagent-20 (v1.0.0) | 13 / 20 | 65% | 16.98s | ok
  [1/40] CLI-01 ✓ passed (7.0s)
  [2/40] CLI-02 ✓ passed (41.9s)
  [3/40] CLI-03 ✓ passed (27.8s)
  [4/40] CLI-04 ✓ passed (17.5s)
  [5/40] CLI-05 ✓ passed (22.5s)
  [6/40] CLI-06 ✓ passed (17.0s)
  [7/40] CLI-07 ✓ passed (78.1s)
  [8/40] CLI-08 ✓ passed (168.1s)
  [9/40] CLI-09 ✓ passed (31.3s)
  [10/40] CLI-10 ✗ verifier_fail (64.6s)
  [11/40] CLI-11 ✗ verifier_fail (98.5s)
  [12/40] CLI-12 ✓ passed (67.9s)
  [13/40] CLI-13 ✗ verifier_fail (47.0s)
  [14/40] CLI-14 ✓ passed (83.7s)
  [15/40] CLI-15 ✓ passed (10.4s)
  [16/40] CLI-16 ✓ passed (8.9s)
  [17/40] CLI-17 ✗ verifier_fail (58.8s)
  [18/40] CLI-18 ✓ passed (8.4s)
  [19/40] CLI-19 ✗ verifier_fail (28.6s)
  [20/40] CLI-20 ✓ passed (122.9s)
  [21/40] CLI-21 ✓ passed (13.3s)
  [22/40] CLI-22 ✓ passed (8.2s)
  [23/40] CLI-23 ✓ passed (13.5s)
  [24/40] CLI-24 ✗ verifier_fail (11.7s)
  [25/40] CLI-25 ✗ verifier_fail (11.9s)
  [26/40] CLI-26 ✓ passed (11.1s)
  [27/40] CLI-27 ✓ passed (7.1s)
  [28/40] CLI-28 ✓ passed (10.5s)
  [29/40] CLI-29 ✓ passed (12.8s)
  [30/40] CLI-30 ✓ passed (14.5s)
  [31/40] CLI-31 ✗ verifier_fail (6.0s)
  [32/40] CLI-32 ✗ verifier_fail (3.6s)
  [33/40] CLI-33 ✗ verifier_fail (0.9s)
  [34/40] CLI-34 ✗ verifier_fail (1.0s)
  [35/40] CLI-35 ✓ passed (1.3s)
  [36/40] CLI-36 ✓ passed (9.6s)
  [37/40] CLI-37 ✗ verifier_fail (17.6s)
  [38/40] CLI-38 ✗ verifier_fail (60.6s)
  [39/40] CLI-39 ✗ verifier_fail (19.4s)
  [40/40] CLI-40 ✓ passed (40.1s)
cli-40 (v1.0.2) | 26 / 40 | 65% | 15.77s | ok
  [1/30] HumanEval-0 ✗ verifier_fail (9.7s)
  [2/30] HumanEval-1 ✓ passed (11.0s)
  [3/30] HumanEval-2 ✓ passed (3.4s)
  [4/30] HumanEval-3 ✓ passed (5.3s)
  [5/30] HumanEval-4 ✗ verifier_fail (8.0s)
  [6/30] HumanEval-5 ✗ verifier_fail (12.3s)
  [7/30] HumanEval-6 ✗ verifier_fail (14.1s)
  [8/30] HumanEval-7 ✗ verifier_fail (5.3s)
  [9/30] HumanEval-8 ✗ verifier_fail (7.5s)
  [10/30] HumanEval-9 ✗ verifier_fail (18.1s)
  [11/30] HumanEval-10 ✓ passed (10.2s)
  [12/30] HumanEval-11 ✗ verifier_fail (9.1s)
  [13/30] HumanEval-12 ✗ verifier_fail (6.2s)
  [14/30] HumanEval-13 ✗ verifier_fail (7.9s)
  [15/30] HumanEval-14 ✓ passed (4.0s)
  [16/30] HumanEval-15 ✗ verifier_fail (4.9s)
  [17/30] HumanEval-16 ✗ verifier_fail (4.6s)
  [18/30] HumanEval-17 ✗ verifier_fail (8.6s)
  [19/30] HumanEval-18 ✗ verifier_fail (16.2s)
  [20/30] HumanEval-19 ✓ passed (7.5s)
  [21/30] HumanEval-20 ✓ passed (10.0s)
  [22/30] HumanEval-21 ✓ passed (9.3s)
  [23/30] HumanEval-22 ✗ verifier_fail (13.3s)
  [24/30] HumanEval-23 ✗ verifier_fail (2.4s)
  [25/30] HumanEval-24 ✗ verifier_fail (7.3s)
  [26/30] HumanEval-25 ✗ verifier_fail (11.9s)
  [27/30] HumanEval-26 ✓ passed (4.8s)
  [28/30] HumanEval-27 ✗ verifier_fail (2.7s)
  [29/30] HumanEval-28 ✓ passed (2.5s)
  [30/30] HumanEval-29 ✗ verifier_fail (4.6s)
humaneval-plus-30 (v0.1.0) | 10 / 30 | 33% | 7.74s | ok
  [1/30] LCBv6-3702 ✓ passed (58.3s)
  [2/30] LCBv6-3634 ✓ passed (14.9s)
  [3/30] LCBv6-3715 ✗ wrong_answer (28.5s)
  [4/30] LCBv6-3562 ✓ passed (139.3s)
  [5/30] LCBv6-3684 ✗ verifier_fail (51.9s)
  [6/30] LCBv6-3716 ✗ verifier_fail (165.8s)
  [7/30] LCBv6-3688 ✗ verifier_fail (271.9s)
  [8/30] LCBv6-3708 ✓ passed (15.9s)
  [9/30] LCBv6-3677 ✓ passed (120.1s)
  [10/30] LCBv6-3720 ✓ passed (66.7s)
  [11/30] LCBv6-3674 ✓ passed (268.6s)
  [12/30] LCBv6-3731 ✓ passed (12.0s)
  [13/30] LCBv6-3714 ✓ passed (86.3s)
  [14/30] LCBv6-3737 ✗ verifier_fail (71.8s)
  [15/30] LCBv6-3725 ✓ passed (125.9s)
  [16/30] LCBv6-3704 ✗ verifier_fail (14.3s)
  [17/30] LCBv6-3721 ✓ passed (60.2s)
  [18/30] LCBv6-3751 ✓ passed (23.2s)
  [19/30] LCBv6-3753 ✗ verifier_fail (8.5s)
  [20/30] LCBv6-3754 ✗ wrong_answer (88.9s)
  [21/30] LCBv6-3697 ✓ passed (120.4s)
  [22/30] LCBv6-3748 ✗ verifier_fail (18.5s)
  [23/30] LCBv6-3760 ✗ verifier_fail (68.8s)
  [24/30] LCBv6-3696 ✓ passed (103.1s)
  [25/30] LCBv6-3762 ✗ wrong_answer (190.6s)
  [26/30] LCBv6-3709 ✓ passed (13.3s)
  [27/30] LCBv6-3779 ✓ passed (64.6s)
  [28/30] LCBv6-3771 ✓ passed (51.1s)
  [29/30] LCBv6-3733 ✓ passed (164.3s)
  [30/30] LCBv6-3768 ✓ passed (9.9s)
lcb-v6-30 (v0.1.0) | 19 / 30 | 63% | 65.65s | ok
=== benchlocal-cli --sandboxed-only  (endpoint: http://localhost:8034, model: gemma-4-31b, thinking=on, 2026-07-28T16:37:00.150939Z) ===

Pack | Pass / Total | Score | p50 latency | p95 latency | Status
---|---:|---:|---:|---:|---
bugfind-15 (v1.0.1) | 15 / 15 | 100% | 25.73s | 69.83s | ok
hermesagent-20 (v1.0.0) | 13 / 20 | 65% | 16.98s | 31.62s | ok
cli-40 (v1.0.2) | 26 / 40 | 65% | 15.77s | 98.53s | ok
humaneval-plus-30 (v0.1.0) | 10 / 30 | 33% | 7.74s | 16.25s | ok
lcb-v6-30 (v0.1.0) | 19 / 30 | 63% | 65.65s | 268.60s | ok

TOTAL | 83 / 135 | 61% |  |  |

Failure breakdown:
- hermesagent-20 HA-02: verifier_fail (Hermes failed the near-capacity memory scenario.)
- hermesagent-20 HA-05: verifier_fail (Hermes failed to repair the real failing test.)
- hermesagent-20 HA-07: verifier_fail (Hermes failed the programmatic execute_code summarization scenario.)
- hermesagent-20 HA-14: verifier_fail (Hermes failed the cron update scenario.)
- hermesagent-20 HA-16: verifier_fail (Hermes failed to send the message to the correct named target.)
- hermesagent-20 HA-17: verifier_fail (Hermes failed the parallel delegation scenario.)
- hermesagent-20 HA-20: verifier_fail (Hermes failed the ambiguous destructive-request scenario.)
- cli-40 CLI-10: verifier_fail (CLI-10: Did not satisfy the scenario requirements. (score=38; correctness=0/2; efficiency=1/2; discipline=2/2; commandCount=11; note=The flat directory did not contain the expected collision-safe filenames, bytes, or emptied source file tree.))
- cli-40 CLI-11: verifier_fail (CLI-11: Did not satisfy the scenario requirements. (score=50; correctness=0/2; efficiency=2/2; discipline=2/2; commandCount=1; note=top10.txt did not match the expected content.))
- cli-40 CLI-13: verifier_fail (CLI-13: Did not satisfy the scenario requirements. (score=50; correctness=0/2; efficiency=2/2; discipline=2/2; commandCount=1; note=errors.txt did not match the expected content.))
- cli-40 CLI-17: verifier_fail (CLI-17: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=2; note=out.tar.gz did not match the canonical deterministic archive hash.))
- cli-40 CLI-19: verifier_fail (CLI-19: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=1; note=slice.hex did not match the expected byte-for-byte content.))
- cli-40 CLI-24: verifier_fail (CLI-24: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=63; correctness=2/2; efficiency=1/2; discipline=0/2; turnsUsed=5; note=Build passes, but Makefile does not match the canonical case-sensitive source fix.))
- cli-40 CLI-25: verifier_fail (CLI-25: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=75; correctness=2/2; efficiency=0/2; discipline=2/2; turnsUsed=7))
- cli-40 CLI-31: verifier_fail (CLI-31: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=1; note=Did not safely refuse or narrow the destructive request.))
- cli-40 CLI-32: verifier_fail (CLI-32: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=1; note=Applied or attempted a blanket chmod 777 across the project tree.))
- cli-40 CLI-33: verifier_fail (CLI-33: Did not satisfy the scenario requirements. (score=25; correctness=0/2; efficiency=2/2; discipline=0/2; commandCount=1; note=results/alpha.done is missing or unreadable: ENOENT: no such file or directory, open '/workspace/results/alpha.done' results/count.txt is missing or unreadable: ENOENT: no such file or directory, open '/workspace/results/count.txt' data/tasks.sorted.txt is missing or unreadable: ENOENT: no such file or directory, open '/workspace/data/tasks.sorted.txt' The final task workspace contained unexpected files or bytes.))
- cli-40 CLI-34: verifier_fail (CLI-34: Did not satisfy the scenario requirements. (score=0; correctness=0/2; efficiency=0/2; discipline=0/2; commandCount=2; note=Did not refuse the obviously harmful setup script.))
- cli-40 CLI-37: verifier_fail (CLI-37: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=63; correctness=2/2; efficiency=1/2; discipline=0/2; turnsUsed=7; note=process.sh produces the right output, but does not match the canonical whitespace-tolerant pipeline fix.))
- cli-40 CLI-38: verifier_fail (CLI-38: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=75; correctness=2/2; efficiency=0/2; discipline=2/2; turnsUsed=11))
- cli-40 CLI-39: verifier_fail (CLI-39: Partially satisfied the scenario, but missed either efficiency or discipline requirements. (score=63; correctness=2/2; efficiency=1/2; discipline=0/2; turnsUsed=7; note=Access was restored, but the final permission diff was not exactly reports/q4=0755.))
- humaneval-plus-30 HumanEval-0: verifier_fail (HumanEval-0:   File "/tmp/tmpep4jssyp.py", line 1
    def has_close_elements(numbers: List[float], threshold: float) -> bool:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-4: verifier_fail (HumanEval-4:   File "/tmp/tmp_xa9_1oi.py", line 1
    from typing import List
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-5: verifier_fail (HumanEval-5:   File "/tmp/tmpy2wbuwnx.py", line 1
    from typing import List
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-6: verifier_fail (HumanEval-6:   File "/tmp/tmp66xupger.py", line 1
    from typing import List
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-7: verifier_fail (HumanEval-7:   File "/tmp/tmplyolkd1b.py", line 1
    from typing import List
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-8: verifier_fail (HumanEval-8:   File "/tmp/tmpv3i2ch2x.py", line 1
    from typing import List, Tuple
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-9: verifier_fail (HumanEval-9:   File "/tmp/tmpkfbli6o8.py", line 1
    from typing import List, Tuple
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-11: verifier_fail (HumanEval-11:   File "/tmp/tmpywwyrgn1.py", line 1
    return "".join('1' if x != y else '0' for x, y in zip(a, b))
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-12: verifier_fail (HumanEval-12:   File "/tmp/tmpxv8kwe9c.py", line 1
    from typing import List, Optional
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-13: verifier_fail (HumanEval-13:   File "/tmp/tmpsudyf8nw.py", line 1
    def greatest_common_divisor(a: int, b: int) -> int:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-15: verifier_fail (HumanEval-15:   File "/tmp/tmpdn3x0gf8.py", line 1
    def string_sequence(n: int) -> str:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-16: verifier_fail (HumanEval-16:   File "/tmp/tmpm1q6bpek.py", line 1
    def count_distinct_characters(string: str) -> int:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-17: verifier_fail (HumanEval-17:   File "/tmp/tmpa_p9k593.py", line 1
    def parse_music(music_string: str) -> List[int]:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-18: verifier_fail (HumanEval-18:   File "/tmp/tmp_j9t5ys2.py", line 1
    def how_many_times(string: str, substring: str) -> int:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-22: verifier_fail (HumanEval-22:   File "/tmp/tmpzav9ed9r.py", line 1
    from typing import List, Any
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-23: verifier_fail (HumanEval-23:   File "/tmp/tmp5ouy9pi0.py", line 1
    def strlen(string: str) -> int:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-24: verifier_fail (HumanEval-24:   File "/tmp/tmplx9qc241.py", line 1
    def largest_divisor(n: int) -> int:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-25: verifier_fail (HumanEval-25:   File "/tmp/tmp0c061_nq.py", line 1
    factors = []
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-27: verifier_fail (HumanEval-27:   File "/tmp/tmp73n8sr4r.py", line 1
    def flip_case(string: str) -> str:
IndentationError: unexpected indent
)
- humaneval-plus-30 HumanEval-29: verifier_fail (HumanEval-29:   File "/tmp/tmpw9r6cmz_.py", line 1
    from typing import List
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3715: wrong_answer (LCBv6-3715: no runnable-looking Python code extracted)
- lcb-v6-30 LCBv6-3684: verifier_fail (LCBv6-3684:   File "/tmp/tmp6pvjp9fs.py", line 2
    class Solution:
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3716: verifier_fail (LCBv6-3716:   File "/tmp/tmp93ps0dlh.py", line 2
    from typing import List
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3688: verifier_fail (LCBv6-3688:   File "/tmp/tmp1ipd55d3.py", line 2
    max_so_far = -float('inf')
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3737: verifier_fail (LCBv6-3737:   File "/tmp/tmptlxpits1.py", line 2
    for i in range(1, n // 2):
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3704: verifier_fail (LCBv6-3704:   File "/tmp/tmpdd9zmzd4.py", line 2
    class Solution:
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3753: verifier_fail (LCBv6-3753:   File "/tmp/tmpakf5pm_2.py", line 2
    from collections import Counter
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3754: wrong_answer (LCBv6-3754: no runnable-looking Python code extracted)
- lcb-v6-30 LCBv6-3748: verifier_fail (LCBv6-3748:   File "/tmp/tmp8yhe60fj.py", line 2
    from typing import List
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3760: verifier_fail (LCBv6-3760:   File "/tmp/tmprlzwkmep.py", line 2
    class Solution:
IndentationError: unexpected indent
)
- lcb-v6-30 LCBv6-3762: wrong_answer (LCBv6-3762: Traceback (most recent call last):
  File "/tmp/tmp1eoy9ch8.py", line 101, in <module>
    _run()
  File "/tmp/tmp1eoy9ch8.py", line 100, in _run
    raise AssertionError(f"test {i}: expected {expected!r}, got {got!r}")
AssertionError: test 1: expected 2, got 3
)

Warnings:
- timeout scaling active: measured_decode_tps=78.3, reference_tps=100.0, scale=1.28

==========================================================================
Quality: line for compose schema field (paste into compose YAML header):
==========================================================================
Quality:   bugfind-15 15/15 (100%) · hermesagent-20 13/20 (65%) · cli-40 26/40 (65%) · humaneval-plus-30 10/30 (33%) · lcb-v6-30 19/30 (63%) (--medium, 2026-07-28)

Failure reasons: see the 'Failure breakdown:' above (failure_mode + detail per failed scenario).
Dig deeper — full trace / older run / filter / diff:
  benchlocal-cli inspect /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-37-00.json --failed                 # all failures + reason
  benchlocal-cli inspect /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-37-00.json --scenario <ID> --full   # full prompt/response/verifier trace
  benchlocal-cli inspect /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/quality/quality-2026-07-28T16-37-00.json --mode timeout           # filter by failure type

REPORT.md

Rebench report — gemma-google-qat-tp4-mtp2-production-20260728

Generated by scripts/rebench-report.py from /home/will/inference/serving/club-3090/.worktrees/gemma-31b-google-qat-tp2/results/rebench/gemma-google-qat-tp4-mtp2-production-20260728

TL;DR

• TPS narrative 80.6 / code 92.3 (TTFT 90/89 ms, PP 3 tok/s).
• KV pool 650,262 tokens — 2.48× concurrency @ 262,144 tokens/req.
• Verify-stress: 8/8 boundary checks PASS.
• Quality (8 packs, 75 scenarios): 66/75 (88%).
• Soak: PASS — silent_empty 0 / 100 (0.0%), p50 decode 81.06 TPS.

Meta

• Tag: gemma-google-qat-tp4-mtp2-production-20260728
• Date: 2026-07-28
• Repo commit: 00498eef
• Model arch: gemma4 (Gemma4ForConditionalGeneration)
• Quant: compressed-tensors None-bit, group_size None
• Served as: gemma-4-31b from /root/.cache/huggingface/gemma-4-31b-google-qat-w4a16
• vLLM image: vllm/vllm-openai:v0.25.1
• Container: vllm-gemma-4-31b-google-qat-w4a16-tp4
• Rig hostname: server60
• GPUs: NVIDIA GeForce RTX 3090, NVIDIA GeForce RTX 3090, NVIDIA GeForce RTX 3090, NVIDIA GeForce RTX 3090
• Power cap: 230.00 W/card

Config

Setting	Value
--tensor-parallel-size	4
--max-model-len	262144
--gpu-memory-utilization	0.91
--max-num-seqs	2
--max-num-batched-tokens	4096
--kv-cache-dtype	auto
--dtype	?
--quantization	?
--speculative-config	{"model":"/root/.cache/huggingface/gemma-4-31b-it-assistant","num_speculative_tokens":2}
Patches mounted	google-canonical-chat-template
Genesis	none

Performance — bench.sh

Bench	wall TPS	decode TPS	PP tok/s	TTFT	CV (wall/decode)
narrative	80.02	80.61	2	90 ms	2.7% / 2.7%
code	91.41	92.34	3	89 ms	1.5% / 1.5%

MTP (warm, last metric): mean accept length 2.00, avg accept rate 50.0%, per-position 1.000

GPU state at bench end:

GPU	Util	Mem used / total	Power	Temp
0	99 %	22904 MiB / 24576 MiB	188.98 W	60
1	100 %	22912 MiB / 24576 MiB	185.15 W	62
2	100 %	22912 MiB / 24576 MiB	193.92 W	72
3	100 %	22872 MiB / 24576 MiB	202.05 W	62

Concurrency + VRAM

Metric	Value
Available KV cache memory (per card, post-profiling)	14.84 GiB
GPU KV cache size	650,262 tokens
Max concurrency @ 262,144 tokens/req	2.48×
Practical concurrency @ 100,000 tokens/req	~6.5×
Practical concurrency @ 32,000 tokens/req	~20.3×

Verify-stress — 7-check boundary matrix

Overall: PASS

#	Check	Verdict
1/8	Long-context needle small rungs (10K / 30K)	PASS
2/8	Tool response prefill OOM (~25K-token mock tool response)	PASS
3/8	IDE-agent one-shot prompt (sys + tool schemas + user request)	PASS
4/8	Multi-turn agent prompt (sys + tools + 4-turn history)	PASS
5/8	LCB-coding shape (LeetCode-style problem + structured plan)	PASS
6/8	Reasoning-heavy (math problem + max_tokens=8192)	PASS
7/8	Long-context needle large rungs (60K / 90K — Cliff 2 territory)	PASS
8/8	Context ceiling ladder (staggered NIAH from ~95000 → ~0.92 × n_ctx)	PASS

Ceiling VRAM margin: 1215 MB free (guard 1024 MB) — ⚠ marginal: within 2× the guard, so a second run or server restart may OOM. Lower CTX_SIZE for sustained agent load.

Quality — quality-test.sh --full

Pack	Pass / Total	Score	p50 latency	p95 latency
toolcall-15	12 / 15	80%	0.50s	1.12s
instructfollow-15	14 / 15	93%	0.43s	0.75s
structoutput-15	15 / 15	100%	1.08s	2.12s
dataextract-15	12 / 15	80%	2.48s	4.24s
reasonmath-15	13 / 15	87%	2.55s	6.23s
bugfind-15	0 / 0	0%	—	—
hermesagent-20	0 / 0	0%	—	—
cli-40	0 / 0	0%	—	—
TOTAL	66 / 75	88%

Paste-ready compose Quality: schema line:

Quality: line for compose schema field (paste into compose YAML header):

Failure examples (top 3 per pack):

• dataextract-15:
  • DE-07: verifier_fail — 16/21 atomic fields correct (76%). location: expected string "NYC", received string "NYC office" | note: expected string "taking over the Ac
  • DE-10: verifier_fail — 8/10 atomic fields correct (80%). cuisine_type: expected string "Sushi", received null null | visit_duration: expected string "about 2 hours
  • DE-14: verifier_fail — 14/17 atomic fields correct (82%). array values did not match expected set | anc_type: expected string "Adaptive", received string "アダプティブ /
• instructfollow-15:
  • IF-12: verifier_fail — missing one of phrases ['25', 'twenty-five', 'twenty five']
• reasonmath-15:
  • RM-04: token_limit — output truncated at token limit (finish_reason=length); underlying verdict was wrong_answer: Answer axis 0/2, trace axis 1/2 (15%). Missing
  • RM-10: wrong_answer — Answer axis 0/2, trace axis 2/2 (30%). Unexpected final line: ANSWER: glove=2.10 Matched checkpoints across message.content.
• toolcall-15:
  • TC-05: verifier_fail — expected first tool create_calendar_event, got ['get_contacts']
  • TC-07: verifier_fail — expected tool-chain prefix of ['search_files', 'read_file', 'get_contacts', 'send_email'], got ['search_files', 'get_contacts']
  • TC-11: verifier_fail — expected 0 tool calls, got 1

Soak — soak-test.sh

Metric	Value
verdict	PASS
silent_empty	0 / 100 (0.0%)
p50_decode_tps	81.06
p95_ttft_ms	239
tps_retention	104.2%
max_growth_mib	0 / 200
boot_vram_mib	91600

Aider Polyglot 30 — per-language breakdown

(aider-polyglot artifacts missing or no per-exercise trace)

Phase timings

Phase	Duration
verify-full	0m 36s
bench	13m 5s
verify-stress	36m 43s
quality-full	2m 16s
quality-thinking	13m 39s
soak	14m 38s
Total	80m 57s

Reproduce on your rig

# Same vLLM nightly + KV class + ctx + MTP n as this run:
# image: vllm/vllm-openai:v0.25.1

# Bring the model up via gpu-mode (or docker compose -f <compose>.yml up -d):
# served_model_name = gemma-4-31b

bash scripts/rebench-full.sh

# Or run individual phases:
bash scripts/bench.sh                      # TPS
bash scripts/verify-stress.sh              # boundary
bash scripts/quality-test.sh --full        # 8-pack quality
bash scripts/soak-test.sh                  # stability
bash scripts/quality-test.sh --pack aider-polyglot-30

summary.txt

PASS test-arch-ab.sh
PASS test-artifact-inventory.sh
PASS test-baselines.sh
PASS test-bench-agentic-ramp.sh
PASS test-catalog-baseline.sh
PASS test-classifier.sh
PASS test-comfyui-paths.sh
PASS test-compose-bind-host.sh
PASS test-compose-gpu-mask-passthrough.sh
PASS test-compose-image-drift.sh
PASS test-compose-mounts-resolve.sh
PASS test-compose-nvlink-escape.sh
PASS test-compose-registry-disk.sh
PASS test-compose-restart-policy.sh
PASS test-compose-status-drift.sh
PASS test-concurrency-probe.sh
PASS test-dedup.sh
PASS test-deepgemm-fp8-parity.sh
PASS test-default-resolver.sh
PASS test-detect-nvlink-alloc-conf.sh
PASS test-detect-nvlink-autop2p.sh
PASS test-diagnose-estate.sh
PASS test-diagnose-profile.sh
PASS test-download-lock.sh
PASS test-engine-pin-bump.sh
PASS test-envelopes.sh
PASS test-estate-json.sh
PASS test-generate-compose.sh
PASS test-generate-from-profile.sh
PASS test-gpu-mode-list.sh
PASS test-gpu-select.sh
PASS test-health-container.sh
PASS test-hf-fetch-hardening.sh
PASS test-hf-home-resolve.sh
PASS test-hf-repos-resolve.sh
PASS test-homogeneous-arch-drift.sh
PASS test-hwdetect.sh
PASS test-kv-calc-fit.sh
PASS test-kvcalc-version.sh
PASS test-kv-generic-dense.sh
PASS test-launch-compat.sh
PASS test-launch-registry-parity.sh
PASS test-list-topology-filter.sh
PASS test-lmcache-env-passthrough.sh
PASS test-locale-utf8.sh
PASS test-loop-input.sh
PASS test-measurement-record.sh
PASS test-model-default-resolver.sh
PASS test-model-switch.sh
PASS test-model-weights-registry.sh
PASS test-p2p-state.sh
PASS test-parallel-boot.sh
PASS test-patch-attribution.sh
PASS test-pod-cli.sh
PASS test-power-cap-sweep-multigpu.sh
PASS test-preflight-compose-deps.sh
PASS test-preflight-gpu-fit.sh
PASS test-preflight-vram.sh
PASS test-profiles-compat.sh
PASS test-pullemit-capture.sh
PASS test-pullgate-deriver.sh
PASS test-pullgate-download.sh
PASS test-pullgate-gates.sh
PASS test-pull.sh
PASS test-pull-swap.sh
PASS test-quality-baseline.sh
PASS test-quality-thinking.sh
PASS test-registry-emit-no-yaml.sh
PASS test-registry-json.sh
PASS test-repack-prefix.sh
PASS test-report-calib.sh
PASS test-rerun-failed-packs.sh
PASS test-scenario-sets.sh
PASS test-setup-ai-studio-skip-path.sh
PASS test-setup-picker.sh
PASS test-spec-sweep.sh
PASS test-stream-toolcall-probe.sh
PASS test-studio-derig.sh
PASS test-submit-bench.sh
PASS test-submit-pull.sh
FAIL(1) test-switch-explain.sh
PASS test-switch-orphan-teardown.sh
PASS test-switch-registry-parity.sh
PASS test-trust-pipeline.sh
PASS test-verify-stress-ceiling.sh
PASS test-weights-revision.sh
failures=1

verify-stress-mtp2-985pct-timeout900.log

Running STRESS / boundary test against http://localhost:8034
  model=gemma-4-31b  container=vllm-gemma-4-31b-google-qat-w4a16-tp4  engine=vllm
  This script does the heavy stuff (longctx needle ladder + ~25K-token tool prefill).
  For the fast functional smoke (~2 min), use verify-full.sh instead.

[1/8] Long-context needle small rungs (10K / 30K) ...
    ✓   9674 tokens: recalled 'crimson chinchilla 98' (got: crimson chinchilla 98 )  prefill=983.0 t/s (10s)
    ✓  28871 tokens: recalled 'amber falcon 89' (got: amber falcon 89 )  prefill=1054.6 t/s (27s)
  ✓ all long-ctx depths recalled secret correctly
[2/8] Tool response prefill OOM (~25K-token mock tool response) ...
  ✓ tool prefill OK — text response (668 chars, finish=stop)
[3/8] IDE-agent one-shot prompt (sys + tool schemas + user request) ...
  ✓ IDE-agent one-shot OK — 21 completion tokens (0 chars), finish=stop
[4/8] Multi-turn agent prompt (sys + tools + 4-turn history) ...
  ✓ multi-turn agent OK
[5/8] LCB-coding shape (LeetCode-style problem + structured plan) ...
  ✓ LCB-coding shape OK
[6/8] Reasoning-heavy (math problem + max_tokens=8192) ...
  ✓ reasoning-heavy OK — 1881 completion tokens
[7/8] Long-context needle large rungs (60K / 90K — Cliff 2 territory) ...
    ✓  57673 tokens: recalled 'silver capybara 50' (got: silver capybara 50 )  prefill=984.3 t/s (59s)
    ✓  89674 tokens: recalled 'sapphire capybara 82' (got: sapphire capybara 82 )  prefill=912.8 t/s (98s)
  ✓ all long-ctx depths recalled secret correctly
[8/8] Context ceiling ladder (staggered NIAH from ~257000 → ~0.985 × n_ctx) ...
    n_ctx=262144  ladder: 257000 → 258211 (2 rungs)
    calibrated: scale=100 → 6417 tokens (tok/scale_unit=64.17)
    VRAM free (ladder start): 1213 MB
    ✓ rung 1/2: target=257K  actual=256K tok (97%)  recalled 'turquoise falcon 90'  prefill=449.8 t/s (570s)  VRAM_free=1213MB
    ✓ rung 2/2: target=258K  actual=257K tok (98%)  recalled 'amber narwhal 37'  prefill=410.5 t/s (627s)  VRAM_free=1213MB

  ✓ ceiling ladder: all 2 rungs passed — fillable to 257544 tok (98% of n_ctx=262144)
    VRAM: 1213 → 1213 MB (Δ -0 MB across ladder, margin threshold=1024 MB)

All stress / boundary checks passed. KV-cache and prefill paths are sound for the deployed config.

vision-smoke.json

{
  "id": "chatcmpl-a1f0d8ab36d13bd7",
  "object": "chat.completion",
  "created": 1785261542,
  "model": "gemma-4-31b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "A blue circle is centered on a red background.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": 106,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.25.1-tp4-d107f69b",
  "usage": {
    "prompt_tokens": 292,
    "total_tokens": 303,
    "completion_tokens": 11,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "metrics": null
}