Mellum 2 Create Tool Bench Report skill

bench_rerport.md

name	bench-report
description	Generate a benchmark report from Prometheus metrics and model output files for a given session_id

You are generating a benchmark report for a sidecar multi-model evaluation run.

The session_id for this report is: $ARGUMENTS

Follow every step in order. Do not skip steps. Do not guess values ? use only data returned by the tools. Do not infer, calculate, or hallucinate values not present in the data. If a value is missing, write N/A or 0. Do not repeat any profile in the Per-Model Analysis section ? each profile appears exactly once. You must write the report with the write or edit tool. Do not put the final report in your chat response.

---

Step 1 ? Query Prometheus for metrics

Prometheus is running at http://localhost:9090.

All metrics are stale (sidecar is a short-lived process). Use last_over_time(...[72h]) for every metric. All metrics carry a session_id label ? filter directly with {session_id="$ARGUMENTS"}.

Run each curl command below using bash. Copy each command EXACTLY as written ? do not rewrite, reformat, or change the quoting. After each command, write the result immediately before continuing to the next. Do not batch ? one command, one result, then next.

1a. Tool calls per profile and tool name

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(tool_calls_total{session_id="$ARGUMENTS"}[72h])'

1b. Tool errors per profile and tool name

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(tool_errors_total{session_id="$ARGUMENTS"}[72h])'

1c. Guard blocks per profile

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(guard_blocks_total{session_id="$ARGUMENTS"}[72h])'

1d. Guard coercions per profile

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(guard_coercions_total{session_id="$ARGUMENTS"}[72h])'

1e. LLM roundtrips per profile

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum by (profile) (last_over_time(llm_requests_total{session_id="$ARGUMENTS"}[72h]))'

1f. Total LLM time (seconds) per profile

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum by (profile) (last_over_time(llm_request_duration_sum{session_id="$ARGUMENTS"}[72h]))'

1g. Input tokens per profile

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum by (profile) (last_over_time(llm_tokens_input_total{session_id="$ARGUMENTS"}[72h]))'

1h. Output tokens per profile

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum by (profile) (last_over_time(llm_tokens_output_total{session_id="$ARGUMENTS"}[72h]))'

---

How to read the Prometheus JSON response

Each query returns JSON in this shape:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": { "profile": "qwen3.5-9b", "tool.name": "bash", "session_id": "..." },
        "value": [1748123456.789, "3"]
      }
    ]
  }
}

• result is an array ? one entry per time series (per label combination).
• metric contains the labels (profile, tool.name, session_id, etc.).
• value[1] is the metric value as a string ? convert to number when computing.
• Empty result array means no data for that metric in this session (value = 0).

If jq is available, extract values like this:

echo "$response" | jq -r '.data.result[] | [.metric.profile, .value[1]] | @tsv'

---

Step 2 ? Find and read model output files

The benchmark script will generate a file called /tmp/${ARGUMENTS}.txt ? read this file. The remaining files are the model-written output files ? one per model. Read each one.

For each file, score the model on these 5 criteria:

#	Criterion	Pass condition
T1	Glob	## Task 1 section is non-empty and lists file paths
T2	Grep	## Task 2 section has file:line:content entries
T3	Read	## Task 3 lists OpenTelemetryProbe tool counter fields and explains guard-block observation mapping
T4	Bash	## Task 4 contains verbatim git log --oneline -5 output (5 commit hashes)
W	Write	The output file exists (proves the model called the write tool)

Score = sum of passed criteria (0?5).

After scoring, validate the run outcome:

• If the expected report file /tmp/${ARGUMENTS}.txt exists, mark report_written = yes.
• If the file is missing /tmp/${ARGUMENTS}.txt, mark report_written = no and say the model only summarized in chat.
• Record whether the benchmark output file exists, whether the benchmark completed, and whether any tool errors or guard blocks were present.

---

Step 3 ? Write the report

Write the following markdown to /tmp/report-$ARGUMENTS.md:

# Sidecar Benchmark Report ? Session `$ARGUMENTS`

**Date:** <today's date>
**Models:** <comma-separated list of profiles found in Prometheus data>
**Task:** 5-task codebase audit ? glob, grep, read, bash, write

---

## 1. Performance

| Profile | LLM Rounds | LLM Duration (s) | Input Tokens | Output Tokens |
|---|---|---|---|---|
| ... | | | | |

(LLM Rounds = llm_requests_total. LLM Duration = llm_request_duration_sum. Tokens from 1g/1h.)
(If token data is 0 or missing, note "N/A ? stream_options not active for this session".)

## 2. Tool Usage

### 2.1 Calls per Tool

| Profile | glob | grep | read | bash | write | Total | Errors |
|---|---|---|---|---|---|---|---|
| ... | | | | | | | |

(Sum tool_calls_total per tool per profile. Sum tool_errors_total for Errors column.)

### 2.2 Guard Activity

List any profile with guard_blocks > 0 or guard_coercions > 0.
State which tool triggered the guard and the count.
If none: "No guard blocks or coercions in this session."

## 3. Output Quality

| Profile | T1 Glob | T2 Grep | T3 Read | T4 Bash | Write | Score |
|---|---|---|---|---|---|---|
| ... | ?/? | ?/? | ?/? | ?/? | ?/? | /5 |

For each task column, add a short note if the model passed but with a quality issue
(e.g. "? relative paths" or "? noisy ? included docs/").

## 4. Per-Model Analysis

For each profile: one paragraph covering what worked, what failed, and any anomalies.
Note if the model violated profile instructions (e.g. used relative paths, wide grep scope).

## 5. Recommendations

For each profile that scored < 5 or had tool_errors > 0:
- What failed
- Specific instruction to add or strengthen in its bundled `.toml` profile

## 6. Run Validation

| Check | Result |
|---|---|
| Benchmark output written | yes/no |
| Report written | yes/no |
| Benchmark completed | yes/no |
| Tool errors | <N> |
| Guard blocks | <N> |
| TPS / duration present | yes/no |

Use the write tool. If write returns file_already_exists, use edit to replace the whole file content. Do not wrap the report in a fenced code block. After the write/edit tool succeeds, respond with only one sentence naming the written path.

---

Do not add commentary outside the report structure. Do not guess values not present in the data. If a Prometheus query returns empty results for a metric, record it as 0 or N/A and continue. Do not finish until the write or edit tool has successfully written the report file.