Skip to content

Instantly share code, notes, and snippets.

@gcavalcante8808

gcavalcante8808/sidecar_session_debugger.md

Last active June 9, 2026 01:22
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save gcavalcante8808/7be2c5e9220fd6ecb7106100b8a4cb93 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save gcavalcante8808/7be2c5e9220fd6ecb7106100b8a4cb93 to your computer and use it in GitHub Desktop.
Download ZIP
Sidecar Session Debugger
Raw
sidecar_session_debugger.md

You are debugging a sidecar session. The session_id to investigate is: $ARGUMENTS

Follow every step in order. Do not skip steps. Do not guess ? use only data from the tools.

Step 0 - Read the report generated by the model

The report would be located on the /tmp/report-${ARGUMENTS}.md. This report is created by the bench-report skill and is a summarization of /tmp/${ARGUMENTS}.txt.

Read both and prepare the correlation of the data with the later sections.

Step 1 ? Read the TransactionLog

The transaction log has all step-by-step generated by the harness, it is verbose.

Read the session log that is at: $HOME/.local/share/sidecar/sessions/$ARGUMENTS.jsonl

Each line is a JSON event. Event types:

  • UserMessage ? user input (text)
  • AssistantMessage ? model response (text + optional tool_calls array)
  • ToolCall ? tool invoked (name, args)
  • ToolResult ? tool output (name, output)

If the file does not exist, report "TransactionLog not found ? session may predate BENCH_SESSION_ID fix or used a different UUID."

Parse and list every event in order using this format:

[N] TYPE  name/preview

Example:

[0] USER  "You are auditing the sidecar codebase..."
[1] TOOL  glob  path=sidecar-core/src/adapters
[2] RES   glob  "sidecar-core/src/adapters/profile/mod.rs..."
[3] ASST  tool_calls=1  ""
[4] TOOL  write  path=/home/.../bench_outputs/gpt-oss-20b.md
[5] RES   write  "wrote 42 lines to ..."
[6] ASST  tool_calls=0  "I have completed all tasks."

Instead of counting the lines yourself, you can use the following command to do so:

jq -r 'to_entries[0].key' $HOME/.local/share/sidecar/sessions/$ARGUMENTS.jsonl | sort | uniq -c

Step 2 ? Query Prometheus

Prometheus is at http://localhost:9090. Use last_over_time(...[72h]) for stale metrics.

Run each command exactly as written. One command, one result, then next.

Prometheus normalizes OpenTelemetry attribute dots to underscores:

  • tool.name becomes tool_name
  • guard.reason becomes guard_reason
  • llm.interface becomes llm_interface
  • model.id becomes model_id
  • server.host becomes server_host

2a. Tool calls total

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(last_over_time(tool_calls_total{session_id="$ARGUMENTS"}[72h]))'

2b. Tool calls breakdown

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(tool_calls_total{session_id="$ARGUMENTS"}[72h])'

2c. Tool errors total

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(last_over_time(tool_errors_total{session_id="$ARGUMENTS"}[72h]))'

2d. Tool errors breakdown

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(tool_errors_total{session_id="$ARGUMENTS"}[72h])'

2e. Guard blocks total

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(last_over_time(guard_blocks_total{session_id="$ARGUMENTS"}[72h]))'

2f. Guard blocks breakdown

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(guard_blocks_total{session_id="$ARGUMENTS"}[72h])'

2g. Guard coercions total

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(last_over_time(guard_coercions_total{session_id="$ARGUMENTS"}[72h]))'

2h. LLM roundtrips total

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(last_over_time(llm_requests_total{session_id="$ARGUMENTS"}[72h]))'

2i. LLM roundtrips breakdown

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=last_over_time(llm_requests_total{session_id="$ARGUMENTS"}[72h])'

2j. LLM failures total

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(last_over_time(llm_failures_total{session_id="$ARGUMENTS"}[72h]))'

If Prometheus returns empty results for all queries, note "No Prometheus data ? session may have run before session_id label fix."

Step 3 ? Correlate and diagnose

Cross-reference the TransactionLog sequence with Prometheus counts. Identify:

  1. Where the session stopped ? what was the last event? Did the model respond with tool_calls=0 (text response instead of tool call)?
  2. Tool errors ? which tool returned an error or guard block? What was the exact error message from the ToolResult?
  3. Missing steps ? which expected tool calls never happened (e.g. write never called after all tasks completed)?
  4. Prometheus vs log discrepancy ? do Prometheus counts match the log event count?
  5. Artifact validation ? if the session is a benchmark or report run, confirm whether the expected output file was actually written and whether the wrapper reported a missing file.

Before writing the diagnosis, compute this reconciliation table using the prometheus provided data:

Check How to compute
log_tool_calls Use the exact jq count command from Step 1
log_tool_results Use the exact jq count command from Step 1
prometheus_tool_calls Numeric value returned by query 2a
prometheus_tool_errors Numeric value returned by query 2c
prometheus_guard_blocks Numeric value returned by query 2e
prometheus_guard_coercions Numeric value returned by query 2g
prometheus_llm_requests Numeric value returned by query 2h
prometheus_llm_failures Numeric value returned by query 2j

If log_tool_calls and prometheus_tool_calls differ, report both values and explain the likely cause using the breakdown labels such as job, profile, and tool_name. Do not invent a merged total.

Step 4 ? Write the debug report

Write the report to: /tmp/debug-$ARGUMENTS.txt

Use the write tool. If write returns file_already_exists, use edit instead.

Structure:

# Session Debug Report ? `$ARGUMENTS`

**Date:** <today>
**Log path:** ~/.local/share/sidecar/sessions/$ARGUMENTS.jsonl
**Total events:** <N>

## Event Timeline

| # | Type | Detail |
|---|---|---|
| 0 | USER | ... |
| 1 | TOOL | ... |
...

## Prometheus Summary

| Metric | Value |
|---|---|
| tool_calls_total | |
| tool_errors_total | |
| guard_blocks_total | |
| guard_coercions_total | |
| llm_requests_total | |
| llm_failures_total | |

## Reconciliation

| Check | Value |
|---|---|
| log_tool_calls | |
| log_tool_results | |
| prometheus_tool_calls | |
| prometheus_tool_errors | |
| prometheus_guard_blocks | |
| prometheus_guard_coercions | |
| prometheus_llm_requests | |
| prometheus_llm_failures | |
| Match? | yes/no + explanation |

## Diagnosis

### What worked
<list>

### What failed / stopped
<explain the last event and why the session did not complete>

### Root cause
<one paragraph ? be specific: error message, guard reason, empty result, text-mode response, etc.>

## Validation

| Check | Result |
|---|---|
| Benchmark/report output written | yes/no/N/A |
| Session completed | yes/no |
| Tool errors | <N> |
| Guard blocks | <N> |
| TPS / duration available | yes/no/N/A |
| Metric/log mismatch | yes/no + brief reason |

## Recommendations

<specific profile instruction or prompt change to fix the identified issue>

## Skill Execution Verdict

| Check | Result |
|---|---|
| Session completed | yes/no |
| Report written | yes/no/N/A |
| Tool errors | <N> |
| Guard blocks | <N> |
| Metric/log mismatch | yes/no |
| Instruction-following issues | <list or none> |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment