Skip to content

Instantly share code, notes, and snippets.

@sanchez314c
Created July 4, 2026 19:39
Show Gist options
  • Select an option

  • Save sanchez314c/e17660f10b21ef5b107b37a023237ca0 to your computer and use it in GitHub Desktop.

Select an option

Save sanchez314c/e17660f10b21ef5b107b37a023237ca0 to your computer and use it in GitHub Desktop.
Fable 5 vs Opus 4.6 benchmark + two-mode agent routing playbook

I benchmarked Fable 5 against Opus 4.6, then blew my monthly quota anyway. Here's the routing playbook that came out of it.

Boys and girls. This is not that complicated. But nobody's writing it down, so here it is.

Two things happened this week. First I ran a real benchmark... Fable 5 at every effort level against Opus 4.6 on max effort, same coding task, blind audits. Second, I let a 116-agent fleet loose on a refactor job and burned 1.6 million tokens in under 4 minutes. Hit my monthly spend cap mid-flight. 75% of my Fable quota gone with 5 days left before reset.

Both taught the same lesson. The model is not the strategy. The ROUTING is the strategy.

The benchmark

One task: "chronosort," a TypeScript dependency-aware task scheduler. DAG cycle detection, topological sort, critical path calculation, parallel wave grouping. 6 source files, 15+ tests, 9 edge cases. Same 3,458-char prompt for every run, each in its own isolated directory, launched headless with claude -p --output-format json. Wall clock, cost, turns, and token counts captured straight off the JSON. Then every run got audited on a 7-dimension rubric (architecture, algorithm correctness, edge cases, error messages, test quality, output format, TypeScript quality) by a separate context. Never let a model grade its own homework. It gets sycophantic the second it does.

Run Actual model Effort Wall Cost Tests Quality /10 Logic bugs
Fable low claude-fable-5 low 186s $2.48 20 8.4 0
Fable medium claude-fable-5 medium 210s $2.60 22 8.6 1
Fable high claude-fable-5 high 248s $2.94 28 9.0 0
Fable xhigh claude-fable-5 xhigh 320s $3.24 33 9.1 0*
Opus 4.6 claude-opus-4-6 max 138s $0.80 22 7.7 1
Opus 4.8 claude-opus-4-8 max 770s $3.61 36 9.1 0
Fable max fable-5 + opus-4.8[1m] max 822s $5.86 42 9.4 0

*Fable xhigh's one flaw was a missing barrel file in package.json, config not logic.

Read that table again. Opus 4.6 on MAX effort was the fastest and cheapest run on the board. 138 seconds. 80 cents. It was also the lowest quality with a real logic bug: its input guard was typeof task.duration !== 'number' || task.duration <= 0... which waves NaN right through, because typeof NaN === 'number' is true and NaN <= 0 is false. NaN propagates into Math.max() and corrupts every wave duration downstream. Every single Fable run validated with Number.isFinite(). Opus 4.6 didn't. It also ran topological sort twice and discarded the first result.

Three findings that actually matter:

  1. The effort ladder has a cliff, and it's medium to high. Fable medium: 8.6 with a bug. Fable high: 9.0, zero bugs. Below high you're getting code that needs human review for correctness. High and above is production-ready. That threshold is worth more than any single benchmark score.

  2. Fable high is the value play. 9.0/10, zero bugs, $2.94, 4 minutes. That replaced Opus 4.6 max as my daily coding default the same night.

  3. Fable max is a trap. $5.86, slowest run at 822 seconds, and the transcript shows it spun up claude-opus-4-8[1m] as a SECOND model under the hood. Dual-model, which is why it costs double. If you're paying for Opus anyway, just use Opus. And don't burn xhigh or max on Fable without a specific reason... xhigh matched Opus 4.8's quality at 42% of the wall time, but high is the sweet spot.

The alias trap (this one WILL get you)

The first "Opus" run in that benchmark wasn't 4.6. I asked for Opus 4.6 max. The run went out as --model opus, and that alias resolves to the LATEST Opus. Which is 4.8. Caught it in the results JSON, re-ran with --model claude-opus-4-6 explicit, kept both rows in the table.

Two days later it got me AGAIN. Two subagents spawned with model "opus" through the agent framework. Both came up 4.8 in the transcripts. I don't run 4.8. It's all over the place, unstable, and I banned it from my entire stack. Didn't matter. The alias walked it right back in through a different door, and this time there was no settings pin behind it to catch the fall.

If you ban a model, the ban is only as good as the aliases you didn't check. Grep everything. Pin exact model IDs everywhere a model gets named: claude-opus-4-6, never opus. Single-version aliases like haiku and fable are safe today and a liability the day a new version ships. The fix that actually holds is an agent definition with the exact ID pinned in frontmatter (opus-worker.md, included below), because then there is exactly one place the model gets resolved and it can't drift.

The quota incident

The refactor job: 58 skill files. My orchestrator spawned one refactor agent per file plus one adversarial reviewer per file. 116 agents, high effort, ~16 concurrent. 1.6 million tokens in under 4 minutes. Monthly spend cap. Dead stop, mid-fleet.

Nothing was wrong with any individual agent. The architecture was wrong. One agent per small file means every agent pays the full context setup cost to do 3 minutes of work. Batching 5-8 files per agent does the same job with 85% fewer agents.

So now it's law in my config, not a lesson I hope I remember:

  1. Batch 5-8 items per agent. Never one agent per small file.
  2. Effort ceilings per seat. Workers cap at high. Reasoning seats default to medium. Max is for irreversible decisions only.
  3. Any fan-out over 10 agents requires a token estimate against remaining quota BEFORE launch.
  4. The expensive model gets at most ~10% of any fleet.

The playbook: two modes, five seats

Here's the part you can steal. Quota is a real constraint. Pretending it isn't means your best model is unavailable exactly when you need it. So the system has two launch modes and the SEATS never change, only who drives.

MODE 2, Fable primary. When the weekly quota is fresh. The strongest reasoner holds the orchestrator seat because orchestration mistakes are the most expensive mistakes in an agent system... one bad dispatch decision multiplies into six figures of wasted subagent tokens. My 116-agent incident was an orchestration mistake, not a worker mistake.

MODE 1, Opus 4.6 primary. When Fable quota runs hot (~50%+) or limits loom. Opus 4.6 on high drives, and Fable gets called out surgically for the moments that need actual thought. You keep most of the benefit at a fraction of the burn.

The seats, in both modes:

Seat Model + effort What it does
REASONER Fable 5, medium architecture, spec review, assessment, adjudication. Genuine thought only
WORKER Opus 4.6, high coding, refactors, file ops, reads, writes, docs
EXTRACTOR Haiku 4.5 grep sweeps, status checks, verification reads. Misfires are cheap and obvious
ESCALATOR Fable 5, high audit-and-fix when a worker's output fails validation
AUDITORS GLM 5.2 + GPT 5.5 independent blind code audits, cross-family on purpose

Why Fable at MEDIUM in the reasoner seat when the benchmark crowned Fable high? Because the benchmark was a coding task and coding goes to workers now. For pure reasoning, a stronger base model at medium effort beats a weaker one at max, and it spends fewer thinking tokens doing it. The table above shows Opus 4.6 burning max effort to land at 7.7. Effort doesn't fix judgment.

The auditors are GLM 5.2 (OpenCode subscription, $87/mo) and GPT 5.5 (Codex, $100/mo). Both flat-rate. Marginal cost per audit: zero. That changes how you think about code review entirely. You stop rationing reviews and start throwing them at everything.

Here's why the headless cross-family setup matters so much for anything you actually care about shipping.

GLM and GPT run as headless CLI processes. opencode run --variant max for GLM, codex exec --ephemeral for GPT. They spin up in their own isolated contexts, audit the code blind (no knowledge of each other's findings, no knowledge of who wrote the code or why), and dump structured output. Your primary Claude session never sees their raw context. No window pollution, no token bleed, no "here's what the other reviewer said" confirmation bias.

Cross-family is the real unlock. Claude reviewing Claude code is like asking your coworker to proofread the email they helped you write. They share training lineage, they share blind spots, they'll miss the same class of bugs and nod at the same plausible-but-wrong patterns. GLM was trained by a completely different team on different data with different optimization targets. GPT, same story, third family. When GLM and GPT independently flag the same line of code, that's three separate training lineages agreeing something is wrong. Those dual-flagged findings are almost always real. Single-model findings? They hallucinate more than you'd think.

The pipeline for every non-trivial code change:

  1. GLM 5.2 and GPT 5.5 run independent blind audits, parallel, headless.
  2. Two Opus instances do a 5-7 turn back-and-forth: what did the auditors find, what's real, what's noise, what should we keep from each.
  3. The Fable reasoner seat issues the final accept/reject/merge.

That's four models across three families touching every serious diff before it lands. $187/mo in flat-rate subscriptions doing the volume work, Claude reserved for the judgment call at the end. The subscriptions are already paid whether you use them or not. Not using them is leaving money on the table.

I ran this exact pipeline on the skill refactor that blew my quota. The GLM pass caught two staleness violations (wrong API endpoint, dead skill references) that both Opus instances missed. The GPT pass caught a template that would have leaked a localhost proxy address into shipped output. Neither would have survived a same-family review because the original code was written by Claude and Claude doesn't flag its own assumptions.

The part nobody told me was possible

Subagents can spawn subagents. Verified it in my own harness this week: an Opus primary spawned a Fable reasoning agent, and that Fable agent spawned its own Haiku worker and relayed the result back up.

That changes the economics completely. The pattern is: cheap primary drives the session, Fable lead gets dispatched with a mission and a batch of context, and the Fable lead runs its OWN fleet of Opus and Haiku workers. Fable tokens go to fleet sizing, defect adjudication, and final verdicts. Volume work happens two levels down on models you can afford to burn. The expensive brain directs. It doesn't type.

Files

Everything referenced is in this gist, verbatim from my running config:

  • 02-benchmark-results.md: full data, dimension scores, throughput, bug forensics
  • 03-MODEL-ROUTING.md: the routing law my agents actually load
  • 04-fable-lead.md: REASONER seat agent definition (spawns its own workers)
  • 05-fable-audit.md: ESCALATOR seat
  • 06-opus-worker.md: WORKER seat, exact model ID pinned, the alias-trap fix

Steal all of it. The model names will age out. The structure won't.

One benchmark, one task, n=1, my rubric. Run your own before you bet your stack on mine. But run SOMETHING, because "the big model on max effort" is not a strategy, and the table above is what it costs to assume it is.

Thank me later.

Benchmark: Fable 5 (all effort levels) vs Opus 4.6 max vs Opus 4.8 max

Date: 2026-07-02/03. Harness: Claude Code 2.1.198, headless claude -p --output-format json --dangerously-skip-permissions, each run in an isolated directory. Wall clock via date +%s%N wrapper. Cost/turns/tokens from the result JSON. Audits done by a separate context against a 7-dimension rubric, 10 points each.

Task: "chronosort"

TypeScript dependency-aware task scheduler. DAG cycle detection, topological sort, critical path calculation, parallel wave grouping. Required: 6 source files (types.ts, parser.ts, graph.ts, scheduler.ts, formatter.ts, cli.ts), 9 edge cases, 15+ tests on node:test. Same 3,458-char prompt for every run.

Results

Run Actual model Effort Wall (s) Cost Turns Output tokens Tests Quality /10 Bugs
Fable low claude-fable-5 low 186 $2.48 18 12,968 20 8.4 0
Fable medium claude-fable-5 medium 210 $2.60 18 14,637 22 8.6 1
Fable high claude-fable-5 high 248 $2.94 24 18,172 28 9.0 0
Fable xhigh claude-fable-5 xhigh 320 $3.24 20 26,056 33 9.1 1*
Opus 4.6 claude-opus-4-6 max 138 $0.80 15 8,501 22 7.7 1
Opus 4.8 claude-opus-4-8 max 770 $3.61 25 63,252 36 9.1 0
Fable max claude-fable-5 + claude-opus-4-8[1m] max 822 $5.86 24 44,229 42 9.4 0

*Config bug (missing index.ts barrel file breaks library consumers), not a logic bug. CLI worked.

Dimension scores (/10 each)

Run Arch Algo Edge ErrMsg Tests Output TS Total /70
Fable low 8 9 9 9 7 9 8 59
Fable medium 9 9 9 9 8 7 9 60
Fable high 9 9 9 10 8 9 9 63
Fable xhigh 9 9 9 10 9 10 8 64
Opus 4.6 max 8 8 7 9 7 8 7 54
Opus 4.8 max 9 9 9 10 9 9 9 64
Fable max (partial: Algo 10, ErrMsg 10, Output 10; other 4 dimensions not individually recorded) 66

Throughput

Run Output tok/s
Fable max 54
Opus 4.6 max 62
Fable low 70
Fable medium 70
Fable high 73
Fable xhigh 81
Opus 4.8 max 82

Bug forensics

Opus 4.6 (logic bug, NaN passthrough): parser.ts guard was typeof task.duration !== 'number' || task.duration <= 0. typeof NaN === 'number' is true, NaN <= 0 is false, so NaN passes validation and propagates into Math.max(), corrupting wave durations. Fix is || !Number.isFinite(task.duration). Also: cli.ts calls topologicalSort() and discards the result before schedule() runs it again internally, and the DFS color constants get declared twice with one raw-integer reference.

Fable medium (logic bug, parallel marker): formatter.ts gated the (parallel) marker on wave.index === 1 && wave.tasks.length > 1. Multi-task waves after Wave 1 lose their marker. The index guard shouldn't exist.

Fable xhigh (config bug): package.json main/types point at a compiled index that has no index.ts source. MODULE_NOT_FOUND for library consumers.

All five Fable runs validated numerics with Number.isFinite(). Opus 4.6 did not.

Notes

  • The run originally labeled "opus max" went out as --model opus, which resolved to claude-opus-4-8 (latest), not 4.6. Caught in the results JSON; an explicit --model claude-opus-4-6 run was added. Both kept.
  • Fable max ran dual-model (claude-fable-5 + claude-opus-4-8[1m] appear in its usage), which explains the cost and wall time.
  • Test count scaled with effort: 20, 22, 28, 33 across the Fable ladder; 22 for Opus 4.6 max; 36 for Opus 4.8; 42 for Fable max.
  • Opus 4.6 was 7.4x more concise than Opus 4.8 on output tokens (8,501 vs 63,252) for the same task.
  • All 7 runs completed. No failures, no timeouts.
  • n=1. One task, one rubric. Directional, not gospel.

MODEL ROUTING — single source of truth

Ratified by User 2026-07-04 (rev 2, post quota incident 14:59). Skills reference seats by NAME, never hardcode models.

Hard rules

  • Opus 4.8 PROHIBITED. Everywhere. No exceptions. (User verdict 2026-07-04: "horrible.") Opus means 4.6.
  • THE ALIAS TRAP (incident 2026-07-04 15:40): the Agent tool's model: "opus" resolves to Opus 4.8 — the BANNED model. There is no availableModels pin in settings.json (the 2026-07-04 morning row claiming one was written is false on disk). Therefore: NEVER pass model: "opus" to the Agent tool or workflow agent(). Opus 4.6 subagents = agentType opus-worker ONLY (pinned by exact ID claude-opus-4-6 in ~/.claude/agents/opus-worker.md). model: "haiku" is safe (resolves to Haiku 4.5, the only Haiku). model: "fable" is safe.
  • NEVER auto-fallback Fable → Opus 4.6 mid-session. Fable's 1M window dumped into 4.6 = immediate context explosion (root cause of the 2026-07-04 "4.6 overload" — it was the fallback, not 4.6-as-primary). Fable classifier trip = PAUSE, User decides.
  • Auditor seats MUST be cross-family. Never put a Claude model in an auditor seat — same-family review destroys triangulation.
  • Route volume to the flat-rate subscriptions, judgment to Claude. Fixed capacity already paid for: GLM 5.2 via OpenCode (~unlimited, $87/mo), GPT 5.5 via Codex xhigh ($100/mo).
  • FABLE CONSERVATION. Fable quota is the scarce resource (5-day reset windows; hit 75% on 2026-07-04). Fable fills ONLY reasoning seats (ARCHITECT/JUDGE/ESCALATOR), always batched. Never Fable for reads, writes, coding, file ops, or mechanical work.

Two launch modes

Mode = which model drives the PRIMARY session, selected by launch alias. Self-detect: check your own model ID at session start.

MODE 1 — DEFAULT (cc) MODE 2 — DEEP (cc-fable)
PRIMARY Opus 4.6 high Fable 5 high
Use when daily driver: orchestration, execution, everything hardest architecture/judgment sessions, Fable quota permitting
Subagent seats identical in both modes (table below) identical in both modes

Primary stays lean in both modes: orchestration + synthesis only, delegate 100%, auto-compact handles the rest.

Quota-state rule (User 2026-07-04): MODE 2 (Fable primary) is the most desirable and becomes the working default whenever the weekly quota is fresh. As Fable quota depletes (~50%+ consumed) or limits loom, drop to MODE 1 — Opus drives, Fable is called out surgically, and you keep most of the benefit at a fraction of the Fable burn. MODE 1 is the always-safe fallback, never a downgrade in structure: the seats don't change, only who drives.

Seats (identical in both modes)

Seat Model + effort Covers Invocation
ARCHITECT Fable 5 medium specs, contracts, API design, security architecture — genuine reasoning Agent tool, agentType fable-lead
JUDGE Fable 5 medium auditreview verdicts, codereview synthesis, A2A moderation, plan verification Agent tool, agentType fable-lead
ESCALATOR Fable 5 high audit-and-improve when a WORKER's output fails validation; irreversible-decision sign-off Agent tool, agentType fable-audit
WORKER / IMPLEMENTER Opus 4.6 high coding, refactors, file ops, reads, writes, docs, packaging Agent tool, model=opus
DEBATER Opus 4.6 xhigh 5-7 turn Architect-vs-Pragmatist verify loop Agent tool, model=opus
EXTRACTOR Haiku 4.5 pure-mechanical sweeps: grep/ls/status checks, verification reads, data extraction — misfires cheap + detectable Agent tool, model=haiku
AUDITOR_A GLM 5.2 independent blind codebase audit opencode run --variant max
AUDITOR_B GPT 5.5 xhigh independent blind codebase audit codex exec --ephemeral -c model_reasoning_effort="xhigh"
MECHANICAL GLM 5.2 (marginal cost zero); EXTRACTOR fallback lint sweeps, docs formatting, bulk batch analysis, boilerplate opencode headless

Rationale (User, 2026-07-04): Fable-medium beats Opus-4.6-max on reasoning while spending fewer thinking tokens; Opus-4.6-high covers all mechanical/production work; failure there is cheap to detect ("we know when it didn't get it right") and escalates to Fable-high audit.

Coding mandate — ALL coding tasks

Every non-trivial code production fires the cross-family process, not just forge phases:

  1. GLM 5.2 (opencode) + GPT 5.5 (codex) independent passes — blind, parallel.
  2. Opus ↔ Opus A2A back-and-forth (5-7 turns) to decide what to use and discard.
  3. JUDGE (Fable medium) issues final accept/reject/merge, verifying flagged files before accepting (hallucinated findings are real; dual-flagged findings are almost always real). Forge seams remain: forge:multiaudit post-build (.forge/FORGE_AUDIT_GLM.md + FORGE_AUDIT_GPT.md), forge:brainstorm Stage 2 cross-review.

Nested delegation

fable-lead / fable-audit agents carry full tools INCLUDING the Agent tool: a Fable reasoning agent spawned by an Opus primary CAN spawn its own WORKER (opus) and EXTRACTOR (haiku) subagents. Canonical pattern: PRIMARY (Opus) → fable-lead (orchestrates + judges) → opus/haiku sub-workers. Keeps Fable tokens on judgment only while it directs volume work.

Fan-out discipline (law, post 2026-07-04 quota incident: 116-agent fleet burned 1.6M tokens in 4 min)

  1. Batch: 5-8 items per agent. NEVER one-agent-per-small-file.
  2. Effort ceilings: WORKER ≤ high. REASONER seats = medium default. xhigh = DEBATER/AUDITOR_B only. max = irreversible decisions only (credential architecture, security sign-off, data-migration design).
  3. Budget gate: any fan-out >10 agents requires a stated token estimate vs remaining quota headroom BEFORE launch.
  4. Fable share of any fleet ≤ ~10% of agents.

Version notes

  • GLM: 5.2 (subscription). Anything referencing GLM-5.1 harnesses is stale.
  • Z.AI endpoint: always api.z.ai/api/coding/paas/v4, never open.bigmodel.cn.
  • Agent definitions: ~/.claude/agents/fable-lead.md (Fable medium, orchestration-capable), ~/.claude/agents/fable-audit.md (Fable high, escalation audit).
name fable-lead
description REASONER seat (MODEL-ROUTING.md). Fable 5 at medium effort for genuine reasoning — architecture, spec review, assessment, adjudication, plan verification, orchestration of sub-fleets. Can spawn its own WORKER (opus) and EXTRACTOR (haiku) subagents via the Agent tool. Use for judgment tasks only; never for mechanical reads/writes/coding — dispatch those to workers.
model fable
effort medium

You are the REASONER seat per ~/.claude/commands/forge/MODEL-ROUTING.md. You are Fable 5 — the scarce, expensive reasoning resource. Operating rules:

  1. Spend your tokens on judgment, not volume. Reading big files, grepping, writing code, moving files — dispatch to subagents via the Agent tool: WORKER = agentType opus-worker (coding/file ops/writing — NEVER model:"opus", that alias resolves to banned Opus 4.8), EXTRACTOR = general-purpose with model=haiku (grep/ls/status/verify sweeps). You orchestrate, verify, and decide.
  2. Batch worker dispatches — 5-8 items per worker, never one-agent-per-small-file. Fan-out >10 agents requires estimating token cost first.
  3. Verify before accepting: check worker returns against the original goal, not just task completion. Failed work goes back with concrete defects or escalates.
  4. Return protocol: end with DONE / DONE_WITH_CONCERNS (list) / BLOCKED (why + what's needed) / NEEDS_CONTEXT (what's missing). Your final message is a report to the orchestrator — compact synthesis, no raw payload dumps; write big artifacts to disk and return paths.
  5. Safety: never rm (AI-Pre-Trash only), never kill without exact PID, Tier-1 backup before edits.
name fable-audit
description ESCALATOR seat (MODEL-ROUTING.md). Fable 5 at high effort — audit-and-improve when a WORKER's output failed validation, and irreversible-decision sign-off (credential architecture, security, data migration). Expensive; dispatch one at a time, only after a cheaper seat has failed or the decision is irreversible.
model fable
effort high

You are the ESCALATOR seat per ~/.claude/commands/forge/MODEL-ROUTING.md — Fable 5 at high effort, the most expensive seat in the system. You are dispatched only when cheaper seats failed or the decision is irreversible. Operating rules:

  1. Root cause, not patch. You were called because a WORKER's output failed validation or the stakes are irreversible. Diagnose why it failed, fix it properly, and state what the worker got wrong so the pattern is correctable.
  2. You may spawn subagents (Agent tool: agentType opus-worker for workers — NEVER model:"opus", that alias resolves to banned Opus 4.8; model=haiku for extractors) for mechanical legwork — keep your own tokens on analysis and the fix itself.
  3. Verify programmatically before reporting. Run the check that proves the fix. No "should work."
  4. Return protocol: DONE / DONE_WITH_CONCERNS / BLOCKED / NEEDS_CONTEXT, with a compact defect analysis. Big artifacts to disk, return paths.
  5. Safety: never rm (AI-Pre-Trash only), never kill without exact PID, Tier-1 backup before edits.
name opus-worker
description WORKER/IMPLEMENTER seat (MODEL-ROUTING.md) — Opus 4.6 PINNED BY EXACT MODEL ID. Coding, refactors, file ops, reads, writes, docs, scraping, packaging. Always use this agentType for Opus work; NEVER pass model:"opus" to the Agent tool — that alias resolves to Opus 4.8, which is banned.
model claude-opus-4-6
effort high

You are the WORKER seat per ~/.claude/commands/forge/MODEL-ROUTING.md: Opus 4.6 high, the production workhorse. Operating rules:

  1. Execute exactly the task given — scope lock, nothing extra.
  2. Verify programmatically before reporting; check outcomes against the original goal, not just step completion.
  3. Return protocol: DONE / DONE_WITH_CONCERNS (list) / BLOCKED (why + needed) / NEEDS_CONTEXT (what). Compact returns — big artifacts to disk, return paths.
  4. Safety: never rm (AI-Pre-Trash only), never kill without exact PID, Tier-1 backup before edits, no credentials in returns.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment