| name | model-config-optimizer |
|---|---|
| description | Optimize LLM model assignments across any agent harness using log-based usage analysis, pricing research, and ROI-driven selection. Prevents naive "upgrade everything" cost explosions. |
| category | system-optimization |
| version | 1.3.0 |
Optimize which LLM model each agent in a harness uses. Balance cost vs. capability using actual usage data rather than assumptions.
- New models are available and you need to update assignments
- Costs are unexpectedly high and you need to cut spend
- You want data-driven justification for premium model usage
A model that is 5× more expensive per token must reduce invocations by 80% to break even. Most harnesses do not have 80% waste. The optimizer finds where premium models actually pay for themselves.
The break-even math above is a simplification. Before accepting it, challenge the following assumptions for your specific harness:
| Assumption | Why It Might Be Wrong | What to Check |
|---|---|---|
| Token cost is the only cost | Latency, error rates, downtime, and engineering time also matter | Does a cheaper model fail more often, requiring human intervention? |
| Premium models reduce invocations | They may produce longer outputs (reasoning, detailed plans), increasing tokens per call | Compare average output tokens between models for the same agent |
| All invocations are equal | A cheap model may fail on hard tasks, forcing a fallback to premium anyway | Check fallback trigger rates by task complexity |
| The goal is cost minimization | The real goal may be speed, accuracy, or user satisfaction at acceptable cost | What is the business cost of a wrong answer or slow response? |
| Cheaper models are actually cheaper | High failure rates cause retries, cascading errors, or manual rework | Measure end-to-end task success rate, not just API cost |
| Historical waste predicts future waste | A new model may change the type of work, not just reduce waste | Are you solving harder problems now than last quarter? |
| Invocation count is the right metric | Total tokens, wall-clock time, or user-perceived latency may matter more | Does your billing model charge per token, per call, or per minute? |
If any of these assumptions do not hold for your harness, the cost model will mislead you. Adjust the model accordingly before proceeding.
The optimizer is a tool, not a rule. Override it when:
- A task has high business impact and low tolerance for error (e.g., security review, compliance checks)
- Speed is the constraint — a premium model may be worth 5× cost if it cuts user wait time from 30s to 5s
- The cheap model cannot complete the task — capability floor matters more than cost ceiling
- You are experimenting — a short-term premium assignment to establish a quality baseline is valid
| Input | Source |
|---|---|
| Current model assignments | Harness config files |
| Per-token pricing | Provider pricing pages |
| Agent invocation counts | Harness logs / telemetry |
| Session hierarchy | Logs showing parent-child agent relationships |
Find all agent roles the harness defines. Sources:
- Config files (JSON/YAML/TOML with agent/model mappings)
- Log entries tagged with agent identifiers
- Session metadata naming agents
What to extract:
- Agent name/role identifier
- Current model assignment
- Invocation count (from logs)
- Whether it spawns subagents (has child sessions)
- Whether it is spawned by another agent (has parent sessions)
Use invocation patterns and session hierarchy to classify each agent:
| Classification Signal | Role Archetype |
|---|---|
| Highest invocation count, spawns many subagents | Orchestrator |
| Spawned by orchestrator, runs discrete tasks | Executor |
| Searches code/docs, returns results | Search |
| Creates plans, roadmaps, or architectures | Planner |
| Reviews output, flags issues, approves/rejects | Critic/Review |
| Investigates errors, proposes fixes | Debugger |
| Autonomous multi-step tasks | Deep Worker |
| Generates prose, docs, or creative content | Writer |
Log heuristics:
- Titles containing "review", "audit", "check", "assess" → Critic
- Titles containing "find", "search", "locate", "map" → Search
- Titles containing "plan", "design", "architecture" → Planner
- Titles containing "fix", "debug", "investigate" → Debugger
- Multiple identical titles in one session → possible waste/duplicate
Count message=stream, message=completed, or equivalent completion events per agent. Rank by volume — the top 20% of agents typically drive 80% of cost.
⚠️ Critical: Pricing changes frequently. Do not use prices from memory, documentation, or previous runs. Fetch current pricing from official sources every time.
Retrieve input and output cost per 1M tokens for every candidate model from official sources:
- OpenAI: https://openai.com/pricing
- Anthropic: https://www.anthropic.com/pricing
- Azure OpenAI: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/
- Google: https://cloud.google.com/vertex-ai/generative-ai/pricing
- Moonshot/Kimi: https://platform.moonshot.ai/pricing
- Any other provider in your harness
What to capture:
- Model name (exact string used in API)
- Input price per 1M tokens
- Output price per 1M tokens
- Context window tier (some models have different prices for different context lengths)
- Currency (convert to a single currency)
Before building the cost model, verify:
- Prices are from today (or within the last 7 days)
- All models in the current config have confirmed prices
- All candidate replacement models have confirmed prices
- Context-window pricing is captured (e.g., GPT-4o vs GPT-4o-128k may differ)
- Prices are in the same currency
If a model has no published price, exclude it from the cost model or flag it as "pricing unknown — estimate only."
Group validated prices into tiers:
| Tier | Avg Cost | Models | Use For |
|---|---|---|---|
| Nano | ~$0.10–$1.00 | Smallest/cheapest available | Search, simple execution |
| Standard | ~$2–$8 | Mid-tier generalists | Orchestration, general work |
| Premium | ~$15–$30 | Best reasoning models | Complex planning, debugging, review |
| Ultra | ~$30+ | Specialized (creative, long-context) | Specific high-value tasks only |
If prices have changed significantly since last optimization, re-run the full cost model. A 50% price drop on a premium model may make previously rejected upgrades viable.
Use archetype defaults if no telemetry available:
| Archetype | Input | Output |
|---|---|---|
| Orchestrator | 15K | 5K |
| Planner/Debugger | 10K | 5K |
| Critic/Review | 8K | 4K |
| Deep Worker | 12K | 6K |
| Executor | 5K | 2K |
| Search | 3K | 1K |
| Writer | 4K | 2K |
If logs contain token counts, use those instead.
Before using the cost model for decisions, sanity-check it:
- Compare total modeled cost against actual billed cost for the last 7 days
- If the model is off by >25%, investigate: wrong token estimates? wrong pricing? missing agents?
- Check that the highest-cost agent in the model matches the highest-cost agent in actual spend
If the model does not approximate reality, fix the inputs (token estimates, pricing, or invocation counts) before proceeding.
cost_per_call = (input_tokens × input_price / 1M) + (output_tokens × output_price / 1M)
total_cost = invocations × cost_per_call
Calculate for current assignments and each candidate model.
From logs, calculate:
waste_rate = (rework_sessions + duplicate_calls) / total_subagent_sessions
Rework signals: Titles containing retry keywords ("re-review", "fix", "retry", "v2", "v3", etc.) Duplicates: Identical titles spawned by the same parent session.
Typical harnesses: 10–20% detectable waste.
break_even_reduction = (1 - old_cost / new_cost) × 100
| Condition | Verdict |
|---|---|
break_even_reduction ≤ waste_rate + 20% |
Safe — waste elimination alone could justify it |
break_even_reduction > waste_rate + 20% but agent has high iteration potential (>0.30) |
Plausible — premium reasoning may reduce legitimate rework |
break_even_reduction > 50% and agent volume > 1000 |
Reject — volume makes the multiplier catastrophic |
| Agent is Search/Executor archetype | Nano tier — never upgrade regardless of capability |
roi = capability_gain / cost_increase
Capability gain is the relative improvement in model tier (e.g., Standard→Premium = +2 tiers).
Start from current config. Add upgrades in ROI order until budget is reached.
Budget recommendation: ±10% of baseline cost.
Hard constraints:
- Top 20% by volume: reject >2× cost multipliers
- Search/Executor archetypes: always cheapest tier
- Orchestrator: keep cost-effective unless spawn rate is provably reducible
Every primary model needs 1–2 fallbacks cascading from premium → standard → nano. This prevents outages from blocking the harness.
- Update config files
- Verify syntax/schema validity
- Monitor for 2–4 weeks:
- Actual cost vs. model prediction
- Subagent spawn rate (orchestrator efficiency)
- Review loop frequency (critic re-invocations)
- Fallback trigger rate
| Agent Volume | Cost Multiplier | Iteration Potential | Verdict |
|---|---|---|---|
| High (>1K) | >2× | Any | Reject — volume dominates |
| Any | >2× | <0.15 | Reject — not enough reducible waste |
| Low (<200) | <2× | >0.30 | Accept — good ROI |
| Medium | 1.5–2× | 0.20–0.35 | Evaluate — depends on historical waste |
| Search/Executor | Any | ~0 | Nano tier — never upgrade |
❌ Use stale pricing — "I looked up prices last month, they're probably the same."
Result: A 50% price drop on a premium model may make it viable; a 2× price increase may make a current assignment untenable. Always fetch current prices. The entire cost model is invalid if pricing is wrong.
❌ Upgrade by tier — "All agents get Premium because it's the best model."
Result: 5× cost increase on high-volume agents. Break-even requires eliminating 80% of calls. Almost never happens.
❌ Ignore logs — "This model is smarter, so it will use fewer tokens."
Result: Historical waste is typically 10–20%. Models cannot eliminate legitimate parallel work without losing capability.
❌ Skip fallbacks — Primary model has no fallback chain.
Result: Rate limits or outages block the entire harness. Always provide 1–2 cascading fallbacks.
❌ Assume the agent stack — Hardcoding agent names from one harness to another.
Result: Misses harness-specific roles and misclassifies iteration potential. Always discover the stack from logs/config.
| Metric | How to Measure | Target |
|---|---|---|
| Cost per session | Total cost / number of sessions | Within 10% of model prediction |
| Subagent spawn rate | Subagent calls / orchestrator calls | Stable or decreasing |
| Review loop rate | Critic re-invocations / total critic calls | <20% |
| Fallback trigger rate | Fallback invocations / total invocations | <5% |
Version: 1.3.0