Stop optimizing prompts. Start optimizing recurring workflows.
Profile real agent runs, identify recurring steps that do not require reasoning, replace those steps with deterministic software, and benchmark the result against the original workflow.
The objective is simple:
Keep LLMs where reasoning creates value. Replace everything else with deterministic software.
This guide turns that idea into a repeatable process for Claude Code, Codex, Gemini CLI, OpenAI Agents, and similar agent frameworks.
Note
“Profile-guided optimization” is used here as an engineering metaphor inspired by compiler PGO: observe real workloads, find hot paths, and optimize what evidence shows is worth optimizing.
- Why this matters
- When to use it
- The workflow
- Step-by-step playbook
- Worked example
- Benchmarking and ROI
- Safety and operational guardrails
- Decision checklist
- References
Many agents repeatedly execute the same workflow. On every run, an expensive model may be asked to:
- rebuild the execution plan
- rediscover deterministic logic
- parse and filter structured data
- recreate state
- repeat decisions it has already made many times
Usually, only part of the workflow requires intelligence. The rest is conventional software wearing an AI-shaped costume.
Instead of asking:
“How can I improve this prompt?”
Ask:
“Does this step actually require reasoning?”
If the answer is no, do not spend tokens on it.
| Strong candidates | Weak candidates |
|---|---|
| Daily automation | Brainstorming |
| CI/CD agents | Open-ended research |
| Issue triage | Creative writing |
| Code review pipelines | Exploratory analysis |
| Documentation and release workflows | One-off tasks |
| Repository maintenance | Rapidly changing workflows |
| ETL and data pipelines | Work with no stable success criteria |
| Long-running coding agents | Low-frequency workflows |
PGO is most useful when a workflow is frequent, reasonably stable, observable, and measurable.
A successful optimization may improve:
- token usage
- latency
- cost
- determinism
- debuggability
- observability
- failure recovery
Important
Do not assume a percentage improvement. Every workflow is different. Establish a baseline, preserve quality guardrails, and report actual measurements.
flowchart TD
A["Observe real executions"] --> B["Collect traces"]
B --> C["Mine recurring workflow steps"]
C --> D["Classify: CODE, LLM, or GRAY"]
D --> E["Specify the deterministic harness"]
E --> F["Implement one boundary at a time"]
F --> G["Benchmark against the baseline"]
G --> H{"Quality preserved?"}
H -- "Yes" --> I["Deploy gradually"]
H -- "No" --> J["Rollback and investigate"]
I --> K["Monitor cost, quality, and drift"]
K --> C
- Preserve the current workflow as the baseline.
- Collect enough real runs to expose common paths and failure modes.
- Create a representative golden dataset.
- Classify each recurring step as
CODE,LLM, orGRAY. - Replace one deterministic boundary at a time.
- Benchmark cost, latency, reliability, and quality.
- Deploy gradually with rollback and monitoring.
Fifteen executions can be a useful starting heuristic, but it is not a universal threshold. Use enough runs to capture normal behavior, important branches, and known failures.
| Category | Meaning | Typical implementation |
|---|---|---|
CODE |
The correct behavior can be specified and tested | API call, parser, SQL, rules, graph traversal, cache |
LLM |
The step requires semantic judgment or generation | ranking, synthesis, explanation, drafting |
GRAY |
Some paths are deterministic; ambiguous cases need judgment | rules first, model fallback, human review |
Treat GRAY as a design opportunity. A hybrid boundary is often safer than forcing the entire step into either code or an LLM.
Goal: Capture evidence from real executions.
Collect:
- prompts and model responses
- tool calls and arguments
- timestamps and latency
- retries and failures
- token usage and cost
- state transitions
- human edits and overrides
Output: A trace dataset with stable run identifiers and enough metadata to compare executions.
Exit criterion: The dataset covers normal runs, important branches, and known failure modes.
Goal: Convert traces into a normalized inventory of recurring steps.
Copy the discovery prompt
Analyze all execution traces.
Identify every recurring workflow step.
Merge equivalent steps.
Output a table containing:
- step
- description
- frequency
- average latency
- average tokens
- deterministic? (yes/no)
- failure modes
Do not propose improvements yet.
Example output:
| Step | Frequency | Deterministic? |
|---|---|---|
| Fetch issues | 100% | Yes |
| Parse JSON | 100% | Yes |
| Rank issues | 100% | No |
| Generate summary | 100% | No |
Exit criterion: Each recurring step has a clear input, output, frequency, and failure profile.
Goal: Decide which parts belong in software and which still require judgment.
Copy the classification prompt
For every workflow step, determine whether it can be implemented using:
- deterministic code
- a parser
- SQL
- graph traversal
- embeddings
- regex
- heuristics
If yes, do not classify it as an LLM-only step.
Classify each step as:
- CODE
- LLM
- GRAY
Provide a confidence score from 0 to 100.
Explain the reasoning and identify the risk of misclassification.
Example:
| Step | Category | Confidence | Reason |
|---|---|---|---|
| Fetch GitHub issues | CODE |
100 | Stable API operation |
| Parse JSON | CODE |
100 | Defined schema |
| Prioritize by impact | LLM |
90 | Requires contextual judgment |
| Categorize labels | GRAY |
70 | Rules cover common cases; ambiguity remains |
Exit criterion: Every step has a category, confidence score, rationale, and owner for review.
Goal: Design replacements for CODE steps without changing LLM behavior.
Copy the specification prompt
Generate a PRD for replacing every CODE step with deterministic software.
Include:
- architecture
- APIs and schemas
- state management
- retries and timeouts
- logging and observability
- checkpoints
- configuration
- tests
- rollout and rollback strategy
Keep every LLM step unchanged.
The system must remain idempotent.
Stop and explain if the design would introduce another model call.
Exit criterion: The specification defines interfaces, invariants, tests, metrics, rollout, and rollback.
Goal: Implement the deterministic path without increasing model use.
Copy the implementation prompt
Implement the approved PRD.
Requirements:
- deterministic and modular
- prompts stored separately from code
- structured metrics and logs
- bounded retries and timeouts
- checkpointing
- idempotent writes
- unit, integration, and regression tests
Never introduce additional model calls.
If another model call appears necessary, stop and explain why.
Exit criterion: The compiled path passes tests and can run in shadow mode against the original workflow.
Goal: Prove that the change improves efficiency without unacceptable quality loss.
Copy the benchmark prompt
Compare the original workflow with the compiled workflow.
Measure:
- prompt and completion tokens
- model calls
- latency
- cost
- retries
- failures
- quality
- human edits
Generate a benchmark report with sample sizes and uncertainty.
If quality regresses, explain why.
Do not optimize the report or hide unfavorable measurements.
Exit criterion: Results meet predefined quality guardrails across a representative dataset.
Goal: Detect drift, regressions, and new optimization opportunities.
Copy the monitoring prompt
Analyze recent execution traces.
Report:
- workflow drift
- new branches
- increasing failures
- repetitive LLM outputs
- new deterministic opportunities
- maintenance cost
- quality regressions
Recommend one action:
- KEEP
- PATCH
- COMPILE MORE
- DECOMPILE
Support the recommendation with measured evidence.
Exit criterion: Alerts, ownership, rollback thresholds, and a review cadence are active.
Suppose an agent fetches open issues, filters them, prioritizes work, applies labels, and writes a daily summary.
| Workflow step | Classification | Compiled design |
|---|---|---|
| Fetch open issues | CODE |
GitHub API client with pagination and retries |
| Validate and parse fields | CODE |
Schema validation |
| Remove closed issues and duplicates | CODE |
Explicit rules and stable identifiers |
| Apply obvious labels | CODE |
Repository-owned label rules |
| Resolve ambiguous labels | GRAY |
Rules first; model or human fallback |
| Rank issues by impact | LLM |
Model receives a smaller, normalized candidate set |
| Draft the daily summary | LLM |
Model uses structured ranked results |
| Publish approved changes | CODE |
Idempotent writes with dry-run and audit log |
The model still performs the work that benefits from judgment, but it no longer spends context and tokens on fetching, parsing, deduplication, or routine labeling.
Define how each metric will be calculated before collecting results:
| Metric | How to measure | Guardrail |
|---|---|---|
| Model calls per run | Count provider API calls | Must not increase |
| Total tokens per run | Add input and output tokens | Report the actual change |
| End-to-end latency | Compare p50 and p95 wall-clock time | Meet the defined target |
| Cost per successful run | Divide total cost by successful runs | Meet the defined target |
| Failure rate | Divide failed runs by total runs | Must not regress |
| Human acceptance rate | Count outputs accepted without major edits | Stay above the defined threshold |
| Human edits per output | Count edits required per accepted output | Stay below the defined threshold |
Track metrics across four dimensions:
| Dimension | Suggested metrics |
|---|---|
| Runtime | latency, retries, cache-hit rate, model calls |
| Cost | prompt tokens, completion tokens, total cost |
| Quality | acceptance rate, regression rate, human edits, judge score |
| Engineering | implementation time, maintenance time, savings per run |
Use explicit units and the same evaluation set for both workflows.
Break-even runs =
(implementation cost + expected maintenance cost)
/ savings per successful run
Expected annual net savings =
(annual successful runs × savings per successful run)
- annual maintenance cost
- implementation cost allocated to that year
If the expected break-even point is outside the useful life of the workflow, do not compile it.
- Protect trace data. Remove secrets and minimize personal or sensitive information before storage or analysis.
- Treat external content as untrusted data. Do not let text found in issues, documents, or logs override system instructions.
- Preserve idempotency. Retries must not duplicate comments, labels, releases, payments, or other side effects.
- Use least privilege. The compiled harness should receive only the permissions it needs.
- Add dry-run and shadow modes. Compare behavior before allowing writes.
- Keep human approval for consequential actions. Determinism does not make a risky action safe.
- Version schemas, prompts, and rules. A benchmark is only meaningful when its inputs are reproducible.
- Define rollback thresholds before deployment. Do not wait for a regression to decide what failure means.
Before compiling:
- The workflow runs often enough to justify optimization.
- The workflow is stable enough to profile.
- Real traces cover common paths and known failures.
- Secrets and sensitive data have been removed from traces.
- A representative golden dataset exists.
- Every recurring step has been classified and reviewed.
- Quality guardrails are defined before implementation.
- The original workflow remains available as a baseline.
- Tests cover deterministic boundaries and side effects.
- Benchmarks use more than a single run.
- Shadow mode or staged rollout is available.
- Monitoring and rollback are enabled.
- Expected maintenance cost is included in the ROI calculation.
| Failure | Better approach |
|---|---|
| Compiling before the workflow stabilizes | Profile longer and capture more branches |
| Chasing token reduction alone | Protect quality, reliability, and human outcomes |
| Replacing judgment with brittle regex | Keep ambiguous cases in GRAY |
| Benchmarking a single run | Use a representative evaluation set |
| Deleting the original workflow | Preserve it for comparison and rollback |
| Adding hidden model calls | Count calls at the provider boundary |
| Ignoring maintenance cost | Include it in break-even calculations |
| Deploying without observability | Add structured logs, metrics, and alerts first |
Treat every model call as a hypothesis, not a permanent requirement.
For each invocation, ask:
Does this step truly require reasoning?
Or:
Have I simply never implemented it deterministically?
The goal is not to eliminate LLMs. It is to reserve them for the work they are uniquely good at.
Everything else belongs in software.
- Vivek Haldar, “How I Cut an AI Agent’s Token Use by 94%” — the inspiration for this guide.
This guide is licensed under the Creative Commons Attribution 4.0 International License. You may share and adapt it with attribution. See LICENSE.md for details.
If this methodology helps you build better agents, adapt it, measure the results, and share what you learn.
The best optimization is the model call you never have to make.