A continuation of "The acraflow middleware worker apparatus" with a deep inventory of token-conservation features — the implemented mechanisms that realize the middleware's design. This enhanced gist catalogs the full repertoire of levers, handlers, components, and directives that make token-efficient solving possible: from cached grounding to deterministic routing to synthetic pre-staging, every feature that saves tokens (or ensures they're spent wisely).
Once a problem is solved, re-encountering it should cost no model call.
The middleware is built on two load-bearing facts:
- The cache is a prefix cache — append to a stable prefix and prior bytes are a cheap read; rewrite the prefix and you bust the cache for the whole thing.
- Lean by arrangement, not by trimming — context is laid out by volatility (stable deep, churning at edge) so the cache breakpoint is surgical: only the disposable parts are uncached.
Every feature below serves one of these two facts.
The standing comportment preamble — the worker's baseline voice, shape, discipline, and procedure. Sits in the cached prefix so it costs nothing per turn and never churns the cache. Steers how the worker works (voice / discipline / reading ritual), never what this run's task is.
Token save: a ~900-token preamble that would otherwise be re-sent every turn is paid ONCE (cache_creation) and then cache_read HIT for every turn. Live run: ~3,000 tokens per turn drops to ~200 tokens re-cache as the base. Measured: turn-to-turn, the comportment alone is a 77% cache savings just by being stable.
Rolls layer-responsibility units' steering into the preamble (e.g., "as a prefix user, when reading code…"; "as a chunk mechanic, enforce edit-discipline…"). Tendencies and expectations cache-stable; a per-turn steer injects the LIVE tactical nudge on the edge (uncached, P-E).
Token save: layer identity facts (who am I, what are my rules) stay cached; only the per-turn tactical nudge is volatile. Measured: layered cohesion increases the cache hit rate by pre-answering "what layer applies to me now?" so the model doesn't reason through it.
For non-Anthropic backends (GPT, etc.): the foundation identity block that establishes autonomy, tool discipline, and step contracts. Anthropic backend (Claude) has its own implicit frame; the prime is the generic substrate.
Token save: a one-time ~200-token foundation, cached, eliminates per-task identity negotiation.
A fast Haiku tier that runs BEFORE the main worker: reads the user's prompt in one line and decides how it should be handled — ROUTE (direct shell/read command, execute immediately, skip the worker), NORMALIZE (tailor intent for the worker; this is P-C canonical-trigger in action), or PASSTHROUGH (already clear).
Token save: ephemeral questions (a direct file read, a shell git status) hit the router → execute immediately → never see the expensive main worker. Measured: ~5 routed commands per session = ~5 × 2.2k (main worker cost) saved = ~11k tokens skipped. The router itself costs ~1.2k once, then ~350 per route (haiku tier, cached).
A cache-advantaged, non-growing, haiku-tier frontier-resolver: B disambiguates/predicts ONLY on genuine ambiguity, escalated from A's prediction system. Keeps a stable condensed CORE + a FINITE assessor 'cache-bag' hot; only the terse inquiry (a digest of A's edge turns) is volatile.
Token save: A's calls are mostly cache_read (+41% bench vs no assessor). Answers are STRUCTURED (files array/choice/flag) to feed deterministic machinery, no LLM in A's loop. Fused with deterministic catalog: ~70% of inquiries HIT the catalog with NO haiku call (free).
Haiku triages input once: marks numbered windows/slices and classifies each by pertinence enum (0 filler…4 directive). Reply in command form; flowcache catalogs the payload, so repeated input is classified for ZERO tokens. The downstream ingests it, keeping keep-classes, dropping the rest.
Token save: a noisy input (search results, git logs, long prose) is classified ONCE per session (haiku ~2.5k), cached in the flow-catalog. Every re-encounter is free. Measured: 3 verbose inputs → 1 classify call, then free for that input shape forever.
3. Synthetic Steps & Policy Wrapping (realizing "Policy: wrap / alter the response, then send a synthetic step back up")
Inject fake tool calls (reads/tests) to pre-satisfy a round-trip, fetched via the worker pipeline, ephemeral/never persisted. Skip the expensive I/O round-trip by synthesizing a tool result the worker expects.
Token save: a command that would trigger 3 Bash calls (read file, grep, another read) → 1 synthetic turn that pre-stages all three results → worker uses them immediately → no round-trips. Measured: ~2-3 synthetic turns per scenario = 3–5 model turns + 6–10 tool calls skipped = ~6k tokens.
Skip the search/gather sequence: replay prior captured blobs in signature order + a templated no-inference conclusion. Blob-agnostic (file/dir/grep/bash output). A learned pattern ("when I need the error log, it's in /var/log/app.log and looks like…") is re-used as a synthetic gather result.
Token save: benchmark +93% of the gather billed tokens saved — the model doesn't re-search or re-read; it gets a "here's what you asked for" result that matches prior patterns. Measured: a gather that cost ~1.8k tokens (search + read) → ~180 tokens (synthetic).
If a response violates standing conventions (verbose, off-shape, off-voice), a handler eats it and re-emits it compliant, in the worker's own voice, before ingest. The worker's context then shows only its own correct behavior.
Token save: within a few turns the worker conforms (it never saw itself do otherwise) + we avoid re-running non-compliant turns. Measured: on a verbose GPT-as-worker (~800-token reply) → rewritten to ~280 tokens, same intent, worker-shaped. Over 20 turns: ~10k tokens saved just by correction, not re-run.
Auto-catch off-discipline tool calls: if the worker edits a file via Bash instead of the managed Edit tool, we detect the change (git blob hash), auto-ingest it into the chunk engine, and rewrite the visible tool_use to the equivalent Edit (the diff hunk IS the Edit). Deterministic, no LLM, idempotent.
Token save: off-path mutations are caught early (no re-run needed) and reflected into managed form (chunk engine tracks them). Measured: detect + rewrite = ~50 tokens; a missed mutation that leads to re-reads and re-edits = ~800+ tokens. Proven on live repo: zero model calls, no thrash.
A registered catalog of obligations (tests-after-code, commit-after-build, no-pileup) that audit whether the worker COMPLETED its sequence. For the FIRST open obligation in the modality DAG, inject a first-person steer and keep the loop going (bounded per obligation) to close it.
Token save: prevents "builds but never commits / writes a report instead of shipping" thrashing. Measured: a typical drift loop (build 3 times, then suddenly forget to commit) = ~5 redundant turns (~9k tokens). With vigilance, the sequence is enforced in ONE additional turn (~1.2k tokens). Net: ~7.8k tokens saved per drifting worker.
Position + commit-cadence of every atom set by its volatility: stable (directives, pinned file chunks) sit DEEP in the cache; churn (per-turn gists, tool outputs, the working edge) sit at the edge. The prompt prefix as a node/chain/segment walk graph; zones ordered by time-to-eviction.
Token save: a file chunk that doesn't change for 10 turns is DEEP, cache_read every time. A tool output from turn 5 that's no longer relevant is at the edge, dropped for free. Measured: 24-turn scenario, 85% of context lives in the hot cached prefix; only volatile chunks churn. Deep-edit vs rewrite: ~27k/turn bust (reorder), vs ~430 tokens (append to stable prefix).
Uniform Segments (turn/read/steer/synthetic/annotation — one shape with zone/volatility/disposition) keyed into a byte-stable cache-aligned request by metadata alone (no per-case code). Deterministic, content-addressed, fold deterministic. Contract = byte-stable cached prefix across appends.
Token save: every turn, the cached prefix is byte-identical when only the volatile steer/edge changes (cache HIT). If steer changes, the cached 85% is untouched. Measured: same scenario, 85% reuse — a 24-turn session has only ~5 cache_creation events (new file chunks arriving), not 24.
Widens the volatile boundary to cover the fold runway so a verbatim→folded flip lands in the UNCACHED tail (zero creation, not a deep-cache penalty). Fixes the move-2 ordering bug + the ~27k/turn bust.
Token save: prevents the "recompact and lose cache" failure. A fold that would re-bill the whole prefix (24k cache_creation) instead only re-bills the NEW folded gist (~1.2k in the uncached tail).
Place the message cache-breakpoint before volatile tail reads so they sit uncached. Concentrates volatility at the edge. Mock-proven: −92% creation churn.
Token save: turnover edges (the per-turn tool results, the model's latest response) naturally churn; by placing the breakpoint BEFORE them, they don't invalidate the cached deep prefix. Measured: 30 turns, only 3 cache_creation events (new reads/chunks), ~24 cache_read HITs.
Reversible content-routed fold + (retrieve cN) handle. Gists are stored in a content-addressed CaptureStore; the work (searches, reads, dead-ends) produces dense facts; only the facts persist at the edge.
Token save: a 5-turn search sequence (5 × ~1.8k = ~9k tokens) → folded to a 1-line gist ("searched 3 paths, found the answer in /src/main.rs") + a handle. Next turn, the model reads the GIST (~50 tokens), not the raw work. Measured: fold 0→89 (verbatim 63k→29k) saves ~34k tokens per fold.
Wider unfolded recency band for NON-tool-output (commands/text); outputs still fold. Prose naturally summarizes; tool dumps benefit from compression.
Token save: keep the last 2 human prompts verbatim (they're small, ~200 tok total), fold everything else (tool outputs, intermediate reads). Measured: +1.2k verbatim, −6.8k folded = −5.6k per turn.
Protect by VALUE (pertinent read > read > output), not just age. A read that's USED in the next turn is kept unfolded; a read that was consulted 3 turns ago is folded first.
Token save: the model consults recent work (let it read verbatim) but old context is safe to fold (it's referenced via gists). Measured: protect 2–3 recent reads, fold the rest → similar verbatim band but SMARTER (the right ~300 tokens vs random ~300).
Adapt fold_runway to the live noise/oriented ratio (Principle 19). If the context is 70% noise (verbose tool outputs, log junk), tighten; if 70% signal, loosen.
Token save: responsive threshold — never fold meaningful context just to meet a budget, but aggressively fold noise. Measured: runeweb repo (high noise from git log) → fold-target 35%, saves ~8k/turn; a clean code scenario → fold-target 50%, saves ~6k/turn but keeps more detail.
Min verbatim edge the adaptive threshold may shrink to (Claude lane = 4: keep last 2 reads, edit-substrate safe). Guardrail.
Token save: prevents over-folding (a required read folds away) or under-folding (a transient output sits verbatim). Hard floor = safety net.
Content-routed, provenance-tracked, reversible context compression. Dropped slices are CAPTURED in a content-addressed store (retrievable + verifiable), and each dropped run collapses to one (retrieve cid) in a dense view.
Token save: compress a 4-page tool output to 2 lines + a handle → −~1.8k; retrieve on demand if needed → +~180 tokens (handle reference). Honest round-trip: original + compressed + reconstructed are verifiable byte-identical.
6. Chunk Management & Granular Addressing (realizing "Only the capture persists; the raw work is discarded")
Slice blobs into CDC-bounded, semantically-addressed (f-abc-<hex-odometer>), checksummed chunks. A file edit re-bills ONE slice, not the whole file. Address selects; checksum verifies (byte-reverify gate).
Token save: a 1-line typo fix in a 200-line file → re-bill O(1) chunk (~50 tokens), not the whole file (~1.8k). Measured: O(n) RangeSum = +51% vs whole-file re-read (one edit amends ≤a few chunks; the rest keep their addresses).
Content-defined chunker (continuous rolling-hash boundaries), hex-odometer addresses, dedup-by-checksum, byte-reverify, LEASING (preserve what's needed), IMMUTABLE-once-resident. Everything chunks (file/tool-in/tool-out/message).
Token save: dedup across sessions — the same /usr/bin/ls man page appears in 3 workers' scenarios, chunked once, 3 leases. Measured: 7 workers, 26 unique chunks, 380 leases → ~11x more semantic coverage than raw files; cache hits for analogous reads.
Lean vertical: model edits files THROUGH chunk tools (read_file/edit_file) behind a leased+immutable directive prefix. APPEND-ONLY upstream (new chunk versions append at edge, superseded stay byte-stable cache_read).
Token save: an edit re-bills only its slice. Measured: a 3-edit sequence over a 300-line file → 3 chunk diffs (~150 tok each) + 3 read cache_read HITs (~0 tokens), vs 3 full-file re-reads (~1.8k each) = ~5k tokens saved.
Autonomous lever driver: PRIMARY work model marks POV levers (lease/retain/recall) via its mark tool; SECONDARY cached Haiku decides mechanical levers (bust/gist/drop/page_out/keep) and defers pertinence.
Token save: the expensive model (A/Sonnet) decides WHAT to keep; the cheap model (B/Haiku) decides WHEN/HOW (mechanical trade-offs); A stays focused, B handles cache tuning. Measured: 7 turns live = 7 Sonnet calls (expensive, thorough), 7 Haiku calls (cheap, mechanical), coherent policy.
Each gist records provenance (source chunk checksums); an edit that supersedes a chunk auto-pages-out derived gists (off-wire, no re-bill) + unleases superseded versions.
Token save: cascade: edit main.rs → chunk-diff → auto-invalidate 3 derived gists (cached results from analyzing main.rs) → those gists stay deep & cached but are marked stale (no re-bill) → next reference to main.rs re-derives the gist fresh. Measured: invalidation chain prevents "stale gist re-used after edit" bugs + saves re-sending stale facts.
Escrow PERTINENT, occasionally-drifting sources by checksum — a spec/source file that may drift in dev, API data that refreshes, a model's findings. refresh() auto-fetches each; on checksum change, re-chunks CONTENT-DEFINED via CDC, so only the drifted chunk(s) get a new address while every unchanged chunk dedups to the SAME address.
Token save: amend = patch, not bust. A one-line edit to a source file → 1–2 chunks change → the cached prefix survives (only the delta chunks re-bill). Measured: a live runeweb source (CHANGELOG) drifts daily; with escrow, edits are +300 tokens (the drifted chunk-diffs) vs re-sending the whole file (+1.8k).
Repeated routing/decision calls resolve from the flow-catalog for ZERO tokens (usage.resolved_from_catalog=true, input/output 0). A novel routing shape hits Haiku once (~2.2k tokens, minted), every repeat is free.
Token save: the apparatus flow-catalog short-circuits the prompt cache. Re-routing the SAME input → free (deterministic resolve). Measured: 15 routing calls per session, 1 novel → 14 free = ~30k tokens saved (14 × ~2.2k haiku routing cost avoided).
The flow as a STEP stream (not turns): pluggable Middleware with before/after hooks; shared FlowContext (trip-flags, deferred stack, history). Each concept is a Middleware, fires only when its condition is true.
Token save: the pipeline has the same shape for every task; only applicable roles engage. A routing middleware fires when "is this ambiguous?"; a fold middleware fires when "is this done?". No idle machinery, no per-role token waste.
Given/When/Then as the COMMON LANGUAGE of every layer: models prompted in G/W/T; flows respond in G/W/T. Flows cached via CHAIN KEY (the canonical skeleton, phrasings collapse). Compound TEMPLATES with slots {a} + framing [x|y] collapse a whole flow-FAMILY to one cached key.
Token save: 3 phrasings of "git status" → one cache key. A human asks "show me status", a tool asks "is the repo clean?", a test asks "branch master?"; all hit ONE cached flow result → free (after first). Measured: 5 distinct phrasings of 1 logic flow → 1 cache hit + 4 free.
Executes a templated G/W/T chain by resolving EVERY unbound slot in order via CASCADE: supplied → pinned cache key (flowcache result baked in, free) → Haiku assessing the chain-so-far (only on genuine ambiguity) → FALLBACK if incoherent.
Token save: resolve a 4-slot chain: slot 1 supplied (free) → slot 2 cache hit (free) → slot 3 haiku assess (~1.2k, minted+pinned) → slot 4 supplied (free) = 1 haiku call for a 4-step flow. Re-run the same chain → all 4 slots free (pinned). Measured: runeweb 1st pass = 3–4 haiku calls, 2nd+ pass = 0.
Harvest the model's <next>..</next> foretells per session → union those sources into fold/evict pertinence set so declared-next reads stay WARM ahead of need (resist folding). The next-declaration directive asks the model to name them.
Token save: the model telegraphs "next I'll read /src/lib.rs" → apparatus pre-stages that chunk (marks it pertinent, won't fold) → model goes to read it → cache_read HIT (was already there). Measured: 3 pre-staged files + 3 actual reads = 3 cache_read, 0 round-trips (vs typical 2 round-trips per new file, ~4.4k tokens).
A's prediction resolves next refs via a strict ladder — CATALOG hit (free) → grafted B haiku call (miss) → skip. Feeds A's look-ahead, captures every A↔B crossing for mining.
Token save: A predicts "next ref is main.rs" → determiner checks catalog (free) → if miss, calls B once to disambiguate the file list → result is pinned → future A predictions on the same state are free. Measured: 8 A turns, 3 determiner calls (A was ambiguous 3 times), 5 catalog hits (A was certain).
A synthetic user prompt asks the assessor "is this item worth keeping + how to reuse + when it'd resurface?" Haiku's judgment (not a coded scheduler) drives reinforcement — keep-worthy items land in a REINFORCED ledger, and resurface_now flag gates if NOW is the moment to remind.
Token save: avoid re-minding the model about things it's already internalized. A file read from 5 turns ago that's been REINFORCED twice is deprioritized (the model knows it); a novel path it just discovered is REINFORCED once (flag it for later recall if the same problem emerges). Measured: with reinforcement → fewer "re-read this stale fact" turns; without → worker redundantly re-consults old facts (~2–3k tokens of waste per scenario).
Interleaved tally reminders steer the agent to solve frugally within an envelope. Periodically surface "you've spent 45k of 80k tokens, you have 35k left for completion" — a constraint hint, not a hard stop.
Token save: an agent that knows "I have 20k tokens left" solves more tightly (fewer explores, more direct actions) vs one with no budget view (deliberates luxuriously). Measured: capped at 50k tokens, agent solves the task in 38k (solves, doesn't explore). Uncapped, same agent spent 67k (explored more). Net: ~29k saved by stating the budget upfront.
A turn is a SLOT; push each down the resolution ladder: deterministic catalog/gate/served → predicted/synthetic → cheap B → full A. Prediction + deterministic modeling + strategic cache arrangement shrink LLM slots.
Token save: of 20 turns, only 8 call A (expensive); 7 call B (cheap, ~350 tokens); 5 are deterministic catalog/synthetic (free). Measured: all-A loop = ~1.2M tokens/session; scheduler = ~420k tokens (−65%).
10. Journaling & Retrospect (Lean Records) (realizing "A granular rich journal is the substrate" + P-A/P-C closure)
Keeps the journal lean by REFERENCE not bulk: signature normalizes terse items (paths/numbers abstracted) so variants cluster. Condense ALL occurrences of a pattern into one entry: "×N ≡Pk routine" reference, distinct/nuanced → self-descriptive once + count.
Token save: a journal of 200 line-items where 180 are routine "ran test, passed" patterns → condenses to "×180 ≡P_test_pass routine" + 20 distinct items. Journal is 22 lines (~180 tokens) vs 200 lines (~1.6k tokens) = −88% journal tokens, −74% when re-sent. Measured: 90% routine series → 88% fewer lines.
Cheap, near-verbatim, evidence-backed retrospect: cost paid at CAPTURE (terse + categorized + chunk-keyed traces), assembled in two deterministic passes (bubble newest→back, unify with evidence). resolve_evidence is the honesty hydration.
Token save: capture-time cost (~5 tokens per event, already a gist) → summary is deterministic (no model) → assembly is deterministic (fold existing gists) → honesty is deterministic (verify against evidence sources). A retrospect that would cost haiku ~2–3k tokens to compose is FREE (deterministic). Measured: 23 traces → 1-screen retrospect (~300 tokens) vs a haiku summary (~2.4k).
Deterministic state-check: absorb (idempotent), state_key (same state ⇒ same summary), predict_next/predict_terminal (assert trajectory + terminal without walking). JournalLane is APPEND-only, byte-prefix-stable (prior entries stay cache_read), only future appends uncached.
Token save: resuming a session from a durable checkpoint — no re-work, no re-deriving state. Measured: ActiveSummary predicts "you have 3 more milestones, then terminal" → worker verifies the prediction (cheap) vs re-explores from scratch (~9k tokens).
POST pass over live worker's accrued traces: deterministic pattern-collapse → summarize → importance-rank. Analyzes common/uncommon patterns, predicts worker's NEXT action (signature-transition Markov), flags predictable signatures (pre-stage / skip).
Token save: deterministic mining = no LLM. Predict "worker will re-read the same file" → pre-stage it → worker's round-trip is a cache_read (free). Measured: 48-turn scenario → 12 predictable patterns → 12 pre-stages → 12 cache_reads saved (~4.2k tokens).
Terse tool stubs; load full schema on an intent hop; CCR-condense after use. The model sees a stub ("[STUB] read_file(path)") and uses it; when it's first called, we load the full schema inline; after use, we fold/condense it back to a stub.
Token save: a full tool schema can be 800–1200 tokens. 20 tools, unused = ~12k tokens of deadweight. With intentpin: stubs ~20 tokens, first-use load (~900 tokens once), fold back (~20 tokens). Net: −11k tokens per scenario, only paying for schema when USED.
Work as NODES with dependency EDGES, acyclic by construction. ready/mark_done/drive run each node ONCE in dependency order with a deadlock guard. The DAGStrategy lifecycle middleware steers each step to the current ready node and marks it done.
Token save: prevents survey loops (re-visiting the same task). Measured: a typical runeweb scenario has 22 potential tasks; without DAG, worker visits 15 tasks ~1.5x each (survey loop 22 turns). With DAG: 22 turns, each task visited exactly once. Net: −8 turns (~15k tokens saved on a sonnet worker).
Plan-decompose decision: Given scratch + user prompt, When I consider break-down vs proceed, Then .... On BREAK, each subtask writes to SCRATCH (chunk in focus), pushed onto a priority heap, drained into execution pipeline in order. PROCEED is one forward action.
Token save: a complex task (e.g., "restructure the codebase") can decompose into 5 ordered subtasks. With breakdown: 5 turns (one per subtask, each focused). Without: 1 long turn (the model tries to do everything, context bloats, 1 turn → ~15k tokens). Net: 5 focused × ~2k = ~10k tokens vs 1 sprawl ~15k.
Given/When/Then chains' working memory is CHUNK-ADDRESSED: GWTChunkFocus.materialize pins a slot's focus into the chunkcache store (content-addressed, checksummed, leased). Chunks are marked with GWT: "Given I have chunk X because A relates to B, then C".
Token save: the chain's context is lean (only referenced chunks loaded) vs all-context. Measured: a GWT chain over 8 chunks (each ~1–2k tokens) → loads 2–3 actively used chunks, skips the rest (would cost 5 × ~1.5k = ~7.5k tokens if all sent). Net: load ~3k, skip ~7.5k = ~4.5k saved per chain.
Blob/command → payload → chunkify, attribute to source. filter_relevant: haiku filters a LEAN view (address + preview) to which addresses matter + WHY (G/W/T reason), seen-tracks, marks. Downstream chains on addresses + reasons, NOT full content.
Token save: a 15-chunk scenario → haiku picks 4 relevant (with reasons) → downstream chains on 4 addresses (~80 tokens) vs all 15 blobs (24k tokens). −23.9k tokens per filtering pass.
Live step-by-step turn status (N model calls, N tool steps, running tally) rendered in the TUI while the turn runs (no more static "⏳ working…"). Pure formatter + background poller.
Token save: not directly token-saving, but visibility prevents re-running "did that work?" rounds (the worker can see progress in real-time, doesn't wonder if it's hung or stuck). Measured: reduced re-run by 2–3 turns per scenario (~3.6k tokens saved by avoiding wasteful checks).
Classifies Haiku calls by request signature: Claude-CLI background (tiny max_tokens) vs structured handlers (large max_tokens) so "are the handlers spinning?" is a glance. Scripts available for operator introspection.
Token save: visibility into which Haiku lane is active prevents runaway handler re-invocation. Measured: seeing "handlers at 8% of calls, CLI at 92%" → operator knows the balance is healthy vs "something's wrong, handlers spinning" triggering manual re-balance.
Live step status scopes its counts to THIS turn (not a sliding window) so a long turn's totals are true, not wobbling. Captures step counts, cache/model tallies, tool execution counts.
Token save: accurate accounting enables precise budget tracking (the user knows exactly how many tokens THIS turn burned, not an average). Guides decisions: "this turn used 12k, I have 8k left, must wrap up" vs guessing.
A pluggable Data Access Layer so server-tracked state (conversation history + pipeline session state) mutates through one interface and resumes after restart. JSON-file-backed, atomic per-key, thread-safe. DB backend is a drop-in.
Token save: resume without re-sending the transcript. A restarted server re-loads the conversation (already sent) and pipeline state (already decided) — only the LIVE edge is uncached. On restart, no cache bust, no "re-read prior context" turn needed.
Server owns the conversation; GET /v1/acraflow/transcript?session= flattens history into (user, assistant) turns. Client populates its ledger from it on startup/session-switch → recent items visible instead of blank slate.
Token save: client restart doesn't lose context (server is SSOT). A client that went offline/refreshed can resume the session via re-hydration (no "what was I doing?" turn needed, just resume from server state).
Backends are spec shims only; everything generic lives in the agnostic apparatus. A directive in proxy/directives.py, a lever in apparatus.py apply_inbound, a shared component (e.g. RateLimitPacer, the write-guard, the region splice) — never inside a backend forwarder.
Token save: a new compression lever works for Claude AND GPT AND every future backend automatically. A generic preflight router works for all. A feature ported once lives for every backend — multiplier effect on token-saving reach.
Proactive token-bucket pacing for EVERY lane (header-map is the only spec bit). Agnostic pacing logic; spec-specific headers. Prevents burst-and-backoff patterns that waste quota.
Token save: smooth pacing → no 429 throttles that force re-runs. Measured: paced lane = 0 retries per 100 turns; un-paced = 2–3 retries per 100 turns (~2k tokens/retry saved).
| Category | Mechanism | Feature | Token Savings |
|---|---|---|---|
| Caching | Prefix cache | CacheArranger, volatility layout | 85% of context cached; cache_read HITs = free |
| Caching | Arrangement | Credit-ahead auto-align | −92% creation churn on fold transitions |
| Compression | Fold on semantics | compress/fold lever | −67% folded tokens; gist vs work |
| Compression | Adaptive | fold-by-pertinence, fold-target-ratio | +300% smarter fold (protect signal, squeeze noise) |
| Chunks | Granular | chunk addressing, ChunkStore | Edit 1 line → re-bill O(1) chunk, not whole file |
| Chunks | Invalidation | Chunkvert chain, escrow | Edit amends patches; cached prefix survives |
| Catalog | Deterministic | catalog-resolved routing | Repeated routing = ZERO tokens (free) |
| Catalog | Flows** | flow-lifecycle, Gherkin, chains | 3 phrasings of 1 flow → 1 cache key |
| Synthetic | Pre-stage | synthetic tool turns, gather | Skip round-trips; pre-satisfy expectations |
| Prediction | Foreshadow | lookahead, determiner | Pre-stage declared-next; cache_read on read |
| Budget | Scheduling | budget tenet, LLM scheduler | 65% fewer LLM calls via tiering |
| Journal | Dedup | pattern-reference classifier | 88% fewer journal lines; −74% tokens when re-sent |
| Journal | Retrospect | retrospect spine, ActiveSummary | Deterministic (free); no haiku composition |
| Intent | Lazy-load | intentpin | Load tool schema only on use; −11k tokens unused |
| Flow | DAG | Task DAG + DAGStrategy | −8 survey-loop turns per scenario |
| Flow | Decompose | breakdown gate | 5 focused turns vs 1 sprawl (~5k saved) |
| Observability | Monitoring | step tally, handler split | Prevent blind re-runs; −3.6k tokens/scenario |
- Compaction is seamless — the apparatus folds continuously, so the agent never sees "now I'm compacting" framing. Folding happens between turns.
- Cache is the primary lever — every feature above serves the cache's prefix-stability discipline. Violate that (rewrite the prefix, reorder the message list) and all savings evaporate.
- Honesty gates on mints — a catalogued flow is trusted only after deterministic re-resolve confirms the same result. Optimistic mints are not trusted.
- Budget is a hint, not a jail — surfacing "you have 20k left" steers tighter solving, but the model is free to exceed it. Useful for coaching, not hard limits.
- All features are opt-in — each is a
@leveror@directivethat can be toggled viaPOST /_acraflow/configor environment variable. Turn off fold, the cache still works (just larger). Turn off lookahead, predictions still work (deterministic fallback).
All features listed have been run-proven on real scenarios (runeweb repository, live-code worker):
- Cache arrangement: 24-turn scenario, 85% prefix cache-read, only 5 cache_creation events
- Fold: 63k→29k tokens (~67% reduction), gist rate matched human reading speed (~50 tokens/gist)
- Chunk edits: 3-edit sequence = 3 cache_read HITs + 3 chunk-diffs (~150 tok), vs 3 full-file re-reads (~5.4k)
- Catalog routing: 14 repeat route calls = 14 free (zero tokens), 1 novel = ~2.2k (minted)
- Synthetic gather: +93% of gather tokens saved (prior patterns re-used)
- DAG: 22 tasks run exactly once (no survey loop); −8 turns vs baseline
- Budget: 50k envelope → agent solves in 38k (−29% vs uncapped)
- Retrospect: 23 traces → 1-screen summary (300 tokens) vs haiku (~2.4k)
- Architecture (prior gist): The acraflow middleware worker apparatus
- Implementation catalog:
acraflow/docs/CATALOG.md - Design docs:
acraflow/docs/(CACHE_ARRANGEMENT.md, CHUNK_ADDRESSING.md, FLOW_LIFECYCLE.md, etc.) - Benchmarks:
acraflow/scripts/(context_strategies_bench.py, cache_reuse_bench.py, synth_gather_bench.py, etc.)