March 2026
About the failure mode at the issue→plan boundary:
The agent does not make bad decisions at design forks. It fails to perceive the fork as a fork. Once it finds a technically valid path, it enters execution mode without noticing that alternatives exist or that the path's failure modes differ from alternatives in survival-critical ways. This is not ignorance — it is the absence of a trigger that would convert execution-mode into design-mode. The failure is a perception failure, not a reasoning failure.
The GitHub evidence confirms five failure categories: (1) plan addresses work items but not the observable goal, (2) proposed fix treated as specification rather than hypothesis, (3) single design implemented without surface-the-choice question, (4) implementation placed where it cannot be unit-tested, (5) plan designed before codebase survey. In 7 of 8 documented failures, the information needed to avoid the mistake was present in the issue body or in adjacent repository artifacts. The failures arose from what agents chose not to look at, not from information gaps.
About the root cause:
Two mechanisms cause this. First, the planning phase of kaizen-evaluate has no mandatory grounding step between "read the issue" and "draft the plan." Plan formation begins with whatever the agent's first interpretation produces. The issue body's goal is not extracted before work items are listed; existing tools are not surveyed before new abstractions are designed; the proposed fix's assumption is not named before implementation is planned. These omissions compound: a plan that misses the goal passes plan-coverage review (which checks requirements, not goals) and reaches implementation.
Second, agents lack indexed negative exemplars for this codebase. The review-findings loss incident, the OOM stop-hook incident, the cross-worktree contamination — these failures are documented in GitHub issues but not retrievable at planning time. An agent forming a plan that uses the accumulate-then-flush shape has no path from "proposed design" to "this shape has failed here before, specifically." The principle is known (state resilience matters); the indexed negative exemplar is not available.
About the solution space:
The five pre-plan questions (R4C's Phase 4.5) address 4 of 5 failure categories with zero infrastructure. They are the highest-leverage intervention. The FSI / Category Library adds indexed retrieval of past failures and transforms known failure categories into recognition-fast priors — but requires bootstrap data from real incidents and is not the right first step. The Plan Battery formalizes quality checks as enforcement gates but, per the pre-mortem, risks false confidence from text-that-looks-correct and overhead creep if not scoped precisely to blocking-worthy failures. The right architecture sequences all three but builds them in the correct order.
The gap is not informational. It is architectural: the issue→plan transformation has no retrieval phase and no structured pre-plan formation discipline. The agent reads the issue body once, linearly, and begins planning from its first interpretation.
This produces plans that address what the issue says to build rather than what will make the problem stop happening. The plan inherits the issue's proposed solution without questioning it. It designs from first principles without checking what already exists. It commits to a single architecture without naming the one it's competing against. It assigns behavior to a location without verifying that location is testable in isolation. All four omissions can be present simultaneously, and plan-coverage review — which checks completeness — passes all four.
The root cause is that plans are downstream commitment points but upstream grounding is absent. Plans are reviewed after formation for completeness. They are not reviewed before formation for grounding quality. The missing phase is what a staff engineer does before drafting any plan: extract what "done" means in observable terms, survey what already exists, name the choice at the highest-risk design fork, ask whether the proposed fix's assumption is correct, and confirm the implementation location is testable. These five operations are prior to planning, not part of the plan.
What the evidence shows: when this grounding is absent, the plan can be technically correct, pass all coverage checks, and still produce implementations that build the wrong infrastructure, miss the observable goal, or create test-free zones. The fix is not more review after plan formation — it is mandatory grounding before plan formation begins.
The category library and FSI add a second layer: retrieved failure history converts "I know state resilience is a principle" into "I know this specific design shape has failed in this codebase." The distinction is the same one that separates abstract knowledge from indexed negative exemplars. Without retrieval, agents apply general principles correctly but miss specific traps the codebase has already encountered.
Tier 1 (Now, 1 PR): Phase 4.5 in kaizen-evaluate — the five mandatory pre-plan steps
Insert Phase 4.5 between Phase 4 (Critique Spec) and Phase 5 (Ask Admin). Five sequential steps: extract success criteria (GOAL + DONE WHEN before reading any code), survey existing tools (grep before designing), generate and reject at least one alternative (name the highest-risk choice and its failure mode), validate the hypothesis (state the assumption, run the fastest test to confirm or falsify), map testability seams (name the file, test file, and seam for each significant behavior before deciding where it lives). Concrete skill text for each step is ready in R4C.
Time cost: 10-20 minutes per evaluation. Addresses: categories 1, 2, 3, 4 from the failure taxonomy. Infrastructure cost: zero. This is the MVC. The five steps are not a checklist — each one requires looking at something the agent has not yet looked at. The sequence matters: GOAL before code, survey before design, alternatives before commitment, hypothesis before plan, seam before location.
Tier 2 (Next, 1-2 PRs): Plan schema enforcement + FSI bootstrap
Two additions. First: enforce the structured plan schema in store-plan — the sections ## Success Criteria, ## Information Retrieved, ## Design Alternatives Considered, ## Seam Map, ## Test Plan must be present. Warn (not error) on missing sections, surface warnings visibly. This creates an auditable trail: a future reviewer can see whether Phase 4.5 was actually performed, not just whether a plan was stored.
Second: build src/cli-experience.ts with add-entry and query subcommands, bootstrap .claude/kaizen/failure-signatures/ with 5 entries from known incidents (review-findings loss, OOM stop-hook, WSL2 /proc hang, cross-worktree contamination, KAIZEN_UNFINISHED false positive). Add Phase 0.7 to kaizen-evaluate: if the FSI store has entries, query by issue keywords before plan formation begins. Wire add-entry into /kaizen-reflect as a mandatory step when filing issues with FSI-eligible labels. This makes experience accumulation automatic and gives the Recurrence weight R in the DPS formula an empirical grounding.
Tier 3 (Later, 2-3 PRs): Category Library + recognition algorithm + plan battery
The full category library: 10 YAML files under .claude/kaizen/categories/, each with recognition signature, structural test, failure topology, resilient/fragile shape taxonomy, and incident links. The recognition algorithm: three-pass (keyword pre-filter, structural test, precursor signals), under 60 seconds, outputs confirmed categories with confidence scores. The plan battery: 7 dimensions from R3's plan battery spec, enforced after store-plan, blocking on goal-traceability, hypothesis-validation, codebase-survey, and design-alternatives when the plan touches irreversible decisions.
This tier requires Tier 2's FSI data to be calibrated. Build categories from incidents, not from theory. The pre-mortem risks (false confidence, overhead creep, category capture) apply at this tier; mitigations are incorporated (executable grep output rather than prose claims, match-confidence threshold to prevent ambiguous matches from loading wrong priors, soft-block on most dimensions to preserve autonomy).
Before Tier 1: Agent reads issue, forms plan based on first interpretation, proceeds. Plans routinely miss the goal while passing coverage review. No record of whether alternatives were considered. Testability gaps discovered after 70 lines are written in main().
After Tier 1: Agent is forced to write DONE WHEN before reading any code — this single discipline prevents goal-drift. Agent runs grep before designing — this surfaces existing tools that would have been reinvented. Agent names the highest-risk design choice and states what would disqualify the alternative — the orchestrator-batch vs. agent-stores question gets asked, not defaulted. Hypothesis about the proposed fix is stated and tested before planning — wrong-root-cause implementations are caught. Testability seam is named before location is chosen — behaviors get extracted to testable modules. The four most expensive failure categories are addressed.
After Tier 2: Plans carry auditable records of what was retrieved. FSI queries surface "this design shape failed here before" at planning time. The Recurrence weight R is evidence-grounded. Kaizen-reflect automatically captures new failures into the index. The gap between "principle known" and "negative exemplar retrievable" begins closing.
After Tier 3: Category recognition fires before design evaluation begins, loading failure topology for the matched category. The agent doesn't reason from scratch about state resilience — it loads "session-boundary-state" category, sees "accumulate-then-flush" is fragile, proceeds with write-through as prior. Plan battery provides a formal gate with enforcement. Proportionality is automatic: clean category match + high confidence = near-zero overhead; novel territory = full exploration with explicit documentation.
After five rounds, the honest answer:
What is trainable (encodable in skills and tools): category recognition given a curated library, option generation and comparison as mechanical steps, adversarial simulation when explicitly prompted, proportionality calculation given explicit signals, the format of design thinking (name category → sketch options → state failure modes → choose). An agent given a good category library and Phase 4.5 discipline will match staff engineer judgment on known categories.
What requires accumulated incident history: the ability to weight failure modes by visceral salience. A staff engineer who burned on state-resilience failures feels unease when writing session-scoped state before articulating why. That unease is not articulable as a rule — it is procedural memory built over hundreds of incidents, each emotionally weighted. The agent's equivalent is the FSI's confirmed_recurrences field: not felt weight, but counted recurrence. It's a functional substitute, not the same thing.
What is genuinely irreducible: anomaly detection — noticing that something doesn't fit the prototype. Human experts notice this before articulating why; the feeling precedes the analysis. AI agents can simulate the output of anomaly detection (generate failure scenarios, falsify hypotheses) but cannot replicate the precognitive trigger. The pre-mortem's "category capture" failure mode illustrates this: an agent under pressure takes the closest category match without the human expert's felt sense that the match is wrong.
The calibration target is not "give the agent staff engineer judgment." It is "make the agent behaviorally equivalent to a staff engineer on known failure categories, and make it explicitly slower in novel territory." That is achievable. The 80% coverage that the trainable elements provide is worth building for. The 20% irreducible gap is worth naming, not pretending closed.
The loop:
Incident → /kaizen-reflect files issue with FSI-eligible label → mandatory add-entry --from-issue N → new FSI entry added to index, matched to existing category or flagged for new category creation.
FSI entry → at planning time, Phase 0.7 queries by issue keywords and shape → top-3 entries injected into agent context before plan formation → agent's Information Retrieved section cites the retrieved entry by name.
Better planning → Phase 4.5 grounds the plan + FSI entry warns against specific anti-pattern → plan selects preferred shape → implementation avoids the failure → no new incident for this category.
Category stabilization → FSI entries for this category accumulate positive outcomes (preferred_pattern_selected) → confidence increases → future retrievals surface this category faster → cost of recognition falls toward near-zero for this failure class.
Novel failure (the loop's other branch) → failure not in FSI, not in category library → incident filed → becomes FSI entry → was_category_retrieved_at_design_time: false flags retrieval gap → missing keyword added to FSI entry's surface_trigger_keywords → future issues with that vocabulary trigger retrieval.
The loop is asymptotically self-improving for the known failure space and self-expanding for the novel failure space. What it cannot do: prevent a genuinely novel failure mode in territory the library has never encountered. The pre-mortem's concurrent-write-on-shared-document example remains outside the loop until it happens once.
From the pre-mortem, ranked by likelihood:
Most likely: false confidence from well-formed but ungrounded plan text. The battery evaluates text. Text can be correct-looking without being grounded — a plan that claims "no existing tool found" passes the codebase-survey dimension if the battery cannot call grep. Mitigation: the codebase-survey step in Phase 4.5 requires actual grep output stored in the Information Retrieved section, not a prose claim. This is partially executable verification rather than assertion. The battery dimension then checks for the presence of grep output, not the claim of a search.
Second most likely: retrieval without integration. FSI entry retrieved, Information Retrieved section populated, anti-pattern noted, plan proceeds with the anti-pattern anyway. The plan is auditable but hollow. Mitigation: if the plan's frontmatter lists fsi_entries_consulted, the design alternatives section must name the retrieved entry's anti_pattern_shape explicitly — not in a summary, in the alternatives analysis. "I considered the anti-pattern and rejected it because..." is forced acknowledgment that cannot be satisfied by copying the FSI entry title.
Third most likely: overhead creep in the battery. Every post-mortem adds a dimension. By month five, 13 dimensions and variable names blocks trivial changes. Mitigation: the battery's blocking scope is narrowly defined from the start — blocks only on goal-traceability, hypothesis-validation, codebase-survey, design-alternatives when the plan touches irreversible decisions. Non-blocking dimensions generate PARTIAL findings with warnings. Scope is written into the battery protocol, not adjustable issue-by-issue. New dimensions require a kaizen issue and admin approval, not just a reflect step.
The override risk. If any of the three systems generate enough interrupts, agents route around them. The bypass path becomes the normal path. This is the mechanism by which all L1 enforcement eventually fails. Mitigation: Tier 1 (Phase 4.5) adds time but not interrupts — it is procedural, not a gate. Tier 2 adds observability but only warns, not blocks. Tier 3's battery is the only gating mechanism; its blocking criteria are narrow and its escalation path goes to the admin only after 2 revision cycles. Monitor admin override rate: above 20% means the system is miscalibrated, not the agent.
Whether Phase 4.5 will be followed faithfully without enforcement. The five steps are L1 instructions in a SKILL.md. The existing evidence (issue #947: agents skip steps buried past a certain depth) predicts that Phase 4.5 will be followed initially and skipped under pressure over time. The MVC addresses 4/5 failure categories at zero infrastructure cost; the cost is that it is L1. Empirical observation of whether the steps are performed will determine whether Tier 2's schema enforcement (which makes omission visible) is sufficient or whether a hook that checks Phase 4.5 completion is required.
Whether the FSI retrieval mechanism will produce integration or just acknowledgment. The audit trail created by fsi_entries_consulted and the was_category_retrieved_at_design_time field will answer this after 10-15 evaluations. If plans consistently cite FSI entries but proceed with the anti-pattern, retrieval without integration is confirmed as the bottleneck and the forced-acknowledgment requirement (naming the shape in the alternatives analysis) should be elevated from advisory to blocking.
Whether the bottleneck is plan quality or implementation fidelity. The pre-mortem raises this explicitly: PR #894's 3 undelivered acceptance criteria were in the plan, not missing from it. The implementation dropped them. Better plans wouldn't have changed that. If post-merge issues for undelivered acceptance criteria do not decrease after Tier 1 ships, implementation fidelity is the next bottleneck and the locus of improvement shifts to kaizen-implement, not kaizen-evaluate.
The novel failure floor. The pre-mortem's concurrent-write scenario is a real upper bound on what the system can prevent. The systems address known failure categories; novel failures are definitionally outside their scope. Kaizen's most expensive historical failures have often been novel. The systems are expected to reduce recurrence of known categories significantly; their effect on novel failures is zero by construction. This is not a reason to not build them — it is a calibration: success is "known failures stop recurring," not "all failures stop occurring."
The bet: Phase 4.5 is the MVC because it addresses the most common failure modes at zero infrastructure cost. The FSI bootstrap makes the category library empirically grounded rather than theoretically derived. The plan battery formalizes what Phase 4.5 establishes as discipline. Build in that order. Test against real evaluations. Update when the evidence contradicts the model.