Forensic research pass. Excludes the 8 already-documented cases (PR #832/#666, PR #816/#814, PR #894/#891, PR #970/#966, issues #940/#957, issue #724, issue #901).
Category: Hypothesis-as-Contract (existing)
What was planned: Issue #685 prescribed a static lint hook (kaizen-lint-kernel-paths.sh) that blocks writes to test files containing /proc/, /sys/, process.kill, etc. PR #708 implemented it fully: 22 tests, hook registered, all passing.
How the plan was inadequate: The issue addressed one specific trigger (/proc/invalid/path hanging on WSL2) rather than the failure class ("tests hang with no circuit breaker"). The proposed solution defended against a threat that wouldn't recur. The correct fix was test timeouts (#684) and a per-run wall-clock budget (#686) — which would handle any hang, not just /proc hangs.
Result: PR #708 merged. Issue #712 filed ("Revert kernel-path lint hook — wrong abstraction level") by admin. PR #736 reverted it entirely (232 lines deleted, zero added). The correct fixes (#684, #686) were implemented separately.
Was the information available? Yes. The issue body itself said "Root cause: no test timeout." The issue also referenced #684 (vitest testTimeout) and #686 (wall-clock budget) as companion work. The plan builder focused on the incident trigger, not the failure class those issues addressed.
Failure signature: Issue prescribes a specific mechanism ("add a hook that blocks X") rather than stating the failure mode. Admin later says "wrong abstraction level" in the revert issue body.
Category: Incomplete Scope (new — see Category 6 below)
What was planned: Issue #790 asked to delete state-utils.sh after all consumers migrated to TypeScript. PR #803 deleted it plus two dead bash hooks, their tests, and updated docs. Net: -1575 lines.
How the plan was inadequate: The plan scoped deletion correctly but did not check whether any bash test files referenced the deleted hooks without consuming their logic directly. Six bash test files (test-pr-kaizen-clear.sh, test-hook-interaction-matrix.sh, test-integration-kaizen-lifecycle.sh, test-waiver-quality.sh, test-kaizen-merge-gate.sh, test-integration-audit-isolation.sh) had require_file guards pointing at the deleted hooks. When the hooks were gone, the guards silently skipped all 3054 lines of tests — resulting in false green CI with zero assertions executing.
Result: Issue #808 filed: "3054 lines of bash tests silently skip after TS migration — false green on hook integration coverage." PR #809 deleted the six dead test files. The integration scenarios those tests covered (cross-hook interaction invariants, full lifecycle tests) had no TS equivalent — the coverage loss was accepted.
Was the information available? Yes. A pre-deletion grep -r "kaizen-pr-reflect-clear" .claude/hooks/tests/ would have revealed all 6 files. The standard deletion checklist (update plugin.json, settings.json, doc references) was followed; "grep for test files that guard on this hook" was not.
Failure signature: Deletion PR checks "all tests pass" but doesn't ask "are any tests silently skipping?"
Category: Hypothesis-as-Contract (existing)
What was planned: Issue #814 said "fix the underlying slowness rather than increasing the timeout." The timing sentinel (runHookTimingSentinel()) benchmarked all hooks by running each bash wrapper, and with 10+ wrappers each spawning npx tsx (1-2s each), the total exceeded the 10s test timeout. Root cause: sentinel was slow; it benchmarked every hook on every test run.
How the plan was inadequate: PR #816 disabled the sentinel in test environments via an env var (HOOK_TIMING_SENTINEL_DISABLED). The sentinel still runs in production. The acceptance criterion — "fix the underlying slowness" — was explicitly not met. The admin-stated requirement and the delivered implementation were directly at odds.
Was the information available? Yes. The issue body stated the acceptance criterion explicitly. The root cause (npx tsx startup time) was identified in the PR body. An alternative approach (benchmark only a subset of hooks, cache results, run asynchronously) was not explored.
Category: Dual Failure Mode Blindness (new — see Category 7 below)
What was planned: PR #718 fixed the "multi-pick loop" in auto-dent where agents kept picking new issues after completing one (run-2 picked 4 issues in one session). Fix: "one issue per run" policy.
How the plan was inadequate: The fix was correct in spirit but wrong in mechanism. /kaizen-deep-dive already had a legitimate bundling capability for related issues. The "one issue per run" rule prevented intentional bundling that was valid workflow. The plan builder focused on eliminating the observed failure mode without asking "what valid behavior does this rule prevent?" — the dual failure mode check prescribed in issue #722.
Result: PR #720 filed immediately after #718 ("fix(auto-dent): plan-first scope — intentional bundling, not blind loop"). PR #720 replaced the rule with a PLAN→EXECUTE→STOP pattern that allowed deliberate bundling while preventing reactive scope expansion.
Was the information available? Yes. The /kaizen-deep-dive skill explicitly documented intentional bundling. Issue #722 ("kaizen-implement: name both failure modes before shipping behavioral/prompt constraints") was filed during this same session, documenting that the agent didn't check both failure modes before shipping.
Failure signature: An L1 constraint fix requires a corrective PR within the same batch. Admin notes that the constraint "was too restrictive."
Category: Scope Leakage (new — see Category 8 below)
What was planned: Issue #849 asked for an adversarial review battery (compare PRs against linked issues). PR #846 delivered the core review battery — plus 4 extra review dimensions (plan-fidelity, dry, test-plan, improvement-lifecycle), a /kaizen-dimensions skill, docs/artifact-lifecycle.md, docs/review-dimensions-research.md, and 3 extra zen principles. None of these extras had corresponding issues.
How the plan was inadequate: The plan had no scope boundary. Work that emerged during implementation got bundled into the PR without lifecycle tracking. Three retroactive issues were filed (#861, #862, #863) just to document what had shipped.
Follow-on costs:
- #861: track
/kaizen-dimensionsskill (shipped untracked) - #862: track 4 extra review dimensions (57% more review cost than planned)
- #863: track 3 docs and 3 zen principles
- #876: DRY/tooling follow-up from review battery review (3 post-review bugs: #881, #882, #883)
- The 4 unplanned dimensions never had replay tests against motivating PRs (tracked in #860, #868)
Was the information available? The problem is structural: no scope boundary was declared before implementation. There was nothing to compare against.
Failure signature: Multiple issues filed retroactively to track what was shipped. Comment on issue: "shipped in PR #846, filed retroactively for lifecycle tracking."
Category: Testability Cutout (existing, extends Case from PR #894/#891)
This is a distinct sub-case from the already-documented PR #894/#891 case, involving the code quality of what was shipped.
What was planned: Issue #891 asked to wire the review-fix loop into auto-dent-run.ts. PR #894 did so, adding ~70 lines of integration code inside main().
How the plan was inadequate: The plan did not include extracting the integration into a testable function first. The wiring code embedded deep inside a 2000-line main() with no dependency injection — untestable without running the full system. Five bugs were discovered by post-hoc audit (#914 meta-issue):
- Fix loop spawned with 0 gaps when all dimensions timeout (#897)
- Budget uses total budget, not remaining (#898)
- Event emission boilerplate repeated 4x (#899)
resume: falsehardcoded, prevents crash recovery (#900)- Zero unit test coverage for the wiring (#896)
Result: Issue #914 documented the pattern. PR #918 extracted runReviewWiring() with dependency injection and fixed all 5 bugs.
Was the information available? Yes. The issue body for #891 explicitly required "Portable zippable run artifacts" and "Test harness supports resume-from-failure." The test plan called for 5 unit tests. Only 4 event schema tests were delivered. The wiring was untestable as written — extracting it into a testable function was the prerequisite step that wasn't done.
Failure signature: Test plan promises unit coverage; delivered PR has only event schema tests. Post-hoc audit reveals bugs in the untested integration path.
Category: Self-Referential Compliance Failure (new, closely related to Hypothesis-as-Contract)
What was planned: Issue #963 established Policy 10: skill changes require behavioral proof (before/after claude -p evidence + smoke test). PR #965 implemented Policy 10 enforcement.
How the plan was inadequate: PR #965 merged without the review battery running (#973). A retrospective review found 5 MUST-FIX items:
- No plan stored in structured data (plan-fidelity gap)
- No actual
claude -pbefore/after transcripts (the very thing the policy requires) - Duplicate imports in test file (dry)
- Worktree path resolution broken in tests
- No behavioral test for the new dimension
PR #974 was filed as a corrective to address all 5 items. The PR that introduced a policy requiring behavioral proof shipped without behavioral proof.
Was the information available? Yes — the policy text was written in the PR body itself. The review battery gate not firing was a separate bug (#973), but the plan builder should have verified the gate would fire before merging.
Failure signature: A policy-enforcement PR violates the policy it introduces. Corrective PR follows immediately.
Category: Single Design (existing) + Self-Defeating Mechanism
What was planned: scope-guard.sh was designed to detect double-installation and block. When it detected kaizen@kaizen in ~/.claude/settings.json, it blocked all tool calls.
How the plan was inadequate: The scope-guard blocked ALL tools — including the Bash tool needed to fix the settings.json. This created a complete deadlock: the fix required Bash access; the guard blocked Bash access. The plan considered only the detection case, not the recovery case.
Result: Real user spent 10+ messages in a deadlocked session, eventually had to run a fix manually outside Claude Code (#757 incident). The guard had to be redesigned to allow the specific fix operation.
Was the information available? Yes. Designing a blocking guard requires asking "can the agent escape when this fires?" That question is not recorded as having been asked during the original design.
Failure signature: Admin discovers deadlock when a real user files an incident report. Issue body says "took 10+ messages to resolve, user had to manually run python3 outside Claude Code."
(Distinct from Case 10 — this captures the evaluation failure in the migration series, not just PR #803)
Category: Codebase Survey Failure (existing)
What was planned: The TS migration series (#786, #789, #798, #801, #803, #804) migrated bash hooks to TypeScript. Each PR ran "all tests pass" and reported correct counts.
How the plan was inadequate: No PR in the migration series checked "are any tests now silently skipping?" The bash test infrastructure uses require_file guards that gracefully skip when the targeted hook is absent — this is the correct behavior for optional hooks, but it creates invisible failures when a hook is deleted. The cumulative effect: 3054 lines of bash tests silently skipping after #803.
What "all tests pass" concealed: 6 test files reporting "0 passed, 0 failed, 0 skipped" when they were actually skipping every assertion. The CI summary showed no failures.
Was the information available? Yes. The require_file pattern was in every test file header. Any review of the test infrastructure would reveal that it gracefully skips on missing hooks. The migration plan should have included: "for each hook being deleted, grep for bash test files that guard on it and decide: port to TS or delete."
Category: Command-Detection-Not-Outcome Enforcement (maps to Category 6)
What was planned: pr-review-loop.ts PostToolUse hook creates a needs_review gate whenever gh pr create is detected in the Bash command. This is supposed to block merge until a review runs.
How the plan was inadequate: PR #965 used a large heredoc body for gh pr create. The command string contained raw newlines, backticks, dollar signs, and special Unicode. When Claude Code serialized this to JSON for the hook stdin, JSON.parse() threw. readHookInput() returned null. The hook exited silently at line 437 with process.exit(0) — no trace, no gate, no review.
Result: PR #965 merged without any review battery firing. Issue #973 filed (incident report). Issue #975 filed (null-input exits silently). PR #976 fixed the silent exit by adding a trace entry and logging raw bytes on parse failure.
Was the information available? The null-input path was discoverable by reading hook-io.ts. The process.exit(0) at line 437 had no trace call before it — visible in the code. No test existed for the null-input path.
Failure signature: Hook fires correctly on simple commands but silently fails on commands with special characters in --body arguments. The failure leaves no trace.
Category: Codebase Survey Failure (existing) + Missing State Check
What was planned: Auto-dent batch batch-260323-1405-5400 created issue tracking. The harness called gh issue create for the batch tracking issue.
How the plan was inadequate: The harness did not check for an existing tracking issue before creating a new one. Two identical issues (#707, #709) were created with identical titles, splitting batch progress across two issues.
Result: Issue #726 filed. PR #825 fixed it by adding findExistingProgressIssue() — a search before create.
Was the information available? Yes. The "check before create" pattern is standard practice for idempotent issue creation. The harness code called gh issue create unconditionally.
Category: Untested Seam (maps to existing + extends to test pyramid)
What was planned: The review gate system (pr-review-loop.ts PostToolUse hook) was supposed to fire whenever gh pr create was called in headless -p mode during auto-dent runs.
How the plan was inadequate: The gate was never verified to work in headless mode. Unit tests existed for the TypeScript functions. No E2E test existed that proved Claude Code would invoke the hooks, deliver output to the agent, and block the session. Batch jolly-marsupial ran 36 sessions and produced 25 PRs — zero review gates fired.
Investigation revealed: Hooks fire from the worktree, not the main checkout. Hook wrappers installed in the main checkout aren't invoked from worktree sessions. The entire enforcement chain was invisible in the mode where it mattered most.
Result: Issue #843 filed. The review battery was then built to work differently — spawning explicit claude -p subagents from the skill rather than relying on hooks to enforce review.
Was the information available? Yes. The hook system is invocation-point-sensitive. A single canary test (write to /tmp/file in a PostToolUse hook, start a headless session, check if the file was written) would have confirmed or denied hook firing in -p mode.
Failure signature: "All unit tests pass" combined with "zero enforcement events in production runs."
Category: Single Design (existing) — Shared Unstated Assumption
What was planned: Five independent systems (vitest.config.ts, review-fix.ts stateDir(), worktree-du.ts cleanupWorktrees(), gate-manager.ts, kaizen-worktree-setup.sh) were each built independently before worktrees became a core primitive.
How the plan was inadequate: Each system was planned and built in isolation. None of them surfaced the shared assumption: "the working directory is stable, long-lived, and owned by a single session." Worktrees violate all three. When a worktree is deleted after merge, five systems fail simultaneously in a cascade.
Symptoms:
vitestcrawled 94 worktrees, polluting test outputreview-fix.tsstate stored inprocess.cwd()/.claude/review-fix/was deleted with the worktreeworktree-du.tsdeleted worktrees unconditionally with no drain or warning- Post-merge stop gate message omitted
/kaizenreflection step - Sessions inside deleted worktrees had all tool calls fail (CWD no longer exists)
Was the information available? Yes. Once worktrees were established as a core primitive (.claude/worktrees/, kaizen-worktree-setup.sh), every new system touching CWD or file paths should have been audited for worktree safety. The audit happened reactively (issue #939) rather than proactively.
Category: Goal vs. Work-Item (existing)
What was planned: The /kaizen-review-pr skill was built to run a review battery and post findings. The skill's SKILL.md described running dimension subagents and classifying findings.
How the plan was inadequate: The SKILL.md had no instruction to call store-review-finding after Phase 3. The review findings existed in-context but were never persisted to the PR. Three consecutive PRs (#933, #956, #965) had zero per-dimension finding comments. The storage step — the mechanism that makes reviews visible and resumable — was completely absent from the skill.
Divergent paths discovered: review-fix.ts CLI (the automated path) called storeReviewFinding() after reviewBattery() — hardcoded in the script. The skill (the manual path) had no such instruction. Two paths, two outcomes, zero consistency.
Was the information available? Yes. The store-review-finding CLI command existed and was documented. review-fix.ts called it. Reading both the skill and the CLI script side by side would have revealed the gap immediately. The skill was written without verifying it produced the same outputs as the automated path.
Category: Recursive Failure (new — see Category 9 below)
What was planned: The kaizen reflection skill had an "Admin teaching embedding" section that said to save corrections as memory files AND codify them in infrastructure.
How the plan was inadequate: The instruction was L1 only. No hook verified that when a correction was given, it was saved. The "write real E2E tests" feedback was given across multiple sessions but never saved as a memory or filed as an issue until the admin explicitly said "can you debug why the lesson on testing is not sticking?" The reflection produces a KAIZEN_IMPEDIMENTS declaration but doesn't force memory saves.
Was the information available? Yes. The reflection skill itself documented the required behavior. The gap was that the required behavior was L1 (optional) — no enforcement verified it happened. This is the system's own self-improvement mechanism failing to capture lessons about itself.
Failure signature: Admin says "I told you" or "how many times" — the signal that a correction was given before but not absorbed.
Category: Solution Collapse (existing)
What was planned: Issue #685 specified "Static lint: block tests that write to /proc, /sys, or kernel interfaces." The issue prescribed the mechanism (lint hook), not the failure mode (tests hang with no circuit breaker). PR #708 implemented the prescribed mechanism faithfully.
How the plan was inadequate: The implementor (kaizen-implement) accepted the spec as ground truth. The spec had already collapsed from failure mode to solution. There was no evaluation step that asked "is this the right solution?" The evaluate skill at the time of PR #708 did not have the 5-question spec critique added by issue #714.
Post-hoc recognition: Issues #713 ("describe the failure mode, not the solution") and #714 ("critically evaluate spec from multiple angles") were filed during the same admin session that caught the kernel-path hook problem. Both document the underlying evaluation gap.
Category: Command-Detection-Not-Outcome (extends existing)
What was planned: Multiple gates (pr-review-loop.ts, bump-plugin-version.ts, pr-kaizen-clear.ts) used command detection as the trigger for state transitions. "If the agent ran gh pr diff, review happened." "If the agent ran git commit, the code is committed."
How the plan was inadequate: None of the gate designs asked "what evidence proves this action succeeded?" A gate that fires on command presence clears when the command runs, regardless of outcome. Concrete symptoms:
bump-plugin-version.tscommitted but didn't push — PR creation failed with "branch ahead of remote" (#919)- Review gate cleared on
gh pr diffwithout verifying dimension agents were spawned or findings stored (#920) - Reflection gate cleared on
echo KAIZEN_IMPEDIMENTS: []without checking substance
Was the information available? Yes. The PostToolUse hook receives the tool result (exit code, stdout, stderr). Every gate had access to outcome evidence. None used it. Issue #921 documents this as a "category error" that spans all L2 gates.
Cat 1: Goal vs. Work-Item — Plan treats deliverable as checked box without verifying the goal is achieved.
- New cases: #22 (review-pr skill missing storage step), #12 (one-issue-per-run over-correction)
- Frequency: High. Appears whenever implementation matches spec language but misses intent.
Cat 2: Hypothesis-as-Contract — Plan accepts unvalidated hypothesis as design fact; testing would refute it.
- New cases: #9 (kernel-path hook), #11 (disable timing sentinel), #24 (spec prescribed wrong solution)
- Frequency: Very high. The most common failure mode across the history.
Cat 3: Single Design — Plan produces one design without exploring alternatives; the one design has a fatal flaw.
- New cases: #16 (scope-guard deadlock), #21 (five systems, shared unstated assumption)
- Frequency: Medium. Appears on architectural decisions; often expensive when it occurs.
Cat 4: Testability Cutout — Plan ships code that works but can't be verified by tests; bugs surface post-merge.
- New cases: #14 (review-fix wiring in main()), #20 (25 PRs without review gate)
- Frequency: High. Appears repeatedly in integration code and headless mode.
Cat 5: Codebase Survey Failure — Plan designs from first principles; existing tools or dependencies overlooked.
- New cases: #10 (deletion without tracing dependents), #17 (migration left 3054 dead test lines), #19 (duplicate issue creation)
- Frequency: High. Common in auto-dent autonomous runs.
Category 6: Incomplete Deletion Scope Deletion is scoped to the target artifact but not to everything that references it. The deleted thing is gone; the references remain, silently broken (tests that skip, docs pointing at deleted files, plugin.json with dead hook paths).
Signature: "All tests pass" after deletion. Later discovery that tests were silently skipping, docs were stale, or CI was running against deleted artifacts.
Cases: #10 (Case 10), #17 (Case 17), plus repeated minor instances in the TS migration series.
Why existing categories miss it: Cat 5 (codebase survey) covers failing to find existing tools before building. Cat 6 covers failing to trace dependents before removing. Different error, different query: not "what already does this?" but "what points at this?"
Category 7: Dual Failure Mode Blindness A constraint fix eliminates one failure mode but creates the symmetric opposite. The plan builder optimizes against the observed failure without naming both ends of the design space.
Signature: A behavioral rule fix is followed immediately by a corrective PR that loosens the rule. Admin notes the fix was "too restrictive" or "prevented valid behavior."
Cases: #12 (one-issue-per-run → plan-first scope), #16 (scope-guard blocking fix operations), issue #722 formally names this pattern.
The test that catches it: Before shipping a behavioral constraint, name both failure modes — "if absent: X" and "if present: Y" — and verify Y is acceptable.
Category 8: Scope Leakage Without Lifecycle Implementation ships features beyond what was tracked in any issue. The extras are coherent with the project but have no lifecycle (no feedback loop, no replay tests, no improvement path). Retroactive issues are filed to track what shipped.
Signature: Multiple retroactive issues filed after merge. PR diff contains substantially more new files than referenced in any issue. Review battery flags "21% unrequested" scope in scope-fidelity dimension.
Cases: #13 (PR #846: 4 extra dimensions, 1 extra skill, 2 docs, 3 zen principles), PR #903 scope-fidelity finding (21% unrequested, all justified post-hoc).
Why it matters: Untracked features have no feedback loop. If they're wrong, nobody knows to fix them. If they drift, nobody tracks the drift. The improvement lifecycle PR #876 filed (DRY/tooling follow-up) was itself triggered by finding untracked elements.
Category 9: Recursive Failure — The System Fails to Capture Lessons About Itself The kaizen reflection and memory system is supposed to capture lessons from admin corrections. When the reflection system itself fails to enforce lesson capture, the same mistakes recur. This is kaizen failing to kaizen itself.
Signature: Admin says "I told you" or "this is not sticking." Memory files exist for the lesson but were never populated. Reflection produced KAIZEN_IMPEDIMENTS but no hook verified memory was saved.
Cases: #23 (E2E test feedback not retained across sessions), issue #781 (admin corrections not persisted), issue #959 (skill chain verification discipline enforced as prose, never as phases).
| Category | Case Count (all, incl. prior 8) | Highest-Cost Instance |
|---|---|---|
| Hypothesis-as-Contract | 6+ | PR #816/#814 (CI broken, timing sentinel disabled) |
| Codebase Survey Failure | 5+ | PR #970/#966 (orchestrator vs. agent-stores design) |
| Testability Cutout | 5+ | Batch jolly-marsupial: 25 PRs, zero review gates |
| Goal vs. Work-Item | 4+ | PR #832/#666 (schema built, 0 SKILL.md files populated) |
| Single Design | 4+ | Issue #758 (scope-guard deadlock — complete system lockout) |
| Incomplete Deletion Scope | 3+ | PR #803 / Issue #808 (3054 dead test lines) |
| Dual Failure Mode Blindness | 2+ | PR #718 → PR #720 (immediate corrective PR) |
| Scope Leakage Without Lifecycle | 2+ | PR #846 (4 unplanned dims, 1 unplanned skill) |
| Recursive Failure | 2+ | Issue #781 (E2E lesson not sticking) |
Most costly by downstream impact (fixes required + time lost):
- Testability Cutout — 25 PRs shipped without review, entire batch unverified. Required redesign of review infrastructure.
- Single Design (Scope-Guard Deadlock) — Real user lost 10+ messages, filed incident report, led to #757 and #758.
- Codebase Survey Failure (Custom Storage) — Led to architecture rework in PR #970, multiple corrective issues.
Most frequent (highest recurrence):
- Hypothesis-as-Contract — 6+ cases; appears in every batch where autonomous work runs without hypothesis testing.
- Codebase Survey Failure — 5+ cases; the default failure mode for autonomous agents designing new features.
For each category, the earliest visible signal in GitHub history that the plan was going wrong:
Cat 1 (Goal vs. Work-Item):
- Signal in PR body: implementation confirms "all acceptance criteria delivered" but one criterion requires behavioral change, not code change. Zero behavioral evidence in the PR.
- Signal in issue: acceptance criterion uses a verb like "stop being" or "make X happen" rather than "implement Y."
- Earliest detection: review battery
requirementsdimension finding MISSING for a criterion that isn't a code artifact.
Cat 2 (Hypothesis-as-Contract):
- Signal in issue body: "likely cause" or "hypothesis" in root cause section, but no test described.
- Signal in plan: solution designed to address the stated mechanism, not the failure class.
- Signal in PR: test plan tests the mechanism works, not that the failure class is prevented.
- Earliest detection: issue #948/#949 — admin added mandatory hypothesis validation phases to skills.
Cat 3 (Single Design):
- Signal in plan: no "alternatives considered" section; single implementation described as "the fix."
- Signal in PR body: no design decision record, no tradeoff statement.
- Earliest detection: issue #979 — the 5-question design pause checklist: "is there more than one design that would work, and do they fail differently?"
Cat 4 (Testability Cutout):
- Signal in PR: test plan says "unit tests" but integration code has no dependency injection.
- Signal in PR: "wiring" or "integration" code lives inside
main()rather than an extracted function. - Signal in review: tooling-fitness dimension flags "integration code in main() — untestable."
- Earliest detection: issue #883 (
launchFix/prefetchnot injectable), PR #876 extraction ofrunFixLoop().
Cat 5 (Codebase Survey Failure):
- Signal in plan: plan mentions "we will build X" where X already exists (e.g.,
write-attachment). - Signal in PR: new functions duplicate functionality of existing CLI commands.
- Signal in review: DRY dimension flags "reimplements utilities already in X."
- Earliest detection: issue #957 — "planning phase doesn't survey existing tools." Review battery
tooling-fitnessdimension catches this post-implementation.
Cat 6 (Incomplete Deletion Scope):
- Signal in PR: "deleted X" but no verification that referencing files were updated.
- Signal in test output: 0 passed, 0 failed (but silently skipping).
- Earliest detection: grep "require_file " before deleting. No automated check existed until post-discovery of issue #808.
Cat 7 (Dual Failure Mode Blindness):
- Signal in issue: solution section specifies a blocking rule without naming what the rule prevents.
- Signal in PR: test plan only tests that the bad case is blocked; no test that the good case still works.
- Earliest detection: issue #722 ("name both failure modes before shipping"). Required adding to evaluate/implement skill.
Cat 8 (Scope Leakage):
- Signal in PR: diff contains new files with no corresponding "Closes #N" or "Fixes #N."
- Signal in review: scope-fidelity dimension finds "X% unrequested" scope.
- Earliest detection: retroactive lifecycle issues filed post-merge (earliest seen: #861, #862, #863 for PR #846).
Cat 9 (Recursive Failure):
- Signal: admin correction given in session, not reflected in any memory file, issue, or policy update.
- Signal: same feedback given in 2+ sessions with no intervening structural change.
- Earliest detection: issue #781 — "recurring admin corrections not persisted — feedback lost between sessions."
When the admin intervenes:
-
"Why didn't you..." — Post-merge, when something that should have been done wasn't. Usually triggers a retroactive issue. Examples: #900 (3 acceptance criteria not filed as follow-ups), #920 (review gate clears without proof of review).
-
"This is the wrong level" — Immediately after discovering the wrong abstraction. Usually triggers a revert. Examples: #712 (kernel-path hook), PR #720 (one-issue-per-run rule). Admin says "wrong abstraction level" or "too prescriptive."
-
"I told you this before" — When a feedback lesson hasn't been retained. Triggers both a fix AND a meta-issue about why lessons aren't sticking. Examples: #781 (E2E testing feedback), issue #959 (verification discipline).
-
"You should have..." — Appears in issue bodies where the admin describes what the implementor should have checked: "a simple grep would have found this," "reading the CLI --help would have shown this."
-
Direct incident report — When a user (sometimes the admin acting as a user) files a detailed incident report with session transcript. High severity, usually reveals a complete failure in a real scenario. Examples: #756 (installation broken for non-Node projects), #757 (repo becomes non-operable after install).
What triggers admin "this plan is wrong" recognition:
- Missing dual failure mode: the fix prevents the bad case but now the good case breaks too. Recognized within one or two uses of the new rule.
- Symptom addressed, failure class not: the fix makes the specific incident not recur, but the next slightly different incident causes the same problem.
- "Check before create" missing: system creates duplicates on retry/restart. Recognized immediately when duplicates appear.
- Behavioral proof missing: the PR says "skill updated" but no evidence the skill produces different behavior. Admin added Policy 10 to enforce this.
- Self-defeating mechanism: a guard or rule that blocks the agent from fixing what the guard is complaining about. Recognized when the agent reports it's stuck.
Admin recognition time:
- Immediate (same session): dual failure mode blindness, self-defeating mechanisms, "this is the wrong level"
- Within 24h: acceptance criteria not delivered, missing follow-up issues
- Weeks later: 3054 dead test lines (only found during an exploration run)
- Never (without tooling): 25 PRs shipped with zero review gate firing — required building the review battery to detect this
Research completed: 2026-03-26. 15 additional cases documented across 5 confirmed categories and 4 newly identified categories.