March 2026 — Synthesis layer
A category is not a label. It is a predictive structure: a named pattern of problem-space geometry that allows an agent to load failure-specific priors before committing to a design.
Formally, a category has five components:
Category: <slug>
recognition_signature:
surface_features: [keywords, design shapes, API patterns]
structural_test: "A question that returns YES/NO when applied to a design"
precursor_signals: [early phrases in specs or issues that predict this category]
failure_topology:
modes: [named failure modes, each with trigger condition and observable effect]
latency: immediate | deferred | session-boundary | cumulative
visibility: always-visible | visible-on-crash | silent
design_taxonomy:
resilient: [named design shapes that survive all known failure modes]
fragile: [named design shapes with documented failure mode associations]
context_modifiers: [environmental factors that change which shape is resilient]
invariants: [architectural rules that make this category's failures structurally impossible]
incidents:
- issue: <number>
shape_that_failed: <slug>
recovery_cost: low | medium | high
recurrence: <count>
confidence: 0.0–1.0 # derived from incident count and recency
The structural test is the critical component that makes a category mechanically usable. It is not "does this look like a state problem?" — it is "does this design store state in a variable whose lifetime is bounded by the session, to be consumed after the session ends?" A structural test with a YES/NO answer takes seconds to apply. A vague description takes minutes and produces uncertain answers.
The failure topology's latency and visibility fields address the second hardest problem in category recognition: many failure modes are invisible in the happy path and only surface at session boundaries or under load. The agent must know this before trusting a design that appears to work in tests.
The following 10 categories are derived from kaizen's incident history (issues #713, #871, #895, #897, #901, #920, #939, #975), the policies in policies.md, and the failure patterns identified in Round 2.
Category: session-boundary-state Recognition signature: any design where state is accumulated in a variable or process memory to be written or consumed after a session boundary. Structural test: "If this process dies at step N of M, is progress from steps 1 through N-1 recoverable without re-execution?" Failure modes: crash-before-flush (data lost), partial-flush (partial data written, corrupt view), read-of-unwritten (consumer sees empty on restart). Latency: session-boundary. Visibility: silent (happy path always passes). Resilient shapes: write-through (agent writes immediately on completion), append-only-log (each agent appends; no batch). Fragile shapes: accumulate-then-flush (orchestrator collects, writes at end), end-of-phase-storage (write deferred to phase completion). Invariants: "All state that must survive session death is persisted atomically by the writing agent before any coordination signal is sent." Incidents: review-findings-loss (orchestrator batch pattern); related to issues from the review-fix work (#883, #896). Confidence: 0.85
Category: worktree-isolation
Recognition signature: any design where hook, state file, or process reads/writes state without filtering by current worktree identity.
Structural test: "If two agents run concurrently in different worktrees, can one agent's state affect the other's gate decisions?"
Failure modes: cross-contamination (worktree A's gate blocks worktree B's work), false-positive-cascade (deleted worktree state poisons new sessions), stale-state-inheritance.
Latency: deferred (appears only under concurrent use). Visibility: visible-on-crash.
Resilient shapes: worktree-keyed-state (state files carry BRANCH field; readers filter by current branch).
Fragile shapes: global-state-scan (iterate /tmp/.pr-review-state/ without branch filter).
Invariants: "All state file iteration goes through isStateForCurrentWorktree; never iterate state directories directly."
Incidents: #939 (worktree ephemeral lifecycle blindness), #871 (false positive MM status).
Confidence: 0.90 (explicit policy exists, multiple incidents)
Category: gate-bypass-via-exit Recognition signature: any enforcement mechanism that fires between actions rather than at completion; any gate the actor can avoid by stopping. Structural test: "If the actor's next action is to exit (stop, close session, finish), does this gate still fire?" Failure modes: exit-before-enforcement (actor stops without triggering gate), silent-skip (gate never reached because workflow terminates early), continuation-assumption (gate assumes actor will keep going). Latency: deferred (appears only when actor exits early). Visibility: silent. Resilient shapes: completion-dependent-gate (fires when actor declares done), post-hoc-correction (harness runs after actor exits, outside actor's lifecycle). Fragile shapes: pretools-only-enforcement (gate fires before next tool use — bypassed by exit), inline-check (check inserted mid-sequence). Invariants: "Completion-critical policies enforce at Stop hooks or post-hoc harnesses, not PreToolUse alone." Incidents: #975 (pr-review-loop exits silently on null hook input), exit-before-enforcement pattern (#895, #966). Confidence: 0.80
Category: gate-false-positive Recognition signature: any hook or gate that matches on a pattern (grep, regex, keyword) where the pattern is shared between the target signal and unrelated legitimate text. Structural test: "Does this pattern match the empty case, the error case, or any legitimate non-target text in the codebase?" Failure modes: false-positive-block (legitimate work blocked), false-positive-cascade (CI breaks on unrelated PRs), pattern-boundary-bleed (matches substring of unrelated token). Latency: immediate. Visibility: always-visible (loudly blocks). Resilient shapes: anchored-regex (match at line boundaries, with word boundaries), negative-lookahead (exclude known false-positive contexts), context-aware-match (check surrounding lines). Fragile shapes: substring-grep (grep for bare string without anchoring), presence-only-check (any line contains X). Invariants: "Every hook pattern must have a test case for the false-positive scenario it is most likely to produce." Incidents: #871 (MM status false positive), KAIZEN_UNFINISHED false positive, #971 (ambiguous gate messages from 3 matchers). Confidence: 0.85
Category: coverage-without-behavior Recognition signature: any test suite that verifies mocks and structure but does not exercise the real execution path (end-to-end seam). Structural test: "If the implementation were completely wrong but returned the right type, would these tests still pass?" Failure modes: silent-wrong (implementation is incorrect; tests pass; production fails), mock-drift (real API changes; mocked API doesn't; tests pass but nothing works), seam-blindness (the gap between units is untested). Latency: deferred (appears when real system is exercised). Visibility: silent. Resilient shapes: boundary-integration-test (exercises real seam with real dependencies), behavior-verification (test asserts observable output, not internal call sequence). Fragile shapes: mock-heavy-unit (mock all collaborators; verify mock was called), structure-test (assert types and fields; never assert behavior). Invariants: "Every execution path has at least one test that exercises the full seam without mocking the seam itself." Incidents: #896 (review-fix wiring had zero unit test coverage), #928 (hook integration test category-level prevention). Confidence: 0.85
Category: schema-coupling Recognition signature: any design where a producer serializes and a consumer deserializes the same data structure, and schema changes in one are not automatically surfaced to the other. Structural test: "If I add a required field to the producer's output schema, does the build fail, or does it fail silently at runtime?" Failure modes: silent-deserialization-failure (consumer silently ignores unknown fields), required-field-break (new required field → all existing consumers fail at runtime), schema-drift (producer and consumer evolve independently, diverge). Latency: deferred (appears on schema change). Visibility: visible-on-crash. Resilient shapes: shared-schema-definition (single type definition imported by both), version-negotiation (producer includes schema version; consumer validates). Fragile shapes: independent-definition (producer and consumer define the same structure separately), field-ignore-by-convention (consumer ignores unknown fields silently). Invariants: "Shared data structures live in one location; both producer and consumer import from it." Incidents: No direct kaizen incidents yet; derived from r2-staff-engineer-model.md category library. Confidence: 0.50 (theoretical)
Category: late-testability-binding Recognition signature: any implementation where the wiring between components (main(), integration loop, CLI dispatch) is not injectable and therefore cannot be unit-tested without running the full runtime. Structural test: "Can I test this component's logic without starting the real server, real filesystem, or real external service?" Failure modes: test-post-hoc (tests written after bugs are discovered, not before), untestable-seam (wiring cannot be reached by any test), coverage-gap-discovered-in-production. Latency: cumulative (cost grows as bugs accumulate untested). Visibility: visible-on-crash. Resilient shapes: dependency-injection (collaborators passed as parameters), extracted-handler (command dispatch extracted from main(), testable in isolation). Fragile shapes: hardwired-main (main() calls implementations directly with no injection points), god-function (single function that initializes, executes, and coordinates). Invariants: "Every module that handles business logic exports a testable function separate from its entry point." Incidents: #883 (launchFix/prefetch not injectable), #922 (cli-structured-data main() refactor for testability). Confidence: 0.80
Category: skill-as-code-unverified Recognition signature: any change to a SKILL.md, prompt template, or workflow document deployed without behavioral proof (before/after claude -p evidence). Structural test: "Is there a passing test that would have caught the previous version's bad behavior?" Failure modes: silent-skill-regression (skill changes behavior invisibly; no test catches it), instruction-drift (skill instructions diverge from actual behavior; agents follow instructions, not behavior), verification-debt (behavioral proof deferred; never arrives). Latency: deferred. Visibility: silent. Resilient shapes: behavioral-proof-with-pr (synthetic case + before/after evidence ships with the PR), smoke-test-in-ci (CI runs skill smoke test on each PR). Fragile shapes: prose-change-only (skill text updated; no behavioral test), deferred-verification ("we'll test it later"). Invariants: "Skill changes require behavioral proof. Synthetic case + smoke test ship with the PR. Never deferred." Incidents: #963 (policy established), #906 (kaizen-write-pr not triggered). Confidence: 0.90 (explicit policy, critical priority)
Category: enforcement-level-mismatch Recognition signature: any fix applied at Level 1 (instructions/docs) for a failure mode that has already occurred once at Level 1. Structural test: "Has this specific failure mode occurred before? If yes, was the previous fix also at Level 1?" Failure modes: recurrence (same failure happens again because instructions are not followed), false-security (documented policy creates confidence that enforcement doesn't warrant), level-plateau (all fixes stay at L1; escalation never happens). Latency: deferred (appears at next recurrence). Visibility: silent. Resilient shapes: escalation-on-recurrence (second occurrence → L2 hook; third → L3 mechanic), hook-as-fix (enforcement point added, not just instruction updated). Fragile shapes: add-to-claudes-md (update instruction text; declare fixed), comment-the-code (add a comment explaining what not to do). Invariants: "If a failure has happened once, the fix is at minimum L2. If it affects humans, the fix is L3." Incidents: Structural invariant behind multiple kaizen reflections; no single issue but present as meta-pattern. Confidence: 0.90
Category: artifact-lifecycle-gap Recognition signature: any system that produces outputs (review findings, run artifacts, structured data) without a defined consumer, storage location, or lifecycle policy. Structural test: "If I run this 10 times, do the outputs accumulate? Is there a reader? Is there a retention policy?" Failure modes: findings-in-void (review written but never posted to PR), artifact-orphan (output file created but never read), retention-absent (outputs accumulate indefinitely). Latency: cumulative. Visibility: silent. Resilient shapes: write-then-confirm-read (writer verifies consumer exists), lifecycle-policy-at-creation (retention, reader, and format specified when artifact is created). Fragile shapes: produce-and-forget (write output; assume something will read it), lifecycle-deferred (we'll figure out retention later). Invariants: "Every artifact has a defined reader, a defined retention policy, and a defined failure mode if the reader is absent." Incidents: #966 (review findings not posted to PRs), #900 (closed issue with 3 acceptance criteria undelivered). Confidence: 0.75
Given a new issue description or design spec, the agent determines which categories apply using a three-pass algorithm. Total target time: under 60 seconds.
Pass 1 — Keyword pre-filter (5 seconds)
Scan the issue/spec for terms that appear in any category's surface_features. Build a candidate set. This is deliberately over-inclusive: false positives are cheaper than false negatives here.
Keyword index (maintained in index.json):
{
"session": ["session-boundary-state", "gate-bypass-via-exit"],
"state": ["session-boundary-state", "worktree-isolation"],
"worktree": ["worktree-isolation"],
"grep": ["gate-false-positive"],
"regex": ["gate-false-positive"],
"test": ["coverage-without-behavior", "late-testability-binding"],
"mock": ["coverage-without-behavior"],
"schema": ["schema-coupling"],
"SKILL.md": ["skill-as-code-unverified"],
"instructions": ["enforcement-level-mismatch"],
"findings": ["artifact-lifecycle-gap", "session-boundary-state"],
"hook": ["gate-bypass-via-exit", "gate-false-positive", "worktree-isolation"]
}Pass 2 — Structural test application (30 seconds)
For each category in the candidate set, apply its structural test as a YES/NO question to the design. A structural test requires reading 1-3 sentences of the design. Categories where the answer is YES are confirmed; others are dropped.
This pass is the discrimination step. It is cheap because structural tests are questions, not analyses. An agent that cannot answer a structural test YES/NO within 10 seconds is either in unknown territory (see Pass 3) or the test is too vague and should be refined.
Pass 3 — Precursor signal scan (15 seconds)
Check for precursor signals in the issue title and first paragraph. Precursor signals are phrases that appear before the problem is fully articulated: "store X after phase Y," "multi-agent with shared," "deferred verification," "later." These catch categories that keyword matching misses because the vocabulary doesn't overlap.
The output of the three passes: a ranked list of confirmed categories (structural test YES) plus precursor-signal candidates (tentative). The agent loads the full category definition for each confirmed category before design evaluation begins. Tentative categories are held for hypothesis formation.
A hypothesis has this mandatory structure:
H: Design D will fail at seam S because property P holds in context C.
Falsification: Construct scenario F in which P holds. If D survives F, the hypothesis is refuted.
Minimum experiment: Trace system state after [specific failure condition]. Record what is lost.
A hypothesis is falsifiable if and only if: (1) there exists a specific scenario that, if it occurred, would prove the hypothesis wrong, and (2) that scenario can be constructed without building the full system.
A hypothesis is vague if it can absorb any outcome without updating. "This might have caching issues" is vague. "This cache will serve stale data because invalidation is synchronous and blocks the writer for 200ms, creating a window where nodes B and C read pre-write data" is falsifiable.
Protocol steps:
Step 1 — One hypothesis per candidate design. For each design option on the table, state one hypothesis about its primary failure mode. The failure mode must come from the loaded category's failure topology if the category matched. If no category matched, the failure mode must be derived from first principles.
Step 2 — Falsification attempt. Construct the specific scenario from the hypothesis. Walk the system state after that scenario executes. Ask: what has been written? what has been lost? is the loss acceptable? The scenario must be specific enough to complete the walk in under 3 minutes.
Step 3 — Outcome classification. If the walk shows the system survives: hypothesis refuted, design survives this round. If the walk shows failure: hypothesis confirmed, design is fragile against this failure mode. Fragile designs are not eliminated yet — they are scored against the category's other failure modes.
Step 4 — Rival hypothesis. For the surviving design(s), form one rival hypothesis: the best argument that this design will fail. Attempt to falsify the rival. If you cannot falsify the rival in 3 minutes of walking, treat the rival as a real risk and surface it explicitly in the design choice.
Step 5 — Dominance check. A design that fails more failure modes than its rival, without compensating advantages, is dominated. Dominated designs are eliminated. Only non-dominated designs reach the selection step.
The key constraint: try to prove yourself wrong. A falsification attempt that you approach hoping to confirm your preferred design is not a falsification attempt — it is a rationalization dressed as rigor. The discipline is to seek the failure, not to fail to find it.
The feedback loop converts session incidents into category library growth. It has four stages.
Trigger. After any /kaizen-reflect run that files a GitHub issue with label type:bug, type:state-resilience, type:regression, or type:pattern, the reflect skill runs a structured extraction step:
npx tsx src/cli-experience.ts add-entry --from-issue <N>This is mandatory when the label condition is met, not optional. The extraction fills the FSI schema (from r2-experience-accumulation.md) and appends to index.json.
Data recorded per incident.
{
"id": "fsi-<N>",
"github_issue": <N>,
"date": "<ISO>",
"matched_category": "<slug>",
"design_shape_that_failed": "<slug>",
"design_shape_preferred": "<slug>",
"seams_involved": ["<seam1>", "<seam2>"],
"recovery_cost": "low|medium|high",
"was_category_in_library": true|false,
"was_category_retrieved_at_design_time": true|false
}The was_category_retrieved_at_design_time field is the feedback loop's diagnostic instrument. If it is consistently false for incidents that match an existing category, the retrieval mechanism is failing, not the category library.
Processing — from incident to category update.
When a new FSI entry is added and its matched_category exists in the library: increment incidents array, update confidence (confidence = min(0.99, 0.5 + 0.1 * incident_count + 0.05 * recurrence_count)), update last_confirmed.
When the incident's matched_category does not exist in the library: flag for human review with the draft category structure pre-filled from the FSI schema. Human review (Aviad) approves or merges with an existing category.
Threshold for new category creation.
A new category is created when: (1) 3 or more FSI entries share the same design_shape_that_failed but do not match any existing category, AND (2) the failure mode is distinct from all existing categories' failure topologies. The threshold of 3 prevents one-off incidents from fragmenting the library.
Category retirement. A category whose last_confirmed is more than 180 days old and whose confidence has never exceeded 0.70 is flagged as speculative. Speculative categories are excluded from retrieval by default but preserved in the library for audit. This prevents false authority from stale categories.
Proportionality is a function of two variables: category match quality and category confidence.
Match quality: How cleanly does the current design map to a single category? Clean match (one category, structural test YES, precursor signals present) = high match quality. Ambiguous match (multiple categories, structural tests uncertain) = low match quality.
Category confidence: The empirical confidence field from the category definition. Derived from incident count, recurrence count, and recency.
The calibration matrix:
| Match quality | Category confidence | Investment level | Action |
|---|---|---|---|
| Clean | High (>0.80) | Near-zero | Load preferred shape, log one sentence, proceed |
| Clean | Moderate (0.50–0.80) | Low | Run one falsification cycle per candidate shape (5 min) |
| Ambiguous | Any | Medium | Run full hypothesis protocol for each confirmed category (15 min) |
| No match | N/A | High | Treat as novel territory: full exploration, create draft FSI entry (20-30 min) |
The critical property: cost scales with novelty relative to the library, not with the absolute complexity of the problem. A problem that appears complex but maps cleanly to a high-confidence category costs near-zero. A problem that appears simple but has no category match costs full exploration. This is exactly the behavior a staff engineer exhibits — familiar problems are fast, unfamiliar problems are slow, and the boundary shifts as experience grows.
Near-zero cost in practice: When match quality is clean and confidence is high, the agent logs: "Category: session-boundary-state (confidence: 0.87). Shape: accumulate-then-flush is fragile in this category. Proceeding with write-through shape." Total overhead: 15 seconds. No design pause. No falsification cycle. The category library has already done the work.
Where category recognition runs: At the start of /kaizen-implement, after the plan is retrieved but before any implementation begins. The plan text is the input to Pass 1 of the recognition algorithm. Category recognition is not optional for plans that touch state, hooks, tests, or skills — these domains have category coverage above 0.80 and should always trigger retrieval.
New files required:
.claude/kaizen/categories/ — one YAML file per category in the formal schema above. Bootstrap with the 10 categories in Section 2. File naming: cat-<slug>.yaml.
.claude/kaizen/categories/index.json — flattened keyword-to-category map for Pass 1, plus category slugs with confidence scores for DPS Recurrence weight computation.
src/cli-experience.ts — subcommands: recognize --text "<plan excerpt>" (runs Passes 1-2, outputs confirmed categories as markdown); add-entry --from-issue <N> (mandatory in reflect); query --category <slug> (loads full category for injection into agent context); update-confidence --id <slug> (recomputes from current incident count).
Changes to SKILL.md files:
/kaizen-implement — Add step 0 before all others: "Run category recognition: npx tsx src/cli-experience.ts recognize --text '<plan summary>'. Load confirmed categories. If any category matches, load its preferred shape before evaluating design options."
/kaizen-reflect — After gh issue create succeeds with FSI-eligible label: "Run npx tsx src/cli-experience.ts add-entry --from-issue <N>. This is mandatory, not optional. Confirm entry ID is written to index.json."
/kaizen-evaluate — Before scope assessment: "Run category recognition on the issue title and description. If category confidence > 0.80 and preferred shape is non-dominated, note this in the evaluation as prior evidence — do not re-derive what the library already knows."
No new schema is needed for hypotheses. Hypotheses are session-local reasoning artifacts. They are not persisted. The output of hypothesis formation is either a design choice (persisted to the plan) or a new FSI entry (persisted to the category library via reflect). The hypothesis itself is scaffolding that is discarded once it has served its function.
The library grows in one direction: from incidents toward prediction. The categories that matter most are the ones filed in anger at 2 AM, generalized in the post-mortem, and retrieved before the next 2 AM arrives.