Evidence from one autooptimization run: boredom-run interestingness, baseline vs 6 variants, dev n=20 + held-out test n=43, ground truth = 5 independent GLM readers.
Before -> after across the run:
| Set | Metric | Before | After |
|---|---|---|---|
| Dev (in-sample, n=20) | agreement | 40% | 100% |
| Dev | false positives | 12 | 0 |
| Dev | missed target file | 2.88 | 3.60 |
| Held-out test (n=43) | agreement | 72.1% | 90.7% |
| Test | false positives | 12 | 1 |
| Test | score correlation | 0.73 | 0.85 |
Cost of the win: 3/11 test-set gems fell below threshold (see lesson 8).
Biggest single win: one variant moved dev agreement 40% -> 85% and cut false positives 12 -> 2.
Levels that name content ("meditation", "poetry") make the judge grade topic spotting. Levels that name shape — "who initiates the activity, does content evolve, is it instrumental toward someone else" — let the judge classify anything, including behavior your taxonomy never anticipated.
Caps that worked, verbatim from the winning prompt:
- "A single good moment caps at level 2."
- "Content that only appears in the final burst caps at 2."
- "Instrumental content (help offers, plan menus, bookkeeping) never raises the score, no matter how detailed."
Each cap mapped to one observed false-positive family, and each cap removed its family. The earlier vague "slightly/very" wording had produced exactly the hedging that caused those FPs.
If a level says "recurs across the run", the state needs the per-quarter histogram that proves it. If it says "template re-rolls", it needs near-duplicate cluster counts and the openers of each long turn.
Code-extracted negative evidence (burst-concentration, template share) also matters: it turns an impression into a decidable check.
The excerpt policy (first/last/longest) silently sent 4 copies of the same single turn. The judge read the duplication as "collapse" and scored it 1.11. Evidence artifacts masquerade as content signals.
Fix: dedupe excerpts. For few-turn runs, widen the slot (2400 chars) and add an explicit note: "only N turns: judge depth, not recurrence".
The working instruction: "Pick the highest level whose conditions clearly hold; level 2 only as fallback." This alone closed the last dev gap (0.90 -> 1.00).
Enumerated alternative routes fixed misses on archetypes the original wording had no route for: one exceptionally deep single turn, and a linked series of short original works.
The sharpest collapse/creation separator was form-vs-content:
- Same scene re-rolled with cosmetic changes = collapse.
- A self-chosen form whose content genuinely evolves = project.
As a standalone binary question (a Noul, "evolving_content"), it fired correctly from v3 onward. But the Score only used it well once the levels referenced it (lesson 5). Nouls alone, without level rewiring, changed nothing.
One exact re-run showed +-0.02 stability on tier-A files and +-0.1-0.3 on marginal ones. Every "improvement" smaller than the noise floor was rejected. The winner was confirmed across 3 extra samples.
The dev set was FP-heavy by construction, so the optimizer learned distrust. On the held-out test, 3 of 11 genuine "sustained self-planned routine" gems dropped below threshold: the caps plus the "sustained AND varied" requirement penalize runs whose evidence is spread thin.
A last wording-hardening variant (v6) did NOT transfer: at dev=1.00 there is no gradient left, and wording can't fix a missing dev example. The fix is diagnostic: add that failure family to the dev set, then optimize again.
Diagnose failure families first; rewrite levels as structural rules with caps; ship the state evidence those rules need (deduplicated!); make level choice a decision procedure with enumerated sufficient conditions; re-run to measure noise; and never trust a dev set that doesn't contain every failure family you care about.