https://github.com/increpare/PuzzleScript/issues/947

info.md

@ -0,0 +1,465 @@

CodeMirror large-file highlighting correctness and responsiveness

Status

Design recommendation after correctness investigation, upstream review, and browser performance testing. The currently retained feature-worktree implementation is the minimal unsliced eager repair; the recommended time-sliced implementation described below has been benchmarked experimentally but has not yet replaced that patch.

Summary

PuzzleScript issue #947 reports that fast navigation through a very large source file can leave chunks with permanently incorrect syntax highlighting. The problem has two related but distinct aspects:

1. Parsing and highlighting a large PuzzleScript source file is expensive. This explains browser warnings such as slow keydown and setTimeout handlers and makes the race easy to trigger.
2. CodeMirror 5 can cache styles generated from approximate parser state. When its precise sequential worker later passes those lines while they are off-screen, it does not normally replace the cached styles. This is why some chunks stay wrong after the parser has caught up.

The permanent visual error is therefore a cache-correctness problem exposed by parser cost, not merely a slow parser.

The simplest correctness fix eagerly repairs every previously cached line when the precise worker reaches it. It is only a two-line control-flow change, but performance testing confirms that it can do substantial redundant work. In the strongest benchmark, it re-highlighted 1,471 cached lines even though every resulting style signature was identical to the cached signature.

The recommended compromise is still eager repair, but limited to a soft 25 ms repair budget per worker callback. A repair-specific yield should schedule the next callback with zero delay, allowing the browser to process rendering and input between callbacks without inserting CodeMirror's normal 100 ms background delay between every repair slice.

User-visible symptoms

The reported console output included warnings of this form:

inputoutput.js:417 [Violation] 'keydown' handler took 151ms
inputoutput.js:417 [Violation] 'keydown' handler took 166ms
[Violation] 'setTimeout' handler took <N>ms
codemirror.js:233 [Violation] 'setTimeout' handler took 51ms
codemirror.js:233 [Violation] 'setTimeout' handler took 70ms
codemirror.js:233 [Violation] 'setTimeout' handler took 63ms
codemirror.js:233 [Violation] 'setTimeout' handler took 88ms
inputoutput.js:417 [Violation] 'keydown' handler took 222ms
inputoutput.js:417 [Violation] 'keydown' handler took 227ms

In the editor, a fast jump into the middle of a large file can initially render a section using parser state reconstructed from too little preceding context. PuzzleScript's mode may then assign prelude or error styles such as cm-METADATA and cm-ERROR to object definitions. If the user leaves before the sequential worker reaches that region, returning later may continue to show the wrong styles indefinitely.

This also has functional implications. PuzzleScript uses token classes such as cm-LEVEL for editor behavior including Ctrl/Cmd-click navigation, so stale token classes are not purely cosmetic.

Environment and upstream investigation

PuzzleScript vendors CodeMirror 5.65.21, the final CodeMirror 5 release line at the time of this investigation.

The upstream sources reviewed were:

• CodeMirror 5 GitHub issues
• The newer self-hosted CodeMirror 5 repository
• CodeMirror issue #3839, the closest upstream report concerning temporarily approximate highlighting

No general CodeMirror 5 fix was found for a line that caches approximate styles, goes off-screen, and is then passed by the precise worker without cache repair. The relevant worker behavior is effectively unchanged from the implementation introduced in 2017. The newer repository did not add per-line style provenance or another mechanism that solves this case.

This is not widely visible in ordinary CodeMirror modes because many modes have compact local state. PuzzleScript's mode is unusually global: its parser state includes the current section, nested-comment state, declared objects, legend entries, sounds, collision layers, rules, win conditions, levels, and multiple name tables. Restarting near an arbitrary viewport can therefore produce a materially incorrect state.

Relevant CodeMirror state

CodeMirror tracks precision globally through frontiers, but not on individual cached lines.

State	Meaning
doc.highlightFrontier	The first line not yet processed by the sequential highlight worker
doc.modeFrontier	How far mode state is considered reliably known
line.styles	Cached token boundaries and CSS classes for one line
line.styleClasses	Cached line-level background/text classes
line.stateAfter	An optional saved parser checkpoint after the line

A line does not record whether its styles were generated:

• from the precise sequential worker;
• from a saved precise checkpoint;
• from a short approximate reconstruction for a distant viewport; or
• before or after a particular highlighting frontier.

There is no existing approximate, provisional, or equivalent per-line flag. Once present and current for modeGen, line.styles is treated as usable regardless of how it was produced.

Root cause

The failure sequence is:

1. The sequential worker has precisely highlighted only the beginning of the document.
2. The user jumps far beyond highlightFrontier.
3. getLineStyles calls getContextBefore to reconstruct mode state for the requested viewport.
4. CodeMirror limits how far it scans backward and may begin from an approximate state.
5. The visible lines are highlighted from that state and the results are cached in line.styles and line.styleClasses.
6. The user scrolls away.
7. The sequential highlightWorker eventually reaches those lines with precise state.
8. Because they are now above display.viewFrom, the worker uses its lightweight off-screen processLine branch. It advances state and saves checkpoints but does not replace the existing style cache.
9. highlightFrontier advances beyond the region.
10. Returning to the region reuses its existing line.styles; no later worker pass is scheduled for it.

Parser speed determines how large the race window is. It does not explain why waiting longer fails to repair the line after the frontier has passed it.

Goals

• A line whose cached styles were generated from approximate state must eventually receive precise styles.
• Revisiting a skipped region must show the correct token classes.
• Lines never rendered by the user should retain CodeMirror's lightweight off-screen parsing path.
• Repair work must not introduce new long main-thread tasks during large-file highlighting.
• Ordinary scrolling before worker catch-up should remain unchanged.
• The fix should remain local to CodeMirror highlighting and avoid redesigning the PuzzleScript parser.

Non-goals

• Making the parser itself faster.
• Eliminating every browser long-task warning already present in the unmodified worker.
• Replacing CodeMirror 5 or converting the mode to CodeMirror 6.
• Reworking PuzzleScript's global parser state.
• Adding a broad editor-level refresh loop.

Approaches evaluated

1. Unsliced eager repair

When the precise worker sees an off-screen line with an existing style cache, run the full highlightLine path and replace the cache. Continue using processLine for off-screen lines without cached styles.

The minimal implementation changes the worker condition from:

if (context.line >= cm.display.viewFrom) {

to:

if (context.line >= cm.display.viewFrom || line.styles) {

and prevents off-screen repairs from registering unnecessary display changes:

if (context.line >= cm.display.viewFrom && ischange) {
  changedLines.push(context.line);
}

Advantages:

• Directly repairs the stale cache at the moment precise state becomes available.
• Very small and easy to audit.
• Revisit cost remains at baseline.

Disadvantages:

• Re-highlights every cached off-screen line, even when its approximate result was already correct.
• Can turn a normal worker callback into a long main-thread task.
• Does not distinguish provisional caches from precise caches because CodeMirror stores no provenance.

2. Lazy cache invalidation

When the precise worker passes a cached off-screen line, clear its cache and continue with lightweight state processing:

if (line.styles) {
  line.styles = null;
  line.styleClasses = null;
}
if (line.text.length <= cm.options.maxHighlightLength) {
  processLine(cm, line.text, context);
}

The line is re-highlighted only if the user returns to it.

Advantages:

• Background catch-up stays near the unmodified worker's cost.
• Avoids full token-style generation for lines never revisited.
• Simple implementation.

Disadvantages:

• Moves the expensive work into user-driven scrolling.
• Invalidates caches that may already be correct.
• A revisit can reconstruct styles for many newly visible lines in one input or rendering action.

The benchmark rejected this approach for PuzzleScript: median per-jump revisit cost rose from approximately 2.8 ms to 17 ms, with p95 and maximum calls near 33-34 ms.

3. Eager repair sliced with the normal workDelay

Keep eager repair but impose a soft 25 ms budget on callbacks that repair cached off-screen lines. On expiry, schedule the next callback through the normal CodeMirror path using workDelay, which defaults to 100 ms.

Advantages:

• Breaks the repair into sub-50 ms tasks in the benchmark.
• Revisit cost remains at baseline.

Disadvantages:

• Total repair CPU is unchanged.
• Eight or nine repair slices incur eight or nine 100 ms gaps.
• The tested file took about one second to settle, leaving stale caches present longer than necessary.

4. Eager repair sliced with immediate continuation

Use the same soft 25 ms repair budget, but schedule a repair-specific continuation with zero delay. A yield still ends the current JavaScript task, giving the browser a scheduling opportunity before the next slice. Normal worker yields unrelated to cached-line repair continue to use the configured workDelay.

The experimental control flow was:

var end = +new Date + cm.options.workTime;
var repairEnd = +new Date + Math.min(cm.options.workTime, 25);

// Inside the worker iteration:
var repairedCachedLine = context.line < cm.display.viewFrom && !!line.styles;

// Run the eager full-highlight branch for visible or cached lines.

var now = +new Date;
var repairYield = repairedCachedLine && now > repairEnd;
if (now > end || repairYield) {
  startWorker(cm, repairYield ? 0 : cm.options.workDelay);
  return true;
}

This is a soft deadline: CodeMirror checks it between lines, so one unusually expensive line may overrun it. It does not make the parser or highlighter faster; it distributes the same CPU work across multiple tasks.

Advantages:

• Preserves eager correctness and baseline revisit speed.
• Produced no greater-than-50-ms worker tasks in the paired benchmark.
• Settled substantially faster than both unsliced eager repair under normal scheduling and slicing with the 100 ms delay.

Disadvantages:

• More control-flow code than the minimal two-line patch.
• Performs the same total re-highlighting work as unsliced eager repair.
• Zero-delay continuations create a short CPU burst, although the browser regains control between slices.

5. Add explicit style provenance

A deeper solution would mark a line's cached styles as provisional when getLineStyles produces them beyond the precise frontier, then repair or invalidate only provisional caches.

This most directly addresses redundant work, but it is a broader CodeMirror modification. Every path that creates, copies, invalidates, edits, splits, joins, or changes the mode generation for a line would have to maintain the flag correctly. It should be considered a separate follow-up rather than folded into the narrow correctness fix without its own design and tests.

6. Refresh from PuzzleScript's editor.js

A viewportChange handler could force a refresh or clear visible styles. This avoids editing vendored CodeMirror code but duplicates scheduler behavior, may redraw correct lines repeatedly, and still needs internal knowledge to distinguish approximate and precise state. It does not address the underlying cache lifecycle and was rejected.

Correctness testing

A deterministic browser test uses full CodeMirror with a small stateful mode:

1. The mode's expected style changes after 200 lines of parser state.
2. Line 250 is rendered while the sequential frontier is still near the start, caching an approximate style.
3. The viewport moves below line 250.
4. The captured sequential worker is run until it settles.
5. The test queries line 250 again and expects precise.

Observed results:

Variant	Result
Unmodified CodeMirror	FAIL: offscreen cache remained approximate
Unsliced eager repair	Pass
Lazy invalidation	Pass
25 ms sliced eager repair	Pass
25 ms sliced repair with immediate continuation	Pass

The existing PuzzleScript Node suite was also run against the retained eager patch:

Passed:  750
Failed:  0
Errors:  0
Total:   750 tests

Performance benchmark methodology

Test document

The benchmark used src/demo/easyenigma.txt, the longest file in src/demo:

• 1,711 source lines;
• 1,712 lines reported by CodeMirror because of the final line boundary.

Workload

The extended harness:

1. Loaded editor.html?demo=easyenigma in a 1,100 by 720 pixel iframe.
2. Captured and blocked CodeMirror's delayed worker before fast navigation.
3. Jumped across 16 widely separated viewport positions in five alternating traversals, for 81 synchronous viewport jumps.
4. Left the viewport at the bottom of the document.
5. Observed a stable frontier at line 132 with 1,471 cached lines beyond it.
6. Measured worker catch-up separately from the first post-catch-up revisit.
7. Repeated the experiment with normal workTime=100 and workDelay=100 scheduling.
8. Used PerformanceObserver to record browser long tasks greater than 50 ms.
9. Separated concurrent revisit jumps by animation frames so sixteen sub-50-ms scroll actions were not incorrectly reported as one synthetic long task.
10. Ran five rotating trials so each primary variant occupied each ordinal position and environmental drift was shared.

Two measurements were used:

• Isolated: worker timers were captured and drained synchronously. This measures total parsing/highlighting CPU and individual callback duration without scheduler delays.
• Normally scheduled: CodeMirror used its normal timer behavior. This measures wall-clock settling, long tasks, first-revisit cost, and end-to-end time.

Style signatures for all originally cached lines were saved before catch-up and compared afterward. This distinguished unchanged caches, changed caches, and invalidated caches without forcing style recomputation during measurement.

Preliminary old-versus-eager benchmark

An earlier, smaller paired harness cached approximately 1,018-1,021 lines ahead of a frontier near lines 52-54. It produced these five-trial results:

Measurement	Old CodeMirror	Unsliced eager fix	Difference
Isolated catch-up median	28.3 ms	84.1 ms	+56.0 ms, about 3x
Normally scheduled settle median	129.3 ms	190.4 ms	+60.9 ms, about 47%
End-to-end paired median delta	-	-	+69.2 ms, about 2.5%
Long tasks above 50 ms	0 of 5 trials	5 of 5 trials	One 83-88 ms task per trial

Fast scroll-call medians and p95 values were effectively unchanged. This established that the two-line eager patch primarily increases background catch-up cost rather than the cost of the scroll calls that create the caches.

Extended benchmark results

All numbers below are medians of five trials unless stated otherwise.

Initial fast scrolling

Before the worker was allowed to catch up, the variants were effectively indistinguishable:

Variant	Median scroll call	Median-of-trials p95
Old CodeMirror	3.0 ms	30.9 ms
Unsliced eager repair	3.1 ms	30.0 ms
Lazy invalidation	3.1 ms	28.6 ms
25 ms sliced eager repair	2.9 ms	29.0 ms

The repair strategy does not affect the initial fast sweep because the worker is blocked during that phase.

Isolated worker CPU and first revisit

Variant	Catch-up CPU	Worker callbacks	Median maximum callback	Revisit total	Median revisit call	Revisit p95	Revisit maximum	End-to-end
Old CodeMirror	67.3 ms	1	67.3 ms	35.3 ms	2.2 ms	3.0 ms	4.0 ms	788.6 ms
Unsliced eager repair	224.1 ms	1	224.1 ms	35.0 ms	2.2 ms	3.1 ms	3.8 ms	932.8 ms
Lazy invalidation	68.3 ms	1	68.3 ms	279.5 ms	15.9 ms	31.8 ms	32.2 ms	1,013.6 ms
25 ms sliced eager repair	229.3 ms	9	29.7 ms	35.9 ms	2.2 ms	3.3 ms	3.7 ms	930.4 ms

The sliced variant's median maximum callback was 29.7 ms; individual-trial maxima ranged as high as 45.8 ms because the budget is checked between lines and ordinary state processing may occur before a cached repair.

Paired median differences from old CodeMirror were:

Variant	Catch-up delta	Revisit delta	End-to-end delta
Unsliced eager repair	+156.4 ms	-0.3 ms	+144.2 ms
Lazy invalidation	approximately 0 ms	+243.1 ms	+230.8 ms
25 ms sliced eager repair	+158.6 ms	-1.4 ms	+139.6 ms

Slicing changes callback shape, not total CPU. Lazy invalidation preserves catch-up CPU but transfers the same class of expensive style generation into scrolling.

Normal CodeMirror scheduling

Variant	Settle time	Revisit total	Revisit maximum call	End-to-end	Long tasks	Median maximum long task
Old CodeMirror	86.3 ms	45.4 ms	4.9 ms	883.0 ms	1	68 ms
Unsliced eager repair	454.4 ms	46.1 ms	4.6 ms	1,261.8 ms	2	101 ms
Lazy invalidation	87.5 ms	293.8 ms	33.9 ms	1,096.3 ms	1	70 ms
25 ms slices with normal 100 ms delay	991.0 ms	45.7 ms	4.6 ms	1,788.1 ms	0	0 ms

Paired median differences from old CodeMirror were:

Variant	Settle delta	Revisit delta	End-to-end delta
Unsliced eager repair	+366.5 ms	+0.2 ms	+342.2 ms
Lazy invalidation	+1.2 ms	+248.4 ms	+220.6 ms
25 ms slices with normal delay	+906.4 ms	+0.9 ms	+922.8 ms

The baseline itself produced a long task in this stronger workload, confirming that parser/state-processing speed accounts for part of the console warnings. Unsliced repair adds substantially more work and more long tasks. Lazy invalidation does not improve the baseline worker task and makes later input/rendering much slower. Ordinary delayed slicing removes long tasks but takes nearly a second to settle because of repeated 100 ms gaps.

Immediate-resume slicing follow-up

The best sliced candidate was rerun in five alternating paired trials against old CodeMirror:

Variant	Settle time	Revisit total	End-to-end	Long tasks	Median maximum long task
Old CodeMirror in paired run	85.8 ms	45.7 ms	909.9 ms	1	71 ms
25 ms slices, immediate repair continuation	269.5 ms	47.1 ms	1,079.3 ms	0	0 ms

Paired median differences for immediate-resume slicing were:

• settle time: +183.7 ms;
• first-revisit total: -0.3 ms;
• end-to-end time: +146.2 ms.

Immediate continuation therefore retained the responsiveness benefit of slicing while avoiding approximately 800 ms of scheduler gaps. It also settled faster than the unsliced eager repair in the normal-scheduling benchmark because unsliced repair hit CodeMirror's 100 ms work limit and then incurred a normal 100 ms delay before continuing.

Redundant-work evidence

The performance workload deliberately stressed cache volume rather than reproducing a particular visibly stale block. Its cached-style comparison found:

Variant	Originally cached	Unchanged after catch-up	Changed after catch-up	Invalidated after catch-up
Old CodeMirror	1,471	1,471	0	0
Unsliced eager repair	1,471	1,471	0	0
Lazy invalidation	1,471	46	0	1,425
Sliced eager repair	1,471	1,471	0	0

Thus, in this particular scroll sequence, eager repair spent approximately 157-162 ms of additional CPU re-generating style arrays that were byte-for-byte equivalent to the existing caches. The deterministic regression test separately proves that approximate and precise styles can differ and that repair is necessary in the failing race.

After the 16-position revisit, lazy invalidation had 894 of the original lines cached again and 577 still invalidated. Every regenerated signature in the visited regions matched its original signature. This demonstrates both sides of the tradeoff: without provenance, neither eager repair nor invalidation knows which caches actually need correction.

Interpretation

The console delays are indeed dominated by parsing, state copying, and token-style generation:

• old CodeMirror needed about 67 ms of isolated CPU to advance the precise state through the benchmark document;
• eager cache repair increased that to about 224-229 ms;
• slicing left the total at about 229 ms but prevented one callback from holding the main thread for the whole duration.

The permanent stale-highlighting bug is nevertheless not just parser speed. The key correctness failure is that CodeMirror advances the precise frontier past an off-screen line while leaving an existing approximate style cache in place.

Recommended design

Implement eager repair with a repair-specific soft time budget and immediate continuation:

1. Preserve the current full-highlight path for visible lines.
2. Use the same path for an off-screen line only when it already has a style cache.
3. Keep lightweight processLine behavior for uncached off-screen lines.
4. Do not register display changes for repaired off-screen lines.
5. Limit repair-bearing callbacks to a soft 25 ms budget.
6. When that repair budget expires, schedule the next worker callback with zero delay.
7. When only CodeMirror's normal workTime expires, retain the configured workDelay.
8. Preserve maxHighlightLength, styleClasses, stateAfter, highlightFrontier, and modeFrontier behavior.

This option is preferred because it:

• fixes the stale cache before the user revisits it;
• keeps revisit scrolling at baseline speed;
• avoids new repair long tasks in the measured workload;
• settles much faster than slicing with the normal 100 ms delay; and
• is still localized to highlightWorker.

The cost is unavoidable redundant CPU until CodeMirror gains style provenance. That follow-up may ultimately be better for extremely large files, but it is more invasive and should not be implemented casually in vendored CodeMirror.

Risks and mitigations

Soft budget overruns

The deadline is checked between lines, so a single expensive line may exceed 25 ms. Keep maxHighlightLength behavior unchanged and retain browser long-task coverage in the regression harness.

Zero-delay timer burst

Immediate continuations can create a short burst of several worker tasks. Each task still yields to the browser event loop. Only repair-specific yields should use zero delay; ordinary worker throttling should continue to honor workDelay.

Vendored CodeMirror divergence

The change modifies vendored CodeMirror 5. Keep the patch narrow, document it next to the worker, and preserve a deterministic browser regression test so a future vendor update cannot silently restore the bug.

Cache provenance remains absent

The recommended fix cannot know whether an existing cache is actually stale. A later provenance design should explicitly specify flag lifetime across edits, line splitting/joining, mode changes, overlays, document replacement, and history operations.

Acceptance criteria

• The deterministic stale-cache test fails on unmodified CodeMirror and passes with the repair.
• A repair-specific continuation is scheduled as a separate JavaScript task after the soft budget expires.
• Repair-specific continuations do not wait for the normal 100 ms workDelay.
• Uncached off-screen lines continue to use processLine.
• Off-screen repairs do not trigger display invalidation.
• First-revisit median and p95 remain close to old CodeMirror.
• The benchmark introduces no greater-than-50-ms repair tasks in five trials of easyenigma.
• The existing 750-test PuzzleScript suite passes.

Implementation state at the end of the investigation

The feature worktree retains the minimal unsliced two-line eager patch and the deterministic browser regression test for inspection. The lazy and sliced candidates were tested in disposable worktrees and removed after measurement. No benchmark harness, local server, or disposable worktree remains. The recommended immediate-resume sliced design still requires a production regression test for its scheduling contract before replacing the minimal patch.