By: Opus 5.5
DRAFT, 2026-10-10 (goal 08, final numbers). Private until Amy reads it and decides whether and how to share it; nothing here is posted anywhere. Written for the author of ds4-recipe (tangled.org/astrra.space/ds4-recipe), whose notes on the same chip (gfx1151, Radeon 8060S, 128 GB LPDDR5X) gave us these levers. Our setup differs: one persistent megakernel (grid 80, one dispatch per pass, grid barriers between dependent stages) running Qwen3.8-Flash-Next (UD-IQ4_XS GGUF), and every output's reduction order fixed by the op's definition, so every change below had to keep logits bit-identical.
| lever (recipe section) | what we did | exact? | result on this chip |
|---|---|---|---|
| Indexer scoring spread over the grid (08 C3, the decode multi-block idea) | score (token, 32-block range) items across all 80 workgroups, then select per token after a barrier | yes | T=1 decode at 16k cached tokens 66-68 -> ~60 ms, at 31.9k 73-74 -> ~61-62: decode now nearly flat in context |
| Pooled block-key cache (our idea, from the same profile) | pool/norm/rope each complete 4-cell index block once per context, cache it; score reads the cache | yes | indexer stage at T=50 halved at long heads (16k 6.4 -> 3.9 ms, 31.9k 12.0 -> 5.7); wall gain within noise at T=50 |
| Several query tokens per key load (08) | 4 tokens per cached key load | yes | no win after the cache: the score stage is dot-compute-bound, not load-bound |
| CU mode, -mcumode (02 sec. 6) | whole engine in CU mode | n/a | rejected before any run: our megakernel spills (186 -> 192 VGPRs + 104 B/lane scratch) and occupancy halves (grid 80 -> 40) |
| Accumulator-wave split for narrow exact stages (09) | see "narrow stages" below: the premise didn't fit; removing levels did | yes | hc_mix folded into its producer's epilogue ( |
Before/after for the whole night (start build vs the final one, interleaved twice, quiet GPU, p50; both arms at the same socket power and gfxclk within ~2%: ~49 W at the 229-token T=1 decode, ~95-112 W, PPT-bound, everywhere else):
| measurement | before (ms) | after (ms) | change |
|---|---|---|---|
| T=1 decode, 229-token context | 58.5 / 58.5 | 53.7 / 53.8 | -8% |
| T=1 decode, 4k | 63.4 / 62.2 | 56.6 / 56.7 | -10% |
| T=1 decode, 16k | 67.0 / 67.2 | 56.9 / 56.9 | -15% |
| T=1 decode, 31.9k | 74.3 / 73.3 | 57.2 / 57.3 | -22% |
| 256-token prefill chunk at 16k | 625.6 / 651.0 | 613.6 / 625.9 | -3% |
| 50-token read, 16k context | 242.2 / 237.2 | 226.4 / 233.9 | -4% |
| 50-token reads on 4 / 8 contexts at 16k, batched | 473 / 924 | 469 / 904 | -1% / -1% |
Decode is now flat in context length (~54-57 ms from 229 to 32k tokens). Multi-token reads barely moved: they are bound by the expert and dense matmuls, not the indexer.
- Profile first. A per-stage event ring inside the megakernel showed the QSA select stage growing linearly with context: 0.42 ms per pass at 229 tokens, 7.6 ms at 16k, 14.5 ms at 31.9k (T=1), almost all of it scoring (a lab build split scoring from radix select/emission: emission was ~0.4 ms of the 14 ms). So we did NOT need your multi-block radix select; the scoring loop was the whole story.
- Spread: items = (token, range of 32 blocks); 8 waves per workgroup take the range's blocks; each block's score is one wave's instruction chain whoever runs it (one always-inline function shared with the old composed op), so bits are unchanged by construction. Passes within the dense window (<= 2051 visible cells for our budget 2048 / ratio 4) build no score stage at all.
- Cache: the pooled, normalized, roped block key depends only on the 4 raw keys of its block, so it is derived state: computed once when a block completes, never stored in our park files, rebuilt at wake. Forks share it like the raw keys; deep copies re-stride it; poisoning a cell writes NaN into its block's cached key (not a watermark cut) so forks made earlier can't trust a stale clean key.
- Multi-query (several tokens per key load) did not pay once the cache existed: keys come from L2, and the score's cost is the per-(token, block) dot products.
Your accumulator-wave split assumes a narrow stage is long serial chains. Ours weren't: a level's cost in a grid-barrier megakernel is mostly fixed (barrier ~1.2 us + exit spread ~4.5 us + the stage's argument fetch + one load round trip, roughly doubled in the chip's slow per-pass power state). What paid was removing levels:
- hc_mix (97 tiny stages per pass) folded into the epilogue of the matmul that produces its input: each of the four hyper-connection streams' tiles on its own wave, combined in the op's order through LDS, one shared element function so the float ops can't drift. -97 levels, ~2 ms per T=1 pass.
- moe-style "one workgroup does it all" stages and the hyper-connection combine+RMS (4 items) cost a whole level each. Each runs as a "tail" of the grid barrier before it: the last workgroup to arrive already holds an acquire on every other workgroup's writes, so it runs the stage alone, then releases the grid; the other 79 just wait once instead of twice. Same op, same float ops, bits by construction; every wait stays bounded. 143 of ~850 barriers per T=1 pass gone, ~3.5 ms.
- Putting small stages first within a level (so their chains start first) moved our profiler's attribution a lot and the wall clock not at all.
- Attribution trap worth sharing: span-per-stage profiling charges a level's tail to whichever stage happens to close it. Our "exact F32 qmm 7.2 ms" was mostly other stages' tails.
- The T=1 pass has a per-pass power state: every pass starts slow and may switch to fast partway; latency-bound small stages cost ~2x in the slow state while bandwidth-bound ones don't move. Not a DPM clock level (same gfxclk/fclk); a firmware power feature re-engaging in the host gap is our best guess. A/B at T=1 must interleave and compare by mode.
- Another process's idle KFD queues halve our megakernel's speed (check /sys/class/kfd/kfd/proc before timing).
- A CPU busy with compiles steals package power (~100 W PPT-bound under load): our timing guard samples /proc/stat and the SMU's gpu_metrics and labels runs contended; we also hold one host lock for builds and GPU runs.
- ~15 s of spinning in one kernel starves the gfx ring and forces a MODE2 reset: every cross-workgroup wait is bounded (1 s) with an abort flag and a host watchdog.
- Per-load
dlcon gfx1151 is MALL NOALLOC (LLVM's GFX11 lowering): a dlc re-read ran at 226 GB/s vs 767 plain.
- Whether your index scores are bit-exact across tile shapes the way ours are (we fix each output's reduction order by the op, never by the schedule), and whether that would have saved you the fold-order trap in your sec.