semble fork vs MinishLab matched benchmarks (2026-07-17, post-merge v0.5.2)

gist-benchmark-results-2026-07-17.md

semble fork vs MinishLab — matched benchmarks (2026-07-17)

Fork: mazulo/semble v0.5.2 (synced to MinishLab main + fork perf)
Upstream: MinishLab/semble current main
Repo under test: local checkout (~normandy), 14,691 files / 341,832 chunks
Matched defaults: _DESIRED_CHUNK_LENGTH_CHARS=750, SEMBLE_MODEL_NAME=minishlab/potion-code-16M-v2
Fork extras: semble-core (Rust BM25 tokenizer), parallel embed, fast cache mtime validate, search-path warmup after load

Profiler: uv run python -m benchmarks.profile_large_repo <repo>

---

Summary

Metric	Upstream	Fork	Δ
Cold build	90.8s	83.4s	-8%
Warm cache hit	6.09s	3.80s	-38%
embed_chunks	29.3s	26.7s	-9%
tokenize (BM25 prep)	7.0s	2.6s	-64%
Search first cold	42.9ms	20.7ms	-52%
Search warm p50	6.8ms	6.3ms	-7%
Search warm p90	8.1ms	6.9ms	-15%
Search warm p99	8.3ms	7.2ms	-13%

Fork wins all headline metrics on a fair (matched chunk size + model) comparison.

---

Upstream (MinishLab)

============================================================
  Profiling: …/normandy
  Files indexed: 14,691
  Chunks created: 341,832
============================================================

[COLD BUILD]  total: 90,790ms
  walk_files                      2,426ms    2.7%
  chunk_source (total)           34,527ms   38.0%
  embed_chunks                   29,302ms   32.3%
  tokenize (BM25 prep)            7,038ms    7.8%
  bm25_index                     10,132ms   11.2%
  (other)                         7,365ms    8.1%

[WARM RUN]    total: 6,087ms  (cache hit)

[SEARCH]
  first query (cold):  42.9ms
  warm  n=99      p50: 6.8ms  p90: 8.1ms  p99: 8.3ms

---

Fork (mazulo/semble v0.5.2)

============================================================
  Profiling: …/normandy
  Files indexed: 14,691
  Chunks created: 341,832
============================================================

[COLD BUILD]  total: 83,417ms
  walk_files                      2,257ms    2.7%
  chunk_source (total)           35,606ms   42.7%
  embed_chunks                   26,713ms   32.0%
  tokenize (BM25 prep)            2,556ms    3.1%
  bm25_index                      8,914ms   10.7%
  (other)                         7,371ms    8.8%

[WARM RUN]    total: 3,802ms  (cache hit)

[SEARCH]
  first query (cold):  20.7ms
  warm  n=99      p50: 6.3ms  p90: 6.9ms  p99: 7.2ms

---

Notes

• Earlier gist (e15866d…) was not matched (fork still on chunk 1500 / older model) — superseded by this run.
• Fork cold-search warmup + float32 cast addresses a one-shot GEMM cost after cache load (model2vec often returns float16).
• Parallel parser preload does not move chunk_source much on this mostly-Python tree; larger multi-language repos may still benefit.