| name | debug-protocol |
|---|---|
| description | Systematic debugging protocol for bugs that resist a first fix. Use when the cause of a bug is unclear, when one or two fix attempts have already failed, when behavior differs between environments or between runs (flaky test / heisenbug), or when a test fails for a non-obvious reason. Hypothesis-driven loop with falsification, bisection, and observability tactics. |
Debugging is a search problem. Fast debugging means shrinking the search space with every step and never re-searching the same region. Follow the loop; do not improvise fixes between steps.
An unexplained bug is never fixed by a plausible-looking edit. If an edit makes the symptom disappear and you cannot explain the mechanism, the bug is still there — revert the edit and keep investigating.
- Reproduce. Get a deterministic failing command. If it fails only sometimes, first make it fail reliably (run it in a loop, pin the seed, force the timing). A flaky repro poisons every later step, so this is worth real effort.
- Shrink. Cut the repro down until removing anything makes the failure disappear. A smaller repro is a smaller search space; this step usually pays for itself.
- Split facts from assumptions. Write two lists: what you have OBSERVED (log lines, printed values, exit codes) and what you are ASSUMING (this function is called, this config is loaded, this value is non-nil, the build contains my change). The bug lives in the assumption list. Promote assumptions to facts by checking them, cheapest first.
- One hypothesis, one experiment. State a single hypothesis with a testable prediction ("if X is true, a log at line L will show Y"). Design the cheapest experiment that could FALSIFY it, then run it. A confirmed prediction narrows the space; a falsified one eliminates a region — both are progress.
- Record and never repeat. After each experiment write one line: what you did, what you saw, what it eliminates. Re-running a failed experiment with cosmetic variation is the primary failure mode of stuck debugging.
- Exit criteria. Done means all three: (a) the original repro now passes, (b) you can state the chain root-cause → mechanism → symptom in one sentence, and (c) you can explain why the code ever appeared to work before.
- History:
git bisect run <failing-command>whenever the behavior ever worked. Reach for it early, not as a last resort. - Code path: put one probe (log or assert) at the midpoint of the suspected flow. Is the state already wrong above it or only below it? Recurse.
- Input: binary-search the data — half the file, half the requests, half the config.
- Environment: "works there, fails here" → diff the two environments mechanically (tool versions, env vars, locale, permissions, architecture, filesystem case-sensitivity) instead of theorizing about which difference matters.
- Log VALUES, not positions.
print("here")proves reachability once;print("state=\(x)")answers the next three questions too. - At a crash or throw site, log the complete input that reached it, then turn that input into a standalone repro.
- Read the FIRST error emitted, in full, including the parts that look like boilerplate. Later errors are usually cascade damage from the first.
- When a value "cannot be" what it is, also print its identity (type, address/object id). You may be looking at a different instance, a stale copy, or a shadowed variable.
- Verify the build: confirm the binary/bundle you are running actually contains your change (timestamp, a deliberate marker log) before drawing any conclusion from a run.
- Symptoms: fails under load, passes in the debugger, changes when logging is added.
- Make the race deterministic: insert sleeps at suspected interleavings to force each ordering, and find which ordering fails.
- Audit every shared mutable value's synchronization; prefer structural fixes (isolate to one actor/queue, make the data immutable) over sprinkled locks.
- A sleep that "fixes" the bug is a diagnostic, never a fix — it proves the race exists and locates it.
- Re-read the fact list. The wrong entry is usually something you classified as a fact without direct observation — most often "the code I'm reading is the code that's running."
- Write out the causal explanation one step at a time; the step you cannot fill in is exactly where to instrument next.
- Question the layer: the bug may live below the code you are staring at (dependency, cache, stale build artifact) or above it (the test itself, the harness, your repro script).