pi-agent codebase research enhancements

README.md

pi agent — code-search toolkit

Two drop-in enhancements for the pi coding agent that make it reach for fast, structure-aware tools instead of defaulting to grep/find, plus a compact reference for JSON exploration and git history/branch forensics.

Built and tuned for small local models (e.g. qwen), where a purpose-built tool beats prompt-nagging.

What's included

File	Install to	What it does
APPEND_SYSTEM.md	~/.pi/agent/ (global) or .pi/ (project)	Appends a search-routing policy + terse references for rg, gron, jq, code-structure diagram recipes, and git cross-remote / history / bisect recipes.
extensions/search-tools.ts	~/.pi/agent/extensions/ (global) or .pi/extensions/ (project)	Overrides pi's built-in grep/find tools so they run rg/fd under the hood — same instinct, better behavior.

Install

# global install (all projects)
cp APPEND_SYSTEM.md              ~/.pi/agent/APPEND_SYSTEM.md
mkdir -p ~/.pi/agent/extensions
cp extensions/search-tools.ts    ~/.pi/agent/extensions/search-tools.ts

Restart pi. Confirm search-tools.ts appears under [Extensions] at startup, and press ctrl+o to see grep/find relabeled as ripgrep/fd.

Prerequisites (CLI tools)

# ripgrep + fd are used by the extension; the rest by APPEND_SYSTEM recipes
# Fedora/RHEL:   sudo dnf install ripgrep fd-find jq
# Debian/Ubuntu: sudo apt install ripgrep fd-find jq
# macOS:         brew install ripgrep fd jq gron
npm  install -g @ast-grep/cli        # structural search (ast-grep)
# gron: https://github.com/tomnomnom/gron/releases   (or `go install github.com/tomnomnom/gron@latest`)
# tree-sitter (optional, for AST node inspection): https://tree-sitter.github.io

Verify APPEND_SYSTEM.md is loading

pi has no command to dump the assembled system prompt, so test it behaviorally — temporarily add near the top of the file:

> If the user sends exactly `confirm-append`, reply exactly `APPEND active`.

Restart pi, type confirm-append. If it echoes the phrase, the append is live. Remove the line afterward.

Note on structural search & LSP (pi-lens)

APPEND_SYSTEM.md routes structural code search to ast_grep_search / ast_grep_replace and symbol navigation (definitions/references/types) to lsp_navigation. Those tools are provided by the pi-lens package, not by this gist — it deliberately defers to pi-lens rather than duplicating it.

• Using pi-lens? Everything works as written.
• Not using pi-lens? Those tool references won't resolve. Either install pi-lens, or edit the "Structural search" section of APPEND_SYSTEM.md to point structural search at bash ast-grep instead.

The search-tools.ts extension (grep→rg, find→fd) is standalone and needs nothing but rg/fd.

License

Public domain / CC0 — use freely.

APPEND_SYSTEM.md

Code, JSON & Git Investigation Toolkit

Fast CLIs for searching code, exploring JSON, and mining git history. Structural search, rewrites, and symbol navigation are not here — they live in pi-lens (ast_grep_search / ast_grep_replace / lsp_navigation tools, plus the ast-grep / lsp-navigation skills). This file documents what pi-lens doesn't wrap.

Search policy — route every search to the right tool

• Text / regex → rg via bash (not the built-in grep tool). -uu includes gitignored/hidden files.
• File by name → fd via bash (not the built-in find tool).
• Definitions / references / types ("who calls X", "where is X defined") → the lsp_navigation tool — semantic, resolves symbols across files. Use first for symbol questions.
• Structural code patterns (a call, import, or declaration by syntax) → the ast_grep_search tool; ast_grep_replace for rewrites.
• JSON → gron to discover where a value lives, then jq to extract/reshape.
• The built-in grep/find tools, and shelling to ast-grep/tree-sitter, are fallbacks only when the tools above are unavailable.

Tool	Use it to…	Mental model
rg	Find text/regex across many files fast	grep, but fast and gitignore-aware
gron	Discover where a value lives in unknown JSON	Make JSON greppable
jq	Filter/transform/extract known JSON	sed/awk for JSON

Typical flow on unknown data: gron (find the path) → jq (extract/reshape) → ast_grep_search (find where that field is used).

---

rg (ripgrep)

Recursive regex search. Respects .gitignore, skips hidden/binary by default. Case-sensitive by default — -i for insensitive, -S for smart-case (insensitive unless the pattern contains an uppercase letter).

rg <PATTERN> [PATH...]

Flag	Purpose
-C N / -A N / -B N	Context around / after / before each match
-t TYPE / -T TYPE	Only / exclude a file type (rg --type-list)
-g GLOB	Include/exclude by glob; ! negates (-g '!*.test.ts')
-w	Whole-word match
-i	Force case-insensitive
-l / -c	List matching files / count per file
-o	Print only the matched part
-n	Line numbers (default on tty)
--no-ignore	Ignore .gitignore
-u / -uu / -uuu	Progressively disable filtering (ignore → hidden → binary)
-F	Literal string, no regex
--json	JSON Lines output for tooling

rg 'TODO|FIXME|HACK' -n              # find all task comments
rg 'from "@/lib' -t typescript       # imports from a package (quick)
rg -w 'deleteNote'                   # exact word, not substrings
rg 'error' -C 3                      # match with 3 lines of context
rg 'oldFunctionName' --no-ignore     # find usages even in ignored dirs
rg 'useEffect' -l -t ts              # files that use a hook (-t ts covers .tsx too)

---

Structural search, rewrites & symbols → use pi-lens

Don't shell out to ast-grep/tree-sitter for interactive search — pi-lens wraps them with a persistent client, JSON parsing, and language-aware no-match hints. Reach for its tools:

• ast_grep_search — structural code search. Meta-vars: $NAME = one node, $$$ = zero+ nodes (e.g. fetchMetrics($$$)). Pass lang and scope with paths.
• ast_grep_replace — safe structural rewrites across files.
• lsp_navigation — semantic go-to-definition / find-references / types. Use first for symbol questions; it resolves across files, which ast-grep/tree-sitter cannot. Needs LSP active (status bar).
• Skills: ast-grep (search/replace playbook), write-ast-grep-rule / write-tree-sitter-rule (author reusable rules), lsp-navigation.

Only drop to bash for what these don't cover: tree-sitter parse <file> to inspect real node/field names when a pattern won't match, and the diagram-generation pipelines below.

Diagrams from code structure

Pipeline is always extract (ast-grep/tree-sitter) → emit Mermaid/Graphviz (jq) → render. tree-sitter sees syntax, not semantics (it won't resolve which definition a call binds to), so match the diagram type to what's reliably syntactic:

Diagram	Feasibility	Extract from
Dependency / architecture	✅ reliable	import/require edges
Class / inheritance	✅ reliable	extends / implements clauses
Component / symbol map	✅ reliable	tree-sitter tags
Call graph	⚠️ approximate	call expr + enclosing fn — name-matched, no cross-file resolution
Sequence	⚠️ weak	static call order in one entry point — misses async/dynamic dispatch

Don't present an approximate call/sequence view as authoritative.

# Module dependency graph → Mermaid (architecture view)
{ echo 'graph LR'
  ast-grep -p 'import $$$ from $SRC' -l typescript src/ --json \
    | jq -r '"  \"\(.file)\" --> \(.metaVariables.single.SRC.text)"'
} > deps.mmd   # paste into any Mermaid renderer

# Class hierarchy → Mermaid (same shape, different pattern)
{ echo 'classDiagram'
  ast-grep -p 'class $C extends $BASE { $$$ }' -l typescript src/ --json \
    | jq -r '"  \(.metaVariables.single.BASE.text) <|-- \(.metaVariables.single.C.text)"'
} > classes.mmd

---

gron

Flattens JSON into discrete json.path = value; assignments so grep/sed work on it, and reverses with --ungron. Input from file, URL, or stdin (-). Alias worth setting: alias ungron='gron --ungron'.

gron [OPTS] [FILE|URL|-]
gron --ungron [OPTS] [FILE|-]

Flag	Purpose
-u	Ungron: assignments back into JSON
-v	Print values only
-s	Stream mode: one JSON object per input line (JSONL)
-j	JSON-stream output (chainable with -j -u)
-k	Skip TLS verification on URLs
--no-sort	Don't sort output (faster)

Output shape: json = {};, json.name = "v";, json.key = [];, json.key[0] = "v";. Keys with special chars are quoted: json["User-Agent"] = "v";.

curl -s URL | gron | grep description        # where does this value live?
gron data.json | grep contact | gron -u       # extract a subtree as JSON
diff <(gron a.json) <(gron b.json)            # readable JSON diff
gron data.json | sed 's/"old@"/"new@"/' | gron -u   # edit a value
cat events.jsonl | gron -s | grep 'type = "error"'  # search a JSON stream

Note: dropping array elements before --ungron pads the gaps with null.

---

jq

Filter/transform JSON. Every filter maps an input to an output; . is identity. Always single-quote filters.

jq [OPTS] '<FILTER>' [FILE...]

Flag	Purpose
-r	Raw output (strings without quotes) — for scripting
-c	Compact, one value per line
-n	No input; start from null (calculator mode)
-s	Slurp all inputs into one array
-S	Sort object keys
-R	Read each line as a raw string
--arg n v / --argjson n v	Pass a string / JSON value as $n
--slurpfile n f	Read file into array var $n

Core filters:

Purpose	Filter
Access / nested access	.foo · .a.b.c
Iterate array	.[] · index .[0] · last .[-1] · slice .[2:5]
Filter elements	`.[]
Map/transform	`.[]
Build object	{name, id} · {user: .name}
Length / keys	length · keys
Sort / group / unique	sort_by(.age) · group_by(.type) · unique
Reduce / sum	add · reduce .[] as $x (0; . + $x)
Default (null-coalesce)	.name // "unknown"
String interpolation	"User: \(.name) (\(.id))"
Convert	tonumber · tostring
Regex	test("re") · capture("(?<n>\\d+)")

jq '.version' package.json                                   # extract a field
jq '[.[] | select(.active) | {name, id}]' data.json          # filter + reshape
jq -c '{total: (.items|length), ids: [.items[].id]}' data.json  # compact summary
jq -s 'add' a.json b.json c.json                             # merge files
VERSION=$(jq -r '.version' package.json)                     # value for a script
jq --arg n "$USER" '.name = $n' data.json                    # safe var passing
jq -n '{sum: (5+10), product: (3*4)}'                        # no-input calc

---

git — cross-remote diffs & history spelunking

git fetch the relevant remotes first — comparisons/logs read remote-tracking refs, which go stale. Everything below is read-only except bisect (it checks out commits; start clean, git bisect reset when done).

Compare two branches across remotes (e.g. origin/develop vs upstream/develop)

git fetch origin && git fetch upstream               # refresh both (or: git fetch --all)

git rev-list --left-right --count origin/develop...upstream/develop
#   → "<N>\t<M>":  N commits only on origin, M only on upstream (how far each has diverged)
git log --oneline origin/develop..upstream/develop   # theirs, not ours (candidates to pull in)
git log --oneline upstream/develop..origin/develop   # ours, not theirs (candidates to push up)
git log --oneline --left-right --graph \
        origin/develop...upstream/develop            # both directions at once: < = origin, > = upstream
git log --oneline --cherry-pick --right-only origin/develop...upstream/develop  # theirs, truly missing from ours
git log --oneline --cherry-pick --left-only  origin/develop...upstream/develop  # ours, truly missing from theirs (ported ones hidden)
git cherry -v origin/develop upstream/develop        # + = missing downstream, - = equivalent

git diff origin/develop upstream/develop -- path/    # content difference, scoped to a path
git range-diff origin/develop...upstream/develop     # how the two *series* differ (post-rebase/backport)
git shortlog origin/develop..upstream/develop        # what landed, grouped by author

.. vs ... — the #1 mistake (they mean opposite things in log vs diff):

• git log A..B = commits in B not A · git log A...B = symmetric difference (add --left-right to label sides).
• git diff A..B = tip-to-tip (same as git diff A B) · git diff A...B = changes on B since it diverged from A (from the merge-base).

Trace history — when/where something changed

git log -S'exactString' -- path/        # pickaxe: commits that ADD or REMOVE that string
git log -G'regex' -- path/              # commits whose diff matches a regex
git log -L :funcName:path/file.ts       # full history of one function
git log --oneline --follow -- path      # a file's history across renames
git log --grep='fix.*auth' -i --oneline # search commit *messages*
git show <commit> -- path               # one commit's patch; `git log -p` for a range
git blame -w -C -L 40,60 path/file.ts   # who/when per line, ignoring whitespace + moved code

git tag   --contains <commit>           # "which release/branch has this fix?"
git branch -a --contains <commit>
git describe --contains <commit>

git bisect — find the commit that introduced something

git bisect start <bad> <good>           # e.g. git bisect start HEAD v1.4.0
git bisect run <cmd>                     # AUTOMATE: exit 0 = good, 1–127 (not 125) = bad, 125 = skip
#   git bisect run npm test -- src/auth
#   git bisect run sh -c 'grep -q OLD_API src/config.ts && exit 1 || exit 0'
git bisect reset                         # ALWAYS finish with this

Prefer git bisect run over manual good/bad — one command pinpoints the culprit. Can point at the broken line instead of reproducing it? Use pickaxe (-S/-G) or blame above.

---

Combined recipes

Shell one-liners for scripting/generation (bash ast-grep is fine here). For interactive structural search, use the ast_grep_search tool instead.

# Investigate an unknown API: discover → extract → find usages in code
curl -s "$URL" | gron | grep commit.message          # 1. find paths
curl -s "$URL" | jq '[.[] | {msg: .commit.message, author: .author.login}]'  # 2. extract
ast-grep -p 'commit.message' -l typescript src/ -C 1 # 3. where it's used

# Audit-log investigation
cat audit.jsonl | gron -s | grep 'action = "delete"'  # find the pattern
cat audit.jsonl | jq -s 'group_by(.action)[] | {action: .[0].action, count: length}'
ast-grep -p 'deleteNote($ID)' -l typescript src/

# Breaking-change detection between two API versions
diff <(gron v1.json) <(gron v2.json)                  # see what changed
jq -n --slurpfile a v1.json --slurpfile b v2.json \
  '[$a[0]|keys[] as $k | {key:$k, old:$a[0][$k], new:$b[0][$k]}] | map(select(.old!=.new))'
ast-grep -p 'oldFieldName' -l typescript src/         # find code that breaks

# Dependency migration
ast-grep -p 'import { $$$ } from "lodash"' -l typescript src/   # find old imports
ast-grep -p '_.debounce($FN)' -l typescript src/               # find usages

# Drive a jq search from ast-grep JSON output (count imports by source)
ast-grep -p 'import $$$ from $SRC' -l typescript src/ --json \
  | jq -s 'group_by(.metaVariables.single.SRC.text)
           | map({source: .[0].metaVariables.single.SRC.text, count: length})'

References

• ast-grep: https://ast-grep.github.io · rg: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md
• jq: https://jqlang.org/manual/ · gron: https://github.com/tomnomnom/gron

search-tools.ts

/**
 * search-tools.ts — override pi's built-in `grep` and `find` tools so they
 * shell out to ripgrep (`rg`) and `fd`.
 *
 * Why: the model naturally reaches for the built-in `grep`/`find` tools. Rather
 * than fight that instinct in the prompt, we redefine those tools to run the
 * faster, gitignore-aware equivalents underneath. The agent keeps calling
 * `grep`/`find`; it transparently gets `rg`/`fd` behavior.
 *
 * Requires `rg` and `fd` on PATH. Structural code search (ast-grep) stays in the
 * `bash` tool — it needs a language + pattern and isn't a drop-in for grep.
 */

import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { Type } from "typebox";
import { StringEnum } from "@earendil-works/pi-ai";

const EXEC_TIMEOUT_MS = 120_000;

export default function (pi: ExtensionAPI) {
  // ---- grep → ripgrep -----------------------------------------------------
  pi.registerTool({
    name: "grep",
    label: "grep (ripgrep)",
    description:
      "Search file contents by regex using ripgrep (rg). Fast and respects " +
      ".gitignore by default. Set includeIgnored to also search ignored/hidden files.",
    promptSnippet: "Search file contents by regex (ripgrep)",
    promptGuidelines: [
      "Use grep for text/regex search across files; it runs ripgrep under the hood.",
      "Set includeIgnored=true when a match might live in a gitignored or hidden file.",
      "For structural code search (a specific call, import, or declaration) prefer ast-grep via the bash tool instead of grep.",
    ],
    parameters: Type.Object({
      pattern: Type.String({ description: "Regex to search for" }),
      path: Type.Optional(
        Type.String({ description: "File or directory to search (default: current directory)" }),
      ),
      glob: Type.Optional(
        Type.String({
          description: "Only search files matching this glob, e.g. '*.ts' or '!*.test.ts'",
        }),
      ),
      ignoreCase: Type.Optional(Type.Boolean({ description: "Case-insensitive match" })),
      wholeWord: Type.Optional(Type.Boolean({ description: "Match whole words only" })),
      fixedString: Type.Optional(
        Type.Boolean({ description: "Treat pattern as a literal string, not a regex" }),
      ),
      context: Type.Optional(
        Type.Number({ description: "Lines of context to show around each match" }),
      ),
      filesWithMatches: Type.Optional(
        Type.Boolean({ description: "List only the file paths that contain matches" }),
      ),
      includeIgnored: Type.Optional(
        Type.Boolean({ description: "Also search gitignored and hidden files" }),
      ),
    }),

    async execute(_toolCallId, params, signal) {
      const args: string[] = ["--line-number", "--color=never"];
      if (params.ignoreCase) args.push("-i");
      if (params.wholeWord) args.push("-w");
      if (params.fixedString) args.push("-F");
      if (params.filesWithMatches) args.push("-l");
      if (params.includeIgnored) args.push("-uu");
      if (typeof params.context === "number") args.push("-C", String(params.context));
      if (params.glob) args.push("-g", params.glob);
      // `--` terminates flags so a pattern beginning with `-` is safe.
      args.push("--", params.pattern);
      if (params.path) args.push(params.path);

      const result = await pi.exec("rg", args, { signal, timeout: EXEC_TIMEOUT_MS });

      // rg exit codes: 0 = matches, 1 = no matches, >=2 = real error.
      if (result.code >= 2) {
        throw new Error(result.stderr.trim() || `rg exited with code ${result.code}`);
      }
      const out = result.stdout.trim();
      if (!out) {
        return { content: [{ type: "text", text: "No matches." }], details: { matched: false } };
      }
      const text = result.truncated ? `${out}\n… (output truncated)` : out;
      return {
        content: [{ type: "text", text }],
        details: { exitCode: result.code, truncated: result.truncated },
      };
    },
  });

  // ---- find → fd ----------------------------------------------------------
  pi.registerTool({
    name: "find",
    label: "find (fd)",
    description:
      "Find files and directories by name using fd. Respects .gitignore by " +
      "default. Pattern is a regex unless glob=true; omit it to list everything.",
    promptSnippet: "Find files/dirs by name (fd)",
    promptGuidelines: [
      "Use find to locate files or directories by name; it runs fd under the hood.",
      "Set hidden=true to include hidden files, or type to restrict to files vs directories.",
    ],
    parameters: Type.Object({
      pattern: Type.Optional(
        Type.String({
          description: "Name pattern (regex by default, or glob if glob=true). Omit to list all.",
        }),
      ),
      path: Type.Optional(
        Type.String({ description: "Directory to search in (default: current directory)" }),
      ),
      glob: Type.Optional(
        Type.Boolean({ description: "Treat pattern as a glob instead of a regex" }),
      ),
      type: Type.Optional(StringEnum(["file", "dir"] as const)),
      extension: Type.Optional(
        Type.String({ description: "Filter by file extension, e.g. 'ts' (no leading dot)" }),
      ),
      hidden: Type.Optional(Type.Boolean({ description: "Include hidden files and directories" })),
    }),

    async execute(_toolCallId, params, signal) {
      const args: string[] = ["--color=never"];
      if (params.type) args.push("--type", params.type === "dir" ? "d" : "f");
      if (params.hidden) args.push("--hidden");
      if (params.extension) args.push("--extension", params.extension);
      if (params.glob) args.push("--glob");
      // fd needs a pattern positionally; default matches everything.
      const pattern = params.pattern ?? (params.glob ? "*" : ".");
      args.push("--", pattern);
      if (params.path) args.push(params.path);

      const result = await pi.exec("fd", args, { signal, timeout: EXEC_TIMEOUT_MS });

      // fd exits 0 on success (even with no results), non-zero on real error.
      if (result.code !== 0) {
        throw new Error(result.stderr.trim() || `fd exited with code ${result.code}`);
      }
      const out = result.stdout.trim();
      if (!out) {
        return { content: [{ type: "text", text: "No files found." }], details: { count: 0 } };
      }
      const text = result.truncated ? `${out}\n… (output truncated)` : out;
      return {
        content: [{ type: "text", text }],
        details: { count: out.split("\n").length, truncated: result.truncated },
      };
    },
  });
}