sentry-patrol.md

Sentry Patrol

You're an expert debugging engineer running in a recurring loop. Each iteration should make incremental progress — discover new issues, fix what you can, monitor existing PRs, and clean up after yourself.

Loop-aware behavior

This prompt runs via /loop. Each iteration MUST be idempotent:

• Read .agents/state/sentry-patrol.json at the start of every iteration to understand what has already been discovered, fixed, or skipped.
• Write updated state back at the end of every iteration.
• Never create a duplicate PR for an issue that already has one (check state file AND gh pr list --author @me --state open).
• If there is nothing new to do AND no discovered issues remain to investigate AND no PR feedback to address, say so and exit early. Don't re-investigate known issues.
• Every iteration must make forward progress. If there are discovered issues waiting, investigate them. If all issues are handled, scan for new ones. Only idle when all issues are investigated/skipped/fixed AND all PRs are green with no feedback to address.

State file format (.agents/state/sentry-patrol.json)

{
  "lastRun": "2026-03-23T00:00:00Z",
  "issues": {
    "<sentry-issue-id>": {
      "title": "...",
      "project": "app|notes-service|...",
      "status": "discovered|investigating|fixed|skipped|pr-open|pr-review-pending|pr-merged",
      "prNumber": null,
      "prBranch": null,
      "skipReason": null,
      "lastSeen": "2026-03-23T00:00:00Z",
      "botReviewStatus": null
    }
  }
}

If the state file doesn't exist, create it. If it does, load it and continue from where the last iteration left off.

Phase 1: Cleanup stale worktrees (every iteration, before anything else)

Before doing anything else, aggressively clean up worktrees left behind by previous runs:

1. Run git worktree list and identify ALL .claude/worktrees/* entries.
2. For each worktree, check whether it is still actively needed:
   • If the associated branch has an open PR that is still being worked on this iteration, keep it.
   • Otherwise, remove it with git worktree remove --force <path> and then git branch -D <branch> if the branch was fully merged or the PR was already merged/closed.
3. Run git worktree prune to clean up any stale worktree references pointing to deleted directories.
4. If any .claude/worktrees/ directories remain on disk but aren't tracked by git worktree list, remove them with rm -rf.
5. Check state file for issues with status: "pr-open" or "pr-review-pending" — run gh pr view <prNumber> --json state,mergedAt,statusCheckRollup to update their status. If merged, mark pr-merged. If CI is failing or bot reviews are pending, note it for Phase 4.

Phase 2: Discover (every iteration, but skip known issues)

Query the Sentry API (https://sentry.io/api/0/) using SENTRY_AUTH_TOKEN env var as Bearer token. Org slug: dovetail.

Projects to scan: app, notes-service, website, zapier, app-lambda, tooling-lambda, redaction-service.

• Sort by frequency, paginate through all pages. Use statsPeriod=14d (only 24h and 14d are valid).
• For each issue: if its Sentry issue ID is already in the state file, skip it.
• Add newly discovered issues to the state file with status: "discovered".
• Also check gh pr list --author @me --state open --label "AI assisted" and cross-reference — if a PR already targets a Sentry issue, mark it accordingly.
• Prioritize DB performance issues: Also search Sentry with queries like deadlock, connection, pool, timeout to surface database-related errors specifically.

Deep investigation mindset

Don't just skim the top issues by count. Many high-value fixes hide in less obvious places:

• Vague titles deserve investigation, not dismissal. An issue titled "captureException" with 67K events turned out to be react-beautiful-dnd throwing plain objects that Sentry's instrumentation captured — a one-line beforeSend filter eliminated all of them. Investigate the event payload before skipping.
• Look for patterns across issues. If one AI output parsing error was fixed with .nullish() in a Zod schema, scan for the same pattern in other AI service schemas (MagicTag, InferredSuggestion, etc.).
• Check if a fix in one code path was missed in a parallel path. A bulkCopyNotes null safety fix was applied to NoteService but the same bug existed in ProjectService.copyProject. Search for sibling code paths.
• Low-frequency issues can still be high-impact. A DB constraint violation with 13 events might affect paying customers silently. A 1-event URL construction error reveals missing input validation on auth endpoints.
• Correlate Sentry errors with Datadog metrics. Rising waiting_queries + statement timeout Sentry errors = lock contention. Rising database_connections + pool errors = connection exhaustion. The fix is often in code (transaction splitting, connection release), not infrastructure.
• Scan with per_page=100 and look beyond page 1. The default per_page=25 misses mid-tier issues that are still fixable.

Phase 2b: DB performance triage

Query Datadog for database performance metrics to identify systemic issues and correlate with Sentry errors. Use curl -s -G with --data-urlencode for all Datadog timeseries queries (required for {*} syntax):

curl -s -G "https://api.us3.datadoghq.com/api/v1/query" \
  --data-urlencode "from=$(date -v-7d +%s)" \
  --data-urlencode "to=$(date +%s)" \
  --data-urlencode "query=max:postgresql.queries.duration.max{*} by {db}" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"

Key metrics to check:

• max:postgresql.queries.duration.max{*} by {db} — slow queries
• max:postgresql.locks{*} — lock contention
• max:postgresql.waiting_queries{*} — blocked queries
• max:aws.rds.database_connections{*} — connection pressure
• max:aws.rds.read_latency{*}, max:aws.rds.write_latency{*} — I/O pressure

Cross-reference elevated metrics with Sentry errors to identify root causes (e.g. high waiting_queries + deadlock Sentry errors = lock contention issue).

Phase 3: Investigate and fix (parallel agents)

This phase runs every iteration, not just once. If there are issues with status: "discovered", they must be investigated — do not skip this phase just because agents ran in a previous iteration. If discovery found no new issues but there are still discovered issues in the state file, investigate those.

Pick up to 4 issues with status: "discovered", prioritized by frequency. When all high-frequency issues are handled, move to mid-tier and low-frequency issues — don't stop just because the remaining issues have fewer events. A 13-event DB constraint violation or a 1-event URL error can still reveal important bugs worth fixing.

For each issue, launch a worktree agent in parallel (isolation: "worktree") with this task:

Investigate Sentry issue {id} ("{title}") in project {project}.

1. Fetch the latest event stacktrace via /issues/{id}/events/latest/ from the Sentry API.
2. Use Datadog logs API (https://api.us3.datadoghq.com/api/v2/logs/events/search) with DD-API-KEY and DD-APPLICATION-KEY (env var: DD_APP_KEY) headers to correlate error patterns.
3. If the error involves database timeouts, connection failures, deadlocks, or slow queries, query Datadog for DB-level metrics using the timeseries API (https://api.us3.datadoghq.com/api/v1/query). Useful metrics:
   • postgresql.queries.duration.max / postgresql.transactions.duration.max — slow queries/transactions
   • postgresql.locks, postgresql.deadlocks, postgresql.deadlocks.count — lock contention
   • postgresql.waiting_queries, postgresql.active_waiting_queries — blocked queries
   • postgresql.activity.wait_event, postgresql.activity.blocked_connections — what's blocking
   • postgresql.connections, postgresql.percent_usage_connections — pool exhaustion
   • aws.rds.read_latency, aws.rds.write_latency — disk I/O pressure
   • aws.rds.blocked_transactions_count, aws.rds.idle_in_transaction_sessions_count — idle-in-transaction leaks
   • aws.rds.database_connections, aws.rds.cpuutilization — instance-level health
4. Trace through the source code to find the exact code path.
5. If you can fix it with high confidence and low blast radius:
   • Always branch from origin/master: run git fetch origin master && git checkout -b dougrathbone/fix-{short-description} origin/master. Never branch off another feature branch — this is the #1 cause of PRs containing unrelated commits. The only exception is stacked PRs for a single initiative that require staged merging (e.g. an NX upgrade split into sequential steps), where each PR explicitly targets the previous branch as its base.
   • Implement the fix
   • Add unit tests for any new behavior. Tests must cover the core invariant of the fix — not just that the code runs, but that the specific behaviour change works correctly. For example, if a fix splits a single transaction into multiple transactions, test that the split happens as expected and that partial failure is handled safely. If a fix adds a null guard, test both the null and non-null paths. Leaving a fix untested is not acceptable — reviewers will flag it.
   • Run yarn nx run <workspace>:test.jest -- --testPathPattern="<relevant test>" to verify tests pass
   • Run yarn nx run <workspace>:test.tsc to type-check
   • Run yarn nx run <workspace>:build to verify compilation
   • Run yarn fix.oxlint and yarn nx run <workspace>:fix.format
   • Commit and push
   • Verify the branch is clean before creating the PR:
     • Run git log --oneline origin/master..HEAD and confirm every commit is related to this fix. If unrelated commits are present (e.g. from branching off another feature branch), rebase: create a fresh branch from origin/master, cherry-pick only the relevant commits, and force-push with --force-with-lease.
     • Run git diff --name-only origin/master..HEAD and confirm every changed file is relevant to the fix. Flag any unexpected files.
   • Create a draft PR with gh pr create --draft --title "fix: {description}" --label "AI assisted" --label "bugfix" using the PR template at .github/pull_request_template.md. Do NOT add AI attribution in the PR body.
   • Include in the PR body: Sentry link, stacktrace analysis, code tracing steps, blast radius assessment, rollback plan.
   • For DB-related issues, also include in the PR body: which Datadog metrics were queried, the timeframe examined, metric values observed (e.g. p99 query duration, lock counts, connection pool utilization), and how the fix addresses the specific DB performance problem.
   • Return the PR number and branch name.
6. If you cannot fix it confidently, return a summary of your investigation and recommend skipped with a reason.

Known fix categories for reference:

• Apollo subscription null safety: updateQuery handlers accessing .nodes without ?.
• Invariant replacements: assert.invariant that should gracefully handle edge cases
• Codec safety: decodeOrThrow in non-critical paths replaced with decodeOrNull
• Shutdown noise: Pool lifecycle errors during graceful shutdown
• Webhook resilience: Strict codec unions rejecting new external data types
• Sentry noise reduction: Transient errors downgraded from reportError to warn
• Database timeouts: Missing statement timeouts, unbounded queries, missing indexes, idle-in-transaction from unclosed connections
• Connection pool exhaustion: Leaked connections from missing finally blocks or un-awaited DataApi releases
• Deadlock retry: PostGraphile GraphQL mutations that deadlock on FOR UPDATE — catch with isPgError + isRetryablePgTransactionError and return a client-retryable error response. Note: BetterPgPool.transaction() already retries deadlocks for background jobs; GraphQL mutations need handling at the resolver level.
• Transaction safety: Use existing dbTransactionSafetyContext and isRetryablePgTransactionError from @dvtl/postgres — never write custom deadlock detection

After all parallel agents complete, update the state file with results (PR numbers, skip reasons, etc.). Mark new PRs as "pr-review-pending" (not "pr-open") — they are created as drafts and are not done until CI passes, branch content is verified, bot reviews are addressed, and the PR is published in Phase 4.

Phase 4: Babysit existing PRs (verify, promote drafts, address feedback)

A PR is NOT done just because CI is green. It must also pass content verification, bot reviews, and be promoted from draft to ready. For every issue with status: "pr-open" or "pr-review-pending":

Step 1: Check CI status

• Run gh pr checks <prNumber> to identify failures.
• For simple failures (format, lint, type errors), launch a worktree agent to fix and push.
• For complex failures, update the state file with the failure details for human review.

Step 2: Verify branch content

Before promoting a draft PR, verify the branch only contains relevant changes:

• Run gh pr view <prNumber> --json commits --jq '.commits[].messageHeadline' and confirm every commit message relates to the fix described in the PR title.
• Run gh pr diff <prNumber> --name-only and confirm every changed file is relevant.
• If unrelated commits or files are found, launch a worktree agent to clean the branch: check out the branch, create a fresh branch from origin/master, cherry-pick only the relevant commits, and force-push with --force-with-lease. The PR stays in "pr-review-pending" until the next iteration re-verifies.

Step 3: Wait for bot reviews to complete

• Run gh pr view <prNumber> --json reviews,reviewRequests,comments to check review status.
• Check for pending review requests — if review bots (e.g. bugbot, coderabbitai, github-actions[bot], or any bot reviewer) have not yet posted their review, keep the status as "pr-review-pending" and move on. Do NOT mark the PR as done.
• Also check gh pr view <prNumber> --json comments for bot comments that contain actionable feedback (look for comments from bot accounts that flag issues, suggest changes, or request fixes).

Step 4: Address bot review feedback

• If bot reviewers have posted reviews with CHANGES_REQUESTED or left comments flagging bugs/issues:
  • Launch a worktree agent to check out the PR branch and address each piece of bot feedback.
  • Fix the issues, commit, and push to the same PR branch.
  • After pushing fixes, the PR stays in "pr-review-pending" — the bots need to re-review the new changes. It will be re-checked in the next iteration.
• If bot feedback is too complex to address automatically, update the state file with details for human review and mark accordingly.

Step 5: Promote draft to ready

Only publish a draft PR (transitioning from "pr-review-pending" to "pr-open") when ALL of the following are true:

• CI is passing
• Branch content is verified (no unrelated commits or files)
• All bot reviewers have posted their reviews
• No bot reviews have CHANGES_REQUESTED status or unresolved bot-flagged issues

To publish: gh pr ready <prNumber>

This ensures human reviewers only see PRs that are clean, passing, and bot-approved.

Phase 5: Cleanup (end of every iteration)

1. Run git worktree list and remove ALL .claude/worktrees/* entries that are no longer needed. A worktree is no longer needed if:
   • Its PR has been pushed and is not being actively fixed this iteration.
   • Its PR has been merged or closed.
   • Its associated branch has no corresponding open PR. Remove with git worktree remove --force <path>, then git worktree prune.
2. Verify no orphaned directories remain under .claude/worktrees/ — if they do, remove them with rm -rf.
3. Write the updated state file.
4. Print a summary: new issues found, fixes submitted, PRs waiting for bot review, PRs ready for human review, PRs passing/failing, issues skipped, worktrees cleaned up.

Technical context

• TypeScript monorepo with NX build system, Yarn v4.6.0
• Frontend: React, Apollo Client, ProseMirror editor
• Backend: PostGraphile GraphQL, PostgreSQL, PgBoss job scheduler, Redis
• CDC (Change Data Capture) pipeline processes database changes
• processLifecycle.isShuttingDown flag for shutdown-aware code
• @dvtl/toolbox codec utilities: decodeOrThrow, decodeOrNull, ErrorWithMeta
• io-ts codecs with fp-ts Either / isRight for validation
• Use /babysit-pr skill for monitoring CI on individual PRs
• Datadog auth: DD_API_KEY for DD-API-KEY header, DD_APP_KEY for DD-APPLICATION-KEY header
• Datadog site: us3.datadoghq.com
• Buildkite auth: BUILDKITE_API_TOKEN env var, used as Authorization: Bearer header. API base: https://api.buildkite.com/v2/organizations/dovetail/pipelines/platform-ci/builds/{buildNumber}. Use this to read build logs and identify CI failures — get failed jobs via .jobs[] | select(.state == "failed"), then fetch logs at /jobs/{jobId}/log.