Database & Application Performance Investigation Loop

Objective

You are a performance engineer investigating database and application performance issues in the Dovetail platform. Use Datadog (APM spans, DBM query metrics, logs, RUM), Sentry error tracking, and the codebase to find real performance problems, then fix them and ship PRs with detailed evidence so reviewers have high confidence in every change.

The bar for every fix is: you can name the dominant cause of the latency, prove it with telemetry, show the code path, and predict the improvement with a number a reviewer can check after deploy. No guesses, no "it's probably a missing index."

Every PR must be the lowest-risk version of the change that still ships the win. A performance fix that causes an outage or changes what users see is a net loss, no matter how much faster it is. Default to no customer risk: prefer strictly-additive changes (a new index, a request-scoped cache), and when a change alters behaviour or results, de-risk it explicitly — split it into backward-compatible stages, gate it behind a LaunchDarkly flag, back it with a resumable backfill, and cover it with unit + e2e tests. You choose that strategy in Risk Minimisation & Safe Rollout (below, right before the Fix phase); it is not optional. If you cannot make a change safe, it goes to Recommendations, not a PR.

Tooling — MCP servers, not API keys

This runbook uses MCP tools for all observability access. There are no API keys, tokens, or curl commands. Do not read DD_API_KEY, DD_APP_KEY, SENTRY_AUTH_TOKEN, or BUILDKITE_API_TOKEN — they are not needed and must not be reintroduced. Authentication, region, org, and site are all handled by the MCP integrations.

Domain	MCP tools (prefix `mcp__…`)
Datadog metrics (DBM query stats, table/instance state)	`datadog__get_datadog_metric`, `datadog__get_datadog_metric_context`, `datadog__search_datadog_metrics`
Datadog APM spans (SQL text, resolver-level N+1, per-call latency)	`datadog__aggregate_spans`, `datadog__search_datadog_spans`
Datadog logs (slow requests, timeouts, deadlocks)	`datadog__search_datadog_logs`, `datadog__analyze_datadog_logs`
Datadog change/deploy correlation	`datadog__get_change_stories`
Datadog RUM (frontend / application-layer latency)	`datadog__search_datadog_rum_events`, `datadog__aggregate_rum_events`
Datadog skill guides (metric catalogs + query templates)	`datadog__list_datadog_skills`, `datadog__load_datadog_skill`
Sentry (errors, perf issues, root-cause)	`sentry__search_issues`, `sentry__search_events`, `sentry__get_sentry_resource`, `sentry__analyze_issue_with_seer`
Buildkite (CI validation)	`buildkite__list_builds`, `buildkite__get_build`, `buildkite__list_jobs`, `buildkite__read_logs`, `buildkite__get_failed_executions`

Fixed context (no lookup needed): Sentry org dovetail (region https://us.sentry.io), Buildkite org dovetail, pipeline platform-ci. Datadog site/region is implicit in the MCP.

Every Datadog tool call takes a telemetry: { intent } argument — a one-line description of why you are making the call (no secrets/PII). Fill it in honestly; it is used for auditing.

Load the Datadog DBM skill before querying metrics

The Datadog MCP ships skill guides with the exact metric names, tag schema, query templates, and thresholds. Load them once at the start of a run — do not guess metric names (wrong names return 400 or empty):

mcp__datadog__load_datadog_skill(skill_name: "datadog/dbm-postgresql",
  telemetry: { intent: "Load the Postgres DBM metric catalog and query templates" })

mcp__datadog__load_datadog_skill(skill_name: "datadog/dbm-postgresql",
  resource_path: "references/metrics-query.md",
  telemetry: { intent: "Load per-query_signature latency/IO/cache templates and thresholds" })

Also useful: references/metrics-table.md (per-table seq/index scans, dead rows, sizes) and references/metrics-activity.md (wait-event / lock attribution). Use list_datadog_skills if the exact name has changed. These skills are the source of truth for metric names, tags, and thresholds — if anything in this runbook disagrees with a freshly-loaded skill, trust the skill.

What the MCP does and does not expose here (verified). This org's Datadog MCP integration exposes the core observability toolset (metrics, spans, logs, RUM, events, change tracking) plus the DBM skill guides and postgresql.* metrics. It does not expose the DBM "assistant" tools (get_datadog_database_explain_plans, get_datadog_database_health_signals, get_datadog_database_recommendations, search_datadog_database_samples). If a future integration adds them, prefer them — an explain plan straight from DBM beats inferring the plan from metrics. Until then, this runbook gets everything it needs from get_datadog_metric on the postgresql.queries.* family plus aggregate_spans.

Worktree Isolation Strategy

All implementation work MUST happen in isolated git worktrees. This prevents your changes from clashing with other in-progress work in the main checkout. The main worktree is used ONLY for read-only discovery (Phase 1-2). All code changes happen in per-fix worktrees.

Workflow overview

Main worktree (read-only)     Worktree A (fix 1)     Worktree B (fix 2)
├── Phase 1: Discovery ──────►├── Phase 3: Fix        ├── Phase 3: Fix
├── Phase 2: Analysis ───────►├── Phase 4: Validate   ├── Phase 4: Validate
│                              ├── Phase 5: PR         ├── Phase 5: PR
│                              └── Cleanup ✓           └── Cleanup ✓
└── Phase 6: Report

Creating a worktree for each fix

Before starting Phase 3 for any issue, create an isolated worktree:

# From the main repo root
FIX_NAME="perf/<short-description>"
WORKTREE_PATH="../platform-worktrees/${FIX_NAME}"

# Create the worktree on a new branch from master
git worktree add -b "${FIX_NAME}" "${WORKTREE_PATH}" origin/master

# Move into it for all subsequent work on this fix
cd "${WORKTREE_PATH}"

# Install dependencies (worktrees share git objects but not node_modules)
yarn install

Yarn cache gotcha (enableGlobalCache: false) — this repo's .yarnrc.yml keeps the cache inside .yarn/cache/, which is copied into each new worktree by git. A specific zip can land truncated (~133 bytes vs the real size), causing yarn install to fail with Libzip Error: Failed to open the cache entry for <pkg>: Not a zip archive. The reliable fix is to copy the good file from the main checkout's cache:

# Replace one corrupted cache entry, then retry yarn install
cp /Users/doug/Code/dovetail-platform/.yarn/cache/<pkg>-npm-<ver>-<hash>-10c0.zip \
   "${WORKTREE_PATH}/.yarn/cache/"
cd "${WORKTREE_PATH}" && yarn install

If the install still fails on a different cache entry, repeat for that entry. Do NOT run yarn cache clean — the cache is the source of truth for zero-install setups in this repo. (On a fresh worktree the cache zips may also be unsmudged Git LFS pointers — if yarn install complains they are not zip archives, run git lfs pull in the worktree first.)

Rules for worktree usage

Never modify files in the main worktree — it stays clean for discovery and reading
Each fix gets its own worktree — never share a worktree between unrelated fixes
Run all build/test/lint commands inside the worktree — they use the worktree's file state
Commit and push from the worktree — git commit and git push work normally inside a worktree
Return to the main worktree between fixes for any additional discovery work

Using subagents for parallel fixes

When multiple fixes are independent, use the Agent tool with isolation: "worktree" to work on them in parallel:

Agent(
  description: "perf: add index on table.column",
  prompt: "<full fix instructions with the evidence dossier>",
  isolation: "worktree"
)

Each agent gets its own worktree automatically. Launch multiple agents in a single message for maximum parallelism. The agent handles branch creation, commits, and pushes within its isolated worktree.

When NOT to parallelize: if two fixes touch the same table or migration sequence, run them sequentially to avoid migration timestamp conflicts.

Worktree cleanup

Worktrees MUST be cleaned up to avoid accumulating stale checkouts that waste disk space and create confusion. Clean up at these trigger points:

After a PR is published (marked ready for review)

cd /path/to/main/repo
git worktree remove "../platform-worktrees/perf/<short-description>" --force
git worktree prune

After CI is green and PR is ready — cleanup immediately

Do not leave worktrees around "just in case." If CI passes and the PR is published, the worktree has served its purpose. The branch still exists on the remote. Exception: for a multi-stage rollout, keep the worktree until every stage (migration → flagged code → cleanup) has merged.

If CI fails and you need to fix

Keep the worktree alive, make fixes inside it, and only clean up after the PR is finally published.

At the end of every investigation run (Phase 6)

After generating the final report, do a sweep to clean up ALL worktrees created during this run:

git worktree list
git worktree list --porcelain | grep -A2 "worktree.*platform-worktrees/perf/" | while read -r line; do
  if [[ "$line" == worktree* ]]; then
    WTPATH="${line#worktree }"
    echo "Removing worktree: ${WTPATH}"
    git worktree remove "${WTPATH}" --force 2>/dev/null || true
  fi
done
git worktree prune
rmdir "../platform-worktrees" 2>/dev/null || true

Cleanup stale worktrees from previous runs

At the start of each new run (before Phase 1), check for and clean up leftover worktrees. Note that background cleanup can prune non-locked worktrees mid-session — commit and push early, and prefer sibling paths under ../platform-worktrees/.

for WT in $(git worktree list --porcelain | grep "worktree.*platform-worktrees/perf/" | sed 's/worktree //'); do
  BRANCH=$(cd "$WT" && git branch --show-current)
  PR_STATE=$(gh pr view "$BRANCH" --json state -q .state 2>/dev/null || echo "NONE")
  if [[ "$PR_STATE" == "MERGED" || "$PR_STATE" == "CLOSED" || "$PR_STATE" == "NONE" ]]; then
    echo "Cleaning up stale worktree: $WT (PR state: $PR_STATE)"
    git worktree remove "$WT" --force 2>/dev/null || true
  else
    echo "Keeping worktree: $WT (PR state: $PR_STATE — still open)"
  fi
done
git worktree prune

State Tracking — `.agents/state/db-patrol.json`

All investigation runs and PRs are tracked in .agents/state/db-patrol.json. This file persists across runs so you know what has already been investigated, what PRs are open, and what has been merged or closed. Always read this file at the start of every run and update it throughout.

Schema

{
  "lastRun": "2026-03-31T10:00:00Z",
  "issues": {
    "<issue-key>": {
      "title": "Short description of the performance issue",
      "source": "dbm-metrics | apm-spans | datadog-logs | sentry | rum",
      "querySignature": "SELECT ... FROM ... (normalized)",
      "service": "notes-service",
      "latencyCause": "scan-io | lock-wait | memory-spill | planning | client-pool | n-plus-1 | cache-pressure | bloat-vacuum",
      "rootCauseConfidence": "high | medium | low",
      "rolloutStrategy": "additive | flagged | multi-stage",
      "featureFlag": "perf.<flag-key> or null",
      "rolloutStage": "single | migration | flagged-code | cleanup",
      "prs": {
        "migration": { "number": null, "branch": null, "status": "not-started" },
        "flagged-code": { "number": null, "branch": null, "status": "not-started" },
        "cleanup": { "number": null, "branch": null, "status": "not-started" }
      },
      "status": "investigating | fix-in-progress | pr-draft | pr-awaiting-review | pr-ci-failing | pr-published | pr-merged | pr-closed | skipped",
      "skipReason": null,
      "metrics": {
        "totalDbTimePerHourSec": 4000,
        "avgLatencyMs": 800,
        "p99LatencyMs": 3500,
        "callsPerHour": 5000,
        "rowsPerExec": 500000,
        "logicalReadsPerExec": 120000,
        "cacheHitRatioPct": 62,
        "deadTupleRatioPct": 4
      },
      "discoveredAt": "2026-03-31T10:05:00Z",
      "lastUpdated": "2026-03-31T11:30:00Z"
    }
  }
}

Notes on the schema:

The <issue-key> should be a stable identifier — the query signature, the service::resolver-method pair for N+1s, a Sentry issue short-ID, or a slug like missing-index-notes-team-id-created-at.
latencyCause and rootCauseConfidence are set in Phase 2. They are the difference between "this is slow" and "we know why it is slow" — never open a PR while rootCauseConfidence is low.
For an additive change, use only prs.migration (index) or a single code PR and leave rolloutStage: "single". For a flagged/multi-stage change, the prs object tracks the ordered stages so a later run can see which stage still owes work (e.g. a merged migration whose code PR hasn't shipped, or a live flag whose cleanup PR is outstanding).

When to read / update the state file

Start of every run — read it. It tells you which issues are already fixed (skip), which PRs are open (check CI/review, clean up merged worktrees), and which were skipped (re-evaluate if conditions changed).
Phase 1 — add new issues with status investigating.
Phase 2 — record metrics, latencyCause, rootCauseConfidence; move to fix-in-progress or skipped (with skipReason).
Risk classification (before Phase 3) — set rolloutStrategy, featureFlag (or null), and rolloutStage.
Phase 3 — as each stage PR is created, fill the matching entry in prs.
Phase 5 — set pr-draft → pr-awaiting-review (while waiting on CI + claudebot) → pr-ci-failing on CI fail → pr-published once CI is green, the review is resolved, and the PR is marked ready.
Cleanup — when a worktree is removed, note it; when all stages have merged, set pr-merged.

Deduplication

Before creating a fix, check the state file. An issue whose query signature or table/column combination is already at status pr-draft, pr-awaiting-review, pr-published, or pr-merged → skip it. A previous pr-closed → investigate why before re-attempting (gh pr view <number> --comments). A previous pr-ci-failing → fix the existing PR rather than opening a new one. For a multi-stage issue, also skip if any stage is still in flight — resume the outstanding stage instead of restarting.

Reconciling state with GitHub at the start of each run

Before discovery, reconcile each open PR's real state:

jq -r '.issues | to_entries[] | .value.prs // {} | to_entries[] | select(.value.number != null) | "\(.key) \(.value.number)"' .agents/state/db-patrol.json | while read -r STAGE PR; do
  STATE=$(gh pr view "$PR" --json state -q '.state' 2>/dev/null || echo "UNKNOWN")
  echo "stage ${STAGE} PR #${PR} → ${STATE}"
  # MERGED → mark stage merged;  CLOSED → pr-closed + clear worktree;
  # OPEN → check CI + claudebot review status (Phase 5).
done

Also reconcile any issue still at pr-draft / pr-awaiting-review between runs so a half-finished PR isn't re-discovered as new work. After reconciling, clean up worktrees for any PRs now merged/closed. The end-of-run sweep (Phase 6) reconciles all entries, not just the filesystem.

Phase 1: Discovery — Gather Performance Data

Discovery has two complementary sources that you cross-reference by SQL text:

DBM query metrics (postgresql.queries.*, tagged by query_signature) — the authoritative source for impact (total DB time), throughput, and the I/O profile (rows/exec, cache hit ratio, block reads). This is where you rank what matters.
APM spans (operation_name:pg.query, resource_name = normalized SQL) — the authoritative source for the full SQL text, per-call latency distribution, and resolver-level N+1 patterns.

Rank with metrics; resolve to SQL and detect N+1 with spans. Match the two by the SQL text (DBM query_signature hashes and APM resource_name are different keys — join on the query text, not the hash).

1a. Rank query signatures by total DB time (impact) — DBM metrics

Total DB time = the amount of database the query actually consumes = the single best impact metric. It is a metric you read directly (sum:postgresql.queries.time), not something you compute from frequency × latency.

mcp__datadog__get_datadog_metric(
  queries: ["sum:postgresql.queries.time{*} by {query_signature}"],
  formulas: ["top(query0, 25, 'sum', 'desc')"],
  from: "now-24h", to: "now",
  response_format: "timeseries",
  telemetry: { intent: "Rank Postgres query signatures by total DB time to find highest-impact work" }
)

postgresql.queries.time is in nanoseconds and is tagged with query_signature, query (truncated obfuscated SQL — for display/grep only), table, command, user, db. Per-instance scope tags (database_instance, replication_role) also exist on the family per the DBM skill.
Scope: {*} sums across all database instances (primary + replicas). To isolate one instance, first confirm the tag actually resolves in this org, then scope by it:
```
mcp__datadog__get_datadog_metric_context(metric_name: "postgresql.queries.time",
  include_tag_values: true, tag_filter: "database_instance",
  telemetry: { intent: "Enumerate Postgres instances to scope query metrics" })
```
If database_instance (or replication_role) returns no values here, fall back to {*} rather than scoping to an empty set. For write-path analysis, replication_role:master narrows to the primary if that tag is present.
Payload discipline: always narrow with top(...) or a small by-cardinality. Grouping by both query_signature and query across all signatures returns a very large response — only add the query tag once you have shortlisted, or resolve the SQL via spans (1c) instead.

1b. Prime index / scan candidates — DBM metrics

A query being slow does not mean an index will help it (it might be lock-bound, or scanning dead tuples). A query doing lots of block reads is scanning — often, but not always, fixable with an index (rule out bloat in Phase 2). Rank by block reads directly:

mcp__datadog__get_datadog_metric(
  queries: ["sum:postgresql.queries.shared_blks_read{*} by {query_signature}"],
  formulas: ["top(query0, 25, 'sum', 'desc')"],
  from: "now-24h", to: "now", response_format: "timeseries",
  telemetry: { intent: "Rank signatures by total shared-buffer block reads — prime indexing candidates" }
)

High total shared_blks_read = cache misses from scanning heap/index pages. These are your best index candidates (pending the Phase 2 triage). Complementary top-N lists: shared_blks_dirtied = top WAL contributors; temp_blks_written = work_mem spills.

1c. Resolve SQL and find N+1 — APM spans

The span resource_name carries the full normalized SQL, and grouping by the JS resolver method (operation_name, e.g. SomeDbService.someMethod) surfaces N+1 patterns that metrics can't see.

Top slow DB spans by count, with full SQL (24h):

mcp__datadog__aggregate_spans(
  query: "service:notes-service operation_name:pg.query @duration:>2000000000",
  computes: [
    { field: "*",         aggregation: "COUNT", output: "n", sort: "desc" },
    { field: "@duration", aggregation: "AVG",   output: "avg_ns" },
    { field: "@duration", aggregation: "P99",   output: "p99_ns" },
    { field: "@duration", aggregation: "MAX",   output: "max_ns" }
  ],
  group_by: { fields: ["resource_name"], limit: 25 },
  from: "now-24h", to: "now",
  telemetry: { intent: "Top slow DB spans by SQL, with latency distribution, for a notes-service perf sweep" }
)

The DB span operation_name is pg.query (not postgres.query/db.query).
Latency lives in @duration (nanoseconds). Aggregating on the bare duration field returns nothing — always use @duration. @duration:>2000000000 = ">2s". COUNT uses field: "*". Percentile aggregations are P50/P75/P90/P95/P99 (not pc99).
Group by resource_name for SQL; group by operation_name for resolver-level N+1 (below).

Resolver fan-out (N+1 detection):

mcp__datadog__aggregate_spans(
  query: "service:notes-service operation_name:pg.query",
  computes: [
    { field: "*",         aggregation: "COUNT", output: "n", sort: "desc" },
    { field: "@duration", aggregation: "AVG",   output: "avg_ns" },
    { field: "@duration", aggregation: "P99",   output: "p99_ns" }
  ],
  group_by: { fields: ["operation_name"], limit: 25 },
  from: "now-24h", to: "now",
  telemetry: { intent: "Rank DB resolver methods by call count to surface N+1 fan-out" }
)

A method with a huge call count and a low average duration is the classic N+1: cheap-but-numerous queries fanning out once per parent row. Confirm it is N+1 by inspecting one real trace — pull the raw spans for a single request and count how many times the resolver runs within it:

mcp__datadog__search_datadog_spans(
  query: "service:notes-service operation_name:<ResolverClass.method>",
  from: "now-24h", to: "now", sort: "-@duration",
  telemetry: { intent: "Inspect individual N+1 resolver spans and their parent trace" }
)

Then fetch the enclosing trace by trace_id to see the fan-out inside one request. Compute the fan-out ratio: child spans ÷ parent requests. A ratio well above 1 that scales with result-set size is the signature of N+1.

1d. Slow requests, timeouts, deadlocks — Datadog logs

mcp__datadog__search_datadog_logs(
  query: "service:notes-service @duration:>2000000000",
  from: "now-24h", to: "now", sort: "-timestamp",
  telemetry: { intent: "Find slow HTTP requests (>2s) to correlate with slow queries" }
)

mcp__datadog__search_datadog_logs(
  query: "service:(notes-service OR tenant-router) (timeout OR deadlock OR \"lock wait\" OR \"statement timeout\" OR \"canceling statement\")",
  from: "now-24h", to: "now", sort: "-timestamp",
  telemetry: { intent: "Find DB timeout / deadlock / lock-wait errors" }
)

For counts and grouping (e.g. how many timeouts per endpoint over time), use analyze_datadog_logs with SQL instead of paging raw logs:

mcp__datadog__analyze_datadog_logs(
  filter: "service:notes-service (timeout OR deadlock OR \"canceling statement\")",
  sql_query: "SELECT message, count(*) AS c FROM logs GROUP BY message ORDER BY count(*) DESC",
  from: "now-24h", to: "now",
  telemetry: { intent: "Aggregate DB error log volume by message" }
)

1e. Errors and root-cause — Sentry

mcp__sentry__search_issues(organizationSlug: "dovetail", regionUrl: "https://us.sentry.io",
  query: "is:unresolved (postgresql OR timeout OR deadlock OR \"statement timeout\")",
  sort: "freq", limit: 25)

For counts/aggregations of error events, use search_events with the appropriate dataset (errors, logs, or spans) — search_issues returns grouped issues only, not statistics.
When you have a specific issue and want a code-level root cause, run Seer: sentry__analyze_issue_with_seer(issueUrl: "https://dovetail.sentry.io/issues/<SHORT-ID>"). Use it deliberately (it can take minutes), not as an automatic follow-up.
Reality check for this org: Sentry's dedicated performance-issue category (issue.category:performance) is typically empty here — do not treat an empty result as "no perf problems." Sentry's value in this loop is error correlation (timeouts/deadlocks tie a slow query to user-facing failures) and root-cause, not perf-issue discovery. Discovery comes from DBM metrics + APM spans.

1f. Correlate a regression with a deploy — change tracking

If a signature's latency got worse, find out what changed. get_change_stories returns deployments, feature-flag flips, DB schema/config changes, and scaling events for a service:

mcp__datadog__get_change_stories(service_name: "notes-service",
  start_ts: "<regression start ISO>", end_ts: "<now ISO>",
  story_types: ["deployment", "database", "feature_flag", "configuration"],
  telemetry: { intent: "Correlate a query latency regression with deploys / schema / flag changes" })

A regression that starts exactly at a deploy or a migration is a different (and often faster) fix than a slow creep from data growth.

1g. (Optional) Application-layer latency — APM + RUM

The title says Application performance, not just database. When the slow user experience isn't DB-bound, look one layer up:

APM (non-DB spans): rank web/GraphQL transactions by p95, then break down where the time goes (DB vs external vs CPU):

mcp__datadog__aggregate_spans(
  query: "service:notes-service operation_name:(express.request OR graphql.execute)",
  computes: [{ field: "@duration", aggregation: "P95", output: "p95", sort: "desc" },
             { field: "*", aggregation: "COUNT", output: "n" }],
  group_by: { fields: ["resource_name"], limit: 25 }, from: "now-24h", to: "now",
  telemetry: { intent: "Rank web/GraphQL transactions by p95 latency" })

RUM (frontend): slow page loads and resource timings (@view.loading_time, @resource.duration in ns):

mcp__datadog__aggregate_rum_events(
  query: "@type:view @view.loading_time:>5000000000",
  computes: [{ field: "@view.loading_time", aggregation: "P75", output: "p75", sort: "desc" },
             { field: "*", aggregation: "COUNT", output: "n" }],
  group_by: { fields: ["@view.url_path_group"], limit: 25 }, from: "now-7d", to: "now",
  telemetry: { intent: "Rank slow front-end page loads by URL group" })

Keep the DB path as the center of gravity — this section is for when the evidence points above the database.

Phase 2: Analysis — Attribute the Latency, Then Correlate with Code

This phase is the heart of the runbook. The most common failure mode in perf work is fixing the wrong thing — adding an index to a query that is slow because of lock contention, or "optimizing" SQL that is actually slow because of connection-pool wait. So the first analysis step is not "find it in the code" — it is attribute the latency to a cause.

2a. Attribute the latency to a dominant cause (triage)

For each candidate signature, pull its I/O profile and classify it before proposing any fix. All templates below come from the datadog/dbm-postgresql → references/metrics-query.md skill; the thresholds are the skill's. Pull these per signature ({query_signature:<SIG>} scope) over the regression window:

Signal	Metric formula	Reads as
Avg latency	`sum:postgresql.queries.time / sum:postgresql.queries.count by {query_signature}` (ns)	symptom; <10ms excellent, 10–100ms typical, 100ms–1s heavy, >1s investigate
Rows / exec	`sum:postgresql.queries.rows / sum:postgresql.queries.count by {query_signature}`	spike >3× typical → missing filter / pagination
Logical reads / exec	`(shared_blks_hit + shared_blks_read) / count`	<10 efficient, 10–100 typical, 100–1000 heavy, >1000 scanning
Cache hit ratio	`100 * shared_blks_hit / (shared_blks_hit + shared_blks_read)`	shared-buffer only (see caveat); 95–99% excellent, <80% investigate (not an automatic fix)
Block read time / exec	`sum:postgresql.queries.blk_read_time / count` (ns)	only meaningful if `track_io_timing=on` (see caveat); >15ms/exec → investigate storage
Temp blocks / exec	`sum:postgresql.queries.temp_blks_written / count`	should be 0 for OLTP; non-zero = work_mem spill
Plan time	`postgresql.queries.mean_plan_time`	only if `track_planning=on`; high vs exec time → planning overhead

Metric caveats — read before you trust a number:

blk_read_time / blk_write_time require the Postgres GUC track_io_timing = on. If it is off, these are identically 0 for every signature — a flat 0 across all queries means "timing disabled," not "no I/O wait." In that case use shared_blks_read and logical-reads-per-exec as the I/O proxy instead.
mean_plan_time requires track_planning = on; treat a flat 0 the same way.
Cache hit ratio is shared-buffers only — it excludes the OS page cache, which serves most "misses" in microseconds. A query at 70–85% shared-buffer hit ratio can be perfectly healthy. Interpret it alongside blk_read_time and absolute shared_blks_read, never as a standalone verdict.

Then classify — this decision tree tells you which fix family applies and, crucially, when NOT to add an index:

Scan / I/O-bound → high logical-reads-per-exec (>1000) and/or high shared_blks_read, low cache hit ratio, rows examined ≫ rows returned — and the table is not bloated (rule out the bloat branch first). → Index or query rewrite. (indexable)
Bloat / dead-tuple → looks like scan-bound (high logical reads, low cache hit ratio) but the table has a high dead-tuple ratio (postgresql.dead_rows vs live_rows by {table}) or degrading index_scans efficiency. Adding an index here just adds more to vacuum. → VACUUM / REINDEX / autovacuum tuning, not a new index. (Also suspect TOAST/detoast cost for wide-column queries — relevant to large text/rich_text-style rows.)
N+1 fan-out (from 1c) → one resolver called N× per parent request, each call cheap. → Batch / request-scoped cache / DataLoader. (the dominant fixable category here)
Memory spill → non-zero temp blocks per exec. → Reduce sort/hash volume, add a supporting index for the sort, or rewrite; occasionally a work_mem question (out of scope for a code PR).
Planning overhead → mean_plan_time large relative to exec time. → Simplify the query / prepared-statement / generic-plan considerations.
Lock / wait-bound → latency high but logical reads and block-read-time low; PK/FOR UPDATE lookups stuck at seconds; the activity/sample stream shows Lock/LWLock waits. → NOT an index. Route to the transaction-scope / deadlock workstream.
Cache pressure → cache hit ratio dropping fleet-wide, not query-specific. → Usually memory/plan, not a per-query PR.
Client / pool / orchestration → query is fast at the DB but slow end-to-end (e.g. per-request set_config RLS setup, connection acquisition). → Caching / pooling, not the query.

Record the winner as latencyCause in the state file. If you cannot pick one confidently, set rootCauseConfidence: low and either gather more evidence or move the issue to Recommendations — do not open a PR.

2b. Get (or infer) the query plan

Understanding the plan is the strongest evidence for a scan-bound diagnosis. This org's MCP does not expose DBM explain plans, so use this order of evidence:

Infer from metrics — this is the primary, always-available evidence and is sufficient to propose a fix. Logical-reads-per-exec ≫ rows-returned plus low cache hit ratio (and I/O-bound, not bloat) is a sequential/heap scan in all but name.
Confirm the index is usable locally (best-effort, not a hard gate). Run EXPLAIN against the local dev DB (psql "$DATABASE_URL" in the notes-service workspace, or the connection the local Docker Postgres exposes). Two important cautions:
- Plan shape is NOT representative at low data volume. Postgres chooses plans from row-count statistics, so on a near-empty local table the planner will pick a Seq Scan for almost everything even when your index is correct and production would use it. Do not expect to observe "Seq Scan → Index Scan" locally, and do not conclude "not a fix" if you don't.
- To confirm the planner can and will use a proposed index locally, either seed representative row counts and ANALYZE, or create the index in a throwaway local session and check with SET enable_seqscan = off; that the index is chosen and lowers the estimated cost. Create the index directly in that local session for the check — do not run the migration (see the "Never run migrations" rule).
- EXPLAIN ANALYZE executes the statement. For any non-SELECT (INSERT/UPDATE/DELETE/UPSERT) or FOR UPDATE query, that mutates data and takes real locks. Default to plain EXPLAIN (plan + estimates, no execution) for write/locking statements, or wrap it: BEGIN; EXPLAIN (ANALYZE, BUFFERS) <stmt>; ROLLBACK;. Reserve EXPLAIN (ANALYZE, BUFFERS) for read-only SELECTs.
The authoritative proof of the fix is the production DBM I/O profile after deploy — logical-reads-per-exec and shared_blks_read dropping into the predicted range (Phase 6). Local EXPLAIN corroborates; production metrics decide.

2c. Find it in the codebase

Search workspaces/services/notes-service/src/ for the query. Queries live in:

.sql files (pgTyped queries) — most common
PostGraphile plugins (*Plugin.ts) and resolver classes
Inline SQL in service files
Migration files (for schema / index issues)

Read the calling code to understand the access pattern: how often is it called, is it inside a loop (N+1), is it missing a filter or index, is it holding a transaction open, could it batch / paginate / cache. Trace the full call chain HTTP handler → service → query.

2d. Index cost, redundancy & bloat check (before proposing an index)

An index is not free — it slows every write to the table and consumes storage. Before adding one:

Rule out bloat first. Check postgresql.dead_rows vs live_rows by {table} (from references/metrics-table.md). A high dead-tuple ratio means the "scan" is scanning dead rows — the fix is VACUUM/autovacuum, not an index.
Check for an existing/overlapping index. Search the migrations for CREATE INDEX on the table. A prefix of a composite index may already serve your query, or your proposed index may be redundant. Don't add (team_id, created_at) if (team_id, created_at, deleted_at) already exists.
Weigh write amplification. Check the table's write rate — postgresql.rows_inserted|updated|deleted by {table}. A high-write, low-read table is a poor index candidate.
Sanity-check selectivity. An index on a low-cardinality column (a boolean, a 3-value status) rarely helps. Prefer the column(s) that filter the result set most.
Confirm the table's scan behavior. postgresql.seq_scans vs postgresql.index_scans by {table} shows whether the table is already being scanned sequentially in production.

2e. Rank by impact and assign confidence

Impact = total DB time (sum:postgresql.queries.time for the signature; 1a already ranked this). A query at 500ms × 10k/hr (1.4 hr of DB time/hr) outranks one at 5s × 1/day.
Weight the critical path. A signature that appears in user-facing web/GraphQL traces (cross-reference APM) outranks a background job at the same DB-time.
Weight tail latency. A p99 that blows past the p50 hurts users even if the average looks fine.
Confidence (rootCauseConfidence): high = cause attributed + metric proof + code located; medium = cause attributed but plan/mechanism partly unconfirmed; low = symptom only. Only high/medium may become PRs.

2f. Build the Evidence Dossier

For every issue you plan to fix, before writing any code, compile a structured dossier (it goes straight into the PR).

Quantitative profile (DBM metrics + spans, with the window and thresholds):

Query signature / normalized SQL (from spans resource_name)
Total DB time (impact); throughput (exec/hr); avg latency (time/count); p50/p95/p99 (from spans @duration percentiles)
Rows/exec, logical-reads/exec, cache hit ratio, block-read-time/exec (note if track_io_timing is off), temp blocks/exec — each with the threshold it crosses
latencyCause and the evidence that pins it
Trend: same metrics for the prior window (e.g. prior 7d) to show improving/stable/degrading, and a get_change_stories hit if a deploy/migration caused it

Error / incident correlation: related Sentry issues (IDs, event counts, users), representative timeout/deadlock log lines, affected endpoints.

Codebase analysis: exact file:line of the query and every caller; the call chain; relevant table schema (columns, existing indexes, approx row count, dead-tuple ratio); why the current approach is slow (tie it to latencyCause).

Proposed fix rationale: what the fix does; why it works (the DB theory — e.g. "composite B-tree on (team_id, created_at) turns the Seq Scan into an index range scan"); local EXPLAIN corroboration where obtainable; and a testable prediction — the metric that will move and by how much (e.g. "logical reads/exec 120k → <200; p50 800ms → <20ms") that the post-deploy check (Phase 6) will verify. The most directly-attributable prediction for an index is the logical-reads/exec drop, not the cache-hit ratio. Plus a risk assessment (CONCURRENTLY build time; plan-regression risk if statistics are stale; write-amplification cost).

Risk Minimisation & Safe Rollout

Do this after the Phase 2 diagnosis and before writing any code in Phase 3. Pick the lowest-risk strategy that still ships the win and record it as rolloutStrategy/featureFlag/rolloutStage in the state file. The goal is that no PR from this loop can cause an outage or a visible-to-customers behaviour change without a fast, safe way to undo it.

Classify the change by blast radius

Change	Customer/outage risk	Default strategy
Add an index (`CONCURRENTLY`)	Very low — strictly-additive access path; worst case is a wasted index	`additive`: one migration PR, `CREATE INDEX CONCURRENTLY`, monitor post-deploy. No flag needed.
Rewrite a query / change SQL (same result set)	Medium — a plan or logic slip can change results or regress latency	`flagged`: dark-launch behind LaunchDarkly, prove result-equivalence with tests, ramp gradually
N+1 → batch / request cache	Medium — caching can serve stale or cross-tenant data if keyed wrong	`flagged` + unit tests asserting identical results and correct cache key/scope
Change a `SECURITY DEFINER` / RLS function	High — can leak data across tenants or shares	`multi-stage` + flag + explicit security review; never bundle
Backfill / data migration	Medium–high — long locks, replication lag, load spikes	Out of the deploy pipeline: resumable batched script or pgboss job (below)
Column/type/constraint change	High — table rewrites and deployment skew	`multi-stage`, backward-compatible ordering (below)

If a change spans two rows, take the higher risk tier.

The four levers for reducing risk

Multi-stage, backward-compatible PRs. There is always a window where old code runs against the new schema. Never break it:
- Migrations ship in a separate PR from the code that needs them (repo rule) and must be backward-compatible: add columns nullable/with defaults; add constraints NOT VALID then VALIDATE in a second migration; add NOT NULL via a validated CHECK then SET NOT NULL; never rename/drop in the first migration — add-new → dual-read → drop-old in a cleanup migration.
- Safe rollout order: backward-compatible migration → code behind a flag → ramp to 100% → remove flag → cleanup migration. Track each stage in the prs object so a later run knows what still owes work.
- One AccessExclusiveLock per migration; CONCURRENTLY (with --! no-transaction) for indexes so builds never take a write lock.
LaunchDarkly feature flags for any behaviour/result change. notes-service evaluates LaunchDarkly server-side (the client is initialised in workspaces/services/notes-service/src/server.ts; config in config/index.ts; e.g. modules/integration-auth/AuthorizationService.ts reads flags server-side). Before choosing flagged, confirm that server-side flag-evaluation pattern exists and follow it — cite the concrete call site you copied, the way the request-cache pattern cites BreadcrumbsRequestCache.ts. If no suitable server-side flag path exists for your code, downgrade the change to Recommendations rather than shipping an unflagged rewrite (never invent a flag that gives false rollback confidence). Pattern once confirmed: new code path behind perf.<flag-key>, default off; branch at the call site on the server-side flag so the old path stays intact; ramp 1% → 10% → 50% → 100% watching DBM metrics + Sentry; remove the flag + old path in a cleanup PR once stable. The flag is your instant rollback that does not require a revert-and-redeploy. (Front-end/web-app flags load post-auth and can't gate the auth-bootstrap layer — keep perf flags on the backend path.)
Backfills as resumable, out-of-pipeline scripts. Never backfill inside a deploy migration (it blocks the pipeline and can lock tables). Write a standalone script or pgboss job that is idempotent, batched (e.g. 1–10k rows/commit), rate-limited, and resumable from its last processed id. Validation splits by read vs write: measure the per-batch SELECT cost, row counts, and lock footprint against a replica (read-only); run the actual write backfill against a staging primary first, watching replication lag, before production. For volatile defaults (gen_random_uuid()): add column no-default → backfill → set default for new rows.
Tests as the correctness proof.
- Unit tests are mandatory for any query-logic change (repo rule; index-only migrations exempt). For a query rewrite, assert the new query returns byte-identical results to the old on representative fixtures — this is what makes a "faster" change safe.
- e2e (Playwright) for user-facing paths the change touches. A perf change that alters a rendered list/order must be exercised end-to-end. Note: a flaky/new e2e can pass platform-ci (soft-fail core-smoke) yet fail post-merge product-e2e-staging and revert the whole PR — validate e2e against staging or land it separately.
- Dark-launch comparison (for flagged changes): prefer the offline unit result-equivalence test plus post-deploy metric/plan comparison. If you shadow-execute old vs new in production to diff results, do it only on sampled traffic (a small %) under monitoring — never run both paths for 100% of a top-DB-time signature, or the validation itself doubles the load on the exact query you're trying to speed up.

Reversibility is mandatory

Every PR states a rollback plan that is fast and does not depend on a hotfix: a flag flip (preferred), a plain revert (additive changes), or "the index is harmless; drop in a follow-up." If the only way to undo a change is a data-repair script, the change is not safe enough yet — add a flag or split it.

Phase 3: Fix — Implement Performance Improvements

One fix per PR, on its own branch and worktree. The fix family must match the latencyCause from Phase 2a and the rolloutStrategy chosen above — an additive fix is a single PR; a flagged/multi-stage fix keeps the old path and gates the new one.

Index additions (`additive` — for scan/I/O-bound signatures)

New migration: cd workspaces/services/notes-service && yarn dbMigrateCreate AddIndex<TableColumn>
Use CREATE INDEX CONCURRENTLY with --! no-transaction at the top; remove ALL template content from the generated file (the SET statement_timeout, SET session_replication_role, and cleanup block) — keep only the directive and your SQL. The runner still appends the session-state reset automatically.
One AccessExclusiveLock per migration — one table per migration file. Split multi-table changes.

Query optimization (`flagged` — for scan / memory-spill signatures)

A query rewrite changes results if you get it subtly wrong, so it does not ship unflagged:

Add the rewritten query alongside the existing one (new .sql file / new branch in the resolver); run yarn nx run notes-service:pgTyped to regenerate types; then build + test.tsc.
Branch at the call site on a server-side LaunchDarkly flag (perf.<flag-key>, default off) so the old query path stays live and is the instant rollback. Confirm the server-side flag pattern exists first (see Risk lever 2).
Add a unit test asserting the new query returns identical results to the old on representative fixtures.
Remove the old path + flag in a cleanup stage PR once ramped to 100% and stable.

N+1 fixes (`flagged` — the dominant fixable category)

Replace per-row queries with a batch query using ANY($1) / IN, or a request-scoped cache. Canonical prior art in this repo: workspaces/services/notes-service/src/modules/graphql/BreadcrumbsRequestCache.ts — a Map<key, Promise<T>> memoised per GraphQL request, attached to Context. It is the exact N+1 pattern: a field on the Context interface (modules/graphql/types.ts), instantiated in GraphQLMiddleware.ts (request-handler and executeQuery paths, ~lines 179 and 439), and called via context.breadcrumbsRequestCache.<method>(...) from the plugins instead of the underlying service. Siblings BillingServiceRequestCache.ts and LiveDatumCountRequestCache.ts follow the same shape.
Gate the new batched/cached path behind a flag and assert result-equivalence + correct cache key/scope (the cache key must include tenant/claims so it can never serve cross-tenant data) with a unit test, exactly as for a query rewrite.

Transaction / lock fixes (for lock-wait signatures — NOT indexes)

Shorten transaction scope; move non-critical work outside the transaction; reduce FOR UPDATE hold time. These are correctness-sensitive — treat as their own change with careful review and, where behaviour changes, a flag.

Bloat / vacuum (for the bloat-vacuum cause — NOT indexes)

This is usually an autovacuum-tuning or one-off VACUUM/REINDEX recommendation, not a code change. Capture it in the dossier and route to the owning team via Recommendations unless there is a clear, safe, reviewable change.

General rules

Follow the project CLAUDE.md strictly — no any, ?? not ||, curly braces always, type-only imports, getColor() for colors, no eslint-disable, no TODOs/commented-out code.
Read the relevant package's AGENTS.md before changing it.

Phase 4: Local Validation — Full Build & Test Suite

Every change must pass the full local validation gauntlet before any commit. Do not skip steps. If any step fails, fix it and re-run from the top of this phase.

4a. Identify affected workspaces

yarn nx show project notes-service --json | jq '.implicitDependencies'

4b. Run the full suite for EACH affected workspace (sequentially)

yarn nx run notes-service:build        # TS compilation — catches pgTyped/import/type errors
yarn nx run notes-service:test.tsc     # stricter type check
yarn fix.oxlint && yarn test.oxlint    # lint (fix, then verify)
yarn nx run notes-service:fix.format && yarn nx run notes-service:test.format
yarn nx run notes-service:test         # full unit + integration suite

Run the same suite for any other affected workspace (libraries, web-app). If you changed web-app, also run yarn nx run oxlint-replacement-rules:check (CI runs ripgrep/ast-grep rules local oxlint misses).

4c. Validate migration files

cat workspaces/services/notes-service/src/db/migrate/<your_migration_file>.sql

--! no-transaction migrations: no template content remains (no -- Write migration here, no BEGIN/COMMIT, no SET statement_timeout).
Regular migrations: SQL is after the template comment; one AccessExclusiveLock only.
Never create migration files by hand or hand-pick timestamps — always yarn dbMigrateCreate, which uses a real Date.now() prefix. For an ordered multi-step change, run it once per step.

4d. Run pgTyped if query files changed

yarn nx run notes-service:pgTyped
yarn nx run notes-service:build && yarn nx run notes-service:test.tsc

CI does not diff pgTyped output — regenerate locally and confirm a no-op diff, or hand-edited IR drift will ship silently.

4e. Verify test coverage of the change

For query optimizations (not index-only changes), confirm a test exercises the modified path:

yarn nx run notes-service:test -- --coverage --collectCoverageFrom='<changed-file>' --findRelatedTests <changed-file>

If the changed query is not exercised, write a test that runs the query path and asserts it returns the same data as before (functional correctness, not performance). Re-run 4b after adding it.

4f. Gate check

build passes · [ ] test.tsc passes · [ ] test.oxlint passes · [ ] test.format passes · [ ] full test passes
query files changed → pgTyped run, build/tsc re-validated, no-op diff confirmed
query logic changed → a test asserts result-equivalence, and the new path is flag-gated
index/query change → local plan corroboration attempted (best-effort; see Phase 2b — production metrics are the authoritative proof)
no unrelated files staged

If any check fails, do NOT proceed. Fix and re-run from 4b.

Phase 5: PR Creation & CI Validation

5·preflight. Local review before committing

Before the final commit, run the repo's dvtl-review skill (or the .agents/skills/dvtl-review review) over the staged diff. It runs the same Dovetail-specific checks the claude-code-review.yml CI bot ("claudebot") runs — architecture violations, migration safety, any/defensive try-catch/hardcoded-colors/eslint-disable/TODO/secret patterns — so you catch them locally instead of round-tripping through CI. Fix everything it surfaces before committing. This is the cheapest place to reduce PR risk. (Skip only for trivial doc-only changes.)

5a. Branch and commit

git checkout -b perf/<short-description> origin/master
git add <specific-files-only>
git commit -m "perf: <description of the optimization>"

5b. Push and create a draft PR with the full dossier

git push -u origin perf/<short-description>
gh pr create --draft --label "AI assisted" \
  --title "perf: <description>" \
  --body-file <(cat <<'PREOF'
## Description

**What** — <1-2 sentence summary of the fix>

**Customer impact** — <faster page loads, fewer timeouts, etc.>

## Performance Analysis

### Problem

| Metric | Value | Source |
| --- | --- | --- |
| Query signature | `<normalized SQL>` | APM spans (`resource_name`) |
| Latency cause | `<scan-io / n-plus-1 / lock-wait / …>` | Phase 2a triage |
| Total DB time | `<X>` s/hr | DBM `postgresql.queries.time` |
| Avg latency (time/count) | `<X>` ms | DBM, last 7d |
| p50 / p95 / p99 | `<X>` / `<X>` / `<X>` ms | APM span `@duration` |
| Throughput | `<X>` /hr | DBM `postgresql.queries.count` |
| Rows / exec | `<X>` | DBM `postgresql.queries.rows` |
| Logical reads / exec | `<X>` (threshold >1000 = scanning) | DBM shared_blks |
| Cache hit ratio | `<X>%` (shared-buffer only; interpret with blk_read_time) | DBM shared_blks |
| Dead-tuple ratio | `<X>%` (rules out bloat as the cause) | DBM `dead_rows`/`live_rows` |
| Trend (7d vs prior 7d) | `<improving/stable/degrading by X%>` | DBM |
| Regression cause | `<deploy/migration/flag @ time, or "data growth">` | `get_change_stories` |
| Related Sentry issues | `<IDs or "none">` | Sentry |
| Timeout/error events (24h) | `<count>` | Datadog logs |
| Affected endpoints | `<list>` | Codebase trace |

### Root Cause

<Why it is slow, tied to the latency cause. Name the table, approx row count, existing indexes (or lack), dead-tuple ratio, and the plan characteristics.>

**Code path:**

(file:line) → (file:line) → (file:line) → SQL:


**Local plan corroboration (best-effort; production metrics are authoritative):**


### Fix

<What the fix does and why it resolves the root cause.>

**Predicted improvement (post-deploy check will verify):**

- logical reads/exec: `<current>` → `~<expected>` (primary attributable win)  ·  p50: `<current>` → `~<expected>` ms

**Risk assessment:**

- <e.g. "CONCURRENTLY build ~5–10 min on prod table size; no write lock">
- <e.g. "strictly-additive access path — no plan-regression risk"; or "flagged: old path stays live, flag flip = instant rollback">
- <write-amplification cost of the new index on this table's write rate>

### Supporting Evidence

<Summarize the DBM metric values and span/log samples backing the table above, with the time window so a reviewer can reproduce via the same MCP tools.>

## Rollout

**Risk tier & strategy** — `<additive | multi-stage | flagged>`. <One line on why this is the lowest-risk way to ship the win.>

**Staging** — <if multi-stage: the ordered PR sequence, e.g. "1/3 migration → 2/3 flagged code → 3/3 flag removal + cleanup". Otherwise "single PR — strictly additive".>

**Feature flag** — `<perf.flag-key, default off, ramp plan, cite the server-side eval site copied>` or "none — additive change, no behaviour difference".

**Backfill** — `<resumable/batched script or pgboss job; SELECT cost verified on replica; write run on staging primary first>` or "none — no data change".

**Rollback plan** — <the *fast* undo: flag flip (preferred) / plain revert (additive) / "index is harmless, drop in follow-up">. Must not require a data-repair hotfix.

**Testing**

- [x] Full test suite passes locally (`yarn nx run notes-service:test`)
- [x] Type check / lint / format / build pass
- [x] `dvtl-review` preflight clean
- [ ] pgTyped regenerated (if query files changed)
- [ ] Unit test asserts result-equivalence (if query logic changed)
- [ ] New/changed behaviour is gated behind a LaunchDarkly flag, or the change is strictly additive
- [ ] e2e (Playwright) covers the user-facing path (if one is affected) — validated vs staging
- [ ] Post-deploy monitoring — confirm the predicted metric improvement within 1h

**Deployment safety check**

- [x] Database migration changes are isolated from application-code changes.
PREOF
)

Draft first, always. Only gh pr ready once CI is green and the review is resolved. Add the "AI assisted" label on every PR. Do NOT add any "Generated with Claude Code" footer — the label is sufficient.

If gh reports insufficient scopes creating the PR, the ambient GITHUB_TOKEN/GH_TOKEN may be a limited fine-grained PAT without PR-write. Fall back to the keyring auth: env -u GITHUB_TOKEN -u GH_TOKEN gh pr create ....

5c. Wait for CI and validate — Buildkite MCP

mcp__buildkite__list_builds(org_slug: "dovetail", pipeline_slug: "platform-ci",
  branch: "perf/<short-description>", per_page: 1)

Take the newest build's number, then poll its state (don't poll faster than every 60s — branch builds take 20–40 min):

mcp__buildkite__get_build(org_slug: "dovetail", pipeline_slug: "platform-ci", build_number: "<n>")

state progresses scheduled → running → passed | failed | blocked | canceled. If it fails, list the build's jobs, find the failed job's id, and read its log:

mcp__buildkite__list_jobs(org_slug: "dovetail", pipeline_slug: "platform-ci", build_number: "<n>")
mcp__buildkite__read_logs(org_slug: "dovetail", pipeline_slug: "platform-ci",
  build_number: "<n>", job_id: "<failed-job-id>", limit: 300)

(buildkite__get_failed_executions is a separate Test Engine call needing test_suite_slug + run_id — not the CI build_number/job_id. For ordinary CI-failure triage, read_logs on the failed job is the path.)

If CI fails: read the log → fix on the same branch → re-run the full local validation gauntlet (Phase 4) → push → re-check. Repeat until green.

Merge-queue builds are separate. After merge, CI runs again on a gh-readonly-queue/master/pr-<n>-<sha> branch that is invisible in the PR's check rollup — if a merge is reverted, look for that build via list_builds with the queue branch.

5d. Trigger, wait for, and address claudebot's review

The automated Dovetail code review ("claudebot", .github/workflows/claude-code-review.yml) is a required risk gate. Its trigger matters: the job runs only when (!draft && action != 'labeled') || contains(labels, 'claude-review'). A --draft PR carrying only the "AI assisted" label therefore does not get reviewed. To review the draft before marking it ready, add the trigger label:

gh pr edit <PR_NUMBER> --add-label "claude-review"

Then wait for positive confirmation the review actually ran — do not treat "no comments yet" as "clean". Poll until a review/comment from the claudebot identity is present (bounded timeout, min wait; the job has a 30-min timeout and continue-on-error, so it won't block CI):

gh pr view <PR_NUMBER> --comments
gh api repos/{owner}/{repo}/pulls/<PR_NUMBER>/reviews --jq '.[] | {user: .user.login, state, submitted_at}'

Set the state to pr-awaiting-review while waiting. "No bot review yet" → keep waiting, never "proceed". Once the review has posted, for each finding:

Assess it on the merits (the repo's receiving-code-review discipline) — don't reflexively agree or dismiss. Verify the claim against the code.
If valid, fix it in code on the same branch, then re-run the preflight (5·preflight) and the full Phase 4 gauntlet before pushing. (A push re-triggers the review on synchronize.)
If a finding is wrong/not applicable, note the reasoning in the PR description, not as a reply.

Do not post comments, reviews, or replies on the PR (gh pr comment / gh pr review) — repo policy is that those appear under the engineer's identity and are only posted when the engineer explicitly asks. You address claudebot's feedback by pushing fixes; you do not converse with it.

5e. Mark PR as ready

Only when CI is green AND claudebot has reviewed AND its actionable findings are resolved:

gh pr ready <PR_NUMBER>

(Marking ready also fires the ready_for_review trigger, so a final review pass runs on the non-draft PR.)

Watch post-merge e2e. A new/flaky Playwright test can pass platform-ci (core-smoke is soft-fail) yet fail post-merge product-e2e-staging and get the PR reverted — see the e2e note in Risk lever 4. Validate any e2e-touching change against staging, or land it separately.

For a multi-stage rollout, repeat 5a–5e per stage, gating each stage's start on the prior stage merging (and, for the cleanup stage, on the flag having ramped to 100% and held).

Phase 6: Report

After all fixes, output a summary:

## Performance Investigation Report — <date>

### Issues Found
| # | Source | Issue | Latency cause | Total DB time/hr | Impact rank | Fix | Confidence |
| - | ------ | ----- | ------------- | ---------------- | ----------- | --- | ---------- |
| 1 | DBM    | Missing index on X.Y | scan-io | 4,000 s/hr | 1 | Added index | high |
| 2 | APM    | N+1 in Z resolver    | n-plus-1 | 600 s/hr | 2 | Request cache (flagged) | high |

### PRs Created
| PR | Title | Rollout | Local validation | CI | claudebot | Predicted improvement |
| -- | ----- | ------- | ---------------- | -- | --------- | --------------------- |
| #NNN | perf: add index on X.Y | additive | ✅ | ✅ | resolved | reads/exec 120k→<200, p50 800→<20ms |

### Recommendations (not fixed)
- <issues that are lock-bound / bloat-vacuum / need schema changes / too risky to auto-fix — include the dossier and the reason>

### Post-Deploy Monitoring Checklist
For each merged PR, verify within 24h by re-querying the same DBM metrics for the signature:
- [ ] PR #NNN — logical-reads/exec and avg latency (`time/count`) dropped into the predicted range
- [ ] PR #NNN — no new Sentry errors related to the change
- [ ] PR #NNN — if flagged: flag ramped to 100% and cleanup PR opened

Verify the post-deploy numbers with the same get_datadog_metric calls from Phase 2a — the prediction in the PR is only credible if you check it.

Repository-Specific Notes (learned from prior runs)

Pragmatic shortcuts discovered on this codebase. Use them to skip dead ends. Treat them as living notes — confirm they still hold with fresh telemetry, since data and code change between runs.

Already-solved patterns (do NOT re-fix)

GraphQL breadcrumbs N+1 — SOLVED. getProjectCategoryBreadcrumbs (called per-row by Project.breadcrumbs, ProjectCategory.breadcrumbs, Dashboard.breadcrumbs, Channel.breadcrumbs, Insight.breadcrumbs, Agent.breadcrumbs) was memoised per GraphQL request in BreadcrumbsRequestCache.ts and is wired into all those plugins + GraphQLMiddleware.ts. Do not re-implement it — it is the canonical prior art for the N+1 request-cache pattern (Phase 3), not an open opportunity.
SELECT * FROM dovetail_private.get_team_product_restrictions ( ? ) — addressed by prior PRs; check CLAUDE.md learnings before re-attempting.

Common skip patterns (don't fix; document and move on)

These recur in slow lists but are not solvable with index/SQL changes — they are latencyCause = client-pool or lock-wait:

select set_config ( ? ) — usually the top signature by call count (thousands/24h). PostGraphile's per-request RLS context setter; latency is connection-pool / claim-resolution cost, not query plans. client-pool. Out of scope.
SELECT * FROM dovetail.rich_text WHERE id = ? FOR UPDATE and other FOR UPDATE PK lookups stuck at seconds — the PK exists; the row is held by a long transaction. lock-wait, not a missing index. Route to deadlock/transaction work. (Confirm with the triage: logical-reads/exec will be tiny — proof it isn't a scan.)
PK lookups at multi-second avg (… rich_text WHERE id = ?, … audio_video_file where id = ? :: uuid) — same story; a sub-ms PK lookup taking seconds is lock/row-lock contention.
Background search / GC queries (CREATE UNLOGGED TABLE dovetail_search.*, dovetail.cached_acl GC, rich_text_search_index integrity) — long-running but not user-facing. Skip unless there's a specific incident.

High-leverage fix patterns

GraphQL N+1 resolvers (PostGraphile) are the dominant fixable category. Group the spans aggregate by operation_name — anything resolving once per parent row is a candidate. The fix is the request-scoped cache pattern (Phase 3), whose canonical example is BreadcrumbsRequestCache.ts (siblings: BillingServiceRequestCache.ts, LiveDatumCountRequestCache.ts). Always flag + result-equivalence-test the new path.
dovetail_private.cached_acls_granting_read() scans dovetail.cached_acl globally (no team_id filter). It's the inner subquery of most RLS SELECT policies — a team scope would be a huge win across many queries, but the blast radius is enormous. Treat as a dedicated multi-stage + flagged change with a thorough rollout plan; never bundle into a "small fix" PR.
Don't bypass RLS in SECURITY DEFINER functions to skip the JOIN to user tables. Tempting on recursive ancestor walks (they only need id/title and are already privileged), but the existing behavior — ancestors the caller can't read are silently dropped — is intentional security behavior. Bypassing it leaks parent folder titles to recipients of a shared link.

Datadog / MCP notes for this org (verified)

DBM query metrics are live and per-signature. The postgresql.queries.* family is tagged by query_signature (plus query, table, command, user, db; instance/role scope tags per the DBM skill). ~115 postgresql.* metrics exist — use the datadog/dbm-postgresql skill references, don't guess names.
The DBM assistant tools are not enabled here (no explain-plan / health-signal / recommendation / sample tools). Get the I/O profile from get_datadog_metric; infer the plan from the I/O profile and corroborate with local EXPLAIN (Phase 2b).
Span latency is @duration in nanoseconds (the bare duration field returns nothing); the DB span operation_name is pg.query; percentiles are P50…P99. (See Phase 1c — canonical.)
Metric payloads can be huge. Group by query_signature alone (not + query) and use top(...) / small limits, or responses overflow.
track_io_timing / track_planning may be off → blk_read_time/mean_plan_time read a flat 0; fall back to shared_blks_read (Phase 2a caveats).
Sentry's issue.category:performance is typically empty here. Use Sentry for error correlation + Seer root-cause, not perf discovery. Org slug dovetail, region https://us.sentry.io.
get_change_stories is the fast path to tie a regression to a deploy/migration/flag.

Buildkite / CI specifics

Org dovetail, pipeline platform-ci. Branch builds take 20–40 min; poll no faster than 60s.
The "AI assisted" label is required on every PR — easy to forget when iterating.
claudebot (claude-code-review.yml) does not review draft PRs unless the claude-review label is present — see Phase 5d.

Important Rules

One fix per PR — don't bundle unrelated optimizations.
Lowest-risk version, always — pick the least-risky rollout that ships the win (additive > flagged > multi-stage), and account for every outage/customer risk with a flag, a stage split, a resumable backfill, or tests. No PR ships a behaviour/result change without a verified LaunchDarkly flag or a strictly-additive argument. Every PR has a fast rollback (flag flip or plain revert).
Trigger and address claudebot — add the claude-review label to the draft, wait for the review to actually run, and resolve its actionable findings before gh pr ready (Phase 5d). Address by pushing fixes; do not post PR comments/replies unless the engineer explicitly asks.
Attribute before you fix — every PR states a latencyCause and only ships if rootCauseConfidence is high/medium. Never add an index to a lock-bound, client-pool, or bloat-vacuum query.
Never run migrations — only create migration files. Migrations run during deploy. (Ask before running any dbMigrate. For a local plan check, create the index in a throwaway local session, not via the migration.)
Don't guess — if you can't find the query in the codebase or can't attribute the cause, note it in Recommendations and move on.
Don't over-optimize — focus on the top 5–10 issues by total DB time.
Evidence is mandatory — no PR without the full Performance Analysis section filled with real MCP data, a testable prediction (the metric that will move and by how much), and local plan corroboration where obtainable. If you can't get metrics, it goes to Recommendations, not a PR.
Local validation is mandatory — never push code that hasn't passed the full Phase 4 gauntlet. No "it's just an index."
Tests are mandatory for query/behaviour changes — result-equivalence unit test for rewrites/N+1; index-only migrations are exempt.
Read CLAUDE.md and the relevant AGENTS.md before any code change.
Only read-only kubectl / DB access during discovery — never rollout restart, delete, apply, scale, or edit unless explicitly asked.
Use MCP tools, never API keys — all Datadog/Sentry/Buildkite access is via the MCP servers; do not reintroduce curl + tokens.

dougrathbone/db-perf-investigator.md