Note: Much of this was extracted and synthesized by Copilot. It's intended as research to guide ideas and discussion — not a completely correct or final implementation plan.
We need an agent that can analyze any CI run and give a 100% accurate account of what happened — which tests failed, which are flaky, which are real regressions, and which should be retried. Today we can't do this.
- CI Doctor — gh-aw agent triggered on
maui-prfailure, files diagnostic issues. Currently broken and needs significant improvement — investigations are session-local (lost between runs), no structured error extraction, no flaky detection, no cross-run correlation - Daily Repo Status — scheduled gh-aw agent for repo intelligence
- AzDO Build Investigator — skill for querying AzDO/Helix results
- Helix Test Scripts — Helix job status and log retrieval
- Test Configuration —
localRerunCount: 2for Controls.*, emptyquarantineRules - Kusto (
engsrvprod/engineeringdata) — 90 days of per-test pass/fail/retry history with statistical regression detection (AzureDevOpsTestAnalysisChi-squared) - Known Build Errors (KBE) — GitHub issues labeled
Known Build Errorwith machine-readable JSON patterns matched against build/test logs by the Build Analysis Processor - Quarantine system —
eng/test-configuration.jsonsupportsquarantineRuleswith matching by test name, assembly, failure message, or callstack (regex/wildcard/contains) — currently empty in MAUI - AzDO Test History API —
_apis/test/test-history/queryandResultDetailsByBuild?$filter=Outcome eq PassedOnRerunfor retry detection
- Fix XHarness blind spot — XHarness exits 0 even when device tests fail, so ADO shows green; fix in the pipeline AND have CI Doctor query Helix aggregated API as fallback
- Improve CI Doctor to extract per-test results — parse TRX from AzDO artifacts + Helix work item APIs instead of regex on console text; get per-test name, outcome, error message, stack trace, duration
- Integrate with Kusto for flaky classification — query
AzureDevOpsTestsSummaryandAzureDevOpsTestAnalysisat analysis time to compare current failures against historical pass rates; no separate database needed - Add error fingerprinting via KBE — use the existing Known Build Error pattern system; create KBE issues for recurring MAUI failures so they're automatically matched and linked
- Populate quarantine rules — use Kusto data to identify tests below pass rate threshold and add them to
eng/test-configuration.jsonquarantineRules; auto-remove when Kusto shows recovery - Auto-file flaky issues — when CI Doctor identifies a new flaky test (not matching any KBE), create a GitHub issue with the
Known Build Errorlabel and structured JSON pattern
Agents parse test output with simple regex ("Failed:\s*(\d+)") and get back a count — no test names, no stack traces, no error messages. XHarness exits code 0 even when device tests fail, so ADO shows green while failures are hidden inside Helix work items.
What's needed:
| Gap | What's Needed | Effort |
|---|---|---|
| Structured test results | Parse TRX from AzDO + Helix work item APIs for per-test detail | Medium |
| XHarness blind spot | Fix pipeline to fail on Helix failures + CI Doctor queries aggregated API | Medium |
| Error fingerprinting | Create KBE issues for recurring failures, leverage existing match engine | Medium |
Kusto already has 90 days of test history with PassOnRetryCount and statistical significance scores. CI Doctor just needs to query it at analysis time.
Classification an accurate CI analyzer would provide:
| Pattern | Classification | Action |
|---|---|---|
| Fails consistently across runs/platforms | Real regression | Block merge, notify author |
| Fails on retry 1, passes on retry 2 | Flaky | File/update KBE issue, don't block |
| Fails only on one platform | Platform-specific | Flag for platform owner |
| Fails with timeout (duration >> normal) | Timing/infrastructure | Retry, likely flaky |
| Matches existing KBE pattern | Known issue | Link to KBE issue, skip |
| New failure, no KBE match | New failure | Investigate, create KBE issue |
Once a test is identified as flaky, it should get auto-assigned to an agent to fix. Today flaky tests are tracked via manual [Skip] attributes with issue URLs in comments — no automation.
CI run → Accurate extraction → Classify flaky vs real
│
┌─────────────────────┤
▼ ▼
FLAKY TEST REAL FAILURE
│ (block merge)
▼
File/update KBE issue
Add to quarantineRules
│
▼
Agent picks up and works on fix
| Capability | How | Used Today |
|---|---|---|
| React to CI failures | workflow_run event on maui-pr completion |
✅ ci-doctor |
| Run on schedule | cron trigger |
✅ daily-repo-status |
| Query GitHub API | Built-in toolsets: repos, issues, PRs, workflow logs | ✅ ci-doctor |
| Query Helix API | web-fetch tool → REST endpoints |
❌ Not yet |
| Query Kusto | web-fetch or scripted query |
❌ Not yet |
| Create/update KBE issues | create-issue safe-output |
❌ Not yet |
Cannot: run dotnet build/test, submit Helix jobs, make code changes
| Capability | How |
|---|---|
| Full .NET MAUI build | SDK + Android SDK + Xcode + JDK provisioned |
| All test types | Unit, integration, device (Helix), UI (Appium) |
| Helix distribution | eng/pipelines/arcade/stage-helix-tests.yml |
| Publish test results | TRX via PublishTestResults@2 |
| Task-level retries | 2-3 retries configured |
| KBE pattern matching | Build Analysis Processor matches logs against KBE issues |
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ gh-aw (Analysis) │ │ AzDO (Execution) │
│ │ │ │
│ • workflow_run → parse results │ │ • Build & run tests │
│ • Query Helix API for truth │ │ • Produce TRX structured output │
│ • Query Kusto for history │ │ • Helix test distribution │
│ • Classify: flaky vs real │ │ • KBE pattern matching │
│ • File/update KBE issues │ │ • Task-level retries │
│ • Update quarantineRules │ │ │
│ • Assign flaky issues to agent │ │ │
└──────────────────────────────────┘ └──────────────────────────────────┘