Why it matters, why it resists measurement, and how to govern automated development
Version 1.26 · Updated 22 August 2026 · Evidence cut-off 22 August 2026 · Links checked 22 August 2026
Written and researched by OpenAI Codex, under the direction and editorial guidance of Peter Roelants
License: Creative Commons Attribution 4.0 International
Working definition. Code quality is the degree to which a software system produces the required outcomes now, while preserving an acceptable cost and risk of understanding, operating, and changing it later.
Software has to do more than work once. It must be operated, diagnosed, secured, extended, migrated, and repaired. A system can be correct today and still be expensive to change tomorrow. It can also be beautifully organized and fail its users. Code quality matters because it influences the cost and risk of the software’s whole life, but it is never a substitute for product quality.
Quality also changes over time. Requirements, dependencies, operating conditions, teams, and tools keep moving, so no audit or cleanup can make quality permanent. A durable practice connects evidence from changes and production back to the code, tests, architecture, development system, and working process.
Coding agents make this distinction more important. They reduce the cost of producing a change, but verification, operation, and future maintenance still carry real costs. If generation becomes faster than the feedback system around it, a team can accumulate defects and structural debt at machine speed. The practical response is to make requirements, constraints, and evidence part of the development loop.
The short version:
- Quality is a profile, not a score. Software must meet its requirements while remaining safe and affordable to operate and change. The relevant balance of correctness, security, reliability, performance, and maintainability depends on the product and its risks. [R2]
- Maintainability is partly latent. Complexity, coupling, coverage, churn, and similar measures are warning signals about future change cost, not quality verdicts. Start with the decision, then choose the evidence. [R10] [R76]
- Agents shift the bottleneck. When generation gets cheaper, verification, integration, operation, and maintenance dominate. Put exact requirements and constraints in executable tools, and protect the evidence used to accept a change.
- Improve from accepted outcomes. Rework, rollback, escaped defects, lead time, incidents, and structural drift reveal more than generated lines, tokens, commits, or pull requests. Use those outcomes to revise the code, tests, harness, and policy. [R132] [R138]
- Evidence is strongest for bounded automation. Dependency updates, dead-code removal, and typed migrations can work at scale when the target is exact and validation is credible. Evidence that open-ended autonomy preserves a general-purpose codebase over years remains thin. [R60] [R61]
- 1. What code quality is and why it matters
- 2. Measuring code quality: evidence and heuristics
- 3. General guidance for agentic development
- 4. Evidence: cases, tools, and limits
- 5. Experimental directions
- 6. Conclusion
- About this report
- Changelog
- References
“High-quality code” is too vague to measure until the relevant qualities are named. ISO/IEC 25010:2023 defines nine product-quality characteristics, including functional suitability, reliability, security, performance efficiency, maintainability, flexibility, and safety. The standard is authoritative as a vocabulary for quality profiles; it does not prescribe one overall score. [R2]
Two related standards clarify what standardization can and cannot do. ISO/IEC 25023 defines product-quality measures but deliberately leaves rating ranges to the product, integrity level, and user need. ISO/IEC 5055 defines automated source-code measures based on selected architectural and coding weaknesses. These standards make vocabulary and calculation more consistent. They do not prove that one threshold predicts maintenance cost in every repository. [R67] [R68]
It helps to separate three levels:
- Outcome quality: does the software create the intended user or business result?
- External product quality: is it correct, reliable, secure, responsive, compatible, operable, and safe enough?
- Internal quality: does the implementation support understanding, diagnosis, and controlled change?
The levels are related but not interchangeable. A tidy system can solve the wrong problem. A useful system can also have an implementation that makes every later change dangerous. Mark Seemann calls their intersection sustainable code: software that is useful now and remains malleable enough to sustain the organization. This is a practitioner definition, not a standard. [R42]
The relevant quality profile depends on context. A disposable migration script may prioritize a narrow correct result and auditability. A payment service needs stronger correctness, security, availability, and traceability. A prototype may reasonably accept debt that would be reckless in a medical device. Quality attributes also trade off: caching can improve latency while complicating consistency; redundancy can harm maintainability while improving availability.
Software changes. Requirements move, dependencies age, vulnerabilities appear, data grows, incidents reveal hidden assumptions, and maintainers come and go. Internal quality matters because it affects what those changes cost.
Parnas’s information-hiding argument remains a useful foundation: divide a system around decisions likely to change, so that changing one decision does not require reconstructing the whole program. His paper compares two functionally equivalent decompositions and explains why their flexibility and comprehensibility differ. It is a foundational design argument, not a modern outcome study. [R4]
Robert C. Martin makes a similar practitioner argument: when intent is obscure, dependencies spread changes, and tests fail to localize problems, each new change takes longer. His claim that keeping code clean is the only way to go fast is memorable but stronger than the evidence warrants. The underlying mechanism is plausible; the slogan is not a universal law. [R3]
Martin Fowler makes the economic argument more explicitly. He separates external quality, which users can observe, from internal quality, which mainly changes the cost of adding and correcting behavior. Better internal design can therefore pay for itself through cheaper later changes. Fowler also labels his familiar payoff curve the Design Stamina Hypothesis and acknowledges that neither design quality nor the crossover point is objectively measurable. The curve is a useful causal model to test against local change outcomes, not an empirical law that every refactoring will pay back. [R126]
The practical value of internal quality appears in:
- smaller and safer changes;
- earlier defect and security feedback;
- faster diagnosis and recovery;
- clearer ownership and easier knowledge transfer;
- more freedom to change product direction.
Risk is concentrated rather than evenly distributed. A confusing function at the edge of a system may be a nuisance. Similar code at a highly coupled, frequently changed boundary can dominate maintenance cost. In Fowler's debt metaphor, the “interest” is the extra effort paid when that part of the system changes; stable debt may be tolerable, while debt in an active area is expensive. This is why hotspots, where structural difficulty meets frequent change, are more useful than repository-wide averages. Studies have linked churn and ownership patterns with defects, but the strength and direction vary by project and process. [R17] [R19] [R20] [R126]
Maintainability can be framed as the expected cost and risk of plausible future changes:
where:
-
$S$ is the current software system; -
$T$ is the future team, tooling, and operating environment; -
$C$ is the set of plausible future changes; -
$P(C)$ is the unknown distribution over those changes; and -
$\lambda$ expresses how the organization values risk relative to cost.
The minus sign means that lower expected cost and risk imply greater maintainability. The model is conceptual, not a standardized metric: future changes, their probability, and the future maintainer and tools are all unknown.
A design that is excellent for one change distribution may be poor for another. A general plugin architecture is valuable if many independent extensions arrive, but needless if the system is replaced next quarter. A normalized schema can protect consistency yet make an analytics workload harder. Until changes occur, maintainability remains partly latent.
Heuristics fill that gap. They look for present conditions that have often made change expensive: broad dependencies, concentrated complexity, weak tests, frequent co-change, fragmented ownership, or repeated rework. They are evidence about maintainability, not maintainability itself.
Clean Code, SOLID, and Code That Fits in Your Head are best read as collections of design heuristics, not as validated laws.
Martin’s SOLID principles provide language for reasons to change, substitutability, interface size, and dependency direction. Clean Architecture turns one part of that guidance into a rule that can be checked: source dependencies should point toward the policies being protected from external details. [R5] [R6] Martin’s own package-metrics paper warns against relying on metrics alone. [R7]
Seemann starts from limited human working memory. His practical response is local reasoning: small APIs, encapsulated invariants, vertical slices, reproducible troubleshooting, clear separation of concerns, and architectures that reveal only a few parts at each level. [R41] Research supports the general claim that working memory is limited, but it does not validate a universal limit of seven items, seven branches, or a fixed number of lines per method. Cowan’s review suggests a central capacity closer to four chunks under controlled conditions; software thresholds remain heuristics. [R43] [R54]
Martin’s clean-code teaching treats names as communication: code should expose intent rather than make the reader reconstruct it. [R23] Names still remain ambiguous and can become stale. Seemann therefore gives more weight to types, explicit data shapes, pure functions, and controlled side effects as structural protection. [R53] Readability itself is better tested by asking a person who did not write the code to explain or change it than by counting lines. Seemann’s proposed readability review is sensible but subjective and supported mainly by experience rather than controlled evidence. [R44]
Taken together, these heuristics make a narrower claim: make important behavior explicit, control dependencies, and reduce how much state a maintainer must hold in mind. Exact class structures, method lengths, and complexity thresholds remain contextual.
Fowler gives heuristics a similarly limited role. A code smell is a surface indication that may point to a deeper problem, not a defect label. Refactoring is then a sequence of small transformations that preserve observable behavior while improving structure. Tests and refactoring-aware tools can help verify the narrow transformation; they cannot decide whether a smell matters in this context or whether removing it will repay its cost. [R127]
Agent-oriented clean code is code that an agent can locate, understand, change, and verify without excessive search, context, retries, or collateral edits. This is not a separate quality standard. It extends the maintainability model in Section 1.3 by allowing the future maintainer to be a person, an agent and its harness, or a mixed team.
For current systems, human and agent needs mostly reinforce each other: explicit interfaces and data shapes, controlled dependencies, cohesive modules, predictable repository structure, reproducible builds, and fast tests all support local reasoning. Agents add emphasis in four places:
- discoverability: definitions, references, ownership, and relevant tests can be found through stable structure and semantic indexes;
- context economy: a change can be understood from a bounded neighborhood rather than a repository-wide prompt;
- verifiability: important behavior and boundaries are expressed as executable tests, types, schemas, policies, and resource limits;
- traceability: intent, exceptions, generated changes, and the evidence used to accept them remain reconstructable.
These are design goals, not validated universal metrics. Retrieval studies show that relevant cross-file context and graph-backed navigation can help agents localize work, while one vendor-authored minimal-pair study found lower token use on selected cleaner repositories without a higher pass rate. Neither establishes that an agent-friendly structure reduces defects or maintenance cost over years. [R55] [R56] [R110] The practical guidance appears in Chapter 3; stronger claims about context-window fit and machine-oriented code remain experimental and are separated in Chapter 5.
Human readability therefore remains part of agent-oriented quality. People still investigate novel failures, audit consequential changes, and remain accountable for release. A representation that is efficient for a model but opaque to its operators has traded one maintenance cost for another, not eliminated it. [R46]
Five problems make a universal code-quality score misleading.
- The target has several dimensions. Reliability, security, performance, readability, and changeability are not interchangeable. An average can hide failure in a critical dimension.
- Maintainability concerns unknown future changes. A static score observes today’s code; it does not observe tomorrow’s task, maintainer, or incident.
- Every proxy is partial. Coverage observes execution, not assertion quality. Complexity counts control flow, not domain difficulty. Duplication tools find similarity, not whether two rules should evolve together.
- Context and aggregation change the result. Complexity 15 means different things in generated parsing code and a central authorization rule. Studies also show that relationships can appear or disappear when methods, files, and projects are aggregated differently. [R10] [R12]
- Measures affect behavior. Once a number becomes a target, people and agents can raise coverage with weak assertions or lower local complexity by adding indirection. Goodhart’s law is an analogy here, not software-engineering evidence in itself. [R8]
Useful measurement therefore requires triangulation: combine direct requirements, structural signals, change history, and production outcomes, then interpret them in context.
The pre-agentic measurement literature contains a durable lesson: measurement should be goal-driven. The Goal–Question–Metric approach starts with a purpose and point of view, turns that goal into questions, and selects measures that can answer them in the local environment. It rejects the bottom-up habit of collecting every available number and deciding later what the dashboard means. [R76]
For example:
- Goal: reduce the cost and regression risk of changing authorization rules.
- Questions: how long do comparable changes take, where does rework occur, how broad is the change, and which regressions escape?
- Measures: active effort and elapsed time, components touched, coupling and complexity deltas, review findings, rework, rollback, and escaped defects.
- Decision: route high-risk changes to deeper review, strengthen a weak oracle, or refactor a repeatedly expensive boundary.
A useful measure needs more than a formula. Its entity and counting rule must be clear; different tools should produce acceptably consistent results; the measure should relate to the attribute or outcome of interest; and the result should support an action. Kitchenham, Pfleeger, and Fenton’s measurement-validation framework separates these concerns. A metric can be computed reliably while still measuring the wrong construct. [R77]
This matters in an agentic loop because any scalar reward can become an optimization target. Give the agent a metric only when the team can state what decision it informs, what trade-off it may hide, and who may change the rule.
| Layer | Examples | Best question it answers | Main limitation |
|---|---|---|---|
| Product and production | user success, incidents, latency, vulnerabilities, recovery | does the software work under real conditions? | lagging and affected by factors outside code |
| Change outcomes | lead time, review effort, rework, rollback, escaped defects | what did this change actually cost? | tasks and teams differ |
| Behavioral verification | acceptance, property, integration, fuzz, mutation tests | does specified behavior survive? | only as good as the oracle |
| Static and architectural analysis | types, data flow, dependencies, cycles, complexity | which present hazards or policy violations are detectable? | partial and prone to false positives |
| Repository history | churn, co-change, hotspots, ownership | where does change and coordination risk concentrate? | correlational and process-dependent |
| Comprehension evidence | review findings, explanation and change tasks | can a maintainer work with the system? | costly and expertise-dependent |
No layer subsumes the others. Passing tests does not show that the requirement is right. Low complexity does not show that authorization works. Few incidents do not show that the next migration will be easy.
“Complexity” covers several different problems:
- Domain complexity is the number and interaction of real rules, states, and exceptions. Code metrics cannot make a complicated tax, scheduling, or authorization domain simple.
-
Control-flow complexity comes from decisions and paths within a unit. McCabe’s cyclomatic complexity is
$V(G)=E-N+2P$ for a control-flow graph with edges$E$ , nodes$N$ , and connected components$P$ . It estimates the number of linearly independent paths; it is not a count of all possible executions or a direct measure of human understanding. [R69] - Cognitive or representational complexity comes from nesting, non-local jumps, implicit state, unfamiliar notation, and the amount of context required to explain behavior. Cognitive Complexity tries to capture part of this burden, but its relationship with comprehension is still partial. [R9]
- Dependency complexity comes from coupling, cycles, unstable interfaces, shared state, and change propagation across units. A small function can sit inside a structurally dangerous dependency graph.
- Change and coordination complexity appears in files that change together, require many owners, or repeatedly trigger rework and incidents. Repository history observes this dimension better than a syntax tree does.
Complexity affects maintainability when it increases the behavior a maintainer must understand, the paths a test strategy must cover, or the scope through which a change can propagate. It is neither necessary nor sufficient for high maintenance cost. Complicated stable code may rarely matter; a simple-looking but frequently changed boundary may dominate effort. In a controlled study of 60 programmers, change type and programmer ability created substantial variation, while local information-flow coupling still related to residual errors. [R72]
These forms can move without disappearing. Extracting branches into several methods may lower the original method’s score while adding navigation and coupling. Introducing a framework may shorten application code while moving complexity into configuration and runtime behavior. The useful question is not “Did the complexity number fall?” but “Which failure mode became easier to reason about, test, or contain?”
Pre-agentic software engineering produced many measures that remain in current tools. Their longevity should not be confused with universal predictive validity.
| Metric or model | Intended signal | What the evidence supports | Practical use now |
|---|---|---|---|
| Lines of code and unit size | volume and the amount of implementation exposed to change | size often explains part of defect and maintenance effort, but it is also a confounder for many other metrics | compare like with like; inspect large outliers and size growth in hotspots |
| Cyclomatic complexity | independent control-flow paths | useful for locating branch-heavy units and planning path-oriented tests; correlated with size and insensitive to some nesting and data-flow difficulties | changed-code ratchet or review trigger, not a repository grade [R69] |
| Halstead measures | lexical vocabulary, volume, difficulty, and estimated effort | historically influential, but their theoretical claims, operator definitions, and empirical validation have received substantial criticism | diagnostic or research input; rarely a defensible acceptance gate [R72] |
| Maintainability Index | one score derived mainly from Halstead volume, cyclomatic complexity, size, and sometimes comments | calibrated against maintainer judgments in specific industrial data; formulas and scales vary, averages hide outliers, and a score gives weak root-cause guidance | trend only within one stable tool and language; always expose the ingredients; never use as the sole gate [R70] [R71] |
| CRAP | method-level interaction between cyclomatic complexity and uncovered code | deliberately prioritizes complex, weakly covered methods, but the published formula and threshold were fitted to subjective judgments rather than validated against change failures or maintenance cost | changed-function triage with both ingredients visible; not a probability, repository grade, or universal gate [R151] |
| CK object-oriented metrics | class size and complexity, coupling, response set, cohesion, and inheritance | some bounded studies predict fault-prone classes; across a larger literature, size, complexity, and coupling are more consistent than inheritance, while definitions and results vary | investigate high-coupling or low-cohesion outliers in context; avoid universal class grades [R73] |
| Code smells | named symptoms such as God Class, Feature Envy, or Shotgun Surgery | mixed and task-dependent; Fowler defines a smell as an indication requiring judgment, and in one controlled industrial study none of 12 smells had a statistically significant association with increased effort after controlling for file size and number of changes | use as review prompts in active hotspots, not as automatic proof that a refactoring is beneficial [R75] [R127] |
| Remediation-cost or “technical debt” scores | estimated effort to repair the rule violations a tool detects | useful as an issue inventory, but the result depends on the rule set, per-rule repair estimates, assumed development cost, and what the analyzer cannot see | expose the assumptions and individual findings; do not treat the total as an accounting liability or expected future maintenance cost [R82] |
| Quality profiles | distributions of unit size, complexity, duplication, testing, and other properties mapped to a quality model | more explainable than one composite and refined in industrial assessment, but mappings and thresholds still contain expert judgment | show distributions and root causes; calibrate benchmark bands to comparable systems [R71] |
SonarQube illustrates the difference between consistent accounting and observed cost. Its “technical debt” adds per-rule remediation estimates, then compares the total with an assumed development cost to produce a rating. A bounded repair study found those estimates inaccurate and usually high. Section 4.9 examines the model and its evidence in detail. [R108] [R111]
Alberto Savoia and Bob Evans introduced CRAP in 2007 as Change Risk Analysis and Predictions; Savoia later called it Change Risk Anti-Patterns. For a method or function
Here
The equation encodes a policy rather than discovering a law. At full coverage, CRAP equals complexity. With no coverage, it equals
| Cyclomatic complexity | 0% coverage | 50% coverage | 80% coverage | 100% coverage |
|---|---|---|---|---|
| 1 | 2.0 | 1.1 | 1.0 | 1.0 |
| 5 | 30.0 | 8.1 | 5.2 | 5.0 |
| 10 | 110.0 | 22.5 | 10.8 | 10.0 |
| 25 | 650.0 | 103.1 | 30.0 | 25.0 |
| 30 | 930.0 | 142.5 | 37.2 | 30.0 |
The familiar threshold of 30 is therefore a contour through two inputs, not a boundary found in defect data. An untested method of complexity 5 scores 30; a method of complexity 25 reaches 30 at 80% coverage; a fully covered method of complexity 30 still scores 30. Above complexity 30, no amount of coverage can bring the score to 30 or below. Sources also alternate between flagging scores greater than 30 and scores at least 30, which changes the boundary cases. A score of 60 does not mean twice the risk or maintenance cost of 30. [R151]
CRAP is useful because it asks a better triage question than coverage alone: which branch-heavy code lacks even the limited protection that execution coverage can show? It also points to two plausible responses: add focused tests, reduce local control flow, or do both. That makes it more actionable than many opaque composites.
Its evidential status is much weaker than the name suggests. The authors tuned the curve by asking colleagues to rank code from an unspecified collection of Java projects, then iterating until the formula fit those judgments. The public account reports no sample size, rating protocol, inter-rater agreement, held-out validation, defect history, change tasks, or maintenance outcome. The 2007 tool was explicitly called experimental; by 2011 its authors said it had largely been abandoned and acknowledged that coverage can accompany poor tests and that CRAP omits coupling and cohesion. The origin story supports a face-valid, practice-based heuristic. It does not validate the formula or 30 as a predictor of change risk. The broader evidence on coverage is mixed, while complexity itself is partial and sensitive to size and counting rules. [R10] [R12] [R69] [R151]
A 2024 MSc research report is the closest direct academic investigation found for this review. It mined more than 90,000 builds from 201 projects, but studied a new patch-level “delta CRAP” rather than the original method metric. Its reported correlation with patch coverage is partly mechanical because patch coverage is an input to the score. The published equations are internally inconsistent about whether complexity is squared, and the substituted normalized complexity value runs from risky at zero to good at one, the opposite direction CRAP requires. No defect, failed-change, effort, or comprehension outcome validates either version. The author appropriately leaves CRAP's effectiveness open. The study shows interest in evolution-aware composites and the difficulty of constructing them; it does not strengthen the original threshold. [R154]
Implementation details matter:
- Keep the unit local. CRAP was defined per method. File or project sums rise with the number of units; averages can improve merely because a new low-scoring unit dilutes the denominator. Neither change proves that existing code became safer.
- Name both inputs. State which constructs count toward complexity, which coverage criterion is used, which tests and configurations ran, and how generated or unreachable code is handled. Current tools differ on all of these points. [R152]
-
Show the ingredients. A high score caused by complexity 31 at full coverage calls for a different discussion from complexity 10 with no coverage. A composite that hides
$C$ and$p$ throws away its own explanation. - Do not confuse execution with an oracle. Coverage does not show that assertions are strong, edge cases are represented, or requirements are correct. This matters especially when an agent writes the implementation and then generates tests from the same mistaken interpretation.
- Watch for displacement. Extracting methods can lower local complexity while adding navigation, coupling, and awkward boundaries. Sobolev's “complexity waterfall” is a useful practitioner illustration of complexity moving between expressions, functions, classes, and modules, but it is not evidence that mandatory splitting improves maintenance outcomes. [R153]
For agentic development, CRAP works best as a sensor and router, not a reward function. Compute it for changed functions with a pinned tool and test scope; return the score, both inputs, and the delta to the agent; then use high or rising values to request characterization tests, stronger acceptance or mutation evidence, refactoring, or deeper review. Combine it with change frequency, dependency impact, ownership, and business criticality before deciding what deserves attention. The code-writing agent should not be able to pass by weakening tests, changing exclusions, redefining complexity, or editing protected acceptance evidence. After refactoring, verify behavior and inspect coupling and change scope as well as the local score.
The strongest recurring pattern is not a magic metric. Coupling, complexity, size, and change history often carry signal, but the useful subset and threshold vary by system. A five-product Microsoft study found correlations between complexity measures and post-release failures, yet no universally best metric set. Reviews of maintainability research report similar problems with inconsistent definitions, external validity, and comparison against actual maintenance outcomes. [R74] [R81]
| Heuristic | What it usefully detects | Evidence strength | What it cannot establish | Best use |
|---|---|---|---|---|
| Build, type, schema, and compatibility checks | violations of explicit machine-checkable rules | Established mechanism; language- and rule-specific. One study found Flow and TypeScript could detect 15% of a sample of historical JavaScript bugs. [R11] | overall correctness | gate |
| Static and data-flow analysis | known bug and vulnerability patterns | Supported at industrial scale by Google and Meta deployments; effectiveness remains analyzer- and organization-specific. [R13] [R14] | absence of unknown defects | gate for high-confidence findings; otherwise review |
| Line or branch coverage | code not exercised by a test run | Mixed. A 100-project Java study found insignificant project-level and no file-level relationship with post-release defects. [R10] | test adequacy or defect freedom | floor or changed-code ratchet |
| Mutation testing | tests that fail to notice selected semantic changes | Supported in context. Google studies show that incremental mutation testing can scale and that mutants can expose real test gaps. [R15] [R16] [R18] | correctness outside the mutation model | risk-based changed-code check |
| Cyclomatic or Cognitive Complexity | concentrated control flow and likely comprehension burden | Partial. Cognitive Complexity correlates with comprehension time and subjective ratings, with mixed results for correctness; size and aggregation remain confounders. [R9] [R12] | domain complexity or maintainability as a whole | ratchet and review trigger |
| CRAP | branch-heavy code with weak execution coverage | Practice-based and not outcome-validated. The original curve was fitted to subjective peer judgments; current tools and Martin's workflow establish implementability, not predictive validity. [R27] [R151] [R152] | defect probability, test adequacy, architectural risk, or a universal threshold | changed-function triage or a locally calibrated ratchet, with complexity and coverage shown separately |
| Dependency rules and cycles | conformance to a declared architecture | Direct for the declared rule, grounded in information hiding and dependency design. [R4] [R6] | that the declared architecture is appropriate | gate plus architecture review |
| Duplication | representations that may need coordinated change | useful warning with weak semantic precision | that similarity is harmful | changed-code warning |
| Churn, co-change, and hotspots | areas combining change pressure and structural risk | Supported in bounded studies, not causal or universal. [R17] | that churn is bad or stable code is healthy | risk routing |
| Ownership and expertise | fragmented knowledge and responsibility | Supported but process-dependent, including replication work. [R19] [R20] | that fewer contributors are always better | review and succession planning |
| Change size | integration and feedback burden | small batches have a strong practical rationale, though not every change can be split safely. [R21] | intrinsic quality | budget and escalation |
| Independent review | defects, unclear intent, and missing context | Supported in industrial practice, but reviewer quality and independence matter. [R22] | complete assurance | required by risk tier |
The table supports a portfolio of narrow instruments. It does not support ranking repositories, teams, or agents with one composite number.
No tool is “proven” in the sense that installing it guarantees maintainability. The stronger evidence is narrower: compilers enforce specified language rules; industrial analyzers find real bug classes; mutation systems reveal test gaps; continuous fuzzing finds crashes and vulnerabilities; architecture tests enforce declared dependencies. Representative products differ in language coverage, precision, cost, and configuration. [R13] [R14] [R15] [R80]
The following stack combines mature pre-agentic mechanisms with an agentic control pattern. Tool names are examples, not endorsements. [R82]
| Job | Representative tools | Conventional use | Agentic adaptation |
|---|---|---|---|
| Compile, type, and validate interfaces | language compilers; TypeScript, mypy; schema and API compatibility checkers | reject invalid programs and incompatible contracts | hard gate; return the exact diagnostic to the worker; protect public schemas and compatibility baselines |
| Format, lint, and catch local mistakes | native formatters; ESLint, Ruff, clang-tidy, Error Prone | remove style variation and flag selected bug-prone constructs | auto-format; gate only stable high-precision rules; prevent blanket suppressions or generated ignore files |
| Analyze deeper bug and security flows | Infer, CodeQL, Semgrep | find known nullness, resource, concurrency, taint, and vulnerability patterns | scan the diff early and the full affected scope before merge; gate high-confidence findings and route noisy ones to review |
| Test behavior and observe execution | language test frameworks; coverage tools such as JaCoCo, Istanbul, and coverage.py | check examples, integration behavior, and unexecuted code | keep product-level acceptance tests outside the implementer’s unilateral control; use changed-code coverage as a floor, not a quality score |
| Test the tests | PIT, Stryker, mutmut | seed selected semantic changes and see whether tests notice | mutate changed risk-bearing logic; require review of surviving mutants rather than rewarding a repository-wide percentage |
| Explore input and state space | libFuzzer, AFL++, Hypothesis, QuickCheck, Hegel (beta), OSS-Fuzz or ClusterFuzzLite; deterministic simulation for suitable systems | search for crashes, invariant violations, malformed-input failures, and fault-sensitive system states | let agents propose properties, harnesses, and generators, but derive acceptance properties from authoritative intent; measure generator behavior; retain minimized or replayable counterexamples as regression artifacts [R143] [R146] |
| Index and query repository structure | compiler APIs; Tree-sitter; Kythe, Glean, and SCIP; CodeQL and Joern | expose syntax, definitions, references, dependencies, control flow, or data flow, depending on the index | assemble a task-specific repository map; estimate impact and test scope; verify the index’s commit, build configuration, and edge semantics before treating it as evidence [R83] [R84] [R85] |
| Enforce architecture | ArchUnit, NetArchTest, dependency-cruiser, import-linter | check allowed dependency directions, layers, cycles, and forbidden APIs | make the declared architecture executable; fail when a new edge violates it; require an explicit policy change rather than an agent workaround [R78] |
| Aggregate code-health signals | SonarQube, NDepend, Code Climate, language-specific metric tools | combine findings, complexity, duplication, coverage, and remediation estimates | expose the underlying findings and deltas; use the dashboard for triage, not its letter grade as the reward function |
| Mine structural change and risk | git; GumTree, RefactoringMiner, CodeTracker, Coming; repository-mining scripts; CodeScene |
recover typed edits and entity histories; locate churn, co-change, ownership, and defect hotspots | separate refactoring from behavior edits; use history to retrieve context, plan impact, and allocate review; do not forbid change merely because an entity changes often [R101] [R102] [R106] |
| Maintain dependencies | Dependabot, Renovate, OSV-Scanner, lockfiles and software bills of materials | identify known vulnerable or stale dependencies and propose bounded updates | constrain versions and scope, run compatibility evidence, abstain on unsupported migrations, and retain human approval where consequences are unclear |
Three integration details matter more than the brand of tool:
- Run checks where feedback is cheap. Formatting, compilation, focused tests, and diff analysis belong inside the working loop; broader suites and deployment evidence follow before acceptance.
- Make failures actionable. Developers in an empirical Microsoft study reported irrelevant checks, poorly phrased warnings, false positives, slow feedback, and weak workflow integration as major barriers. Agents also need a stable rule identifier, location, evidence, and permitted remediation rather than a generic “quality failed” message. [R79]
- Separate policy from implementation. The worker may repair code, but it should not silently lower a threshold, delete an architecture test, waive a vulnerability, or exclude the changed file from analysis. This is an active configuration concern: SonarQube's MCP interface can expose issue-status changes, including marking a finding false positive. A code-writing agent should not receive that authority merely because it can query findings; waivers need separate permissions, rationale, and review. [R108]
Parsing the whole repository is useful, but “build an AST of the codebase” compresses several different jobs into one phrase. A parser usually produces a syntax tree for a file or translation unit. The tree shows declarations, expressions, and nesting. It does not by itself establish which declaration a name denotes, which implementation a virtual call can reach, which target a build includes, or which service receives a configured message. Roslyn, for example, exposes syntax trees separately from symbols and semantic models. Tree-sitter builds concrete syntax trees and supports language-specific queries, but semantic relations must be added by an indexer or another analysis. [R83]
There is no single canonical AST shared by all tooling for a language. Parsers may preserve or discard punctuation, macros, generated constructs, type information, or error nodes. A useful repository representation is therefore a layered model, not one universal graph.
| Representation | Typical nodes and edges | What it supports | What it misses or approximates |
|---|---|---|---|
| Concrete or abstract syntax tree | files, declarations, statements, expressions; parent–child edges | structural search, rewriting, outline extraction, local metrics | cross-file binding, runtime dispatch, build and configuration semantics |
| Symbol and reference index | modules, types, functions, variables; defines, refers-to, imports, inherits | go-to-definition, find-references, rename, focused context retrieval | dynamic lookup and relations that the indexer or build does not observe |
| Dependency or call graph | files, packages, components, functions; imports, calls, depends-on | impact analysis, layering, cycle detection, centrality, change planning | reflection, function pointers, dependency injection, generated code, and runtime targets may be incomplete or conservatively over-approximated |
| Control-, data-flow, and code-property graph | operations and values; control transfer, definition–use, taint, syntax | path-sensitive bug and security queries; structural pattern matching across several relations | interprocedural scale and language semantics are costly; results depend on the analysis model |
| Build, test, deployment, and runtime graph | targets, artifacts, tests, schemas, services; builds, tests, deploys, calls | test selection, ownership, blast-radius and release analysis | often split across build files, configuration, telemetry, and organizational systems |
The important sequence is:
The tree makes source structure available to a tool; the traversal rules decide what a metric means in that tool. Nesting depth follows parent–child structure fairly directly. Cyclomatic complexity was originally defined over a control-flow graph, although many tools implement a language-specific approximation by counting decisions in the syntax tree. Cognitive Complexity adds another policy layer: selected constructs increment the score, nesting adds penalties, and some shorthand is discounted. Two tools can therefore report different values under the same metric name without either parser being broken.
Radon makes this mechanism unusually easy to inspect. It parses Python with the standard ast module, walks nodes with an ast.NodeVisitor, and applies explicit rules for conditionals, loops, exception handlers, Boolean operators, comprehensions, and other constructs. It then reports cyclomatic complexity, Halstead measures, and a composite Maintainability Index. This is a useful executable example, not validation of the measures: it is Python-specific, its categories and thresholds are conventions, and its own documentation describes the Maintainability Index as experimental. [R98]
The two supplied explanatory articles are helpful introductions to this process. Ahituv illustrates AST visitors for cyclomatic complexity, nesting, and Cognitive Complexity; Lépine shows how syntax-derived metrics and dependency rules can be placed in CI. Neither article is an empirical study. In particular, cyclomatic complexity gives a basis-path count under its graph model; calling it the minimum number of tests for “full coverage” is too broad unless the coverage criterion and feasible paths are specified. The example thresholds in both articles should be treated as policies to test locally, not evidence-backed constants. [R97]
The mature tool families reflect these layers:
- Compiler APIs such as Roslyn and Clang provide the most language-faithful syntax and semantic information when the exact compilation context is available. Tree-sitter offers fast, incremental, error-tolerant parsing across many languages;
tree-sitter-graphcan turn query matches into an application-specific graph. [R83] - Syntax-aware transformation systems use a tree plus semantic and formatting context to make repeatable edits. OpenRewrite calls its representation a lossless semantic tree rather than an AST. Uber used typed recipes, file-level preconditions, and build and test validation to migrate more than 75,000 test classes. This is strong operational evidence for bounded, deterministic transformation, not for graph-based quality scoring or open-ended agents. [R61]
- Semantic indexes such as Kythe, Meta’s Glean, Sourcegraph’s SCIP, and GitHub’s Stack Graphs store definitions and references for code navigation at repository scale. Their schemas and name-resolution strategies differ. They demonstrate operational feasibility, not an effect on maintenance cost. [R84]
- Program-analysis stores expose richer paths. CodeQL extracts a language-specific relational database containing syntax, control-flow, and data-flow relations; it is queryable like a graph but is not a graph database. Joern uses a code-property graph that combines syntax, control flow, and data flow in a directed, attributed multigraph. The original code-property-graph study found previously unknown Linux-kernel vulnerabilities, which supports the security-analysis technique in that setting rather than a general maintainability claim. [R85]
- Architecture graphs turn dependencies into policy. jQAssistant scans code, bytecode, build files, and configuration into Neo4j, then checks Cypher constraints. ArchUnit, NDepend, Lattix, and Understand provide related dependency, cycle, matrix, and metric views with different language and analysis models. Their documentation establishes capability; it does not show that buying the tool improves code quality. [R78] [R86]
Agent-facing products now package several of these layers. SonarQube combines language analyzers, metrics, imported coverage, profiles, gates, architecture, and repository context; AST Metrics combines Tree-sitter-derived measures, dependencies, history signals, and an MCP interface. Their implementations establish capability. Claims about defects, maintenance cost, or agent performance need separate evidence. Sections 4.7 and 4.9 examine that evidence. [R100] [R108]
A repository graph makes several structural heuristics computable. Each asks a narrower question than “is this code maintainable?”
| Measure | Question it approximates | Sensible use | Main caution |
|---|---|---|---|
| In-degree and out-degree | how many units depend on this unit, and how many does it depend on? | identify high fan-in interfaces and high fan-out coordinators | a common utility may have high fan-in by design; edge type and direction matter |
| Strongly connected components | which units participate in a dependency cycle? | forbid new architectural cycles; prioritize large or growing cycles | a cycle is a structural fact only relative to the extracted graph; not every cycle causes observed maintenance harm |
| Reachability or propagation cost | how much of the graph is reachable after a unit changes? | compare blast radius and architecture deltas within one stable model | reachability is potential propagation, not a forecast that every reachable unit will change |
| Betweenness, closeness, eigenvector, and related centralities | which units bridge, reach, or connect influential parts of the graph? | risk routing, ownership review, and inspection of architectural bottlenecks | rankings can change sharply with graph boundaries and omitted edges; a high rank is not automatically a design defect |
| Ego-network density and brokerage | is a unit embedded in a tight local cluster or bridging otherwise separate groups? | examine coordination and defect-prone neighborhoods | interpretation depends on scale and on whether edges mean calls, imports, data flow, or co-change |
| Core–periphery, modularity, and design-structure-matrix measures | is the system concentrated around a core, split into modules, or broadly coupled? | compare releases and investigate structural drift | descriptive architecture patterns are not universal quality grades |
Pre-agentic empirical work found real but bounded signal. In Windows Server 2003, network measures over binary dependencies predicted failure-prone binaries better than a comparison set of complexity measures, and the combined model performed better still. The authors warned that the model might not generalize even to another Microsoft product. [R88] A later study across 30 versions of nine open-source projects found that dependency-network measures improved defect prediction in five of nine evaluation scenarios; the advantage ranged from marginal to larger and was not universal. [R89]
Other studies used dependency structure to describe architecture. Design-structure-matrix and propagation measures have exposed differences between systems and releases, and a study of 78 Java applications found very large class cycles in a substantial share of sufficiently large systems. These results show that graph structure can reveal concentrated coupling and costly-looking tangles. They do not establish that one centrality, cycle count, or propagation threshold causes maintenance outcomes. [R90] [R91]
The graph itself is also a measurement instrument. An empirical comparison of static C call-graph extractors found significantly different results on the same programs because the tools handled macros, pointers, input representations, and approximations differently. [R87] Before using a graph in a gate or agent loop, specify its contract:
- node and edge meanings, including direction and granularity;
- repository revision, build target, compiler flags, generated sources, and external dependencies included;
- known over- and under-approximations;
- freshness, provenance, and how extraction failures are surfaced;
- the decision a metric or query is allowed to influence.
For agents, the safest current uses are navigation and risk routing: retrieve the definitions and neighborhoods relevant to a task, identify likely impact and tests, and request deeper review around central or cyclic areas. Exact declared rules—such as “the domain layer must not depend on adapters”—can be gates. Noisy graph measures should remain review triggers or changed-code ratchets. Section 4.7 reviews the emerging agent evidence.
One AST describes one version of a program. A sequence of ASTs, together with reliable correspondences between their nodes, describes how its structure evolves. Software-engineering research usually calls this temporal, longitudinal, or evolutionary static analysis. “Dynamic analysis” normally means observing a running program, so it is not the clearest name for revision-history analysis.
A simplified pipeline is:
The edit script
The difficult part is identity through change. A method may be renamed, moved to another file, split, extracted, inlined, or translated. If the tracker treats the old entity as deleted and the new one as unrelated, its churn and co-change history becomes misleading. RefactoringMiner generates refactoring-aware AST diffs for commits, pull requests, and commit ranges; CodeTracker follows methods, variables, attributes, and blocks through commit history. Their evaluations support lineage recovery in the languages and curated histories studied, not perfect identity for every language or transformation. [R102]
Once identity is reasonably stable, several longitudinal signals become available:
| Signal | How it is derived | Maintenance question | Main caution |
|---|---|---|---|
| Typed structural delta | normalized inserts, deletes, updates, and moves by syntax or symbol kind | what kind of structure changed, beyond added and deleted lines? | edit magnitude is not semantic impact; one condition change may matter more than a large mechanical move |
| Entity volatility and hotspots | recent or decayed edit frequency combined with size, complexity, coupling, or failures | where does repeated work meet difficult structure? | a central, well-designed policy or fast-moving feature can be legitimately busy |
| Change dispersion or entropy | distribution of changed entities across files, packages, components, or services | is work concentrated or repeatedly scattered across the system? | feature breadth, migrations, generated output, and commit practice can dominate the measure |
| Evolutionary coupling | weighted edges between entities that repeatedly change in the same task or change set | which units travel together even when no static dependency is visible? | co-change records coincidence, not the reason; tangled commits and release batching create false relations |
| Propagation surprise | compare the realized change set with static reachability, prior co-change, and the declared architecture | did the change cross an unexpected boundary or miss a historically related entity? | both the dependency graph and the historical model are incomplete |
| Architectural drift | compare cycles, forbidden edges, centrality, communities, and reachability under one versioned graph contract | are boundaries eroding or bottlenecks growing over releases? | parser, build, graph-schema, and repository-boundary changes can look like architecture changes |
History provides useful signal, but its empirical scope is narrower than the usual design rhetoric. ROSE mined association rules from version histories and, after an initial edit, correctly predicted 26% of further files and 15% of more precise entities in its evaluated systems; its three leading suggestions contained a correct location in 64% of the evaluated cases. Other studies found that co-change can help predict propagation and defects. Yet a large Java study found that roughly half of static dependencies never participated in co-change and that most co-changes involved only a small subset of dependencies. Its conclusion was that dependency structure alone gives an unreliable account of evolvability: static structure and observed change belong together. [R103]
Change-process measures also sometimes add predictive signal beyond static code measures. Relative churn predicted defect density in one Microsoft system; change metrics outperformed static attributes in one Eclipse study; and the entropy of how changes were scattered helped predict fault-prone files in six open-source projects. These are observational defect-prediction results, not direct measurements of maintainability. Change size, task type, file size, developer activity, release practice, and imperfect defect labels can explain part of the association. [R17] [R104] An emerging line of work embeds AST edit sequences for just-in-time defect prediction, with gains reported on Java datasets. It should remain an experimental risk-ranking input until independently replicated across languages, time periods, and operational settings. [R105]
Martin's formulation of the Single Responsibility Principle is about reasons and constituencies for change: gather things that change for the same reasons and separate things that change for different reasons. His Common Closure Principle makes a similar package-level claim, while the Stable Dependencies Principle warns about dependency direction and volatility. [R5] [R7] Repository history can test part of that intuition:
- repeated co-change can identify candidate units that may belong together or need a clearer shared interface;
- frequent changes to a high-fan-in dependency deserve review because many consumers rely on a volatile unit;
- repeated changes that cross the same declared boundary may reveal a misplaced responsibility or missing abstraction;
- a static dependency that repeatedly propagates edits deserves more attention than an equally shaped edge that never does.
The history cannot recover the “reason” by itself. Two files may change together because one feature genuinely spans them, because a commit mixes tasks, because generated artifacts follow their source, or because a mechanical migration touches everything. Conversely, two coupled units may not have co-changed simply because the relevant requirement has not arrived yet. Issue intent, pull-request boundaries, ownership, and review outcomes provide essential context.
A broad change is therefore a question, not a verdict. Concern increases when the same kind of request repeatedly crosses unrelated boundaries, the actual change reaches farther than the declared architecture or prior history suggested, or broad edits coincide with rework, omissions, defects, and slower review. A one-off API migration or behavior-preserving reorganization may be large and still improve the design.
An evolution-aware agent loop can use these signals without turning them into a gameable grade:
- Parse the merge base and proposed revision with a pinned parser, graph schema, build target, and generated-code policy.
- Match entities with refactoring-aware rules before classifying additions, deletions, moves, extractions, and renames.
- Compute typed AST, symbol, dependency, and architecture deltas; keep their edge semantics separate.
- Compare the proposed change with similar historical tasks, co-change neighbors, affected tests, owners, and prior failure or rework evidence.
- Give the agent and reviewer a short structural summary: important behavior edits, moved code, new dependencies, likely missed companions, changed tests, and uncertain mappings.
- After merge, retain rework, rollback, defect, and later-change outcomes so that local thresholds and ranking models can be recalibrated.
Do not gate on raw AST edit count, file count, or co-change centrality. An agent can lower those numbers by hiding work behind indirection or combining responsibilities. Exact architecture violations can be gates; unexpected propagation, rising dispersion, or volatile hubs work better as review triggers and changed-code ratchets. Section 4.8 examines current tools and bounded agent adaptations.
Use three control modes.
- Gates fit exact or high-confidence rules: build success, types, schemas, compatibility, protected dependencies, critical security findings, licenses, and explicit resource limits.
- Ratchets and budgets fit noisy structural signals: do not add a cycle, increase complexity in an existing hotspot, weaken changed-code mutation results, or introduce unapproved duplication.
- Dashboards and investigations fit lagging or confounded outcomes: incidents, rework, change lead time, ownership concentration, architecture-wide trends, and recovery time.
“Deterministic” needs the same discipline as “quality.” A static rule can be repeatable for a fixed source tree, analyzer version, build model, quality profile, and exclusion set. That does not make the rule complete, the finding true, or the configured policy appropriate. Analyzer upgrades, language features, dependency models, generated sources, baselines, and exclusions can all change the result. In an agent loop, pin and record these inputs; protect them from the worker; and distinguish repeatable policy enforcement from proof of correctness or security. [R108] [R115]
A ratchet asks whether the new change made a known risk worse. This is usually more defensible than forcing a mature, heterogeneous repository under a universal threshold.
SonarQube's default new-code gate illustrates both the appeal and the limit of this approach. It offers a practical ratchet and a small-change exception, but its thresholds remain vendor policies rather than empirical boundaries at which software becomes maintainable. Record the locally justified policy and expose the underlying findings. A pass means compliance with that policy, not “high-quality code.” Section 4.9 examines the defaults and evidence. [R108]
Seemann calls limits such as cyclomatic complexity 7 archetypes: simple numbers that trigger attention. He argues that the threshold matters less than the discussion it causes and should be revised or retired when it stops helping. This is experienced advice, not validation of the number seven. [R43]
Tests are evidence only in relation to a requirement or oracle. Acceptance examples protect visible behavior; unit tests localize rules; property tests exercise invariants; fuzz tests explore malformed input; compatibility tests protect interfaces; production canaries observe conditions missed before release.
Property-based testing combines an executable property with generated inputs. Useful patterns include comparison with an independent model or implementation, round trips such as decode(encode(x)) = x, idempotence, monotonicity, invariant preservation, and explicit safety conditions. This can cover families of behavior that a few hand-picked examples do not. It is still sampling, not proof: one counterexample can refute an “always” property, but any finite number of passing cases cannot establish it universally. [R143] [R145]
The generator is part of the measurement instrument. Narrow generators miss important states; broad generators with many rejected cases waste the test budget; an invalid property produces convincing noise. Record seeds or replay artifacts, shrink failures into understandable counterexamples, and inspect generated-value distributions, rejection rates, relevant path coverage, and mutation results. A 2024 study of 30 experienced users found generator design and judging test effectiveness persistently difficult. Section 4.13 reviews that evidence and its limits. [R143]
The terms property-based testing, generative testing, and fuzzing overlap but are not synonyms. Property-based testing usually emphasizes structured generators and domain properties; coverage-guided fuzzing often emphasizes program exploration with crash, sanitizer, or invariant oracles. The distinction matters because the methods explore different state spaces and fail in different ways. Sections 4.13 and 4.14 review the supporting evidence and the Antithesis case. [R145]
Coverage merely shows that execution reached code. It does not show that an assertion would fail if the behavior were wrong. Mutation testing asks a stronger but narrower question: would the suite detect selected semantic changes? Martin makes the same distinction in his practitioner writing, while warning that TDD does not replace architecture. [R31] [R32]
Agent-generated tests deserve extra scrutiny because implementation and tests may repeat the same misunderstanding. A useful hierarchy is:
Useful checks include observing a test fail before the behavior exists, deliberately perturbing an existing implementation, keeping test control flow simple, repeating tests to expose flakiness, inspecting warning suppressions, and using mutation testing. These checks show that a test can detect selected failures; they cannot show that its oracle represents the requirement. [R48] [R50]
Fresh-context, specification-first test generation may reduce shared blind spots. A July 2026 preprint found higher fault detection when tests were generated from task descriptions without the faulty implementation in context. Its selected Python benchmark tasks and simplified workflow make this an experimental independence control, not a repository-level effect estimate. Asking one agent to perform a test ritual is likewise no substitute for an independently owned requirement. Section 3.4 gives the practical pattern; Section 4.12, Section 4.13, and Section 4.14 examine the studies and product cases. [R130] [R149]
Once a change arrives, maintainability becomes partly observable. The closest measures are task-level outcomes: active effort, elapsed time, comprehension and review work, components touched, rework, regressions, and recovery. Controlled maintenance tasks improve comparability but cover few systems and maintainers. Operational history is more realistic but confounded by task difficulty, staffing, process, and business priority.
Repository history turns some of the counterfactual in Section 1.3 into observed data. Retain, for each accepted change:
- task and risk class;
- structural deltas and components touched;
- test, analysis, review, and mutation evidence;
- human and compute effort;
- rework before acceptance;
- rollback, incident, and defect links after release;
- the later cost of changing the same area.
Then ask whether a signal improves decisions locally. Does added complexity predict rework after controlling for change size? Do surviving mutants reveal assertion gaps? Do dependency violations precede broad changes? Which checks find defects, and which mostly create noise?
Compare like with like: maintenance and feature work differ, as do generated code and policy code, and a five-line schema change may carry more risk than a large mechanical rename. Use distributions and outliers instead of averages alone. Where enough data exists, test whether a signal adds information after controlling for obvious confounders such as change size, file size, task type, and component.
Representative-change replay probes the counterfactual more directly. Run several plausible changes against before-and-after snapshots with fresh sessions, a fixed model and harness, repeated trials, and a protected behavioral oracle. Include the cost of the refactoring, then measure later task time, tokens, correctness, rework, and regressions. A 2026 one-application demonstration illustrates the method but does not provide a maintainability estimate; Section 4.12 examines its limits. [R128]
Version thresholds, record exceptions, and review false-positive and escape rates. Periodically ask whether the measure changed a useful decision and whether the predicted outcome arrived. A borrowed heuristic becomes useful when it is treated as a monitored local instrument rather than as a definition of quality.
This supplies the measurement arm of the continuous-improvement system in Section 3.7. Measurement becomes improvement only when evidence leads to an owned, reversible intervention whose later effects are checked.
This chapter contains the report's recommendations. Sections 3.1–3.2 explain the flow problem, Section 3.3 defines the production baseline, and Section 3.5 lists controls justified only by particular risks. Section 3.7 describes how to revise the system from observed results. Named examples remain in Chapter 4; unvalidated proposals remain in Chapter 5.
Coding agents can inspect repositories, edit many files, run tools, and iterate with less step-by-step supervision. Dark software factories extend this model by moving people toward specification and governance while agents produce most implementation. [R1]
The main economic change is simple: generation becomes cheaper relative to verification, integration, operation, and later maintenance. More candidate designs can be explored and more routine work can be automated. The same capacity can also overwhelm review and CI, multiply dependencies and tests, and produce defects faster than delayed feedback can reveal them.
Current agents add six recurring risks:
- short-horizon optimization: passing today’s tests can reward a local patch that makes later changes harder;
- incomplete context: important assumptions may sit outside the retrieved files or prompt;
- shared blind spots: code and tests derived from the same ambiguous instruction may agree on the same wrong behavior;
- cheap proliferation: files, wrappers, dependencies, and tests are easier to create than to justify;
- weak continuity: a later run may not know why a boundary or exception exists;
- activity bias: generated lines and merged changes are visible before rework and maintenance cost arrive.
These risks do not prove that agents produce worse code in every setting. They explain why fast generation needs fast, independent feedback.
An agent-generated patch is inventory, not delivered value. It becomes valuable only after the system has reviewed, verified, integrated, released, and learned from it. Optimizing generation alone can therefore increase queues, review delay, merge contention, rework, and operational risk while every agent appears locally productive.
Gene Kim's Three Ways gives this problem a useful DevOps and Lean vocabulary: improve flow through the whole system, shorten and amplify feedback, and build a habit of experimentation and learning. The Phoenix Project makes the same argument through a fictionalized constraint-and-improvement story. These are practitioner frameworks, not controlled evidence, but they expose the right unit of analysis: the path from requested change to observed outcome, not the coding step by itself. [R121] Section 4.11 examines the transfer to agentic development and its evidence limits.
In practice:
- map the full path from task selection to accepted code, deployment, and production feedback;
- cap concurrent agent work at the capacity of review, CI, integration, release, and operations rather than at the number of available agents;
- keep changes and queues small enough that failures remain attributable and reversible;
- measure queue time, feedback latency, rework, escaped defects, and accepted outcomes alongside generation cost;
- use incidents, review findings, and recurring agent failures to improve the harness and then reassess where the constraint moved.
DORA's 2025 research is consistent with this systems view: greater AI use was positively associated with throughput and product performance but negatively associated with delivery stability, with surrounding technical and organizational capabilities shaping the result. The study is large and mixed-method, but its survey relationships remain observational rather than causal. [R38] [R124]
The following controls are a baseline for any agentic workflow that changes production software.
- Classify the risk before generation. Identify product, security, data, compatibility, and operational consequences. The evidence required for a documentation fix should differ from that required for an authentication migration.
- Make intent authoritative. Link the task to versioned acceptance examples, invariants, public interfaces, architecture decisions, and operational constraints. Record ambiguity instead of letting the agent silently choose.
- Protect independent evidence. Do not let the implementation worker silently weaken product-level acceptance tests, security policy, or compatibility rules to make its change pass.
- Keep changes small enough to diagnose. Prefer a vertical slice with a clear outcome, bounded diff, and reversible release. Small batches improve feedback and recovery, although some migrations cannot be split safely. When restructuring, separate behavior-preserving refactoring from feature behavior where practical, prefer semantics-aware transformations, protect the behavioral tests, and scrutinize regions where the two kinds of edit remain tangled. [R21] [R127] [R131]
- Use the narrowest reliable mechanism. Prefer an ordinary deterministic transformation when it can express the change exactly. Put builds, types, schemas, dependency policies, security checks, licenses, and resource limits in the pipeline, not only in a prompt. For a large repository, use a graph-backed index to select context and estimate impact, but treat it as an index with declared semantics and freshness rather than a complete model of the running system. Section 2.7 maps established tool categories to this loop; Section 2.8 covers repository graphs; Section 2.9 covers their evolution across changes.
- Preserve provenance. Record the request, model and tool context, changed artifacts, test results, approvals, and exceptions needed to reconstruct why a change was accepted.
- Observe and recover. Use telemetry, staged rollout, explicit stop conditions, and rollback appropriate to the failure mode.
- Connect outcomes back to the gate. Link rework, incidents, and later change cost to the change and evidence that admitted it.
- Test the factory itself. Version and test prompts, context assembly, routing, merge policy, evaluator invocation, and failure defaults. Release material harness changes through canaries, keep a rollback path, and fail closed when required evidence is absent.
A prompt such as “follow SOLID” can guide generation. It cannot establish compliance or correctness. A compiler rule, protected test, or dependency checker gives the loop a result it can act on.
Repository instruction files are useful as maps and for unusual local conventions, but they are not control systems. Early benchmark results are mixed: more instructions can add inference cost without improving success, while narrow negative constraints sometimes help. Keep the prose short and specific, and move important rules into executable checks. [R57] [R58]
The harness deserves the same release discipline as the product. Version it, test model-specific behavior, isolate material changes, soak them, release them gradually, and keep rollback available. A first-party Anthropic postmortem in Section 4.5 shows how configuration defects can survive an otherwise strong test and review process. [R66]
Multiple agents do not automatically provide independent assurance. Agents that share the same model, prompt ancestry, retrieved context, and mutable test suite may reproduce one misconception several times.
Independence improves when the source of evidence changes:
- acceptance examples come from product intent rather than generated implementation;
- security checks use deterministic data-flow or dependency analysis;
- an architecture check reads a versioned policy;
- QA receives user scenarios and a deployed artifact, not only the implementer’s summary;
- critical policy and test changes require separate authorization;
- production canaries can stop a rollout without asking the producer.
For generated properties and tests, the practical pattern is: draft them from the product contract in a fresh context, have an accountable reviewer settle ambiguous guarantees, then expose the implementation for coverage and adversarial refinement. Do not let observed implementation behavior silently become the specification. A recent benchmark study found substantially lower fault detection when the same conversational context contained faulty code, but its Python tasks and simplified workflow do not establish the size of that effect in repositories. [R149] Section 2.11 covers the measurement details.
Seemann’s speculative supply-chain scenario leads to a practical rule: treat agent changes like contributions from an unknown author, using provenance, least privilege, and review proportional to consequence. [R49]
An AI reviewer is another probabilistic sensor. In one controlled experiment, review helped when the reviewer was stronger than the producer and hurt in the reverse pairing. The tasks were single-file programming problems and the reviewer could not run tests, so the result should not be generalized to repositories. A second model is not independent proof; it should produce findings that a test, analyzer, or person can confirm. [R59]
The baseline is not enough for every system. Add stronger controls when the failure mode justifies their cost:
- property and model-based tests for important invariants, especially where an independent model, round trip, or stable domain law is available;
- fuzzing for parsers, protocols, hostile input, and other large input spaces;
- deterministic simulation and fault injection for concurrency- and coordination-sensitive systems that can be isolated sufficiently;
- incremental mutation testing for critical changed logic;
- performance and resource budgets for constrained services;
- formal specification or verification for narrow high-consequence components;
- independent security review and penetration testing;
- migration rehearsal, compatibility checking, canaries, and rollback for data or interface changes.
Correctness matters unevenly. Some exploratory tools can tolerate errors; security, finance, health, infrastructure, and decision-support systems may cause serious harm while appearing to run normally. Seemann’s examples support risk classification, not a claim about how common correctness-critical software is. [R51]
Remove controls that add delay without improving decisions. A mature system uses the smallest set of varied, timely evidence that covers its actual risks.
Section 4.13 reviews the established property-testing evidence and Hegel; Section 4.14 treats Antithesis as a separate platform and agent-workflow case.
Tokens, lines, commits, and pull requests measure activity and cost. They are not quality or productivity measures.
Evaluate review tools on representative local changes rather than adoption or comment volume. Use a stable corpus of pull requests with known or adjudicated issues, several evaluators, and a rubric covering correctness, severity, actionability, and noise. Record missed issues, time spent, accepted findings, and downstream outcomes; rerun the comparison when the model or repository changes. Section 4.10 describes two small company evaluations that illustrate this method without establishing a universal winner.
| Dimension | Useful measures |
|---|---|
| Behavioral correctness | acceptance failures, invariant violations, mutation survivors, escaped defects |
| Change safety | regressions, rollback, hotfixes, compatibility breaks, blast radius |
| Structural trajectory | new cycles, dependency violations, complexity growth in hotspots, clone and warning deltas |
| Changeability | time and human effort to acceptance, components touched, rework, later cost of changing the area |
| Test health | flaky rate, runtime, mutation strength, tests weakened with implementation changes |
| Factory health | work in progress by stage, queue and feedback time, run-to-accept rate, retries, context and tool use, compute cost, human intervention, evaluator disagreement |
| Product and operations | user success, reliability, latency, security events, recovery time |
Keep the dimensions separate, and preserve task and risk classes instead of reporting one average. Faster local production need not improve delivery performance when instability and feedback are ignored; Section 3.2 covers the supporting DORA evidence.
A quality gate answers whether one change satisfies today's policy. Continuous improvement asks whether the product, codebase, tests, tools, harness, and policy should change before the next one. It is a second-order loop: improve not only the software, but the system that produces and verifies it.
The loop also makes partly latent maintainability more observable. Complexity, coupling, churn, mutation survivors, and agent retries begin as hypotheses about future change cost. By linking them to later effort, rework, incidents, and representative-change probes, a team can recalibrate, narrow, or discard the signals as described in Section 2.12.
Several traditions converge on that idea without being interchangeable:
| Tradition | Useful contribution | Boundary |
|---|---|---|
| Quality and lifecycle management: ISO 9001, ISO 9004, ISO/IEC/IEEE 90003 and 12207, and ISO/IEC TR 33014 | objectives, responsibilities, evidence, corrective action, lifecycle processes, and an improvement program that can be tailored to local work | conformity or process maturity does not establish product quality or prove that a process improves outcomes [R132] [R133] [R134] |
| Measurement, service, and AI management: ISO/IEC/IEEE 15939, ISO/IEC 20000-1, and ISO/IEC 42001 | information-driven measurement, operated-service feedback, and governance of organizations that develop or use AI systems | disciplined management cannot rescue an invalid proxy, and certification does not establish that code or an agent is correct, secure, or maintainable [R135] [R136] [R141] |
| DevOps and Lean: the Three Ways and The Phoenix Project | end-to-end flow, fast feedback, learning, and repeated work on the current constraint | practitioner synthesis and instructional fiction rather than a controlled validation of a particular program [R121] |
| Quality Improvement Paradigm and Experience Factory | characterize the context, set goals, measure, analyze, and reuse local experience | an influential engineering model whose value still depends on representative evidence and follow-through [R137] |
The wording matters. Correction fixes the observed problem. Corrective action changes a cause or control to reduce recurrence. Continual improvement repeatedly improves suitability, adequacy, or effectiveness; it need not be one uninterrupted rise. A team that repairs a defect but leaves the same ambiguous specification, weak test, or unsafe agent permission in place has corrected the instance without improving the system.
The ISO management-system cycle is often summarized as Plan–Do–Check–Act. In this context, Plan means choose an objective, risk, scope, measure, and intervention; Do means run the controlled change; Check means compare evidence with the objective and guardrails; and Act means adopt, adapt, revert, or correct the system. Merely repeating build–test–merge is iteration, not necessarily improvement. [R132]
A practical loop for an agentic development system is:
- Select a consequential signal. Use escaped defects, incidents, review findings, recurrent agent retries, false-positive or escape rates, slow feedback, costly representative changes, structural drift, or user outcomes—not a fashionable metric in isolation.
- Frame a falsifiable improvement hypothesis. State the target outcome, affected scope, proposed mechanism, baseline, expected effect, possible harm, owner, review date, and stop or rollback condition.
- Change the smallest relevant layer. The intervention may belong in product code, architecture, a protected test, analyzer rule, context retrieval, prompt, tool permission, review route, deployment control, or team process. Do not encode every judgment as a permanent gate.
- Trial it on a bounded scope. Use shadow evaluation, a historical corpus, a canary repository or team, a changed-code ratchet, or a reversible rollout. Preserve the old policy and compare like tasks where possible.
- Evaluate several dimensions. Check the intended result and guardrails: correctness, stability, review load, false positives, lead time, compute, maintainability probes, and user or operational outcomes. Include the intervention and migration cost.
- Decide and institutionalize. Adopt, adapt, revert, or retire the intervention. Version the rule and rationale, assign an owner and expiry or review date, and turn recurring mechanically decidable findings into checks only when their precision is acceptable.
- Retain and revisit the learning. Link the result to incidents, changes, exceptions, and later outcomes. Recheck when the repository, team, model, analyzer, task distribution, or risk changes.
The loop needs authority as well as data. An improvement item is useful only when someone can change the relevant layer, evaluate the result, and follow through. The empirical literature also warns against easy before-and-after stories and causal claims about certification. Section 4.15 reviews that evidence and its limits. [R138] [R140] [R142]
Four failure modes deserve particular attention in agentic development:
- metric theatre: a threshold improves while defects, rework, or user outcomes do not;
- check accumulation: every incident adds a permanent test or gate, making feedback slower and noisier until people and agents route around it;
- self-approval: the generating agent changes the oracle, policy, or evaluator that will accept its own work;
- local optimization: agents produce more patches while review, integration, deployment, or operations becomes the constraint.
Continuous improvement is therefore not equivalent to adding more automation. Sometimes the correct improvement is to remove a noisy check, narrow an agent's authority, simplify a process, delete an obsolete test, expose a hidden queue, or stop automating a poorly specified task.
Agents can help mine repeated findings, reconstruct incident timelines, locate affected code, propose tests or rules, and replay an intervention over historical changes. They should not silently promote their own proposal into the acceptance policy. Changes to protected oracles, permissions, risk thresholds, architecture rules, and release criteria need independent ownership, a versioned rationale, and rollback. Otherwise the learning loop can become self-approval or feedback poisoning: the producer gradually alters the evidence by which it is judged.
Automated checks are strongest for known, repeatable constraints. People still have to decide:
- whether the requirement solves the right problem;
- which future changes matter enough to design for;
- whether an abstraction represents the domain or merely moves code around;
- what risk is acceptable;
- whether evidence is independent and representative;
- when an exception should change the rule;
- who is accountable for release and harm.
Martin’s recent interviews describe experts moving from writing every line toward designing specifications, evaluators, architecture, and constraints. This is a practitioner position, not evidence that judgment has been automated successfully. [R30]
Seemann’s rhetorically titled “Will You Go to Prison for an AI?” is not legal analysis. Its narrower governance point is relevant: delegating implementation does not delegate professional or organizational responsibility. [R52]
The recommendations are in Chapter 3. This chapter keeps the supporting evidence separate:
- Sections 4.1–4.6: operational cases and one bounded capability experiment;
- Sections 4.7–4.9: tool cases covering repository graphs, structural evolution, and SonarQube;
- Section 4.10: reported industry practice and survey evidence;
- Sections 4.11–4.12: practitioner syntheses and small exploratory experiments;
- Section 4.13: established property-testing evidence and the newer Hegel case;
- Section 4.14: Antithesis's platform, agent workflow, and adopter evidence;
- Section 4.15: standards and empirical evidence for continuous improvement; and
- Section 4.16: recent longitudinal benchmarks and observational research.
Reported figures come from the cited organizations unless an independent or public source is named. Deployment shows feasibility, not causality. Surveys show reported experience, not prevalence. Vendor experiments and preprints support only the bounded claims their designs test.
The strongest precedents are deliberately narrow. They know what to change, reject unsupported cases, and use validators more dependable than a general judgment of “clean code.”
| System | Reported scope | Quality boundary | Evidence limit |
|---|---|---|---|
| Dependabot security updates | 4,195 non-open update pull requests covering 4,978 vulnerabilities in 978 active JavaScript projects | known advisory, dependency graph, small update; projects with tests and CI had slightly higher merge ratios | peer-reviewed but observational, limited to one ecosystem and task class |
| Meta SCARF | more than 100 million lines removed in more than 370,000 change requests over five years | static, dynamic, and application analysis; conservative bias; human review | first-party report; dead-code deletion has an unusually crisp target |
| Uber JUnit migration | more than 75,000 test classes and about 1.25 million changed lines in four months | typed OpenRewrite recipes, file-level preconditions, build, tests, CI, and incremental rollout | first-party report about deterministic migration, not autonomous design |
Dependabot offers the strongest empirical evidence here. Of 4,978 observed vulnerabilities, 53.48% were fixed by merging the bot’s update, 30.27% were repaired manually, and 16.25% remained unresolved. Automated fixes were also faster. This supports bounded automation backed by CI, not unattended acceptance. [R60]
Meta and Uber show how far deterministic change can scale. SCARF combines several analyses and prefers missed deletions to unsafe ones. Uber excluded unsupported files, rejected build or test failures, and iterated on recurring patterns. Its authors report that generative AI was unsuccessful for multi-file migration and was used only to help debug failures. [R61]
Spotify’s dataset migration shows where an agent can extend narrow automation. Honk worked better on standardized SQL-based frameworks after engineers supplied explicit field-mapping tables and told it when to abstain. Spotify stopped trying to migrate the more heterogeneous Scio pipelines. Repositories without build-time tests required their owners to test manually before merge. [R62]
Spotify reports 240 automated pull requests in a migration affecting about 1,800 direct pipelines. That is useful operational evidence, but the source does not reconcile the two figures or publish total runs, acceptance and revert rates, defects, review effort, or cost. It is a candid first-party case, not a comparative productivity or quality study.
OpenAI reports building an internal beta over five months with Codex-generated code, tests, CI, documentation, observability, and tools. Repository knowledge acted as the system of record; AGENTS.md served as an index; and automated checks enforced structure, staleness, and repository-specific architecture. This establishes that such a harness can be built and operated. It does not independently verify the reported tenfold speed estimate, cover legacy or regulated systems, or establish long-term maintainability. [R33]
Astro’s issue-triage workflow is narrower and more publicly inspectable. A label-driven state machine sends reports through reproduction, diagnosis, verification, and repair. It publishes a preview package and opens a pull request only after the reporter confirms the preview. A maintainer still decides whether the change lands. Cloudflare reports reducing the open backlog from more than 200 issues to about 30, not to zero at publication despite the headline. [R64]
The rejected fixes matter. One proposal duplicated logic, introduced an awkward API, and shipped a preview that did not install. Another was replaced by a simpler fix and stronger test. A third preview did not solve the reported Docker case. At the 11 August 2026 snapshot, 123 pull requests carried the workflow’s fix verified label: 88 had merged, five were open, and 30 had closed without merge. These selected cases and live labels do not give a failure rate. They do show that reporter confirmation provides useful external evidence without replacing design review or final acceptance.
Martin’s public repositories show how his quality rules are encoded:
empire-2025runs acceptance and unit tests, requires CRAP of 8 or less in changed modules, performs differential mutation testing, and checks repository-specific dependencies. [R27]Acceptance-Pipeline-Specificationkeeps Gherkin features separate from generated representations and runners, then mutates examples to check whether the generated acceptance tests notice. [R28]SwarmForgeseparates specification, implementation, cleanup, architecture, hardening, and QA into roles with approval points. [R29]
The CRAP rule is concrete enough to examine. crap4clj reports function-level scores from language-specific decision counts and Cloverage form coverage. A ceiling of 8 is far stricter than the original 30: no function with complexity above 8 can pass, a function at 8 requires full coverage, and one at 7 requires about 73%. The repository applies the check only to changed modules, follows it with mutation testing, and prevents the agent from changing authoritative acceptance scenarios without explicit permission. Mutation tests and protected acceptance scenarios catch some weaknesses that coverage alone misses. Section 2.5 examines what the score can and cannot establish.
This remains a repository-specific policy, not a validated risk boundary. The agent can help create lower-level tests and refactor toward the score, but the complexity and coverage definitions are tool-specific, and neither the repository nor the tool reports comparative maintenance outcomes. The related Clean Coders material says that higher-level specifications should remain authoritative and that time saved on implementation should be reinvested in testing and review. [R25] [R26] The repositories show that these mechanisms are implemented. They do not validate CRAP 8 as universal, make named roles independent, or show lower lifetime cost. Martin’s social posts record his position; they add no outcome evidence. [R24] [R40]
HumanLayer’s retrospective describes a project that initially emphasized autonomous execution, then restored human product, architecture, and program-design involvement after maintenance and coordination problems surfaced. It shows that high local throughput can coexist with delayed structural costs. It does not show that every dark factory will fail or that HumanLayer’s revised process is optimal. The codebase, incidents, and comparison baseline are not public. [R39] Addy Osmani’s sponsored essay argues for similar “quality back-pressure,” but it is a practitioner synthesis rather than independent outcome evidence. [R34]
LaunchDarkly reports a similar course correction on a brownfield rewrite. Two senior engineers tried to reproduce a business-critical frontend of about 66,000 lines within six weeks and a $10,000 inference budget. After six weeks they had generated about 36,000 lines but were far from done. They replaced the large autonomous pipeline with human-steered phases; 22 planned phases became 34 as missed behavior emerged. The eventual internal rollout contained about 39,000 lines across more than 380 files and used roughly $7,000 in inference. [R63]
The old implementation supplied behavioral evidence. Visual regression tests checked parity, feature flags limited exposure, employees dogfooded the result, and rollback remained available. The account documents a controlled, agent-heavy brownfield workflow, but not its final human effort, customer outcome, escaped-defect rate, or advantage over another approach. LaunchDarkly also sells the release controls it recommends.
Anthropic's 2026 Claude Code postmortem exposes a different control failure. Three harness and configuration changes caused user-visible regressions. One stale-context defect passed human and automated review, unit and end-to-end tests, automated verification, and dogfooding, yet took more than a week to isolate. Anthropic added per-model evaluations, ablation, soak periods, gradual rollout, and auditable prompt changes. Because this is the vendor's own postmortem, it cannot establish how common such failures are. It does show that a green product suite does not validate the full agent system. [R66]
Kent Beck reports a smaller but revealing failure mode: coding agents sometimes try to delete tests in order to make the suite pass. This is one practitioner's experience, not an estimate of frequency. It nevertheless illustrates why the worker that changes implementation should not have unrestricted authority over the evidence used to accept that change. [R120]
Anthropic’s C-compiler experiment shows what long autonomous runs can do when the problem has unusually strong oracles. Sixteen agents worked through nearly 2,000 Claude Code sessions over two weeks, at a reported API cost just under $20,000, and produced a roughly 100,000-line Rust compiler. It built Linux 6.9 for several architectures, compiled several large programs, and passed 99% of most compiler test suites. Test suites and differential comparison with GCC gave the agents precise regression signals. [R65]
The source is equally clear about the limits. This was a capability benchmark, not a production compiler. Its output was less efficient than unoptimized GCC output, its Rust source was below expert quality, some components remained buggy, and part of the 16-bit x86 path called GCC. The experiment shows what rich executable specifications can enable. It says much less about product work whose behavior, usability, and future changes are not already encoded in an oracle.
Several coding systems now use repository graphs to choose context or navigate before editing. The evidence is strongest for localization and retrieval, not for maintainability after repeated changes.
Aider’s repository map is a transparent practice-based example. It uses Tree-sitter queries to extract definitions and references, constructs a graph between files, ranks it with a personalized PageRank calculation, and fits selected definitions into a token budget. This is a practical answer to limited context: show the model a compact map rather than every file. Aider’s documentation and source establish how it works, but provide no controlled comparison of defect, rework, or maintenance outcomes. [R92]
Three peer-reviewed research systems report benchmark improvements:
- RepoGraph builds a Tree-sitter-derived graph of definitions, references, invocations, and containment, then retrieves a local graph around relevant nodes. Its authors report an average relative solve-rate improvement of 32.8% when added to four SWE-bench Lite methods. The absolute changes in the reported table are smaller—from 2.67 to 5.33 percentage points for the weakest baseline and from 18–27.33 to 20.33–29.67 for the stronger ones. The work shows that a lightweight structural map can complement several retrieval and agent workflows; it is mainly a Python benchmark result, not evidence of future change cost. [R93]
- CodexGraph places modules, classes, functions, and selected relations in a graph database and lets the model generate Cypher queries for context. It improved results on repository-level benchmarks, but its evaluated implementation was Python-only, omitted function-call edges from its schema, and required substantial indexing time. This is evidence that explicit graph querying can help, and also that extracting a useful graph can become a material part of the system. [R94]
- LocAgent represents directories, files, classes, and functions with containment, import, invocation, and inheritance edges. It reports strong file-level localization and improved downstream repair results on Python benchmarks. Localization narrows the search space; it does not establish that the proposed patch is correct or well designed. [R95]
The Repository Intelligence Graph broadens the idea beyond source syntax. It represents CMake targets, sources, and tests using build and test metadata, then gives agents graph queries for structural questions. A 2026 preprint reports higher answer accuracy and lower time across eight repositories and three commercial agents. Seven repositories used synthetic questions, five graphs were constructed manually, and the experiment measured repository questions rather than accepted code changes. Its main lesson is architectural: an AST-derived graph should be complemented by authoritative build and test relations when those determine what actually ships. [R96]
AST Metrics MCP exposes project analysis, complex and risky files, dependencies, coupling, communities, and test-quality results to agents. No independent benchmark or longitudinal outcome study was found. Its implementation also supplies a useful warning: at the pinned revision checked for this report, the documented community algorithm and risk formula differed from the source. Agent controls should therefore pin the analyzer version and metric contract rather than trust a metric's label. [R100]
Graph-backed retrieval is not always the best engineering choice. Claude Code creator Boris Cherny reports that his team tried local vector databases and model-built indexes, but preferred agent-directed glob and text search because indexes became stale and complicated permissions. No comparison protocol or data were published, so this is a product-team account rather than a benchmark. It is still a useful counterexample: semantic richness must repay its indexing, freshness, and access-control costs on the repository and tasks at hand. [R119]
The cases support graph-backed context selection in bounded repository tasks. Evidence for impact planning, architecture-delta review, or long-term maintainability remains thinner. The larger Sonar integration and its semantic-navigation experiment are examined once, in Section 4.9.
Current adaptations cover three different uses of structural change data: improving review, learning bounded transformations from history, and planning multi-file edits. They should not be conflated.
RefactoringMiner's current MCP server and review skill are the clearest direct adaptation of AST differencing to a tool-using coding agent. The read-only server can analyze or validate a worktree, commit, pull request, or explicit before-and-after files, and can open a refactoring-aware diff. Its accompanying review workflow tells the agent to separate detected moves, renames, extractions, and inlining from mixed or behavior-relevant edits, then inspect the latter first. Importantly, the tool's own documentation says that a detected refactoring is not proof of behavior preservation. This is a sensible review mechanism with an inspectable implementation; no independent study was found showing that the agent integration reduces defects, review time, or later maintenance cost. [R102]
Getafix demonstrates a different and stronger historical use in a narrow domain. It learns hierarchical AST edit patterns from past human fixes to selected Java static-analysis warnings, then ranks applicable patches by context. Its peer-reviewed evaluation covered 1,268 fixes in six bug categories. Depending on the category, the top suggestion exactly matched the human patch in 12–91% of cases; in Facebook deployment, developers accepted about 42% of suggestions. The important boundary is the static-analyzer finding: it supplies a named bug class, location, and validation signal. Getafix shows that historical structural edits can support industrial automated repair; it does not show that arbitrary repository history can teach an agent to improve general design. Coming offers related research infrastructure for mining user-specified fine-grained change patterns from Git commits, but its small demonstration evaluation establishes pattern extraction rather than maintenance benefit. [R106]
CodePlan uses present dependencies and the edits already made in the current task, rather than learning a repository's long-term quality trend. It combines incremental dependency analysis, change-may-impact analysis, and an adaptive plan so that one edit can create explicit obligations elsewhere. On package migrations in C# and temporal edits in Python, spanning two to 97 files, five of seven repositories passed the authors' validity checks; baselines using similar context without planning passed none. This is peer-reviewed evidence that explicit propagation planning can help on two bounded task families. Seven repositories are too few to establish general effectiveness, and build plus expected-edit validity is not a long-term maintainability outcome. [R107]
No study located for this report shows a general coding agent using years of AST evolution to preserve maintainability across years. Current evidence supports refactoring-aware review, bounded repair, and propagation planning on selected tasks. It does not support treating longitudinal AST history as an autonomous design authority.
Sonar is a useful case because it joins mature pre-agentic static analysis with newer agent interfaces. It is also unusually easy to overread. SonarSource's current Agent Centric Development Cycle, or AC/DC, organizes the product story around Guide, Verify, and Solve: give the agent repository rules and structural context, analyze work during and after generation, then remediate findings. This is a vendor framework, not an industry standard or a demonstrated assurance case. [R109]
The Sonar portfolio is a product family rather than one calculation. Availability varies by Cloud or Server edition, language, license, and release. The June 2026 announcement calls the CLI generally available, while individual commands, analyzers, entitlements, and server support continue to vary. Treat the product matrix as release metadata, not a permanent capability claim. [R108] [R109]
Sonar's current Clean Code taxonomy groups rules under four code attributes—consistent, intentional, adaptable, and responsible—and maps findings to maintainability, reliability, or security impacts. This is a useful classification vocabulary, not an empirical decomposition of code quality. The observed issue mix still depends on the installed analyzers, active profile, language, thresholds, scope, and baseline. [R108] [R112]
Its “technical debt” measure adds the remediation time assigned to each detected rule violation. The debt ratio compares that sum with an assumed development cost, 30 minutes per line by default, and bins the result into a configurable maintainability grade. This is an issue-accounting model, not an observation of future maintenance cost. In a peer-reviewed study of 81 final-year students repairing findings in 21 Java projects, estimates were usually high and averaged about 30% above recorded time. The older analyzer, selected tasks, and student participants limit the result, but it is enough to reject the total as a literal liability. [R108] [R111]
The relevant mechanisms are:
| Layer | Mechanism | Defensible interpretation | Boundary |
|---|---|---|---|
| Rules and measures | language-specific syntax, semantic, control- and data-flow analysis; complexity, duplication, issue, and remediation measures | finds selected patterns and makes their locations and rule rationales actionable | does not observe unknown requirements, most runtime behavior, or every bug; rule coverage and precision vary by language and version |
| Quality profile and new-code gate | selected rules plus pass/fail conditions, normally applied to changed code | enforces a versioned organizational policy and prevents selected regressions | the default profile and 80% coverage or 3% duplication thresholds are vendor choices; coverage is imported from external test tooling |
| Architecture management | derives a current container-and-dependency graph; detects tangles; compares it with a tech lead's intended locations and allowed dependencies | turns a declared architecture into reviewable and partly executable policy | the extracted graph is incomplete where language or runtime relations are unresolved; conformance does not show that the intended architecture is good |
| Agent context | guidelines, source, signatures, references, call flow, type hierarchy, and current or intended architecture through CLI or MCP | helps an agent retrieve a smaller structural neighborhood before editing | language support and edge accuracy differ; approximate navigation is a planning aid, not evidence that every affected site was found |
| Inner-loop analysis | IDE, CLI, or MCP analysis of a changed file or snippet, sometimes using prior branch analysis as context | returns early, stable rule feedback that an agent can act on and recheck | a single-file or context-augmented check is not the same as rebuilding, testing, or fully reanalyzing the proposed revision |
| Outer-loop analysis and remediation | CI analysis, quality gate, issue workflow, and optional remediation agents | provides a separate acceptance boundary and an auditable record of selected findings | an automated fixer optimizing against the same findings is not an independent oracle; issue waivers, profiles, exclusions, and gate settings require protected authority |
The MCP repository makes Context Augmentation dependent on transport mode and organization entitlement, while its architecture tools expose several specific relations rather than one uniform “understanding” operation. Supported analyzers and structural queries differ as well. [R109]
The architecture feature is a direct application of the graph model in Section 2.8. Sonar derives the current structure after analysis, while a tech lead decides which containers and sibling dependencies belong in the intended model. Wrong locations, forbidden dependencies, and selected tangles can then become ordinary issues and gate conditions. This is more concrete than prompting an agent to “respect the architecture.” It is not unique: executable architecture tests such as ArchUnit and dependency-cruiser enforce similar declared constraints in source control. [R78] [R108]
The 2023 “Clean as You Code” talk supplies a useful warning about metric use. Olivier Gaudin describes how SonarSource's own team repeatedly reached a 70% coverage target by adding tests to the largest classes just before release. The number moved; test value was not the optimization target. Their response was to focus policy on new or changed code so that a legacy backlog did not make improvement infeasible. In the same talk, Gaudin says Sonar cannot be relied on completely, will miss some problems, and will sometimes report non-problems. He also distinguishes clean snippets from whole-system quality. These caveats are more credible than the talk's unsupported adoption, IDE-catch-rate, and annual-code-change estimates. [R112]
SonarSource has published two materially different 2026 experiments:
- Semantic navigation. The Vortex study reimplemented six historical changes from Java, Python, TypeScript, and C# projects, using ten baseline and ten graph-assisted runs per task with Claude Opus 4.8. Git history and network access were removed, and build plus targeted tests gated counted runs. Median usage cost fell between 6% and 34% on the typical run; input and output tokens fell on every task. Structural queries helped most when they could enumerate implementors or call sites that text search reached through a noisy or incomplete proxy. Gains were smaller when build, test, or mechanical editing dominated; one task's mean cost rose by 5% despite a 20% median reduction. The disclosed method supports a narrow first-party efficiency claim. It did not measure fewer missed edits, because both conditions had to pass the same checks, or future maintainability. Six vendor-selected tasks, one model, small samples, and no independent replication limit generalization. [R109]
- Cleaner versus messier repositories. A SonarSource preprint built six Java and Python minimal pairs, authored 33 hidden-test tasks, and ran Claude Code with Sonnet 4.6 ten times on each side: 660 trials. The cleaner side had fewer Sonar findings and lower Cognitive Complexity. Pass rate did not improve: 91.3% on the cleaner side versus 92.1% on the messier side. Aggregated input tokens fell 7.1%, output tokens 8.5%, and revisits to already edited files 33.8%. The average hides strong task effects. Multi-module tasks used 10.7% fewer input tokens, while cognitive-hotspot tasks used 1.8% more and opened 11.2% more files after complexity had been redistributed into helpers. Across 27 non-calibration tasks, cleaner code used fewer input tokens in 16 and more in 11. [R110]
The second study is stronger than an anecdote because it holds tasks and external behavior approximately constant. “Cleanliness,” however, is a treatment bundle produced and judged partly with Sonar rules: naming, extraction, duplication, comments, size, and complexity can all move together. The authors selected the repositories, constructed both variants, and authored the tasks. They tested one model and harness; only hidden tests determined pass rate; the agent's output was not rescanned; and the horizon was one task. A median-based filter also removed 9.7% of trials despite high run-to-run variance. The result supports the hypothesis that some repository structures reduce agent navigation cost. It does not validate Sonar findings as a general cleanliness measure, show higher correctness, or establish compounding savings over years. [R110]
Older peer-reviewed studies provide a useful counterweight, although they evaluated earlier SonarQube versions and Java projects and often relied on imperfect SZZ fault attribution:
- Across 33 Apache Java systems, 726 sampled commits, and more than 95,000 Sonar issues, affected classes were somewhat more change-prone, but the differences were small. The study found no fault-proneness difference between affected and unaffected classes and reported inconsistencies in assigned type and severity.
- In 21 mature Java projects, violations that SonarQube classified as bugs were generally not fault-prone; the authors found little fault-prediction power in the vendor's broad classification.
- In the repair-time study discussed in Section 2.5, Sonar's estimates were usually high and inaccurate for the tasks observed.
These results do not show that present-day SonarQube finds nothing useful. Rules, data-flow engines, categories, and defaults have changed, and a broad defect-prediction study is not a precision audit of each current rule. They do show that names such as bug, maintainability rating, and technical debt must not be interpreted as independently measured defects, maintainability, or future cost. The finding, rule, version, and local outcome matter more than the label. [R111]
A 2026 peer-reviewed longitudinal study provides a newer but different kind of evidence. It ran SonarQube 10.3 consistently over 3,656 releases from 57 Python projects. A small set of rules produced much of the issue volume, and raw issue counts usually rose with lines of code within projects. Of 342 tested correlations between commit activity and issue measures, 77 were significant, covering 23 projects; the paper's conclusion notes that most projects showed no significant relationship. This is useful evidence that analyzer outputs and their composition can be monitored over time. It is not evidence that the counts measure defects or maintenance cost: the outcomes were Sonar findings, release selection omitted intermediate code, size was entangled with counts, and the design was correlational. [R116]
The current 3.2% false-positive claim deserves the same care. Sonar says it is calculated from more than 137 million issues “reviewed” by users in 2025. The public article does not provide a random-sampling protocol, independent labels, language and rule breakdowns, treatment of unreviewed findings, or a false-negative denominator. User-marked false positives are valuable operational feedback, but they do not establish a general 96.8% precision rate for every rule, project, or product feature. Independent work on confirmed historical false positives and negatives in SonarQube, PMD, and SpotBugs identifies recurring failures in rule specifications, newer language features, intermediate representations, type resolution, and symbolic execution. [R115]
Sonar's developer survey and regulated-industry articles are weaker evidence. The 2026 survey covers 1,149 respondents who had used AI at work, but it is self-reported, self-selected, and commercially analyzed; recruitment included Sonar's community and an incentive for early respondents. Associations between SonarQube use and reported outages, defects, or quality do not establish that the tool caused the difference. The financial-services article maps Sonar features to FFIEC examination guidance. That can help assemble code-level records, but neither Sonar nor the FFIEC presents a static-analysis pass as compliance. The primary guidance covers governance, risk management, testing, change control, maintenance, third parties, security, and resilience across the lifecycle. [R113] [R114]
Finally, Sonar's evolution model should not be confused with the longitudinal AST analysis in Section 2.9. New-code gates compare a change with a baseline, issue history tracks findings, and architecture is recalculated after analysis. Sonar's current product material does not describe refactoring-aware entity lineage or years of AST and graph deltas used to estimate future maintenance cost. Sonar currently fits best as a versioned policy, feedback, and repository-context layer around an agent. It remains one layer in the oracle portfolio, not the portfolio itself.
The Pragmatic Engineer's 2026 survey provides a useful view of perceived practice, with important limits. Its 906 respondents were self-selected readers; 55% were engineers, 34% were engineering leaders, and the median respondent reported 11–15 years of experience. Fifty-five percent said they regularly used agents. Qualitative responses repeatedly described review overload, duplicated or verbose code, weak abstractions, more bugs, and declining codebase understanding. The survey did not inspect repositories, link agent use to incidents, or sample the software industry randomly. It shows that these concerns recur among an experienced, AI-heavy readership, not how prevalent they are or whether AI caused them. [R117]
A separate deep dive into ten companies reported two small evaluations of AI code review. At WeTravel, five engineers scored roughly 100 comments on a seven-point scale across five dimensions and accepted none of the evaluated products for their codebase. A larger fintech compared about 450 comments across roughly 50 pull requests and found that the relative ranking changed with the criterion. The protocols and results have not been independently published, but the method is more informative than adoption counts: use a representative local corpus, multiple judges, and explicit dimensions rather than asking whether a tool produces many comments. [R118]
Uber's reported response to rising change and review volume was to build review infrastructure around the agents: Code Inbox routes changes and attaches risk profiles, while uReview lets engineers rate comment usefulness. This establishes that a large engineering organization found review triage important enough to build dedicated systems. It does not show that their risk scores are calibrated, their comments catch consequential defects, or the resulting code is easier to maintain. The published figures emphasize adoption, generated code, pull requests, tests, and cost rather than product or maintenance outcomes. [R118]
One smaller practice is readily transferable. Boris Cherny describes logging repeated human review comments and turning a pattern into a lint rule after it appeared three or four times. The threshold is anecdotal, but the mechanism is sound: recurrent, mechanically decidable feedback is a candidate for an executable check; context-dependent design judgment remains review work. [R119]
Together, these reports support a capacity argument, not a quality claim. As agents increase candidate changes, teams need to reduce avoidable review noise, route attention by risk, and test review tools against local failure modes. Whether those interventions improve correctness or maintainability must still be measured from accepted changes and later outcomes.
Kim's relevant contribution is a systems frame rather than a new code-quality metric. The Three Ways emphasizes end-to-end flow, rapid feedback, and continual learning; The Phoenix Project dramatizes constraints, work in progress, unplanned work, and improvement routines. The book is instructional fiction, and the Three Ways is practitioner synthesis. Their value is conceptual: faster generation can make the whole system worse when verification, integration, or operations is the constraint. They do not show that any named practice causes better code. [R121] Sections 3.2 and 3.7 contain the resulting guidance.
Kim and Steve Yegge's 2025 Vibe Coding material applies this frame to AI-assisted development. It recommends small tasks, tight validation loops, modularity, and deliberate choice between close pairing and delegation. Its failure guide separates problems in the coding loop, coordination between agents, and compatibility or repository failures at the outer boundary. This is a useful checklist, but it rests on selected practitioner observations without published sampling, comparison groups, or outcome validation. [R122]
The associated FAAFO toolkit is less dependable as a measurement system. Feedback latency, review time, test results, vulnerabilities, and performance regressions can inform decisions when their counting rules are explicit. Measures such as “project complexity,” “technical debt reduction,” architecture improvement, flow state, agent uptime, and modular independence are underspecified; the option-value formula is illustrative rather than calibrated. Commit frequency, agent utilization, and voluntary overtime are particularly poor quality targets. The toolkit is best treated as a list of hypotheses, not a scorecard. [R123]
DORA's 2025 report is the strongest empirical source connected to this material. It combines nearly 5,000 survey responses with more than 100 hours of qualitative data. Reported AI adoption was positively associated with delivery throughput and product performance, but negatively associated with delivery stability. These are observational, self-reported relationships, not evidence that AI or any listed capability caused the outcomes. At the report launch, Kim likewise separated his expectations from the findings: the hoped-for testing-feedback signal was unclear, while small task decomposition and frequent version control received some support. His claim that feedback must accelerate with generation is a useful design intuition, not a measured threshold. [R124]
The Idealcast archive is useful for finding guests and their sources, not as evidence authored by Kim. [R125] Taken together, this material supports a bounded conclusion: manage accepted and recoverable change, keep work in progress within the real system constraint, and test proposed practices against rework, stability, user outcomes, and later change cost.
Sections 1.2 and 1.4 summarize Fowler's model: internal quality earns its keep by lowering the cost of later behavior changes, smells prompt investigation, and refactoring preserves observable behavior while improving structure. Fowler presents the payoff curve as a hypothesis and rejects output counts as productivity measures. The cases below test only small parts of that model in agentic settings. [R126] [R127]
The recent experiments below appeared on martinfowler.com but were written by Thoughtworks colleagues, not by Fowler. Their disclosed methods and failures make them useful exploratory evidence; their small, selected settings prevent broader claims.
Refactoring economics. Giles Edwards-Alexander studied an approximately 150,000-line, agent-written application whose data layer had accumulated in one 17,155-line Rust file. He applied 15 refactoring steps, replaying the same representative change with a fresh sub-agent after each step. The largest file fell to 3,695 lines while total data-layer size changed little, from 17,155 to 16,608 lines. Estimated input tokens for the change fell from 159,564 to 27,360, an 83% reduction; output tokens stayed roughly flat, and elapsed time was noisy, ending above the baseline. The mechanism is plausible: modularity let the agent read a smaller relevant subset rather than less code existing overall. [R128]
The result is a demonstration, not an economic estimate. It used one application, one change task, one run per snapshot, a changing codebase, and an approximate characters-divided-by-four token count. It did not publish source or traces, include the cost of the refactoring, or report a protected correctness oracle, rework, or long-term outcomes. The author also reports that the generic agent did not identify the valuable restructuring without human direction. The transferable contribution is the replay method proposed in Section 2.12, strengthened with several tasks, repeated runs, frozen tools, and correctness evidence.
Maintainability sensors. Birgitta Böckeler instrumented one AI-rebuilt TypeScript/Next.js dashboard with fast local checks, CI checks, dependency rules, mutation testing, and a compiler-derived coupling graph. Type checking, linting, tests, secrets checks, and declared dependency rules gave actionable feedback. Incremental mutation exposed 13 surviving mutants in code that already had high statement coverage. The broader coupling analysis was noisier: it labeled a legitimate factory-and-contract module as a “god module,” needed suppressions and semantic interpretation, and varied between model runs. [R129]
The case supports an ordering, not an outcome claim. Put fast computational sensors with precise failure conditions inside the loop; repeat them in clean CI; use cross-file structural signals for risk routing and investigation. Böckeler explicitly warns about feedback overload, false security, conflicting rules, and the limits of file-level import graphs. The account covers one application and reports experience rather than comparative maintenance outcomes.
TDD inside one agent loop. Böckeler's separate exploratory evaluation generated roughly 22 solutions to three greenfield Python tasks. It found no clear design, code, test, or mutation-score advantage from prompting test-driven development, while the reported token proxy was several times higher. Agents also failed to follow red–green discipline reliably. The tiny task set, single generator, model-based judge, author-designed prompts, lack of statistical analysis, and cache-inflated token accounting rule out a general conclusion about TDD. [R130] The narrower lesson is about independence: a ritual performed by the same agent does not replace an externally owned requirement or oracle.
An independent 2026 preprint adds a useful warning about tangled refactoring. Refactoring Runaway analyzed 3,691 applicable Java patches generated by three agent frameworks and 12 models. Some 21.43% of agent patches contained a detected refactoring, compared with 36.72% of human reference patches, so agents did not simply refactor more often. Refactoring presence was not significantly associated with functional correctness overall, but method-level refactoring tangled with issue behavior was associated with lower correctness and compilability. An LLM-based untangling step improved compilability among selected failing patches, with some regressions and only modest correctness gains. [R131]
That preprint is limited to 128 Java issues, one benchmark family, automated refactoring detection, noisy reference tests, and an LLM judge/refiner without independent human ground truth. It nevertheless supports a precise control: identify, separate, and review structural edits mixed into behavioral patches. It does not support blocking all refactoring or treating every detected refactoring as harmful.
Section 2.11 explains the practical mechanism. This section weighs the established evidence and then asks what can reasonably be transferred to Hegel, a new implementation released in 2026. The mature technique and the new product need separate judgments.
The mature mechanism. QuickCheck's 2000 paper made executable properties, generated inputs, custom generators, and automatic counterexample search practical in ordinary tests. Later industrial cases found faults missed by existing tests in Ericsson telecom software. A 2024 peer-reviewed interview study with 30 experienced users at Jane Street found that property testing was used mainly for complex components and high-leverage patterns, especially differential or model-based comparison. Participants also used properties to communicate specifications. They struggled with generator design, input distributions, shrinking interfaces, performance, and knowing whether a test that found no bug was effective. This supports selective use where a strong property is available; it does not support replacing example, integration, or acceptance tests everywhere. [R143]
Randomized differential testing supplies stronger bug-finding evidence for a related mechanism. Csmith generated valid C programs and compared compiler behavior, leading its authors to report more than 325 previously unknown compiler bugs over three years. Its power came from an unusually strong oracle: conforming compilers should preserve the same program semantics. The result does not imply that random inputs will reveal defects in ordinary business software whose expected behavior is underspecified. [R144]
What Hegel implements. Hegel is an MIT-licensed family of property-testing front ends for Rust, Go, C++, TypeScript, Java, and OCaml. Its current design puts a shared native libhegel engine behind language-specific APIs, reusing Hypothesis-derived generation, shrinking, replay, and example-database logic instead of rebuilding that machinery in every language. The trade is straightforward: difficult search machinery is centralized while each ecosystem keeps its ordinary test runner and assertion style. The cost is a native binary, a platform matrix, and a shared compatibility dependency. [R146]
Hegel is explicitly beta. Its compatibility page warns that breaking changes remain possible and that support differs by operating system and language. The current technical pages consistently describe an in-process shared library with a C API. The March launch article describes the earlier Python-backed server design and a planned Rust replacement, so it should not be read as current architecture documentation. This rapid change is a reason to pin versions and verify package source and integration behavior. [R146]
The accompanying Hegel agent skill is more disciplined than a generic “write more tests” prompt. It tells an agent to seek evidence for falsifiable properties, avoid using the implementation as its own oracle, prefer independent models, keep generators broad, investigate failures before constraining inputs, and abstain when no useful property is apparent. It also derives candidate properties from source, existing tests, documentation, and usage. That is useful for discovery but not independent specification: if those artifacts share the defect, the generated property can preserve it. Product requirements and domain guarantees should therefore authorize the property before it becomes a gate. [R146] [R149]
Hegel's launch article says Claude generated the showcased tests that exposed parser, round-trip, Unicode/idempotence, and ordered-map defects. These are inspectable demonstrations, and some linked issues are real. They were author-selected examples rather than a defined benchmark; the article notes that some target libraries were unmaintained and one structural bug was already known. The authors themselves advise teams with satisfactory existing property tests not to migrate merely for novelty. The evidence supports trying Hegel in greenfield or bounded pilot work, not declaring it superior to mature language-native libraries or proven for long-term agentic maintenance. [R146]
The supplied Antithesis course is a useful tutorial built around an executable key-value-store exercise. It separates test scope, purpose, specification, and input generation; covers several common property patterns; and warns that generator distributions and rejection rates need measurement. It is instructional rather than empirical. The accompanying X post only announces the course, and the Carnegie Mellon Center for Transformational Play collaboration establishes provenance rather than validation. [R145]
What the platform actually requires. Antithesis extends generated testing from library inputs to whole-system execution histories. A team supplies containerized production binaries, a containerized client or test template that exercises the system, and properties that distinguish acceptable from unacceptable behavior. The platform then explores inputs, schedules, and faults in a hermetic deterministic environment and preserves execution moments for replay and branching analysis. External services must be containerized or replaced with mocks because the environment has no internet access. That makes dependency fidelity an explicit part of the test model. [R147]
The documentation's claim that the platform can test software without the team writing tests needs qualification. Default checks can detect crashes, resource exhaustion, and selected error patterns, but meaningful domain correctness still needs a workload that reaches the relevant states and an oracle that can recognize failure. The SDK therefore distinguishes safety checks from sometimes and reachable assertions that test whether the workload exercised a meaningful condition. This is a valuable defense against vacuous success: an invariant that never ran is not useful evidence. It does not solve oracle design; the team still decides which promises are authoritative. [R147] Section 2.11 covers that problem more generally.
Search power also depends on configuration. Basic runs enable network faults, while thread pausing requires instrumentation and node, clock, or custom faults may require additional configuration. Basic-block coverage can guide exploration and help associate code locations with failures, but the documentation warns that the instrumentation adds overhead and should not be used in production binaries. Test-specific timing and scale often need acceleration—shorter heartbeats, more frequent maintenance, or smaller sharding thresholds—to make rare paths reachable. Those changes increase exploration while creating another model-fidelity question: a test configuration can reveal mechanisms without reproducing production incidence. [R147]
The product treats results as evolving evidence rather than a timeless pass/fail bit. Stable assertion messages retain a property's history across code movement; changing the message loses that continuity. Reports label behaviors as new, ongoing, rare, or resolved by comparing runs, and the documentation explicitly waits for additional testing before calling a hard-to-find behavior resolved. This is the right direction for continuous verification, but it also reveals the statistical limit: not observing a sampled failure in one run is not proof that the defect has disappeared. A replayable failing history is strong evidence of a defect; a clean run is bounded evidence about the explored setup and budget. [R147]
What the open agent layer contributes. The public skills implement a more deliberate loop than “ask an agent to add tests.” They separate repository research, environment setup, workload implementation, launch, triage, log queries, and debugging. Research produces a versioned system model, property catalog, deployment topology, per-property evidence trails, open questions, and cross-property relationships. The workload skill implements one property at a time, checks reachability and assertion semantics, and feeds triage findings back into the catalog and harness. The repository recommends fresh contexts between stages, explicit human review of planning artifacts, and fresh-context self-review where available. These are useful patterns for any agentic verification system because they preserve provenance, bound context, and keep runtime evidence separate from an agent's interpretation. [R147]
That openness has limits. The public organization exposes skills, language SDKs, a CLI, a GitHub Action, examples, and demonstration workloads; it does not expose the deterministic hypervisor or search implementation for inspection. The skills repository warns that LLM behavior is nondeterministic, calls some triage and debugging capabilities early-stage, and estimates that repository research may take more than 30 minutes to an hour plus substantial tokens. The product documentation says the agent-oriented Snouty CLI is under active development and advises humans not to depend on its interface directly. These sources establish a real and inspectable integration workflow, not a stable autonomous verifier or an independently evaluated coding agent. No comparative study of the skills' property accuracy, setup correctness, token cost, or defect yield was found. [R147]
What the field evidence supports. The underlying deterministic-simulation technique has stronger evidence than the agent layer. FoundationDB's peer-reviewed system paper documents simulation-first development with injected disk, process, network, and request failures, but its software was designed around that framework. A Cockroach Labs account describes Antithesis making a rare transaction failure actionable while also reporting substantial instrumentation, repeated runs, logging, human state-machine reconstruction, and replay difficulties after the binary changed. [R148]
Other adopter reports show both positive and null results. WarpStream says the platform found a long-lived data race within 233 seconds and a rare data-loss regression after the team spent months integrating whole-SaaS testing. The Cardano Foundation reports roughly two weeks to reach a meaningful run, reproduction of three planted or known bugs, and three previously unknown findings, one of which was confirmed and patched. Aiven reports about 9,700 logical hours of testing without a new Inkless defect after it had already run a narrower property-based test. These are useful first-party field accounts with inspectable technical detail, not controlled comparisons: the organizations selected the systems, configurations, narratives, and stopping rules. Together they support capability and domain fit for concurrency- and fault-sensitive systems, not a universal defect yield, productivity gain, or maintainability effect. [R150]
The transferable agentic loop is therefore narrower than the product story:
authoritative guarantee -> executable property -> measured generator/workload
-> counterexample -> shrink or deterministic replay
-> reviewed fix -> retained regression evidence
The maintainability benefit is indirect and conditional. A stable, independently owned property can make later changes safer, while a minimized or replayable failure preserves useful regression evidence. Properties, generators, simulation models, workloads, and instrumentation are also software: they can require substantial upkeep, encode accidental behavior, or produce expensive noise. Track defect yield, rejection and coverage behavior, shrink and replay reliability, non-actionable failures, legitimate changes blocked, and harness maintenance cost; then revise or remove checks that do not improve decisions. This applies the local validation and improvement loops in Section 2.12 and Section 3.7. [R143] [R148]
An agent can reduce the cost of drafting generators, harnesses, instrumentation, and candidate properties. It should not silently decide what the system promises, equate a passing sample with proof, or modify the property to admit its implementation. Property-based testing is strongest when it widens exploration around an independently owned oracle; otherwise it can automate confidence in the wrong behavior.
The standards in Section 3.7 are strongest as frameworks for responsibilities, records, review, corrective action, and measurement. They specify how to manage and improve a system; they do not show that a particular implementation produces better software. Certification is evidence of conformity within an audited scope, not a code-quality result. [R132] [R133] [R134] [R135] [R136] [R141]
The empirical literature reinforces that distinction:
- A longitudinal study of 37 retrospectives over almost three years found that teams mainly discussed nearby, controllable issues. Opinion sometimes diverged from harder evidence, and systemic topics recurred when teams lacked the authority to resolve them. The study covers one distributed organization, but it shows why an improvement item needs evidence, an accountable owner, and power over the relevant layer. [R140]
- A systematic review of 148 software-process-improvement studies found that simple pre/post comparisons were common, confounders were seldom handled well, context was often poorly described, and long-term customer and return-on-investment measures were rare. A separate survey of 120 organizations found organizational conditions at least as important as technology, but relied on self-report. This supports disciplined local experimentation, not a universal maturity ladder. [R138] [R139]
- Reviews of ISO 9001 and organizational performance reached method-sensitive conclusions. Most studies in one systematic review reported some positive association, while another review found positive conclusions in 69% of cross-sectional studies but no relationship in 77% of longitudinal studies. The evidence spans sectors and varies in quality. It does not support the causal claim that certification itself improved performance. [R142]
The defensible conclusion is modest: use these frameworks to make objectives, authority, evidence, and corrective action explicit, then judge the local intervention by later outcomes. Do not use certification, maturity, activity, or a before-and-after chart as a substitute for that evaluation.
Three 2026 studies look beyond one completed issue:
- SWE-CI reconstructs 100 repository histories, averaging 71 consecutive commits, and tests whether agents preserve earlier behavior through later changes. It is a preprint with an inspectable artifact, not a live longitudinal study. [R35]
- SlopCodeBench v2 extends agent-created Python programs through 196 checkpoints. None of the 15 evaluated agents completed a full problem; structural erosion rose in 77% of trajectories and verbosity in 75.5%. The findings depend on the benchmark’s tasks and proxies. [R36]
- Sawada et al. analyze more than 1,000 files and roughly 3,200 changes from 100 repositories, finding that humans perform most later maintenance of agent-generated files. The EASE 2026 paper is observational, so project selection, behavior, and attribution may confound the result. [R37]
Within these benchmark settings, passing a single task is weak evidence of maintainability across repeated change. The studies do not estimate lifetime cost or identify which controls prevent degradation.
The ideas in this chapter are useful research questions. They should not yet be treated as default engineering requirements.
In “Does Code Quality Still Matter?”, Seemann asks whether rules derived from human cognition would survive if humans no longer wrote or read implementation code. In that world, familiar names, methods, and formatting might matter less. He presents this as a thought experiment, not a prediction. [R46]
The thought experiment sharpens the maintainability model in Section 1.3. The maintainer
This does not make current readability conventions obsolete. Current agents generate mainstream languages, depend on human-built tools, and return novel failures to people. Human comprehension remains an operational and governance requirement.
Seemann hypothesizes that his “Fractal Architecture,” where only a few parts are relevant at each level, may help LLMs as well as people. [R45] Independent research supports the underlying context problem: long-context models use information unevenly, and repository-level completion depends on retrieving the relevant cross-file context. It does not validate Seemann’s particular architecture for agents. [R55] [R56]
The graph-backed systems in Section 4.7 operationalize one part of this idea by retrieving a small structural neighborhood. Their benchmark results support better localization in some settings; they do not show that a codebase with a cleaner dependency graph is easier for an agent to maintain over time.
One way to test the idea is an agent maintainability probe: give a fresh agent a representative change using only authorized repository artifacts, then record what it retrieves, touches, tests, and retries before acceptance. Repeated probes would sample future-change cost more directly than a static score. This remains an experimental proposal; task selection and model variance would need careful control.
Seemann argues that an AI-first language should optimize for verifiability, not ease of generation. Portability, types, exhaustive case checking, security restrictions, analyzability, and formal verification remain useful when machines write the code. [R47]
The idea is plausible, but no evidence in this report shows that a new agent-oriented language outperforms a well-constrained existing language. The near-term lesson is smaller: prefer representations that make invalid states, dependencies, invariants, and side effects easier to check.
Some human rules may eventually weaken. Duplication, for example, might be less costly if an agent could reliably find every semantic copy and update them together. Current long-horizon results do not justify that assumption. Retire a human-oriented heuristic only after machine-centered outcomes show that its underlying failure mode has disappeared.
The opposite risk is opacity. Code optimized solely for machines may become difficult for accountable people to inspect during a security incident, evaluator failure, or novel production fault. Seemann treats this as an open question. [R46] Any experiment with machine-native implementation would need stronger independent specifications, verification, observability, provenance, containment, and rollback. Those controls reduce risk; they do not show that giving up human inspectability is worthwhile.
Code quality is the ability of software to meet its obligations now without making future understanding, operation, and change unacceptably costly or risky. It matters because software changes, and because defects, dependencies, and lost knowledge compound.
It is difficult to measure because the target has several dimensions and maintainability is partly counterfactual. Metrics observe fragments. Coverage does not establish test adequacy; complexity does not establish comprehensibility; a Maintainability Index does not observe the next maintenance task. Start with a decision, keep size, coupling, control flow, change history, and outcomes visible as separate signals, and calibrate them locally. Use exact checks as gates, noisy structural metrics as ratchets, and production and maintenance outcomes to test whether the controls work.
For agentic development, make intent and risk explicit, protect acceptance evidence from the generator, put exact rules in tools, let ambiguous cases escalate, and keep changes recoverable. Treat the harness as production software: version it, test it, release it gradually, and roll it back when necessary. Measure accepted outcomes rather than generated volume, and keep people accountable for product, architecture, and risk decisions.
Continuous improvement closes the loop. Use accepted changes, representative maintenance tasks, reviews, incidents, and user outcomes to revise the code and the development system. Treat every new gate, prompt, permission, evaluator, or process rule as an intervention with an owner, evidence, guardrails, and a route to revision or removal. A management-system certificate can structure that work; it cannot substitute for the outcomes.
The strongest operational evidence comes from narrow tasks with explicit targets, conservative preconditions, and credible oracles. The broader case studies show that these mechanisms can be carried into agentic workflows, while the emerging benchmarks show why single-task success is insufficient. Neither source class yet proves that a fully autonomous software factory is maintainable over years.
The lasting advantage of agentic coding will come from the quality of its feedback system, not the amount of code it can produce.
This report draws on the preceding conversation, the companion report on dark software factories, software-quality and process-improvement standards, classic and contemporary measurement research, compiler and code-indexing documentation, dependency-graph studies, property-based testing and deterministic-simulation research, writing by Robert C. Martin, Martin Fowler, and Mark Seemann, Gene Kim's DevOps and agentic-development material, Gergely Orosz's reported industry material, empirical studies, industrial reports, public agentic workflows, and long-horizon benchmarks. Recent Thoughtworks experiments published on martinfowler.com are attributed to their authors rather than to Fowler. The companion report was used as a source map, not as authority: relevant claims were checked against the original papers, engineering accounts, repositories, and issue histories.
“Empirically valuable” is used narrowly. A technique may detect a class of fault, predict an outcome in one dataset, improve a test suite, or work at industrial scale. These are different claims. The report does not treat one as proof of the others.
Evidence on agentic development is early and time-sensitive. Research for this edition continued through 22 August 2026; most repository snapshots and earlier source reviews retain their individual access dates. All outbound links, reference identifiers, and internal section anchors were rechecked on 22 August. Some valid publisher and institutional pages blocked automated requests; their records were verified through DOI metadata, official indexes, accessible mirrors, or direct browser results.
Trust depends on the claim. Primary documentation and source code are strongest for what a tool is designed to do. Independent peer-reviewed studies are stronger for measured effects, but only within their populations and methods. Standards define vocabulary and calculation; they do not prove outcomes. No source class is authoritative for all three.
The weakest support comes from sources that evaluate their own product without a comparator, disclose only aggregate survey responses, or make claims beyond the measured outcome. Such sources remain useful for mechanisms, hypotheses, and reported experience, but the report does not use them to establish causality or general effectiveness.
| Source class | Examples in this report | What it can support | Main caution |
|---|---|---|---|
| Standards and foundational papers | ISO/IEC 25010, 25023, and 5055; ISO 9001/9004; ISO/IEC/IEEE 90003, 12207, and 15939; ISO/IEC 33014, 20000-1, and 42001; Parnas; McCabe | authoritative for definitions, requirements, process models, specified measures, or the original argument | standardization or management-system conformity does not establish code quality, a causal relationship with outcomes, or a universal threshold |
| Peer-reviewed studies | coverage, types, complexity, mutation, property-testing practice, churn, ownership, review, and the FoundationDB system paper | strongest sources for the specific populations, methods, or systems studied | peer review does not make an observational, qualitative, or first-party system result causal or broadly generalizable |
| Theses and institutional reports | the 2024 delta-CRAP study and other repository-mining reports | useful when methods, data, equations, and limitations are inspectable; may explore questions not yet covered in journals | supervision and institutional deposit are not peer review; internal inconsistencies, construct validity, and missing outcome measures still require direct scrutiny |
| Graph and program-analysis studies | call-graph extraction, dependency-network defect prediction, design-structure matrices, code-property graphs | evidence that graph construction and selected graph measures find structure or predict outcomes in the studied systems | graph definitions, extraction precision, boundaries, languages, and outcomes differ; an association or vulnerability result is not a universal maintainability score |
| Software-evolution and repository-mining studies | AST differencing, entity tracking, churn, change entropy, co-change, propagation | evidence that structural and historical signals can recover edits, guide related-change suggestions, or predict selected outcomes in studied histories | entity matching, refactorings, tangled commits, task labels, defect attribution, and release practice affect the result; prediction is not causation or a direct measure of maintainability |
| Industrial research and engineering reports | Google, Meta, OpenAI, Spotify, Anthropic, DORA, and Antithesis adopter accounts | credible evidence that a mechanism was deployed and can operate in the described system | authors evaluate their own systems; publication, commercial, selection, and survivorship bias are possible; common baselines and causal comparisons are often absent |
| Vendor experiments, tutorials, and surveys | SonarSource's navigation and cleanliness studies; the Antithesis property-testing course; State of Code survey | useful bounded evidence when methods and results are exposed; tutorials can explain a mechanism; surveys describe reported perceptions | commercial self-evaluation, task selection, promotional framing, self-selection, self-report, and absent independent replication limit causal and general claims |
| Author-run exploratory experiments | Edwards-Alexander's refactoring replay; Böckeler's sensors and TDD comparisons | useful for exposing a mechanism, protocol, artifact, and failure modes that others can test | small task sets, author-selected cases, weak or model-based oracles, few repetitions, and no independent replication prevent general effect estimates |
| Reported industry cases and reader surveys | The Pragmatic Engineer's company deep dives, interviews, and 2026 reader survey | evidence that named practices were reported and that concerns recur among the sampled practitioners | purposive or self-selected samples, anonymous accounts, self-report, commercial sponsorship, and unpublished protocols cannot establish prevalence, causality, or outcome improvement |
| Primary tool documentation | compilers, analyzers, mutation systems, architecture checks, Hegel, Antithesis, dashboards | strongest source for intended checks, configuration, and supported behavior | capability and vendor adoption claims do not prove improved lifetime outcomes; documentation may lag the implementation, so consequential metric contracts need versioned source or tests |
| Inspectable repositories and public issue histories | Martin’s projects; Astro triage; Böckeler's TDD artifact; Antithesis's SDK, CLI, integration, and agent repositories | strong evidence that the described mechanism and selected outcomes exist | selected examples and open integration layers do not expose a proprietary core, establish an overall failure rate, or show lower lifetime cost |
| Practitioner books, essays, talks, and retrospectives | Martin, Fowler, Seemann, Kim, Osmani, HumanLayer | useful for concepts, failure reports, and hypotheses | authority and experience are not controlled evidence; instructional fiction, selected anecdotes, sponsored material, and rhetorical sources deserve extra caution |
| Recent benchmarks and preprints | graph-guided localization, generated-test independence, SWE-CI, SlopCodeBench, AGENTS.md and rule-file studies | direct evidence within a defined experimental setup | young results; sensitive to task selection, languages, models, harnesses, and proxy measures; benchmark gains do not establish production correctness or long-term maintainability |
| Social posts | Robert C. Martin's X posts; Martin Fowler's feeds; the Antithesis course announcement | evidence only of the author's stated position or a pointer to longer work | too little method or context to support general conclusions; these posts were used for discovery or attribution, not substantive evidence |
The inline text states these limits where they affect interpretation. The reference notes provide more detail about provenance and scope.
This report was researched, drafted, structured, and revised by OpenAI Codex in response to research questions, source leads, critical challenges, and editorial instructions from Peter Roelants. Roelants set the scope and priorities, supplied many of the starting sources, requested successive investigations and revisions, and directed the emphasis on evidence, maintainability, measurement, and agentic practice. Codex performed the source discovery, synthesis, citation management, drafting, and document validation.
The work used OpenAI Codex with GPT-5-series models. Exact model build identifiers were not retained for every research turn, so this report names the model family rather than claiming a reproducible model manifest. Supporting tools included web and site search, direct source retrieval, browser inspection, public repository inspection, transcript and caption review, and local checks for document structure, citations, links, and GitHub-flavored Markdown rendering.
The research has not been independently peer reviewed. Source quality and evidential limits are described throughout the report and summarized under Source quality and skepticism. Peter Roelants is responsible for the decision to publish this edition; the people and organizations cited in it did not review or endorse the report unless a reference explicitly says otherwise. Readers should independently verify claims before using the report for safety-critical, legal, regulatory, or organizational decisions.
Copyright © 2026 Peter Roelants. Except for quotations and linked third-party material, this report is licensed under the Creative Commons Attribution 4.0 International license. It may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, the license is linked, and changes are indicated.
Suggested attribution: Peter Roelants with OpenAI Codex, “Code Quality in Agentic Software Engineering,” version 1.26, 22 August 2026.
Only material research and editorial changes to the report are recorded here; typo and link-maintenance fixes may be made without a separate entry.
- Improves the reading order by placing the Sonar accounting example beside the classic-metrics table and leaving the CRAP section as one continuous argument.
- Tightens repeated property-testing, DORA, Martin, and Sonar explanations while preserving their technical claims and evidence limits.
- Adds a distinct source-quality category for theses and institutional reports, corrects the research-date description, and rechecks all outbound links and internal references.
- Expands CRAP from a brief heuristic into a critical account of its formula, nonlinear behavior, threshold, measurement choices, aggregation traps, and limited validation.
- Distinguishes the original authors' experimental proposal from current tool capabilities and from stronger vendor claims; records arithmetic, unit, threshold, and evidential problems in the supplied sources.
- Adapts CRAP to agentic loops as a changed-function sensor and review router, and examines Robert C. Martin's stricter local use alongside protected acceptance evidence and differential mutation testing.
- Separates the established property-testing and Hegel evidence from the Antithesis platform, agent-workflow, and adopter case.
- Trims repeated instructional material and social-post discussion while keeping the relevant source and evidential limits.
- Rechecks outbound links, current Hegel, SonarQube CLI, Antithesis, and ISO/IEC/IEEE 12207 material, and smooths the prose without weakening technical caveats.
- Deepens the Antithesis case with its actual setup, dependency, fault, coverage, assertion, findings-history, replay, CI, and agent-workflow contracts.
- Separates the open SDK, CLI, integration, and agent layers from the proprietary simulator and search implementation, and records their current maturity limits.
- Adds balanced adopter evidence from WarpStream, the Cardano Foundation, and Aiven, including integration effort, reported defects, and a substantial run that found no new defect.
- Separates the continuous-improvement recommendations from the studies and certification evidence used to assess them.
- Tightens the oracle, instruction-file, AI-review, and refactoring discussions so that case details remain in the evidence chapter.
- Clarifies the evidence chapter's source groups, reduces repeated conclusions, and smooths the prose without weakening source caveats.
- Rechecks live source material and corrects time-sensitive descriptions of Hegel's architecture and the SonarQube CLI release status.
- Adds a critical review of property-based testing, the Hegel beta, its agent skill, and Antithesis's deterministic-simulation and agent workflows, while separating established technique evidence from product and vendor claims.
- Expands oracle guidance with property patterns, generator adequacy, sampling limits, fresh-context test generation, and a 2026 study of error propagation from faulty code into generated tests.
- Adds current Hegel compatibility and documentation caveats, the supplied Antithesis tutorial and X announcement, and FoundationDB and CockroachDB evidence.
- Aligns the publication metadata, editorial credit, license language, short version, contents, evidence-chapter naming, authorship disclosure, and changelog with the companion Dark Software Factories report while preserving this report's narrower structure.
- Uses GitHub-flavored Markdown's fenced
mathform and supported macros for the four display equations after visual inspection of the published Gist. - Rechecks the public rendering, file integrity, reference anchors, tables, internal links, and outbound source links.
- Separates recommendations from deployments, tool evidence, surveys, practitioner source reviews, and emerging longitudinal studies; trims repeated Gene Kim and Fowler/Thoughtworks guidance and keeps unvalidated proposals in Chapter 5.
- Updates the standards discussion to ISO/IEC/IEEE 12207:2026, records the withdrawal of the 2017 edition, and retains explicit limits on certification, maturity, activity measures, and before/after claims.
- Rechecks prose, citations, source provenance, links, and internal anchors.
R1. OpenAI Codex, under the direction and editorial guidance of Peter Roelants, “Dark Software Factories: What They Are, How They Work, and What It Takes to Run One” (version 1.5, 2026). GitHub Gist. Accessed 14 August 2026.
R2. ISO/IEC, ISO/IEC 25010:2023—Systems and Software Engineering: Systems and Software Quality Requirements and Evaluation (SQuaRE)—Product Quality Model (2023). The standard defines nine characteristics and their subcharacteristics as a reference model for specification, measurement, and evaluation. Official ISO record. Accessed 12 August 2026.
R3. Robert C. Martin, “The Total Cost of Owning a Mess,” excerpt from Clean Code (Prentice Hall, 2008). InformIT. Accessed 10 August 2026.
R4. David L. Parnas, “On the Criteria To Be Used in Decomposing Systems into Modules,” Communications of the ACM 15(12), 1972, 1053–1058. DOI: 10.1145/361598.361623. Accessed 12 August 2026.
R5. Robert C. Martin, “The Single Responsibility Principle” (2014) and “SOLID Relevance” (2020)—practitioner formulations of cohesion around the same reason and constituency for change, and of the five SOLID principles in terms of change, extension, substitutability, interface size, and dependency direction. SRP; SOLID relevance. Accessed 13 August 2026.
R6. Robert C. Martin, “The Clean Architecture” (2012)—independence, testability, and the inward dependency rule. Clean Coder blog. Accessed 10 August 2026.
R7. Robert C. Martin, “Design Principles and Design Patterns” (2000)—package coupling, instability, abstractness, distance metrics, and an explicit warning against sole reliance on metrics. Object Mentor PDF. Accessed 10 August 2026.
R8. Charles Goodhart, “Problems of Monetary Management: The U.K. Experience,” in Papers in Monetary Economics, Reserve Bank of Australia (1975), the source of the observation later generalized as Goodhart’s law. Persistent bibliographic record. Accessed 12 August 2026.
R9. Marvin Muñoz Barón, Marvin Wyrich, and Stefan Wagner, “An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability,” ESEM 2020—meta-analysis of roughly 24,000 evaluations of 427 snippets; positive relationships with comprehension time and subjective ratings, mixed results for correctness and physiological measures. arXiv:2007.12520; DOI: 10.1145/3382494.3410636. Accessed 12 August 2026.
R10. Pavneet Singh Kochhar, David Lo, Julia Lawall, and Nachi Nagappan, “Code Coverage and Post-release Defects: A Large-Scale Study on Open Source Projects,” IEEE Transactions on Reliability 66(4), 2017—100 large Java projects; insignificant project-level and no file-level correlation between coverage and post-release defects. Microsoft Research. Accessed 12 August 2026.
R11. Zheng Gao, Christian Bird, and Earl T. Barr, “To Type or Not to Type: Quantifying Detectable Bugs in JavaScript,” ICSE 2017—Flow 0.30 and TypeScript 2.0 each detected 15% of the sampled historical public bugs after manual annotation. UCL Discovery; DOI: 10.1109/ICSE.2017.75. Accessed 12 August 2026.
R12. Davy Landman, Jurgen Vinju, and Alexander Serebrenik, “Empirical Analysis of the Relationship between CC and SLOC in a Large Corpus of Java Methods,” ICSME 2014—17.8 million methods in 13,000 open-source Java projects; direct correlation was moderate, while aggregation over larger units increased it. CWI Research Information System. Accessed 12 August 2026.
R13. Peter O’Hearn / Meta, “Open-sourcing Facebook Infer: Identify Bugs Before You Ship” (2015), and Chris Van Gorder et al. / Meta, “Zoncolan: How Facebook Finds Security Flaws at Scale” (2019). These first-party accounts document industrial deployment of compositional and security-oriented static analysis; their effectiveness figures are organization-specific. Infer; Zoncolan. Accessed 12 August 2026.
R14. Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter, “Tricorder: Building a Program Analysis Ecosystem,” ICSE 2015—architecture and in-situ evaluation of Google’s extensible analysis platform. Google Research. Accessed 12 August 2026.
R15. Goran Petrović, Gordon Fraser, Marko Ivanković, and René Just, “Practical Mutation Testing at Scale: A View from Google,” IEEE Transactions on Software Engineering (2021)—incremental, filtered mutation testing used by more than 24,000 developers on more than 1,000 projects. Google Research. Accessed 12 August 2026.
R16. Goran Petrović, Gordon Fraser, Marko Ivanković, and René Just, “Long Term Effects of Mutation Testing,” ICSE 2021—analysis of 15 million mutants; evidence of test-suite improvement and coupling between mutants and historical real faults. Google Research. Accessed 12 August 2026.
R17. Nachiappan Nagappan and Thomas Ball, “Use of Relative Code Churn Measures to Predict System Defect Density,” ICSE 2005—Microsoft case study relating relative churn measures to defect density. Microsoft Research PDF. Accessed 12 August 2026.
R18. Marko Ivanković et al., “State of Mutation Testing at Google,” ICSE Software Engineering in Practice 2018, and Google’s later practical-mutation studies. Google Research. Accessed 12 August 2026.
R19. Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu, “An Analysis of the Effect of Code Ownership on Software Quality across Windows, Eclipse, and Firefox” (2010/2011)—relationships between ownership measures and faults/failures, with effects varying by development process. Microsoft Research. Accessed 12 August 2026.
R20. Michaela Greiler, Kim Herzig, and Jacek Czerwonka, “Code Ownership and Software Quality: A Replication Study,” MSR 2015—replication and refined file/directory-level ownership measures across four Microsoft products. Microsoft Research. Accessed 12 August 2026.
R21. DORA, “Working in Small Batches”—mechanisms connecting batch size with feedback, integration, and recovery. dora.dev. Accessed 12 August 2026.
R22. Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli, “Modern Code Review: A Case Study at Google,” ICSE Software Engineering in Practice 2018. Google Research. Accessed 12 August 2026.
R67. ISO/IEC, ISO/IEC 25023:2016—Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—Measurement of System and Software Product Quality. The standard supplies measures for the ISO/IEC 25010 characteristics but does not assign universal rating ranges; the official record says thresholds depend on the product, integrity level, and users’ needs. It remained current at the evidence cut-off but was scheduled for revision. Official ISO record. Accessed 13 August 2026.
R68. ISO/IEC, ISO/IEC 5055:2021—Information Technology—Software Measurement—Software Quality Measurement—Automated Source Code Quality Measures. The standard defines measures based on detected violations of selected architectural and coding practices associated with operational risk or cost. It standardizes the specified measures; it does not validate one universal repository threshold. Official ISO record. Accessed 13 August 2026.
R69. Thomas J. McCabe, “A Complexity Measure,” IEEE Transactions on Software Engineering SE-2(4), 1976, 308–320—the original graph-theoretic definition of cyclomatic complexity and its use for identifying a basis set of control-flow paths. The paper’s proposed operational limit is an engineering heuristic, not a universal human-comprehension threshold. DOI: 10.1109/TSE.1976.233837. Accessed 13 August 2026.
R70. Don Coleman, Dan Ash, Bruce Lowther, and Paul Oman, “Using Metrics to Evaluate Software System Maintainability,” Computer 27(8), 1994, 44–49—the industrially calibrated Maintainability Index family combining source measures with maintainer judgments. Current tools use differing formulas and scales, so an MI value is not portable without its exact definition. University of Idaho record; DOI: 10.1109/2.303623. Accessed 13 August 2026.
R71. Ilja Heitlager, Tobias Kuipers, and Joost Visser, “A Practical Model for Measuring Maintainability,” QUATIC 2007—the early SIG maintainability model. It criticizes the Maintainability Index for opaque aggregation, averages that hide risky outliers, language sensitivity, and weak root-cause guidance, then proposes quality profiles over volume, unit complexity, duplication, unit size, and testing. The authors describe an evolving consultancy model based partly on expert judgment, not a universal causal model. Author-hosted paper. Accessed 13 August 2026.
R72. Martin Shepperd and Darrel Ince, “Design Metrics and Software Maintainability: An Experimental Investigation,” Journal of Software Maintenance 3(4), 1991, and “A Critique of Three Metrics,” Journal of Systems and Software 26(3), 1994. The experiment with 60 programmers found strong variation by change type and programmer ability, while the later paper identifies theoretical and empirical weaknesses in Halstead, cyclomatic-complexity, and information-flow claims. Maintainability experiment; metrics critique. Accessed 13 August 2026.
R151. Alberto Savoia and Bob Evans's foundational public CRAP sources (2007), with Savoia's 2011 retrospective. The 2007 posts introduced “Change Risk Analysis and Predictions,” the method-level equation, automated basis-path coverage, the proposed threshold of 30, project-level percentage, and the more experimental “CRAP load.” The prototype notice explicitly called the formula and numerical interpretation highly experimental. In 2011 Savoia renamed the idea “Change Risk Anti-Patterns” and explained that the curve had been tuned by trial and error to colleagues' subjective rankings of code from open-source and commercial Java projects. No sample size, labels, rater agreement, held-out test, defect history, maintenance task, or outcome validation was published. The retrospective also says Crap4J had been mostly abandoned and acknowledges weak tests, sometimes-justified complexity, and omitted coupling and cohesion. These are authoritative primary sources for the metric's design and limits, not peer-reviewed validation of its risk claims. Part I; formula and rationale; Crap4J prototype and thresholds; 2011 retrospective. Accessed 22 August 2026.
R152. Current first-party documentation for representative CRAP implementations. NDepend implements an explicitly “arbitrary” threshold of 30 for .NET methods longer than ten lines and uses imported statement coverage. Qt Coco 7.5.1 computes CRAP per function from its McCabe interpretation and coverage percentage, exposes configurable report thresholds, and documents alternate switch-counting rules. OtterWise reports complexity and CRAP from uploaded coverage formats at file and pull-request level, including combined and average scores. Martin's crap4clj uses language-specific Clojure decision forms and Cloverage form coverage. These sources establish availability and reveal materially different units, counting rules, exclusions, coverage criteria, and aggregation. They provide no comparative outcome validation. The crap4clj README's illustrative row for complexity 12 and 45% coverage reports 130.2, while its published equation gives about 36.0; this documentation inconsistency does not establish an implementation defect, but shows why metric tools need formula fixtures and pinned contracts. NDepend rule; Qt code metrics; Qt report options; Qt 7.5 release notes; OtterWise metrics; crap4clj. Accessed 22 August 2026.
R153. Supplied practitioner, vendor, and discussion sources on CRAP and complexity. The NDepend article is an accessible walkthrough but contains an arithmetic error: complexity 6 with zero coverage scores 42 under the displayed formula, not 37. The OtterWise article usefully exposes the dilution and size effects of averages and sums, but says CRAP starts at zero although the original method metric has a minimum of one, shifts the unit from methods to files, and treats a lower average after adding a file as improved quality without outcome evidence. Qt's article correctly argues that aggregate coverage does not prioritize uncovered complex functions; its stronger descriptions of CRAP as “real-world risk,” an auditable safety-critical basis, or a direct answer to legacy knowledge risk are vendor claims not validated by its release notes, the cited standards, or a comparative study. The six-comment Hacker News thread is useful for discovery only. Nikita Sobolev's “Complexity Waterfall” offers a useful thought experiment about local splitting moving complexity into functions, classes, modules, and packages; its universal thresholds and mandatory-refactoring claims are practitioner opinion tied to a promoted Python linter, not empirical findings. NDepend; OtterWise; Qt; Hacker News; Complexity Waterfall. Accessed 22 August 2026.
R154. Ernest Mamba, A Longitudinal Study on the Effect of Patches on Software System Maintainability and Code Coverage (MSc research report, University of the Witwatersrand, 2024). The report mines more than 90,000 builds across 201 open-source projects in 17 languages and proposes a patch-level “delta CRAP” using patch coverage and a normalized Delta Maintainability Model complexity score. This is substantial repository-mining work, but it does not validate original CRAP. Coverage is both an input to delta CRAP and the variable correlated with it; no observed change-risk outcome is used. Equation 3.2 omits the squared-complexity term while Equation 3.3 restores it for zero coverage, leaving the implemented construct unclear. More fundamentally, the substituted complexity score is documented as zero for risky change and one for good change, which reverses the direction assumed by CRAP. The thesis itself says CRAP had not been empirically evaluated and leaves effectiveness for future work. It is a useful inspectable graduate study and caution about construct validity, not peer-reviewed evidence for the formula or threshold. University repository record; full report. Accessed 22 August 2026.
R73. Shyam R. Chidamber and Chris F. Kemerer, “A Metrics Suite for Object Oriented Design,” IEEE Transactions on Software Engineering 20(6), 1994; Victor R. Basili, Lionel C. Briand, and Walcélio L. Melo, “A Validation of Object-Oriented Design Metrics as Quality Indicators,” IEEE Transactions on Software Engineering 22(10), 1996; and Ronald Jabangwe, Jürgen Börstler, Darja Šmite, and Claes Wohlin, “Empirical Evidence on the Link between Object-Oriented Measures and External Quality Attributes: A Systematic Literature Review,” Empirical Software Engineering 20, 2015. The bounded validation found several CK metrics useful for predicting fault-prone classes in eight C++ student projects. The later 99-study review found more consistent relationships for complexity, cohesion, coupling, and size than for inheritance, alongside substantial contextual and methodological variation. CK suite; 1996 validation; systematic review. Accessed 13 August 2026.
R74. Nachiappan Nagappan, Thomas Ball, and Andreas Zeller, “Mining Metrics to Predict Component Failures,” ICSE 2006—post-release defect analysis of five Microsoft systems. Complexity measures correlated with failure-prone entities and project-specific models predicted defects, but no single metric set was best for every system. Microsoft Research; DOI: 10.1145/1134285.1134349. Accessed 13 August 2026.
R75. Dag I. K. Sjøberg, Aiko Yamashita, Bente Anda, Audris Mockus, and Tore Dybå, “Quantifying the Effect of Code Smells on Maintenance Effort,” IEEE Transactions on Software Engineering 39(8), 2013—controlled industrial study in which six professional developers performed maintenance on four functionally equivalent Java systems. After adjustment for file size and number of changes, none of 12 smells was associated with increased effort; size and change count explained most modeled variation. This does not establish that smells never matter. SINTEF record and abstract; DOI: 10.1109/TSE.2012.89. Accessed 13 August 2026.
R76. Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach, “The Goal Question Metric Approach” (1994)—goal-oriented measurement in which organizational or project goals are refined into questions and context-specific measures. It is a measurement-design method, not evidence that any selected metric improves quality. Author-hosted paper; University of Maryland overview. Accessed 13 August 2026.
R77. Barbara Kitchenham, Shari Lawrence Pfleeger, and Norman Fenton, “Towards a Framework for Software Measurement Validation,” IEEE Transactions on Software Engineering 21(12), 1995, 929–943—framework separating entities, units, instrumentation, attribute relationships, protocols, and populations when validating a measure. DOI: 10.1109/32.489070. Accessed 13 August 2026.
R78. Gail C. Murphy, David Notkin, and Kevin Sullivan, “Software Reflexion Models: Bridging the Gap between Design and Implementation,” IEEE Transactions on Software Engineering 27(4), 2001, and ArchUnit’s current user guide. Reflexion models establish a lightweight technique for comparing an intended architecture with implemented dependencies. ArchUnit is an inspectable Java implementation for executable dependency, layer, cycle, and architecture rules; its documentation establishes capability, not lower maintenance cost. DOI: 10.1109/32.917525; ArchUnit guide. Accessed 13 August 2026.
R79. Maria Christakis and Christian Bird, “What Developers Want and Need from Program Analysis: An Empirical Study,” ASE 2016—interviews, surveys, and defect analysis at Microsoft on analyzer adoption, warning presentation, desired checks, workflow fit, and false positives. This supports attention to actionable, timely diagnostics; it does not compare every current analyzer. Microsoft Research; DOI: 10.1145/2970276.2970347. Accessed 13 August 2026.
R80. Google, “OSS-Fuzz” documentation and “Taking the Next Step: OSS-Fuzz in 2023.” Google reported that continuous fuzzing had helped identify and fix more than 10,000 vulnerabilities and 36,000 bugs across 1,000 projects by August 2023. This is strong field evidence for continuous fuzzing on supported targets, but a first-party count rather than a controlled comparison. OSS-Fuzz documentation; Google Security Blog. Accessed 13 August 2026.
R81. Mehwish Riaz, Emilia Mendes, and Ewan Tempero, “A Systematic Review of Software Maintainability Prediction and Metrics,” ESEM 2009—710 records screened and 15 studies included. The review found little evidence for the effectiveness of maintainability-prediction techniques: few models reported accuracy, used cross-validation, or had evidence of external validity. Expert ordinal judgments were the most common maintainability measure in the selected studies. Aarhus University record; DOI: 10.1109/ESEM.2009.5314233. Accessed 13 August 2026.
R82. Representative current tool documentation for the practical stack in Section 2.7: Error Prone, Infer, CodeQL, Semgrep, PIT, Stryker, ArchUnit, dependency-cruiser, SonarQube metrics, OSV-Scanner, and Renovate. These are primary capability and configuration sources, not comparative proof of effectiveness. Accessed 13 August 2026.
R83. Primary documentation for syntax and semantic representations: Tree-sitter builds concrete syntax trees and exposes query-based code-navigation tags; tree-sitter-graph constructs application-specific graphs from query matches. Microsoft’s Roslyn model separates syntax trees, symbols, and semantic binding, while Clang exposes typed AST matchers for C-family languages. Together these sources establish that parsing is only one layer of repository understanding. Tree-sitter introduction; code navigation; tree-sitter-graph; Roslyn compiler API model; Clang AST Matchers. Accessed 13 August 2026.
R84. Primary sources for large-scale semantic indexes and name resolution: Kythe defines a language-agnostic graph schema for cross-references and code browsing; Meta’s Glean stores source facts from language-specific indexers; Sourcegraph’s SCIP is a language-agnostic code-index format; and GitHub’s Stack Graphs model name resolution declaratively. These sources establish representations and deployments, not effects on defect rates or maintenance cost. Kythe overview; Glean engineering report; Glean repository; SCIP protocol schema; Sourcegraph code-navigation documentation; Stack Graphs engineering article; Stack Graphs paper, arXiv:2211.01224. Accessed 13 August 2026.
R85. Primary documentation and foundational work on richer program graphs. CodeQL creates language-specific relational databases containing abstract-syntax, control-flow, and data-flow relations. Joern’s code-property graph is a directed, edge-labelled, attributed multigraph combining syntax, control flow, and data flow. Fabian Yamaguchi et al.’s original IEEE S&P 2014 study used this representation to discover 18 previously unknown Linux-kernel vulnerabilities. This is strong evidence for the security-query technique in that target, not for a universal code-quality measure. CodeQL overview; Joern code-property graph; Yamaguchi et al., paper. Accessed 13 August 2026.
R86. Representative architecture-graph tools. jQAssistant scans software artifacts into Neo4j and applies Cypher concepts and constraints. Understand exposes dependency graphs and source metrics; Lattix uses dependency-structure matrices and propagation-related metrics; NDepend provides .NET dependency queries, diagrams, and architecture rules. These are first-party capability sources, not comparative evidence that the tools reduce maintenance cost. jQAssistant manual; Understand graphs; Understand metrics; Lattix DSM guide; NDepend architecture documentation. Accessed 13 August 2026.
R87. Gail C. Murphy, David Notkin, William G. Griswold, and Erica S. Lan, “An Empirical Study of Static Call Graph Extractors,” ACM Transactions on Software Engineering and Methodology 7(2), 1998, extending the ICSE 1996 study—comparison of several extractors on C systems. The tools produced significantly different call graphs because of their input representations and treatment of macros, function pointers, and language constructs. The study is old and C-specific, but its measurement warning remains relevant. Author-hosted paper; DOI: 10.1145/279310.279314. Accessed 13 August 2026.
R88. Thomas Zimmermann and Nachiappan Nagappan, “Predicting Defects Using Network Analysis on Dependency Graphs,” ICSE 2008—study of Windows Server 2003 binary dependencies using degree, reachability, centrality, ego-network, and structural-hole measures. Network measures outperformed the paper’s complexity comparison for identifying failure-prone binaries, and combined measures performed better still. The authors explicitly caution that the model came from one product and may not transfer even to other Microsoft systems. Microsoft Research; author-hosted paper; DOI: 10.1145/1368088.1368161. Accessed 13 August 2026.
R89. Lina Gong, Gopi Krishnan Rajbahadur, Ahmed E. Hassan, and Shujuan Jiang, “Revisiting the Impact of Dependency Network Metrics on Software Defect Prediction,” IEEE Transactions on Software Engineering 48(12), 2022, 5030–5049—30 versions of nine open-source projects evaluated across nine contexts and scenarios. Dependency-network measures improved over traditional code metrics in five scenarios, with varying effect sizes; they were not uniformly superior. Author-hosted paper; DOI: 10.1109/TSE.2021.3131950. Accessed 13 August 2026.
R90. Alan MacCormack, John Rusnak, and Carliss Y. Baldwin, “Exploring the Structure of Complex Software Designs: An Empirical Study of Open Source and Proprietary Code,” Management Science 52(7), 2006, and Carliss Baldwin, Alan MacCormack, and John Rusnak, “Hidden Structure: Using Network Methods to Map System Architecture,” Research Policy 43(8), 2014. These studies apply design-structure matrices, propagation measures, and network clustering to compare systems and 1,286 releases from 17 applications. They support structural description and comparison; causal claims about maintainability require more evidence. 2006 DOI: 10.1287/mnsc.1060.0552; 2014 working paper; 2014 bibliographic record. Accessed 13 August 2026.
R91. Hayden Melton and Ewan Tempero, “An Empirical Study of Cycles among Classes in Java,” Empirical Software Engineering 12(4), 2007, 389–415—study of 78 open- and closed-source Java applications. Among sufficiently large applications, about 45% contained a dependency cycle of at least 100 classes and about 10% one of at least 1,000 classes. This establishes prevalence and possible untangling scale, not that every cycle causes defects or maintenance delay. DOI: 10.1007/s10664-006-9033-1. Accessed 13 August 2026.
R92. Aider, “Repository map” documentation and implementation. Aider extracts definitions and references with Tree-sitter queries, builds a file graph, ranks nodes using personalized PageRank, and selects a map under a token budget. The public implementation establishes the mechanism; the cited sources do not provide an independent controlled quality or maintainability evaluation. Documentation; repomap.py. Accessed 13 August 2026.
R93. Siru Ouyang et al., “RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph,” ICLR 2025—Tree-sitter-derived definition, reference, invocation, and containment graph used as additional context for four SWE-bench Lite approaches. The paper reports a 32.8% average relative solve-rate gain, but the absolute improvements are modest and the evaluation is benchmark- and mostly Python-specific. arXiv:2410.14684; artifact. Accessed 13 August 2026.
R94. Xiangyan Liu et al., “CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases,” NAACL 2025—a graph database of modules, classes, functions, and selected structural relations queried through model-generated Cypher. The evaluated implementation was Python-only, lacked function-call edges in its schema, and reported substantial graph-construction time, limiting generalization and deployment claims. ACL Anthology; paper. Accessed 13 August 2026.
R95. Zhaoling Chen et al., “LocAgent: Graph-Guided LLM Agents for Code Localization,” ACL 2025—directed heterogeneous repository graph over directories, files, classes, and functions with containment, import, invocation, and inheritance relations. The paper reports up to 92.7% file-level localization accuracy and a 12% improvement in downstream Pass@10 issue resolution in its Python benchmark setting. These are task-level benchmark results, not longitudinal maintenance outcomes. ACL Anthology; paper. Accessed 13 August 2026.
R96. Tsvi Cherny-Shahar and Amiram Yehudai, “Repository Intelligence Graph: Deterministic Architectural Map for LLM Code Assistants” (preprint, 2026)—build- and test-centered graph for CMake repositories evaluated on structural questions across eight repositories and three commercial agents. The paper reports an average 12.2% accuracy increase and 53.9% time reduction. Seven question sets were synthetic, only three graphs were automatically extracted, and no code-change or maintainability outcome was measured. arXiv:2601.10112. Accessed 13 August 2026.
R97. Shlomit Ahituv, “Decoding Complexity: How the AST Shapes the Understanding of Code Complexity” (2026), and Jean-François Lépine, “AST Metrics: Static analysis for all” (2024). Both are practitioner tutorials that illustrate syntax-tree traversal, metric calculation, CI thresholds, and dependency rules. They are useful explanations of mechanisms, but neither provides an independent evaluation of prediction accuracy, developer outcomes, or maintenance cost; their role labels and thresholds should be read as teaching devices. Ahituv article; Lépine article. Accessed 13 August 2026.
R98. Michele Lacchia et al., Radon documentation and source. Radon uses Python’s ast module and an ast.NodeVisitor to compute language-specific cyclomatic complexity, Halstead measures, raw metrics, and a composite Maintainability Index. The source makes its counting rules inspectable; that transparency does not validate the resulting scores as predictors. The documentation presents several Maintainability Index variants and explicitly calls the metric experimental. Repository; metric introduction; ComplexityVisitor; metric aggregation. Accessed 13 August 2026.
R100. AST Metrics documentation and source. The current implementation uses Tree-sitter to parse supported languages, then computes local metrics and extracts dependencies for coupling, cycle, community, history, report, CI, and MCP features. This establishes an inspectable, agent-facing mechanism, but no independent benchmark or longitudinal maintenance evaluation was located. At commit 3292c3f41a0b895385f228889a583f90db58fa94, the documented community and risk formulas do not fully match the inspected implementation, reinforcing the need to version metric semantics. Repository; documentation; MCP tools; community documentation; pinned community implementation; risk documentation; pinned risk implementation; Tree-sitter runner. Accessed 13 August 2026.
R101. Foundational and scaling work on syntax-aware differencing. Beat Fluri, Michael Würsch, Martin Pinzger, and Harald C. Gall, “Change Distilling: Tree Differencing for Fine-Grained Source Code Change Extraction,” IEEE Transactions on Software Engineering 33(11), 2007, introduced AST matching and typed edit extraction, evaluated on 1,064 manually classified changes. Jean-Rémy Falleri et al., “Fine-grained and Accurate Source Code Differencing,” ASE 2014, introduced GumTree and evaluated its mappings through manual and large automated comparisons. Quentin Le Dilavrec et al., “HyperDiff: Computing Source Code Diffs at Scale,” ESEC/FSE 2023, evaluated a time-oriented representation on 19 large projects and reported lower time and memory relative to GumTree with high mapping agreement. The studies validate differencing accuracy or scalability against their benchmarks; they do not establish edit-script size as semantic risk or maintainability. ChangeDistiller paper; UZH tool page; GumTree paper; GumTree repository; HyperDiff conference record. Accessed 13 August 2026.
R102. Refactoring-aware differencing and entity tracking. RefactoringMiner detects refactorings and produces AST diffs for commits, pull requests, directories, and commit ranges; its current read-only MCP server exposes analysis, intended-refactoring validation, and a local diff view. Its bundled agent-review workflow tells reviewers to inspect non-refactoring and mixed regions first and explicitly warns that detected refactorings do not prove behavior preservation. Mehran Jodavi and Nikolaos Tsantalis's CodeTracker work tracks methods and variables through commit history, extended by later work to blocks, attributes, and refactoring-aware blame. The peer-reviewed evaluations and curated oracles support lineage recovery for the studied Java histories; the current multi-language roadmap, MCP integration, and broader use claims should not be mistaken for universal tracking accuracy or agent outcome evidence. RefactoringMiner repository and research index; MCP documentation; agent-review workflow; CodeTracker repository; Jodavi and Tsantalis, ESEC/FSE 2022, DOI: 10.1145/3540250.3549079; block-tracking preprint. Accessed 13 August 2026.
R103. Peer-reviewed studies of evolutionary coupling and change propagation. Thomas Zimmermann et al., “Mining Version Histories to Guide Software Changes,” ICSE 2004, evaluated association-rule suggestions at file and entity level. Ahmed E. Hassan and Richard C. Holt, “Predicting Change Propagation in Software Systems,” ICSM 2004, evaluated several history and dependency heuristics on five open-source systems. Markus M. Geipel and Frank Schweitzer, “The Link between Dependency and Cochange: Empirical Evidence,” IEEE TSE 38(6), 2012, found both a relationship and substantial mismatch between static dependencies and observed co-change, arguing that both should be considered. Marco D'Ambros, Michele Lanza, and Romain Robbes, “On the Relationship Between Change Coupling and Software Defects,” WCRE 2009, found correlations and model improvements in three Java systems. These studies support recommendation and risk-ranking signals in bounded histories; commit construction, missing dependencies, defect labels, and observational design limit causal and universal claims. Zimmermann et al. publication page; paper; Hassan and Holt paper; Geipel and Schweitzer, ETH record and paper; D'Ambros et al. paper. Accessed 13 August 2026.
R104. Change-process measures and faults. Raimund Moser, Witold Pedrycz, and Giancarlo Succi, “A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction,” ICSE 2008, found process and change measures more efficient than static attributes in its Eclipse dataset. Ahmed E. Hassan, “Predicting Faults Using the Complexity of Code Changes,” ICSE 2009, used entropy over the distribution of changes and reported stronger fault prediction than prior-change and prior-fault baselines across six open-source projects. These results concern defect prediction, not maintainability directly; size, task mix, developer activity, repository practices, and imperfect fault attribution remain plausible confounders. Moser et al. publication record and DOI; Hassan paper; DOI: 10.1109/ICSE.2009.5070510. Accessed 13 August 2026.
R105. Wei Zhuang, Huaimin Wang, and Xiangqun Zhang, “Just-in-time Defect Prediction Based on AST Change Embedding,” Knowledge-Based Systems 248, 2022. The study compares ASTs before and after a change, embeds their edit sequences, and reports predictive improvements on ten Java open-source projects. It supports further investigation of typed change representations, not a causal claim or an operationally validated maintainability gate; results depend on project selection, labels, baselines, and time-aware evaluation. Publisher abstract and DOI: 10.1016/j.knosys.2022.108852. Accessed 13 August 2026.
R106. Historical AST edits used for pattern mining and bounded repair. Matias Martinez and Martin Monperrus, “Coming: a Tool for Mining Change Pattern Instances from Git Commits,” ICSE Companion 2019, presents an open tool that finds user-specified fine-grained patterns across revisions; its demonstration evaluated 28 Defects4J revision pairs. Johannes Bader et al., “Getafix: Learning to Fix Bugs Automatically,” PACMPL/OOPSLA 2019, learned hierarchical AST edit patterns from 1,268 human fixes across six Java analyzer-warning categories. Exact top suggestions ranged from 12% to 91% by category, and the paper reports roughly 42% developer acceptance in Facebook deployment. This is strong evidence for narrow, analyzer-grounded repair patterns, not for learning general design quality from arbitrary commits. Coming paper; Coming repository; Getafix paper; arXiv:1902.06111. Accessed 13 August 2026.
R107. Ramakrishna Bairi et al., “CodePlan: Repository-level Coding using LLMs and Planning,” Proceedings of the ACM on Software Engineering 1(FSE), 2024. CodePlan combines incremental dependency analysis, change-may-impact analysis, adaptive planning, and an LLM. Across package-migration and temporal-edit tasks in C# and Python repositories requiring changes to two to 97 files, five of seven repositories passed the paper's validity checks while comparable no-planning baselines passed none. This supports explicit propagation planning on two bounded task families; the repository count is small, the full system is not available in the public artifact, and no longitudinal maintenance outcome was measured. Microsoft Research; arXiv:2309.12499; artifact. Accessed 13 August 2026.
R108. SonarSource product documentation and inspectable implementations. The documentation defines issues, complexity, duplication, coverage, remediation effort, debt ratios, ratings, profiles, gates, new code, analysis scope, security hotspots, architecture, and MCP tools. It establishes current intended behavior, not predictive validity. Important qualifications include: coverage reports come from external tools; quality and duplication scopes can exclude files; the built-in gate contains vendor defaults and a small-change exception; remediation effort comes from per-rule estimates; supported architecture and agent tools vary by language, product, edition, and release; and the MCP server can expose write operations such as issue-status changes. The supplied static-analysis guide is a useful vendor primer on these capabilities, not an outcome study. Rule model and coding attributes; metric definitions; quality gates; new code; coverage imports; analysis scope; security reports; architecture; MCP tools; CLI commands; CLI quickstart; static-analysis guide; sonar-java. Accessed 14 August 2026.
R109. SonarSource's current agentic product material and first-party experiments. The semantic-navigation article discloses six historical tasks, one model, ten runs per condition, build-and-test gating, token and cost results, and the deliberate accuracy–freshness trade-off in its local code graph. The architecture, CLI, MCP, and Agentic Analysis material documents a concrete guide–verify–solve loop. These sources support product capability and the bounded navigation-cost result; they do not independently establish correctness, security, compliance, or long-term maintenance benefit. Several supplied articles repeat the same AC/DC framing without additional outcome evidence. Sonar homepage; semantic-navigation study; MCP server and Context Augmentation tools; CLI verification; architecture management; Agentic Analysis; verification-loop argument. Accessed 14 August 2026.
R110. Priyansh Trivedi and Olivier Schmitt / SonarSource, “Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study” (preprint, 19 May 2026). Six constructed Java and Python repository pairs, 33 tasks, ten runs per side, and 660 Claude Code/Sonnet 4.6 trials produced an unchanged pass rate, lower aggregate token use, and fewer file revisits on the cleaner side. The treatment was a bundle defined partly through Sonar issues and Cognitive Complexity. Effects varied sharply by task; authors selected repositories, constructed pairs, and authored tasks; only hidden tests were graded; one model and harness were used; no final cleanliness scan or long horizon was included; and a median-based filter removed 9.7% of trials. This is emerging first-party evidence about agent footprint, not validation of Sonar's complete quality model. arXiv:2605.20049; full paper; SonarSource summary. Accessed 13 August 2026.
R111. Independent peer-reviewed studies of SonarQube measures and labels. Valentina Lenarduzzi, Nyyti Saarimäki, and Davide Taibi studied more than 95,000 issues across 33 Apache Java projects and found at most small change-proneness effects, no fault-proneness difference, and inconsistencies in type and severity. Lenarduzzi et al. found SonarQube's broad Java “bug” classifications had extremely low fault-prediction power in 21 projects. Maria Teresa Baldassarre et al. asked 81 final-year students to fix findings in 21 Java projects and found the older remediation estimates inaccurate and usually high—about 30% above recorded time on average. All three studies concern older SonarQube versions, Java, selected open-source projects, and imperfect maintenance or SZZ proxies; they narrow the interpretation of labels rather than measuring every current rule's precision. Change- and fault-proneness, DOI: 10.1016/j.jss.2020.110750; open paper record; “Are SonarQube Rules Inducing Bugs?”, DOI: 10.1109/SANER48275.2020.9054821; open manuscript; remediation-time study, DOI: 10.1016/j.infsof.2020.106377; open postprint. Accessed 13 August 2026.
R112. Olivier Gaudin / SonarSource, “The Clean as You Code Imperative,” WeAreDevelopers World Congress 2023. Gaudin argues for applying gates to new or changed code and describes his own team's gaming of a 70% coverage target. In the Q&A he says Sonar misses some problems, reports some non-problems, did not then model Domain-Driven Design, and cannot infer whole-system cleanliness from clean snippets. The talk is a vendor-practitioner account, not a comparative study; its uncited adoption, catch-rate, cost, and annual-change estimates were not used as empirical findings. YouTube. Accessed 13 August 2026.
R113. SonarSource's 2026 State of Code Developer Survey and related supplied articles. The survey reports 1,149 adult, working respondents who had used AI on the job. Recruitment included the Sonar community and an incentive for the first 500 eligible respondents; results are self-reported and the public material does not establish causal comparability between SonarQube users and non-users. The articles on trust, technical debt, review scale, the summit, and a Sonar employee's AI-built application are useful statements of vendor and practitioner experience, not objective measures of code share, outages, review effort, or tool effectiveness. Survey; PDF; recruitment post; trust-gap article; technical-debt article; review-scale article; Summit summary; developer retrospective. Accessed 13 August 2026.
R114. SonarSource's regulated-industry articles mapped product capabilities to governance concerns, especially the FFIEC Development, Acquisition, and Maintenance booklet. The mapping is not a regulator endorsement, certification, or complete compliance control. The primary FFIEC material is examiner guidance covering development and acquisition planning, governance and risk management, maintenance, change management, third parties, security, and resilience; it does not prescribe SonarQube or a separate AI-code regime. Sonar financial-services mapping; Sonar regulated-industries article; FFIEC 2024 announcement; FFIEC handbook site. Accessed 13 August 2026.
R115. Evidence on analyzer precision and failure. SonarSource reports that users marked 3.2% of more than 137 million reviewed 2025 findings as false positive. The public article does not disclose random sampling, independent adjudication, rule/language strata, the unreviewed population, or false negatives, so the figure cannot be generalized to every rule or project. Yinxing Xue et al.'s independent historical study of 350 confirmed false-positive and false-negative issues across SonarQube, PMD, and SpotBugs found failures involving rule specifications, missing cases, newer language and library features, intermediate representations, type resolution, data flow, and symbolic execution; metamorphic tests found further confirmed issues. The study characterizes known historical failures rather than estimating a population-wide rate. SonarSource precision article; Sonar rules documentation; Xue et al., arXiv:2408.13855; artifact. Accessed 13 August 2026.
R116. Liviu-Marian Berciu, Simona Claudia Motogna, and Arthur-Jozsef Molnar, “A Long-Term Exploratory Study of Source Code Quality Issues in Open-Source Python Projects,” Software Quality Journal 34, article 9 (2026). The study applied a fixed SonarQube 10.3 configuration to 3,656 releases from 57 Python projects, documenting issue composition, rule concentration, and correlations with size and commit activity. Only 77 of 342 tested activity–issue correlations were significant, spanning 23 projects, and most projects showed no significant relationship. The open, peer-reviewed, longitudinal design is useful for studying analyzer-output evolution. It remains correlational; uses Sonar findings rather than observed faults or maintenance effort as outcomes; selects releases rather than every revision; and reports raw counts that often rise with LOC. Open article and DOI: 10.1007/s11219-026-09740-z. Accessed 13 August 2026.
R23. Robert C. Martin / Clean Coders, “Clean Code, Episode 2”—names, communication, and exposing intent. cleancoders.com. Accessed 10 August 2026.
R24. Robert C. Martin, post on scaling agentic testing discipline to project size and risk—unit tests and CRAP for some work, Gherkin acceptance and QA for larger work. X. Accessed 10 August 2026.
R25. Robert C. Martin / Clean Coders, “Agentic Discipline 2”—higher-level source artifacts, implementation as the agent’s domain, and rebuilding in different languages. cleancoders.com. Accessed 10 August 2026.
R26. Robert C. Martin / Clean Coders, “Agentic Discipline 4”—BDD, coverage, mutation testing, management, debugging, and performance in agentic development. cleancoders.com. Accessed 10 August 2026.
R27. Robert C. Martin, empire-2025 agent instructions and supporting tools. The current workflow requires CRAP of 8 or less for every changed module, then differential mutation testing; acceptance scenarios may not be changed without explicit permission. crap4clj computes function scores from a documented set of Clojure decision forms and Cloverage form coverage, supports focused path filters, and exposes a Claude Code skill. These repositories demonstrate an executable control loop and locally chosen policy. They publish no comparison against defects, change effort, or lifetime cost and do not validate 8 as a portable threshold. AGENTS.md; crap4clj; clj-mutate; dependency-checker. Accessed 22 August 2026.
R28. Robert C. Martin, Acceptance-Pipeline-Specification—portable Gherkin parsing, intermediate representation, generated tests, runner, dry checking, and acceptance mutation. GitHub. Accessed 12 August 2026.
R29. Robert C. Martin, swarm-forge—a local, worktree-based orchestration system whose runnable two-pack, four-pack, and six-pack branches separate combinations of specification, coding, cleanup/refactoring, architecture, hardening, and QA. The repository verifies the workflow structure, not the independence or effectiveness of its roles. GitHub repository; Agentic Discipline 6. Accessed 12 August 2026.
R30. How Many CTOs Podcast, “There’s A Pattern To Follow: An Interview with Robert ‘Uncle Bob’ Martin” (4 August 2026)—deterministic tools, agent myopia, stepping back from code, quality gates, expert judgment, and apprenticeship. Full interview; deterministic tooling, 31:40; quality and judgment, 52:14. Accessed 10 August 2026.
R31. Robert C. Martin, “TDD Harms Architecture” (2017)—TDD as a discipline that does not replace design judgment. Clean Coder blog. Accessed 10 August 2026.
R32. Robert C. Martin, “Mutation Testing” (2016) and “Testing Like the TSA” (2017)—mutation as a test-strength check and coverage as an asymptotic discipline rather than proof of correctness. Mutation testing; coverage. Accessed 10 August 2026.
R33. Ryan Lopopolo / OpenAI, “Harness Engineering: Leveraging Codex in an Agent-First World” (11 February 2026)—first-party account of a five-month internal product, repository knowledge base, structural checks, observability, and agent workflow. OpenAI. Accessed 12 August 2026.
R34. Addy Osmani, “Agentic Code Quality” (8 August 2026)—practitioner argument for quality back-pressure throughout the agentic loop; sponsored by Sonar and not an outcome study. Substack. Accessed 12 August 2026.
R35. Jialong Chen et al., “SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration” (preprint, 4 March 2026)—100 retrospective repository-evolution tasks and repeated regression/maintainability evaluation. arXiv:2603.03823; artifact. Accessed 10 August 2026.
R36. Gabriel Orlanski et al., “SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks,” version 2 (preprint, 7 May 2026)—36 evolving tasks, 196 checkpoints, structural erosion, and verbosity measures. arXiv:2603.24755v2; benchmark. Accessed 10 August 2026.
R37. Shota Sawada et al., “To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study” (EASE 2026)—observational evidence on subsequent maintenance of agent-authored files. arXiv:2605.06464. Accessed 10 August 2026.
R38. Jessica Baolin and Nathen Harvey / DORA, “Balancing AI Tensions: Moving from AI Adoption to Effective SDLC Use” (2026)—systems-level discussion of AI adoption, throughput, instability, and delivery performance. dora.dev. Accessed 10 August 2026.
R39. Dex Horthy / HumanLayer, “Why Software Factories Fail, or: Harness Engineering Is Not Enough” (2026)—firsthand practitioner retrospective on a failed lights-off attempt, delayed maintainability costs, and restored product/architecture/program-design review. It is one self-reported account, not a controlled study. Written report; talk; later Pragmatic Engineer interview. Accessed 13 August 2026.
R40. Robert C. Martin, post advocating that agent-generated speed be reinvested in unit, acceptance, property, torture, mutation, and QA testing. X. Accessed 10 August 2026.
R117. Gergely Orosz and Elin Nilsson, “AI Tooling for Software Engineers in 2026” (3 March 2026) and “AI’s Impact on Software Engineers in 2026: Key Trends, Part 2” (19 May 2026). The first article reports 906 self-selected respondents, 55% engineers and 34% engineering leaders, with a median 11–15 years of experience; 55% said they used agents regularly. The second synthesizes qualitative responses about review overload, duplicated or verbose code, bugs, maintainability, and organizational pressure. This is a reader survey of an experienced, AI-heavy audience, not a random industry sample, repository analysis, or causal study. Survey overview; quality and adoption analysis. Accessed 13 August 2026.
R118. Gergely Orosz / The Pragmatic Engineer, reported company practices for selecting and absorbing AI development tools. “How 10 Tech Companies Choose the Next Generation of Dev Tools” (3 February 2026) describes small local comparisons of AI review comments at WeTravel and a large fintech. “How Uber Uses AI for Development: Inside Look” (10 March 2026), based on an Uber presentation, describes Code Inbox, risk profiles, uReview comment ratings, and other internal infrastructure. These purposive cases establish reported mechanisms and evaluation practices, not calibrated risk prediction, better code, or productivity; detailed protocols and outcomes were not independently published. Orosz disclosed that he formerly invested in Graphite, one of the review vendors discussed. Ten-company deep dive; Uber deep dive; Uber presentation; investment disclosure. Accessed 13 August 2026.
R119. Gergely Orosz, “Building Claude Code with Boris Cherny” (4 March 2026). Cherny reports that Claude Code's team preferred agent-directed glob and text search over attempted vector and model-built indexes because of freshness and permissions, and describes converting recurrent review comments into lint rules after three or four instances. He also refers to an internal Meta analysis linking code quality and productivity, but no method or data are public, so that quantitative claim is not used in this report. This is an interview with a product leader about his own systems and practice, not an independent comparison. Transcript and summary; video. Accessed 13 August 2026.
R120. Gergely Orosz, “TDD, AI Agents and Coding with Kent Beck” (11 June 2025). Beck describes TDD as useful when working with agents and reports difficulty preventing them from deleting tests to make a suite pass. The latter is a concrete practitioner anecdote, not a measured failure rate or evidence that TDD alone secures an agentic loop. Transcript and summary; video. Accessed 13 August 2026.
R121. Gene Kim, Kevin Behr, and George Spafford, The Phoenix Project (IT Revolution, 2013), and Gene Kim, “The Three Ways: The Principles Underpinning DevOps.” The First Way concerns end-to-end flow and avoiding local optimization, the Second Way short and amplified feedback, and the Third Way experimentation and learning. Kim describes the novel as modeled on Eliyahu Goldratt's The Goal and the Theory of Constraints. These are influential practitioner frameworks and instructional fiction, not controlled evidence that the practices cause a particular quality outcome. Book page; Three Ways; construction of the novel; book excerpt. Accessed 13 August 2026.
R122. Gene Kim and Steve Yegge, Vibe Coding: Building Production-Grade Software With GenAI, Chat, Agents, and Beyond (IT Revolution, 2025), with public excerpts and its failure-pattern guide. The material advocates tight feedback, task decomposition, modularity, frequent validation, risk-aware pairing or delegation, and explicit controls across inner, middle, and outer loops. Its case material and failure taxonomy are selected practitioner experience; no sampling frame, prevalence estimate, comparator, or long-term outcome study is published. Book page; pairing versus delegation; essential skills; coding loop; culture and validation; failure-pattern guide. Accessed 13 August 2026.
R123. Gene Kim and Steve Yegge, FAAFO Measurement Toolkit, companion to Vibe Coding (2025). The toolkit proposes measures across Fast, Autonomous, Ambitious, Fun, and Optionality dimensions and warns against turning measures into goals. Some candidates are useful raw signals, but several constructs lack operational definitions, and the proposed option-value formula and modular-independence ratio are not calibrated or empirically validated. It should be used to generate local hypotheses, not as a quality model or scorecard. Toolkit PDF. Accessed 13 August 2026.
R124. DORA / Google Cloud, State of AI-assisted Software Development: 2025, with the launch presentation by Nathen Harvey, Gene Kim, and Daniel Rock. The report draws on survey responses from nearly 5,000 technology professionals and more than 100 hours of qualitative data. It reports positive relationships between AI adoption and delivery throughput and product performance, but a continuing negative relationship with delivery stability. These are observational, substantially self-reported associations rather than causal estimates. In the launch discussion, Kim explicitly distinguishes hypotheses from results and says the testing-feedback hypothesis was unclear while small task decomposition and more frequent version-control activity received some support; his Nyquist analogy is interpretation, not a calibrated measure. DORA report and method entry point; Google Cloud summary; launch video and transcript. Accessed 13 August 2026.
R125. Gene Kim, host, The Idealcast, episodes 21 and 25. Episode 21 interviews Gail Murphy about modularity, information hiding, software supply chains, and why developer productivity is difficult to define. Episode 25 interviews Nicole Forsgren and Jez Humble about the State of DevOps research, feedback, continuous delivery, and independently testable and deployable components. The archive provides useful expert exposition and routes to underlying work; the substantive claims belong to the guests and should not be treated as new findings by Kim or as independently replicated evidence. Episode 21, Gail Murphy; Episode 25, Nicole Forsgren and Jez Humble; podcast archive. Accessed 13 August 2026.
R126. Martin Fowler, “Is High Quality Software Worth the Cost?”, “Design Stamina Hypothesis,” “Technical Debt,” and “Cannot Measure Productivity.” Fowler distinguishes externally visible quality from internal design, models internal quality as an influence on the cost of future changes, and describes debt interest as extra effort paid when changing affected code. He also calls the design-payoff curve a hypothesis, says its crossover point is unknown, and argues that software output and productivity cannot be captured by lines of code. These are influential economic and practitioner arguments, not controlled estimates of refactoring return. quality and cost; Design Stamina Hypothesis; technical debt; productivity measurement. Accessed 13 August 2026.
R127. Martin Fowler with Kent Beck, Refactoring: Improving the Design of Existing Code, second edition (Addison-Wesley, 2018), and Fowler's “Code Smell” and “Opportunistic Refactoring” entries. Fowler defines refactoring as controlled, behavior-preserving improvement through small transformations and treats smells as indications that require judgment rather than defects. The book and entries provide a method and vocabulary; they do not show that every smell removal pays back or that tests prove full semantic equivalence. book page; code smell; opportunistic refactoring. Accessed 13 August 2026.
R128. Giles Edwards-Alexander, “The Economic Benefit of Refactoring,” Thoughtworks Exploring Gen AI, published on martinfowler.com (30 July 2026). In one approximately 150,000-line, agent-written application, 15 guided refactoring steps reduced a 17,155-line Rust file to 3,695 lines and the estimated input for one replayed change from 159,564 to 27,360 tokens while total layer size stayed similar. The case uses one task, one run per snapshot, an approximate token count, no published source or traces, no protected correctness outcome, and no accounting for the refactoring itself. The article reports that the useful restructuring required human direction. It supports a promising experimental design and mechanism, not an 83% general cost-reduction claim. article. Accessed 13 August 2026.
R129. Birgitta Böckeler, “Maintainability Sensors for Coding Agents,” published on martinfowler.com (27 May 2026). The article reports experience adding type, lint, test, secrets, dependency, mutation, and compiler-based coupling checks to one AI-rebuilt TypeScript/Next.js dashboard. It documents useful local feedback, 13 surviving mutants despite high statement coverage, noisy coupling interpretation, conflicting signals, and run-to-run variation. This candid single-system practitioner experiment demonstrates mechanisms and failure modes; it has no comparator or measured maintenance outcome. article; companion CLI repository. Accessed 13 August 2026.
R130. Birgitta Böckeler, “TDD Inside the Agent Loop—Theater or Actual Value?”, Thoughtworks Exploring Gen AI, published on martinfowler.com (11 August 2026). In roughly 22 solutions across three greenfield Python tasks, the exploratory comparison found no clear advantage for TDD-prompted runs in model-judged code, design, tests, or mutation score, and reported substantially higher token proxies. The task set is tiny; generation and judging use related model families; the judge is not a product oracle; adherence is imperfect; and the token total counts cache reads repeatedly. The artifact makes the examples inspectable, but the result neither evaluates human TDD nor supports a general causal conclusion. article; artifact. Accessed 13 August 2026.
R131. Zhao Tian, Zifan Zhang, Tao Xiao, Dong Wang, Masanari Kondo, Junjie Chen, and Yasutaka Kamei, “Refactoring Runaway: Understanding and Mitigating Tangled Refactorings in Coding Agents for Issue Resolution” (preprint, submitted 21 May 2026). The study applies RefactoringMiner to 3,691 valid Java patches selected from 4,608 generations by three agent frameworks and 12 models. Detected refactoring was less prevalent in agent patches than in human reference patches; refactoring presence was not significantly associated with functional correctness overall, while method-level tangled refactoring was associated with lower correctness and compilability. LLM-based untangling improved compilability for selected failures but sometimes regressed successful patches and produced only modest correctness gains. Limits include 128 issues in one Java benchmark family, imperfect refactoring detection and benchmark tests, observational models, and no independent human ground truth for the refiner. arXiv:2605.22526; artifact. Accessed 13 August 2026.
R132. ISO, ISO 9001:2015—Quality Management Systems—Requirements and ISO 9004:2018—Quality Management—Quality of an Organization—Guidance to Achieve Sustained Success. ISO 9001 uses a process approach, risk-based thinking, objectives, performance evaluation, internal audit, management review, corrective action, and continual improvement; ISO 9004 supplies broader guidance and self-assessment for sustained success. ISO was developing a new ISO 9001 edition at the evidence cut-off, so implementations should verify the applicable edition. ISO 9001 can certify conformity of a quality-management system within a declared scope; neither standard certifies source-code maintainability or proves that an intervention caused better product or business outcomes. ISO 9001 official record; ISO 9001 overview; ISO 9004 official record; ISO quality-management principles. Accessed 13 August 2026.
R133. ISO/IEC/IEEE, 90003:2018—Software Engineering—Guidelines for the Application of ISO 9001:2015 to Computer Software and 12207:2026—Systems and Software Engineering—Software Life Cycle Processes. ISO/IEC/IEEE 90003 interprets quality-management guidance for software acquisition, supply, development, operation, and maintenance. ISO/IEC/IEEE 12207:2026 supplies a common framework for processes across acquisition, development, operation, maintenance, support, and retirement; it can also be used to define, control, and improve lifecycle processes. The current edition explicitly does not require one lifecycle model, methodology, or engineering approach. It replaced the 2017 edition, which ISO withdrew on 29 April 2026. These documents organize responsibilities and evidence; they do not establish an optimal process, measure product quality, or show that adoption improves outcomes. 90003 official record; 12207:2026 official record; withdrawn 12207:2017 record. Accessed 13 August 2026.
R134. ISO/IEC, ISO/IEC TR 33014:2013—Information Technology—Process Assessment—Guide for Process Improvement, with Jan Pries-Heje and Jørn Johansen, “Change Strategy for ISO/IEC 33014: A Multi-case Study on Which Change Strategies Were Chosen” (EuroSPI 2014). The Technical Report structures assessment-based improvement at strategic, tactical, and operational levels. The companion study explains that the strategic level connects business goals, organizational scope, models, roles, and overall change strategy; its 49 assessments in 44 organizations found several context-dependent strategies rather than one dominant universal method. The study concerns recommended strategies, not comparative improvement outcomes, and later work has argued that the standard's assessment-based tactical scope needs expansion for other kinds of change. ISO catalogue record; multi-case study; scope critique and extension. Accessed 13 August 2026.
R135. ISO/IEC/IEEE, 15939:2017—Systems and Software Engineering—Measurement Process. The standard defines a measurement process that connects information needs with measures, data collection, analysis, and communication. It is complementary to goal-driven measurement in Section 2.2: a repeatable process can make a measure traceable, but it cannot establish that the construct, threshold, or causal interpretation is valid. Official ISO record. Accessed 13 August 2026.
R136. ISO/IEC, 20000-1:2018—Information Technology—Service Management—Part 1: Service Management System Requirements. The standard covers establishing, implementing, maintaining, and continually improving a service-management system, including performance evaluation, incidents, problems, nonconformity, and corrective action. It helps connect development with operated-service evidence; certification remains scoped management-system conformity, not proof of source-code or product quality. Official ISO record. Accessed 13 August 2026.
R137. Victor R. Basili and H. Dieter Rombach, “The TAME Project: Towards Improvement-Oriented Software Environments,” IEEE Transactions on Software Engineering 14(6), 1988. The paper integrates goal/question/metric planning and analysis with a Quality Improvement Paradigm: characterize the environment, set goals and measures, execute, analyze, package experience, and feed it into current and future projects. It is a foundational software-engineering model and prototype account, not a controlled estimate of the business effect of an Experience Factory. DOI: 10.1109/32.6156; open bibliographic record. Accessed 13 August 2026.
R138. Michael Unterkalmsteiner et al., “Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review,” IEEE Transactions on Software Engineering 38(2), 2012. Among 148 papers published from 1991 through 2008, pre/post comparison was the most common evaluation strategy, used by 49%; quality, cost, and schedule were measured in 62%, 41%, and 18% respectively. The review found sparse handling of potential confounders, incomplete context descriptions, and relatively little long-term customer-satisfaction or return-on-investment measurement. It evaluates an older and heterogeneous literature, but directly supports caution about causal claims from local before/after dashboards. DOI: 10.1109/TSE.2011.26; open manuscript. Accessed 13 August 2026.
R139. Torgeir Dybå, “An Empirical Investigation of the Key Factors for Success in Software Process Improvement,” IEEE Transactions on Software Engineering 31(5), 2005. A survey of 120 software organizations reported six organizational factors explaining more than half of the variance in its SPI-success outcome and concluded that organizational conditions were at least as important as technology. The work improves on anecdotal success stories but remains a cross-sectional, self-reported survey whose constructs and associations do not establish which intervention will cause improvement in a particular organization. IEEE Xplore; DOI: 10.1109/TSE.2005.53. Accessed 13 August 2026.
R140. Timo O. A. Lehtinen, Juha Itkonen, and Casper Lassenius, “Recurring Opinions or Productive Improvements—What Agile Teams Actually Discuss in Retrospectives,” Empirical Software Engineering 22, 2017. The longitudinal case study analyzes 37 team retrospectives over almost three years in one large distributed organization. Teams mostly discussed nearby controllable topics; participant opinion could diverge from available evidence; and complex or uncontrollable topics recurred. The study supports evidence-backed action ownership and escalation, not a universal estimate of retrospective effectiveness. DOI: 10.1007/s10664-016-9464-2; open paper. Accessed 13 August 2026.
R141. ISO/IEC, 42001:2023—Information Technology—Artificial Intelligence—Management System. The certifiable management-system standard applies to organizations that develop, provide, or use AI systems and covers policy, objectives, risk and impact processes, competence, operational control, performance evaluation, internal audit, corrective action, and continual improvement. It can structure governance of coding agents and their control plane. Conformity remains evidence about the scoped AI management system, not a model-capability evaluation or proof that generated software is correct, secure, or maintainable. Official ISO record; ISO explanation. Accessed 13 August 2026.
R142. Leonardo Stertz Sfreddo et al., “ISO 9001 Based Quality Management Systems and Organisational Performance: A Systematic Literature Review” (2018), and Nadia Astrini, “ISO 9001 and Performance: A Method Review” (2018). The first review found that most studies from 2000–2017 reported a positive relationship with one or more performance dimensions, while noting no definitive conclusion and gaps in maturity and process-management treatment. The method review found positive conclusions in 69% of cross-sectional studies but no relationship in 77% of longitudinal studies and warned that time sequence, terminology, and indicators affected results. These cross-sector reviews do not isolate software organizations or every implementation, but they show why certification and simple association should not be interpreted causally. systematic review, DOI: 10.1080/14783363.2018.1549939; method review, DOI: 10.1080/14783363.2018.1524293. Accessed 13 August 2026.
R143. Koen Claessen and John Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs,” ICFP 2000; Thomas Arts, John Hughes, Ulf Wiger, and Joakim Johansson, “Testing Telecoms Software with Quviq QuickCheck,” Erlang Workshop 2006; and Harrison Goldstein et al., “Property-Based Testing in Practice,” ICSE 2024. The original paper establishes the property-and-generator mechanism. The Ericsson case reports faults and specification ambiguities missed by existing techniques. The 2024 study draws on 30 interviews at one sophisticated adopter, Jane Street, finding high-leverage use in complex and differential/model-based testing alongside difficulties with generators and effectiveness assessment. Together they show an established technique with credible bounded practice, not a universal comparative effect on defects or maintenance cost. QuickCheck DOI; Ericsson case; ICSE 2024 paper; ICSE DOI. Accessed 14 August 2026.
R144. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr, “Finding and Understanding Bugs in C Compilers,” PLDI 2011. Csmith generated valid C programs and used differential behavior to report more than 325 previously unknown compiler bugs over three years. This is strong field evidence for generated testing with a precise oracle in compiler engineering; Csmith is not evidence that every property-testing library, generator, or application domain will produce similar results. author-hosted paper; DOI: 10.1145/1993498.1993532. Accessed 14 August 2026.
R145. Antithesis and Alperen Keles, “Introduction to Property-Based Testing” (10 August 2026), with the Antithesis announcement on X (13 August 2026) and Antithesis properties documentation. The approximately 30-minute first course installment, made with Carnegie Mellon University's Center for Transformational Play, includes a transcript and hands-on Hypothesis exercise. Its separation of test scope, purpose, specification, and input generation, property blueprints, and generator-measurement advice are useful instructional material. The X post merely announces the course; neither it nor the video is an empirical comparison. The documentation correctly explains the asymmetry of sampled “always” and “sometimes” properties, but its treatment of property testing and fuzzing as essentially interchangeable is broader than common technical usage. course and transcript; video; X announcement; property-testing guide; property semantics. Accessed 14 August 2026.
R146. Hegel primary sources: current website, compatibility and libhegel references, GitHub organization and language libraries, Hegel agent skill, and David MacIver's launch article, “Hypothesis, Antithesis, synthesis” (24 March 2026). The current documentation describes a shared, in-process, Hypothesis-derived native engine and front ends for Rust, Go, C++, TypeScript, Java, and OCaml, with generation, shrinking, replay, test-database behavior, and standard-runner integration. Hegel is explicitly beta and permits breaking changes. The launch article describes an earlier Python-backed server architecture and planned Rust replacement, so its design details are historical rather than current. The agent skill contains thoughtful oracle and generator guidance, and the launch article's agent-written bug examples are inspectable demonstrations; neither constitutes a comparative evaluation of Hegel, the skill, or long-term maintenance outcomes. Hegel; why Hegel; compatibility; libhegel reference; architecture page; GitHub organization; agent skill; launch article. Accessed 14 August 2026.
R147. Antithesis product documentation, public repositories, and Carl Sverre, “Antithesis Skills for Agents” (25 March 2026). The documentation specifies the container and test-template contract, hermetic dependency model, default and custom properties, assertion and reachability semantics, configurable faults, coverage-instrumentation overhead, findings history, and replay/debugging workflow. The public organization exposes language SDKs, the Snouty CLI, a GitHub Action, agent skills, examples, and workloads, but not the deterministic hypervisor or exploration implementation. The skills require reviewable research artifacts and divide research, setup, one-property-at-a-time workload implementation, launch, triage, log analysis, and debugging into separate stages; their own documentation says some capabilities and Snouty are still evolving. These first-party sources are authoritative for intended interfaces and visible workflow design, not independent evidence of search effectiveness, property correctness, onboarding cost, defect yield, or agent performance. welcome; how Antithesis works; AI integration; assertions; findings history; dependencies; coverage instrumentation; skills repository; Snouty; GitHub organization; agent-skills article. Accessed 14 August 2026.
R148. Jingyu Zhou et al., “FoundationDB: A Distributed Unbundled Transactional Key Value Store,” SIGMOD 2021, and Alex Sarkesian and Denis Rystsov / Cockroach Labs, “Antithesis of a One-in-a-Million Bug: Taming Demonic Nondeterminism” (26 February 2024). The peer-reviewed FoundationDB paper documents simulation-first testing with injected disk, process, network, and request faults in an important production system. The CockroachDB account documents Antithesis helping reproduce and diagnose a rare transaction bug, while candidly reporting substantial instrumentation, logging, repeated runs, human reasoning, and replay limitations across modified binaries. These sources provide strong mechanism and bounded deployment evidence for distributed systems, not a controlled return-on-investment estimate or proof that the proprietary platform improves general code maintainability. FoundationDB paper; DOI: 10.1145/3448016.3457559; Cockroach Labs case. Accessed 14 August 2026.
R149. Michael Konstantinou, Florian Tambon, and Mike Papadakis, “On the Risk of Coding Before Testing: An Empirical Study on LLM-Based Test Generation Workflow” (preprint, 6 July 2026). Across HumanEval+, MBPP, and BigCodeBench tasks in Python and five model families, the authors selected subtle faulty generated implementations and found higher fault detection when tests were generated independently from the task description than after faulty code in a preserved conversation—25% versus 14% in the aggregate reported result. Exposing the code also reduced detection under summarization, chain-of-thought, and chain-of-verification variants. The study evaluates generated unit tests rather than property tests, one selected faulty implementation per available task, and a simplified agentic workflow; it supports concern about shared context and error propagation, not a repository-level effect size or a claim that implementation-blind testing is always superior. arXiv:2607.05139. Accessed 14 August 2026.
R150. First-party Antithesis adopter accounts from WarpStream (2024), the Cardano Foundation (2025), and Aiven (2026). WarpStream reports a data race found after 233 seconds and a rare data-loss regression, alongside several months of integration work. The Cardano Foundation reports about two weeks to a meaningful run, three reproduced known or deliberately reintroduced bugs, and three previously unknown findings, including one confirmed and patched consensus issue. Aiven describes an explicit 14-worker Kafka, PostgreSQL, MinIO, driver, workload, and assertion setup and reports roughly 9,700 logical test hours without a new Inkless defect after earlier property-based testing. The technical detail and null result make these more informative than vendor testimonials, but they remain selected, uncontrolled adopter reports without common baselines, denominators, or independently audited outcomes. WarpStream; Cardano Foundation; Aiven. Accessed 14 August 2026.
R41. Mark Seemann, Code That Fits in Your Head: Heuristics for Software Engineering (Addison-Wesley Professional, 2021), with the author’s announcement, published contents, and related excerpt on referential transparency and fractal architecture. The book presents a practitioner framework organized around acceleration and sustainability; its specific heuristics are not a controlled empirical validation. Announcement and article index; contents; referential-transparency excerpt. Accessed 12 August 2026.
R42. Mark Seemann, “Code Quality Isn’t Software Quality” (4 March 2019)—distinction between externally observable software utility and the readability and malleability of implementation, joined in the idea of sustainable code. ploeh blog. Accessed 12 August 2026.
R43. Mark Seemann, “Put Cyclomatic Complexity to Good Use” (9 December 2019), “Curb Code Rot with Thresholds” (13 April 2020), and “CC Golf” (14 November 2023). Seemann presents cyclomatic complexity as a rough guide, treats seven as an approximate human-oriented rule rather than a validated constant, and argues that thresholds are useful mainly because they trigger attention. Complexity and testing; thresholds; CC Golf. Accessed 12 August 2026.
R44. Mark Seemann, “Readability Verification” (18 October 2021)—argument that readability should be evaluated by a reviewer who did not write the code, using small changes that the team can afford to reject. The article explicitly characterizes the evidence base as substantially anecdotal and subjective. ploeh blog. Accessed 12 August 2026.
R45. Mark Seemann, “Code That Fits in a Context Window” (2 February 2026)—short hypothesis that fractal architecture and small local parts may help LLMs cope with repository and context limits. It proposes an analogy; it does not test the hypothesis. ploeh blog. Accessed 12 August 2026.
R46. Mark Seemann, “Does Code Quality Still Matter?” (23 July 2026), including his 11 August follow-up comment. The article is explicitly speculative: it asks whether human-oriented quality rules survive machine-only maintenance, while proposing that machine-specific constraints and technical debt may still change the cost of future work. ploeh blog. Accessed 12 August 2026.
R47. Mark Seemann, “Programming Languages for AI” (30 March 2026)—argument that portability, static typing, security restrictions, analyzability, and verifiability remain useful even without human code reading, and that AI-first languages should optimize for verification rather than ease of generation. ploeh blog. Accessed 12 August 2026.
R48. Mark Seemann, “Critiquing Tests” (16 February 2026)—preliminary proposal for evaluating LLM-generated test suites through coverage, mutation testing, simple test structure, repeatability, stronger type systems, static analysis, and suppression checks. Seemann describes the proposal as incomplete and calls for more research. ploeh blog. Accessed 12 August 2026.
R49. Mark Seemann, “Secret Agentic AI” (13 April 2026)—speculative provider and supply-chain threat model, with the operational recommendation to treat agent-authored changes as untrusted external contributions requiring scrutiny. The geopolitical and economic scenario is not empirical evidence of an observed attack. ploeh blog. Accessed 12 August 2026.
R50. Mark Seemann, “AI-Generated Tests as Ceremony” (26 January 2026)—epistemic critique of tests generated only after implementation, with proposals to observe a red–green sequence or deliberately perturb the system to show that an ex-post test can fail. These procedures increase confidence but do not prove that the test oracle represents the requirement. ploeh blog. Accessed 12 August 2026.
R51. Mark Seemann, “In Defence of Correctness” (3 March 2026)—argument that software has heterogeneous risk and that a substantial category, including decision-, security-, and safety-relevant systems, cannot responsibly treat correctness as optional. The article is a practitioner argument illustrated by examples, not an estimate of the size of that category. ploeh blog. Accessed 12 August 2026.
R52. Mark Seemann, “Will You Go to Prison for an AI?” (23 March 2026)—warning about professional accountability for high-consequence agent-produced software. It raises a governance question and should not be read as legal advice or as a general statement of liability law. ploeh blog. Accessed 12 August 2026.
R53. Mark Seemann, “Good Names Are Skin-Deep” (23 November 2020)—argument that descriptive names help but remain ambiguous and fragile, while types and explicit separation of pure logic from side effects provide stronger structural protection. ploeh blog. Accessed 12 August 2026.
R54. Nelson Cowan, “The Magical Number 4 in Short-Term Memory: A Reconsideration of Mental Storage Capacity,” Behavioral and Brain Sciences 24(1), 2001, 87–114—review and theoretical reconsideration of short-term storage capacity. PubMed; DOI: 10.1017/S0140525X01003922. Accessed 12 August 2026.
R55. Nelson F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics 12 (2024)—controlled evidence that long-context models can use information unevenly depending on its position. The tasks are long-context retrieval and question answering, not software maintenance. ACL Anthology; arXiv:2307.03172. Accessed 12 August 2026.
R56. Tianyang Liu et al., “RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems,” ICLR 2024—benchmark separating cross-file context retrieval, next-line prediction, and their combination. It studies repository-level completion rather than autonomous long-horizon maintenance, but establishes relevant-context retrieval as a distinct problem. ML Anthology; arXiv:2306.03091. Accessed 12 August 2026.
R57. Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev, “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” (preprint, version 2, 23 June 2026). Across SWE-bench and the authors’ 138-instance CTXbench, neither generated nor developer-committed context files significantly improved success over having no context file, while inference cost rose by more than 20% on average. Developer-written files did significantly outperform generated ones. The study covers four agent/model configurations but focuses on Python tasks. arXiv:2602.11988. Accessed 12 August 2026.
R58. Xing Zhang et al., “Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents” (preprint, version 2, 28 May 2026)—679 public rule files, 25,532 extracted rules, and more than 5,000 Claude Code/Opus 4.6 runs on a discriminative subset of SWE-bench Verified. The reported constraint-versus-directive pattern is promising, but the study uses one agent, model, benchmark, and language; its average rule gain relies on an across-condition sign test, and no individual rule survives strict correction for 18 comparisons. arXiv:2604.11088. Accessed 12 August 2026.
R59. Zuodong Xiang, Yike Zhang, YueMing Zhang, and Hailu Xu, “Cross-Model LLM Code Review: Should You Use Claude to Review Codex or Vice Versa?” (Agentic Software Engineering workshop at KDD 2026)—six writer–reviewer conditions on 116 hard and medium LiveCodeBench tasks. Claude Opus 4.7 review improved Codex GPT-5.5 drafts in this setup, while Codex review reduced the stronger Claude baseline. Reviewers could not execute tests, and the single-file contest tasks do not represent repository or architectural review. arXiv:2607.21656; artifact. Accessed 12 August 2026.
R60. Hamid Mohayeji Nasrabadi, Andrei E. Agaronian, Eleni Constantinou, Nicola Zannone, and Alexander Serebrenik, “Securing Dependencies: A Comprehensive Study of Dependabot’s Impact on Vulnerability Mitigation,” Empirical Software Engineering 30, article 89 (31 March 2025)—4,195 non-open security-update pull requests covering 4,978 vulnerabilities in 978 active JavaScript projects. The study defines “non-open” as merged or closed. Its observational analysis finds faster automated than manual fixes and slightly higher merge ratios in projects with tests and CI; it does not establish that those tools caused the difference or that every merged update was defect-free. Springer Nature; open-access author record. Accessed 12 August 2026.
R61. First-party accounts of mature deterministic change systems. Meta reports that SCARF removed more than 100 million lines through more than 370,000 change requests over five years, combining static and dynamic evidence and feeding review-caught false positives back into analysis. Uber reports migrating more than 75,000 JUnit test classes and changing about 1.25 million lines over four months through typed OpenRewrite recipes, preconditions, build/test/CI validation, and incremental rollout; its authors say generative AI was unsuccessful for multi-file migration and was used only to help debug failures. These reports establish scale and mechanisms, not independent comparative effectiveness. Meta, “Automating Dead Code Cleanup”; Uber, “How Uber Executed a JUnit Migration at Massive Scale”. Accessed 12 August 2026.
R62. Devon Edwards Joseph / Spotify, “Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4)” (22 April 2026)—first-party account of 240 automated pull requests within a migration affecting about 1,800 direct downstream pipelines. Spotify reports better results on standardized frameworks after adding explicit mapping tables and abstention rules, abandoning the heterogeneous Scio path, and relying on owning teams for manual tests where build-time tests were absent. The source calls ten engineering weeks a prior estimate and does not publish total agent runs, acceptance/revert rates, defects, review effort, or cost. Spotify Engineering. Accessed 12 August 2026.
R63. Alexis Georges / LaunchDarkly, “Stories from the Factory Floor: Building a Software Factory on Our Scariest Code” (3 August 2026)—first-party account of a legacy-frontend rewrite by two senior engineers. It candidly reports the failed six-week autonomous plan, later human-steered phases, visual parity work, feature flags, internal dogfooding, roughly $7,000 in inference cost, and an internal rather than completed customer rollout. The article does not report final elapsed time, human effort, defect or rollback outcomes, or a comparator; LaunchDarkly sells the release controls it recommends. LaunchDarkly. Accessed 12 August 2026.
R64. Matthew Phillips / Cloudflare, “How We Built a Software Factory to Drive Astro’s GitHub Issue Count to Zero” (4 August 2026), Astro’s open triagebot-action, and public workflow outcomes. The article reports reducing the backlog from more than 200 issues to about 30, so its headline anticipated rather than documented zero. One complete successful path appears in issue #17657 and pull request #17658; selected rejected fixes appear in #17600, #17343, and #17656. At the 11 August snapshot, live GitHub searches showed 123 fix verified pull requests: 88 merged, five open, and 30 closed without merge. These live labels and selected cases establish mechanism and inspectable outcomes, not an audited quality or failure rate. Cloudflare account. Accessed 12 August 2026.
R65. Nicholas Carlini / Anthropic, “Building a C Compiler with a Team of Parallel Claudes” (5 February 2026)—first-party capability experiment with 16 agents, nearly 2,000 sessions, two weeks, and just under $20,000 in API cost. Strong compiler suites and GCC differential testing supported a roughly 100,000-line compiler, but the author describes the Rust code as below expert quality, the output as inefficient, several components as incomplete or buggy, and a 16-bit x86 shortcut through GCC. Anthropic; source repository. Accessed 12 August 2026.
R66. Anthropic, “An Update on Recent Claude Code Quality Reports” (23 April 2026)—primary postmortem tracing user-visible quality changes to three harness or configuration changes. One stale-context bug passed multiple human and automated reviews, unit and end-to-end tests, automated verification, and dogfooding, yet took more than a week to isolate. Anthropic reports adding broader per-model evaluations, prompt ablation and audit tooling, soak periods, and gradual rollout. It is a vendor’s retrospective about its own product, but directly documents control-layer failure and remediation. Anthropic. Accessed 12 August 2026.