This reviews each of four Modelplane PRs five times with the review bot, without posting, and measures each run's share of the distinct findings that the five runs raised between them. It then judges whether a maintainer would want each finding posted, and traces where each run lost the findings it didn't post. See modelplaneai/modelplane#622 for the results.
review-eval.ymlis the workflow that runs the reviews. It's the review job fromreview.ymlat 7f50acb, changed to run every entry inruns.jsonin parallel and to post nothing. Each run reviews the whole PR, which can be in another repo. It fetches each PR's context once, before any run starts, and leaves the bot's own reviews, and replies to them, out of it. The runs reported in the issue used an earlier version that fetched context in each job, so a