TL;DR
- On the Claude Code team, about 59% of what the AI reviewer kept flagging across rounds was already there at its first pass.
- Four rules changed re-review, and mean reviews to pass fell from 2.08 to 1.32 on internal pull requests.
- Two of the four are built into Code Review for Claude Code; the other two are an instruction to the reviewer and a count you keep.
An agent opens a pull request, a reviewer agent posts findings, and the author agent fixes each one. Every fix is a new commit, so the reviewer reads the change again and posts more. A person waves off a nit in seconds; an agent fixes it, and the fix restarts the round. The Claude Code team ran git blame on what its reviewer kept flagging and found about 59% had been there since the first pass. Each round was turning up a few more old bugs, and the team had made that worse by running several rounds by default; one CLI pull request went through more than fifty rounds overnight before its author stopped it. The team’s fix, written up here, is four rules on what a re-review may see and say.
Re-review only the diff
The first few passes read the whole diff against the repository; after them the reviewer sees only what changed since the last round, so it has fewer chances to surface old bugs in code it already judged. In Code Review a later review that starts on its own focuses on what changed since the last one, posts fewer nits and pre-existing findings, and counts what it left out in a line ending “not posted”; @claude review runs a full pass on request. Those first passes carry the weight, since once re-review narrows, a defect the diff cannot show waits for a requested full pass or for the team’s nightly sweep over existing code, and the reviewer reads code without building or running it.
Post no finding twice
On MCR-Bench, 2,269 multi-round review tasks, the largest class of false positives, 32.5%, was a defect already resolved and flagged again as new; the models are older, so the rate belongs to them and the failure to any reviewer that keeps no record. An open issue on the Claude Code GitHub Action reports reviews that re-raise comments already fixed in later commits; a user found in the logs that its final prompt includes every pull request comment, resolved or not. Code Review keeps the record in the threads instead. The next review resolves a finding’s thread if it judges the issue fixed, a reply alone does not dismiss one, resolving the thread yourself does, and the next review reads the replies and can close the thread. If the same kind of finding keeps coming back, the docs send you to REVIEW.md.
Flag the class of bug
When the reviewer finds one instance of a pattern, it names the class, so the author fixes every instance, not one line. Naming the class on one pull request is the small form of a routine the team runs repository-wide: when the reviewer keeps flagging the same thing, the fix moves into a skill, a lint rule or the instruction files, and for review that file is REVIEW.md. The good ones say what failure looks like in that repository, redefine what Important means there, cap the nits (the docs’ example allows five), skip generated code and whatever CI already checks, and require a file and line as evidence for a behaviour claim. Code Review reads the file from the base or default branch, never from the pull request’s own changes, and honours instructions to lower a severity, exclude findings or cap nits only when it reads them from a protected branch.
After three rounds, ask the author to rethink
After three rounds the reviewer asks the author to step back and reconsider the approach. That is an escalation, not a stop and not an approval. Several rounds can be worthwhile on a risky change, the team says, but not on every pull request, and approval scales the same way: a classifier sets a tier from the change, a low-risk change from a trusted author can merge with no person when CI and both automated reviews are clear, no finding is open, and neither a second classifier nor a protected path calls for one, and an agent’s approval never counts as a person’s. Code Review’s docs describe no round count, so in an Action or your own harness the count is yours to keep; Sourcery caps automatic re-reviews at five per pull request.
Rounds-to-pass beside recall
The team tracks rounds-to-pass beside recall, because a loop that is shorter and catches fewer bugs has only moved the cost. With all four rules in place, the share of internal pull requests needing five or more rounds fell from 8.9% to 1.5% and mean reviews to pass from 2.08 to 1.32. That is one before-and-after for all four rules together.
The team counts noise by what people do with each comment: fixed it, called it wrong, called the behaviour intentional, or decided it did not matter. It also runs the reviewer on pull requests known to be clean, counting reports that would have led to a harmful change. Past incidents form an eval set the reviewer is scored against daily, and a change to its prompts, tools or model stays only if the score rises.
Implications
Anthropic’s rule stops at asking the author to reconsider the approach after three rounds. Our own rule goes one step further and hands the change to a person. Routine changes from trusted authors merge with no human approval once CI, the automated reviews and every open finding are clear, and the time that saves pays for deep review and a human approval on billing, CI workflows and any path an owner lists. The price is a capped loop whose cost in missed bugs is unmeasured, which we accept on routine changes because that is where a missed bug costs least. Revert and incident rates show whether that price was paid, and since Anthropic has no circuit breaker yet that tightens approval when they move, a person applies that rule for now.
Takeaways
- Move a finding that keeps coming back into REVIEW.md, or into a lint rule when CI can check it, and tell the reviewer to skip what CI enforces.
- Cap the loop at three rounds and hand the change to a person; on a path owners mark as risky, keep the rounds and the human approval.
- Let a routine change from a trusted author merge with no human approval once the automated reviews are green and every finding is resolved; the lighter route ends when the path’s revert or incident rate rises.
- Measure rounds-to-pass beside recall before changing re-review; until recall is measured, keep the rounds on any change where a missed bug is expensive.