Hi Simon, On Jul 24, 2026 at 1:12:26 AM, Simon Jakobi via ghc-devs < ghc-devs@haskell.org> wrote:
So I think that it would be worth experimenting with LLM-generated reviews, in order to reduce the load on human reviewers and in order to catch more bugs and performance issues before they make it into the code base. But if we require that the person who prompts the review and reports it on the MR has to take full responsibility for that review, I think that such an experiment is unlikely to stand a chance. In particular, the person providing the review would then probably be required to understand the generated review, and possibly the MR too. As a consequence we would be less likely to get LLM-generated reviews on the more complex MRs where they could be particularly helpful.
I'm not sure if there have been any problems with LLM-generated reviews so far? My suggestion would be to drop the responsibility requirement for clearly marked LLM-generated MR reviews.
I have been using Claude Code (Opus) to assist with code reviews lately. I find that it is of great assistance in doing code reviews, but in no way can be left to stand on its own as a reviewer. It will quickly find several issues. I then examine each one. As needed, I ask Claude for explanations and justifications, and consult the code myself, to make sure I understand. I find that most of the issues it finds are legitimate, but some are not actually issues - it may have misunderstood something about the code or the intent. I will then continue to review the code myself, and often will find major issues that Claude missed. Although I think Claude is of great assistance in getting up to speed on the merge request, and with questions on the code, it is no magic bullet. There is no way to ensure that it doesn’t generate false positives, or false negatives. Thus I think that generating reviews completely automatically with LLMs is likely to cause frustration to the MR author. I’m fine with someone doing an LLM-assisted review, but agree with the Simon PJ’s proposed notion of someone needing to take responsibility for it. We should avoid “success at all costs”. — Lyle