Dear Simon,

Sorry for the late reply, I had been enjoying some time in Ha Long Bay with my family.
Let me start by stating that I do not contest the potential value of LLM reviews. I use them
myself as well. And I think providing this service to interested parties is great! It's presumably
something you enjoy working on, and one more option for those who are interested. That
seems like a win-win to me.

I don't think I can quantify or qualify if, and how much a secondary review by an LLM to
supplement one's own review would impact negatively.  I believe everyone will have to
experiment with this themselves and figure out what works best for them. It might not even
be the same for everyone.

My concern is that machine generated output ends up verbatim, or largely verbatim in
issue trackers, and pull requests. As I outlined in the statement with Julian, to both of
us (and this might be very different for others!) the participation in open source projects
is valuable due to the human collaboration, which specifically includes the fact that
humans end up getting into a conversation about a change.

Reading a completely machine (auto)generated blurb in that conversation feels, to some,
rather offensive, and may look like pure noise. Whether there was a human acting as
conduit to a LLM or not is fairly irrelevant to that.

This becomes substantially different (at least to me), if someone (potentially) read some
secondary review, and after careful review of that, incorporated parts, of it into their own
review. Sure maybe his own review, and the secondary might coincide, maybe even
largely. But they would need to make these their own, verify the claims, ... and then
post their review. This is a very subtle line, and fairly hard to properly articulate I
believe.

Let me try to bring up two common examples (CodeRabbit, and CoPilot) on GitHub.
PR's at that point looks like:

[some comment by a human]
[ lots of ]
[ noise ]
[ by ]
[ some ]
[ machine(s) ]
[ maybe another comment by a human ]
...

The (human) signal to noise ratio goes down fast.

Now I do personally like to just ask CoPilot for a review of the code (on my repos),
just as a quick cross validation.  My personal preference would be that this creates
comments only visible to me. But that's not how GH works, so I end up requesting
a CoPilot review and go through the findings, then hide/delete all the comments from
copilot to get rid of the noise, such that the PR discussion can focus on the human
collaboration and not the tool noise interleaved.

CodeRabbit, and other tools also have this annoying habit of posting information
as comments, which makes them very disruptive (imo) to the human discourse
around a PR.  If GH offered a better way to attach metadata or a separate track
for these kinds of information, I might feel different about this.

This (for me!) is the reason, why I'm very skeptical of automatically added data into
the human conversation around a MR on gitlab am, and why I am suggesting to
keep this in a second medium (e.g. email, or external web interface) completely 
outside of the lane of human discussion around any PR.

If we all were using git as intended :D, via email, this would be rather simple, one
could simply forward the patch emails to a service email address, that would respond
in private with the annotated LLM review. Similarly how one would facilitate a private
review of a patch from a coworker, before sending ones own review to the shared
mailing list.

I hope this somewhat clarifies where _I_ stand on this topic, and my thinking around
it.

Best,
  Moritz

On Fri, 31 Jul 2026 at 00:14, Simon Jakobi <simon.jakobi@gmail.com> wrote:
Moritz,

let me loop in the mailing list again.

I understand that my service really doesn't have the value of a proper
human code review. However, judging from my private trial runs, it
does seem to catch bugs that neither the testsuite nor human reviewers
caught. MR authors could of course use LLMs themselves to find the
same bugs, but this doesn't seem to be widespread practice yet.
Potential reasons for this: cost or lack of access for other reasons,
insufficient quality of the review output.

I also agree that there is a potential risk that my service could
reduce the current human review practice. My suggestion would be to
make a small survey in three months or so where we evaluate how my
service affects this project, including the social interactions.

Potential tweaks on how to use this service:
* MR authors can request a private review for their draft MRs in order
to catch issues before human reviewers spend time on them.
* Only once the "human review" has concluded, we do a second GlaHiCo
run, and the result is shared publicly on the MR.

Does this sound better to you?

Best,
Simon

Am Do., 30. Juli 2026 um 02:38 Uhr schrieb Moritz Angermann
<moritz.angermann@gmail.com>:
>
> Hi Simon,
>
> I want to highlight the following:
>>
>>
>> 1. The information in the review is intended for all people who are involved in an MR: Both authors and reviewers.
>
>
> I personally find this to be straddling the line where we force exclusively LLM generated content on others and it starts to blur the line between where the human is.
>
> If a human receives (on request) a LLM review, treats it as probabilistic text output, and reviews the review for potential useful highlight, and then selectively makes those highlights their own before fixing them themselves or discussing them with others has a qualitatively difference to “look at the LLM generated review at <url>.” The latter one (at least to me), comes across as lacking respect towards others.
>
> If you use a tool to perform a review, I don’t really care, this is your agency, your freedom to do as you please, but if you pass this off to me as: here look at this output an llm generated, instead of reading the output making it your own and then passing it on as your own with your own comments, … who is it with whom I interact?
> Why don’t I instead just ask the llm myself?
>
>
> Now if everyone agrees that they want to collectively ask a llm, sure.  Even though I do use LLMs, I wouldn’t want unedited llm output pass on to me. There is no value in that to me; I can do that myself. I want to interact with humans and their ideas, intellect and curiosity.
>
> It’s the same fundamental conflict I have wirh coderabbit and copilot reviews. I find them intellectually exceptionally boring. Distracting even. And having them in PR discussions (especially as comments like on GitHub), I feel myself losing interest extremely quickly. This is distinctly different from me using a LLM on the side as assertive tool to support my understanding, and form my opinion (and maybe word choice) when writing a comment.
>
> My primary concern is that while I’m perfectly ok with using LLMs, this opens the door towards verbatim llm output as a normalized way to interact with others; something which I oppose.
>
> Best,
>   Moritz
>
>>
>> Am Mo., 27. Juli 2026 um 19:38 Uhr schrieb Simon Jakobi
>> <simon.jakobi@gmail.com>:
>> >
>> > Hi devs!
>> >
>> > Inspired by the recent discussion on the LLM policy, I'd like to try
>> > introducing LLM-generated code reviews in GHC, as an opt-in service.
>> >
>> > My motivation:
>> >
>> > * Catch issues before human reviewers spend time on them.
>> > * Reduce the number of bugs merged.
>> >
>> >
>> > What this is — and what it isn't
>> > --------------------------------
>> >
>> > This is tool output: requested on the MR, produced by a model, posted verbatim
>> > and clearly marked as machine-generated. This is not a review in the sense of
>> > draft LLM policy.
>> >
>> > I'm not vouching for any of it. Think of it as a linter the author could have
>> > run themselves, except on my LLM budget. Nobody is required to act on
>> > it even read it.
>> >
>> > I mention this explicitly because the policy asks reviewers to take full
>> > responsibility for each line of their review. I can't do that here — it would
>> > mean checking every finding myself, which defeats the purpose and would
>> > restrict the service to areas of the compiler I already know well.
>> >
>> >
>> > How to request one
>> > ------------------
>> >
>> > * Ping me on the MR with a comment like "@sjakobi llm-review please".
>> >   * Feel free to request particular aspects the review should cover.
>> > * Authors can request a review for their own MRs. Maintainers can request one
>> >   on any MR.
>> > * If I have enough usage left on my plan, I'll confirm that I'm on it.
>> > * I'll then prompt the model, and post the result on the MR.
>> > * Feedback on the quality of the review is of course very welcome.
>> >
>> >
>> > What I promise
>> > --------------
>> >
>> > * Every review is labelled as machine-generated, with model, effort level and
>> >   the exact prompt used.
>> > * Reviews are posted verbatim. I don't edit or curate them, so what you see is
>> >   exactly what the model produced.
>> > * Only the public MR diff and public repository context go to the model.
>> >   Nothing else.
>> > * I'll keep working on the prompts and the choice of model to improve the
>> >   signal-to-noise ratio.
>> > * If the signal-to-noise ratio stays inacceptably bad, we can simply
>> > stop this service.
>> >
>> >
>> > Review coverage
>> > ---------------
>> >
>> > I'll probably start with a fairly basic review prompt, primarily aimed at
>> > correctness issues.
>> >
>> > Possible extensions:
>> >
>> > * Documentation consistency: check that documentation stays consistent,
>> >   including Notes elsewhere that reference the changed code.
>> > * Performance: check for performance issues — potentially including checking
>> >   the -ddump-simpl output for the changed code for unnecessary
>> >   allocation etc.
>> > * Feel free to suggest anything else!
>> >
>> >
>> > Format
>> > ------
>> >
>> > Initially a single comment per review. Once we reach a good signal-to-noise
>> > ratio, we can consider inline comments on the diff.
>> >
>> >
>> > Caveats
>> > -------
>> >
>> > This is a volunteer service I intend to provide in my spare time. If I'm AFK
>> > or on vacation, reviews will take longer. If I can't keep up with requests, I
>> > will prioritise which MRs I run. I might take a break entirely.
>> >
>> >
>> > Getting involved
>> > ----------------
>> >
>> > Is anyone interested in joining this effort? If so we could form a team
>> > (@llm-reviewers?), and share prompting techniques, model choices, etc.
>> >
>> > If the service turns out to be genuinely useful, we can look into
>> > automating it and – if necessary – possible funding.
>> >
>> >
>> > Cheers,
>> > Simon
>> _______________________________________________
>> ghc-devs mailing list -- ghc-devs@haskell.org
>> To unsubscribe send an email to ghc-devs-leave@haskell.org