Moritz,
let me loop in the mailing list again.
I understand that my service really doesn't have the value of a proper
human code review. However, judging from my private trial runs, it
does seem to catch bugs that neither the testsuite nor human reviewers
caught. MR authors could of course use LLMs themselves to find the
same bugs, but this doesn't seem to be widespread practice yet.
Potential reasons for this: cost or lack of access for other reasons,
insufficient quality of the review output.
I also agree that there is a potential risk that my service could
reduce the current human review practice. My suggestion would be to
make a small survey in three months or so where we evaluate how my
service affects this project, including the social interactions.
Potential tweaks on how to use this service:
* MR authors can request a private review for their draft MRs in order
to catch issues before human reviewers spend time on them.
* Only once the "human review" has concluded, we do a second GlaHiCo
run, and the result is shared publicly on the MR.
Does this sound better to you?
Best,
Simon
Am Do., 30. Juli 2026 um 02:38 Uhr schrieb Moritz Angermann
<moritz.angermann(a)gmail.com>:
>
> Hi Simon,
>
> I want to highlight the following:
>>
>>
>> 1. The information in the review is intended for all people who are involved in an MR: Both authors and reviewers.
>
>
> I personally find this to be straddling the line where we force exclusively LLM generated content on others and it starts to blur the line between where the human is.
>
> If a human receives (on request) a LLM review, treats it as probabilistic text output, and reviews the review for potential useful highlight, and then selectively makes those highlights their own before fixing them themselves or discussing them with others has a qualitatively difference to “look at the LLM generated review at <url>.” The latter one (at least to me), comes across as lacking respect towards others.
>
> If you use a tool to perform a review, I don’t really care, this is your agency, your freedom to do as you please, but if you pass this off to me as: here look at this output an llm generated, instead of reading the output making it your own and then passing it on as your own with your own comments, … who is it with whom I interact?
> Why don’t I instead just ask the llm myself?
>
>
> Now if everyone agrees that they want to collectively ask a llm, sure. Even though I do use LLMs, I wouldn’t want unedited llm output pass on to me. There is no value in that to me; I can do that myself. I want to interact with humans and their ideas, intellect and curiosity.
>
> It’s the same fundamental conflict I have wirh coderabbit and copilot reviews. I find them intellectually exceptionally boring. Distracting even. And having them in PR discussions (especially as comments like on GitHub), I feel myself losing interest extremely quickly. This is distinctly different from me using a LLM on the side as assertive tool to support my understanding, and form my opinion (and maybe word choice) when writing a comment.
>
> My primary concern is that while I’m perfectly ok with using LLMs, this opens the door towards verbatim llm output as a normalized way to interact with others; something which I oppose.
>
> Best,
> Moritz
>
>>
>> Am Mo., 27. Juli 2026 um 19:38 Uhr schrieb Simon Jakobi
>> <simon.jakobi(a)gmail.com>:
>> >
>> > Hi devs!
>> >
>> > Inspired by the recent discussion on the LLM policy, I'd like to try
>> > introducing LLM-generated code reviews in GHC, as an opt-in service.
>> >
>> > My motivation:
>> >
>> > * Catch issues before human reviewers spend time on them.
>> > * Reduce the number of bugs merged.
>> >
>> >
>> > What this is — and what it isn't
>> > --------------------------------
>> >
>> > This is tool output: requested on the MR, produced by a model, posted verbatim
>> > and clearly marked as machine-generated. This is not a review in the sense of
>> > draft LLM policy.
>> >
>> > I'm not vouching for any of it. Think of it as a linter the author could have
>> > run themselves, except on my LLM budget. Nobody is required to act on
>> > it even read it.
>> >
>> > I mention this explicitly because the policy asks reviewers to take full
>> > responsibility for each line of their review. I can't do that here — it would
>> > mean checking every finding myself, which defeats the purpose and would
>> > restrict the service to areas of the compiler I already know well.
>> >
>> >
>> > How to request one
>> > ------------------
>> >
>> > * Ping me on the MR with a comment like "@sjakobi llm-review please".
>> > * Feel free to request particular aspects the review should cover.
>> > * Authors can request a review for their own MRs. Maintainers can request one
>> > on any MR.
>> > * If I have enough usage left on my plan, I'll confirm that I'm on it.
>> > * I'll then prompt the model, and post the result on the MR.
>> > * Feedback on the quality of the review is of course very welcome.
>> >
>> >
>> > What I promise
>> > --------------
>> >
>> > * Every review is labelled as machine-generated, with model, effort level and
>> > the exact prompt used.
>> > * Reviews are posted verbatim. I don't edit or curate them, so what you see is
>> > exactly what the model produced.
>> > * Only the public MR diff and public repository context go to the model.
>> > Nothing else.
>> > * I'll keep working on the prompts and the choice of model to improve the
>> > signal-to-noise ratio.
>> > * If the signal-to-noise ratio stays inacceptably bad, we can simply
>> > stop this service.
>> >
>> >
>> > Review coverage
>> > ---------------
>> >
>> > I'll probably start with a fairly basic review prompt, primarily aimed at
>> > correctness issues.
>> >
>> > Possible extensions:
>> >
>> > * Documentation consistency: check that documentation stays consistent,
>> > including Notes elsewhere that reference the changed code.
>> > * Performance: check for performance issues — potentially including checking
>> > the -ddump-simpl output for the changed code for unnecessary
>> > allocation etc.
>> > * Feel free to suggest anything else!
>> >
>> >
>> > Format
>> > ------
>> >
>> > Initially a single comment per review. Once we reach a good signal-to-noise
>> > ratio, we can consider inline comments on the diff.
>> >
>> >
>> > Caveats
>> > -------
>> >
>> > This is a volunteer service I intend to provide in my spare time. If I'm AFK
>> > or on vacation, reviews will take longer. If I can't keep up with requests, I
>> > will prioritise which MRs I run. I might take a break entirely.
>> >
>> >
>> > Getting involved
>> > ----------------
>> >
>> > Is anyone interested in joining this effort? If so we could form a team
>> > (@llm-reviewers?), and share prompting techniques, model choices, etc.
>> >
>> > If the service turns out to be genuinely useful, we can look into
>> > automating it and – if necessary – possible funding.
>> >
>> >
>> > Cheers,
>> > Simon
>> _______________________________________________
>> ghc-devs mailing list -- ghc-devs(a)haskell.org
>> To unsubscribe send an email to ghc-devs-leave(a)haskell.org
