Thanks Simon.

I see they rely on a series of fairly specific prompts related to
various parts of the
review. Some of them encoding quite a bit of domain knowledge.

So there might be something to this approach. At least I can imagine
this reducing false positives
to a rate where this becomes more valuable to use by default. I do
wonder how expensive this
is to run at that point.

I might look further into this. Thanks!


On 28/07/2026 01:09, Simon Jakobi wrote:
> Hi Andreas!
>
> Regarding Sashiko, I think the README is quite informative:
> https://github.com/sashiko-dev/sashiko#readme
> If by "workflow" you mean how developers and maintainers interact with
> it, it's apparently mostly via mailing lists:
> https://github.com/sashiko-dev/sashiko/blob/main/MAINTAINERS_GUIDE.md.
>
> https://lore.kernel.org/sashiko-reviews/ lists the (some?) review
> threads. But the actual reviews are easier to view at
> https://sashiko.dev/. (At the bottom of each review page, there's a
> link "View Raw Log", where one can read the prompts and the internal
> communication of the "subagents" producing the review.)
>
> The claimed false positive rate around 20% sounds pretty good. Initial
> findings have to pass through some adversarial filtering
> (involving a "false positive guide" like
> https://github.com/sashiko-dev/sashiko/blob/main/third_party/prompts/kernel/false-positive-guide.md).
>
> But it's quite clear that a lot of work went into this system. To
> adapt it for GHC would be a pretty large task. A mini version for GHC
> might be more realistic. I think the largest task would be to create a
> good false positive guide.
>
>
> As for my "service", I may consider the side-channel option.
>
> Cheers,
> Simon
>
> Am Mo., 27. Juli 2026 um 23:18 Uhr schrieb Andreas Klebinger via
> ghc-devs <ghc-devs@haskell.org>:
>> The noise is mostly inherent to the models. There is no good way to
>> "work on it to fix it" imo.
>>
>> They miss identify things as bugs due to missing context or simply a
>> lack of understanding fairly frequently.
>> For example I used Opus 5 to audit some GHC code for a specific kind of
>> bug today. It flagged up:
>>
>> * A "bug" that can only happen if there are broken preconditions that
>> are already checked by lint.
>> * A "bug" that it flagged but marked safe as "only due to" a core
>> invariant of GHCs register allocator design.
>> * Four "Benign issues" that are all correct and pointless noise.
>> * One real bug that I already knew about. (I was just doing a sanity check).
>>
>> Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts
>> to work through those false reports.
>> But someone can easily spend an hour or much longer just working through
>> those false positives.
>>
>> If someone who isn't very familiar with the NCG reads through only the
>> LLM output but not the actual code their understanding
>> of the NCG might actually be worse than before. Because they will
>> remember hallucinated reasoning about how things work that
>> is embedded in between those flagged issues.
>>
>> It can still be valuable to a reviewer that is aware of those
>> circumstances. So I think people who want to try this should take
>> you up on it. But unless one does non-trivial post processing in the
>> prompt like spelling out invariants the model ignored,
>> certain design decisions etc. that require domain knowledge the reviews
>> are typically not worth putting on a MR.
>>
>> I do find your offer very generous! But for those reasons I think its
>> not very useful to embed these artefacts directly on gitlab.
>> As others mentioned I think performing this experiment via some side
>> channel seems fine to me though.
>>
>> I would also be interested to hear what the workflow for the review tool
>> the kernel uses is in case you have a good writeup readily
>> at hand.
>>
>> Andreas
>>
>> On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
>>> Hi Cheng,
>>>
>>> Thanks for your kind reply! :)
>>>
>>> Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
>>>> i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
>>>>
>>>> thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly.
>>> If the problem is with visual noise, I don't mind putting the blobs
>>> into `<details>` tags so they don't take up as much space.
>>>
>>> However if the goal is to reduce maintainers' workload, I think the MR
>>> is still the best place, so maintainers can use the presence of the
>>> LLM-generated review as a signal.
>>>
>>> A decent amount of visibility would probably also help to improve the
>>> LLM review system more quickly.
>>>
>>>> i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches.
>>> If the signal-to-noise ratio is indeed so bad, I'm ready to improve
>>> the system until the ratio is more acceptable. It would be great to
>>> have some collaborators in this, but I'll also give a try on my own.
>>>
>>> Cheers,
>>> Simon
>>>
>>>> cheers,
>>>> cheng
>>>>
>>>> On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote:
>>>>> I am strongly opposed to this. I also don't see the value in
>>>>> it. If you want to ask an LLM to review your code, you can
>>>>> probably do this much quicker by just asking one. Today you
>>>>> are never more than a few clicks away from a text box that
>>>>> will put you in direct contact with one.
>>>>>
>>>>> If you want to maintain a shared repository of common prompts,
>>>>> skills, guidance etc for LLMs regarding work on GHC, I have
>>>>> no objection to this (I would prefer to only have documentation
>>>>> and prose targetting humans in the main GHC repository though).
>>>>>
>>>>> In fact I would like to propose a policy in the exact opposite
>>>>> direction:
>>>>>
>>>>> Disallow _purely autonomous_ systems from posting on the GHC
>>>>> Gitlab, unless such a system is cleared in advance by the
>>>>> administrators of the instance and/or contributors to GHC.
>>>>>
>>>>> What this won't disallow:
>>>>>
>>>>> * Getting an LLM to isolate and create a minimal reproducer
>>>>> or bug description and posting this.
>>>>>
>>>>> * Getting an LLM to propose a patch or fix for a particular
>>>>> issue
>>>>>
>>>>> * Asking an LLM about the details of how some particular
>>>>> external system works, and then posting the results.
>>>>>
>>>>> * Asking an LLM to catalogue a bunch of related issues and
>>>>> extract a common pattern
>>>>>
>>>>> * etc.
>>>>>
>>>>> What this will disallow:
>>>>>
>>>>> Autonomous systems without humans in the loop writing and
>>>>> interacting with the gitlab.
>>>>>
>>>>> My reasons
>>>>>
>>>>> 1. Code review is a tool for building a shared understanding
>>>>> and a common mental model of the codebase and problem
>>>>> domain. An LLM model cannot do that, or even if you believe
>>>>> it can, this internal model is lost as soon as that
>>>>> particular session is garbage collected. At best you can try
>>>>> to reconstruct some approximation of this model later by
>>>>> feeding the same inputs into a fresh session on a probably
>>>>> different model.
>>>>>
>>>>> 2. I would find it a very sad if the GHC Gitlab became a
>>>>> place where bots talk to other bots who act upon and those
>>>>> comments to make addtional changes in response to this in
>>>>> an uncontrolled feedback loop. I predict that in the limit
>>>>> any such feedback loop becomes quickly detached from the
>>>>> original problem and devolves into a lot of noise totally
>>>>> divorced from any real human concerns.
>>>>>
>>>>> We can already see examples of how this goes wrong **today**
>>>>> Take a look at a prominent example of a codebase that went
>>>>> "all in" on mostly unsupervised autonomous LLM development,
>>>>> the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
>>>>>
>>>>> It would kill any motivation I would have to contribute to
>>>>> GHC and collaborate with others if the merge request queue
>>>>> looked anything approximating that. I use the gitlab to
>>>>> talk to and collaborate with actual humans, who I hope will
>>>>> actually engage their brains and help get us to a better
>>>>> and more corret mental model of the space.
>>>>>
>>>>> If I wanted to talk to an LLM, I have countless other
>>>>> options for ways in which to do that.
>>>>>
>>>>> 3. Prompt injection. Gitlab comments are an untrusted
>>>>> input source and LLMs fundamentally cannot discriminate
>>>>> between instructions and data. You can imagine prompt
>>>>> injection attacks where hidden text in a gitlab comment,
>>>>> attached bug reproducer, link etc. gets the model to do
>>>>> various destructive things, like spamming the gitlab,
>>>>> editing/deleting your prior comments, mining bitcoin on the
>>>>> CI runners, attempting to inject malicious code into the
>>>>> compiler, testcases, reproducers etc.
>>>>>
>>>>> That doesn't mean that there isn't value for LLM assisted
>>>>> autonomous processes on the gitlab, given that they are
>>>>> desiged properly keeping in mind the concerns about human
>>>>> centeredness, resource usage and prompt injection.
>>>>>
>>>>> Here are some examples of LLM assisted system/bots that
>>>>> could actually be useful if audited and pre-approved with
>>>>> the community:
>>>>>
>>>>> * A bot that automatically bisects regressions and identifies
>>>>> the change responsible for it.
>>>>> In principle this could be done with a purely deterministic
>>>>> script, but often you run into problems with the toolchain
>>>>> or commits not being individually buildable etc that require
>>>>> some adjustements on the fly.
>>>>>
>>>>> * A bot that automatically minimises reproducers and posts
>>>>> the results after the testcase can been verified on a deterministic
>>>>> system
>>>>>
>>>>> On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
>>>>>> Hi devs!
>>>>>>
>>>>>> Inspired by the recent discussion on the LLM policy, I'd like to try
>>>>>> introducing LLM-generated code reviews in GHC, as an opt-in service.
>>>>>>
>>>>>> My motivation:
>>>>>>
>>>>>> * Catch issues before human reviewers spend time on them.
>>>>>> * Reduce the number of bugs merged.
>>>>>>
>>>>>>
>>>>>> What this is — and what it isn't
>>>>>> --------------------------------
>>>>>>
>>>>>> This is tool output: requested on the MR, produced by a model, posted verbatim
>>>>>> and clearly marked as machine-generated. This is not a review in the sense of
>>>>>> draft LLM policy.
>>>>>>
>>>>>> I'm not vouching for any of it. Think of it as a linter the author could have
>>>>>> run themselves, except on my LLM budget. Nobody is required to act on
>>>>>> it even read it.
>>>>>>
>>>>>> I mention this explicitly because the policy asks reviewers to take full
>>>>>> responsibility for each line of their review. I can't do that here — it would
>>>>>> mean checking every finding myself, which defeats the purpose and would
>>>>>> restrict the service to areas of the compiler I already know well.
>>>>>>
>>>>>>
>>>>>> How to request one
>>>>>> ------------------
>>>>>>
>>>>>> * Ping me on the MR with a comment like "@sjakobi llm-review please".
>>>>>>    * Feel free to request particular aspects the review should cover.
>>>>>> * Authors can request a review for their own MRs. Maintainers can request one
>>>>>>    on any MR.
>>>>>> * If I have enough usage left on my plan, I'll confirm that I'm on it.
>>>>>> * I'll then prompt the model, and post the result on the MR.
>>>>>> * Feedback on the quality of the review is of course very welcome.
>>>>>>
>>>>>>
>>>>>> What I promise
>>>>>> --------------
>>>>>>
>>>>>> * Every review is labelled as machine-generated, with model, effort level and
>>>>>>    the exact prompt used.
>>>>>> * Reviews are posted verbatim. I don't edit or curate them, so what you see is
>>>>>>    exactly what the model produced.
>>>>>> * Only the public MR diff and public repository context go to the model.
>>>>>>    Nothing else.
>>>>>> * I'll keep working on the prompts and the choice of model to improve the
>>>>>>    signal-to-noise ratio.
>>>>>> * If the signal-to-noise ratio stays inacceptably bad, we can simply
>>>>>> stop this service.
>>>>>>
>>>>>>
>>>>>> Review coverage
>>>>>> ---------------
>>>>>>
>>>>>> I'll probably start with a fairly basic review prompt, primarily aimed at
>>>>>> correctness issues.
>>>>>>
>>>>>> Possible extensions:
>>>>>>
>>>>>> * Documentation consistency: check that documentation stays consistent,
>>>>>>    including Notes elsewhere that reference the changed code.
>>>>>> * Performance: check for performance issues — potentially including checking
>>>>>>    the -ddump-simpl output for the changed code for unnecessary
>>>>>>    allocation etc.
>>>>>> * Feel free to suggest anything else!
>>>>>>
>>>>>>
>>>>>> Format
>>>>>> ------
>>>>>>
>>>>>> Initially a single comment per review. Once we reach a good signal-to-noise
>>>>>> ratio, we can consider inline comments on the diff.
>>>>>>
>>>>>>
>>>>>> Caveats
>>>>>> -------
>>>>>>
>>>>>> This is a volunteer service I intend to provide in my spare time. If I'm AFK
>>>>>> or on vacation, reviews will take longer. If I can't keep up with requests, I
>>>>>> will prioritise which MRs I run. I might take a break entirely.
>>>>>>
>>>>>>
>>>>>> Getting involved
>>>>>> ----------------
>>>>>>
>>>>>> Is anyone interested in joining this effort? If so we could form a team
>>>>>> (@llm-reviewers?), and share prompting techniques, model choices, etc.
>>>>>>
>>>>>> If the service turns out to be genuinely useful, we can look into
>>>>>> automating it and – if necessary – possible funding.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Simon
>>>>>> _______________________________________________
>>>>>> ghc-devs mailing list -- ghc-devs@haskell.org
>>>>>> To unsubscribe send an email to ghc-devs-leave@haskell.org
>>>>> _______________________________________________
>>>>> ghc-devs mailing list -- ghc-devs@haskell.org
>>>>> To unsubscribe send an email to ghc-devs-leave@haskell.org
>>> _______________________________________________
>>> ghc-devs mailing list -- ghc-devs@haskell.org
>>> To unsubscribe send an email to ghc-devs-leave@haskell.org
>> _______________________________________________
>> ghc-devs mailing list -- ghc-devs@haskell.org
>> To unsubscribe send an email to ghc-devs-leave@haskell.org
_______________________________________________
ghc-devs mailing list -- ghc-devs@haskell.org
To unsubscribe send an email to ghc-devs-leave@haskell.org