Hi Iavor, some kind of batching might make sense when there are multiple MRs in the same area of the codebase, but that's not very typical. However we can already profit from caching, e.g. when several verifier agents look at findings in the same area. Luckily it does seem that the models that I've been using are already trained on GHC's codebase and are therefore quite capable at navigating it. Cheers, Simon Am Di., 28. Juli 2026 um 09:47 Uhr schrieb Iavor Diatchki <iavor.diatchki@gmail.com>:
Hello,
I am by no means an expert on this, but doing it from scratch for every review might indeed get expensive, as the agent will have to "relearn" the code base each time. It might be worth investigating some sort of design where you can batch the reviews as caching should help significantly.
Cheers, Iavor
On Tue, Jul 28, 2026, 10:15 AM Andreas Klebinger via ghc-devs <ghc-devs@haskell.org> wrote:
Thanks Simon.
I see they rely on a series of fairly specific prompts related to various parts of the review. Some of them encoding quite a bit of domain knowledge.
So there might be something to this approach. At least I can imagine this reducing false positives to a rate where this becomes more valuable to use by default. I do wonder how expensive this is to run at that point.
I might look further into this. Thanks!
On 28/07/2026 01:09, Simon Jakobi wrote:
Hi Andreas!
Regarding Sashiko, I think the README is quite informative: https://github.com/sashiko-dev/sashiko#readme If by "workflow" you mean how developers and maintainers interact with it, it's apparently mostly via mailing lists: https://github.com/sashiko-dev/sashiko/blob/main/MAINTAINERS_GUIDE.md.
https://lore.kernel.org/sashiko-reviews/ lists the (some?) review threads. But the actual reviews are easier to view at https://sashiko.dev/. (At the bottom of each review page, there's a link "View Raw Log", where one can read the prompts and the internal communication of the "subagents" producing the review.)
The claimed false positive rate around 20% sounds pretty good. Initial findings have to pass through some adversarial filtering (involving a "false positive guide" like https://github.com/sashiko-dev/sashiko/blob/main/third_party/prompts/kernel/...).
But it's quite clear that a lot of work went into this system. To adapt it for GHC would be a pretty large task. A mini version for GHC might be more realistic. I think the largest task would be to create a good false positive guide.
As for my "service", I may consider the side-channel option.
Cheers, Simon
Am Mo., 27. Juli 2026 um 23:18 Uhr schrieb Andreas Klebinger via ghc-devs <ghc-devs@haskell.org>:
The noise is mostly inherent to the models. There is no good way to "work on it to fix it" imo.
They miss identify things as bugs due to missing context or simply a lack of understanding fairly frequently. For example I used Opus 5 to audit some GHC code for a specific kind of bug today. It flagged up:
* A "bug" that can only happen if there are broken preconditions that are already checked by lint. * A "bug" that it flagged but marked safe as "only due to" a core invariant of GHCs register allocator design. * Four "Benign issues" that are all correct and pointless noise. * One real bug that I already knew about. (I was just doing a sanity check).
Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts to work through those false reports. But someone can easily spend an hour or much longer just working through those false positives.
If someone who isn't very familiar with the NCG reads through only the LLM output but not the actual code their understanding of the NCG might actually be worse than before. Because they will remember hallucinated reasoning about how things work that is embedded in between those flagged issues.
It can still be valuable to a reviewer that is aware of those circumstances. So I think people who want to try this should take you up on it. But unless one does non-trivial post processing in the prompt like spelling out invariants the model ignored, certain design decisions etc. that require domain knowledge the reviews are typically not worth putting on a MR.
I do find your offer very generous! But for those reasons I think its not very useful to embed these artefacts directly on gitlab. As others mentioned I think performing this experiment via some side channel seems fine to me though.
I would also be interested to hear what the workflow for the review tool the kernel uses is in case you have a good writeup readily at hand.
Andreas
On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote: > I am strongly opposed to this. I also don't see the value in > it. If you want to ask an LLM to review your code, you can > probably do this much quicker by just asking one. Today you > are never more than a few clicks away from a text box that > will put you in direct contact with one. > > If you want to maintain a shared repository of common prompts, > skills, guidance etc for LLMs regarding work on GHC, I have > no objection to this (I would prefer to only have documentation > and prose targetting humans in the main GHC repository though). > > In fact I would like to propose a policy in the exact opposite > direction: > > Disallow _purely autonomous_ systems from posting on the GHC > Gitlab, unless such a system is cleared in advance by the > administrators of the instance and/or contributors to GHC. > > What this won't disallow: > > * Getting an LLM to isolate and create a minimal reproducer > or bug description and posting this. > > * Getting an LLM to propose a patch or fix for a particular > issue > > * Asking an LLM about the details of how some particular > external system works, and then posting the results. > > * Asking an LLM to catalogue a bunch of related issues and > extract a common pattern > > * etc. > > What this will disallow: > > Autonomous systems without humans in the loop writing and > interacting with the gitlab. > > My reasons > > 1. Code review is a tool for building a shared understanding > and a common mental model of the codebase and problem > domain. An LLM model cannot do that, or even if you believe > it can, this internal model is lost as soon as that > particular session is garbage collected. At best you can try > to reconstruct some approximation of this model later by > feeding the same inputs into a fresh session on a probably > different model. > > 2. I would find it a very sad if the GHC Gitlab became a > place where bots talk to other bots who act upon and those > comments to make addtional changes in response to this in > an uncontrolled feedback loop. I predict that in the limit > any such feedback loop becomes quickly detached from the > original problem and devolves into a lot of noise totally > divorced from any real human concerns. > > We can already see examples of how this goes wrong **today** > Take a look at a prominent example of a codebase that went > "all in" on mostly unsupervised autonomous LLM development, > the javascript runtime Bun: https://github.com/oven-sh/bun/pulls > > It would kill any motivation I would have to contribute to > GHC and collaborate with others if the merge request queue > looked anything approximating that. I use the gitlab to > talk to and collaborate with actual humans, who I hope will > actually engage their brains and help get us to a better > and more corret mental model of the space. > > If I wanted to talk to an LLM, I have countless other > options for ways in which to do that. > > 3. Prompt injection. Gitlab comments are an untrusted > input source and LLMs fundamentally cannot discriminate > between instructions and data. You can imagine prompt > injection attacks where hidden text in a gitlab comment, > attached bug reproducer, link etc. gets the model to do > various destructive things, like spamming the gitlab, > editing/deleting your prior comments, mining bitcoin on the > CI runners, attempting to inject malicious code into the > compiler, testcases, reproducers etc. > > That doesn't mean that there isn't value for LLM assisted > autonomous processes on the gitlab, given that they are > desiged properly keeping in mind the concerns about human > centeredness, resource usage and prompt injection. > > Here are some examples of LLM assisted system/bots that > could actually be useful if audited and pre-approved with > the community: > > * A bot that automatically bisects regressions and identifies > the change responsible for it. > In principle this could be done with a purely deterministic > script, but often you run into problems with the toolchain > or commits not being individually buildable etc that require > some adjustements on the fly. > > * A bot that automatically minimises reproducers and posts > the results after the testcase can been verified on a deterministic > system > > On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote: >> Hi devs! >> >> Inspired by the recent discussion on the LLM policy, I'd like to try >> introducing LLM-generated code reviews in GHC, as an opt-in service. >> >> My motivation: >> >> * Catch issues before human reviewers spend time on them. >> * Reduce the number of bugs merged. >> >> >> What this is — and what it isn't >> -------------------------------- >> >> This is tool output: requested on the MR, produced by a model, posted verbatim >> and clearly marked as machine-generated. This is not a review in the sense of >> draft LLM policy. >> >> I'm not vouching for any of it. Think of it as a linter the author could have >> run themselves, except on my LLM budget. Nobody is required to act on >> it even read it. >> >> I mention this explicitly because the policy asks reviewers to take full >> responsibility for each line of their review. I can't do that here — it would >> mean checking every finding myself, which defeats the purpose and would >> restrict the service to areas of the compiler I already know well. >> >> >> How to request one >> ------------------ >> >> * Ping me on the MR with a comment like "@sjakobi llm-review please". >> * Feel free to request particular aspects the review should cover. >> * Authors can request a review for their own MRs. Maintainers can request one >> on any MR. >> * If I have enough usage left on my plan, I'll confirm that I'm on it. >> * I'll then prompt the model, and post the result on the MR. >> * Feedback on the quality of the review is of course very welcome. >> >> >> What I promise >> -------------- >> >> * Every review is labelled as machine-generated, with model, effort level and >> the exact prompt used. >> * Reviews are posted verbatim. I don't edit or curate them, so what you see is >> exactly what the model produced. >> * Only the public MR diff and public repository context go to the model. >> Nothing else. >> * I'll keep working on the prompts and the choice of model to improve the >> signal-to-noise ratio. >> * If the signal-to-noise ratio stays inacceptably bad, we can simply >> stop this service. >> >> >> Review coverage >> --------------- >> >> I'll probably start with a fairly basic review prompt, primarily aimed at >> correctness issues. >> >> Possible extensions: >> >> * Documentation consistency: check that documentation stays consistent, >> including Notes elsewhere that reference the changed code. >> * Performance: check for performance issues — potentially including checking >> the -ddump-simpl output for the changed code for unnecessary >> allocation etc. >> * Feel free to suggest anything else! >> >> >> Format >> ------ >> >> Initially a single comment per review. Once we reach a good signal-to-noise >> ratio, we can consider inline comments on the diff. >> >> >> Caveats >> ------- >> >> This is a volunteer service I intend to provide in my spare time. If I'm AFK >> or on vacation, reviews will take longer. If I can't keep up with requests, I >> will prioritise which MRs I run. I might take a break entirely. >> >> >> Getting involved >> ---------------- >> >> Is anyone interested in joining this effort? If so we could form a team >> (@llm-reviewers?), and share prompting techniques, model choices, etc. >> >> If the service turns out to be genuinely useful, we can look into >> automating it and – if necessary – possible funding. >> >> >> Cheers, >> Simon >> _______________________________________________ >> ghc-devs mailing list -- ghc-devs@haskell.org >> To unsubscribe send an email to ghc-devs-leave@haskell.org > _______________________________________________ > ghc-devs mailing list -- ghc-devs@haskell.org > To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org