LLM-generated code review as a service
Hi devs! Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service. My motivation: * Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged. What this is — and what it isn't -------------------------------- This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy. I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it. I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well. How to request one ------------------ * Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome. What I promise -------------- * Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service. Review coverage --------------- I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues. Possible extensions: * Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else! Format ------ Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff. Caveats ------- This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely. Getting involved ---------------- Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc. If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding. Cheers, Simon
On Mon, Jul 27, 2026 at 07:38:17PM +0200, Simon Jakobi via ghc-devs wrote:
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
I appreciate this offer! I have been astonished by the bugs that LLM-generated code review has found in my code and the code of my colleagues. This could be a very useful service for GHC devs. Tom
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one. If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though). In fact I would like to propose a policy in the exact opposite direction: Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC. What this won't disallow: * Getting an LLM to isolate and create a minimal reproducer or bug description and posting this. * Getting an LLM to propose a patch or fix for a particular issue * Asking an LLM about the details of how some particular external system works, and then posting the results. * Asking an LLM to catalogue a bunch of related issues and extract a common pattern * etc. What this will disallow: Autonomous systems without humans in the loop writing and interacting with the gitlab. My reasons 1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model. 2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns. We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space. If I wanted to talk to an LLM, I have countless other options for ways in which to do that. 3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc. That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection. Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community: * A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly. * A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so. thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. cheers, cheng On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs < ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Hi Cheng, Thanks for your kind reply! :) Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly.
If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space. However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal. A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches.
If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own. Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
<detail> tag with a big "oh fable found this oh sol found that here's full diagnosis and fix" caption is not a visual noise, it's a cognitive one. a human reviewer, no matter pro or against llms, would have this temptation with one click away, and this is a constant distraction when they spend their own personal time looking at that patch. and regardless of if the reviewer is organic or not, a pre-existing blob of text there will diverge their attention towards what's already been spotted, away from what might not have been spotted yet (deeper architectural issues, something shouldn't be done in the first place, etc etc). again, i don't think the gitlab instance is the right data plane for llm generated reviews, or for "experiments" that you propose. On Mon, Jul 27, 2026 at 10:41 PM Simon Jakobi <simon.jakobi@gmail.com> wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered
auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab
instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly.
If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches.
If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <
ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted
verbatim
and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
I think Simon is offering a free service here. If you don't want to see these reviews for your MRs, then don't request them. I'd surely find them helpful, although I am fortunate enough to have a Claude subscription of my own. I don't think discarding the idea on the grounds that "LLM reviews are too noisy" is sensible. Firstly, linters and warnings are noisy as well, yet they bring our validate pipelines to a halt. This is *much* noisier than an LLM review you requested but now don't want to read for some reason. Secondly, if a contributor requests an LLM "review" that has a false positive (which happens, but far less frequently than true positives!), then either the contributor is able to label it as such or will discuss it with the maintainers. At the end of such a discussion, often, both contributor and maintainer will have learned something. Am Mo., 27. Juli 2026 um 22:54 Uhr schrieb Cheng Shao via ghc-devs < ghc-devs@haskell.org>:
<detail> tag with a big "oh fable found this oh sol found that here's full diagnosis and fix" caption is not a visual noise, it's a cognitive one. a human reviewer, no matter pro or against llms, would have this temptation with one click away, and this is a constant distraction when they spend their own personal time looking at that patch. and regardless of if the reviewer is organic or not, a pre-existing blob of text there will diverge their attention towards what's already been spotted, away from what might not have been spotted yet (deeper architectural issues, something shouldn't be done in the first place, etc etc).
again, i don't think the gitlab instance is the right data plane for llm generated reviews, or for "experiments" that you propose.
On Mon, Jul 27, 2026 at 10:41 PM Simon Jakobi <simon.jakobi@gmail.com> wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered
auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the
gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly.
If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches.
If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <
ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted
verbatim
and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Hi Simon, I think your offer is generous! I am however not a fan adding purely machine generated text to GitLab. I think this could work much better as a hosted service outside of GitLab. I believe what we fundamentally want is to build a community of human interaction, not of machine interaction. Having pages of machine output in the midst of this, is imo undermining the human collaboration and trust and joy to work on such projects. Why contribute if it ends up just being machine generated interactions anyway? That’s I think very emotionally draining. I will not argue against the potential value of having assistive tooling used to read over a proposed change and give suggestions what one might want to look at. In the same sense as I find the proliferation of CodeRabbit and CoPilot reviews on GitHub troublesome. They add a lot of noise and make reading PRs and discussions extremely annoying and tiresome to me. As such, I believe there is value in asking machines to provide suggestions. Maybe they highlight something one has overlooked? But I think this should stay strictly in a lane outside of the human collaboration one. It is imo a private matter to ask for help, not one that should be forced onto others. Therefore, I’d strongly suggest to offer this as a hosted service. Especially as it may provide those who do want to see what a machine might highlight across their contribution, but are not fortunate enough to have access to these tools. Maybe an email offer can work? You offer contributors the opportunity to email you their MR link, and then in private respond to their email with the machine generated review? I think what such an email should contain is that this is a machine generated review and it should be considered suggestive and not at face value. Also that posting it as-is into discussion with others is not permissible behavior? After all (and the policies so far say this as well), we consider the project one of human collaboration not machine collaboration. Created by humans for humans. This were now a lot of words to say: yes, I think such a service can be valuable; but no, I don’t think such a service belongs into a shared medium like gitlab. Best, Moritz On Tue, Jul 28, 2026 at 4:29 AM Sebastian Graf via ghc-devs < ghc-devs@haskell.org> wrote:
I think Simon is offering a free service here. If you don't want to see these reviews for your MRs, then don't request them. I'd surely find them helpful, although I am fortunate enough to have a Claude subscription of my own.
I don't think discarding the idea on the grounds that "LLM reviews are too noisy" is sensible. Firstly, linters and warnings are noisy as well, yet they bring our validate pipelines to a halt. This is *much* noisier than an LLM review you requested but now don't want to read for some reason. Secondly, if a contributor requests an LLM "review" that has a false positive (which happens, but far less frequently than true positives!), then either the contributor is able to label it as such or will discuss it with the maintainers. At the end of such a discussion, often, both contributor and maintainer will have learned something.
Am Mo., 27. Juli 2026 um 22:54 Uhr schrieb Cheng Shao via ghc-devs < ghc-devs@haskell.org>:
<detail> tag with a big "oh fable found this oh sol found that here's full diagnosis and fix" caption is not a visual noise, it's a cognitive one. a human reviewer, no matter pro or against llms, would have this temptation with one click away, and this is a constant distraction when they spend their own personal time looking at that patch. and regardless of if the reviewer is organic or not, a pre-existing blob of text there will diverge their attention towards what's already been spotted, away from what might not have been spotted yet (deeper architectural issues, something shouldn't be done in the first place, etc etc).
again, i don't think the gitlab instance is the right data plane for llm generated reviews, or for "experiments" that you propose.
On Mon, Jul 27, 2026 at 10:41 PM Simon Jakobi <simon.jakobi@gmail.com> wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered
auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the
gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly.
If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches.
If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <
ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model,
posted verbatim
and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
I too think this is a generous offer, and I might well take you up on it. Some thoughts - I agree with others that I don't want a lot of real-estate on my screen taken up with such a review when scrolling through a MR. So *definitely *under <details> or even elsewhere with a link. (I don't mind which, but I suppose the latter might reduce load times.) - I agree that we should not allow LLMs to autonomously affect GHC's GitLab; and indeed Simon is not proposing that. - Is it too distracting to have a three-line comment in an MR saying "LLM review: follow this link"? I don't think so. If you don't want to be distracted, don't follow the link. Would such a short comment really be unacceptable in the MR comment stream? Personally I don't think so. - Although LLMs will get things wrong (thanks Andreas for the actual data point) they are already spookily good and getting better fast. Moreover, detecting even one bug at birth, before it can do damage, has very high value. So I for one am willing to give it a try. If the signal to noise ratio is too low, I'll quickly stop. It's opt-in so there is no 'tax' on anyone else. - A speculative point. I often ask authors for a Note explaining their design. I wonder if the false positives from an LLM review might be a useful driver to improve documentation so that even a stupid LLM could not make that mistake next time. Simon On Mon, 27 Jul 2026 at 22:29, Sebastian Graf via ghc-devs < ghc-devs@haskell.org> wrote:
I think Simon is offering a free service here. If you don't want to see these reviews for your MRs, then don't request them. I'd surely find them helpful, although I am fortunate enough to have a Claude subscription of my own.
I don't think discarding the idea on the grounds that "LLM reviews are too noisy" is sensible. Firstly, linters and warnings are noisy as well, yet they bring our validate pipelines to a halt. This is *much* noisier than an LLM review you requested but now don't want to read for some reason. Secondly, if a contributor requests an LLM "review" that has a false positive (which happens, but far less frequently than true positives!), then either the contributor is able to label it as such or will discuss it with the maintainers. At the end of such a discussion, often, both contributor and maintainer will have learned something.
Am Mo., 27. Juli 2026 um 22:54 Uhr schrieb Cheng Shao via ghc-devs < ghc-devs@haskell.org>:
<detail> tag with a big "oh fable found this oh sol found that here's full diagnosis and fix" caption is not a visual noise, it's a cognitive one. a human reviewer, no matter pro or against llms, would have this temptation with one click away, and this is a constant distraction when they spend their own personal time looking at that patch. and regardless of if the reviewer is organic or not, a pre-existing blob of text there will diverge their attention towards what's already been spotted, away from what might not have been spotted yet (deeper architectural issues, something shouldn't be done in the first place, etc etc).
again, i don't think the gitlab instance is the right data plane for llm generated reviews, or for "experiments" that you propose.
On Mon, Jul 27, 2026 at 10:41 PM Simon Jakobi <simon.jakobi@gmail.com> wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered
auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the
gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly.
If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches.
If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <
ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model,
posted verbatim
and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
The noise is mostly inherent to the models. There is no good way to "work on it to fix it" imo. They miss identify things as bugs due to missing context or simply a lack of understanding fairly frequently. For example I used Opus 5 to audit some GHC code for a specific kind of bug today. It flagged up: * A "bug" that can only happen if there are broken preconditions that are already checked by lint. * A "bug" that it flagged but marked safe as "only due to" a core invariant of GHCs register allocator design. * Four "Benign issues" that are all correct and pointless noise. * One real bug that I already knew about. (I was just doing a sanity check). Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts to work through those false reports. But someone can easily spend an hour or much longer just working through those false positives. If someone who isn't very familiar with the NCG reads through only the LLM output but not the actual code their understanding of the NCG might actually be worse than before. Because they will remember hallucinated reasoning about how things work that is embedded in between those flagged issues. It can still be valuable to a reviewer that is aware of those circumstances. So I think people who want to try this should take you up on it. But unless one does non-trivial post processing in the prompt like spelling out invariants the model ignored, certain design decisions etc. that require domain knowledge the reviews are typically not worth putting on a MR. I do find your offer very generous! But for those reasons I think its not very useful to embed these artefacts directly on gitlab. As others mentioned I think performing this experiment via some side channel seems fine to me though. I would also be interested to hear what the workflow for the review tool the kernel uses is in case you have a good writeup readily at hand. Andreas On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Hi Andreas! Regarding Sashiko, I think the README is quite informative: https://github.com/sashiko-dev/sashiko#readme If by "workflow" you mean how developers and maintainers interact with it, it's apparently mostly via mailing lists: https://github.com/sashiko-dev/sashiko/blob/main/MAINTAINERS_GUIDE.md. https://lore.kernel.org/sashiko-reviews/ lists the (some?) review threads. But the actual reviews are easier to view at https://sashiko.dev/. (At the bottom of each review page, there's a link "View Raw Log", where one can read the prompts and the internal communication of the "subagents" producing the review.) The claimed false positive rate around 20% sounds pretty good. Initial findings have to pass through some adversarial filtering (involving a "false positive guide" like https://github.com/sashiko-dev/sashiko/blob/main/third_party/prompts/kernel/...). But it's quite clear that a lot of work went into this system. To adapt it for GHC would be a pretty large task. A mini version for GHC might be more realistic. I think the largest task would be to create a good false positive guide. As for my "service", I may consider the side-channel option. Cheers, Simon Am Mo., 27. Juli 2026 um 23:18 Uhr schrieb Andreas Klebinger via ghc-devs <ghc-devs@haskell.org>:
The noise is mostly inherent to the models. There is no good way to "work on it to fix it" imo.
They miss identify things as bugs due to missing context or simply a lack of understanding fairly frequently. For example I used Opus 5 to audit some GHC code for a specific kind of bug today. It flagged up:
* A "bug" that can only happen if there are broken preconditions that are already checked by lint. * A "bug" that it flagged but marked safe as "only due to" a core invariant of GHCs register allocator design. * Four "Benign issues" that are all correct and pointless noise. * One real bug that I already knew about. (I was just doing a sanity check).
Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts to work through those false reports. But someone can easily spend an hour or much longer just working through those false positives.
If someone who isn't very familiar with the NCG reads through only the LLM output but not the actual code their understanding of the NCG might actually be worse than before. Because they will remember hallucinated reasoning about how things work that is embedded in between those flagged issues.
It can still be valuable to a reviewer that is aware of those circumstances. So I think people who want to try this should take you up on it. But unless one does non-trivial post processing in the prompt like spelling out invariants the model ignored, certain design decisions etc. that require domain knowledge the reviews are typically not worth putting on a MR.
I do find your offer very generous! But for those reasons I think its not very useful to embed these artefacts directly on gitlab. As others mentioned I think performing this experiment via some side channel seems fine to me though.
I would also be interested to hear what the workflow for the review tool the kernel uses is in case you have a good writeup readily at hand.
Andreas
On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Thanks Simon. I see they rely on a series of fairly specific prompts related to various parts of the review. Some of them encoding quite a bit of domain knowledge. So there might be something to this approach. At least I can imagine this reducing false positives to a rate where this becomes more valuable to use by default. I do wonder how expensive this is to run at that point. I might look further into this. Thanks! On 28/07/2026 01:09, Simon Jakobi wrote:
Hi Andreas!
Regarding Sashiko, I think the README is quite informative: https://github.com/sashiko-dev/sashiko#readme If by "workflow" you mean how developers and maintainers interact with it, it's apparently mostly via mailing lists: https://github.com/sashiko-dev/sashiko/blob/main/MAINTAINERS_GUIDE.md.
https://lore.kernel.org/sashiko-reviews/ lists the (some?) review threads. But the actual reviews are easier to view at https://sashiko.dev/. (At the bottom of each review page, there's a link "View Raw Log", where one can read the prompts and the internal communication of the "subagents" producing the review.)
The claimed false positive rate around 20% sounds pretty good. Initial findings have to pass through some adversarial filtering (involving a "false positive guide" like https://github.com/sashiko-dev/sashiko/blob/main/third_party/prompts/kernel/...).
But it's quite clear that a lot of work went into this system. To adapt it for GHC would be a pretty large task. A mini version for GHC might be more realistic. I think the largest task would be to create a good false positive guide.
As for my "service", I may consider the side-channel option.
Cheers, Simon
Am Mo., 27. Juli 2026 um 23:18 Uhr schrieb Andreas Klebinger via ghc-devs <ghc-devs@haskell.org>:
The noise is mostly inherent to the models. There is no good way to "work on it to fix it" imo.
They miss identify things as bugs due to missing context or simply a lack of understanding fairly frequently. For example I used Opus 5 to audit some GHC code for a specific kind of bug today. It flagged up:
* A "bug" that can only happen if there are broken preconditions that are already checked by lint. * A "bug" that it flagged but marked safe as "only due to" a core invariant of GHCs register allocator design. * Four "Benign issues" that are all correct and pointless noise. * One real bug that I already knew about. (I was just doing a sanity check).
Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts to work through those false reports. But someone can easily spend an hour or much longer just working through those false positives.
If someone who isn't very familiar with the NCG reads through only the LLM output but not the actual code their understanding of the NCG might actually be worse than before. Because they will remember hallucinated reasoning about how things work that is embedded in between those flagged issues.
It can still be valuable to a reviewer that is aware of those circumstances. So I think people who want to try this should take you up on it. But unless one does non-trivial post processing in the prompt like spelling out invariants the model ignored, certain design decisions etc. that require domain knowledge the reviews are typically not worth putting on a MR.
I do find your offer very generous! But for those reasons I think its not very useful to embed these artefacts directly on gitlab. As others mentioned I think performing this experiment via some side channel seems fine to me though.
I would also be interested to hear what the workflow for the review tool the kernel uses is in case you have a good writeup readily at hand.
Andreas
On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Hello, I am by no means an expert on this, but doing it from scratch for every review might indeed get expensive, as the agent will have to "relearn" the code base each time. It might be worth investigating some sort of design where you can batch the reviews as caching should help significantly. Cheers, Iavor On Tue, Jul 28, 2026, 10:15 AM Andreas Klebinger via ghc-devs < ghc-devs@haskell.org> wrote:
Thanks Simon.
I see they rely on a series of fairly specific prompts related to various parts of the review. Some of them encoding quite a bit of domain knowledge.
So there might be something to this approach. At least I can imagine this reducing false positives to a rate where this becomes more valuable to use by default. I do wonder how expensive this is to run at that point.
I might look further into this. Thanks!
On 28/07/2026 01:09, Simon Jakobi wrote:
Hi Andreas!
Regarding Sashiko, I think the README is quite informative: https://github.com/sashiko-dev/sashiko#readme If by "workflow" you mean how developers and maintainers interact with it, it's apparently mostly via mailing lists: https://github.com/sashiko-dev/sashiko/blob/main/MAINTAINERS_GUIDE.md.
https://lore.kernel.org/sashiko-reviews/ lists the (some?) review threads. But the actual reviews are easier to view at https://sashiko.dev/. (At the bottom of each review page, there's a link "View Raw Log", where one can read the prompts and the internal communication of the "subagents" producing the review.)
The claimed false positive rate around 20% sounds pretty good. Initial findings have to pass through some adversarial filtering (involving a "false positive guide" like
https://github.com/sashiko-dev/sashiko/blob/main/third_party/prompts/kernel/... ).
But it's quite clear that a lot of work went into this system. To adapt it for GHC would be a pretty large task. A mini version for GHC might be more realistic. I think the largest task would be to create a good false positive guide.
As for my "service", I may consider the side-channel option.
Cheers, Simon
Am Mo., 27. Juli 2026 um 23:18 Uhr schrieb Andreas Klebinger via ghc-devs <ghc-devs@haskell.org>:
The noise is mostly inherent to the models. There is no good way to "work on it to fix it" imo.
They miss identify things as bugs due to missing context or simply a lack of understanding fairly frequently. For example I used Opus 5 to audit some GHC code for a specific kind of bug today. It flagged up:
* A "bug" that can only happen if there are broken preconditions that are already checked by lint. * A "bug" that it flagged but marked safe as "only due to" a core invariant of GHCs register allocator design. * Four "Benign issues" that are all correct and pointless noise. * One real bug that I already knew about. (I was just doing a sanity
check).
Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts to work through those false reports. But someone can easily spend an hour or much longer just working through those false positives.
If someone who isn't very familiar with the NCG reads through only the LLM output but not the actual code their understanding of the NCG might actually be worse than before. Because they will remember hallucinated reasoning about how things work that is embedded in between those flagged issues.
It can still be valuable to a reviewer that is aware of those circumstances. So I think people who want to try this should take you up on it. But unless one does non-trivial post processing in the prompt like spelling out invariants the model ignored, certain design decisions etc. that require domain knowledge the reviews are typically not worth putting on a MR.
I do find your offer very generous! But for those reasons I think its not very useful to embed these artefacts directly on gitlab. As others mentioned I think performing this experiment via some side channel seems fine to me though.
I would also be interested to hear what the workflow for the review tool the kernel uses is in case you have a good writeup readily at hand.
Andreas
On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao
<terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs < ghc-devs@haskell.org> wrote:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
We can already see examples of how this goes wrong **today** Take a look at a prominent example of a codebase that went "all in" on mostly unsupervised autonomous LLM development, the javascript runtime Bun: https://github.com/oven-sh/bun/pulls
It would kill any motivation I would have to contribute to GHC and collaborate with others if the merge request queue looked anything approximating that. I use the gitlab to talk to and collaborate with actual humans, who I hope will actually engage their brains and help get us to a better and more corret mental model of the space.
If I wanted to talk to an LLM, I have countless other options for ways in which to do that.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote: > Hi devs! > > Inspired by the recent discussion on the LLM policy, I'd like to try > introducing LLM-generated code reviews in GHC, as an opt-in service. > > My motivation: > > * Catch issues before human reviewers spend time on them. > * Reduce the number of bugs merged. > > > What this is — and what it isn't > -------------------------------- > > This is tool output: requested on the MR, produced by a model, posted verbatim > and clearly marked as machine-generated. This is not a review in the sense of > draft LLM policy. > > I'm not vouching for any of it. Think of it as a linter the author could have > run themselves, except on my LLM budget. Nobody is required to act on > it even read it. > > I mention this explicitly because the policy asks reviewers to take full > responsibility for each line of their review. I can't do that here — it would > mean checking every finding myself, which defeats the purpose and would > restrict the service to areas of the compiler I already know well. > > > How to request one > ------------------ > > * Ping me on the MR with a comment like "@sjakobi llm-review please". > * Feel free to request particular aspects the review should cover. > * Authors can request a review for their own MRs. Maintainers can request one > on any MR. > * If I have enough usage left on my plan, I'll confirm that I'm on it. > * I'll then prompt the model, and post the result on the MR. > * Feedback on the quality of the review is of course very welcome. > > > What I promise > -------------- > > * Every review is labelled as machine-generated, with model, effort level and > the exact prompt used. > * Reviews are posted verbatim. I don't edit or curate them, so what you see is > exactly what the model produced. > * Only the public MR diff and public repository context go to the model. > Nothing else. > * I'll keep working on the prompts and the choice of model to improve the > signal-to-noise ratio. > * If the signal-to-noise ratio stays inacceptably bad, we can simply > stop this service. > > > Review coverage > --------------- > > I'll probably start with a fairly basic review prompt, primarily aimed at > correctness issues. > > Possible extensions: > > * Documentation consistency: check that documentation stays consistent, > including Notes elsewhere that reference the changed code. > * Performance: check for performance issues — potentially including checking > the -ddump-simpl output for the changed code for unnecessary > allocation etc. > * Feel free to suggest anything else! > > > Format > ------ > > Initially a single comment per review. Once we reach a good signal-to-noise > ratio, we can consider inline comments on the diff. > > > Caveats > ------- > > This is a volunteer service I intend to provide in my spare time. If I'm AFK > or on vacation, reviews will take longer. If I can't keep up with requests, I > will prioritise which MRs I run. I might take a break entirely. > > > Getting involved > ---------------- > > Is anyone interested in joining this effort? If so we could form a team > (@llm-reviewers?), and share prompting techniques, model choices, etc. > > If the service turns out to be genuinely useful, we can look into > automating it and – if necessary – possible funding. > > > Cheers, > Simon > _______________________________________________ > ghc-devs mailing list -- ghc-devs@haskell.org > To unsubscribe send an email to ghc-devs-leave@haskell.org _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Hi Iavor, some kind of batching might make sense when there are multiple MRs in the same area of the codebase, but that's not very typical. However we can already profit from caching, e.g. when several verifier agents look at findings in the same area. Luckily it does seem that the models that I've been using are already trained on GHC's codebase and are therefore quite capable at navigating it. Cheers, Simon Am Di., 28. Juli 2026 um 09:47 Uhr schrieb Iavor Diatchki <iavor.diatchki@gmail.com>:
Hello,
I am by no means an expert on this, but doing it from scratch for every review might indeed get expensive, as the agent will have to "relearn" the code base each time. It might be worth investigating some sort of design where you can batch the reviews as caching should help significantly.
Cheers, Iavor
On Tue, Jul 28, 2026, 10:15 AM Andreas Klebinger via ghc-devs <ghc-devs@haskell.org> wrote:
Thanks Simon.
I see they rely on a series of fairly specific prompts related to various parts of the review. Some of them encoding quite a bit of domain knowledge.
So there might be something to this approach. At least I can imagine this reducing false positives to a rate where this becomes more valuable to use by default. I do wonder how expensive this is to run at that point.
I might look further into this. Thanks!
On 28/07/2026 01:09, Simon Jakobi wrote:
Hi Andreas!
Regarding Sashiko, I think the README is quite informative: https://github.com/sashiko-dev/sashiko#readme If by "workflow" you mean how developers and maintainers interact with it, it's apparently mostly via mailing lists: https://github.com/sashiko-dev/sashiko/blob/main/MAINTAINERS_GUIDE.md.
https://lore.kernel.org/sashiko-reviews/ lists the (some?) review threads. But the actual reviews are easier to view at https://sashiko.dev/. (At the bottom of each review page, there's a link "View Raw Log", where one can read the prompts and the internal communication of the "subagents" producing the review.)
The claimed false positive rate around 20% sounds pretty good. Initial findings have to pass through some adversarial filtering (involving a "false positive guide" like https://github.com/sashiko-dev/sashiko/blob/main/third_party/prompts/kernel/...).
But it's quite clear that a lot of work went into this system. To adapt it for GHC would be a pretty large task. A mini version for GHC might be more realistic. I think the largest task would be to create a good false positive guide.
As for my "service", I may consider the side-channel option.
Cheers, Simon
Am Mo., 27. Juli 2026 um 23:18 Uhr schrieb Andreas Klebinger via ghc-devs <ghc-devs@haskell.org>:
The noise is mostly inherent to the models. There is no good way to "work on it to fix it" imo.
They miss identify things as bugs due to missing context or simply a lack of understanding fairly frequently. For example I used Opus 5 to audit some GHC code for a specific kind of bug today. It flagged up:
* A "bug" that can only happen if there are broken preconditions that are already checked by lint. * A "bug" that it flagged but marked safe as "only due to" a core invariant of GHCs register allocator design. * Four "Benign issues" that are all correct and pointless noise. * One real bug that I already knew about. (I was just doing a sanity check).
Now I somewhat know my way around the NCG, so it cost me maybe 5 minuts to work through those false reports. But someone can easily spend an hour or much longer just working through those false positives.
If someone who isn't very familiar with the NCG reads through only the LLM output but not the actual code their understanding of the NCG might actually be worse than before. Because they will remember hallucinated reasoning about how things work that is embedded in between those flagged issues.
It can still be valuable to a reviewer that is aware of those circumstances. So I think people who want to try this should take you up on it. But unless one does non-trivial post processing in the prompt like spelling out invariants the model ignored, certain design decisions etc. that require domain knowledge the reviews are typically not worth putting on a MR.
I do find your offer very generous! But for those reasons I think its not very useful to embed these artefacts directly on gitlab. As others mentioned I think performing this experiment via some side channel seems fine to me though.
I would also be interested to hear what the workflow for the review tool the kernel uses is in case you have a good writeup readily at hand.
Andreas
On 27/07/2026 22:41, Simon Jakobi via ghc-devs wrote:
Hi Cheng,
Thanks for your kind reply! :)
Am Mo., 27. Juli 2026 um 22:18 Uhr schrieb Cheng Shao <terrorjack@type.dance>:
i think simon jakobi's proposal is not installing some label-triggered auto-review bot on our gitlab instance, it's about him using his own account to paste llm-generated reviews to patches where authors explicitly request so.
thanks simon, it's indeed a generous offer, but i don't think the gitlab instance is the right place for pasting large blobs of llm-generated reviews. imho it's better to move these to e.g. a self-hosted gerrit instance, or a discord/matrix channel etc, where people who want this service can ping you, get a piece of review to take time to digest and improve their patch accordingly. If the problem is with visual noise, I don't mind putting the blobs into `<details>` tags so they don't take up as much space.
However if the goal is to reduce maintainers' workload, I think the MR is still the best place, so maintainers can use the presence of the LLM-generated review as a signal.
A decent amount of visibility would probably also help to improve the LLM review system more quickly.
i personally find llm-generated reviews very useful in my own work, but the signal-to-noise ratio of review outputs by even the best models/harnesses today is simply low, the insight needs to be mined from the text, and this can be a huge distraction and prevent a human reviewer from entering the flow when looking at patches. If the signal-to-noise ratio is indeed so bad, I'm ready to improve the system until the ratio is more acceptable. It would be great to have some collaborators in this, but I'll also give a try on my own.
Cheers, Simon
cheers, cheng
On Mon, Jul 27, 2026 at 9:40 PM Zubin Duggal via ghc-devs <ghc-devs@haskell.org> wrote: > I am strongly opposed to this. I also don't see the value in > it. If you want to ask an LLM to review your code, you can > probably do this much quicker by just asking one. Today you > are never more than a few clicks away from a text box that > will put you in direct contact with one. > > If you want to maintain a shared repository of common prompts, > skills, guidance etc for LLMs regarding work on GHC, I have > no objection to this (I would prefer to only have documentation > and prose targetting humans in the main GHC repository though). > > In fact I would like to propose a policy in the exact opposite > direction: > > Disallow _purely autonomous_ systems from posting on the GHC > Gitlab, unless such a system is cleared in advance by the > administrators of the instance and/or contributors to GHC. > > What this won't disallow: > > * Getting an LLM to isolate and create a minimal reproducer > or bug description and posting this. > > * Getting an LLM to propose a patch or fix for a particular > issue > > * Asking an LLM about the details of how some particular > external system works, and then posting the results. > > * Asking an LLM to catalogue a bunch of related issues and > extract a common pattern > > * etc. > > What this will disallow: > > Autonomous systems without humans in the loop writing and > interacting with the gitlab. > > My reasons > > 1. Code review is a tool for building a shared understanding > and a common mental model of the codebase and problem > domain. An LLM model cannot do that, or even if you believe > it can, this internal model is lost as soon as that > particular session is garbage collected. At best you can try > to reconstruct some approximation of this model later by > feeding the same inputs into a fresh session on a probably > different model. > > 2. I would find it a very sad if the GHC Gitlab became a > place where bots talk to other bots who act upon and those > comments to make addtional changes in response to this in > an uncontrolled feedback loop. I predict that in the limit > any such feedback loop becomes quickly detached from the > original problem and devolves into a lot of noise totally > divorced from any real human concerns. > > We can already see examples of how this goes wrong **today** > Take a look at a prominent example of a codebase that went > "all in" on mostly unsupervised autonomous LLM development, > the javascript runtime Bun: https://github.com/oven-sh/bun/pulls > > It would kill any motivation I would have to contribute to > GHC and collaborate with others if the merge request queue > looked anything approximating that. I use the gitlab to > talk to and collaborate with actual humans, who I hope will > actually engage their brains and help get us to a better > and more corret mental model of the space. > > If I wanted to talk to an LLM, I have countless other > options for ways in which to do that. > > 3. Prompt injection. Gitlab comments are an untrusted > input source and LLMs fundamentally cannot discriminate > between instructions and data. You can imagine prompt > injection attacks where hidden text in a gitlab comment, > attached bug reproducer, link etc. gets the model to do > various destructive things, like spamming the gitlab, > editing/deleting your prior comments, mining bitcoin on the > CI runners, attempting to inject malicious code into the > compiler, testcases, reproducers etc. > > That doesn't mean that there isn't value for LLM assisted > autonomous processes on the gitlab, given that they are > desiged properly keeping in mind the concerns about human > centeredness, resource usage and prompt injection. > > Here are some examples of LLM assisted system/bots that > could actually be useful if audited and pre-approved with > the community: > > * A bot that automatically bisects regressions and identifies > the change responsible for it. > In principle this could be done with a purely deterministic > script, but often you run into problems with the toolchain > or commits not being individually buildable etc that require > some adjustements on the fly. > > * A bot that automatically minimises reproducers and posts > the results after the testcase can been verified on a deterministic > system > > On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote: >> Hi devs! >> >> Inspired by the recent discussion on the LLM policy, I'd like to try >> introducing LLM-generated code reviews in GHC, as an opt-in service. >> >> My motivation: >> >> * Catch issues before human reviewers spend time on them. >> * Reduce the number of bugs merged. >> >> >> What this is — and what it isn't >> -------------------------------- >> >> This is tool output: requested on the MR, produced by a model, posted verbatim >> and clearly marked as machine-generated. This is not a review in the sense of >> draft LLM policy. >> >> I'm not vouching for any of it. Think of it as a linter the author could have >> run themselves, except on my LLM budget. Nobody is required to act on >> it even read it. >> >> I mention this explicitly because the policy asks reviewers to take full >> responsibility for each line of their review. I can't do that here — it would >> mean checking every finding myself, which defeats the purpose and would >> restrict the service to areas of the compiler I already know well. >> >> >> How to request one >> ------------------ >> >> * Ping me on the MR with a comment like "@sjakobi llm-review please". >> * Feel free to request particular aspects the review should cover. >> * Authors can request a review for their own MRs. Maintainers can request one >> on any MR. >> * If I have enough usage left on my plan, I'll confirm that I'm on it. >> * I'll then prompt the model, and post the result on the MR. >> * Feedback on the quality of the review is of course very welcome. >> >> >> What I promise >> -------------- >> >> * Every review is labelled as machine-generated, with model, effort level and >> the exact prompt used. >> * Reviews are posted verbatim. I don't edit or curate them, so what you see is >> exactly what the model produced. >> * Only the public MR diff and public repository context go to the model. >> Nothing else. >> * I'll keep working on the prompts and the choice of model to improve the >> signal-to-noise ratio. >> * If the signal-to-noise ratio stays inacceptably bad, we can simply >> stop this service. >> >> >> Review coverage >> --------------- >> >> I'll probably start with a fairly basic review prompt, primarily aimed at >> correctness issues. >> >> Possible extensions: >> >> * Documentation consistency: check that documentation stays consistent, >> including Notes elsewhere that reference the changed code. >> * Performance: check for performance issues — potentially including checking >> the -ddump-simpl output for the changed code for unnecessary >> allocation etc. >> * Feel free to suggest anything else! >> >> >> Format >> ------ >> >> Initially a single comment per review. Once we reach a good signal-to-noise >> ratio, we can consider inline comments on the diff. >> >> >> Caveats >> ------- >> >> This is a volunteer service I intend to provide in my spare time. If I'm AFK >> or on vacation, reviews will take longer. If I can't keep up with requests, I >> will prioritise which MRs I run. I might take a break entirely. >> >> >> Getting involved >> ---------------- >> >> Is anyone interested in joining this effort? If so we could form a team >> (@llm-reviewers?), and share prompting techniques, model choices, etc. >> >> If the service turns out to be genuinely useful, we can look into >> automating it and – if necessary – possible funding. >> >> >> Cheers, >> Simon >> _______________________________________________ >> ghc-devs mailing list -- ghc-devs@haskell.org >> To unsubscribe send an email to ghc-devs-leave@haskell.org > _______________________________________________ > ghc-devs mailing list -- ghc-devs@haskell.org > To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Hi Zubin! Am Mo., 27. Juli 2026 um 21:39 Uhr schrieb Zubin Duggal <zubin@well-typed.com>:
I am strongly opposed to this. I also don't see the value in it. If you want to ask an LLM to review your code, you can probably do this much quicker by just asking one. Today you are never more than a few clicks away from a text box that will put you in direct contact with one.
If you want to maintain a shared repository of common prompts, skills, guidance etc for LLMs regarding work on GHC, I have no objection to this (I would prefer to only have documentation and prose targetting humans in the main GHC repository though).
In fact I would like to propose a policy in the exact opposite direction:
Disallow _purely autonomous_ systems from posting on the GHC Gitlab, unless such a system is cleared in advance by the administrators of the instance and/or contributors to GHC.
What this won't disallow:
* Getting an LLM to isolate and create a minimal reproducer or bug description and posting this.
* Getting an LLM to propose a patch or fix for a particular issue
* Asking an LLM about the details of how some particular external system works, and then posting the results.
* Asking an LLM to catalogue a bunch of related issues and extract a common pattern
* etc.
What this will disallow:
Autonomous systems without humans in the loop writing and interacting with the gitlab.
Interesting! I don't want to discuss your policy proposal in depth here, because IMHO it belongs on the LLM policy thread. However if there's a deep opposition to autonomous systems on GitLab (with the exception of Marge Bot I guess), I don't mind keeping this non-autonomous for a few months while we find out whether the LLM-generated reviews are helpful or not. My idea is somewhat inspired by the Sashiko system in the Linux kernel (see "Email from Sashiko" at https://lwn.net/SubscriberLink/1083275/d5518174ec4dcbcb/), and there it appears that at least some maintainers appreciate that it reduces their workload.
My reasons
1. Code review is a tool for building a shared understanding and a common mental model of the codebase and problem domain. An LLM model cannot do that, or even if you believe it can, this internal model is lost as soon as that particular session is garbage collected. At best you can try to reconstruct some approximation of this model later by feeding the same inputs into a fresh session on a probably different model.
Yes, I certainly don't want to take away these human reviews. I fully agree that they are very important. However I think that an initial LLM-generated review could be useful to filter out basic issues, broken conventions etc. And point out bugs. GHC is a complex system and judging by the current rate of bug reports, it's quite clear that human code review is insufficient to catch many issues.
2. I would find it a very sad if the GHC Gitlab became a place where bots talk to other bots who act upon and those comments to make addtional changes in response to this in an uncontrolled feedback loop. I predict that in the limit any such feedback loop becomes quickly detached from the original problem and devolves into a lot of noise totally divorced from any real human concerns.
I agree that this would be sad, and that's why I'm in favour of upholding the "human conversation" part of the current draft LLM policy.
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection.
That doesn't mean that there isn't value for LLM assisted autonomous processes on the gitlab, given that they are desiged properly keeping in mind the concerns about human centeredness, resource usage and prompt injection.
Here are some examples of LLM assisted system/bots that could actually be useful if audited and pre-approved with the community:
* A bot that automatically bisects regressions and identifies the change responsible for it. In principle this could be done with a purely deterministic script, but often you run into problems with the toolchain or commits not being individually buildable etc that require some adjustements on the fly.
* A bot that automatically minimises reproducers and posts the results after the testcase can been verified on a deterministic system
Thanks, these are good suggestions. :)
On 26/07/27 19:38, Simon Jakobi via ghc-devs wrote:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection.
I would advise against giving any LLM system "write access" to the gitlab, where it can autonoumously post things, push commits, trigger pipelines and so on without a "human in the loop".
Recent models are also increasingly resistant to prompt injection.
I find this to be untrue in my experience. LLMs process an undifferentiated input stream, and while the harnessnes try to impose some structure on it, it comes through things approximating xml tags, json etc. However, tool output can produce these "structured tags" just as easily and this gets injected into the context stream with no escaping/validation etc so the model has no chance of differentiating between "legitimate" instructions like `<system-reminders>` etc that actually come from the harness and those that come from running arbitrary commands etc. I have a few proof of concepts of this I can show you off-thread, you don't even have to be too clever about it.
I think the offer was for Simon to copy paste the output from Gitlab<->Model<->Gitlab manually. On 27/07/2026 22:39, Zubin Duggal via ghc-devs wrote:
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection.
I would advise against giving any LLM system "write access" to the gitlab, where it can autonoumously post things, push commits, trigger pipelines and so on without a "human in the loop".
Recent models are also increasingly resistant to prompt injection.
I find this to be untrue in my experience. LLMs process an undifferentiated input stream, and while the harnessnes try to impose some structure on it, it comes through things approximating xml tags, json etc. However, tool output can produce these "structured tags" just as easily and this gets injected into the context stream with no escaping/validation etc so the model has no chance of differentiating between "legitimate" instructions like `<system-reminders>` etc that actually come from the harness and those that come from running arbitrary commands etc. I have a few proof of concepts of this I can show you off-thread, you don't even have to be too clever about it. _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Just to confirm: Yes, that's my offer. Am Mo., 27. Juli 2026 um 22:46 Uhr schrieb Andreas Klebinger via ghc-devs <ghc-devs@haskell.org>:
I think the offer was for Simon to copy paste the output from Gitlab<->Model<->Gitlab manually.
On 27/07/2026 22:39, Zubin Duggal via ghc-devs wrote:
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection.
I would advise against giving any LLM system "write access" to the gitlab, where it can autonoumously post things, push commits, trigger pipelines and so on without a "human in the loop".
Recent models are also increasingly resistant to prompt injection.
I find this to be untrue in my experience. LLMs process an undifferentiated input stream, and while the harnessnes try to impose some structure on it, it comes through things approximating xml tags, json etc. However, tool output can produce these "structured tags" just as easily and this gets injected into the context stream with no escaping/validation etc so the model has no chance of differentiating between "legitimate" instructions like `<system-reminders>` etc that actually come from the harness and those that come from running arbitrary commands etc. I have a few proof of concepts of this I can show you off-thread, you don't even have to be too clever about it. _______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
Am Mo., 27. Juli 2026 um 22:40 Uhr schrieb Zubin Duggal <zubin@well-typed.com>:
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection.
I would advise against giving any LLM system "write access" to the gitlab, where it can autonoumously post things, push commits, trigger pipelines and so on without a "human in the loop".
I agree, and I the service I'm offering doesn't require any of this.
Recent models are also increasingly resistant to prompt injection.
I find this to be untrue in my experience. LLMs process an undifferentiated input stream, and while the harnessnes try to impose some structure on it, it comes through things approximating xml tags, json etc. However, tool output can produce these "structured tags" just as easily and this gets injected into the context stream with no escaping/validation etc so the model has no chance of differentiating between "legitimate" instructions like `<system-reminders>` etc that actually come from the harness and those that come from running arbitrary commands etc. I have a few proof of concepts of this I can show you off-thread, you don't even have to be too clever about it.
I would be interested in running these against Claude Opus 5 which is claimed to be quite resistant.
Thanks for all the feedback, everyone! Regarding the choice of channel, how about I upload the reviews as snippets to e.g. https://gitlab.haskell.org/sjakobi/llm-reviews/-/snippets/? The advantage of snippets is that people can comment on them, so I can easily collect feedback on the quality of the reviews. It's important to me though that there is some visible connection from the MR to the review: 1. The information in the review is intended for all people who are involved in an MR: Both authors and reviewers. 2. Similarly, I'd really love to get feedback from everyone who cares to read the reviews. This way I can improve the service more quickly. 3. I want some visibility, so more people are likely to give the service a try. In case you're worried about your AI agents ingesting the generated reviews, I think it shouldn't be too hard to block access to that repo for them. In other news, Anthropic has generously accepted my application for their open source program (https://claude.com/contact-sales/claude-for-oss), so for the next 6 months, I do have a decent amount of usage that I can spend on reviews (including access to Fable 5!). Consequently, I have started vibe-coding a primitive "mini-sashiko" for GHC: GlaHiCo (https://gitlab.haskell.org/sjakobi/GlaHiCo/-/merge_requests/1). Right now there's just a single "finder" agent, and each finding gets checked separately by a "verifier" agent. The code is a mess already of course, but it's slowly getting useful too. If you have a feature request or want to contribute in other ways, please let me know! :) Cheers, Simon Am Mo., 27. Juli 2026 um 19:38 Uhr schrieb Simon Jakobi <simon.jakobi@gmail.com>:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon
Hi again! Am Do., 30. Juli 2026 um 00:33 Uhr schrieb Simon Jakobi <simon.jakobi@gmail.com>:
In other news, Anthropic has generously accepted my application for their open source program (https://claude.com/contact-sales/claude-for-oss), so for the next 6 months, I do have a decent amount of usage that I can spend on reviews (including access to Fable 5!).
Just to prevent any confusion about this: I didn't apply for this program specifically for GlaHiCo (and in fact I applied before my idea for this service was fully formed). I mainly wanted to get free access to some good models, and used my maintainership of unordered-containers with its many reverse dependencies as the primary argument on why I should be accepted. While I do think that catching bugs in GHC is a good use of my free Claude plan, there's no endorsement or anything like this from Anthropic. Apart from this I think that GlaHiCo is ready for some trial runs, so please let me know if there is an MR that I should run it on. By default, I will deliver the report as a snippet in the GlaHiCo repo and a link in a comment below your request. If you have a different preference on this, I will try to accomodate you. If you have any special requests regarding models, prompts etc, just let me know, and I'll try to accomodate them too. Also please note that for now, GlaHiCo is made for a "high-trust" environment. If you want to experiment with prompt injections or anything like that, please run it on your own machine and your own LLM plan, at your own risk. Cheers, Simon
Consequently, I have started vibe-coding a primitive "mini-sashiko" for GHC: GlaHiCo (https://gitlab.haskell.org/sjakobi/GlaHiCo/-/merge_requests/1). Right now there's just a single "finder" agent, and each finding gets checked separately by a "verifier" agent.
The code is a mess already of course, but it's slowly getting useful too. If you have a feature request or want to contribute in other ways, please let me know! :)
Cheers, Simon
Am Mo., 27. Juli 2026 um 19:38 Uhr schrieb Simon Jakobi <simon.jakobi@gmail.com>:
Hi devs!
Inspired by the recent discussion on the LLM policy, I'd like to try introducing LLM-generated code reviews in GHC, as an opt-in service.
My motivation:
* Catch issues before human reviewers spend time on them. * Reduce the number of bugs merged.
What this is — and what it isn't --------------------------------
This is tool output: requested on the MR, produced by a model, posted verbatim and clearly marked as machine-generated. This is not a review in the sense of draft LLM policy.
I'm not vouching for any of it. Think of it as a linter the author could have run themselves, except on my LLM budget. Nobody is required to act on it even read it.
I mention this explicitly because the policy asks reviewers to take full responsibility for each line of their review. I can't do that here — it would mean checking every finding myself, which defeats the purpose and would restrict the service to areas of the compiler I already know well.
How to request one ------------------
* Ping me on the MR with a comment like "@sjakobi llm-review please". * Feel free to request particular aspects the review should cover. * Authors can request a review for their own MRs. Maintainers can request one on any MR. * If I have enough usage left on my plan, I'll confirm that I'm on it. * I'll then prompt the model, and post the result on the MR. * Feedback on the quality of the review is of course very welcome.
What I promise --------------
* Every review is labelled as machine-generated, with model, effort level and the exact prompt used. * Reviews are posted verbatim. I don't edit or curate them, so what you see is exactly what the model produced. * Only the public MR diff and public repository context go to the model. Nothing else. * I'll keep working on the prompts and the choice of model to improve the signal-to-noise ratio. * If the signal-to-noise ratio stays inacceptably bad, we can simply stop this service.
Review coverage ---------------
I'll probably start with a fairly basic review prompt, primarily aimed at correctness issues.
Possible extensions:
* Documentation consistency: check that documentation stays consistent, including Notes elsewhere that reference the changed code. * Performance: check for performance issues — potentially including checking the -ddump-simpl output for the changed code for unnecessary allocation etc. * Feel free to suggest anything else!
Format ------
Initially a single comment per review. Once we reach a good signal-to-noise ratio, we can consider inline comments on the diff.
Caveats -------
This is a volunteer service I intend to provide in my spare time. If I'm AFK or on vacation, reviews will take longer. If I can't keep up with requests, I will prioritise which MRs I run. I might take a break entirely.
Getting involved ----------------
Is anyone interested in joining this effort? If so we could form a team (@llm-reviewers?), and share prompting techniques, model choices, etc.
If the service turns out to be genuinely useful, we can look into automating it and – if necessary – possible funding.
Cheers, Simon
participants (9)
-
Andreas Klebinger -
Cheng Shao -
Iavor Diatchki -
Moritz Angermann -
Sebastian Graf -
Simon Jakobi -
Simon Peyton Jones -
Tom Ellis -
Zubin Duggal