Am Mo., 27. Juli 2026 um 22:40 Uhr schrieb Zubin Duggal <zubin@well-typed.com>:
3. Prompt injection. Gitlab comments are an untrusted input source and LLMs fundamentally cannot discriminate between instructions and data. You can imagine prompt injection attacks where hidden text in a gitlab comment, attached bug reproducer, link etc. gets the model to do various destructive things, like spamming the gitlab, editing/deleting your prior comments, mining bitcoin on the CI runners, attempting to inject malicious code into the compiler, testcases, reproducers etc.
I don't quite understand how this is connected to my offer. I may have to apply some safeguards in the review but it's still running on _my_ system. Regarding the "untrusted input source": accounts already need to be verified. Recent models are also increasingly resistant to prompt injection.
I would advise against giving any LLM system "write access" to the gitlab, where it can autonoumously post things, push commits, trigger pipelines and so on without a "human in the loop".
I agree, and I the service I'm offering doesn't require any of this.
Recent models are also increasingly resistant to prompt injection.
I find this to be untrue in my experience. LLMs process an undifferentiated input stream, and while the harnessnes try to impose some structure on it, it comes through things approximating xml tags, json etc. However, tool output can produce these "structured tags" just as easily and this gets injected into the context stream with no escaping/validation etc so the model has no chance of differentiating between "legitimate" instructions like `<system-reminders>` etc that actually come from the harness and those that come from running arbitrary commands etc. I have a few proof of concepts of this I can show you off-thread, you don't even have to be too clever about it.
I would be interested in running these against Claude Opus 5 which is claimed to be quite resistant.