I did find that some jobs were being retried repeatedly. I have deployed a workaround to prevent this from continuing. The problem was related to T18623 behaving strangely, and I've opened https://gitlab.haskell.org/ghc/ghc/-/issues/23139 so GHC devs can have a look at it.

On Sun, 19 Mar 2023 at 13:27, Bryan Richter <bryan@haskell.foundation> wrote:
I'm back at my computer and am investigating. I don't see the problem I feared, but I do see some anomalies. I'll update once it's back to normal.

On Sat, 18 Mar 2023 at 17:55, Bryan Richter <bryan@haskell.foundation> wrote:
I'm away from my computer for the day, but yes there were some jobs that got stuck in a restart loop. See https://gitlab.haskell.org/ghc/ghc/-/issues/23094#note_487426 . Unfortunately I don't know if there are others, but I did fix the root cause of that particular loop.

On Sat, 18 Mar 2023, 15.06 Sam Derbyshire, <sam.derbyshire@gmail.com> wrote:
I think there's a problem with jobs restarting, on my renamer MR there were 5 full pipelines running at once. I had to cancel some of them, but also it seems some got cancelled by some new CI pipelines restarting.

On Sat, 18 Mar 2023 at 13:59, Simon Peyton Jones <simon.peytonjones@gmail.com> wrote:
All GHC CI pipelines seem stalled, sadly


Can someone unglue it?

Thanks!

Simon
_______________________________________________
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs