Re: Consistent CI failure in job nightly-i386-linux-deb9-validate

28 Sep 2022

      Aha: while i386-linux-deb9-validate sets no extra XZ options,
*nightly*-i386-linux-deb9-validate
(the failing job) sets "XZ_OPT = 9".

A revert would fix the problem, but presumably so would tweaking that
option. Does anyone have information that would lead to a better decision
here?

On Wed, Sep 28, 2022 at 2:02 PM Cheng Shao  wrote:
...
Sure, in which case pls revert it. Apologies for the impact, though
I'm still a bit curious, the i386 job did pass in the original MR.
On Wed, Sep 28, 2022 at 1:00 PM Bryan Richter 
wrote:
...
Yep, it seems to mostly be xz that is running out of memory. (All recent
builds that I sampled, but not all builds through all time.) Thanks for
pointing it out!
...
I can revert the change.
On Wed, Sep 28, 2022 at 11:46 AM Cheng Shao  wrote:
...
Hi Bryan,
This may be an unintended fallout of !8940. Would you try starting an
i386 pipeline with it reversed to see if it solves the issue, in which
case we should revert or fix it in master?
On Wed, Sep 28, 2022 at 9:58 AM Bryan Richter via ghc-devs
 wrote:
...
Hi all,
For the past week or so, nightly-i386-linux-deb9-validate has been
...
...
...
They show up on the failure dashboard because the logs contain the
...
...
...
I haven't looked yet to see if they always fail in the same place,
but I'll do that soon. The first example I looked at, however, has the line
"xz: (stdin): Cannot allocate memory", so it's not GHC (alone) causing the
...
...
...
As a consequence of showing up on the dashboard, the jobs get
restarted. Since they fail consistently, they keep getting restarted. Since
failing consistently.
phrase "Cannot allocate memory".
problem.
the jobs keep getting restarted, the pipelines stay alive. When I checked
just now, there were 8 nightly runs still running. :) Thus I'm going to
cancel the still-running nightly-i386-linux-deb9-validate jobs and let the
pipelines die in peace. You can still find all examples of failed jobs on
the dashboard:
...
...
...
https://grafana.gitlab.haskell.org/d/167r9v6nk/ci-spurious-failures?orgId=2&from=now-90d&to=now&refresh=5m&var-types=cannot_allocate
...
To prevent future problems, it would be good if someone could help me
look into this. Otherwise I'll just disable the job. :(
...
_______________________________________________
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs