Hi all,
For the past week or so, nightly-i386-linux-deb9-validate has been failing consistently.
They show up on the failure dashboard because the logs contain the phrase "Cannot allocate memory".
I haven't looked yet to see if they always fail in the same place, but I'll do that soon. The first example I looked at, however, has the line "xz: (stdin): Cannot allocate memory", so it's not GHC (alone) causing the problem.
As a consequence of showing up on the dashboard, the jobs get restarted. Since they fail consistently, they keep getting restarted. Since the jobs keep getting restarted, the pipelines stay alive. When I checked just now, there were 8 nightly runs still running. :) Thus I'm going to cancel the still-running nightly-i386-linux-deb9-validate jobs and let the pipelines die in peace. You can still find all examples of failed jobs on the dashboard:
To prevent future problems, it would be good if someone could help me look into this. Otherwise I'll just disable the job. :(