I am still trying to track this down, in the context of D913. I am hoping that the extra logging that will be availiable once @thomie has finalised D926 will help. There seems to be some shared resource when using the GHC API, and it cannot tolerate running multiple tests at the same time. Alan On Fri, May 29, 2015 at 10:42 PM, Joachim Breitner

wrote:

Hi,

Am Donnerstag, den 28.05.2015, 17:46 -0700 schrieb Edward Z. Yang:

...

To whoever manages our Travis setup https://travis-ci.org/ghc/ghc/builds

that would be me.

...

We seem to be failing because our build takes longer than 50min.

that happens occasionally, yes, and is the main reason we still don’t send failure mails to the commiter (and instead to me).

But the real reason why the Travis setup is currently unusable is this:

Unexpected failures: driver T2507 [bad stderr] (normal) driver T8959a [bad stderr] (normal) ghc-api T6145 [bad exit code] (normal) ghc-api T8639_api [bad exit code] (normal) ghc-api/annotations T10278 [bad exit code] (normal) ghc-api/annotations T10357 [bad exit code] (normal) ghc-api/annotations T10358 [bad exit code] (normal) ghc-api/annotations T10396 [bad exit code] (normal) ghc-api/annotations T10399 [bad exit code] (normal) ghc-api/annotations boolFormula [bad exit code] (normal)

all of which fail with

collect2: ld terminated with signal 9 [Killed] make[3]: *** [t10358] Error 1

Unfortunately, when this first appeared, there were other build failures in HEAD, so I could not (easily) identify the first commit causing this.

I raised this on this list (“api annotations test failures”, May 15). Alan Zimmerman looked into it, but it seems without success.

Greetings, Joachim

-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I think the annotations tests are failing because the Makefile looks like this (excerpt) ------------------- CheckAnnotations: CheckAnnotations.hs rm -f CheckAnnotations CheckAnnotations.hi CheckAnnotations.o '$(TEST_HC)' $(TEST_HC_OPTS) --make -w -v0 -package ghc CheckAnnotations .PHONY: T10358 T10358: CheckAnnotations ./CheckAnnotations "`'$(TEST_HC)' $(TEST_HC_OPTS) --print-libdir | tr -d '\r'`" Test10358 .PHONY: T10396 T10396: CheckAnnotations ./CheckAnnotations "`'$(TEST_HC)' $(TEST_HC_OPTS) --print-libdir | tr -d '\r'`" Test10396 ------------------------ If separate processes are used to execute "make T10396" and "make T10358", they will each independently try to build CheckAnnotations. This causes problems. Alan On Sat, May 30, 2015 at 12:00 PM, Alan & Kim Zimmerman wrote:

Your mail seems to indicate that the processes were sent a SIGKILL during the link phase.

To my knowledge this can happen in two ways: from the test runner for a timeout or from the kernel for an out of memory condition.

I suspect the timeout is not the culprit.

Is there any way to check for out of memory kills on the travis box?

Or is there another way it can be killed?

Alan

On Fri, May 29, 2015 at 10:42 PM, Joachim Breitner < mail@joachim-breitner.de> wrote:

...

Hi,

Am Donnerstag, den 28.05.2015, 17:46 -0700 schrieb Edward Z. Yang:

...

To whoever manages our Travis setup https://travis-ci.org/ghc/ghc/builds

that would be me.

...

We seem to be failing because our build takes longer than 50min.

that happens occasionally, yes, and is the main reason we still don’t send failure mails to the commiter (and instead to me).

But the real reason why the Travis setup is currently unusable is this:

Unexpected failures: driver T2507 [bad stderr] (normal) driver T8959a [bad stderr] (normal) ghc-api T6145 [bad exit code] (normal) ghc-api T8639_api [bad exit code] (normal) ghc-api/annotations T10278 [bad exit code] (normal) ghc-api/annotations T10357 [bad exit code] (normal) ghc-api/annotations T10358 [bad exit code] (normal) ghc-api/annotations T10396 [bad exit code] (normal) ghc-api/annotations T10399 [bad exit code] (normal) ghc-api/annotations boolFormula [bad exit code] (normal)

all of which fail with

collect2: ld terminated with signal 9 [Killed] make[3]: *** [t10358] Error 1

Unfortunately, when this first appeared, there were other build failures in HEAD, so I could not (easily) identify the first commit causing this.

I raised this on this list (“api annotations test failures”, May 15). Alan Zimmerman looked into it, but it seems without success.

Greetings, Joachim

-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

With a lot of help from @thomie .. On Mon, Jun 1, 2015 at 1:25 PM, Joachim Breitner wrote:

Hi,

yay, travis is green again. Thanks to Alan for fixing this.

Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I suspect this is different, I am building a local master now, at current HEAD a27fb46ff1ea46a45e0084c3db92a24475e3bab5 to see what I find. Alan On Mon, Jun 1, 2015 at 8:22 PM, Joachim Breitner wrote:

Hi,

Am Montag, den 01.06.2015, 13:44 +0200 schrieb Alan & Kim Zimmerman:

...

On Mon, Jun 1, 2015 at 1:25 PM, Joachim Breitner < mail@joachim-breitner.de> wrote: yay, travis is green again. Thanks to Alan for fixing this.

...

With a lot of help from @thomie ..

hmm, looks like it is not fixed after all:

Unexpected failures: ghc-api T6145 [bad exit code] (normal) ghc-api T8628 [bad exit code] (normal) ghc-api T8639_api [bad exit code] (normal) ghc-api/annotations T10278 [bad exit code] (normal) ghc-api/annotations T10313 [bad exit code] (normal) ghc-api/annotations T10354 [bad exit code] (normal) ghc-api/annotations T10396 [bad exit code] (normal) ghc-api/annotations T10399 [bad exit code] (normal) ghc-api/annotations boolFormula [bad exit code] (normal) ghc-api/annotations-literals parsed [bad exit code] (normal)

https://s3.amazonaws.com/archive.travis-ci.org/jobs/64945879/log.txt

Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Is is possible to limit the number of concurrent tests? Instead of running on all available cores. On Mon, Jun 1, 2015 at 9:16 PM, Austin Seipp wrote:

Looks like:

collect2: ld terminated with signal 9 [Killed]

which I'd say with fair certainty means that the jobs failed because the linker ran out of memory on Travis, and the OOM killed it. So, that's a thing now.

Perhaps for travis builds, we could disable SplitObjs by default to relieve the memory pressure? I'm not sure if it's on during Travis's builds, but it's a thought.

On Mon, Jun 1, 2015 at 1:30 PM, Alan & Kim Zimmerman wrote:

...

I suspect this is different, I am building a local master now, at current HEAD a27fb46ff1ea46a45e0084c3db92a24475e3bab5 to see what I find.

Alan

On Mon, Jun 1, 2015 at 8:22 PM, Joachim Breitner < mail@joachim-breitner.de> wrote:

...

Hi,

Am Montag, den 01.06.2015, 13:44 +0200 schrieb Alan & Kim Zimmerman:

...

On Mon, Jun 1, 2015 at 1:25 PM, Joachim Breitner wrote: yay, travis is green again. Thanks to Alan for fixing this.

...

With a lot of help from @thomie ..

hmm, looks like it is not fixed after all:

Unexpected failures: ghc-api T6145 [bad exit code] (normal) ghc-api T8628 [bad exit code] (normal) ghc-api T8639_api [bad exit code] (normal) ghc-api/annotations T10278 [bad exit code] (normal) ghc-api/annotations T10313 [bad exit code] (normal) ghc-api/annotations T10354 [bad exit code] (normal) ghc-api/annotations T10396 [bad exit code] (normal) ghc-api/annotations T10399 [bad exit code] (normal) ghc-api/annotations boolFormula [bad exit code] (normal) ghc-api/annotations-literals parsed [bad exit code] (normal)

https://s3.amazonaws.com/archive.travis-ci.org/jobs/64945879/log.txt

Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Regards,

Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Hi, Am Montag, den 01.06.2015, 14:26 -0500 schrieb Austin Seipp:

I don't think there's any built-in way to do that in the current testsuite infrastructure, other than setting THREADS=1, which, if it isn't set already, would probably push us back over the Travis build limit.

we are running validate with CPU=2, which I believe translates to THREADS=1. But we never had memory problems before, and 3GB is still quite a bit. Are we sure that there is no bug here? Or is our official requirement now above 3GB? Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

And I am preparing a test branch on ghc-api/annotations to use this flag, will push it once I have checked its sanity locally On Mon, Jun 1, 2015 at 9:42 PM, Austin Seipp wrote:

On Mon, Jun 1, 2015 at 2:34 PM, Joachim Breitner wrote:

...

Hi,

Am Montag, den 01.06.2015, 14:26 -0500 schrieb Austin Seipp:

...

I don't think there's any built-in way to do that in the current testsuite infrastructure, other than setting THREADS=1, which, if it isn't set already, would probably push us back over the Travis build limit.

we are running validate with CPU=2, which I believe translates to THREADS=1.

It's actually the opposite - if you check ./validate, CPU=2 translates to THREAD=3! The reasoning is 'CPUS' is the number of physical cores you have, and it adds 1 to saturate all your cores, while the extra thread gets scheduled during I/O overlap on the underlying disk from the other threads, overall improving throughput. Normally it's a win on a completely un-contested machine, as some overlap exists.

...

But we never had memory problems before, and 3GB is still quite a bit. Are we sure that there is no bug here? Or is our official requirement now above 3GB?

Well, it's possible there's something else lurking here. Maybe the merge of D913 caused things to spike for some reason, during the cleanup? Possible there was an error in the refactoring. But given that the buildbot only has 3GB and executed (in essence) with THREADS=3, I'm almost certain that it's just because they all ran at once, and the OOM killed them. We may have just gotten lucky?

Also, Thomas has alerted me that the testsuite actually has an option called `high_memory_usage` which will serialize the tests and run them one at a time. So actually, that can work as a solution too!

...

Greetings, Joachim

-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Regards,

Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Looks like https://github.com/ghc/ghc/tree/wip/high_memory_usage has passed. Interestingly it gathered all the tests marked that way for the end, and ran them one at a time. See https://api.travis-ci.org/jobs/64977719/log.txt?deansi=true On Mon, Jun 1, 2015 at 9:54 PM, Joachim Breitner wrote:

HI,

Am Montag, den 01.06.2015, 14:42 -0500 schrieb Austin Seipp:

...

On Mon, Jun 1, 2015 at 2:34 PM, Joachim Breitner

...

we are running validate with CPU=2, which I believe translates to THREADS=1.

It's actually the opposite - if you check ./validate, CPU=2 translates to THREAD=3!

Eh, sorry, that’s what I meant.

...

Also, Thomas has alerted me that the testsuite actually has an option called `high_memory_usage` which will serialize the tests and run them one at a time. So actually, that can work as a solution too!

Good idea. I just pushed a wip/ branch to test that.

Greetings, Joachim

-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs