Validation Failures on aarch64

Hello GHC devs, When attempting to validate a patch on aarch64, it seems there are a large number of validation failures: SUMMARY for test run started at Mon Apr 8 07:19:05 2019 UTC 0:15:35 spent to go through 6890 total tests, which gave rise to 17169 test cases, of which 10018 were skipped 41 had missing libraries 3151 expected passes 150 expected failures 11 caused framework failures 0 caused framework warnings 0 unexpected passes 3798 unexpected failures 0 unexpected stat failures The failures seem consistent on recent-ish master, specifically the neighborhood of 6113d0d4540af7853c71e9f42a41c3b0bab386fd. Is this to be expected? Thanks, Travis Whitaker

What kinds of tests are failing? Have they always failed?
On Mon, Apr 8, 2019 at 8:55 PM Travis Whitaker
Hello GHC devs,
When attempting to validate a patch on aarch64, it seems there are a large number of validation failures:
SUMMARY for test run started at Mon Apr 8 07:19:05 2019 UTC 0:15:35 spent to go through 6890 total tests, which gave rise to 17169 test cases, of which 10018 were skipped 41 had missing libraries 3151 expected passes 150 expected failures 11 caused framework failures 0 caused framework warnings 0 unexpected passes 3798 unexpected failures 0 unexpected stat failures
The failures seem consistent on recent-ish master, specifically the neighborhood of 6113d0d4540af7853c71e9f42a41c3b0bab386fd. Is this to be expected?
Thanks,
Travis Whitaker _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Just a wild guess, but considering the time it's taken to run through the
testsuite you're running it on a reasonable AArch64 machine.
You may be hitting the weak memory ordering bugs
https://gitlab.haskell.org/ghc/ghc/issues/15449 , though I haven't followed
the ticket very closely...
On Mon, Apr 8, 2019 at 8:55 PM Travis Whitaker
Hello GHC devs,
When attempting to validate a patch on aarch64, it seems there are a large number of validation failures:
SUMMARY for test run started at Mon Apr 8 07:19:05 2019 UTC 0:15:35 spent to go through 6890 total tests, which gave rise to 17169 test cases, of which 10018 were skipped 41 had missing libraries 3151 expected passes 150 expected failures 11 caused framework failures 0 caused framework warnings 0 unexpected passes 3798 unexpected failures 0 unexpected stat failures
The failures seem consistent on recent-ish master, specifically the neighborhood of 6113d0d4540af7853c71e9f42a41c3b0bab386fd. Is this to be expected?
Thanks,
Travis Whitaker _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

The patch I’m validating is actually to fix those memory barrier issues.
It turns out these failures were all due to GHC complaining about the LLVM
version on stderr; sorry for rubber ducking the list!
On Mon, Apr 8, 2019 at 1:26 PM Phyx
Just a wild guess, but considering the time it's taken to run through the testsuite you're running it on a reasonable AArch64 machine. You may be hitting the weak memory ordering bugs https://gitlab.haskell.org/ghc/ghc/issues/15449 , though I haven't followed the ticket very closely...
On Mon, Apr 8, 2019 at 8:55 PM Travis Whitaker
wrote: Hello GHC devs,
When attempting to validate a patch on aarch64, it seems there are a large number of validation failures:
SUMMARY for test run started at Mon Apr 8 07:19:05 2019 UTC 0:15:35 spent to go through 6890 total tests, which gave rise to 17169 test cases, of which 10018 were skipped 41 had missing libraries 3151 expected passes 150 expected failures 11 caused framework failures 0 caused framework warnings 0 unexpected passes 3798 unexpected failures 0 unexpected stat failures
The failures seem consistent on recent-ish master, specifically the neighborhood of 6113d0d4540af7853c71e9f42a41c3b0bab386fd. Is this to be expected?
Thanks,
Travis Whitaker _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Indeed this is much worse than it used to be and quite unexpected. Last I knee there were only 100 or so failures, mostly external-interpeter related. Any investigation you could do would be greatly appreciated. I would be happy to offer access to hardware if this helps.
Cheers,
- Ben
On April 8, 2019 3:55:18 PM EDT, Travis Whitaker
Hello GHC devs,
When attempting to validate a patch on aarch64, it seems there are a large number of validation failures:
SUMMARY for test run started at Mon Apr 8 07:19:05 2019 UTC 0:15:35 spent to go through 6890 total tests, which gave rise to 17169 test cases, of which 10018 were skipped 41 had missing libraries 3151 expected passes 150 expected failures 11 caused framework failures 0 caused framework warnings 0 unexpected passes 3798 unexpected failures 0 unexpected stat failures
The failures seem consistent on recent-ish master, specifically the neighborhood of 6113d0d4540af7853c71e9f42a41c3b0bab386fd. Is this to be expected?
Thanks,
Travis Whitaker
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Ben Gamari
Indeed this is much worse than it used to be and quite unexpected. Last I knee there were only 100 or so failures, mostly external-interpeter related. Any investigation you could do would be greatly appreciated. I would be happy to offer access to hardware if this helps.
Whoops. It looks like I was a bit late to the party here. Thanks for pinning down the issue, Travis. That being said, it seems that our CI job is failing due to missing bzip2. Fixed in https://gitlab.haskell.org/ghc/ci-images/merge_requests/10. It would be great if we could fix/mark-as-broken enough tests that we can remove allow_failure from the aarch64 job. Cheers, - Ben
participants (4)
-
Ben Gamari
-
Matthew Pickering
-
Phyx
-
Travis Whitaker