[GHC] #10510: Testsuite driver does not run tests in parallel on Windows
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Keywords: | Operating System: Windows Architecture: | Type of failure: None/Unknown Unknown/Multiple | Blocked By: Test Case: | Related Tickets: Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- `testsuite/driver/runtests.py` contains the following 3 lines: {{{ if windows: print("Warning: Ignoring request to use threads as running on Windows") config.use_threads = 0 }}} They were introduced in commit e9b3d6cc65c0b69bd4a50bfa7bc7e1d3536446a8: {{{ Author: Ian Lynagh <> Date: Wed Oct 28 17:54:21 2009 +0000 Don't use threads on Windows It seems to cause some sort of deadlock }}} I tried removing them (Windows 64 bit, using msys2), but then running validate doesn't get any further than: {{{ ... =====> Overlap3(normal) 1605 of 4525 [2, 7, 0] =====> Overlap4(normal) 1606 of 4525 [2, 7, 0] =====> Overlap5(normal) 1607 of 4525 [2, 7, 0] =====> Overlap6(normal) 1608 of 4525 [2, 7, 0] =====> Overlap7(normal) 1609 of 4525 [2, 7, 0] =====> Overlap9(normal) 1610 of 4525 [2, 7, 0] }}} It would be great if we could run the testsuite in parallel again, since running validate takes forever currently. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows
-------------------------------------+-------------------------------------
Reporter: thomie | Owner:
Type: bug | Status: new
Priority: high | Milestone:
Component: Test Suite | Version: 7.10.1
Resolution: | Keywords:
Operating System: Windows | Architecture:
Type of failure: None/Unknown | Unknown/Multiple
Blocked By: | Test Case:
Related Tickets: | Blocking:
| Differential Revisions:
-------------------------------------+-------------------------------------
Comment (by Thomas Miedema
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: Type of failure: None/Unknown | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by Phyx-): So what I know so far is that this seems to be a bug in the `msys-2.0` runtime that happens sporadically. In my case it manifests itself inside the `diff.exe` that is being called by the Python testsuite scripts. [[Image(http://i.imgur.com/IWciMQM.png)]] `diff.exe` gets stuck in an infinite loop while starting up. Eventually, you'll either run out of cores or the tests finish but it'll wait on the one that's stuck. Looking at the stack of the application shows: {{{ msys-2.0.dll!Ordinal1816+0x1b7 msys-2.0.dll!Ordinal208+0x3e984 msys-2.0.dll!Ordinal130+0x171c msys-2.0.dll!Ordinal1816+0x1b7 ntdll.dll!RtlInitializeCriticalSection+0x10e ntdll.dll!RtlInitializeCriticalSection+0x88 ntdll.dll!RtlIsCriticalSectionLockedByThread+0x2a5 ntdll.dll!RtlIsCriticalSectionLockedByThread+0x1ed ntdll.dll!RtlGetVersion+0x7c0 ntdll.dll!RtlInitializeHandleTable+0xe89 ntdll.dll!RtlInitializeHandleTable+0x45 ntdll.dll!LdrInitializeThunk+0x10 }}} and it never exits from `msys-2.0.dll!Ordinal1816`. Curiously `msys-2.0.dll` does not export anything with ordinal `1816` so I am not sure what happens here. Unfortunately because this is happening inside a critical section the debuggers won't attach. `strace`, `gdb` and `windbg` just wait to break in. Eventually `windbg` will time out and suspend the program to allow you to look, but it doesn't allow you to step to see what may be causing the loop. I'm currently attempting to find the `msys-2.0 runtime` version that introduced the issue. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by thomie): Phyx-: It would be *great* if you could fix this. We would like to validate all patches on Windows as well Linux at some point, but it's sort of a problem when a single validate (--slow) takes hours. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by Phyx-): I have some time set aside this weekend to try to get to the bottom of this. Or at least get closer. The ordinals resolve to: {{{ 1816 717 000EDB02 strptime 208 CF 000EC4BE _ecvtbuf 130 81 000EBFB4 _aclsort }}} But not really sure why `strptime` would be calling `_aclsort` unless I'm missing something. We don't seem to be the only ones with this issue though. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by Phyx-): I have been trying to reproduce this ever since upgrading to Windows 10 but I don't seem to be able to. Can someone who has a Windows 8.1 or earlier machine check to see if this is still an issue? If so i'll install a VM and try to track it down now that I'm able to build the msys2 dll. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by thomie): This is still an issue for me. {{{ $ uname -a MINGW64_NT-6.3 ghc-msys2 2.0.0(0.275/5/3) 2014-10-09 11:14 x86_64 Msys }}} {{{ $ systeminfo Host Name: GHC-MSYS2 OS Name: Microsoft Windows Server 2012 R2 Standard OS Version: 6.3.9600 N/A Build 9600 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Server OS Build Type: Multiprocessor Free Registered Owner: Rackspace Registered Organization: Rackspace Product ID: 00252-70000-00000-AA535 Original Install Date: 10/4/2014, 8:10:48 AM System Boot Time: 9/12/2015, 11:29:35 AM System Manufacturer: Xen System Model: HVM domU System Type: x64-based PC Processor(s): 4 Processor(s) Installed. [01]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2600 Mhz [02]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2599 Mhz [03]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2599 Mhz [04]: Intel64 Family 6 Model 62 Stepping 4 GenuineIntel ~2599 Mhz BIOS Version: Xen 4.1.5, 11/28/2013 Windows Directory: C:\Windows System Directory: C:\Windows\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC) Coordinated Universal Time Total Physical Memory: 4,092 MB Available Physical Memory: 3,503 MB Virtual Memory: Max Size: 6,140 MB Virtual Memory: Available: 5,496 MB Virtual Memory: In Use: 644 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: N/A Hotfix(s): 59 Hotfix(s) Installed. [01]: KB2894852 [02]: KB2894856 [03]: KB2896496 [04]: KB2918614 [05]: KB2919355 [06]: KB2919442 [07]: KB2920189 [08]: KB2926765 [09]: KB2931358 [10]: KB2931366 [11]: KB2937220 [12]: KB2938066 [13]: KB2938772 [14]: KB2939087 [15]: KB2939471 [16]: KB2939576 [17]: KB2949621 [18]: KB2950153 [19]: KB2954879 [20]: KB2955164 [21]: KB2956575 [22]: KB2957189 [23]: KB2958262 [24]: KB2959626 [25]: KB2959977 [26]: KB2961072 [27]: KB2962409 [28]: KB2964718 [29]: KB2965500 [30]: KB2965788 [31]: KB2966826 [32]: KB2966828 [33]: KB2967917 [34]: KB2971203 [35]: KB2972213 [36]: KB2973114 [37]: KB2973201 [38]: KB2973351 [39]: KB2973448 [40]: KB2975061 [41]: KB2975719 [42]: KB2976627 [43]: KB2976897 [44]: KB2977174 [45]: KB2977629 [46]: KB2977765 [47]: KB2978668 [48]: KB2979582 [49]: KB2981580 [50]: KB2982791 [51]: KB2984006 [52]: KB2988948 [53]: KB2989647 [54]: KB2990532 [55]: KB2993100 [56]: KB2993651 [57]: KB2994897 [58]: KB2995004 [59]: KB2998527 Network Card(s): 2 NIC(s) Installed. [01]: Citrix PV Network Adapter Connection Name: public0 DHCP Enabled: No IP address(es) [01]: 104.130.171.101 [02]: fe80::8c4e:3953:209a:d97e [03]: 2001:4802:7801:104:903:8a71:7342:99a1 [02]: Citrix PV Network Adapter Connection Name: private0 DHCP Enabled: No IP address(es) [01]: 10.209.1.81 [02]: fe80::8d64:adaf:4a4b:b531 Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed. }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows -------------------------------------+------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by Phyx-): Hmm could someone who's still getting this issue get a memory dump of the process that's blocking? I'd like to compare it to the one I had. Maybe that will give me more information on where in the msys2 runtime this happens.. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows ---------------------------------+---------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by bgamari): * milestone: => 8.2.1 Comment: It doesn't look like this will happen for 8.2. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows ---------------------------------+---------------------------------------- Reporter: thomie | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.1 Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by bgamari): I have also been unable to reproduce the hang on Windows 10. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#10510: Testsuite driver does not run tests in parallel on Windows
---------------------------------+----------------------------------------
Reporter: thomie | Owner:
Type: bug | Status: new
Priority: high | Milestone: 8.2.1
Component: Test Suite | Version: 7.10.1
Resolution: | Keywords:
Operating System: Windows | Architecture: Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: |
---------------------------------+----------------------------------------
Comment (by Ben Gamari
#10510: Testsuite driver does not run tests in parallel on Windows ---------------------------------+---------------------------------------- Reporter: thomie | Owner: Type: bug | Status: closed Priority: high | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: This has been fixed. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/10510#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC