[GHC] #12661: Testsuite driver fails on Windows
#12661: Testsuite driver fails on Windows --------------------------------------+--------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Keywords: | Operating System: Windows Architecture: x86_64 (amd64) | Type of failure: None/Unknown Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: --------------------------------------+--------------------------------- I encountered the following failure in the testsuite driver while bringing up the new Windows build bot, {{{ PYTHON="python2" "python2" ../../driver/runtests.py -e ghc_compiler_always_flags="'-dcore-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -dno- debug-output'" -e config.compiler_debugged=True -e ghc_with_native_codegen=1 -e config.have_vanilla=True -e config.have_dynamic=False -e config.have_profiling=False -e ghc_with_threaded_rts=1 -e ghc_with_dynamic_rts=0 -e config.have_interp=False -e config.unregisterised=False -e config.ghc_dynamic_by_default=False -e config.ghc_dynamic=False -e ghc_with_smp=1 -e ghc_with_llvm=0 -e windows=True -e darwin=False -e config.in_tree_compiler=True --threads=3 -e config.cleanup=True -e config.local=False --rootdir=. --configfile=../../config/ghc -e 'config.confdir="../../config"' -e 'config.platform="x86_64-unknown- mingw32"' -e 'config.os="mingw32"' -e 'config.arch="x86_64"' -e 'config.wordsize="64"' -e 'config.timeout=int() or config.timeout' -e 'config.exeext=".exe"' -e 'config.top="/home/drydock/ghc/testsuite"' --config 'compiler="/home/drydock/ghc/inplace/bin/ghc-stage1.exe"' --config 'ghc_pkg="/home/drydock/ghc/inplace/bin/ghc-pkg.exe"' --config 'haddock="/home/drydock/ghc/inplace/bin/haddock.exe"' --config 'hp2ps="/home/drydock/ghc/inplace/bin/hp2ps.exe"' --config 'hpc="/home/drydock/ghc/inplace/bin/hpc.exe"' --config 'gs="gs"' --config 'timeout_prog="../../timeout/install-inplace/bin/timeout.exe"' -e "config.stage=1" --summary-file "../../../testsuite_summary_stage1.txt" --no-print-summary 1 \ \ \ \ \ \ \ Warning: Ignoring request to use threads as running on Windows Traceback (most recent call last): File "../../driver/runtests.py", line 174, in <module> if mydll.kernel32.SetConsoleCP(65001) == 0: File "/usr/lib/python2.7/ctypes/__init__.py", line 435, in __getattr__ dll = self._dlltype(name) File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__ self._handle = _dlopen(self._name, mode) OSError: No such file or directory }}} From the repl it indeed seems that `ctypes.cdll.kernel32` is broken, {{{ $ python
import ctypes ctypes.cdll.kernel32 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.4/ctypes/__init__.py", line 421, in __getattr__ dll = self._dlltype(name) File "/usr/lib/python3.4/ctypes/__init__.py", line 351, in __init__ self._handle = _dlopen(self._name, mode) OSError: No such file or directory }}}
Tamar, do you have any idea what is going on here? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Phyx- Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by Phyx-): * owner: => Phyx- Comment: I haven't encountered it before but: File not Found will also be returned by the system loader when it couldn't find a dependency of a library it's trying to load. In this case I suspect you've set up the buildbot on a completely new Windows install. In which case you're probably missing the proper Microsoft C Runtime. (Windows update would eventually get to this). According to the docs : https://docs.python.org/3.4/using/windows.html Python 3.5 has a dependency on the VC2015 runtime https://www.microsoft.com/en-us/download/details.aspx?id=48145 and Python 3.4 on the VC2008 runtime https://www.microsoft.com/en- us/download/details.aspx?id=15336 The load is probably failing because ctypes loads a dll that needs the runtime. It should have recorded the missing dependency in the Windows Event Viewer as well. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Phyx- Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari):
In this case I suspect you've set up the buildbot on a completely new Windows install. In which case you're probably missing the proper Microsoft C Runtime. (Windows update would eventually get to this).
Indeed you are right, this is a clean installation. Unfortunately the problem seems to persist after installing the VC2008 runtime (both 32- and 64-bit for good measure). Moreover, there is no sign of any relevant events in Event Viewer. Sadly my usual `strace` approach seems powerless on Windows: I can see an `open` system call for `ctypes/__init__.py` but nothing else interesting. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by Phyx-): * cc: Phyx (removed) * cc: Phyx- (added) * owner: Phyx- => Comment: Whoops, meant to correct the CC not assign it to myself. With 3.4 installed your python 3.4 example should work, can you try the repl again with the python 3.4? The 2.7 the testsuite is automatically detecting is a big more vague as it support multiple compilers https://docs.python.org/2.7/using/windows.html Typically on Windows the easiest way to debug such issues is using glfags https://msdn.microsoft.com/en- us/library/windows/hardware/ff549557(v=vs.85).aspx which has an option to "enable loader snaps" This will tell the kernel to display loader debug messages *only* when the process given is running in a debugger. But now that you have the VC2008 runtime try the 3.4 repl (unless that's the one you said still doesn't work?) I'll check the dependency chain when I get home. Usually the python installer takes care of all of this, but I guess because python was installed via pacman it can't pull in the MSVCRT dependencies. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): Indeed 3.4.3 is the failing python version, {{{ $ strace -o log python Python 3.4.3 (default, Mar 4 2016, 10:23:36) [GCC 4.9.2] on msys Type "help", "copyright", "credits" or "license" for more information. p>>> import ctypes
ctypes.cdll.kernel32 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.4/ctypes/__init__.py", line 421, in __getattr__ dll = self._dlltype(name) File "/usr/lib/python3.4/ctypes/__init__.py", line 351, in __init__ self._handle = _dlopen(self._name, mode) OSError: No such file or directory }}}
-- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Phyx-): Right, so I see the problem... You're using the msys version of python, which doesn't correctly search for the SO name. {{{ 6ef8:5d44 @ 584664718 - LdrLoadDll - ENTER: DLL name: kernel32. 6ef8:5d44 @ 584664718 - LdrpLoadDllInternal - ENTER: DLL name: kernel32 6ef8:5d44 @ 584664718 - LdrpFindKnownDll - ENTER: DLL name: kernel32 6ef8:5d44 @ 584664718 - LdrpFindKnownDll - RETURN: Status: 0xc0000135 6ef8:5d44 @ 584664718 - LdrpSearchPath - ENTER: DLL name: kernel32 6ef8:5d44 @ 584664718 - LdrpComputeLazyDllPath - INFO: DLL search path computed: E:\ghc- dev\msys64\usr\bin;C:\WINDOWS\SYSTEM32;C:\WINDOWS\system;C:\WINDOWS;.;E:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\arcade;C:\ruby215\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\WIDCOMM\Bluetooth Software\;C:\Program Files\WIDCOMM\Bluetooth 6ef8:5d44 @ 584664718 - LdrpResolveDllName - ENTER: DLL name: E:\ghc- dev\msys64\usr\bin\kernel32 6ef8:5d44 @ 584664718 - LdrpResolveDllName - RETURN: Status: 0xc0000135 6ef8:5d44 @ 584664718 - LdrpResolveDllName - ENTER: DLL name: C:\WINDOWS\SYSTEM32\kernel32 6ef8:5d44 @ 584664718 - LdrpResolveDllName - RETURN: Status: 0xc0000135 6ef8:5d44 @ 584664718 - LdrpResolveDllName - ENTER: DLL name: C:\WINDOWS\system\kernel32 6ef8:5d44 @ 584664718 - LdrpResolveDllName - RETURN: Status: 0xc0000135 6ef8:5d44 @ 584664718 - LdrpResolveDllName - ENTER: DLL name: C:\WINDOWS\kernel32 }}} it doesn't add the ".dll" to it. so the loader doesn't find it. What you want, is the mingw-w64 version of python (which is also compiled with GCC so MSVC runtime not needed). {{{ pacman -S mingw-w64-$(uname -m)-python3 }}} Odd, the installation of Sphinx should have installed it as well. It would be python3.5. The msys python is a bit broken (which is unsurprising, the msys versions are usually the bare minimum to get the msys shell itself working). correct the link from python3 -> mingw python3.5 and you should be good to go. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): Replying to [comment:5 Phyx-]:
Right, so I see the problem...
You're using the msys version of python, which doesn't correctly search for the SO name.
Indeed it looks like that is the issue. Great catch!
it doesn't add the ".dll" to it. so the loader doesn't find it. What you want, is the mingw-w64 version of python (which is also compiled with GCC so MSVC runtime not needed).
{{{ pacman -S mingw-w64-$(uname -m)-python3 }}}
Odd, the installation of Sphinx should have installed it as well. It would be python3.5. The msys python is a bit broken (which is unsurprising, the msys versions are usually the bare minimum to get the msys shell itself working).
Judging from the output from that `pacman` command I suspect that the correct python was already installed. However...
correct the link from python3 -> mingw python3.5 and you should be good to go.
Ahh, I suspect this is what I missed. In fact, shouldn't it also be necessary to symlink `python` to `python3.5` as well to ensure that no references to the broken python leak through? This is a rather subtle step, do you think you could mention this on [[Building/Preparation/Windows]]? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): I'm actually quite confused. It looks like the testsuite driver will not run with python3. {{{ *** WARNING: running testsuite using Python 3. *** Python 3 support is experimental. See Trac #9184. Traceback (most recent call last): File "../driver/runtests.py", line 216, in <module> for line in pkginfo.split('\n'): TypeError: a bytes-like object is required, not 'str' }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: Indeed simply installing `mingw-w64-$(uname -m)-python2` seems to fix things. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): Thanks for your help Phyx! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Phyx-): Yes, though this advice of prefering the mingw packages over the msys ones doesn't just apply for `python`. I've amended the section at https://ghc.haskell.org/trac/ghc/wiki/Building/Preparation/Windows#Configuri... The way I always do it is to make sure in my profile that `/mingw64/bin` is the first thing on the `$PATH`. I need to make some time to re-evaluate that page, as I am not sure we need any of those msys versions of the tools we're telling people to install. They've bitten us quite a few times already. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): Thanks for amending the Wiki discussion, Tamar. That explanation clears things up nicely. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Phyx-): No problem, you might also hit #12554, haven't figured that one out yet. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:12> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by bgamari): Apparently this is only an issue with `msys2-runtime >= 1.5.1`. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:13> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Phyx-): I believe you mean `msys2-runtime >= 2.5.1` -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:14> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by bgamari): * status: closed => new * resolution: fixed => Comment: This really isn't fixed per se; we just found a workaround. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:14> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: upstream Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by bgamari): * status: new => upstream Comment: I've opened `msys2-packages` [[https://github.com/Alexpux/MSYS2-packages/issues/707|Issue #707]] to track this issue. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: upstream Priority: normal | Milestone: Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"17d696f1a8dcc28074886a2830036144eced6d87/ghc" 17d696f/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="17d696f1a8dcc28074886a2830036144eced6d87" validate: Allow user to override Python interpreter Due to #12554 and #12661 we must be quite picky about our choice of Python interpreter on Windows. Allow the user to override it. Test Plan: `PYTHON=/usr/bin/python2 ./validate` on Windows Reviewers: austin, Phyx Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2603 GHC Trac Issues: #12554, #12661 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:16> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by bgamari): * status: upstream => patch * milestone: => 8.2.1 Comment: RyanGlScott has a nice [[https://github.com/Alexpux/MSYS2-packages/issues/707#issuecomment-256648193|workaround]] for this. I'm working on integrating this in Phab:D2641. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:17> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"cc4710af723ca4bbcf77172ff715af22f9ce8419/ghc" cc4710af/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="cc4710af723ca4bbcf77172ff715af22f9ce8419" testsuite: Simplify kernel32 glue logic On Windows the testsuite driver calls kernel32 to set the current terminal codepage. The previous implementation of this was significantly more complex than necessary, and was wrong in the case of MSYS2, which requires that we explicitly load the library using the name of its DLL, including its file extension. Test Plan: Validate on Windows Reviewers: austin, RyanGlScott, Phyx Reviewed By: RyanGlScott, Phyx Subscribers: thomie Differential Revision: https://phabricator.haskell.org/D2641 GHC Trac Issues: #12661 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:18> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"0ce59be3a2723f814a3e929fd32a44ff4e890a49/ghc" 0ce59be/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="0ce59be3a2723f814a3e929fd32a44ff4e890a49" Fix testsuite threading, timeout, encoding and performance issues on Windows In a land far far away, a project called Cygwin was born. Cygwin used newlib as it's standard C library implementation. But Cygwin wanted to emulate POSIX systems as closely as possible. So it implemented `execv` using the Windows function `spawnve`. Specifically ``` spawnve (_P_OVERLAY, path, argv, cur_environ ()) ``` `_P_OVERLAY` is crucial, as it makes the function behave *sort of* like execv on linux. the child process replaces the original process. With one major difference because of the difference in process models on Windows: the original process signals the caller that it's done. this is why the file is still locked. because it's still running, control was returned because the parent process was destroyed, but the child is still running. I think it's just pure dumb luck, that the older runtimes are slow enough to give the process time to terminate before we tried deleting the file. Which explains why you do have sporadic failures even on older runtimes like 2.5.0, of a test or two (like T7307). So this patch fixes a couple of things. I leverage the existing `timeout.exe` to implement a workaround for this issue. a) The old timeout used to start the process then assign it to the job. This is slightly faulty since child processes are only assigned to a job is their parent were assigned at the time they started. So this was a race condition. I now create the process suspended, assign it to the job and then resume it. Which means all child processes are not running under the same job. b) First things, Is to prevent dangling child processes. I mark the job with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` so when the last process in the job is done, it insures all processes under the job are killed. c) Secondly, I change the way we wait for results. Instead of waiting for the parent process to terminate, I wait for the job itself to terminate. There's a slight subtlety there, we can't wait on the job itself. Instead we have to create an I/O Completion port and wait for signals on it. See https://blogs.msdn.microsoft.com/oldnewthing/20130405-00/?p=4743 This fixes the issues on all runtimes for me and makes T7307 pass consistenly. The threading was also simplified by hiding all the locking in a single semaphore and a completion class. Futhermore some additional error reporting was added. For encoding the testsuite now no longer passes a file handle to the subprocess since on windows, sh.exe seems to acquire a lock on the file that is not released in a timely fashion. I suspect this because cygwin seems to emulate console handles by creating file handles and using those for std handles. So when we give it an existing file handle it just locks the file. I what's happening is that it's not releasing the handle until all shared cygwin processes are dead. Which explains why it worked in single threaded mode. So now instead we pass a pipe and do not interpret the resulting data. Any bytes written to stdin or read out of stdout/stderr are done so in binary mode and we do not interpret the data. The reason for this is that we have encoding tests in GHC which pass invalid utf-8. If we try to handle the data as text then python will throw an exception instead of a test comparison failing. Also I have fixed the ability to override `PYTHON` when calling `make tests`. This now works the same as with `.\validate`. Finally, after cleaning up the locks I was able to make the abort behavior work correctly as I believe it was intended: when you press Ctrl+C and send an interrupt signal, the testsuite finishes the active tests and then gracefully exits showing you a report of the progress it did make. So using Ctrl+C will not just *die* as it did before. These changes lift the restriction on which python version you use (msys/mingw) or which runtime or python 3 or python 2. All combinations should now be supported. Test Plan: PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH && PYTHON=/usr/bin/python THREADS=9 make test THREADS=9 make test PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH && PYTHON=/usr/bin/python ./validate --quiet --testsuite-only Reviewers: erikd, RyanGlScott, bgamari, austin Subscribers: jrtc27, mpickering, thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2684 GHC Trac Issues: #12725, #12554, #12661, #12004 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:19> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed Comment: I think this is now as fixed as it is going to be. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:20> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Tamar Christina <tamar@…>): In [changeset:"a3704409acc3bd237d3e872f640686918fb51f5f/ghc" a3704409/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="a3704409acc3bd237d3e872f640686918fb51f5f" Fix various issues with testsuite code on Windows Summary: Previously we would make direct calls to `diff` using `os.system`. On Windows `os.system` is implemented using the standard idiom `CreateProcess .. WaitForSingleObject ..`. This again runs afoul with the `_exec` behaviour on Windows. So we ran into some trouble where sometimes `diff` would return before it's done. On tests which run multiple ways, such as `8086` what happens is that we think the diff is done and continue. The next way tries to set things up again by removing any previous directory. This would then fail with and error saying the directory can't be removed. Which is true, because the previous diff code/child is still running. We shouldn't make any external calls to anything using `os.system`. Instead just use `runCmd` which uses `timeout`. This also ensures that if we hit the cygwin bug where diff or any other utility hangs, we kill it and continue and not hang the entire test and leave hanging processes. Further more we also: Ignore error lines from `removeFile` from tools in the testsuite. This is a rather large hammer to work around the fact that `hsc2hs` often tries to remove it's own file too early. When this is patched the workaround can be removed. See Trac #9775 We mark `prog003` as skip. Since this test randomly fails and passes. For stability it's disabled but it is a genuine bug which we should find. It's something with interface files being overwritten. See Trac #11317 when `rmtree` hits a readonly file, the `onerror` handler is raised afterwards but not during the tree walk. It doesn't allow you to recover and continue as we thought. Instead you have to explicitly start again. This is why sometimes even though we call `cleanup` before `os.mkdirs`, it would sometimes fail with an error that the folder already exists. So we now do a second walk. A new verbosity level (4) will strip the silent flags from `MAKE` invocations so you can actually see what's going on. Test Plan: ./validate on build bots. Reviewers: bgamari, austin Reviewed By: bgamari Subscribers: mpickering, thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2894 GHC Trac Issues: #12661, #11317, #9775 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:21> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#12661: Testsuite driver fails on Windows ---------------------------------+-------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Windows | Architecture: x86_64 (amd64) Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+-------------------------------------- Comment (by Tamar Christina <tamar@…>): In [changeset:"1f366b8d15feaa05931bd2d81d8b0c5bae92f3b8/ghc" 1f366b8d/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="1f366b8d15feaa05931bd2d81d8b0c5bae92f3b8" Add delete retry loop. [ci skip] Summary: On Windows we have to retry the delete a couple of times. The reason for this is that a `FileDelete` command just marks a file for deletion. The file is really only removed when the last handle to the file is closed. Unfortunately there are a lot of system services that can have a file temporarily opened using a shared readonly lock, such as the built in AV and search indexer. We can't really guarantee that these are all off, so what we can do is whenever after a `rmtree` the folder still exists to try again and wait a bit. Based on what I've seen from the tests on CI server, is that this is relatively rare. So overall we won't be retrying a lot. If after a reasonable amount of time the folder is still locked then abort the current test by throwing an exception, this so it won't fail with an even more cryptic error. The issue is that these services often open a file using `FILE_SHARE_DELETE` permissions. So they can seemingly be removed, and for most intended purposes they are, but recreating the file with the same name will fail as the FS will prevent data loss. The MSDN docs for `DeleteFile` says: ``` The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED. ``` Retrying seems to be a common pattern, SQLite has it in their driver http://www.sqlite.org/src/info/89f1848d7f The only way to avoid this is to run each way of a test in it's own folder. This would also have the added bonus of increased parallelism. Reviewers: austin, bgamari Reviewed By: bgamari Subscribers: thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2936 GHC Trac Issues: #12661, #13162 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12661#comment:22> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC