#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by enolan): * Attachment "break011.run.stdout" added. break011 output - master branch -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by enolan): * Attachment "PatBind.comp.stderr" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by enolan): * Attachment "plugins01.run.stdout" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by enolan): * Attachment "T9405.run.stdout" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Changes (by enolan): * Attachment "T11223_simple_duplicate_lib.run.stderr" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by Phyx-): Hi @enolan, Windows hasn't validated cleanly in a long while. We're working on getting it to, but for now it doesn't. The GHC 8.0.1 tests for `rts/T11223` should all pass, except if you tested for `x86` since there's a slight difference in the output. I'll have to change my sanitize function to account for it. in terms of head: ghci.debugger/scripts break011 - New output just needs to be accepted ghci/prog003 prog003 - Known segfault, there's a ticket for this already partial-sigs/should_compile PatBind [exit code non-0] (normal) plugins plugins01 [bad exit code] (normal) - These two I haven't looked into yet rts T7037 [bad stdout] (normal) - This is a sporadic failure due to Python and how the files are flushed. There's a diff to fix this but hasn't been committed yet/ rts T9405 [bad exit code] (normal) rts testmblockalloc [bad exit code] (normal) - These two might be new, don't remember them.. rts/T11223 T11223_simple_duplicate_lib [bad exit code] (normal) - This will be fixed soon, the test output was changed but the Windows test results not updated. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | ---------------------------------+---------------------------------------- Comment (by Phyx-): `T9405` seems to be a timing issue. the process is killed after 0.2s and it seems that when it is killed the RTS hasn't finished processing the flags in RtsFlags. Increasing the timeout makes it work but I'm having some issue with the bash commands. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures ---------------------------------+---------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2684 Wiki Page: | ---------------------------------+---------------------------------------- Changes (by Phyx-): * differential: => Phab:D2684 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures -------------------------------------+------------------------------------- Reporter: enolan | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2684 Wiki Page: | Phab:D2759 -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"0ce59be3a2723f814a3e929fd32a44ff4e890a49/ghc" 0ce59be/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="0ce59be3a2723f814a3e929fd32a44ff4e890a49" Fix testsuite threading, timeout, encoding and performance issues on Windows In a land far far away, a project called Cygwin was born. Cygwin used newlib as it's standard C library implementation. But Cygwin wanted to emulate POSIX systems as closely as possible. So it implemented `execv` using the Windows function `spawnve`. Specifically ``` spawnve (_P_OVERLAY, path, argv, cur_environ ()) ``` `_P_OVERLAY` is crucial, as it makes the function behave *sort of* like execv on linux. the child process replaces the original process. With one major difference because of the difference in process models on Windows: the original process signals the caller that it's done. this is why the file is still locked. because it's still running, control was returned because the parent process was destroyed, but the child is still running. I think it's just pure dumb luck, that the older runtimes are slow enough to give the process time to terminate before we tried deleting the file. Which explains why you do have sporadic failures even on older runtimes like 2.5.0, of a test or two (like T7307). So this patch fixes a couple of things. I leverage the existing `timeout.exe` to implement a workaround for this issue. a) The old timeout used to start the process then assign it to the job. This is slightly faulty since child processes are only assigned to a job is their parent were assigned at the time they started. So this was a race condition. I now create the process suspended, assign it to the job and then resume it. Which means all child processes are not running under the same job. b) First things, Is to prevent dangling child processes. I mark the job with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` so when the last process in the job is done, it insures all processes under the job are killed. c) Secondly, I change the way we wait for results. Instead of waiting for the parent process to terminate, I wait for the job itself to terminate. There's a slight subtlety there, we can't wait on the job itself. Instead we have to create an I/O Completion port and wait for signals on it. See https://blogs.msdn.microsoft.com/oldnewthing/20130405-00/?p=4743 This fixes the issues on all runtimes for me and makes T7307 pass consistenly. The threading was also simplified by hiding all the locking in a single semaphore and a completion class. Futhermore some additional error reporting was added. For encoding the testsuite now no longer passes a file handle to the subprocess since on windows, sh.exe seems to acquire a lock on the file that is not released in a timely fashion. I suspect this because cygwin seems to emulate console handles by creating file handles and using those for std handles. So when we give it an existing file handle it just locks the file. I what's happening is that it's not releasing the handle until all shared cygwin processes are dead. Which explains why it worked in single threaded mode. So now instead we pass a pipe and do not interpret the resulting data. Any bytes written to stdin or read out of stdout/stderr are done so in binary mode and we do not interpret the data. The reason for this is that we have encoding tests in GHC which pass invalid utf-8. If we try to handle the data as text then python will throw an exception instead of a test comparison failing. Also I have fixed the ability to override `PYTHON` when calling `make tests`. This now works the same as with `.\validate`. Finally, after cleaning up the locks I was able to make the abort behavior work correctly as I believe it was intended: when you press Ctrl+C and send an interrupt signal, the testsuite finishes the active tests and then gracefully exits showing you a report of the progress it did make. So using Ctrl+C will not just *die* as it did before. These changes lift the restriction on which python version you use (msys/mingw) or which runtime or python 3 or python 2. All combinations should now be supported. Test Plan: PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH && PYTHON=/usr/bin/python THREADS=9 make test THREADS=9 make test PATH=/usr/local/bin:/mingw64/bin:$APPDATA/cabal/bin:$PATH && PYTHON=/usr/bin/python ./validate --quiet --testsuite-only Reviewers: erikd, RyanGlScott, bgamari, austin Subscribers: jrtc27, mpickering, thomie, #ghc_windows_task_force Differential Revision: https://phabricator.haskell.org/D2684 GHC Trac Issues: #12725, #12554, #12661, #12004 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12004: Windows unexpected failures -------------------------------------+------------------------------------- Reporter: enolan | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: Component: Compiler | Version: Resolution: fixed | Keywords: Operating System: Windows | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2684 Wiki Page: | Phab:D2759 -------------------------------------+------------------------------------- Changes (by Phyx-): * status: new => closed * resolution: => fixed Comment: None of these currently fail on HEAD. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12004#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler