#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: new Priority: low | Milestone: 8.6.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by monoidal): There's already a Python version (timeout.py), it's apparently needed sometimes on Windows; see TIMEOUT_PROGRAM in Makefile. This specialcasing is from 2009, it's likely it can be removed but it needs checking. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: new Priority: low | Milestone: 8.6.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by lantti: Old description:

When trying to understand the GHC test suite I noticed some small improvements that I could do to it. This would probably work nicely as my first task on GHC:

Rewrite the timeout.hs in python, it doesn't seem to do anything that would strictly need the Haskell libraries to be used and it is the only part of the test suite driver that is not written in python.

Or;

See if the timeout scripts could be eliminated altogether as the python subprocess module that we are using can now (since python 3.3) handle timeouts by itself and using the timeout scripts effectively doubles the number of processes we need to create for each test case. Notice that the timeout scripts do more than just generating the timeout.

New description: When trying to understand the GHC test suite I noticed some small improvements that I could do to it. This would probably work nicely as my first task on GHC: Rewrite the timeout.hs in python, it is used for Windows runs but it doesn't seem to do anything that would strictly need the Haskell libraries to be used and it is the only part of the test suite driver that is not written in python. Or; See if the timeout scripts could be eliminated altogether as the python subprocess module that we are using can now (since python 3.3) handle timeouts by itself and using the timeout scripts effectively doubles the number of processes we need to create for each test case. Notice that the timeout scripts do more than just generating the timeout. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: new Priority: low | Milestone: 8.6.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by lantti): Using winpty instead of mintty on Windows allows the ctrl-c event to propagate through the process tree, but even then the clean up routines are not run and the results from the tests already completed are not displayed. At the moment I presume this is because the timeout.hs is not able to react to the ctrl-c event while waiting on the child process. When run in isolation it only reacts after the child returns (or never if the child ignores ctrl-c). As part of the testsuite it doesn't get a chance to react before being killed. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: new Priority: low | Milestone: 8.6.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Great news. Do you have a patch to share? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: new Priority: low | Milestone: 8.6.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Phyx-): timeout.hs goes through great lengths to be able to reliably detect when all child processes exit using job groups. replacing this with the standard python process api will re-introduce threading issues we spent a great deal of time debugging so be careful with that. It also deals with quite a few corner cases with regards to how msys implements pipes. You would essentially have to rewrite it using ctypes because python's `os` package only provides process grouping for posix OSes. Before such a patch is accepted I'd like to see some evidence that it works correctly when the system is under heavy load. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: patch Priority: low | Milestone: 8.8.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by lantti): * status: new => patch Comment: I was playing around some more with the Windows process groups but eventually went the Job Objects way. The Windows console process groups are supported by subprocess module and os module, resulting in an almost total elimination of the diverging code path and working to my satisfaction on a light load, but unfortunately not working under mintty without additional winpty wrapping and still breaking apart on heavier loads due to Windows Console signals not being guaranteed to be delivered in such cases. The Job Objects on the other hand require a long diverging code path, but were absolutely robust against my limited stress tests. A preliminary patch is submitted to Phabricator: https://phabricator.haskell.org/D5107 Some simple tests that I used to evaluate that: https://github.com/lantti/ghc/tree/testsuite-tests/testsuite/tests/aaa The patch follows the approach from timeout.hs for Windows and uses subprocess module for Posix. As the Windows code path doesn't use subprocess anymore the IO redirection and environment string generation are handled by this patch too. All in all the diverging code paths didn't get shorter, unlike I had hoped. According to my few test runs the testsuite runs slightly faster on Posix now, on Windows I don't have any coherent results of it running faster or slower. Also interrupting the test run on Windows is as sketchy as it used to be, it only works because mintty kills our process tree for us. No signals get delivered to our testsuite driver and none of our cleanup or results collecting code runs. Some related observations I made while working on this: - in addition to the signals/console incompatibility between mintty-mingw and native Windows programs, current versions of mintty-mingw have problems handling unicode environment variables, the variables get mangled already before they are passed to the python interpreter. - our Posix code (before and after the patch) leave child process management to the test process in cases of normal termination and keyboard interrupt, meaning that if the test program itself does not signal and wait for its children to terminate then nobody will. The only case where the testsuite driver takes action and signals the process group is after a timeout. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: patch Priority: low | Milestone: 8.8.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5107 Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonpj): * differential: => Phab:D5107 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: patch Priority: low | Milestone: 8.8.1 Component: Test Suite | Version: 8.4.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5107 Wiki Page: | -------------------------------------+------------------------------------- Comment (by lantti): I agree with this summary. To add some more (possibly superfluous) details, the original problem is not only Windows specific. In order to maintain a consistent interface the current POSIX code path also shells out an external process (a separate python interpreter) to handle the timeout and interrupting functionality even when {{{subprocess}}} functionality would be adequate on POSIX. To issue 1.: I have nothing to add. To issue 2.: I'd like to add that even {{{timeout.hs}}} itself is not fully functional at the moment, failing to handle interrupts signals, although {{{winbindings.py}}} doesn't do any better job either. Interrupting the test run on Windows works only because {{{mintty}}} simply kills our whole process tree without consulting our code at all, resulting in different but still effective end to the test run (compared to POSIX). The results for the tests already run won't get displayed in that case but fortunately this functionality doesn't strike me as essential. To issues 3,4,5: If this is the direction we should be going I'll happily align my efforts with it instead. I was under the impression that currently hs->py was the preferred way as according to the git logs (e5063a042c9a1701ea7273da7bacb530d5c077d3) GHC used to have a testsuite language and appropriate interpreter implemented all in Haskell. The stated motivation to move away from the Haskell code was maintainability but presumably we have a lot better Haskell libraries now than back then and the maintainability would perhaps not become such an issue anymore. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15363: Do some cleaning up of the testsuite driver -------------------------------------+------------------------------------- Reporter: lantti | Owner: lantti Type: task | Status: closed Priority: low | Milestone: 8.8.1 Component: Test Suite | Version: 8.4.3 Resolution: wontfix | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5107 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Phyx-):

To issue 2.: I'd like to add that even timeout.hs itself is not fully functional at the moment, failing to handle interrupts signals, although winbindings.py doesn't do any better job either. Interrupting the test run on Windows works only because mintty simply kills our whole process tree without consulting our code at all, resulting in different but still effective end to the test run (compared to POSIX). The results for the tests already run won't get displayed in that case but fortunately this functionality doesn't strike me as essential.

This isn't a testsuite problem. It's simply because mintty suppresses all windows signals in order to implement the pseudo posix signal handlers. Mintty is simply incompatible with windows programs. That's why interfaces like winpty exist but that are ultimately a hack. The simplest solution here is simply not to use mintty. Any other terminal emulator and call bash directly. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15363#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler