[GHC] #11980: Testsuite: run each test in its own /tmp directory, after copying required files
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- == Plan == For each TEST: * create a directory `<testdir>` inside `/tmp` * link/copy all source files that the test needs into `<testdir>` * run the test inside `<testdir>` * delete `<testdir>` == Benefits == * running tests in parallel always works, even when two tests write to the same file (for example .hi/.o files, from sharing a .hs file). This should reduce the number of unsuspected and annoying validate failures. There is no longer any need to specify `-outputdir` explicitly. * there is no longer any need to add entries to the perpetually out-of- date `testsuite/.gitignore`. * there is no longer any need to specify `extra_clean` or `clean_cmd` setup functions for tests. Sometimes you //will// have to specify which extra files a test needs, using the (new) `extra_files` setup function, but it will be immediately clear when to do so (i.e. the test won't run if you don't). By default only files with the name `<testname>*` are copied to `<testdir>`. * it could become possible to run //ways// for a single test in parallel (i.e. `make TEST=<testname> slow` would become faster) * `clean`ing the testsuite won't be necessary (except for a few files in `testsuite/mk` perhaps) Currently the only way to clean the testsuite is via `make CLEANUP=1 CLEAN_ONLY=1`, which requires a call to the testsuite, which requires building mk/ghc-config first. There are 2 problems with this: * you don't expect a call to `make clean` to start building stuff. Even less so in a pristine source distribution. * most of the time it doesn't clean all files, because lots of tests are missing are have incomplete `extra_clean`s. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: patch Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * status: new => patch * differential: => Phab:D1187 @@ -33,7 +33,1 @@ - building mk/ghc-config first. - - There are 2 problems with this: - * you don't expect a call to `make clean` to start building stuff. Even - less so in a pristine source distribution. - * most of the time it doesn't clean all files, because lots of tests are - missing are have incomplete `extra_clean`s. + building mk/ghc-config first. This will all become much simpler. New description: == Plan == For each TEST: * create a directory `<testdir>` inside `/tmp` * link/copy all source files that the test needs into `<testdir>` * run the test inside `<testdir>` * delete `<testdir>` == Benefits == * running tests in parallel always works, even when two tests write to the same file (for example .hi/.o files, from sharing a .hs file). This should reduce the number of unsuspected and annoying validate failures. There is no longer any need to specify `-outputdir` explicitly. * there is no longer any need to add entries to the perpetually out-of- date `testsuite/.gitignore`. * there is no longer any need to specify `extra_clean` or `clean_cmd` setup functions for tests. Sometimes you //will// have to specify which extra files a test needs, using the (new) `extra_files` setup function, but it will be immediately clear when to do so (i.e. the test won't run if you don't). By default only files with the name `<testname>*` are copied to `<testdir>`. * it could become possible to run //ways// for a single test in parallel (i.e. `make TEST=<testname> slow` would become faster) * `clean`ing the testsuite won't be necessary (except for a few files in `testsuite/mk` perhaps) Currently the only way to clean the testsuite is via `make CLEANUP=1 CLEAN_ONLY=1`, which requires a call to the testsuite, which requires building mk/ghc-config first. This will all become much simpler. -- -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: patch Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"3f3dc23ea64573a12e2f4bfdaaa3aa536ad3188d/ghc" 3f3dc23/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="3f3dc23ea64573a12e2f4bfdaaa3aa536ad3188d" Testsuite: run tests in /tmp after copying required files Major change to the testsuite driver. For each TEST: * create a directory `<testdir>` inside `/tmp`. * link/copy all source files that the test needs into `<testdir>`. * run the test inside `<testdir>`. * delete `<testdir>` Extra files are (temporarily) tracked in `testsuite/driver/extra_files.py`, but can also be specified using the `extra_files` setup function. Differential Revision: https://phabricator.haskell.org/D1187 Reviewed by: Rufflewind, bgamari Trac: #11980 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: patch Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"1a9ae4b39c057d2a21192f6be033c8545702f345/ghc" 1a9ae4b3/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="1a9ae4b39c057d2a21192f6be033c8545702f345" Testsuite: delete old cleanup code (#11980) }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * status: patch => closed * resolution: => fixed -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"2230c8822233d6d68f930170cd51d96169649056/ghc" 2230c882/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="2230c8822233d6d68f930170cd51d96169649056" Testsuite: fix T12010 for real * Use `extra_files` instead of (the deprecated) `extra_clean` (#11980). * Don't depend on generated files from build tree (libraries/base/include/HsBaseConfig.h). Running 'make test TEST=T12010' should work, even without building GHC first (it will use the system installed ghc). Test Plan: 'make test TEST=T12010' on Linux and Windows. Reviewed by: Phyx Differential Revision: https://phabricator.haskell.org/D2256 GHC Trac Issues: #12010 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"35053eb6c3f9058020f9818bf0de672aef6894c9/ghc" 35053eb6/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="35053eb6c3f9058020f9818bf0de672aef6894c9" Testsuite: delete check_files_written The CHECK_FILES_WRITTEN feature is no longer necessary, since tests don't write to the source directory anymore (#11980). Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2162 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"36d254afadb49a855dea031c036b1a371a567f16/ghc" 36d254af/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="36d254afadb49a855dea031c036b1a371a567f16" Testsuite: run tests in /tmp/ghctest-xxx instead of /tmp/ghctest/xxx This fixes a bug of not being able to create files or directories in /tmp/ghctest if it was created by a different user. Trac ticket #11980. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"f72f23f9f6ff2914ec99fc86f67c89927f18ba47/ghc" f72f23f/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="f72f23f9f6ff2914ec99fc86f67c89927f18ba47" Testsuite: run tests in <testdir>.run instead of /tmp As discussed in Phab:D1187, this approach makes it a bit easier to inspect the test directory while working on a new test. The only tests that needed changes are the ones that refer to files in ancestor directories. Those files are now copied directly into the test directory. validate still runs the tests in a temporary directory in /tmp, see `Note [Running tests in /tmp]` in testsuite/driver/runtests.py. Update submodule hpc. Reviewed by: simonmar Differential Revision: https://phabricator.haskell.org/D2333 GHC Trac Issues: #11980 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11980: Testsuite: run each test in its own /tmp directory, after copying required files -------------------------------------+------------------------------------- Reporter: thomie | Owner: thomie Type: task | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D1187 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"3b49f8faa1cbd3a04f1d6aa817a315a853e0cd79/ghc" 3b49f8fa/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="3b49f8faa1cbd3a04f1d6aa817a315a853e0cd79" Testsuite: remove `-fforce-recomp` from default flags (#11980) There is no need for this flag anymore, since each test runs in a newly created directory. Removing it cleans up testlib.py a bit. There is a small risk that this renders some tests useless. It's hard to know. Those tests should have specified -fforce-recomp` explicitly anyway, so I'm not going to worry about it. I've fixed the ones that failed without -fforce-recomp. Reviewed by: bgamari Differential Revision: https://phabricator.haskell.org/D2346 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC