[GHC] #11980: Testsuite: run each test in its own /tmp directory, after copying required files

26 Apr 2016

      #11980: Testsuite: run each test in its own /tmp directory, after copying required
files
-------------------------------------+-------------------------------------
           Reporter:  thomie         |             Owner:  thomie
               Type:  task           |            Status:  new
           Priority:  normal         |         Milestone:  8.2.1
          Component:  Test Suite     |           Version:  8.0.1
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  None/Unknown
  Unknown/Multiple                   |
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 == Plan ==
 For each TEST:
 * create a directory `<testdir>` inside `/tmp`
 * link/copy all source files that the test needs into `<testdir>`
 * run the test inside `<testdir>`
 * delete `<testdir>`

 == Benefits ==

 * running tests in parallel always works, even when two tests write to the
 same file (for example .hi/.o files, from sharing a .hs file). This should
 reduce the number of unsuspected and annoying validate failures. There is
 no longer any need to specify `-outputdir` explicitly.

 * there is no longer any need to add entries to the perpetually out-of-
 date `testsuite/.gitignore`.

 * there is no longer any need to specify `extra_clean` or `clean_cmd`
 setup functions for tests. Sometimes you //will// have to specify which
 extra files a test needs, using the (new) `extra_files` setup function,
 but it will be immediately clear when to do so (i.e. the test won't run if
 you don't). By default only files with the name `<testname>*` are copied
 to `<testdir>`.

 * it could become possible to run //ways// for a single test in parallel
 (i.e. `make TEST=<testname> slow` would become faster)

 * `clean`ing the testsuite won't be necessary (except for a few files in
 `testsuite/mk` perhaps)

 Currently the only way to clean the testsuite is via `make CLEANUP=1
 CLEAN_ONLY=1`, which requires a call to the testsuite, which requires
 building mk/ghc-config first.

 There are 2 problems with this:
 * you don't expect a call to `make clean` to start building stuff. Even
 less so in a pristine source distribution.
 * most of the time it doesn't clean all files, because lots of tests are
 missing are have incomplete `extra_clean`s.

--
Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11980
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler