#16026: No way to run a subset of the testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: feature request | Status: new Priority: high | Milestone: 8.8.1 Component: Build System | Version: 8.6.2 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by alpmestan): Indeed, at the moment we support specifying a test with `--only`, and this feature is probably getting closer and closer to the top of our priority queue, with the work we've all been doing lately. I may take a stab at implementing this. Do we want to keep specifying all the tests through an env var? A CLI argument? Both? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16026#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#16026: No way to run a subset of the testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: feature request | Status: new Priority: high | Milestone: 8.8.1 Component: Build System | Version: 8.6.2 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by alpmestan): I'm afraid we want the "list of tests to run" to be supplied at runtime, as rebuilding hadrian whenever I want to look at, say, only a subset of those 10 tests that I was running before, would be a bit annoying. The way I see it, `UserSettings` are useful when you want to do things that you'd usually do in `build.mk`, i.e add some extra options for GHC or the C compiler here and there, things like that. It's not something you'll be editing a lot while working on a particular patch, probably more of a "edit once at the beginning and not much afterwards" kind of workflow. While names of tests to run can definitely change from a minute to the next, when you're looking into group of failing tests "per theme" for example. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16026#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#16026: No way to run a subset of the testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: feature request | Status: new Priority: high | Milestone: 8.8.1 Component: Build System | Version: 8.6.2 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by alpmestan): Well, it's not much different from `make TEST="t123 t456 t789"`, is it? We can support both quite easily, really. The important thing is that you have a quick way to take the testsuite output that lists the failing tests and ask hadrian to run just those, or a only subset. We might have to do something about the `TEST` environment variable to make sure that it makes it through the `build.*` scripts, to the hadrian executable. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16026#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#16026: No way to run a subset of the testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: alpmestan Type: feature request | Status: new Priority: high | Milestone: 8.8.1 Component: Build System | Version: 8.6.2 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): Just happened to come across this. As an avid user of the feature you're describing, I would want a way to do this on the command line. Other options are good, too (e.g. setting an env var so that way I can run the same tests over and over), but inessential (for me). Separate from the `make TEST=...` functionality, I also make regular use of the fact that running `make` in different subdirectories of `tests` runs only subsets. For example, I'll go into `testsuite/tests/typecheck` and run `make` just to run type-checking tests. Sometimes I even just want the `should_fail` directory of that. I would certainly miss the ability to run subsets like this if it were gone. It's so easy (just change dirs) and so useful. Thanks for all the work you're putting into this! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16026#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#16026: No way to run a subset of the testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: alpmestan Type: feature request | Status: patch Priority: high | Milestone: 8.8.1 Component: Build System | Version: 8.6.2 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5431 Wiki Page: | -------------------------------------+------------------------------------- Changes (by alpmestan): * status: new => patch * differential: => Phab:D5431 Comment: Got a patch ([https://phabricator.haskell.org/D5431 D5431]) that makes hadrian support `TEST="..." hadrian/build.sh test` and/or `hadrian/build.sh test --only="..."`. Both can be used at the same time and hadrian will happily concatenate both lists of tests and run them all. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16026#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#16026: No way to run a subset of the testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: alpmestan Type: feature request | Status: closed Priority: high | Milestone: 8.8.1 Component: Build System | Version: 8.6.2 (Hadrian) | Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5431 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16026#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler