On Thu, Feb 16, 2012 at 10:10 AM, Johan Tibell wrote:

I don't think we want to run the tests of all dependent packages e.g. if I do

   cabal install --enable-tests --only-dependencies

if that's what you're trying to do. Or at least we need a way to say install all dependencies of this package, including its test dependencies.

No, that's not what I'm trying to do. I had introduced a patch that ran tests for target packages and aborted installation if the tests failed. For example,

cabal install --enable-tests network

would install all network's dependencies (but not their tests), then build, test, and install network. It didn't change how dependencies were handled at all. However, the test runner in Cabal wasn't designed to be invoked from cabal-install: it sometimes exits on success, which cabal-install interprets as an error, even when it isn't. This was stopping packages without tests from being installed with 'cabal install --enable-tests', which shouldn't happen. -- Thomas Tuegel

Hi Thomas.

would install all network's dependencies (but not their tests), then build, test, and install network. It didn't change how dependencies were handled at all. However, the test runner in Cabal wasn't designed to be invoked from cabal-install: it sometimes exits on success, which cabal-install interprets as an error, even when it isn't. This was stopping packages without tests from being installed with 'cabal install --enable-tests', which shouldn't happen.

Actually, the new solver will filter out --enable-tests for packages that don't define any. If the old solver could do the same, would that be an alternative to patch the Cabal lib? Cheers, Andres -- Andres Löh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com

Can you explain why the modular solver filters out '--enable-tests' for packages without tests? It doesn't seem like it should affect the generated dependency list.

It's the natural thing to do. The modular solver will not even generate a choice node in the search tree if there aren't any test stanzas, so an --enable-test constraint will simply have no effect on such a package. The only other option would be to generate such a node for every single package, but that'd just make the search tree larger for no benefit.

(I still want to patch the Cabal lib for this because it made me aware of the fact that the 'test' phase behaves differently from every other phase, but now I will postpone it until I do the other fixes the test runner needs.)

Right. I agree. But it should become an independent fix then, right? Cheers, Andres -- Andres Löh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com

Hi Thomas.

On Sun, Feb 12, 2012 at 5:55 AM, Andres Löh wrote:

...

Modular solver support for optional stanzas is done. Regarding your patches, there's one aspect I don't understand:

In Distribution/Client/Install.hs:

   configBenchmarks = toFlag False,    configTests = toFlag (TestStanzas `elem` stanzas)

You first had configBenchmarks analogous to configTests, then changed it back. I don't fully understand why the two should be handled differently. Could you please explain?

These lines control whether a package is actually built with its tests and benchmarks. Although the resolver pulls in benchmark dependencies now, I didn't enable actually building the benchmarks because we wouldn't do anything with the results anyway.

Actually, since we won't be running test suites automatically (until the next release of Cabal), they should probably be disabled, too.

I'm still not sure if I understand this. If you enable tests and benchmarks, shouldn't they then at least be installed? Whereas if you pull in the dependencies, but then don't build them, what did you pull in the dependencies for?

I'm getting this warning in the modular solver:

...

Distribution/Client/Dependency/Modular/Message.hs:80:1:     Warning: Pattern match(es) are non-exhaustive              In an equation for `showFR':                  Patterns not matched: _ (MalformedStanzaChoice _)

It seems innocuous, but you would know better than I. Otherwise, everything looks good.

It's an internal error that shouldn't occur, but nevertheless a case I missed. Thanks for spotting it. Now fixed.

I'm working on the patch for Cabal that we need in order to run tests automatically. Hopefully, I'll send it to the list this afternoon. Then, after the cabal-install release, we can turn automatic tests back on.

Got it. Is it actually wise to run tests automatically and fail the installation if tests fail? Don't you think there might be users who'd like to have the test suites installed, be able to run them in their own time, and look closely at the results, even if some of them might fail? Cheers, Andres -- Andres Löh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com

Hi Thomas.

...

...

Actually, since we won't be running test suites automatically (until the next release of Cabal), they should probably be disabled, too.

I'm still not sure if I understand this. If you enable tests and benchmarks, shouldn't they then at least be installed? Whereas if you pull in the dependencies, but then don't build them, what did you pull in the dependencies for?

The dependencies are pulled in for the case where you do 'cabal install --enable-benchmarks --only-dependencies' to quickly pull in all the deps for a project you are developing.

Yes, I was under the mistaken assumption that tests and benchmarks are installed, too.

...

...

I'm working on the patch for Cabal that we need in order to run tests automatically. Hopefully, I'll send it to the list this afternoon. Then, after the cabal-install release, we can turn automatic tests back on.

Got it. Is it actually wise to run tests automatically and fail the installation if tests fail? Don't you think there might be users who'd like to have the test suites installed, be able to run them in their own time, and look closely at the results, even if some of them might fail?

Now I understand what you mean. However: tests and benchmarks never get installed. They never live outside the build directory, so if they don't get run before install takes place, they'll be deleted with the build directory. This is why I don't know what we'd do with benchmarks if we built them: we could run them, but what would we do with the results?

So, let me ask whether it's really the right decision that tests and benchmarks are never installed? I agree that they're *mainly* useful for development, but aren't there cases where one might want to have tests and benchmarks be run on a client machines, as part of debugging a problem? Cheers, Andres -- Andres Löh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com

What I think they want to say is

   cabal configure-and-build-why-does-cabal-make-me-write-so-much-stuff-manually --enable-tests --also-pull-in-deps

but we don't really have a way to say that.

Aha. I'm looking for something similar from time to time. I think (despite being misnamed) something like cabal install --stop-after-building --enable-tests in a development directory might be what I (and perhaps you) want. I'd also like cabal get/clone/whatever pkgname that's similar to "cabal unpack" but uses the repository information from the .cabal file to clone the development repository. Anyway, all the patches regarding --enable-test from this thread are now in the cabal-install repository. Testing and further feedback welcome :) Cheers, Andres -- Andres Löh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com