Re: Parallel building multiple targets

On Wed, Jan 22, 2014 at 12:25 AM, Simon Marlow wrote:

On 05/01/2014 23:48, John Lato wrote:

...

(FYI, I expect I'm the source of the suggestion that ghc -M is broken)

First, just to clarify, I don't think ghc -M is obviously broken. Rather, I think it's broken in subtle, unobvious ways, such that trying to develop a make-based project with ghc -M will fail at various times in a non-obvious fashion, at least without substantial additional rules.

If I understand you correctly, you're not saying that ghc -M is broken, but that it would be easier to use if it did more. Right? Maybe you could make specific suggestions? Saying it is "broken" is a bit FUD-ish. We use it in GHC's build system, so by an existence proof it is certainly not broken.

This is more-or-less true. To be a bit more precise, I'm saying that the raw output from ghc -M is insufficient in several interesting/useful cases. I'm also not convinced that it's worth anyone's time to add the necessary features to ghc -M to cover these cases, or indeed that it's even necessarily the correct place to fix it. Some specific shortcomings of ghc -M are: 1. no support for .cmm file dependencies (though in fairness, --make doesn't track these either) 2. no support for Language.Haskell.TH.Syntax.addDependentFile 3. no support for preprocessing, such as .hsc or #include'd files. (I think it would work to write a .hs file that uses -pgmF to call hsc2hs, but that seems rather non-obvious and I'm not sure it's a good idea). However, these are all rather obviously fixable as part of the build system. For me, the worst problems have to do with cleaning. If you're using a Makefile, typically you want to leave intermediate object files around and only rebuild them when the sources have changed. However, there are various issues with ghc batch-mode that make this difficult (e.g. https://ghc.haskell.org/trac/ghc/ticket/8029 ). The workarounds to deal with this are not as straightforward. The alternative is to live with the occasional build error that can only be fixed by blowing away the entire build dir (a remedy that I often need with ghc's source tree, as even make maintainer-clean doesn't always cut it. Hopefully my experience here is unique, but I do not believe it is). Also, the most common use case seems to be for parallel building of modules. As ghc-7.8 provides this with --make, I'd expect the demand for ghc -M will be greatly reduced. That's why I'm not certain it's worth the time it would take to resolve these issues. Cheers, John

Cheers, Simon

For an example of some of the extra steps necessary to make

...

something like this work, see e.g. https://github.com/nh2/multishake (which is admittedly for a more complicated setup, and also has some issues). The especially frustrating part is, just when you think you have everything working, someone wants to add some other tool to a workflow (hsc2hs, .cmm files, etc), and your build system doesn't support it.

ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build. Of course you could do the same without cabal, but it's a convenient way to create a common build directory and manage multiple targets. This is the approach I would take to building multiple executables from the same source files.

ghc doesn't do any locking of build files AFAIK. Running parallel ghc commands for two main modules that have the same import, using the same working directory, is not safe. In pathological cases the two different main modules may even generate different code *for the imported module*. This sort of situation can arise with the IncoherentInstances extension, for example.

The obvious approach is of course to make a library out of your common files. This has the downsides of requiring a bit more work on the developer's part, but if the common files are relatively stable it'll probably lead to the fastest builds of your executables. Also in this case you could run multiple `ghc --make`s in parallel, using different build directories, since they won't be rebuilding any common code.

John L.

On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes mailto:sami.liedes@iki.fi> wrote:

Hi,

I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD.

As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles.

However I have a question about ghc --make that is also relevant without parallel ghc --make:

If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other?

Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in parallel?

Sami

[1] http://stackoverflow.com/questions/20938894/generating- correct-link-dependencies-for-ghc-and-makefile-style-builds

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users