Parallel building multiple targets
Hi, I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD. As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles. However I have a question about ghc --make that is also relevant without parallel ghc --make: If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other? Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in parallel? Sami [1] http://stackoverflow.com/questions/20938894/generating-correct-link-dependen...
(FYI, I expect I'm the source of the suggestion that ghc -M is broken) First, just to clarify, I don't think ghc -M is obviously broken. Rather, I think it's broken in subtle, unobvious ways, such that trying to develop a make-based project with ghc -M will fail at various times in a non-obvious fashion, at least without substantial additional rules. For an example of some of the extra steps necessary to make something like this work, see e.g. https://github.com/nh2/multishake (which is admittedly for a more complicated setup, and also has some issues). The especially frustrating part is, just when you think you have everything working, someone wants to add some other tool to a workflow (hsc2hs, .cmm files, etc), and your build system doesn't support it. ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build. Of course you could do the same without cabal, but it's a convenient way to create a common build directory and manage multiple targets. This is the approach I would take to building multiple executables from the same source files. ghc doesn't do any locking of build files AFAIK. Running parallel ghc commands for two main modules that have the same import, using the same working directory, is not safe. In pathological cases the two different main modules may even generate different code *for the imported module*. This sort of situation can arise with the IncoherentInstances extension, for example. The obvious approach is of course to make a library out of your common files. This has the downsides of requiring a bit more work on the developer's part, but if the common files are relatively stable it'll probably lead to the fastest builds of your executables. Also in this case you could run multiple `ghc --make`s in parallel, using different build directories, since they won't be rebuilding any common code. John L. On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes@iki.fi> wrote:
Hi,
I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD.
As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles.
However I have a question about ghc --make that is also relevant without parallel ghc --make:
If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other?
Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in parallel?
Sami
[1] http://stackoverflow.com/questions/20938894/generating-correct-link-dependen...
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
John Lato wrote:
ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build.
As long as the ghc-options flags are the same for all the different component sections (I've been bitten by this). Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/
On Sun, Jan 5, 2014 at 3:54 PM, Erik de Castro Lopo <mle+hs@mega-nerd.com>wrote:
John Lato wrote:
ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build.
As long as the ghc-options flags are the same for all the different component sections (I've been bitten by this).
Yes, good point. This is one case where putting the common code in a library will help a lot.
On 05/01/2014 23:48, John Lato wrote:
(FYI, I expect I'm the source of the suggestion that ghc -M is broken)
First, just to clarify, I don't think ghc -M is obviously broken. Rather, I think it's broken in subtle, unobvious ways, such that trying to develop a make-based project with ghc -M will fail at various times in a non-obvious fashion, at least without substantial additional rules.
If I understand you correctly, you're not saying that ghc -M is broken, but that it would be easier to use if it did more. Right? Maybe you could make specific suggestions? Saying it is "broken" is a bit FUD-ish. We use it in GHC's build system, so by an existence proof it is certainly not broken. Cheers, Simon For an example of some of the extra steps necessary to make
something like this work, see e.g. https://github.com/nh2/multishake (which is admittedly for a more complicated setup, and also has some issues). The especially frustrating part is, just when you think you have everything working, someone wants to add some other tool to a workflow (hsc2hs, .cmm files, etc), and your build system doesn't support it.
ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build. Of course you could do the same without cabal, but it's a convenient way to create a common build directory and manage multiple targets. This is the approach I would take to building multiple executables from the same source files.
ghc doesn't do any locking of build files AFAIK. Running parallel ghc commands for two main modules that have the same import, using the same working directory, is not safe. In pathological cases the two different main modules may even generate different code *for the imported module*. This sort of situation can arise with the IncoherentInstances extension, for example.
The obvious approach is of course to make a library out of your common files. This has the downsides of requiring a bit more work on the developer's part, but if the common files are relatively stable it'll probably lead to the fastest builds of your executables. Also in this case you could run multiple `ghc --make`s in parallel, using different build directories, since they won't be rebuilding any common code.
John L.
On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes@iki.fi <mailto:sami.liedes@iki.fi>> wrote:
Hi,
I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD.
As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles.
However I have a question about ghc --make that is also relevant without parallel ghc --make:
If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other?
Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in parallel?
Sami
[1] http://stackoverflow.com/questions/20938894/generating-correct-link-dependen...
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org <mailto:Glasgow-haskell-users@haskell.org> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
On Wed, Jan 22, 2014 at 12:25 AM, Simon Marlow <marlowsd@gmail.com> wrote:
On 05/01/2014 23:48, John Lato wrote:
(FYI, I expect I'm the source of the suggestion that ghc -M is broken)
First, just to clarify, I don't think ghc -M is obviously broken. Rather, I think it's broken in subtle, unobvious ways, such that trying to develop a make-based project with ghc -M will fail at various times in a non-obvious fashion, at least without substantial additional rules.
If I understand you correctly, you're not saying that ghc -M is broken, but that it would be easier to use if it did more. Right? Maybe you could make specific suggestions? Saying it is "broken" is a bit FUD-ish. We use it in GHC's build system, so by an existence proof it is certainly not broken.
This is more-or-less true. To be a bit more precise, I'm saying that the raw output from ghc -M is insufficient in several interesting/useful cases. I'm also not convinced that it's worth anyone's time to add the necessary features to ghc -M to cover these cases, or indeed that it's even necessarily the correct place to fix it. Some specific shortcomings of ghc -M are: 1. no support for .cmm file dependencies (though in fairness, --make doesn't track these either) 2. no support for Language.Haskell.TH.Syntax.addDependentFile 3. no support for preprocessing, such as .hsc or #include'd files. (I think it would work to write a .hs file that uses -pgmF to call hsc2hs, but that seems rather non-obvious and I'm not sure it's a good idea). However, these are all rather obviously fixable as part of the build system. For me, the worst problems have to do with cleaning. If you're using a Makefile, typically you want to leave intermediate object files around and only rebuild them when the sources have changed. However, there are various issues with ghc batch-mode that make this difficult (e.g. https://ghc.haskell.org/trac/ghc/ticket/8029 ). The workarounds to deal with this are not as straightforward. The alternative is to live with the occasional build error that can only be fixed by blowing away the entire build dir (a remedy that I often need with ghc's source tree, as even make maintainer-clean doesn't always cut it. Hopefully my experience here is unique, but I do not believe it is). Also, the most common use case seems to be for parallel building of modules. As ghc-7.8 provides this with --make, I'd expect the demand for ghc -M will be greatly reduced. That's why I'm not certain it's worth the time it would take to resolve these issues. Cheers, John
Cheers, Simon
For an example of some of the extra steps necessary to make
something like this work, see e.g. https://github.com/nh2/multishake (which is admittedly for a more complicated setup, and also has some issues). The especially frustrating part is, just when you think you have everything working, someone wants to add some other tool to a workflow (hsc2hs, .cmm files, etc), and your build system doesn't support it.
ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build. Of course you could do the same without cabal, but it's a convenient way to create a common build directory and manage multiple targets. This is the approach I would take to building multiple executables from the same source files.
ghc doesn't do any locking of build files AFAIK. Running parallel ghc commands for two main modules that have the same import, using the same working directory, is not safe. In pathological cases the two different main modules may even generate different code *for the imported module*. This sort of situation can arise with the IncoherentInstances extension, for example.
The obvious approach is of course to make a library out of your common files. This has the downsides of requiring a bit more work on the developer's part, but if the common files are relatively stable it'll probably lead to the fastest builds of your executables. Also in this case you could run multiple `ghc --make`s in parallel, using different build directories, since they won't be rebuilding any common code.
John L.
On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes@iki.fi <mailto:sami.liedes@iki.fi>> wrote:
Hi,
I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD.
As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles.
However I have a question about ghc --make that is also relevant without parallel ghc --make:
If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other?
Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in parallel?
Sami
[1] http://stackoverflow.com/questions/20938894/generating- correct-link-dependencies-for-ghc-and-makefile-style-builds
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org <mailto:Glasgow-haskell-users@haskell.org> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
On 23/01/14 03:52, John Lato wrote:
However, these are all rather obviously fixable as part of the build system. For me, the worst problems have to do with cleaning. If you're using a Makefile, typically you want to leave intermediate object files around and only rebuild them when the sources have changed. However, there are various issues with ghc batch-mode that make this difficult (e.g. https://ghc.haskell.org/trac/ghc/ticket/8029 ). The workarounds to deal with this are not as straightforward. The alternative is to live with the occasional build error that can only be fixed by blowing away the entire build dir (a remedy that I often need with ghc's source tree, as even make maintainer-clean doesn't always cut it. Hopefully my experience here is unique, but I do not believe it is).
You said "various issues", but you've only mentioned *one* specific issue so far: #8029, and we concluded that was not a bug, although I do see how it could require manually deleting a .hi file if you have a module that shadows a package module and then remove it. This seems a rare occurrence to me, but perhaps it is something you do often. If it really hurts, then you could have a way to tell your build system about a file when it is removed from the project, so that it can delete the build artifacts that go with it. Anyway, are there other problems you'd like to bring to our attention? Cheers, Simon
Also, the most common use case seems to be for parallel building of modules. As ghc-7.8 provides this with --make, I'd expect the demand for ghc -M will be greatly reduced. That's why I'm not certain it's worth the time it would take to resolve these issues.
Cheers, John
Cheers, Simon
For an example of some of the extra steps necessary to make
something like this work, see e.g. https://github.com/nh2/__multishake <https://github.com/nh2/multishake> (which is admittedly for a more complicated setup, and also has some issues). The especially frustrating part is, just when you think you have everything working, someone wants to add some other tool to a workflow (hsc2hs, .cmm files, etc), and your build system doesn't support it.
ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build. Of course you could do the same without cabal, but it's a convenient way to create a common build directory and manage multiple targets. This is the approach I would take to building multiple executables from the same source files.
ghc doesn't do any locking of build files AFAIK. Running parallel ghc commands for two main modules that have the same import, using the same working directory, is not safe. In pathological cases the two different main modules may even generate different code *for the imported module*. This sort of situation can arise with the IncoherentInstances extension, for example.
The obvious approach is of course to make a library out of your common files. This has the downsides of requiring a bit more work on the developer's part, but if the common files are relatively stable it'll probably lead to the fastest builds of your executables. Also in this case you could run multiple `ghc --make`s in parallel, using different build directories, since they won't be rebuilding any common code.
John L.
On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes@iki.fi <mailto:sami.liedes@iki.fi> <mailto:sami.liedes@iki.fi <mailto:sami.liedes@iki.fi>>> wrote:
Hi,
I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD.
As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles.
However I have a question about ghc --make that is also relevant without parallel ghc --make:
If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other?
Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in parallel?
Sami
[1] http://stackoverflow.com/__questions/20938894/generating-__correct-link-depe... <http://stackoverflow.com/questions/20938894/generating-correct-link-dependencies-for-ghc-and-makefile-style-builds>
_________________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.__org <mailto:Glasgow-haskell-users@haskell.org> <mailto:Glasgow-haskell-users@__haskell.org <mailto:Glasgow-haskell-users@haskell.org>> http://www.haskell.org/__mailman/listinfo/glasgow-__haskell-users <http://www.haskell.org/mailman/listinfo/glasgow-haskell-users>
_________________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.__org <mailto:Glasgow-haskell-users@haskell.org> http://www.haskell.org/__mailman/listinfo/glasgow-__haskell-users <http://www.haskell.org/mailman/listinfo/glasgow-haskell-users>
On Jan 23, 2014 1:28 AM, "Simon Marlow" <marlowsd@gmail.com> wrote:
On 23/01/14 03:52, John Lato wrote:
However, these are all rather obviously fixable as part of the build system. For me, the worst problems have to do with cleaning. If you're using a Makefile, typically you want to leave intermediate object files around and only rebuild them when the sources have changed. However, there are various issues with ghc batch-mode that make this difficult (e.g. https://ghc.haskell.org/trac/ghc/ticket/8029 ). The workarounds to deal with this are not as straightforward. The alternative is to live with the occasional build error that can only be fixed by blowing away the entire build dir (a remedy that I often need with ghc's source tree, as even make maintainer-clean doesn't always cut it. Hopefully my experience here is unique, but I do not believe it is).
You said "various issues", but you've only mentioned *one* specific issue
so far: #8029, and we concluded that was not a bug, although I do see how it could require manually deleting a .hi file if you have a module that shadows a package module and then remove it. This seems a rare occurrence to me, but perhaps it is something you do often. If it really hurts, then you could have a way to tell your build system about a file when it is removed from the project, so that it can delete the build artifacts that go with it. It seems uncommon until you're developing in a branch that does so, and try to go back and forth between that branch and another.
Anyway, are there other problems you'd like to bring to our attention?
If the one bug I linked to earlier is to be closed as "not a bug" (seems correct to me), there doesn't seem much point to raising other issues relating to out-of-date intermediate files. The general solution is exactly as you suggested, leading to an increasingly baroque build system. I don't think any of this detracts from my original thrust, which is that something that looks like an afternoon's work is much more complicated. Plus, you'll end up fighting with/hacking on a build system instead of what you meant to work on.
Cheers, Simon
Also, the most common use case seems to be for parallel building of modules. As ghc-7.8 provides this with --make, I'd expect the demand for ghc -M will be greatly reduced. That's why I'm not certain it's worth the time it would take to resolve these issues.
Cheers, John
Cheers, Simon
For an example of some of the extra steps necessary to make
something like this work, see e.g. https://github.com/nh2/__multishake
<https://github.com/nh2/multishake> (which is admittedly for a
more
complicated setup, and also has some issues). The especially frustrating part is, just when you think you have everything working, someone wants to add some other tool to a workflow (hsc2hs, .cmm files, etc), and your build system doesn't support it.
ghc --make doesn't allow building several binaries in one run, however if you use cabal all the separate runs will use a shared build directory, so subsequent builds will be able to take advantage of the intermediate output of the first build. Of course you could do the same without cabal, but it's a convenient way to create a common build directory and manage multiple targets. This is the approach I
would
take to building multiple executables from the same source files.
ghc doesn't do any locking of build files AFAIK. Running parallel ghc commands for two main modules that have the same import, using the same working directory, is not safe. In pathological cases the two different main modules may even generate different code *for the imported module*. This sort of situation can arise with the IncoherentInstances extension, for example.
The obvious approach is of course to make a library out of your common files. This has the downsides of requiring a bit more work on
the
developer's part, but if the common files are relatively stable it'll probably lead to the fastest builds of your executables. Also in this case you could run multiple `ghc --make`s in parallel, using different build directories, since they won't be rebuilding any common
code.
John L.
On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes@iki.fi <mailto:sami.liedes@iki.fi> <mailto:sami.liedes@iki.fi <mailto:sami.liedes@iki.fi>>> wrote:
Hi,
I have a Haskell project where a number of executables are produced from mostly the same modules. I'm using a Makefile to enable parallel builds. I received advice[1] that ghc -M is broken, but that there is parallel ghc --make in HEAD.
As far as I can tell, ghc --make does not allow building several binaries in one run, so I think it may not still be a full replacement for Makefiles.
However I have a question about ghc --make that is also relevant without parallel ghc --make:
If I have two main modules, prog1.hs and prog2.hs, which have mutual dependencies (for example, both import A from A.hs), is it safe to run "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there some kind of locking to prevent both from building module A at the same time and interfering with each other?
Is there a good way (either in current releases or HEAD) to build multiple binaries partially from the same sources in
parallel?
Sami
[1]
http://stackoverflow.com/__questions/20938894/generating-__correct-link-depe...
<
http://stackoverflow.com/questions/20938894/generating-correct-link-dependen...
_________________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.__org <mailto:Glasgow-haskell-users@haskell.org> <mailto:Glasgow-haskell-users@__haskell.org <mailto:Glasgow-haskell-users@haskell.org>> http://www.haskell.org/__mailman/listinfo/glasgow-__haskell-users<;
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users>
_________________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.__org <mailto:Glasgow-haskell-users@haskell.org> http://www.haskell.org/__mailman/listinfo/glasgow-__haskell-users<;
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users>
participants (4)
-
Erik de Castro Lopo -
John Lato -
Sami Liedes -
Simon Marlow