Hi all, SimonPJ asked me to summarise the problems I've been having with GHC's new build system with the idea that we could perhaps talk a bit about this during today's IRC meeting. The context is the Data Parallel Haskell project and in particular the dph packages which need a number of not quite straightforward things from the build system (this is only going to get worse in the future). A lot of the problems stem from Cabal's lack of extensibility. This means that whenever I need some new Cabal functionality for dph I have to ask Ian or Duncan to implement it. They both are very helpful but this just doesn't scale! IMO, at the moment Cabal is simply not mature enough for building complex projects. But wait, we do have those Setup.hs files which are customisable so why don't I just do what I want with them? Firstly, they aren't customisable enough. And secondly, I have no idea how to do what I want. The development version of Cabal changes on a daily basis and isn't documented. For instance, the black magic that Ian implemented in dph/dph/Setup.hs a month ago (and which I don't really understand) has been broken once already. Fortunately, I was able to fix that myself but if something more complex changes, I'll have to ask Ian again. In effect, I'm not able to properly maintain my own library! I don't think I understand how GHC itself is built any longer, either. What does cabal-bin do? What is runghc.wrapper? What is the difference between make.library.* and build.library.* and why do we need both (and why doesn't one of them work for dph)? It is important to have an idea how the build system works when hacking on a project but how do I find out? And finally, there is the problem of the HEAD being broken quite frequently. I have decided to pull as little as possible and it turns out that others have independently arrived at the same conclusion. This is bad! For instance, I would have spotted the OS X breakage much earlier if I pulled more frequently; now, I have to wade through a couple of weeks' worth of patches to find the culprit (that darcs unpull doesn't work in my repo doesn't help). IMO, the main issue is that we have two moving targets: the GHC build system and the tool it relies on (Cabal). Both change rapidly and there is just no chance, for me at least, to keep up. If I understand correctly, the situation is this: - Ultimately, Cabal will be used for building GHC. - At the moment, Cabal doesn't provide the necessary functionality. The current approach seems to be to gradually implement stuff in Cabal and to immediately use it in the build system. I think this just doesn't work for a highly collaborative project such as GHC. I have two suggestions how to improve things. 1. I think it would be better to implement, document and release all the necessary functionality in Cabal first and only then to modify GHC's build system to use this functionality. In effect, this means that the Cabal version used for building GHC would be frozen at the beginning of a development cycle and not at the end. Critical stuff could still be merged, perhaps, but that should happen very infrequently and be backwards compatible. 2. Ian, it would be great if you could add a couple of lines to build system patches explaining what they do and why. This could ultimately mean less work for you because I will bug you less often :-) Sorry for the long rant, I simply seem to spend too much time wrestling with the build system. Roman
For infrequent updaters like myself, it would also be nice just to have a
HEADS UP before and after periods of larger than usual instability
(such as build system replacements).
And, talking about #ghc, I've got the suspicion that this channel is
partly to blame for the occasional disappearance of critical information:
did I just miss the messages here on cvs-ghc or was the discussion on
#ghc instead? With no logs for #ghc to browse, I've got no idea what
useful information #ghc inhabitants assume to be widely known!-)
Couldn't lambdabot or some other bot file daily logs for #ghc somewhere,
similar to the logs for #haskell? Then I could check whether my issues
are new or well-known.
I used to bug folks here with my "head doesn't build" messages before it
finally dawned on me that the system is in such a state of upheaval that it
will simply take time for things to settle down (rendering my failure
messages redundant). Now I'm just waiting for the HEADS UP that tells
us that things are back to normal, and that 'make;make binary-dist' is
supposed to work again (on windows):
http://www.haskell.org/pipermail/cvs-ghc/2008-July/043797.html
http://www.haskell.org/pipermail/cvs-ghc/2008-July/043798.html
Claus
----- Original Message -----
From: "Roman Leschinskiy"
Hi all,
SimonPJ asked me to summarise the problems I've been having with GHC's new build system with the idea that we could perhaps talk a bit about this during today's IRC meeting. The context is the Data Parallel Haskell project and in particular the dph packages which need a number of not quite straightforward things from the build system (this is only going to get worse in the future).
A lot of the problems stem from Cabal's lack of extensibility. This means that whenever I need some new Cabal functionality for dph I have to ask Ian or Duncan to implement it. They both are very helpful but this just doesn't scale! IMO, at the moment Cabal is simply not mature enough for building complex projects.
But wait, we do have those Setup.hs files which are customisable so why don't I just do what I want with them? Firstly, they aren't customisable enough. And secondly, I have no idea how to do what I want. The development version of Cabal changes on a daily basis and isn't documented. For instance, the black magic that Ian implemented in dph/dph/Setup.hs a month ago (and which I don't really understand) has been broken once already. Fortunately, I was able to fix that myself but if something more complex changes, I'll have to ask Ian again. In effect, I'm not able to properly maintain my own library!
I don't think I understand how GHC itself is built any longer, either. What does cabal-bin do? What is runghc.wrapper? What is the difference between make.library.* and build.library.* and why do we need both (and why doesn't one of them work for dph)? It is important to have an idea how the build system works when hacking on a project but how do I find out?
And finally, there is the problem of the HEAD being broken quite frequently. I have decided to pull as little as possible and it turns out that others have independently arrived at the same conclusion. This is bad! For instance, I would have spotted the OS X breakage much earlier if I pulled more frequently; now, I have to wade through a couple of weeks' worth of patches to find the culprit (that darcs unpull doesn't work in my repo doesn't help).
IMO, the main issue is that we have two moving targets: the GHC build system and the tool it relies on (Cabal). Both change rapidly and there is just no chance, for me at least, to keep up. If I understand correctly, the situation is this:
- Ultimately, Cabal will be used for building GHC.
- At the moment, Cabal doesn't provide the necessary functionality.
The current approach seems to be to gradually implement stuff in Cabal and to immediately use it in the build system. I think this just doesn't work for a highly collaborative project such as GHC. I have two suggestions how to improve things.
1. I think it would be better to implement, document and release all the necessary functionality in Cabal first and only then to modify GHC's build system to use this functionality. In effect, this means that the Cabal version used for building GHC would be frozen at the beginning of a development cycle and not at the end. Critical stuff could still be merged, perhaps, but that should happen very infrequently and be backwards compatible.
2. Ian, it would be great if you could add a couple of lines to build system patches explaining what they do and why. This could ultimately mean less work for you because I will bug you less often :-)
Sorry for the long rant, I simply seem to spend too much time wrestling with the build system.
Roman
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Claus Reinke wrote:
For infrequent updaters like myself, it would also be nice just to have a HEADS UP before and after periods of larger than usual instability (such as build system replacements). And, talking about #ghc, I've got the suspicion that this channel is partly to blame for the occasional disappearance of critical information: did I just miss the messages here on cvs-ghc or was the discussion on #ghc instead?
We don't send announcements or "heads-up"-type messages to #ghc, so don't worry you're not missing anything by not watching the channel 24/7.
With no logs for #ghc to browse, I've got no idea what useful information #ghc inhabitants assume to be widely known!-) Couldn't lambdabot or some other bot file daily logs for #ghc somewhere, similar to the logs for #haskell? Then I could check whether my issues are new or well-known.
Hmm, I thought we *did* have IRC logs, but I'm not sure where they're put. We certainly used to have them. Perhaps someone responsible for lambdabot knows? Cheers, Simon
We can talk about the general issues on IRC. But I thought I'd answer a few of the specific questions quickly: Roman Leschinskiy wrote:
I don't think I understand how GHC itself is built any longer, either. What does cabal-bin do?
It's effectively a pre-compiled Setup.hs for packages that have no need for a custom Setup.hs. cabal-install does the same job for end-users, but we can't rely on having cabal-install in the GHC build system.
What is runghc.wrapper?
it's a template used to make a shell wrapper for a binary. There seems to be new functionality in Cabal to support this.
What is the difference between make.library.* and build.library.* and why do we need both
This one is my fault. build.* runs 'cabal-bin build', whereas make.* runs 'cabal-bin makefile' followed by 'make'. That is, build.* ends up using ghc --make, whereas make.* ends up using a traditional makefile with ghc -M for dependencies and individual single-module compilations. The latter was added so that we could (a) use make -j and (b) compile single modules for testing/debugging purposes. We don't *need* both - the build system normally only uses make.* (and it would also work perfectly well using build.*, except you'd get no parallelism). One day make.* will go away when Cabal provides the necessary functionality via 'cabal build'.
(and why doesn't one of them work for dph)?
no idea about that, sorry. Cheers, Simon
On Wed, 2008-07-30 at 14:46 +0100, Simon Marlow wrote:
I don't think I understand how GHC itself is built any longer, either. What does cabal-bin do?
It's effectively a pre-compiled Setup.hs for packages that have no need for a custom Setup.hs. cabal-install does the same job for end-users, but we can't rely on having cabal-install in the GHC build system.
The point being that linking default Setup.hs scripts all the time is a waste (especially since it doesn't parallelise).
What is runghc.wrapper?
it's a template used to make a shell wrapper for a binary. There seems to be new functionality in Cabal to support this.
Ian added this the other day. I can't say I understand it yet or if it's generally applicable to all packages. Obviously the intention is to allow relocatable binaries on unix systems by using wrapper scripts. Duncan
On 30/07/2008, at 23:46, Simon Marlow wrote:
We can talk about the general issues on IRC. But I thought I'd answer a few of the specific questions quickly:
Thanks, Simon!
Roman Leschinskiy wrote:
I don't think I understand how GHC itself is built any longer, either. What does cabal-bin do?
It's effectively a pre-compiled Setup.hs for packages that have no need for a custom Setup.hs. cabal-install does the same job for end- users, but we can't rely on having cabal-install in the GHC build system.
I see. So it looks at the Build-Type in the package description and calls the right defaultMain if it's not Custom. And if my Setup.hs isn't standard then it's my responsibility to set the Build-Type to Custom in the .cabal file and this isn't checked, right? I wonder, though. What exactly does this buy us? Duncan says:
The point being that linking default Setup.hs scripts all the time is a waste (especially since it doesn't parallelise).
But the time spent in compiling those Setup.hs is negligible and don't understand the "it doesn't parallelise" bit.
What is runghc.wrapper?
it's a template used to make a shell wrapper for a binary. There seems to be new functionality in Cabal to support this.
I see. Is runghc the only program we do this for? Or will others be added gradually?
What is the difference between make.library.* and build.library.* and why do we need both
This one is my fault. build.* runs 'cabal-bin build', whereas make.* runs 'cabal-bin makefile' followed by 'make'. That is, build.* ends up using ghc --make, whereas make.* ends up using a traditional makefile with ghc -M for dependencies and individual single-module compilations. The latter was added so that we could (a) use make -j and (b) compile single modules for testing/debugging purposes.
Ok, this answers the following question:
(and why doesn't one of them work for dph)?
Because it does non-standard stuff in Setup.hs and hence no Makefile can be generated. Hence the SUBDIRS_BUILD hackery in libraries/ Makefile, I assume. Thanks again! Roman
On Thu, Jul 31, 2008 at 12:55:09AM +1000, Roman Leshchinskiy wrote:
I see. So it looks at the Build-Type in the package description and calls the right defaultMain if it's not Custom. And if my Setup.hs isn't standard then it's my responsibility to set the Build-Type to Custom in the .cabal file and this isn't checked, right?
Nothing checks that Setup.hs behaves as the Build-Type claims it does, no.
I wonder, though. What exactly does this buy us? Duncan says:
The point being that linking default Setup.hs scripts all the time is a waste (especially since it doesn't parallelise).
But the time spent in compiling those Setup.hs is negligible
The time taken and disk space used weren't entirely negligible, especially on arches without object splitting. I think it also makes the Makefiles simpler overall.
What is runghc.wrapper?
it's a template used to make a shell wrapper for a binary. There seems to be new functionality in Cabal to support this.
I see. Is runghc the only program we do this for? Or will others be added gradually?
No, hsc2hs, ghc-pkg and ghc also have wrappers. I'm adding them as I move utils over to build with Cabal rather than the old build system. The reason there was a patch just adding a wrapper for runghc is that I forgot to "darcs add" it in an earlier patch. n.b. these wrappers are essentially the same as the old C and shell script wrappers that the old build system made.
(and why doesn't one of them work for dph)?
Because it does non-standard stuff in Setup.hs and hence no Makefile can be generated. Hence the SUBDIRS_BUILD hackery in libraries/ Makefile, I assume.
Exactly. Thanks Ian
participants (6)
-
Claus Reinke -
Duncan Coutts -
Ian Lynagh -
Roman Leschinskiy -
Roman Leshchinskiy
-
Simon Marlow