Make-based incremental build for arch-haskell
Hi, I have created a fairly simple GNUmakefile to perform incremental builds of the arch-haskell repository (a.k.a. "habs"). The code can be accessed by means of Git as follows: git clone http://git.cryp.to/arch-haskell The repository is very small -- currently, it builds only pandoc, hledger, and uulib (plus all packages that these three depend on). The set of packages is determined by a bunch of symbolic links that are part of the repository: ~/src/arch-haskell$ ls -l src/*.cabal lrwxrwxrwx 1 simons users 27 Oct 22 13:44 src/binary.cabal -> binary/0.5.0.2/binary.cabal lrwxrwxrwx 1 simons users 19 Oct 22 13:44 src/csv.cabal -> csv/0.1.1/csv.cabal lrwxrwxrwx 1 simons users 27 Oct 22 13:44 src/digest.cabal -> digest/0.0.0.8/digest.cabal lrwxrwxrwx 1 simons users 28 Oct 22 13:44 src/hledger.cabal -> hledger/0.12.1/hledger.cabal lrwxrwxrwx 1 simons users 36 Oct 22 13:44 src/hledger-lib.cabal -> hledger-lib/0.12.1/hledger-lib.cabal [...] As you can see, these links refer to the Hackage database, which is supposed to be copied into the src/ directory as well. The make target "update" will do that automatically. Given those files, "make all" ... - generates a PKGBUILD for every Cabal package, - compile the package, and - and registers the package in the Pacman database. If you'd like to try out the build, run these commands in a freshly checked-out copy of the http://git.cryp.to/arch-haskell repository: # Populate the src/ directory with a recent version of Hackage. make update # Unregister Haskell packages from Pacman to make sure that the # following builds don't have any hidden dependencies. make clean # Build all packages (pandoc, hledger, and uulib). make Instead of building all targets, it's also possible to build only selected targets by running "make haskell-zlib" or "make hledger". Now, the GNUmakefile needs to map package names from Cabal to ArchLinux. Currently, this is accomplished using a hard-coded list. For example, hledger needs the following variable assignment hledger_name = hledger to tell make that this package deviates from the normal "haskell-NAME" scheme. Obviously, this list ought to be generated automatically from the information contained in the haskell-Ashlin's library, but I haven't yet automated the task. Furthermore, make needs dependency information about the packages. This know-how, too, is currently hard-coded in the GNUmakefile. For example: $(pandoc_tarball) : $(HTTP_tarball) $(xhtml_tarball) $(texmath_tarball) $(zip-archive_tarball) $(network_tarball) : $(parsec_tarball) $(HTTP_tarball) : $(network_tarball) $(texmath_tarball) : $(xml_tarball) $(parsec_tarball) $(zip-archive_tarball) : $(digest_tarball) $(zlib_tarball) $(binary_tarball) Maintaining that information manually is out of question, IMHO, and we really ought to generate that information from the Cabal files. I'm not yet sure how to do it, though. Anyway, let me know what you think. Take care, Peter
On 2010/10/22 Peter Simons
As you can see, these links refer to the Hackage database, which is supposed to be copied into the src/ directory as well. The make target "update" will do that automatically. Given those files, "make all" ...
- generates a PKGBUILD for every Cabal package,
Remember that except for dependency management you can use manycabal2arch for this kind of task.
- compile the package, and
Due to variations in makepkg.conf, the packages generated by makepkg may end up in a different directory. I suggest you mention what you are expecting to find in makepkg.conf to make it work. In addition, you may want to build packages in a chroot, because we need to check whether external dependencies (C libraries) are correctly encoded in the PKGBUILD.
- and registers the package in the Pacman database.
Now, the GNUmakefile needs to map package names from Cabal to ArchLinux. Currently, this is accomplished using a hard-coded list. For example, hledger needs the following variable assignment
hledger_name = hledger
This name can be obtained by the following Haskell function: archname :: GenericPackageDescription -> IO String archname cabal = do sysProvides <- getDefaultSystemProvides case preprocess cabal sysProvides of Nothing -> return "" Just pkg -> arch_pkgname $ fst $ cabal2pkg pkg sysProvides
to tell make that this package deviates from the normal "haskell-NAME" scheme. Obviously, this list ought to be generated automatically from the information contained in the haskell-Ashlin's library, but I haven't yet automated the task.
Furthermore, make needs dependency information about the packages. This know-how, too, is currently hard-coded in the GNUmakefile. For example:
$(pandoc_tarball) : $(HTTP_tarball) $(xhtml_tarball) $(texmath_tarball) $(zip-archive_tarball) $(network_tarball) : $(parsec_tarball) $(HTTP_tarball) : $(network_tarball) $(texmath_tarball) : $(xml_tarball) $(parsec_tarball) $(zip-archive_tarball) : $(digest_tarball) $(zlib_tarball) $(binary_tarball)
Maintaining that information manually is out of question, IMHO, and we really ought to generate that information from the Cabal files. I'm not yet sure how to do it, though.
This is mainly why I chose shell scripts over Makefiles for my own system. Dependency checks should not necessarily be done according to the modification time of file, but according to the pkgrel (which should be bumped when necessary). I would personally add a "make depend" rule that runs an external (Haskell or shell) program that generates the needed info in a separate file depends.mk. It is not necessary that depends.mk uses the Cabal names, since everything will be expanded anyway. So you could use something like #!/bin/bash #This is makedepends.sh for pkg in * do source $pkg/PKGBUILD echo -n "\$(${pkg}_tarball):" for dep in $depends[@] $makedepends[@] do dep=${dep%>*} dep=${dep%=*} dep=${dep%<*} [ -d "$dep" ] && echo -n " \$(${dep}_tarball)" done echo done -- Rémy.
Hi Rémy,
- generates a PKGBUILD for every Cabal package,
Remember that except for dependency management you can use manycabal2arch for this kind of task.
yes, that it is true, but manycabal2arch doesn't scale nearly as well on multi-core machines as "make -j<n>" will.
- compile the package, and
Due to variations in makepkg.conf, the packages generated by makepkg may end up in a different directory.
What variations do you mean exactly?
In addition, you may want to build packages in a chroot, because we need to check whether external dependencies (C libraries) are correctly encoded in the PKGBUILD.
Unfortunately, I don't know much about chroot builds in ArchLinux, so it's not obvious to me how to add that feature. It sure sounds like a good idea, though.
hledger_name = hledger
This name can be obtained by the following Haskell function: [...]
It would be nice if cabeal2arch would have a flag that could be used to obtain that information, as in the following (imaginary) examples: $ cabal2arch --arch-name hledger hledger $ cabal2arch --arch-name HUnit haskell-hunit
$(pandoc_tarball) : $(HTTP_tarball) $(xhtml_tarball) $(texmath_tarball) $(zip-archive_tarball) $(network_tarball) : $(parsec_tarball) $(HTTP_tarball) : $(network_tarball) $(texmath_tarball) : $(xml_tarball) $(parsec_tarball) $(zip-archive_tarball) : $(digest_tarball) $(zlib_tarball) $(binary_tarball)
Maintaining that information manually is out of question, IMHO, and we really ought to generate that information from the Cabal files.
This is mainly why I chose shell scripts over Makefiles for my own system.
I don't see how that would make much of a difference. An accurate dependency graph is required no matter whether bash or make is used to build the packages. The two tools just differ in the way that they represent those dependencies. IMHO, make is clearly superior for this kind of task because it's been designed to traverse dependency graphs.
Dependency checks should not necessarily be done according to the modification time of file, but according to the pkgrel (which should be bumped when necessary).
Well, cabal2arch generally sets pkgrel=1, and I am not aware of any way to change that behavior. So I don't see how anyone could use $pkgrel for anything at all, really? Am I missing something?
#!/bin/bash #This is makedepends.sh for pkg in * do source $pkg/PKGBUILD echo -n "\$(${pkg}_tarball):" for dep in $depends[@] $makedepends[@] do dep=${dep%>*} dep=${dep%=*} dep=${dep%<*} [ -d "$dep" ] && echo -n " \$(${dep}_tarball)" done echo done
I would prefer to generate the dependency graph from the Cabal files, because the information found in PKGBUILDs cannot be considered reliable until the test builds have succeeded. It seems to be easy enough to write a short cabal2make converter, so from the looks of it that's the solution I favor at the moment. Take care, Peter
On Fri, Oct 22, 2010 at 15:35, Peter Simons
>> hledger_name = hledger > > This name can be obtained by the following Haskell function: [...]
It would be nice if cabeal2arch would have a flag that could be used to obtain that information, as in the following (imaginary) examples:
$ cabal2arch --arch-name hledger hledger
$ cabal2arch --arch-name HUnit haskell-hunit
Raise a bug against cabal2arch on github. I'm in the middle of wiring in some better command line parsing into cabal2arch so requests like this one will hopefully not be too difficult to satisfy in the future. /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus＠therning．org Jabber: magnus＠therning．org http://therning.org/magnus identi.ca|twitter: magthe
participants (3)
-
Magnus Therning -
Peter Simons -
Rémy Oudompheng