Hi Gour,

Let's first see who wants to be part of the 'team' or everything was just one-man-band.

I would like to contribute some time and effort. There are a number of Haskell packages on AUR that I need in my daily work, and I'd sure be able to help keeping them (and their dependencies) up-to-date. I wonder is what is going to happen to the arch-haskell user on AUR? Don, would you be willing to share the password so that other people you can actually update packages on AUR? Also, it would be great if the e-mail address of the arch-haskell user could point to this mailing list so that everyone who is subscribed here receives out-of-date notifications and comments that are left on AUR. Having that information publicly available for everyone would greatly improve our ability to function as a team.

Magnus> Do we give up on having all of Hackage in AUR and instead rely Magnus> on tools like bauerbill?

This is interesting idea...

Bauerbill's Hackage support works great. However, in my understanding Bauerbill will generally install the latest version of every package. This is a problem for us, because sometimes we need to withhold updates for a while until all users of the package have updated their packages to cope with the latest version. This is easy to do on AUR, where we can choose which packages to update in which order, but I don't see how that could be accomplished with Bauerbill, unless Xyne is willing to add some major features to the program.

Magnus> Do we try to do something like what Xyne suggested--set up a Magnus> Haskell ABS and publish pre-compiled packages in [arch-haskell]?

...however, in the spirit of Arch (in comparison with the Gentoo which I left 3yrs ago), I consider that having kind of 'Haskell overlay' with binary packages would be very nice...

Yes, I agree that this is probably the best solution. Arch was designed to work that way. How difficult is it to set up a repository? Does anyone know how to do that? Take care, Peter

Magnus Therning :

On 08/10/10 23:04, Peter Simons wrote:

...

...

...

Do we try to do something like what Xyne suggested--set up a Haskell ABS and publish pre-compiled packages in [arch-haskell]?

...however, in the spirit of Arch (in comparison with the Gentoo which I left 3yrs ago), I consider that having kind of 'Haskell overlay' with binary packages would be very nice...

Yes, I agree that this is probably the best solution. Arch was designed to work that way. How difficult is it to set up a repository? Does anyone know how to do that?

I think there are a few Arch devs reading this list, at least a TU or two, hopefully they can offer some more information.

AFAIU it's not much to it really.  An ABS-like tree (perhaps kept in a Darcs repo for extra points :-) and a place to upload binary packages and the repo DB to.

I support that idea. I would really like to host that on the same server as [community] and [multilib] repos, if the total size of the repositories is not too big, but I fear it would be several gigabytes. I imagine: - keeping ghc + several basic modules in extra/community for bootstrapping - a darcs repo one for the scripts * a script that builds packages (possibly inspired by devtools, for example an analogue of archbuild taking multiple arguments) * a script handling rebuilds (build all reverse dependencies of something in the right order) possibly just an option to the previous script * a script that updates a given collection of PKGBUILDs by calling cabal2arch * a script that moves built packages to the repositories (like commitpkg in devtools) - a darcs repo for the PKGBUILDs * one dir per package, and subdirs $package/trunk, $package/repo (holding the current WIP version of the PKGBUILD and one corresponding to the binary package in repo) * people are supposed to do only a partial checkout of the darcs repo, I know Git can do that, but that said, a full working copy is only a few thousand files. Is darcs as efficient as Git for storage ? I expect the transfer size for a full cloning to be less than 5MB. Leaving xmonad and its dependencies in the main repos in probably fine. -- Rémy

On Sat, 09 Oct 2010 22:13:50 +0100, Magnus Therning wrote:

On 09/10/10 09:43, Rémy Oudompheng wrote:

...

Magnus Therning :

...

On 08/10/10 23:04, Peter Simons wrote:

...

...

...

Do we try to do something like what Xyne suggested--set up a Haskell ABS and publish pre-compiled packages in [arch-haskell]?

...however, in the spirit of Arch (in comparison with the Gentoo which I left 3yrs ago), I consider that having kind of 'Haskell overlay' with binary packages would be very nice...

Yes, I agree that this is probably the best solution. Arch was designed to work that way. How difficult is it to set up a repository? Does anyone know how to do that?

I think there are a few Arch devs reading this list, at least a TU or two, hopefully they can offer some more information.

AFAIU it's not much to it really. An ABS-like tree (perhaps kept in a Darcs repo for extra points :-) and a place to upload binary packages and the repo DB to.

I support that idea. I would really like to host that on the same server as [community] and [multilib] repos, if the total size of the repositories is not too big, but I fear it would be several gigabytes. I imagine: - keeping ghc + several basic modules in extra/community for bootstrapping - a darcs repo one for the scripts * a script that builds packages (possibly inspired by devtools, for example an analogue of archbuild taking multiple arguments) * a script handling rebuilds (build all reverse dependencies of something in the right order) possibly just an option to the previous script * a script that updates a given collection of PKGBUILDs by calling cabal2arch * a script that moves built packages to the repositories (like commitpkg in devtools) - a darcs repo for the PKGBUILDs

1. Are you suggesting we keep binary versions of *all* of hackage in a repo, or 2. we keep PKGBUILDs for all of hackage in an ABS tree, and only provide binaries for a subset of packages?

...

* one dir per package, and subdirs $package/trunk, $package/repo (holding the current WIP version of the PKGBUILD and one corresponding to the binary package in repo) * people are supposed to do only a partial checkout of the darcs repo, I know Git can do that, but that said, a full working copy is only a few thousand files. Is darcs as efficient as Git for storage ? I expect the transfer size for a full cloning to be less than 5MB.

It's worth clarifying here that while git does support partial checkout it doesn't support partial cloning. darcs supports lazy cloning, and I think git does as well.

Just to get some numbers I downloaded the cabal files for for the latest version of all packages on Hackage. Then I ran cabal2arch on it all. After that I attempted to put the results in darcs and git.

Adding all files, 100 at a time:

* darcs: 522.18s user 5.49s system 99% cpu 8:48.96 total * git: 1.90s user 0.77s system 97% cpu 2.726 total

Record/commit of initial changeset:

* darcs: NA, it seems frozen with: 5548 done, 5480 queued * git: 0.33s user 0.32s system 57% cpu 1.128 total

I would suggest to use git for this kind of larges repos, however darcs will be a lot happier with smaller patches (even one per new file), making tags from time to time helps as well since it trunks the inventory and makes some operations cheaper. -- Nicolas Pouillard http://nicolaspouillard.fr

Rémy Oudompheng wrote:

I imagine: - keeping ghc + several basic modules in extra/community for bootstrapping - a darcs repo one for the scripts * a script that builds packages (possibly inspired by devtools, for example an analogue of archbuild taking multiple arguments) * a script handling rebuilds (build all reverse dependencies of something in the right order) possibly just an option to the previous script * a script that updates a given collection of PKGBUILDs by calling cabal2arch * a script that moves built packages to the repositories (like commitpkg in devtools)

As a proof of concept, here is a set of scripts I used today: * archhaskellbuild is a customised version of archbuild from devtools * makechrootpkg-aufs is makechrootpkg using aufs as it did several months ago, mounting a local repo inside the chroot * print-cabal-urls.hs is a script printing filenames for latest versions of packages in Hackage * batch-cabal2arch downloads the tarballed cabal files from Hackage and runs cabal2arch on it. * buildlibs/buildall do a batch compilation of all packages in naive alphabetical order (buildlibs does only haskell-*) They can be used to power a buildbot with minimal feature set. -- Rémy.

On Tue, Oct 12, 2010 at 08:42, Nicolas Pouillard wrote:

On Mon, 11 Oct 2010 21:28:05 +0100, Magnus Therning wrote:

...

On 08/10/10 22:14, Magnus Therning wrote:

...

Now that Don has decided to focus his Herculean powers in other directions I wonder what people feel should be the plan for the future.

Do we try to fix up Don's tools and scripts so that they are more conducive to team work?

Do we give up on having all of Hackage in AUR and instead rely on tools like bauerbill?

Do we try to do something like what Xyne suggested--set up a Haskell ABS and publish pre-compiled packages in [arch-haskell]?

Here's what I'd suggest based on the last few days' discussion:

• Create an ABS-like tree for all hackage packages. • Populate it with the current versions of packages as found on AUR. • Keep the tree on github.

This will allow us to, in the short term:

• Distribute the work of keeping up with Hackage. • Still upload packages to AUR.

In the longer term I'd like to evolve the tools to work with this setup.  At some point I'd love to see this:

• Some advanced packages builders are able to work straight against the ABS-tree on github, e.g. bauerbill. • cabal2arch modified to work against a local version of the ABS-tree. • A tool capable of determining the reverse dependencies of packages kept in the ABS-tree (so that all packages depending on the updated one can be rebuilt). • An (official) repository for binary Haskell packages, maintained by a team of people.

What do you think?

That's a great plan, I hope that we are not far from a binary repo, I would be a big step forward. In some way, once cabal2arch has done its job and that we have this ABS tree, tools should be the same than for others repository?

Thanks. I'll push ahead with it, despite the relative lack of responses so far... it's only been half a day or so though so maybe there'll be more comments later on. In any case, nothing I'm doing is irreversible :-) Yes, I think once we have an ABS tree the same old established tools can be used in that tree as in other ABS trees. I still see a use for cabal2arch though when it comes to keeping up with hackage, but it has to be changed and improved a little bit to make it fit better. /M -- Magnus Therning                        (OpenPGP: 0xAB4DFBA4) magnus＠therning．org          Jabber: magnus＠therning．org http://therning.org/magnus         identi.ca|twitter: magthe

On Tue, Oct 12, 2010 at 10:23, Rémy Oudompheng wrote:

Magnus Therning wrote:

...

Thanks.  I'll push ahead with it, despite the relative lack of responses so far... it's only been half a day or so though so maybe there'll be more comments later on.  In any case, nothing I'm doing is irreversible :-)

Yes, I think once we have an ABS tree the same old established tools can be used in that tree as in other ABS trees.  I still see a use for cabal2arch though when it comes to keeping up with hackage, but it has to be changed and improved a little bit to make it fit better.

What do you mean ? cabal2arch seems to fit exactly its role, it translates a single cabal file to a single PKGBUILD, I would rather say the helper programs in haskell-archlinux have to be changed.

Oh, don't get me wrong, some parts are already perfect. The kind of things I was thinking of was • modify cabal2arch to accept the location of the local copy of the ABS tree so that the new PKGBUILD is created in the correct place • modify cabal2arch to be aware of the layout of the ABS tree (especially I expect that we would want some way of separating binary and source packages) • modify cabal2arch in the directions that Don has already pointed out, i.e. to remove the static, compiled-in lists (e.g. naming of C packages for dependencies), instead those lists should be kept outside the binary (maybe even make it possible to get them from a URL) I hope that makes my statement clearer. /M -- Magnus Therning                        (OpenPGP: 0xAB4DFBA4) magnus＠therning．org          Jabber: magnus＠therning．org http://therning.org/magnus         identi.ca|twitter: magthe

On Tue, Oct 12, 2010 at 12:09, Rémy Oudompheng wrote:

Magnus Therning wrote:

...

Oh, don't get me wrong, some parts are already perfect.  The kind of things I was thinking of was

• modify cabal2arch to accept the location of the local copy of the ABS tree so that the new PKGBUILD is created in the correct place • modify cabal2arch to be aware of the layout of the ABS tree (especially I expect that we would want some way of separating binary and source packages) • modify cabal2arch in the directions that Don has already pointed out, i.e. to remove the static, compiled-in lists (e.g. naming of C packages for dependencies), instead those lists should be kept outside the binary (maybe even make it possible to get them from a URL)

I am strongly bothered by the fact that the PkgBuild "Show" function is implemented in cabal2arch and not in the archlinux module, is there anybody against merging them together ?

I was actually more bothered about the number of executables that are contained in archlinux; my initial thought was to break them out and put them in cabal2arch, thereby making archlinux a pure library package. ;-) /M -- Magnus Therning                        (OpenPGP: 0xAB4DFBA4) magnus＠therning．org          Jabber: magnus＠therning．org http://therning.org/magnus         identi.ca|twitter: magthe

On Tue, Oct 12, 2010 at 13:22, Rémy Oudompheng wrote:

On 2010/10/12 Magnus Therning wrote:

...

On Tue, Oct 12, 2010 at 12:09, Rémy Oudompheng wrote:

...

I am strongly bothered by the fact that the PkgBuild "Show" function is implemented in cabal2arch and not in the archlinux module, is there anybody against merging them together ?

I was actually more bothered about the number of executables that are contained in archlinux; my initial thought was to break them out and put them in cabal2arch, thereby making archlinux a pure library package. ;-)

I personally think that archlinux itself has too many dependencies. I'd like to have three packages - archlinux: a set of libraries needed for cabal2arch and managing the repos - cabal2arch: cabal2arch + basic scripts to find upstream updates and do database-like operations - archlinux-report: library+programs used to upload to AUR, interact with website, update web pages etc.

I like this split. Is it something that you would like to prototype? /M -- Magnus Therning                        (OpenPGP: 0xAB4DFBA4) magnus＠therning．org          Jabber: magnus＠therning．org http://therning.org/magnus         identi.ca|twitter: magthe

On Rémy Oudompheng wrote:

I split archlinux into archlinux and archlinux-web, feel free to pull my repo [*]. archlinux is now buildable with only GHC. To ease moving files around, I've put everything together.

This commit [$] replaces the computation of md5 hashes (using a system call to wget and a Haskell implementation of MD5) by a system call to "makepkg -g". This drops the dependency on haskell-puremd5. As a result the effective dependencies of cabal2arch become ghc and haskell-archlinux, which I think will greatly simplify bootstrapping. -- Rémy.