Peter Simons wrote:

Xyne, you are a trusted user in ArchLinux, right? Is there any way that I could maybe submit updated PKGBUILD files and have someone, anyone, put them into the abs repository?

Take care, Peter

TUs only have access to the [community] repository so I wouldn't be able to update anything in [extra]. That stated, I could contact the devs to request access specifically to maintain Haskell packages for the Arch-Haskell team (either alongside Vegai or replacing him if he is no longer interested). If others on this list approve then I could send a request to the dev list. It may help if we formally designate the new Arch-Haskell team first to give some context. I'm sure that recommendations from the other members would be taken into consideration (including from Don). Regards, Xyne

On 2010/10/19 Vesa Kaihlavirta wrote:

On Mon, Oct 18, 2010 at 8:37 PM, Xyne wrote:

...

TUs only have access to the [community] repository so I wouldn't be able to update anything in [extra]. That stated, I could contact the devs to request access specifically to maintain Haskell packages for the Arch-Haskell team (either alongside Vegai or replacing him if he is no longer interested).

The replacing sounds good. Any volunteers?

I also volunteer to help for [extra] : if we ever have a binary repository for arch-haskell, I suggest we only keep in [extra]/[community] the needed libraries to build xmonad and other popular Haskell programs. -- Rémy

Magnus Therning wrote:

...

I also volunteer to help for [extra] : if we ever have a binary repository for arch-haskell, I suggest we only keep in [extra]/[community] the needed libraries to build xmonad and other popular Haskell programs.

I think it's worth thinking a bit about how to organise having popular Haskell programs in [extra] and [community]. And how to harmoniously have a [archhaskell] repository.

For instance: • what do we do with Haskell Platform? (several of the packages in HP are already in [extra]/[community], but I don't think all are) • how do we deal with updates that trigger re-compilation across repositories? (I often bump into haskell-http it seems, some package foo is compiled against one version, another package bar is compiled against another, and then I get scary messages from GHC when I try to compile baz which relies on both foo and bar)

I suspect there are no easy solutions to this particular problem.

/M

[edit] Below I come to the conclusion that a different approach is better, but I have left the beginning here to make my train of thought clear. [/edit] * Topological Rebuilds Spanning Multiple Repos This is not an issue as we only need to worry about [extra] and [community] (see comments about AUR and [arch-haskell] repo below). Topological rebuilds can be pushed to both [extra] and [community] simultaneously. If arch-haskell manages the full topological set then this is not an issue. The more likely scenario is that others outside of arch-haskell will maintain packages that depend on Haskell packages (probably in [community]... we should be able to adopt everything in [extra]). We could: - adopt an official upgrade policy (including tools) and leave it up to others to keep up with us... this may cause issues though - receive permission from other maintainers to rebuild their packages as necessary (it's possible to update others' packages), e.g. all haskell packages in [extra] and [community] could be rebuilt automatically as needed - request a monopoly on Haskell packages to avoid such considerations and to control the inclusion of Haskell packages in [extra] and [community] The AUR will be left to the end users who build their own packages and tools such as bauerbill can be updated to simplify the process. I think [extra] packages need to be signed off though, unlike [community] packages, which might make scripted rebuilds more painful than they need to be. We would have to look into the procedure of using the corresponding testing repos when necessary. The reason that I suggested a separate [arch-haskell] repo in the first place was to avoid all of these considerations. I still think it might be much simpler to remove all Haskell packages from [extra] and most from [community] and maintain them in a separate repo. If it were official like the [multilib] repo then it could be enabled by default in pacman.conf. Basically any package that would need to be part of a topological rebuild would be in [arch-haskell]. This would exclude some packages from [community] but this would have no effect on the end user as it would be enabled by default in pacman.conf and just as accessible. [community] and [arch-haskell] could also be codependent, i.e. deps from one could be in the other, as both could be expected to be enabled by default. The more I think about it, the more I am convinced that this is the optimal approach. This guarantees that we can maintain internal consistency with topological (and maybe even automatic) rebuilds and we do not need to worry about others remaining synchronized with our build cycle. It keeps everything in one place and would also gain the benefits of being mirrored with the rest of the official repos. I have more thoughts concerning parallel repos for different subsets of package versions, but I'll leave that for another post to avoid distraction from the current topic. Regards, Xyne

On 2010/10/19 Magnus Therning wrote:

On Tue, Oct 19, 2010 at 13:31, Xyne wrote: [...]

...

The reason that I suggested a separate [arch-haskell] repo in the first place was to avoid all of these considerations. I still think it might be much simpler to remove all Haskell packages from [extra] and most from [community] and maintain them in a separate repo. If it were official like the [multilib] repo then it could be enabled by default in pacman.conf.

Basically any package that would need to be part of a topological rebuild would be in [arch-haskell]. This would exclude some packages from [community] but this would have no effect on the end user as it would be enabled by default in pacman.conf and just as accessible. [community] and [arch-haskell] could also be codependent, i.e. deps from one could be in the other, as both could be expected to be enabled by default.

The more I think about it, the more I am convinced that this is the optimal approach. This guarantees that we can maintain internal consistency with topological (and maybe even automatic) rebuilds and we do not need to worry about others remaining synchronized with our build cycle. It keeps everything in one place and would also gain the benefits of being mirrored with the rest of the official repos.

I agree with you about this.  However, this raises the question of how we make this is to happen.  Who do we have to convince?  What do we have to do?

I suggest we set up our infrastructure first: when everything is ready, we can think about changing how packages are managed in [extra] and [community]. However, I disagree with the possibility that [community] packages could depend on [arch-haskell], just like there is no way a [community] package can depend on a [multilib] package. I think there is no harm having a full [arch-haskell] repository that people can put at top of pacman.conf, so that it is used with higher priority than [extra] and [community]. So first we have a robust and tested system, and then we discuss about how it integrates with the mainstream package set. -- Rémy.

Hi Magnus,

Sure, but that brings back the biggest of issues with the binary-packages-plan: online storage.

from past experience, I expect a hosting site to materialize eventually. We could start out providing a *small* repository that includes, say haskell-platform. Once people start using that and find it useful, some interested party will show up and offer to host it. We just need to generate enough momentum to get the whole thing started. In order to do that, we need to set up a process that builds and maintains such a repository. This is the first step, and before we have accomplished that, there's no need to worry about disk space or bandwidth, IMHO. The same process that would maintain a binary repository can be used to maintain AUR, so any effort we put into that task pays off. Right now, the biggest problem we ought to solve is that a lot of our packages don't build because the version constraints cannot be satisfied. The following solutions to this problem have been proposed: 1) Generate a consistent package set computationally. 2) Maintain the package set manually, and use automated test builds to verify that all constraints are satisfied. In the long run, we should have both, but (2) is a must-have if we aim to provide a binary repository. So I wonder whether anyone is working on an implementation of either approach? And if, then what is the state of those efforts? Take care, Peter

On Fri, Oct 22, 2010 at 01:19, Rémy Oudompheng wrote:

On 2010/10/21 Peter Simons wrote:

...

Right now, the biggest problem we ought to solve is that a lot of our packages don't build because the version constraints cannot be satisfied. The following solutions to this problem have been proposed:

 1) Generate a consistent package set computationally.

 2) Maintain the package set manually, and use automated test builds    to verify that all constraints are satisfied.

In the long run, we should have both, but (2) is a must-have if we aim to provide a binary repository.

So I wonder whether anyone is working on an implementation of either approach? And if, then what is the state of those efforts?

The build system I suggest is the following: http://github.com/remyoudompheng/archhaskell-build

I would like to add a feature to manycabal2arch that reads a list of packages with specified pkgrels and generates PKGBUILDs with that pkgrel, this would reduce bumping pkgrels for rebuilds to a trivial script.

I still have to add functions to do topological rebuilds (all needed programs do already exist), and include the script that checks for Cabal updates.

Would you mind clarifying the use cases that these tools address? /M -- Magnus Therning                        (OpenPGP: 0xAB4DFBA4) magnus＠therning．org          Jabber: magnus＠therning．org http://therning.org/magnus         identi.ca|twitter: magthe

simons:

Hi Rémy,

...

http://github.com/remyoudompheng/archhaskell-build

this is really quite impressive! Great job. I have some question, and I hope that you can help me understand how the whole things works:

1) The file PKGLIST determines the set of Cabal files to be converted into PKGBUILDs. At the same time, PKGLIST is generated from the set of available PKGBUILDs. This is looks like a circular dependency to me. Am I missing something?

2) The main build step appears to be "makeworld". Is that script capable of performing incremental builds?

To give you a concrete example, assume that makeworld has run successfully and that the entire binary repository is up-to-date. Now, there arrives an update to "hledger-lib", which necessitates that two packages ought to be re-built, namely "hledger-lib" and "hledger". Will running "makeworld" again do that?

Can someone publish what consisten set you've found? It would be interesting to then compare against what cabal says is consistent.

On 2010/10/22 Peter Simons wrote:

Hi Rémy,

 > http://github.com/remyoudompheng/archhaskell-build

this is really quite impressive! Great job. I have some question, and I hope that you can help me understand how the whole things works:

 1) The file PKGLIST determines the set of Cabal files to be converted    into PKGBUILDs. At the same time, PKGLIST is generated from the set    of available PKGBUILDs. This is looks like a circular dependency to    me. Am I missing something?

The generation of PKGLIST from the habs tree was only there to take a snapshot of the current state of our packages.

 2) The main build step appears to be "makeworld". Is that script    capable of performing incremental builds?

   To give you a concrete example, assume that makeworld has run    successfully and that the entire binary repository is up-to-date.    Now, there arrives an update to "hledger-lib", which necessitates    that two packages ought to be re-built, namely "hledger-lib" and    "hledger". Will running "makeworld" again do that?

makeworld does not rebuild an already built package, based on its filename. The missing part is: * use the revdeps script [#] to compute reverse dependencies * use some sed magic or shell script to increment pkgrel in the PKGLIST. * add to manycabal2arch the ability to: - not regenerate already generated packages - read pkgrel from the PKGLIST * makeworld checks only existence of files to know whether something has to be rebuilt, if the pkgrel has changed, the package filename changes, hence the package has to be rebuilt. [#] http://github.com/archhaskell/archlinux/blob/master/scripts/reverse_deps.hs

Take care, Peter

On 2010/10/22 Peter Simons wrote:

Hi Rémy,  > The missing part is:  > * use the revdeps script [#] to compute reverse dependencies  > * use some sed magic or shell script to increment pkgrel in the PKGLIST.  > * add to manycabal2arch the ability to:  >   - not regenerate already generated packages  >   - read pkgrel from the PKGLIST  > * makeworld checks only existence of files to know whether something  > has to be rebuilt, if the pkgrel has changed, the package filename  > changes, hence the package has to be rebuilt.

I may be misunderstanding things, of course, but this sounds very much like the short answer to my question is "no".

Yes.

Does it even make sense to implement all those features that you've mentioned? From the looks of it, you are about to re-invent Make.

It is necessary to increment $pkgrel in order to get pacman to know an update is available (if the goal is a binary repo). This means that any version bump on a package needs a regeneration for all dependent PKGBUILDs. I don't really know a 'make' way of doing that. Regards, Rémy.

On 23/10/10 18:30, Peter Simons wrote:

Hi Rémy,

...

It is necessary to increment $pkgrel in order to get pacman to know an update is available (if the goal is a binary repo). [...] Would it be possible to move development from your archhaskell-build repository into one that's accessible to all members of the organization? I have a few minor patches to make the scripts work on my i686-only machine.

For the moment you can always create a fork to share your patches with Remy directly. There's no need to wait for an official repo. This would take the place of the current habs repo. I think it's best to create a new repo for it though, any suggestion for the name? Otherwise it's likely to be archhaskell/habs2 :-) Just let me know when you think there is a repo that is in good enough state to make it official.

The way I see it now, our "job" is maintain the PKGLIST file. That file provides a mapping from Cabal to ArchLinux. For every package, we know

- Cabal package name, - Cabal package version, - ArchHaskell package name, and

I thought this was dealt with automatically by cabal2arch. Is that functionality not good enough to keep on using?

- ArchHaskell package release number.

Using that file, an automated process can build the entire repository to guarantee consistency and to provide binary packages. Very nice!

Excellent job on this. Now I just need to take a closer look at it. :-) /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus＠therning．org Jabber: magnus＠therning．org http://therning.org/magnus identi.ca|twitter: magthe

On 23/10/10 20:17, Peter Simons wrote:

Hi Magnus,

...

For the moment you can always create a fork to share your patches with Remy directly. There's no need to wait for an official repo.

you are right, there is no need to create an official repo for the build tool chain yet. Is there any particular reason why you feel we should wait?

No real reason, just that I get the impression on the mailinglist that you are both still working out kinks. That may of course be a misconception on my part :-)

...

This would take the place of the current habs repo.

Yes, I agree. We could import Remy's work into a branch of habs, say "new-master", and merge that into "master" eventually.

...

I think it's best to create a new repo for it though, any suggestion for the name?

Do we really need another repository? Personally, I feel it's sufficient to create a branch in haps.

Well, if I understand what you are doing, then the branch would be completely disconnected from the existing master branch. That would mean we have two unconnected branches in one repo. That thought makes me cringe ;-)

...

Just let me know when you think there is a repo that is in good enough state to make it official.

I think Remy's system is ready for that. I tried to build my own arch-haskell repo with his toolchain, and it worked like a charm in less than an hour. His script just ties the various ArchHaskell devtools together; it's quite simple and straight-forward. Based on this experience, I feel the system is good and ready to be used.

Ah, that's a good report.

...

...

- Cabal package name, - Cabal package version, - ArchHaskell package name, and

I thought this was dealt with automatically by cabal2arch. Is that functionality not good enough to keep on using?

Yes, I completely agree. cabal2arch is getting the job done fine. We will probably have to extend it, though, so that we can pass an ArchLinux pkgrel number that it ought to use to generate the PKGBUILD file. The alternative is that we patch those files in a separate phase after cabal2arch has run, but my impression is that it would be cleaner to let cabal2arch generate the proper version directly.

Yes, the format of the PKGLIST file has to influence the creation of the PKGBUILD (pkgrel and possibly configuration/compilation flags). I'm about to push some changesets that adds command line parsing to cabal2arch, including support for subcommands which means cabal2arch subsumes manycabal2arch. /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus＠therning．org Jabber: magnus＠therning．org http://therning.org/magnus identi.ca|twitter: magthe

Hi guys,

...

...

I could contact the devs to request access specifically to maintain Haskell packages for the Arch-Haskell team (either alongside Vegai or replacing him if he is no longer interested).

The replacing sounds good. Any volunteers?

I also volunteer to help for [extra].

this is good news! I'd also be willing to update packages in [extra], of course, but since I've plenty of stuff to do I wouldn't mind if Xyne and Rémy take care of that for the time being. As far as I'm concerned, the most important matter right now is to enable the Haddock documentation for Haskell packages in [extra]. If we update the PKGBUILD altogether in the process, then that would be great, too! The relevant packages are: haskell-binary haskell-dataenc haskell-deepseq haskell-extensible-exceptions haskell-hashed-storage haskell-haskeline haskell-haskell-src haskell-html haskell-hunit haskell-mmap haskell-network haskell-packedstring haskell-parallel haskell-regex-base haskell-regex-compat haskell-regex-posix haskell-stm haskell-terminfo haskell-utf8-string haskell-xhtml haskell-zlib These packages seem to have support for Haddock already, so we can probably leave them alone for now: haskell-cgi haskell-http haskell-mtl haskell-parsec haskell-quickcheck Take care, Peter