2012/11/30 Magnus Therning

On Thu, Nov 29, 2012 at 9:50 PM, Ramana Kumar wrote:

...

[haskell] and [haskell-web] are out of sync at the moment.

I would say that github repository of [haskell] is out of sync with [haskell], so [haskell-web] can't be updated.

...

Do we have a mechanism in place whereby we only push changes publicly if they are all compatible with each other?

No, we haven't.

Yes, that's exactly what `cblrepo` does through its package database. Of course it only does this with the packages in a single database, there is no support for multiple databases. There is not even any support for merging databases.

We don't need to merge databases. `cblrepo` is a very cool tool, but a single human can reasonably manage 150 packages, more or less. If we want a bigger, almost complete haskell repository (do we?) we need to work in team.

...

I suppose the right way to do this would be to only offer one endpoint repo, which combines the results of all the small repos people are maintaining.

Please elaborate a little more on the details of how that would work. I'm more than open to making changes to `cblrepo` in this direction if necessary.

This are my suggestions: 1. Magnus keep a [haskell-base] repository. This MUST be an Arch repo in sync with a github repo. 2. Both repositories (Arch and git) should be available for maintainers of other repository. 3. Maintainer of [haskell-web] (me) MUST update it's repo to keep in sync with base in a reasonable time. 4. After this time a “global maintainer” (Magnus, Ramana, me, whoever) can grab all packages from both repositories and put them in a new one: [haskell] 5. Only [haskell] is intended for end users. A “reasonable time” could be 3 days. I saw that Magnus updates the repo on Wednesday and on Saturday/Sunday: so before the next update [haskell-web] should be in sync. I developed [haskell-web] in a way that it will not duplicate packages from [haskell]: I use them as DistroPkg. Updating is easy with `cbladmin` [^1], there's no need to modify `cblrepo`, maybe you could merge (and develop) some ideas from there. But this is not the main point. The main point is to be in sync, or else I'm just wasting my time, as [haskell-web] will never be really useful. Fabio [^1]: https://github.com/EffeErre/habs-web

On Fri, Nov 30, 2012 at 02:41:51PM +0100, Fabio Riga wrote:

2012/11/30 Magnus Therning

...

On Thu, Nov 29, 2012 at 9:50 PM, Ramana Kumar wrote:

...

[haskell] and [haskell-web] are out of sync at the moment.

I would say that github repository of [haskell] is out of sync with [haskell], so [haskell-web] can't be updated.

Yeah, that's my bad. I sometimes start updates and builds in the morning and then check in on them during the day. If they succeed I push the new packages to the repo, but I only have write access to the github repos from my home computer. I always plan to push the changes as soon as I get home, but you know what they say about the best laid schemes of mice and men; I sometimes forget and it might go days before I remember.

...

Yes, that's exactly what `cblrepo` does through its package database. Of course it only does this with the packages in a single database, there is no support for multiple databases. There is not even any support for merging databases.

We don't need to merge databases. `cblrepo` is a very cool tool, but a single human can reasonably manage 150 packages, more or less. If we want a bigger, almost complete haskell repository (do we?) we need to work in team.

...

...

I suppose the right way to do this would be to only offer one endpoint repo, which combines the results of all the small repos people are maintaining.

Please elaborate a little more on the details of how that would work. I'm more than open to making changes to `cblrepo` in this direction if necessary.

This are my suggestions:

1. Magnus keep a [haskell-base] repository. This MUST be an Arch repo in sync with a github repo. 2. Both repositories (Arch and git) should be available for maintainers of other repository. 3. Maintainer of [haskell-web] (me) MUST update it's repo to keep in sync with base in a reasonable time. 4. After this time a “global maintainer” (Magnus, Ramana, me, whoever) can grab all packages from both repositories and put them in a new one: [haskell] 5. Only [haskell] is intended for end users.

A “reasonable time” could be 3 days. I saw that Magnus updates the repo on Wednesday and on Saturday/Sunday: so before the next update [haskell-web] should be in sync.

I developed [haskell-web] in a way that it will not duplicate packages from [haskell]: I use them as DistroPkg. Updating is easy with `cbladmin` [^1], there's no need to modify `cblrepo`, maybe you could merge (and develop) some ideas from there. But this is not the main point. The main point is to be in sync, or else I'm just wasting my time, as [haskell-web] will never be really useful.

The big piece that's missing above is how to handle failures to upgrade a package caused by a dependant not allowing the upgrade. Do we hold off the base-test repo then? I currently upgrade everything I can in one transaction, what if one of the packages requires holding back due to a dependant. Would that require a partial roll-back in the base-test repo? It's questions like these that I don't have a good answer to. And yes, both situations have occurred in the past. They both cause a bit of work (though not so much if they are caught already at 'clbrepo updates' time). /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus I invented the term Object-Oriented, and I can tell you I did not have C++ in mind. -- Alan Kay

2012/12/4 Magnus Therning

Yeah, that's my bad. I sometimes start updates and builds in the morning and then check in on them during the day. If they succeed I push the new packages to the repo, but I only have write access to the github repos from my home computer. I always plan to push the changes as soon as I get home, but you know what they say about the best laid schemes of mice and men; I sometimes forget and it might go days before I remember.

Sometimes it happens to me too, but no other repo is depending on mine, so nobody happens to know this!

...

This are my suggestions:

1. Magnus keep a [haskell-base] repository. This MUST be an Arch repo in sync with a github repo. 2. Both repositories (Arch and git) should be available for maintainers of other repository. 3. Maintainer of [haskell-web] (me) MUST update it's repo to keep in sync with base in a reasonable time. 4. After this time a “global maintainer” (Magnus, Ramana, me, whoever) can grab all packages from both repositories and put them in a new one: [haskell] 5. Only [haskell] is intended for end users.

A “reasonable time” could be 3 days. I saw that Magnus updates the repo on Wednesday and on Saturday/Sunday: so before the next update [haskell-web] should be in sync.

I developed [haskell-web] in a way that it will not duplicate packages from [haskell]: I use them as DistroPkg. Updating is easy with `cbladmin` [^1], there's no need to modify `cblrepo`, maybe you could merge (and develop) some ideas from there. But this is not the main point. The main point is to be in sync, or else I'm just wasting my time, as [haskell-web] will never be really useful.

The big piece that's missing above is how to handle failures to upgrade a package caused by a dependant not allowing the upgrade. Do we hold off the base-test repo then? I currently upgrade everything I can in one transaction, what if one of the packages requires holding back due to a dependant. Would that require a partial roll-back in the base-test repo?

It's questions like these that I don't have a good answer to. And yes, both situations have occurred in the past. They both cause a bit of work (though not so much if they are caught already at 'clbrepo updates' time).

I think here you are right, we don't have a real solution to this. In these days there are some discussions going in the haskell-cafè mailing list about the mess with cabal. The problem is not cabal, cblrepo or our different approaches to packaging for Arch. The problem is hackage. Every developer use different version of dependencies: someone is fast in keeping up-to-date, someone is more conservative, someone simply hasn't enough time to dedicate to his project and some projects are abandoned. We have to decide how many packages we want to include in [haskell] repo, and this depends on the approach we take: the up-to-date or the conservative. I call conservative what seems to have most consensus in haskell community: haskell-platform. There are A LOT of packages that simply don't work with ghc-7.6, because it's bleeding edge (the assumption is: if it is newer than HP, it's bleeding edge). This also was Arch approach until the switch from ghc-7.04 to 7.4.1. I feel that if we use HP, our repository can be much more bigger and Arch haskellers can compile the most of hackage, if not packaged in archhaskell. The BIG drawback is that we'll have to avoid updating for the most active packages, and we will easily end up with the dependency nightmare. The up-to-date approach is what both of us are doing. I feel is more like in the Arch way and, IMHO, the right thing to do. There will be less packages in [haskell], but they are the most active ones. When a change in a dependency break a dependant, we usually should wait only few days. If the broken package persist, we will decide if taking it out from the repo or to switch to the conservative approach. Yes, this will happen and could require a partial roll-back, but I think it will be less frequent and easier to solve than trying to keep HP with less up-to-date packages. There are two other solutions: a more static approach and nix/cabal-dev/something similar. The static way could be: take Haskell Platform and build a repo with as much packages as possible, the most updated version that work, then stop. The next HP release (or every 6 months) repeat again. Isn't it the Ubuntu way? Maybe it's easier, but I don't like it. If somebody is for the cabal-dev approach, he surely would find the archhaskell effort useless. Still I don't see the point in cabal-dev, as projects are compiled statically. But this is off-topic here. Sorry for my verbosity, and this isn't a real answer to the problem, but I wanted to explain my view to ask: what is our goal? Why can't we try to follow the path we have already taken? Let's go on, after all, merging the repos is not such a big change! Cheers, Fabio

On Fri, Dec 21, 2012 at 8:41 AM, Ramana Kumar wrote:

I agree that the up-to-date approach is the way to go: it fits well with Arch. I suggest we continue that way.

Three items on the roadmap, then:

Document the jobs of the maintainers of the merged repo, and of the satellite repos. Set up the merged repo (and rename current [haskell] to [haskell-core] (or -base)) Document how to create new repos like [haskell-web].

Fabio wrote something for 1 in a previous email; I will add it to the wiki and extend if I can.

Magnus, can you proceed with 2? If necessary I may be able to provide another server.

I modified the existing repo so that the following pacman entry also works: [haskell-core] Server = http://www.kiwilight.com/haskell/core/$arch Server = http://xsounds.org/~haskell/core/$arch The old name ([haskell]) will continue to work too, at least for the time being. I think [haskell-core] should be the preferred name so I've gone ahead and made changes to the Arch Wiki to reflect that. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus