2008/8/6 Duncan Coutts :

On Tue, 2008-08-05 at 22:12 -0700, Don Stewart wrote:

...

marlowsd:

...

Following lots of useful discussion and evaluation of the available DVCSs out there, the GHC team have made a decision: we're going to switch to git.

Hooray, this will generate a lot of open source good will, and help make GHC more accessible to the outside world.

Heh, you still need darcs to build it, because all the libs are using darcs, and that's not going to change any time soon.

One thing that might be a good idea is setting up Git mirrors of the libraries etc that we cannot convert to Git since other people depend on them. This would give us nice integration with Gits submodule support, allowing us to check out a consistent snapshot of the entire tree (including the libraries, Cabal etc) at any point in time straightforwardly. Of course, as a bonus you wouldn't have to install Darcs to clone. Cheers, Max

Max Bolingbroke:

2008/8/6 Duncan Coutts :

...

On Tue, 2008-08-05 at 22:12 -0700, Don Stewart wrote:

...

marlowsd:

...

Following lots of useful discussion and evaluation of the available DVCSs out there, the GHC team have made a decision: we're going to switch to git.

Hooray, this will generate a lot of open source good will, and help make GHC more accessible to the outside world.

Heh, you still need darcs to build it, because all the libs are using darcs, and that's not going to change any time soon.

One thing that might be a good idea is setting up Git mirrors of the libraries etc that we cannot convert to Git since other people depend on them. This would give us nice integration with Gits submodule support, allowing us to check out a consistent snapshot of the entire tree (including the libraries, Cabal etc) at any point in time straightforwardly. Of course, as a bonus you wouldn't have to install Darcs to clone.

I seriously hope the plan is to move all *core* libraries (including GHC's cabal repo) etc over to git, too. In other word, everything that you need to build the development version of GHC should come via git. Having a mix of VCSs would be the worst option of all. Manuel

On Fri, 2008-08-08 at 14:03 +0100, Ian Lynagh wrote:

On Fri, Aug 08, 2008 at 12:04:15PM +1000, Manuel M T Chakravarty wrote:

...

I seriously hope the plan is to move all *core* libraries (including GHC's cabal repo) etc over to git, too. In other word, everything that you need to build the development version of GHC should come via git. Having a mix of VCSs would be the worst option of all.

No, the plan is to move only the GHC and testsuite repos to git, as the others are also used by hugs, nhc98, etc.

It would be possible to move GHC's Cabal repo over too, as that is private to GHC, but given the other libraries will be using darcs anyway I think it is simpler to keep all darcs repos using the same VCS.

If there's some way of having automated git mirrors of the upstream darcs repos then that's might be convenient for people building ghc. Asking the maintainers of all other libs to switch is a bit much though. Duncan

Ian Lynagh:

On Fri, Aug 08, 2008 at 12:04:15PM +1000, Manuel M T Chakravarty wrote:

...

I seriously hope the plan is to move all *core* libraries (including GHC's cabal repo) etc over to git, too. In other word, everything that you need to build the development version of GHC should come via git. Having a mix of VCSs would be the worst option of all.

No, the plan is to move only the GHC and testsuite repos to git, as the others are also used by hugs, nhc98, etc.

It would be possible to move GHC's Cabal repo over too, as that is private to GHC, but given the other libraries will be using darcs anyway I think it is simpler to keep all darcs repos using the same VCS.

I think all *core* libraries must switch. Seriously, requiring GHC developer to use a mix of two vcs during development is a Very Bad Idea. Don was excited about getting more people to look at the source when it is in git (see the comments he posted from reddit). By requiring two vcs you will get *less* people to look at the source. This is not only to get the sources to hack them, but you effectively require developers to learn the commands for two vcs (when they are already reluctant to learn one). For example, often enough somebody who changes something in GHC will modify the base package, too. Then, to commit the overall work, you need to commit using both vcs. If you need to branch for your work, you need to create branches in two vcs (no idea whether the semantics of a branch in git and darcs is anywhere similar). When you merge your branch, you need to merge in both vcs. You can't seriously propose such a set up! Duncan wrote,

If there's some way of having automated git mirrors of the upstream darcs repos then that's might be convenient for people building ghc. Asking the maintainers of all other libs to switch is a bit much though.

I am not talking about all libs, I am talking about the core libs. Most developers of the core libs are also GHC developers. So, you ask them to change already by changing the vcs of GHC. Asking them to work with two vcs at the same time is worse IMHO. I *strongly* object to moving to git before this isn't sorted out. As Roman said before, GHC is heading into a dangerous direction. It gets progressively harder to contribute to the project at the moment. First, changing the build system to Cabal. Now, proposing to use two vcs. Somebody who is new to the project not only has to learn the internals of GHC, but they also have to learn two new vcs, and if they need to change the build system, they need to learn a new build tool. Raising the bar for developers to contribute to a project has been proven to be a very bad idea many times. Let's not take GHC down that path. Manuel

Duncan Coutts wrote:

On Sat, 2008-08-09 at 15:46 +1000, Manuel M T Chakravarty wrote:

...

Raising the bar for developers to contribute to a project has been proven to be a very bad idea many times. Let's not take GHC down that path.

I don't especially relish having to learn another vcs tool or raising the bar for contributions to Cabal either (we have lots of people who make small one-off contributions).

I wonder how many of the libraries are "core" in that they need to be changed a lot for GHC? - all the ones that depend on GHC internals, such as base. (Except the current system has many of them use preprocessor conditionals so that can they depend on various compilers' internals, including nhc98 and hugs? Because a lot of that code is actually shared between implementations) - Cabal, since it's needing a lot of extension to make GHC work with it. Do boot-libraries like unix typically need work by GHC devs? On the other hand, it's looking like there's enough intersection between GHC and other-haskell that it's not such a helpful path to pursue. not quite related: I wonder about various haskell libs switching to darcs2 format. A few new programs use it already. As distros include darcs2, it should become less painful. The conversion is less painful for code that's branched less. So maybe in the future a lot of Haskell libs will be in the superior darcs2 format. what an unpleasant situation! But cross-converting between darcs and git format for the same repo is probably even worse. Last time I tried the darcs-all script (maybe a month ago, using darcs 2.0.2), IIRC, it hung, or had some other problem in one of the libraries. Even though it was a clean copy that I'd only ever pulled into (many times, and was getted by darcs-1.0.9, but still). And darcs-all on the libraries has always been a slow sequential task. So I'm not actually all that enamoured of darcs for ghc development, even for the libs. Since I couldn't update anymore (despite going into ghc-head/libraries/something and mucking around with darcs-revert and such), I just deleted the tree and decided to wait until GHC switches VCS before getting a new copy. (trying git-cloning ghc.git sometime, took about 10 minutes, nearly no CPU time, and 80 MB, so I'm pretty happy about that random experience, but I didn't try to do anything with the repo) -Isaac

Duncan Coutts:

On Sat, 2008-08-09 at 15:46 +1000, Manuel M T Chakravarty wrote:

...

Raising the bar for developers to contribute to a project has been proven to be a very bad idea many times. Let's not take GHC down that path.

I don't especially relish having to learn another vcs tool or raising the bar for contributions to Cabal either (we have lots of people who make small one-off contributions).

I don't think it matters what vcs Cabal uses. GHC does already for a while use a separate repo for its version of Cabal, and the GHC Cabal repo needs to be explicitly updated to ensure that changes to Cabal do not randomly break GHC. To be honest, if I had to say anything, I would say that GHC has to uses fixed, stable versions of Cabal (like it does of gmp). So, it really doesn't matter what vcs Cabal uses. A completely different matter are libraries like base which are deeply connected to GHC. Manuel

Ian Lynagh:

On Sun, Aug 10, 2008 at 02:16:25PM +1000, Manuel M T Chakravarty wrote:

...

Duncan Coutts:

...

I don't especially relish having to learn another vcs tool or raising the bar for contributions to Cabal either (we have lots of people who make small one-off contributions).

I don't think it matters what vcs Cabal uses. GHC does already for a while use a separate repo for its version of Cabal, and the GHC Cabal repo needs to be explicitly updated to ensure that changes to Cabal do not randomly break GHC. To be honest, if I had to say anything, I would say that GHC has to uses fixed, stable versions of Cabal (like it does of gmp). So, it really doesn't matter what vcs Cabal uses.

Unless we do get to a point where we are literally using tarballs[1] of Cabal, I don't think using a mixture of VCSs for Cabal is a good idea. Having to convert patches from one VCS format to the other sounds like a recipe for a lot of pain and suffering.

[1] which I think is a bad idea anyway, as it makes it a lot more hassle to fix Cabal bugs that GHC+bootlibs expose.

The hassle that having two different repo types for Cabal head and Cabal GHC is part of the price of switching from darcs to git for ghc. Incidentally, that you are concerned about Cabal devel in the GHC tree is a consequence out of using GHC as a guinea pig for Cabal development, which by itself is IMHO a Very Bad Idea. Cabal is supposed to be a tool like Happy or Alex. If Cabal *were* mature enough to be used in GHC's build system in the way it is now, GHC would just use the latest stable release of Cabal and we wouldn't have a problem. So, let's please not use one bad idea (using an immature and constantly changing build tool whose use in GHC's build tree barely anybody understands) to justify another bad idea (using two vcs for one project). Manuel

Ian Lynagh:

On Sat, Aug 09, 2008 at 03:46:50PM +1000, Manuel M T Chakravarty wrote:

...

I am not talking about all libs, I am talking about the core libs. Most developers of the core libs are also GHC developers.

I'm not sure that's true. e.g. Malcolm and Ross both commit to the bootlibs, and we get a lot of patches from various people in the community.

Ross does commit patches to ghc (according to darcs changes). So, either he stops that or has to learn git anyway. I don't think we are talking about random contributions from the community. If anything, we need to compare two numbers (1) developers who need to start using git when the ghc repo changes and (2) library developers (ie, people with commit bits regularly contributing to the boot libs) who do not contribute to ghc and hence could avoid learning git if the boot libs stay in a darcs repo.

...

I *strongly* object to moving to git before this isn't sorted out.

FWIW, personally I would prefer staying with darcs. I prefer its underlying philosophy, and I find its UI far more intuitive and easy to use.

Personally, I am more than happy to stay with darcs, too, but my understanding was that at least the Simons decided that we are going to move from darcs to git. All I am saying is that whatever vcs ghc uses, you need to be able to *easily* get, modify, and commit patches to the HEAD and the boot libs with *just one* vcs. Using two vcs is going to make the current situation worse, not better. For example, SimonPJ said one reason for switching vcs is that interns had trouble getting started because they did have trouble obtaining the head as darcs caused them grief. If the boot libs stay under darcs control. Nothing is one, the same interns still won't get going any quicker. Presumably, they are going to take even longer, because they can now get into trouble with darcs and git. We want to lower the barrier to entry, not raise it. By effectively adding a complications (namely git) and not removing any, matters will get worse. Manuel

On Sat, Aug 9, 2008 at 10:44 PM, Roman Leshchinskiy wrote: Maybe investing some time in fixing the most obvious darcs problems would be

a better solution?

We're working on that over at Darcs HQ, but there is no guarantee that we'd come close to fixing the problems within the 4-5 week window that Ian mentioned. Supposing that the main problems GHC has with darcs 2 format get solved in the next month, would that give GHC reason enough to keep using darcs? It seems many of you are eager to use git; perhaps even if darcs was working to satisfaction. People will be working on making darcs work better with the GHC repo as a test case either way. And personally, since I'm not a GHC dev, the decision doesn't affect my life. Having said that, I'm still obviously biased. I'd love for darcs to work well enough that this never came up. Let me throw out one more idea: What if, as a GHC contributor, I could pick equally between git and darcs? My understanding is that, while not optimal, you could use tailor[1] to synchronize a darcs repository with a git one. Offer up both repositories and keep them in sync. Let the masses decide? Jason [1] http://progetti.arstecnica.it/tailor

Jason Dagit:

On Sat, Aug 9, 2008 at 10:44 PM, Roman Leshchinskiy

...

wrote:

Maybe investing some time in fixing the most obvious darcs problems would be a better solution?

We're working on that over at Darcs HQ, but there is no guarantee that we'd come close to fixing the problems within the 4-5 week window that Ian mentioned. Supposing that the main problems GHC has with darcs 2 format get solved in the next month, would that give GHC reason enough to keep using darcs? It seems many of you are eager to use git; perhaps even if darcs was working to satisfaction.

People will be working on making darcs work better with the GHC repo as a test case either way. And personally, since I'm not a GHC dev, the decision doesn't affect my life. Having said that, I'm still obviously biased. I'd love for darcs to work well enough that this never came up.

Same here, and fwiw I won't change any of my many other darcs repos any time soon. However, as I have said before, if ghc is to switch, it must be a clean switch, and no messy use of two vcs at the same time for ghc and boot libs.

Let me throw out one more idea: What if, as a GHC contributor, I could pick equally between git and darcs? My understanding is that, while not optimal, you could use tailor[1] to synchronize a darcs repository with a git one. Offer up both repositories and keep them in sync. Let the masses decide?

I don't think that this technical feasible. I used tailor once to convert a CVS repo to darcs, and while that was better than throwing away the history, it was pretty messy and nothing that you would want to do on a regular basis. Besides, even if the actual conversion would work smoothly (which I strongly doubt), you'd immediately be faced with problems of atomicity and associated race conditions of commits to the two repos. Manuel

dons:

chak:

...

Ian Lynagh:

...

On Fri, Aug 08, 2008 at 12:04:15PM +1000, Manuel M T Chakravarty wrote:

...

I seriously hope the plan is to move all *core* libraries (including GHC's cabal repo) etc over to git, too. In other word, everything that you need to build the development version of GHC should come via git. Having a mix of VCSs would be the worst option of all.

No, the plan is to move only the GHC and testsuite repos to git, as the others are also used by hugs, nhc98, etc.

It would be possible to move GHC's Cabal repo over too, as that is private to GHC, but given the other libraries will be using darcs anyway I think it is simpler to keep all darcs repos using the same VCS.

I think all *core* libraries must switch. Seriously, requiring GHC developer to use a mix of two vcs during development is a Very Bad Idea.

I agree with this.

As Audrey says, you have to lower the barrier to entry. That means:

* one build system * one vcs

to build ghc (and anything it requires, such as the core libraries).

This is a chance to make a big step towards accessibility, let's make that step.

I just want to add to this. We're offering an unusual product: a lazy, purely functional language. This already separates us from the mainstream. So how to we ensure we minimise the stress of adopting something so unfamiliar? By ensuring that in all other respects the environment they have to learn is familiar and simple. I think we risk isolating oursevles yet further -- and creating new barriers to adoption, beyond those we can't avoid -- by adding more complicated dependencies (two revision control systems, one of which, darcs, is now firmly out of the maintstream). Instead, if we just use ubiquitous, common tools -- like git -- for everything, we minimise the pain for people, and sit firmly in the mainstream of open source. If anything has been learnt by Spencer and I while working on xmonad, is: dependencies, dependencies, dependencies reduce these, and you gain eyeballs, and ultimately developers. Increase these, and you end up isolated and marginalised. So let's capitalise on this switch to git, and take the opportunity to remove one big dependency from the system. -- Don

On 10/08/2008, at 05:38, Don Stewart wrote:

Instead, if we just use ubiquitous, common tools -- like git -- for everything, we minimise the pain for people, and sit firmly in the mainstream of open source.

While I agree with this in general, I'm not sure it really applies to vcs (especially darcs) all that much. I don't think anyone who has ever worked with a vcs will need more than a day to learn how to use darcs (or any other sane vcs, for that matter). Really, the problem with darcs is not that it is not mainstream; rather, it's just that it simply doesn't work sometimes. Roman

As a very part-time, temporarily inactive GHC developer I will offer some opinions which should carry no weight: * When I saw the announcement, I cheered! Last fall, I lost 2 weeks of a 9-week visit to darcs hell. While the alleged features may be alluring, the software simply doesn't do what it says on the box. The highly touted 'theory of patches' is not published anywhere in any form that can be understood and checked by anyone with a little bit of mathematics (e.g., group theory or algebra). All these truths make me eager to be rid of darcs. * It seems clear that git offers a richer user interface than darcs and that the UI is harder to master. Moreover, because everyone I have talked to finds the git documentation less than completely helpful, git is probably harder to learn than darcs. Git's advantages are that it is robust, fast, and popular. * A number of people I trust and respect have urged me to adopt git. Since 2005, nobody has urged me to adopt darcs. A number of people I trust and respect have said they wished to abandon darcs. Nobody has suggested abandoning git. * I violently agree with whomever (Don? Malcolm?) said that the Haskell community will prosper to the degree that we have *one* build system and *one* version-control system. And when the build system or version-control system is standard, we gain eyeballs and developers. I haven't found a standard build system that I am willing to use, but I think git is good enough to be used. * Our long-term goal should be to get the *entire* Haskell development community to agree on a version-control system---one that is not darcs. We should expect this process to take several years, and we should expect it to cost money. Would Microsoft or Galois or York or other large players be willing to donate part of an expert's time to migrate to the new version-control system? I'm no particular fan of git. But in a worse-is-better sort of way, I think it's in---it will fill the niche of free, distributed version control. It would be good to identify a way of helping to smooth the path not only for GHC and *all* the libraries but for Hugs, York, xmonad, everybody. Norman, whose most popular open-source software still lives under RCS!

On Sat, Aug 09, 2008 at 06:56:23PM -0400, Norman Ramsey wrote:

...

* Our long-term goal should be to get the *entire* Haskell development community to agree on a version-control system---one that is not darcs. We should expect this process to take several years, and we should expect it to cost money. Would Microsoft or Galois or York or other large players be willing to donate part of an expert's time to migrate to the new version-control system?

It is, of course, up to people with money what they spend it on, but personally I would much prefer to see money spent on making darcs better, for reasons I won't repeat again.

I missed them and wouldn't mind receiving a private note. For the last year I have been hoping to make 'a new darcs-like thing, with a real theory founding it' an important part (one of three) of a grant proposal in distributed computing. So you can see I am in favor of spending money to create a better darcs (which is not quite the same thing as making darcs better; I want to start with a new theory). But I am having second thoughts because I think by the time a proposal reaches a review committee, git may be so firmly entrenched (worse is better) that the work would be considered not worth funding. I realize that I am now firmly off topic, but if people here have opinions, I would be grateful to receive them (perhaps off-list). Norman

Brandon S Allbery wrote:

On 2008 Aug 10, at 20:17, Norman Ramsey wrote:

...

For the last year I have been hoping to make 'a new darcs-like thing, with a real theory founding it' an important part (one of three) of a grant proposal in distributed computing. So you can see I am in favor of spending money to create a better darcs (which is not quite the same thing as making darcs better; I want to start with a new theory).

Can you elucidate what's wrong with the current one?

Noone knows how to formalise it, and (AFAIK) only David understands the new conflicts handling, and hasn't managed to completely communicate that understanding to anyone else. Ganesh ============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ==============================================================================

Ian, I completely agree with you. I love the darcs vcs model, too. However, we have three discussions here: (1) Do we want darcs vcs model? Except Thomas Schilling, who seems to be dead set to get rid of darcs, everybody who voiced their opinion seems to be in favour of the darcs model. (2) Is the current implementation of darcs up to a project the size of ghc? Due to problems in the past & performance regressions with darcs2, a serious number of (important) people believe that the current implementation is not good enough. (3) If we change the vcs for the ghc repo, do we change the vcs for the boot libs, too. This is just an open-source project maintenance question. It has nothing to do with which vcs is better. This is the only point I have been arguing: *if* GHC's repo changes, all boot lib repos must change, too. Manuel Ian Lynagh:

On Sun, Aug 10, 2008 at 08:17:50PM -0400, Norman Ramsey wrote:

...

...

On Sat, Aug 09, 2008 at 06:56:23PM -0400, Norman Ramsey wrote:

...

personally I would much prefer to see money spent on making darcs better, for reasons I won't repeat again.

I missed them and wouldn't mind receiving a private note.

OK, I'll send to the list so that I have somewhere convenient to point people if this comes up in the future:

* A lot of darcs's functionality could be refactored into generally usable Haskell libraries, e.g. LCS-finding, downloading-with-libcurl.

* darcs was once a flagship Haskell application, supporting the idea that Haskell can be used in the real world. That image has mostly faded away now due to the problems it has, but I think we can get it back if we can get a high quality darcs out there. That would be good for the community's image.

* darcs has (in my opinion, at least) a much simpler, more intuitive interface than the other version control systems. I don't think I'm alone here, as I think this is where a lot of the resistance against moving to git is coming from.

* I think darcs is the Obvious, Right way to do version control. Phil Wadler (at least, I think it was him; and probably many others too) has said that the lambda calculus is universal, in the sense that if we were to meet a sufficiently advanced alien culture, it is almost inconceivable that they would not have also discovered the lambda calculus. Darcs-style patch theory, before conflicting patches are introduced, falls into the same category in my opinion. (I'm not yet sure if it can be extended to include some definition of conflictors too). By contrast, the heuristics and multiple merge algorithms of other systems feels very ad-hoc.

Thanks Ian

Iavor Diatchki wrote:

I also don't think that the darcs model has much to offer over git, in fact I find that it lacks some useful features (not counting a reliable implementation). Examples include good support for branching, and being able to easily determine the version of the software that is in a repository (git uses a hash of the content to identify the current state, so it is easy to check if we two developers have the same version of the content).

I think these things are possible in darcs's model, just not its implementation. For example, under _darcs it could have enough info in various states to allow one to switch branches within the same physical directory tree (and if there aren't many changes between the two branches/patchsets, the switch can be quick). And if it weren't for the varying ways the same patch can be stored, hashes of history ought to work too (although that's certainly very built in to the current implementation of darcs; whether it's technically part of the model probably depends whether you can provide the exact same interface, semantics, and computational complexity with a different representation). And I wonder why (it sounds like) Git doesn't have tools to do some kind of smart cherrypicking, using a heuristic to decide which patches in a branch are definitely dependencies of the cherry-picked patch. In any case, I notice a few times with ghc/darcs/Trac tickets, more than one commit has to be listed explicitly to be merged into the stable branch. Maybe it's not very useful/reliable for these purposes anyway? Since I've only ever used Darcs (besides read-only CVS/SVN/etc.), I personally can't speak to what model is better for me! -Isaac

On Wed, Aug 13, 2008 at 2:49 AM, Manuel M T Chakravarty wrote:

Ian, I completely agree with you. I love the darcs vcs model, too. However, we have three discussions here:

(1) Do we want darcs vcs model?

Except Thomas Schilling, who seems to be dead set to get rid of darcs, everybody who voiced their opinion seems to be in favour of the darcs model.

I'm also in favor of the switch to Git. The Git model has proved to be both more productive and more reliable. And the interface, as far as I'm concerned, is *better*. Cheers, Johan

Malcolm Wallace:

...

...

...

...

I seriously hope the plan is to move all *core* libraries (including GHC's cabal repo) etc over to git, too.

...

* one build system * one vcs This is a chance to make a big step towards accessibility, let's make that step.

Ultimately, I don't think git would make ghc any more accessible to new contributors. Darcs is not especially offputting to any beginner who already knows something about VCS in general.

What the move to git is about, is making life easier for the *existing* HQ and core contributors. Evaluate it on that basis, and not in terms of unknown (and unknowable) benefits to current non- contributors. Indeed, you should also consider how many contributors you might lose in a move.

I am not advocating to move. I am just saying, if ghc moves, every component needs to move on which the HEAD build depends and that is needed in its current development form (eg, *not* alex, happy, cabal).

I do hear some significant current contributors having doubts. I can certainly appreciate that having to run 2 VCS in parallel might be confusing and simply make matters worse than at present.

It is confusing and it is going to make matters worse as two failure points are worse than one. And two extra tools to learn worse than one.

The libraries question is a difficult one. We have made a lot of effort over the last 5 years to build infrastructure and code that is shared and portable across multiple implementations of the language. Is this the time to fork those supposedly "common" core libraries into ghc versions vs the rest?

It would be a pity to fork, but to be honest, I'd rather fork the libs than have to use two vcs for GHC. The only other alternative is to decouple more library releases from ghc releases. Manuel

I had my share of problems with Darcs; working on the GHC API I constantly have to avoid conflicts. My temporary workaround is to not update at all. Maybe switching to Darcs 2 format would help here, but there are other issues. I initially converted GHC to Git to be able to more easily checkout older versions (e.g., to find a build bug using git-bisect) but with external core libraries this just doesn't work. Right now, there is simply no practical way to check out an old, building version of GHC! Even if we'd switch to Darcs 2 this problem could not be solved. We would also still need turn to the Git repo to get change histories for specific files or to run commands such as 'git-blame' (unless you don't mind getting a cup of coffee and some biscuits each time you run those commands). I think we can make things easier for existing library contributors by providing a darcs/git cheat sheet or even a command line wrapper. Previous attempts at creating such a wrapper have been abandoned, possibly because some commands cannot easily be modelled in Git. However, if we accept some limitations this is doable. In particular the tricky commands are: darcs pull -- (save) cherry picking requires patch dependency information darcs push -- same as above (darcs pull -a and darcs push -a both can be modelled easily) darcs replace -- not directly supported in Git, but could be modelled -- with a script. If these missing features don't feel like too big a handicap the change should be fairly easy for existing contributors. (And with some time they can start and learn Git's other features.) For our build woes integrating the libraries and the main GHC repo in one Git repo will be very helpful, since we can now just instruct build bots to try and build revision 12345deadbeef and be happy. / Thomas -- My shadow / Change is coming. / Now is my time. / Listen to my muscle memory. / Contemplate what I've been clinging to. / Forty-six and two ahead of me.

Thomas Schilling:

I had my share of problems with Darcs; working on the GHC API I constantly have to avoid conflicts. My temporary workaround is to not update at all. Maybe switching to Darcs 2 format would help here, but there are other issues.

I initially converted GHC to Git to be able to more easily checkout older versions (e.g., to find a build bug using git-bisect) but with external core libraries this just doesn't work. Right now, there is simply no practical way to check out an old, building version of GHC!

Correct me if I am wrong, but this sounds as if you support my point that switching the GHC repo to git without doing the same for the core libs (in an integrated way) would not address the problems you experienced with darcs. Manuel

Thomas Schilling wrote:

(I am also no longer convinced that Darcs' automatic patch dependency calculations are actually a good idea. Just because two patches don't touch the same files, doesn't mean they aren't semantically dependent. Take for example "monadification" patches, which are typically submitted split up for each file. A branch captures those dependencies just fine.)

But the darcs approach to dependency is what underlies cherry-picking, which many people consider the most worthwhile feature of darcs. In fact many people would like it to be possible to override even the dependencies that darcs *does* find to cherry-pick patch A without patch B that A depends on, at the expense of producing a conflict that then has to be fixed up by hand. Ganesh ============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ==============================================================================

-----Original Message----- From: Thomas Schilling [mailto:nominolo@googlemail.com] Sent: 11 August 2008 12:18 To: Sittampalam, Ganesh Cc: Manuel Chakravarty; Don Stewart; Ian Lynagh; Simon Peyton-Jones; GHC Users Mailing List Subject: Re: Version control systems Thomas Schilling wrote:

On 11 Aug 2008, at 12:38, Sittampalam, Ganesh wrote:

...

Thomas Schilling wrote:

...

(I am also no longer convinced that Darcs' automatic patch dependency calculations are actually a good idea. Just because two patches don't touch the same files, doesn't mean they aren't semantically dependent. Take for example "monadification" patches, which are typically submitted split up for

...

...

each file. A branch captures those dependencies just fine.)

But the darcs approach to dependency is what underlies cherry-picking, which many people consider the most worthwhile feature of darcs. In fact many people would like it to be possible to override even the dependencies that darcs *does* find to cherry-pick patch A without patch B that A depends on, at the expense of producing a conflict that then has to be fixed up by hand.

Cherry-picking just a single patch is simple in Git: "git cherry-pick <commit-id>"[1].

I wasn't saying that Git doesn't support cherry-picking, just that you would expect dependencies to restrict what you can and can't cherry-pick; if you specify dependencies just in a linear fashion along each branch (i.e. each patch depends on all those before it on that branch) as I thought you were suggesting, then you enormously restrict what cherry-picks are possible. Ganesh ============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ==============================================================================

On 11 Aug 2008, at 13:00, Duncan Coutts wrote:

It's not clear to me that we've really bothered to find out. The last evaluation in relation to ghc that I'm aware of was prior to the 2.0 release. My impression is that we've all complained about the darcs v1 problems (justly) but spent the most effort investigating things other than darcs v2 which would be the easiest to upgrade to and not have the problems of using two different systems for ghc vs other libs.

I converted the ghc repo to darcs2 (locally): Getting file local history: * darcs changes --last 20 compiler/main/HscTypes.lhs very quick but prints only two patches * darcs changes compiler/hsSyn/HsTypes.lhs 1m22s (16s for the second time) Git <1s * darcs get ghc2 ghc-test (creating a *local* branch) real 13m25.365s user 0m14.677s sys 0m29.541s (at least it seems it actually worked, though) git clone ghc g2 (the slow method of creating a local branch) real 0m6.742s user 0m0.335s sys 0m0.652s * I haven't tested a remote pull yet. At 80 Kb/s, it should take about 15min to clone via Git (70 MB). A test of darcs would be interesting. Finally, of course, we have to hope that Darcs2's conflict problems are actually solved. I also had some weird automerges with Darcs when pulling from Max' repository, so Darcs isn't flawless there, either (this seemed to be one of the main critiques of Git). / Thomas -- Awareness is the enemy of sanity, for once you hear the screaming, it never stops.

Duncan Coutts wrote:

It's not clear to me that we've really bothered to find out. The last evaluation in relation to ghc that I'm aware of was prior to the 2.0 release. My impression is that we've all complained about the darcs v1 problems (justly) but spent the most effort investigating things other than darcs v2 which would be the easiest to upgrade to and not have the problems of using two different systems for ghc vs other libs.

I promised to put together our reasoning on why we don't think moving to darcs2 would help enough. Here's a summary: - using the darcs2 format may well fix the exponential-time merge problem, but the UI for merging conflicts is still lacking in many important ways in darcs: * The conflict markers are not annotated with the patch that they came from, and the ordering of patches in conflict markers is non-deterministic (when I asked about this problem, I was told it was hard to fix). * The 'darcs changes' output only shows one of the patches that is conflicting, you have to guess at the other one(s). Also, it doesn't show which patches are conflict resolutions. - Performance. darcs2 regressed in performance for many operations we commonly use. I've submitted some measurements for some things, but it's pretty easy to find your own test cases: things like "darcs add", "darcs whatsnew", "darcs unrecord" are all slower than darcs 1. When simple operations take multiple seconds to complete, it really slows down your workflow. - I still can't use 'darcs annotate' because it's too slow. Also, we can't browse the GHC repository on the web because the web interface wants to do 'darcs changes <file>', and that takes minutes. It's possible with caching, but you still have to regenerate the cache after a change. - why can I do a complete git clone of a remote GHC repo in a few minutes, but it takes hours to do a complete 'darcs get'? - Bugs. Many bugs have been fixed in darcs2, which is great, but we did already encounter one (hard to reproduce) bug on Windows, when trying to get an up-to-date repo. Perhaps bugs will be less of an issue in the future, but we have had painful experiences particularly on Windows and I know the darcs developers are still not actively testing on Windows. FWIW, I'd also like to stay with darcs because it has the right model, but unfortunately the current implementation is not useable for us, and it's holding us back. I'll say something about core libs in a separate mail. Cheers, Simon

Duncan Coutts:

On Mon, 2008-08-11 at 13:57 +0100, Simon Marlow wrote:

...

- Performance. darcs2 regressed in performance for many operations we commonly use. I've submitted some measurements for some things, but it's pretty easy to find your own test cases: things like "darcs add", "darcs whatsnew", "darcs unrecord" are all slower than darcs 1. When simple operations take multiple seconds to complete, it really slows down your workflow.

Turns out that the reason for slow darcs whatsnew is ghc bug #2093

http://hackage.haskell.org/trac/ghc/ticket/2093

because getSymbolicLinkStatus is broken on 32bit systems in 6.8.2 it means that the 'stat' optimisation does not work so darcs has to read the actual contents of many files. Obviously that's very slow, especially over nfs. That explains why it worked for me in 0.2 seconds but for you took several seconds user time and (even more real time due to nfs).

LOL - that is funny. GHC devel slowed down by slow darcs due to GHC bug. The bug is fixed, isn't it? So, recompiling darcs with 6.8.3 should improve matters. Manuel

2008/8/11 Thomas Schilling

(I am also no longer convinced that Darcs' automatic patch dependency calculations are actually a good idea. Just because two patches don't touch the same files, doesn't mean they aren't semantically dependent. Take for example "monadification" patches, which are typically submitted split up for each file. A branch captures those dependencies just fine.)

Darcs has a feature to deal with patches that are unrelated in patch theory but are related from the user's point of view. When you record you can use --ask-deps to specify dependent patches. These dependencies are then artificially enforced in commute (where dependencies are normally detected). Note: I'm not trying to advocate anything here, I just wanted to let you know that others noticed this and added a feature for it long ago. Jason

On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:

The main obstacle with just switching the core libraries is that they are shared by other implementations and other maintainers. So I see no alternative but to create forks of those repositories for use by GHC, unless/until the other projects/maintainers want to migrate to git.

Forking is much worse than using multiple vcs's, and if we don't fork, anyone working on those libraries will have to use git at least to get GHC HEAD to check that they're not breaking it. And clearly GHC developers outnumber developers of other implementations. (I don't think a move to git will lead to more GHC developers, but I buy the interns argument.) My concern is that there are rather more developers of libraries and assorted other packages, and this will place an arbitrary divide across those. Unless everyone moves to git, of course.

...

...

...

...

"Manuel" == Manuel M T Chakravarty writes:

Manuel> In other words, the decision to move the ghc repo affects all Manuel> core library developers anyway. No use pretenting that changing Manuel> only the ghc repo (and leaving the rest in darcs) would make Manuel> anything simpler for anybody. I'd say that moving GHC to git affects the WHOLE haskell community and we can already think about having all the HackageDB running git. I'm not at all pleased with it (personally I switched from darcs to bzr), bet better to face what the future brings instead of putting the head in the sand. :-) Sincerely, Gour -- Gour | Zagreb, Croatia | GPG key: C6E7162D ----------------------------------------------------------------

| It is worth pointing out that I *never* validate against ghc head when | I commit to the core libraries. I think that's perfectly reasonable for the reasons you explain. Simon

Manuel wrote:

...

| It is worth pointing out that I *never* validate against ghc head when | I commit to the core libraries.

Sorry, but I think the only reason its halfway acceptable is that Malcolm didn't break the GHC build yet. If he does, I'll be screaming as loudly as for anybody else.

Whilst I'm in no way saying that a working nhc98 head is anything like as important as a working ghc head, are you saying that I should scream louder everytime someone breaks nhc98 too? It is happening several times a week at the moment. It can be jolly frustrating when I have other things I could be doing. But I accept that it is simply the price to pay for keeping up-to-date with the libraries everyone else is using. Ghc has no monopoly on the "core" libraries. They are a shared resource.

to be honest, I don't think its a valid reason for us to go to the trouble of having two vcs for ghc.

Well indeed, I don't want to stand in the way of ghc. There are far more people contributing to it, so their needs have greater weight. But I am raising the libraries question, because I think it has an impact much more widely than just ghc (or Hugs or nhc98, for that matter). Git may turn out to be sufficiently easy to use that this will all seem like a storm in a teacup once the dust has settled. (I'm not filled with confidence by blog postings that say "granted, git is a usability disaster zone", and "[you] may find git to be hostile, unfriendly and needlessly complex", but those seem to be minority opinions.) Regards, Malcolm Regards, Malcolm

- Why does NHC98 break so often? Is it because people are checking in code that is not Haskell 98 compatible?

Yes, there is a bit of that. Also, as you point out, there is quite a lot of CPP conditionally compiled code in the libraries, and I would say that it is the major contributor to breakage. It is often unclear which parts of the code are shared and which separate, so a lot of breakage arises from e.g. exporting a name that is defined for ghc only. In addition, there are some (once obscure) bugs in nhc98 that are now triggered increasingly frequently. (We can't blame anyone except nhc98 for those of course.) These include complex import renaming resolution, and contexts on pattern variables.

- It seems to me that implementations "share" libraries using CPP. That seems like a bad approach to me.

Agreed. The CPP was always intended to be as temporary as possible, with the goal to share as much as possible. One of the problems is that the primitives provided by compilers are different. Really, there should be a package below 'base' in the dependency tree, specific to each compiler, and non-shared. Then everything from base upwards would be cleaner and more portable.

If it's so difficult to share code without continuously breaking the build then we're better of keeping the code separate.

I don't agree. The only way to achieve convergence is to start from some semi-merged point, and work to eliminate the differences. Igloo is doing a fantastic job of determining the dependencies and gradually moving stuff around to enable this to happen. Regards, Malcolm

Hello Johan, Wednesday, August 13, 2008, 3:43:15 PM, you wrote:

...

...

- Why does NHC98 break so often? Is it because people are checking in code that is not Haskell 98 compatible?

Can we make sure that these libraries are always built with some Haskell 98 compatibility flag by GHC so people find out when they add non Haskell 98 stuff?

on ghc, we use many non-h98 features. actually, there are even non-h98 features that are common to all 3 compilers (f.e. rank-2 types, afair) we need to check that non-nhc features are not used in *shared* code and the best tool to do such checks is nhc itself -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Tue, Aug 12, 2008 at 10:10:31AM +0100, Malcolm Wallace wrote:

On 12 Aug 2008, at 01:35, Manuel M T Chakravarty wrote:

...

Ah, good point! Changing ghc to git means *all* developers of boot libraries need to use git *regardless* of what repo format the boot libraries are in. After all, they need to validate against the current ghc head before pushing.

It is worth pointing out that I *never* validate against ghc head when I commit to the core libraries.

Also, all of the people who send us patches don't need to validate (and I suspect most of them don't), as we validate the patches before pushing them. Thanks Ian

Hello Manuel, Wednesday, August 13, 2008, 4:39:25 AM, you wrote:

Well, its up to you whether you want to validate for other people, but I don't think that is the right policy. Everybody (including Malcolm) should validate.

as far as we have people validating patches for their platforms (Igloo for GHC, Malcolm for nhc...) this reduces amount of time required to support things working. ideal solutions are too expensive for real world :) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Wed, Aug 13, 2008 at 1:52 PM, Malcolm Wallace wrote:

...

I don't think that is the right policy. Everybody (including Malcolm) should validate.

If you contribute code to the linux kernel, comprehensive testing of the code is a requirement, too.

The analogy is flawed. It is like asking the developers of _gcc_ to ensure that the Linux kernel still builds after every modification to the gcc project code base. The projects are different, so the suggested requirement would be an unreasonable burden.

I think an even better analogy is probably comparing it to developer of GCC changing the libc implementation of another compiler or vice versa. -- Johan

On Wed, Aug 13, 2008 at 3:13 PM, Malcolm Wallace wrote:

...

I think an even better analogy is probably comparing it to developer of GCC changing the libc implementation of another compiler or vice versa.

Our shared libraries do not belong to any one compiler. They are joint creations, with a lot of community (non-compiler-hacker) involvement.

I'm very grateful these people took the time to write these libraries. However, how these modules were created is irrelevant when it comes to addressing the current problem. Parts of their implementation is compiler dependent and having compiler specific code live together is bound to lead to problems because the people hacking on those modules are likely to use and validate on only one compiler. It would be difficult to require them to do otherwise too. To avoid this problem either the compiler dependent code has to be abstracted out from these modules so people can ignore the differences or the implementations of these files need to be kept separate. Consider the following scenario. GHC hackers implement Data.Array in A.hs, Hugs and NHC98 hackers develop separate implementations in B.hs and C.hs respectively. We now run something like diff A.hs B.hs C.hs | sed <diff markers with if-defs> > X.hs X.hs could be likened to what current implementation of these files look like. If the three groups don't validate their changes on all compilers they risk breaking someones build. Especially note the poor scaling properties of this approach were each new implementation adds one more compiler for everyone to verify on. I think the reason this works at all right now is that most work is happening on GHC and that's also were the most users are. Cheers, Johan

kili:

On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote: [...]

...

As for Cabal - we had a thread on cvs-ghc last week, and as I said there we'd love to hear suggestions for how to improve things, including wild and crazy ideas for throwing it all away and starting again. However, as I explained, there are good reasons for the way things are done now, the main one being that the build system for packages is not written twice.

Well, at least the Makefile creation was a step (the first step?) into the wrong direction, IMHO. I'll run a GHC build to get some of those generated Makefiles and followup on cvs-ghc, but for a starter, Cabal shouldn't know anything about implementation-specific internal build systems; instead it should rely only on it's own metadata. Implementation-specific stuff (such as how to run the compiler) should be supplied by the implementation, not by Cabal.

I see more and more workarounds for workarounds for an unmaintainable (and unusable) build system, and after the latest discussions about git vs. darcs, maintaining GHC-specific branches of libraries etc., I think I'll just drop maintainership from all GHC-related OpenBSD ports until the GHC build system chaos settles down a little bit.

Ian, please read this. The inability to build GHC reliably is a problem. Can someone with a plan please describe what measures are in place to ensure GHC emerges buildable, and the tree regains the status of a tree that *does not break*? -- Don

On Tue, 2008-08-12 at 01:15 +0200, Johan Henriksson wrote:

I see an increasing problem in that every community comes up with their own package system, instead of reusing existing frameworks.

That's because there are no usable existing frameworks. It would be wonderful of course if there were some standard language neutral build and packaging system where each language just wrote some lib and could integrate nicely into multi-language systems.

dependencies to other non-haskell libraries has to be addressed for every other coexisting package system (such as apt-get), if it is addressed at all. likewise, other languages depending on haskell will have trouble resolving dependencies.

so my point is, if there will be any bigger reworking of cabal, I think one should consider how it could work as a module in a bigger (maybe future) meta-packaging framework, lifting up binaries to for example .deb, .exe-installer, .dmg or whatever is the most native for the platform.

There are tools to convert Cabal packages to native packages for rpm, deb, ebuild and arch. The Cabal format was designed to allow this translation. This includes dependencies on C libs and external programs. Note that this is in contrast to existing frameworks like autoconf which do not allow the automatic extraction of dependencies to allow automatic conversion into native packages.

I see a point in language specific package systems as they have more insight into the build process, but the current implementations assume a very ideal world in which there are no other dependencies involved.

I don't think this is true. Duncan

Norman Ramsey wrote:

At the time of the wonderful GHC Hackathon in Portland, where the GHC API was first introduced to the public, I urged Simon PJ to consider taking ghc --make and generalising it to support other languages. I still think this would be a good project.

As well as supporting any mix of languages (and platforms, since it's often difficult to separate the two). I would love to see such a tool! As flexible as make if necessary, but as easy as doing nothing if possible. Sean

Matthias Kilian:

On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote: [...]

...

As for Cabal - we had a thread on cvs-ghc last week, and as I said there we'd love to hear suggestions for how to improve things, including wild and crazy ideas for throwing it all away and starting again. However, as I explained, there are good reasons for the way things are done now, the main one being that the build system for packages is not written twice.

Well, at least the Makefile creation was a step (the first step?) into the wrong direction, IMHO. I'll run a GHC build to get some of those generated Makefiles and followup on cvs-ghc, but for a starter, Cabal shouldn't know anything about implementation-specific internal build systems; instead it should rely only on it's own metadata. Implementation-specific stuff (such as how to run the compiler) should be supplied by the implementation, not by Cabal.

I see more and more workarounds for workarounds for an unmaintainable (and unusable) build system, and after the latest discussions about git vs. darcs, maintaining GHC-specific branches of libraries etc., I think I'll just drop maintainership from all GHC-related OpenBSD ports until the GHC build system chaos settles down a little bit.

Thanks for demonstrating my point... Complicated build infrastructure and lack of portability used to be a big problem for GHC in the past. Over the last years, the situation got much better (to a large extent due to SimonM sanitising the makefile-based build system). Why are we so keen to throw it all away now? Manuel

On Tue, Aug 12, 2008 at 11:59:37AM +0100, Simon Marlow wrote:

...

Well, at least the Makefile creation was a step (the first step?) into the wrong direction, IMHO. I'll run a GHC build to get some of those generated Makefiles and followup on cvs-ghc, but for a starter, Cabal shouldn't know anything about implementation-specific internal build systems; instead it should rely only on it's own metadata.

I'm not completely sure, but I think you may have misunderstood how Cabal's makefile generation currently works. It has no specific knowledge of GHC's build system, and it does rely on its own metadata.

I mean the GHC-specific template used for building the Makefile (Distribution/Simple/GHC/Makefile.in) and the function `makefile` in Distribution/Simple/GHC.hs (this function even spills out some some make rules in addition to what's in Makefile.in, which looks very wrong to me). Yes, it relies only on the Cabal metadata, but the output is a Makefile only useful for building GHC. It'd be better to be able to run $ ./Setup mkmetadata -prefix foo- which just produces some simple variable declarations like foo-impl-ghc-build-depends = rts foo-impl-ghc-exposed-modules = Data.Generics Data.Generics.Aliases ... ... foo-exposed-modules = Control.Applicative Control.Arrow ... ... foo-c-sources = cbits/PrelIOUtils.c cbits/WCsubst.c ... ... foo-windows-extra-libaries = wsock32 msvcrt kernel32 user32 shell32 foo-extensions = CPP foo-ghc-options = -package-name base foo-nhc98-options = -H4M -K3M Basically, the .cabal file is just converted into some other format that may be included by another Makefile. And since it's a really simple output format, it could even be used by different implementations of make(1) or even other build tools. The `foo-' prefix just shields variables in the including Makefile. Take this output, write it to some cabalmetadata.mk, and then use a (GHC-specific) Makefile copied over into all library directories that does an include cabalmetadata.mk ... GHC_OPTS += $(foo-ghc-options) EXPOSED_MODULES = $(foo-exposed-modules) $(foo-impl-ghc-exposed-modules) EXTRA_LIBS = $(foo-extra-libraries) $(foo-$(HostOS_CPP)-extra-libraries) Thus, Cabal dumps the metadata, without knowing how it's used. All the remaining stuff are some (implementation specific) Makefiles relying on recursive variable expansion. I'll implement this for GHC when I've a little bit more spare time (in three or four weeks).

(in my other message I'm suggesting moving the Makefile generation into GHC's build system so that it could be made specific to GHC, though).

Generated files should be as simple, primitive and portable as possible. Generating complete Makefiles make things more difficult. And it doesn't matter wether they're generated by Cabal or by GHC's build system. If you've to tweak the build system, you don't want to tweak generators but just an existing Makefile.

...

Implementation-specific stuff (such as how to run the compiler) should be supplied by the implementation, not by Cabal.

This is what makes me unsure. Implementation of what?

The Haskell compiler. Or, to be more exact, the Cabal library shipped with the Haskell compiler (or some supplementary compiler-specific library -- I didn't think much about this part yet). However, my main concern is the usage of Cabal from within the GHC build system, so please just forget this part ;-)

Are you suggesting a redesign of Cabal, or just changing the way something works?

I don't think that a large redesign is necessary. It should just try to be as implementation-agnostic as possible. Ciao, Kili -- do yourself a favor and let the le(4)s rot on the junkpile of history... -- Henning Brauer

Matthias Kilian wrote:

I mean the GHC-specific template used for building the Makefile (Distribution/Simple/GHC/Makefile.in) and the function `makefile` in Distribution/Simple/GHC.hs (this function even spills out some some make rules in addition to what's in Makefile.in, which looks very wrong to me).

Yes, it relies only on the Cabal metadata, but the output is a Makefile only useful for building GHC.

Ok, this statement is plainly not true, since I can use 'cabal makefile' to build any package outside of the GHC build tree. So perhaps I've misunderstood your point?

It'd be better to be able to run

$ ./Setup mkmetadata -prefix foo-

which just produces some simple variable declarations like

foo-impl-ghc-build-depends = rts foo-impl-ghc-exposed-modules = Data.Generics Data.Generics.Aliases ... ... foo-exposed-modules = Control.Applicative Control.Arrow ... ... foo-c-sources = cbits/PrelIOUtils.c cbits/WCsubst.c ... ... foo-windows-extra-libaries = wsock32 msvcrt kernel32 user32 shell32 foo-extensions = CPP foo-ghc-options = -package-name base foo-nhc98-options = -H4M -K3M

Yes, we could use this to implement GHC's build system. It's somewhat similar to the scheme I suggested in the other thread, but more generic. I'd be completely happy to do it this way if the functionality would be useful to others outside GHC too. Cheers, Simon

On Mon, Aug 11, 2008 at 04:17:59PM +0100, Simon Marlow wrote:

One way we could create the forks would be to create a git repo for each package with two branches: the master branch that GHC builds, and a separate branch that tracks the main darcs repository, and is synced automatically whenever patches are pushed to the main darcs repo. We'd have to explicitly merge the tracking branch into the master branch from time to time. When we want to make changes locally, we can just commit them to the GHC branch and push the changes upstream in a batch later (and then we'd end up having to merge them back in to the GHC branch... but hopefully git's merge is clever enough to avoid manual intervention here). This is complicated and ugly of course; better suggestions welcome.

I don't think that this will work well. At some point we'll have to resolve conflicts that accumulate (by rebasing GHC-only patches after conflicting upstream patches come in? This may not be trivial, as the rebaser may not be familiar with the patches), and I suspect that the forks will end up diverging for significant periods of time, as there's little impetus to merge them. Even the current situation with Cabal is a bit of a pain, as it's easy to forget to push patches upstream as well as GHC's repo, and that's just with 2 repos of the same VCS. I actually think the original plan, where only the ghc repo (plus one or two others) is in git is preferable. You may have to use a different VCS for different subprojects, but after that it's downhill all the way. You don't have to worry about patches being converted from one VCS to another, moved to another repo, converted back and merged back into the first repo. I expect people will be using different VCSs for different /projects/, if not subprojects, anyway. Thanks Ian

On Tue, Aug 12, 2008 at 2:46 AM, Manuel M T Chakravarty wrote:

Ian Lynagh: Having two vcs for one project is bad. One reason to switch to git (I am told) is that people had problems with darcs on some platforms (windows and Solaris, for example). How is that going to be any better if part of the project is still in darcs? So, can we please make up our mind? If darcs has problems on some platforms, then we should not use darcs at all for ghc. If darcs does not have problems on some platforms, then there is one less reason to switch.

I switched all my Haskell projects over to Git, as a developer on Linux, is that I wasted way too much time fighting with Darcs that I should have spent programming. It's way too slow. I've run in to exponential merges with only two developers commiting to the same repository. It randomly freezes -- exponential merge, general sluggishness, who knows! -- or crashes. And merging, perhaps the most important operation in a DVCS, is a pain. I don't trust Darcs to keep my source code safe anymore. Cheers, Johan

Hello, On Mon, Aug 11, 2008 at 8:20 PM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

Simon Marlow:

...

Manuel M T Chakravarty wrote:

I think all *core* libraries must switch. Seriously, requiring GHC

...

developer to use a mix of two vcs during development is a Very Bad Idea. Don was excited about getting more people to look at the source when it is in git (see the comments he posted from reddit). By requiring two vcs you will get *less* people to look at the source. This is not only to get the sources to hack them, but you effectively require developers to learn the commands for two vcs (when they are already reluctant to learn one). For example, often enough somebody who changes something in GHC will modify the base package, too. Then, to commit the overall work, you need to commit using both vcs. If you need to branch for your work, you need to create branches in two vcs (no idea whether the semantics of a branch in git and darcs is anywhere similar). When you merge your branch, you need to merge in both vcs. You can't seriously propose such a set up!

I completely agree this is a problem. The main obstacle with just switching the core libraries is that they are shared by other implementations and other maintainers. So I see no alternative but to create forks of those repositories for use by GHC, unless/until the other projects/maintainers want to migrate to git. Some of the repositories are not shared - for example ghc-prim, integer-gmp, template-haskell, and these don't need to be forked.

One way we could create the forks would be to create a git repo for each package with two branches: the master branch that GHC builds, and a separate branch that tracks the main darcs repository, and is synced automatically whenever patches are pushed to the main darcs repo. We'd have to explicitly merge the tracking branch into the master branch from time to time. When we want to make changes locally, we can just commit them to the GHC branch and push the changes upstream in a batch later (and then we'd end up having to merge them back in to the GHC branch... but hopefully git's merge is clever enough to avoid manual intervention here). This is complicated and ugly of course; better suggestions welcome.

Yes, it's a pain. However, it is better than two vcs for one project.

I *strongly* object to moving to git before this isn't sorted out. As

...

...

Roman said before, GHC is heading into a dangerous direction. It gets progressively harder to contribute to the project at the moment. First, changing the build system to Cabal. Now, proposing to use two vcs. Somebody who is new to the project not only has to learn the internals of GHC, but they also have to learn two new vcs, and if they need to change the build system, they need to learn a new build tool. Raising the bar for developers to contribute to a project has been proven to be a very bad idea many times. Let's not take GHC down that path.

I'm not completely convinced we need to have this all worked out before GHC switches, although it would be nice of course. We currently have infastructure in place for the build to work with a mixture of darcs and git repositories, and existing developers already have to learn git anyway. They just need to remember to use darcs for libraries and git for the main GHC repo, and this is only a temporary situation.

As far as I am concerned, building GHC is turning into a big mess. We discussed ways to improve it again, BUT I'd rather not see it getting any messier before it gets better. Hence, please let's have a complete plan that we are convinced will work before making any more changes.

As for Cabal - we had a thread on cvs-ghc last week, and as I said there

...

we'd love to hear suggestions for how to improve things, including wild and crazy ideas for throwing it all away and starting again. However, as I explained, there are good reasons for the way things are done now, the main one being that the build system for packages is not written twice.

Yes, we need cabal for packages because we don't want two build systems. However, this does not justify the use of Cabal outside of libraries/. Nobody explained to me why that was necessary. Why change all the rest of the build system. What is the benefit for the ghc project?

To be honest, if you ask me, I'd go back to the old makefile based system and remove Cabal from everywhere except building of the library packages.

Manuel

I am a new developer with GHC and most of my background is with C programming and Makefile based build systems, such as with the Linux Kernel. Thus, it was much easier for me to get started hacking on GHC to only need to modify Makefiles as compared to learning an entirely different build system; therefore, I think you lower the barrier to entry for gaining new GHC developers if you stick with the Makefile build system, which is far more common, stable, robust, and definitely taught/used in many university projects. (Some of these reasons may be the same reasons GHC repo is switching to git.) As I said, I am new to hacking on GHC, so I am not sure what reasons there are to switch to Cabal for the build system; but, I am not an expert in build systems and I was able to figure out the GHC Makefiles to add static and run-time flags to GHC, etc. I definitely think the current Makefile build system could be improved, but overall I did find it quite manageable for my needs. Disclaimer: I really have very little experience with Cabal other than using it for installing packages, so take everything I have said with a grain of salt. __ Donnie Jones

Simon Marlow:

Manuel M T Chakravarty wrote:

...

To be honest, if you ask me, I'd go back to the old makefile based system and remove Cabal from everywhere except building of the library packages.

I wouldn't object to dropping the use of Cabal for other tools in the build tree; the reasons for using it elsewhere are certainly not as compelling as for packages.

Ian, I realise this means backing out a lot of the work you've been doing recently, and it would mean that we'd lose a lot of time in the runup to 6.10.1, but perhaps it's a step that we need to take to get us back on the right track again?

I do realise that this would mean backing out a lot of Ian recent work, and that's why I haven't proposed going back to the old system before you explicitly asked. However, I am increasingly getting the feeling that the move to Cabal was pre-mature, and the overall loss will be minimised by backing out now. In a sense, it was an interesting experiment and it should still be useful to the development of Cabal. In fact, I see no reason why the experiment cannot be continued on a branch. Who knows, maybe Cabal is sufficiently mature in a year to make a switch worthwhile? I just object to using the whole GHC developer community as guinea pigs. Manuel

On Wed, Aug 13, 2008 at 04:19:37PM +1000, Manuel M T Chakravarty wrote:

Simon Marlow:

...

Manuel M T Chakravarty wrote:

...

To be honest, if you ask me, I'd go back to the old makefile based system and remove Cabal from everywhere except building of the library packages.

I wouldn't object to dropping the use of Cabal for other tools in the build tree; the reasons for using it elsewhere are certainly not as compelling as for packages.

Ian, I realise this means backing out a lot of the work you've been doing recently, and it would mean that we'd lose a lot of time in the runup to 6.10.1, but perhaps it's a step that we need to take to get us back on the right track again?

I do realise that this would mean backing out a lot of Ian recent work, and that's why I haven't proposed going back to the old system before you explicitly asked. However, I am increasingly getting the feeling that the move to Cabal was pre-mature, and the overall loss will be minimised by backing out now.

We're only talking about "other tools", not the libraries (including the GHC library), right? This seems like it would be a step backwards to me (after all, I wouldn't have spent the time moving to Cabal if I didn't think it was a step forwards), and I'm not really sure what benefit you see in it: how much time do you spend working in utils/? Thanks Ian

Ian Lynagh:

On Tue, Aug 12, 2008 at 10:20:14AM +1000, Manuel M T Chakravarty wrote:

...

To be honest, if you ask me, I'd go back to the old makefile based system and remove Cabal from everywhere except building of the library packages.

Manuel

PS: Just for some more collateral damage. Did anybody check whether the Mac OS installer support and the -unfortunately, only partially working- support to compile for older OS X versions that I added to the *makefiles* still works with the Cabal-based system? I doubt it. Took me quite a while to get all this going, and I am not very well motivated to spend a lot of time to figure out how it might work with Cabal. IMHO using Cabal for anything but the libraries was a step back for no good reason.

Do you mean the "rebuilding the tools with stage2" stuff? If so, that's an interesting example to pick, as that was the impetus behind changing how the build system worked for all the non-libraries/ghc.

Rebuilding with stage1 was already needed to build GHC with a builtin readline. In general, it is a bad idea to build distributed binaries of Haskell programs with the *bootstrap compiler*. It must be done with the stage1 compiler. (If you are unsure why, I'll happily elaborate.) What I was mainly refer to is the building of GHC.framework with xcodebuild and the accompanying packing with packagemaker. Building for older versions of Mac OS X requires the MACOSX_DEPLOYMENT_TARGET and related infrastructure.

Those changes made the build non-idempotent: we would build something with the bootstrapping compiler, build some other stuff, then come back, clean it, and build it again with the in-tree compiler. This was a little annoying at the best of times, as e.g. rerunning make at the top level would needlessly rebuild some stuff.

However, when my local changes meant that programs built by GHC segfaulted, it was especially irritating to find that after (hopefully) fixing the bug I couldn't just run make in compiler/ or rts/, because ghc-pkg etc now just segfaulted!

It was at that point that I half-reverted the changes, and later I reimplemented something similar using Cabal. Now we make, for example, ghc-pkg with the bootstrapping compiler in utils/ghc-pkg/dist-inplace, and then later on we make it with the stage1 compiler in utils/ghc-pkg/dist-install.

It's of course much cleaner to build inplace versions of everything with the bootstrap compiler and separate distributeable versions with stage1. I think we briefly talked about that during the run up to 6.8.3.

To answer your actual question: No, not having OS X yet I haven't tested it, but I did make an effort to keep it working. In mk/cabal- flags.mk we say:

USE_STAGE_CONFIGURE_FLAGS = \ ... $(addprefix --cc-option=,$(MACOSX_DEPLOYMENT_CC_OPTS)) \ $(addprefix --ld-option=,$(MACOSX_DEPLOYMENT_LD_OPTS))

which will hopefully do the trick, and (IMO) in a much cleaner, more maintainable way than would have been possible with the old build system.

I appreciate that you tried to preserve it, but things like those usually don't work until explicitly tested and debugged. I think this illustrates the issue I am having with the current process. I don't think large changes that have not been properly tested should be committed to the head. I appreciate that you cannot test everything for every patch and don't have all the platforms at hand. That's why major rejigging of the build system should be done on a branch. Then, you can ask other people to test it, once it is all working well for you. Ripping the guts out of the head and leaving some of them on the floor just means everybody else is going to trip over them. Manuel

Friends | > I see more and more workarounds for workarounds for an unmaintainable | > (and unusable) build system, and after the latest discussions about | > git vs. darcs, maintaining GHC-specific branches of libraries etc., | > I think I'll just drop maintainership from all GHC-related OpenBSD | > ports until the GHC build system chaos settles down a little bit. | | Ian, please read this. | The inability to build GHC reliably is a problem. | | Can someone with a plan please describe what measures are in place | to ensure GHC emerges buildable, and the tree regains the status of a | tree that *does not break*? I don't think we should over-react here. There's been lots of email on this thread, some of which IMHO makes things sound rather worse than they really are. Let me say how it looks to me. There are two separate but loosely-related conversations going on. 1. Changes to GHC's build system. Cabal is used to build Haskell libraries. We started to use it to build the libraries that come with GHC; and we recently moved over to Cabal to build GHC itself (which is, these days, just another library). The old makefile-based system was essentially duplicating much of the functionality of Cabal, and that duplication was painful. In retrospect, we should have made this change in a branch, and tested it thoroughly before applying it to the HEAD. Build systems tend to be platform dependent, so testing on one platform isn't enough. Nor did we consult, or even communicate, enough before going ahead. And we need more Wiki documentation about how to drive the new system. The net effect of these omissions has been a lot of pain to our collaborators. I am very sorry about that. But I think it'd be a pity to confuse the pain of transition with the destination. The build system is settling down. For the moment, it probably makes sense not to aggressively pull patches from the GHC repo if you don't have to, but we absolutely do not expect that situation to persist. We'll make an announcement when we're ready for you to give it a try. The clear goal is: it simply builds flawlessly. There is an element of "dogfoooding" here. GHC is a stress test for what Cabal can do, and is itself not fully mature. But the pain we experience thereby leads to bug-fixes and significant features for Cabal that are useful for everyone. Perhaps we made the move too early though! The new design is not set in stone, and we are actively thinking about ways to improve it, *including* backing off from Cabal in places where it appears too inflexible. Of course, any such further changes would extend the period of upheaval, but (a) we'll publish a design before executing, and (b) we'll do it on a branch. 2. The version control system (VCS) At the same time, we had an extended conversation about changing the version control system we use for GHC. There was a lot of consultation here, as a result of which we chose git. I won't rehearse again the reasons we are unhappy with darcs, except to say that darcs is a thing of beauty, but the scale of GHC's repository seems to flush out many darcs bugs and performance problems that have proved difficult to fix. Unlike the build system, we have not yet executed this decision. In particular, the earlier discussions focused mainly on the relative merits of the various systems. But it's more complicated than that. GHC needs "core libraries" without which it cannot be built. It is obviously highly desirable that a developer can build GHC with just one VCS, which suggests that the core libraries should be in git too. But those same core libraries are used by nhc98 and Hugs (I think that's all), and the last thing we want to do is to impose new costs on other implementations. Diversity in implementation is a Good Thing. It's unclear exactly what to do about this. The most plausible possibility is to keep the core libraries that are shared with other implementations in darcs as now, and mirror them in git for GHC developers. That will impose pain on GHC developers to keep the git stuff in sync with the darcs master copies; but at least other developers would be unaffected. It's a hard judgement call to say which pain is greatest: the pain of staying with darcs, or the pain of managing the two-VCS problem. Regardless, though, if all you want to do is build GHC from scratch, it absolutely will be a question of getting the relevant VCS, installing support software (Happy, Alex, an earlier GHC), and typing 'make'. You won't have to know about funny branches. We (GHC HQ) are still learning the transition to wider participation in building and hacking on GHC, which we *very much* welcome. Bear with us if we don't get it right first time. We're trying! Simon

On 12 Aug 2008, at 15:46, Simon Peyton-Jones wrote:

It's unclear exactly what to do about this. The most plausible possibility is to keep the core libraries that are shared with other implementations in darcs as now, and mirror them in git for GHC developers. That will impose pain on GHC developers to keep the git stuff in sync with the darcs master copies; but at least other developers would be unaffected.

FWIW, I started a wiki page that tries a direct comparison between Darcs and Git: http://hackage.haskell.org/trac/ghc/wiki/GitForDarcsUsers Some mappings are simple, some are more complicated and will require adopting a different workflow. I still recommend reading a tutorial, but this cheat sheet should be a good start if you don't want to spend much time to learn Git just yet. Where no directly corresponding command exists or emulating it would be too messy, I try to hint towards other work flows. I encourage everyone to add useful tips and examples both from users who already use Git and later on, once we have gathered more experience. I believe that Git has some features which can improve our productivity and I'd like this page to also collect tips how to do so. / Thomas -- Push the envelope. Watch it bend.

On Tue, Aug 12, 2008 at 03:17:59PM -0400, Isaac Dupree wrote:

Thomas Schilling wrote:

...

I encourage everyone to add useful tips and examples both from users who already use Git and later on, once we have gathered more experience. I believe that Git has some features which can improve our productivity and I'd like this page to also collect tips how to do so.

what about `darcs send --dry-run`? It's not perfect, but I use it in my old repos in conjunction with `darcs wh [-l]` to find out what of value I'd lose by deleting an old checkout. (e.g., patches merged into HEAD aren't of value. But they still aren't of value even if they've been amend-recorded, rewritten, or equivalent by simon/ian/etc., but Darcs can't tell this, unfortunately.)

-Isaac

Hi Isaac, git rebase can do this partially. See this example that's what I know about (make sure you don't have important data in /tmp/xx) How intelligent git behaves on partially applied / cherry picked commits I don't know. #!/bin/sh echO(){ echo; echo " >>>> $@"; echo 'return to continue'; read; } evaL(){ echo; echo "cmd: $@"; eval "$@"; } cd /tmp/xx || exit 1 rm -fr * .* set -e git init addfile(){ echo $1 > $1 git add $1 git commit -m $1 -a } evaL 'addfile a' evaL 'addfile b' evaL 'addfile c' evaL 'addfile d' echO 'a,b,c,d recorded succesfully' evaL 'git checkout HEAD~2' echO 'gone back two commits' evaL 'git checkout -b mutate' echO 'branch mutate created' evaL 'addfile new' echO 'new file new added which would be lost' evaL 'git cherry-pick master' evaL 'git cherry-pick master^' echO 'cherry picked d c in reverse order, look at popping up gitk now (you may want to keep it open)' evaL 'gitk --all &' echO 'continue after gitk has popped up, you should see one branch' evaL 'git checkout -b rebased' evaL 'git rebase master rebased' echO 'tried rebasing, data which would be lost should be ahead of master now' echO 'opening second gitk showing current repo state' evaL 'gitk --all' echO 'if this is not enough, you can always use git-diff:' evaL 'git diff mutate master'

Isaac see third

FWIW, I started a wiki page that tries a direct comparison between Darcs and Git:

http://hackage.haskell.org/trac/ghc/wiki/GitForDarcsUsers

Some mappings are simple, some are more complicated and will require adopting a different workflow. I still recommend reading a tutorial, but this cheat sheet should be a good start if you don't want to spend much time to learn Git just yet. Where no directly corresponding command exists or emulating it would be too messy, I try to hint towards other work flows.

I encourage everyone to add useful tips and examples both from users who already use Git and later on, once we have gathered more experience. I believe that Git has some features which can improve our productivity and I'd like this page to also collect tips how to do so.

Hi Thomas: Great work! There is not much I could add (although I've used git during the last weeks quite often..) However I'm missing four small tips: first man git-rev-parse (or git rev-parse help) HEAD HEAD^ HEAD^^ .. is equal to HEAD HEAD~1 HEAD~2 ..> So to drop one of the last ten commits (don't remember which one) git rebase -i HEAD~10 ... second : you forgot to mention gitk. It helps you getting an overview about when which branches have been created You can use google pictures search to see how it looks like or just play around (try the script in my other post).. You can have a look at the history and branches easily.. You can even highlight commits by changes made to filepath (must be relative to repo path!) or by adding/ removing strings etc.. And it's a nice tool to just keep all hashes in memory in case you mess up your repo by accident :-) But recent gitk can do more. When getting some conflicts on git merge or git rebase gitk --merge will show you all commits causing this conflict. third: #git on freenode.. I bet you'll get help there as well.. I got the last tip there as well < doener> MarcWeber: you could, for example, do "git log or git rev-list or gitk --left-right --cherry-pick A...B" lists all commits beeing present on the one or the other branch, but not in both 4th: You should know one thing about git history There used to be no difference between git-log (now depreceated, does no longer work in the git git version) and git log Thus git log --help = git-log --help = man git-log (more convinient to type) The only execption: git-clone doesn't work in all cases, git clone does (?) (Don't ask me why) maybe git show commit-id:file is of interest as well (you told about git show) Sincerly Marc Weber

Manuel M T Chakravarty wrote:

Everybody who contributes to the boot/core libraries needs to validate their patches. If the GHC version of the libraries is in git, then all library code needs to be validated against the git version of the libraries before it can enter the master repository. I don't see how that makes anything easier for anybody.

As I said before, I believe there is exactly one sane solution: all boot libraries use the same vcs as ghc.

I don't think this completely sane :-) It's not fair or reasonable for the GHC project to require everyone else contributing to the core libraries to validate their changes against GHC. We need to be branching the shared repositories, so that we can keep the GHC branches working and sync up with the shared repositories as necessary. Morally this is the right thing; technically its a lot more difficult than not forking. What would make it much easier is for both the original shared repository and GHC's branch to be git repositories (or branches of the same repo): git is really good at having two parallel lines of development that sync occasionally. Having one in darcs and one in git would be painful, but doable. So I suggest we propose moving all the core packages to git, and we translate all those for which nobody objects to the change. For the others, we'll keep them in darcs and live with the pain. Cheers, Simon

Neil Mitchell:

If it really makes the life easier for people who are having lots of VCS pain at the moment, then its hard to object. But many of the comments in this discussion, about how everyone is going to flock to GHC just as soon as it switches to Git, seem overly optimistic. I think GHC is a few years off becoming drive-by hacker friendly, for many other reasons.

It's not about becoming "drive-by hacker friendly". It is about not becoming even less friendly as it is right now. Manuel

Thomas Schilling:

Are you advocating for ease of use by new developers or for existing developers? Current GHC hackers have to learn Git anyways and know Darcs already. Library patches still have to be recorded separately, so it would be a bit weird, but not much harder, really.

I am arguing for both. It would be more than weird. For example, if you branch ghc, you usually need to branch the core libraries, too. Doing that in two different vcs sounds like a mess to me. Moreover, as I wrote a few times before, some reasons for switching in the first place are invalidated by not having the core libraries in git, too. For example, one complaint about darcs is that it either doesn't build (on the Sun Solaris T1 and T2 machines) or is buggy (on Mac OS with MacPorts), and hence people have trouble getting the sources out of darcs in the first place. How is that going to be addressed if some crucial code still needs to be obtained using darcs? Manuel

On Fri, Aug 15, 2008 at 1:59 AM, Manuel M T Chakravarty wrote:

...

Neil Mitchell:

...

If it really makes the life easier for people who are having lots of VCS pain at the moment, then its hard to object. But many of the comments in this discussion, about how everyone is going to flock to GHC just as soon as it switches to Git, seem overly optimistic. I think GHC is a few years off becoming drive-by hacker friendly, for many other reasons.

It's not about becoming "drive-by hacker friendly". It is about not becoming even less friendly as it is right now.

Gregory Wright:

On Aug 14, 2008, at 9:12 PM, Manuel M T Chakravarty wrote:

...

Moreover, as I wrote a few times before, some reasons for switching in the first place are invalidated by not having the core libraries in git, too. For example, one complaint about darcs is that it either doesn't build (on the Sun Solaris T1 and T2 machines) or is buggy (on Mac OS with MacPorts), and hence people have trouble getting the sources out of darcs in the first place. How is that going to be addressed if some crucial code still needs to be obtained using darcs?

Regarding darcs on OS X from MacPorts, I am not aware (or have been sent any bug reports) that there are problems with the latest darcs-2.0.0 port. Is there something that I should know (and try to fix)?

The latest port defaults to wget instead of libcurl since I have noticed darcs spinning endlessly when using libcurl. I haven't had time to dtrace what is going on but I'm guessing the underlying problem is likely some misunderstanding of the signal handling API or some corner case of blocking/nonblocking IO.

Well, that "spinning endlessly" is the bug I am referring to. I re- checked my MacPorts darcs2 installation and, you are right, there was an update that removes the use of libcurl. It seems to work *much* better now. Thanks for the fix! You may want to publicise this a bit further. When I asked on #darcs about the problem a few days ago, nobody knew about this update to the port. Manuel

2008/8/15 Isaac Dupree :

So let's figure out how it would work (I have doubts too!) So, within the directory that's a git repo (ghc), we have some other repos, git (testsuite) and darcs (some libraries). Does anyone know how git handles nested repos even natively?

You can explicitly tell Git about nested Git repos using http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html. This essentially associates a particular version of each subrepo with every version of the repo that contains them, so e.g. checking out GHC from 2 weeks ago could check out the libraries from the same point in time. AFAIK, nothing in Git caters for subrepos of a different VCS.

Then, adding complexity, git branches are normally done by switching in-place. So how does this interact with VCS like darcs that doesn't have a concept of in-place switching of branches?

Since we will set up Git to ignore the contents of the Darcs repos, it will simply leave them unmodified. This is exactly like the current situation, where rolling back / patching the GHC repo does not affect the others. If you want Darcs-like behaviour (one branch per repo) you are free to do this in Git as well, in which case since you never switch branches the nested Darcs repos should never be inappropriate for your branch. Personally, since I only ever hack GHC and tend to leave the libraries alone, I could still use the in-place branching without difficulty.

(Now, I wouldn't be surprised if git, the monstrosity that it is, has already invented answers for these sort of questions :-) But we need to figure out the answers for whatever situation we choose for the 6.11 development cycle, and probably document them somewhere on the wiki (that I lazily didn't bother to check again before writing this message).

The situation above is pretty much the whole story, if we are taking the route where we just convert the GHC+testsuite repo to Git. I don't think it's particularly confusing, but maybe that's because I've spent too long thinking about VCSs :-). This thread has got quite large, and doesn't appear to have made much progress towards a resolution. Let me try and sum up the discussion so far. There seem to be four stakeholders in this switch: a) Current GHC developers b) Future GHC developers c) People who just contribute to the libaries d) Maintainers of other compilers GHC shares repos with And there are at least 5 options for how to proceed: 1) Convert just GHC and Testsuite to Git, leave everything else in Darcs Pros: - No change in habits required for stakeholders c, d - Resolves all Darcs issues discussed at length before, pleasing stakeholders a, b Cons: - Requires two VCSs to be installed and learnt (more points of failure, makes source tree less accessible, doesn't solve any Darcs' build+install problems), affecting stakeholders a and b - Difficult to check out a consistent version of the source tree (no submodules), affecting stakeholders a and b 2) Wait for Darcs2 to get better Pros: - No change in habits required for any stakeholders (though we still have one-off switching cost) - Potentially resolves all Darcs issues, pleasing stakeholders a, b - Only option that will not require a workflow change for GHC developers (more topic branches rather than "spontaneous branches" and cherry-picking), pleasing stakeholders a Cons: - Darcs will probably continue to be less popular and well supported than Git (see Debian popcon graphs for the trend difference). Reduced popularity will affect the ability of stakeholders b to contribute (learning barrier), and less support/real world use may potentially lead to a higher incidence of bugs encountered, affecting stakeholders a-d. This point is certainly debatable. - Apparently somewhat vaporware at the moment 3) Convert all repos to Git Pros: - Native Git submodule integration, makes life easier for stakeholders a-b - Single (popular) command set to learn, single thing to install: makes life better for stakeholder b at least Cons: - Significant inconvenience for stakeholders c-d as they have to change their own projects 4) Branch all repos into Git but leave the Darcs repos alone and push Darcs patches into the Git repos automatically. Never push to these Git repos in any other way, similar to Cabal repo currently Pros: - As option 3 - Stakeholders c-d do not need to do anything Cons: - Makes it harder to hack on the libraries within a GHC checkout, affecting a, b - Automatic synchronisation will require occasional maintenance by someone 5) Branch all repos into Git and then set up a manual merging / sync process that tries to turn Git commits into Darcs patches and vice-versa Pros: - As option 3 - Hack on the libraries in a GHC checkout with ease, pleasing a, b - Stakeholders c-d do not need to do anything Cons: - Synchronisation much more fragile than 4), will likely require constant maintenance This summary is probably incomplete and inaccurate. However, if people find it useful for organising the various lines of discussion on this issue, perhaps someone could Wikify it so we can get a complete, clear picture? My personal preference is for 3), but that's because I'm a stakeholder "a" who isn't a great fan of spontaneous branches! Anyway, there are good arguments on every side, so I don't want to advocate a particular position (and indeed, my opinions quite rightly do not carry any weight! :-). However I'd really like for us to work out what is going on so we have a clear plan for moving away from Darcs 1, which is an inadequate VCS for GHC for reasons that have been discussed to death. I hope (perhaps naively) that this email can provide a framework for reaching a consensus agreeable to all parties. All the best, Max

On Fri, Aug 15, 2008 at 4:38 PM, Ian Lynagh wrote:

One way that it is worse is that you will get a lot more "automatic merge" commits when you pull changes from the central repo into a repo in which you have local commits. I don't think that there is anything bad about these, as such; they're just noise in the history. (I'm not sure if it's possible to automatically rebase these away, or something?).

This is the use case for "git pull --rebase". Instead of creating an automatic merge commit, it rebases your local changes on top of the newly pulled changes (ignoring patches already present, which could happen if you had sent one change as a patch via mail.) The timestamp issue seems tricky, though.

On Fri, Aug 15, 2008 at 04:24:12PM +0100, Ian Lynagh wrote:

On Fri, Aug 15, 2008 at 05:09:55PM +0200, Thomas Schilling wrote:

...

On Fri, Aug 15, 2008 at 4:38 PM, Ian Lynagh wrote:

...

One way that it is worse is that you will get a lot more "automatic merge" commits when you pull changes from the central repo into a repo in which you have local commits. I don't think that there is anything bad about these, as such; they're just noise in the history. (I'm not sure if it's possible to automatically rebase these away, or something?).

This is the use case for "git pull --rebase". Instead of creating an automatic merge commit, it rebases your local changes on top of the newly pulled changes

Hmm, last night the conversation went:

< nominolo> malcolmw: so i'm advocating "git pull --rebase" for that use case < glguy_> rebasing can be less successful than merging when dealing with big changes < glguy_> since the rebase happens one commit at a time

so I'm confused as to what the best practice is.

We discussed this in #ghc, and the conclusion seems to be: If you have lots of local changes (e.g. the sorts of long-running branch that gives darcs 1 problems), then you need to use merge. If you use rebase then you might end up with lots of conflicts to manually resolve. Using merge gives you automatic merge commits, If you think these are ugly (opinion is divided on that amongst git people; I guess for GHC we'd want to make a global decision about that) then you can use rebase when you have few local changes, and thus you are unlikely to get many conflicts. Using merge you also get a more accurate reflection of the project history, i.e. you can see that the two branches were being developed independently. Thanks Ian

Ian Lynagh:

On Fri, Aug 15, 2008 at 04:24:12PM +0100, Ian Lynagh wrote:

...

On Fri, Aug 15, 2008 at 05:09:55PM +0200, Thomas Schilling wrote:

...

On Fri, Aug 15, 2008 at 4:38 PM, Ian Lynagh wrote:

...

One way that it is worse is that you will get a lot more "automatic merge" commits when you pull changes from the central repo into a repo in which you have local commits. I don't think that there is anything bad about these, as such; they're just noise in the history. (I'm not sure if it's possible to automatically rebase these away, or something?).

This is the use case for "git pull --rebase". Instead of creating an automatic merge commit, it rebases your local changes on top of the newly pulled changes

Hmm, last night the conversation went:

< nominolo> malcolmw: so i'm advocating "git pull --rebase" for that use case < glguy_> rebasing can be less successful than merging when dealing with big changes < glguy_> since the rebase happens one commit at a time

so I'm confused as to what the best practice is.

We discussed this in #ghc, and the conclusion seems to be:

If you have lots of local changes (e.g. the sorts of long-running branch that gives darcs 1 problems), then you need to use merge. If you use rebase then you might end up with lots of conflicts to manually resolve.

Using merge gives you automatic merge commits, If you think these are ugly (opinion is divided on that amongst git people; I guess for GHC we'd want to make a global decision about that) then you can use rebase when you have few local changes, and thus you are unlikely to get many conflicts.

Using merge you also get a more accurate reflection of the project history, i.e. you can see that the two branches were being developed independently.

Sorry for being a git n00b, but does using merge mean that we need to use in-place branch switching (which you earlier said won't work well for ghc anyways)? Manuel

Sorry for being a git n00b, but does using merge mean that we need to use in-place branch switching (which you earlier said won't work well for ghc anyways)?

You have to kind of "branches" : local ones and remote ones. remote ones represent the state of remote ones. The only way I know of to change them is by using git-fetch (which is called by git pull as well) or by editing the files manually On the other hand you normally push your local ones. So if you have /tmp/a/.git (heads master and mybranch) than do cd /tmp git clone a b git will setup .git/refs/remotes/{master,mybranch} .git/refs/heads/master Now you can make mybranch local as well by git branch mybranch remotes/mybranch (indeed you have 4 branches, 2 tracking the remote repo and two you are working with) Now you can do git checktout master; git merge mybranch # merge in place within the same repo or git merge remotes/mybranch # merge with remote branch which is what you'll do when using darcs branch style etc.. After comitting to mybrach git push will change the head of the remote repository. You can setup each local head branch to "track" a remote oone automaticaly so that git pull will rebase or merge depending on your settings (AFAIK) So if you have an active project you end up having dozens of remote branches but only some "heads" you are working on or you want to backup (in case someone else rewrits history or such) HTH Marc Weber

2008/8/15 Ian Lynagh :

...

You can explicitly tell Git about nested Git repos using http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html. This essentially associates a particular version of each subrepo with every version of the repo that contains them, so e.g. checking out GHC from 2 weeks ago could check out the libraries from the same point in time.

We were talking about this last night on #ghc, and AIUI this doesn't play well with the in-tree branching style that is advocated, e.g. if you want to branch ghc and base then as you change between ghc branch X and Y, git won't automatically change base between branches X' and Y'.

If you change the submodules in branch X to point to the X' commit in base, and do the corresponding thing for Y and Y', I believe you /would/ get this behaviour (though you might have to remember to do "git submodule update" when switching.. this can probably be automated). Provisio: I'm also not a Git expert, but this is my understanding of how it works. Cheers, Max

you don't use local branches? On Sat, Aug 16, 2008 at 12:04 AM, Johan Tibell wrote:

On Fri, Aug 15, 2008 at 4:38 PM, Ian Lynagh wrote:

...

One way that it is worse is that you will get a lot more "automatic merge" commits when you pull changes from the central repo into a repo in which you have local commits. I don't think that there is anything bad about these, as such; they're just noise in the history. (I'm not sure if it's possible to automatically rebase these away, or something?).

I'm not sure if this is what you want but I always use git pull --rebase when I'm pulling to have my local commits lie on top of the one in the published repo.

-- Johan _______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

From what you are saying, it seems that one "advantage" of git (in- place branch switching) is not going to be useful to GHC in any case (because we use nested repositories). Manuel Ian Lynagh:

On Fri, Aug 15, 2008 at 01:01:08PM +0100, Max Bolingbroke wrote:

...

2008/8/15 Isaac Dupree :

...

So let's figure out how it would work (I have doubts too!) So, within the directory that's a git repo (ghc), we have some other repos, git (testsuite) and darcs (some libraries). Does anyone know how git handles nested repos even natively?

You can explicitly tell Git about nested Git repos using http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html. This essentially associates a particular version of each subrepo with every version of the repo that contains them, so e.g. checking out GHC from 2 weeks ago could check out the libraries from the same point in time.

We were talking about this last night on #ghc, and AIUI this doesn't play well with the in-tree branching style that is advocated, e.g. if you want to branch ghc and base then as you change between ghc branch X and Y, git won't automatically change base between branches X' and Y'.

...

...

Then, adding complexity, git branches are normally done by switching in-place. So how does this interact with VCS like darcs that doesn't have a concept of in-place switching of branches?

The in-tree branching style also sounds like it won't work well with trees you are working in: If you have a tree built with branch X, and then you swap to branch Y for a minute and then back to branch X, then the timestamps on any source files that differ between the branches will have changed, so the build won't think it is up-to-date any more and you will get needless recompilation.

Working only in the "master" branch, and using different repos for branches (i.e. doing what we do with darcs), is an option, although git users seem to think it is a worse way to work; I'm not really clear on the main reasons why.

One way that it is worse is that you will get a lot more "automatic merge" commits when you pull changes from the central repo into a repo in which you have local commits. I don't think that there is anything bad about these, as such; they're just noise in the history. (I'm not sure if it's possible to automatically rebase these away, or something?).

Hopefully a git person will correct me if I've got something wrong!

Thanks Ian

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Ian Lynagh:

On Mon, Aug 18, 2008 at 12:21:47PM +1000, Manuel M T Chakravarty wrote:

...

From what you are saying, it seems that one "advantage" of git (in- place branch switching) is not going to be useful to GHC in any case

Yes.

...

(because we use nested repositories).

That does make it harder, but the main problem is that switching between branches changes the timestamp of files that differ, meaning the build system thinks that recompilation needs to be done.

Also, if you have 2 in-place branches of GHC then only one of them can be built at any one time, as they share a working directory.

That doesn't sound like GHC-specific issues. So, if inplace branches are useful for other projects -such as the Linux kernel- why shouldn't it be useful for us? Manuel

Manuel M T Chakravarty wrote:

From what you are saying, it seems that one "advantage" of git (in-place branch switching) is not going to be useful to GHC in any case (because we use nested repositories).

As far as I can tell, in-place branches are not a lot of use to us compared to just having separate checkouts for each local branch. For one thing, having separate source trees lets you keep multiple builds, whereas with in-place branches you can only have one build at a time, and switching branches probably requires a complete rebuild. However, I think I am convinced that using in-place branches for the master repo makes sense. That way we don't need to publish the names of new branches when we make them, and everyone can easily see which branches of GHC are available from the main repo. Cheers, Simon

Max Bolingbroke:

...

Then, adding complexity, git branches are normally done by switching in-place. So how does this interact with VCS like darcs that doesn't have a concept of in-place switching of branches?

Since we will set up Git to ignore the contents of the Darcs repos, it will simply leave them unmodified. This is exactly like the current situation, where rolling back / patching the GHC repo does not affect the others. If you want Darcs-like behaviour (one branch per repo) you are free to do this in Git as well, in which case since you never switch branches the nested Darcs repos should never be inappropriate for your branch.

This ignores that the ability to have branches, switch between them, and merge has been cited as one of the reasons for switching to git. Embedded darcs library repos would hence nullify, or at least reduce, one of the advantages. Manuel

On 16/08/2008, at 00:12, Ian Lynagh wrote:

On Fri, Aug 15, 2008 at 11:12:20AM +1000, Manuel M T Chakravarty wrote:

...

Moreover, as I wrote a few times before, some reasons for switching in the first place are invalidated by not having the core libraries in git, too. For example, one complaint about darcs is that it either doesn't build (on the Sun Solaris T1 and T2 machines)

I don't remember seeing this mentioned before, and googling for "Solaris T1" darcs doesn't find anything. What goes wrong? I'd expect darcs to build anywhere GHC does.

I only vaguely remember what was wrong but IIRC, the problem was that darcs 1.0.? didn't build with GHC 6.8.? because of some incompatibility in the libs and darcs 2 built ok but didn't work, probably because of libcurl issues. At that point I gave up. Roman

Duncan Coutts wrote:

On Fri, 2008-08-15 at 15:12 +0100, Ian Lynagh wrote:

...

On Fri, Aug 15, 2008 at 11:12:20AM +1000, Manuel M T Chakravarty wrote:

...

Moreover, as I wrote a few times before, some reasons for switching in the first place are invalidated by not having the core libraries in git, too. For example, one complaint about darcs is that it either doesn't build (on the Sun Solaris T1 and T2 machines) I don't remember seeing this mentioned before, and googling for "Solaris T1" darcs doesn't find anything.

That's probably because there entire world are probably only two T1/T2 machines that people are using to run ghc. :-)

One of them is at UNSW and the other was recently donated by Sun to the community and is just about to go online at Chalmers.

...

What goes wrong? I'd expect darcs to build anywhere GHC does.

So would I usually, though I've had to turn down cc flags to get darcs to build on ia64 before (SHA1.hs generates enormous register pressure).

We should really use a C implementation of SHA1, the Haskell version isn't buying us anything beyond being a stress test of the register allocator. Cheers, Simon

Git 1.6.0 was just released [1]. Might be of interest given the current discussion. I cherry picked some highlights that might matter to us: * Source changes needed for porting to MinGW environment are now all in the main git.git codebase. * even more documentation pages are now accessible via "man" and "git help". * "git-add -i" has a new action 'e/dit' to allow you edit the patch hunk manually. 1. http://lkml.org/lkml/2008/8/17/174 Cheers, Johan

On 19/08/2008, at 8:57 PM, Ian Lynagh wrote:

On Mon, Aug 18, 2008 at 09:20:54PM +1000, Ben Lippmeier wrote:

...

Ian: Did this problem result in Intel CC / GCC register allocator freakouts?

Have you got me confused with someone else? I don't think I've ever used Intel CC.

Sorry, I couldn't find the rest of the preceding message. Someone wrote that they had to turn down cc flags to get SHA1.hs to compile on IA64. What C compiler was being used, and what were the symptoms? SHA1.hs creates vastly more register pressure than any other code I know of (or could find), but only when -O or -O2 is enabled in GHC. If -O and -prof are enabled then the linear allocator runs out of stack slots (last time I checked). I'm wondering three things: 1) If the C compiler could not compile the C code emitted by GHC then maybe we should file a bug report with the CC people. 2) If the register pressure in SHA1.hs is more due to excessive code unfolding than the actual SHA algorithm, then maybe this should be treated as a bug in the simplifier(?) (sorry, I'm not familiar with the core level stuff) 3) Ticket #1993 says that the linear allocator runs out of stack slots, and the graph coloring allocator stack overflows when trying to compile SHA1.hs with -funfolding-use-threshold20. I'm a bit worried about the stack over-flow part. The graph size is O(n^2) in the number of vreg conflicts, which isn't a problem for most code. However, if register pressure in SHA1.hs is proportional to the unfolding threshold (especially if more than linearly) then you could always blow up the graph allocator by setting the threshold arbitrarily high. In this case maybe the allocator should give a warning when the pressure is high and suggest turning the threshold down. Then we could close this issue and prevent it from being re-opened. Cheers, Ben.

I personally don't know Git, and while I'm sure I'll be learning at some point, I'm always nervous about learning a VCS on something I care about, as mistakes can go quite wrong. If I can lend you (or someone else) a hand don't hesitate to contact me. (I'm not a git guru though..) With git you can't get too much wrong because it's very cheap to create additional pointerns / branches.. So if you clone a branch before taking any action you can always reset the messed up branch to the "backup" by git reset --{soft or hard} backupbranchname Or you could write a two line sh script writing all hashes to a temp file etc.. If you just start gitk it will keep all hashes in memory.. so you can recover from those as well (unless using the update menu item or running the garbage collector)

Sincerly Marc Weber

Claus Reinke wrote:

Perhaps it would be useful for GHC HQ to have a GHC project blog,

Actually we have talked about doing that, and it's highly likely we'll set one up in due course. I think it's worth letting the current discussion(s) run their course and then we'll have a set of concrete decisions to act upon, one of which will probably be to set up a blog so that GHC devs can communicate what they're up to. Cheers, Simon

Alexander Dunlap wrote:

On Tue, Aug 5, 2008 at 2:23 AM, Simon Marlow wrote:

...

(notice how fast that is :-)

git clone has gone about 45 minutes so far without finishing...is that an improvement over darcs?

I think http is still bandwidth-throttled on darcs.haskell.org. You should get better results cloning the github mirror: git://github.com/ghc-hq/ghc.git Thomas Schilling set this up yesterday. Cheers, Simon

2008/8/6 david48 :

cat: _darcs/prefs/defaultrepo: No such file or directory Couldn't work out defaultrepo at ./darcs-all line 27.

You can't yet build from the Git repo, alas. I've added the necessary patches and scripts (you need sync-all, not darcs-all) to http://hackage.haskell.org/trac/ghc/wiki/DarcsConversion but they haven't yet been commited to HEAD. Cheers, Max

On 6 Aug 2008, at 12:35, Samuel Tardieu wrote:

...

...

...

...

...

"Simon" == Simon Marlow writes:

Simon> We already have an up-to-date git mirror thanks to Thomas Simon> Schilling:

Simon> git clone http://darcs.haskell.org/ghc.git

Simon> (notice how fast that is :-)

It would be even much faster if you (Thomas?) setup a git server. It is as easy as "touch git-daemon-export-ok" in the GIT repository and launching "git-daemon /path/to/parent/of/git/repo" at boot time, as shown by Chris Double at

http://www.bluishcoder.co.nz/2007/09/how-to-publish-git- repository.html

Then the "git://" protocol can be used, which makes intelligent decisions on what needs to be transferred.

Thanks, I will look into it. I need to talk to our admin anyway. / Thomas -- My shadow / Change is coming. / Now is my time. / Listen to my muscle memory. / Contemplate what I've been clinging to. / Forty-six and two ahead of me.