On 10/01/2011 13:02, Max Bolingbroke wrote:

On 10 January 2011 11:19, Simon Marlow wrote:

...

Let us know what you think - would this make life harder or easier for you? Would it make you less likely or more likely to contribute?

Well, as a sometime-contributor I would certainly be happier hacking on GHC if it were git based. When working on a GHC branch, it is rather irritating to spend time work around the inevitable Darcs bugs rather than hacking on the code.

However, I remember the last time this came up there were some issues that might make migration painful. From the top of my head:

1) Some people expressed concern that they would have to use two revision control systems to work on GHC, because not all GHC dependencies would be git-based.

It would be a prerequisite to switching that a GHC developer only has to use one VCS. So we either migrate dependencies to git, or mirror them in GHC-specific git branches.

2) There was also concern that Git isn't so great on Windows. I have heard that this is less of an issue now, but I never personally suffered from any problems, so can't be sure. (FWIW I used Git on Windows industrially ~1 year ago for 3 months and didn't have problems, though the people around me occasionally had issues with e.g. case insensitivity causing obscure error messages).

Again, it would be a prerequisite that all our workflows work on Windows too. We'd have to do some research to check for problems.

3) The git transition has the potential to make Ian's workflow (i.e. maintaining branches for old GHC releases) harder.

AFAIK currently Ian just "darcs pulls" a particular patch from GHC HEAD into e.g. the GHC 7 branch. Darcs automatically works out which of the earlier patches are dependent on that patch and deals with pulling them in as well (if they are not already present).

If GHC HQ transitioned to Git but committed all patches to master then Ian's job might be significantly harder because he would have to use "git cherry-pick" to pick out any bug fix patches that should be merged back to e.g. GHC 7. Unfortunately, Git does not provide any mechanism for automatically working out which earlier patches commits on the cherry-picked commit, so this will fail quite often. Ian will then have to manually identify the dependents.

To make Ian's life easier GHC HQ could adopt a new workflow. For example, bug fixers could fix their bugs on *new* branches (one peg bug) which start from the last major GHC release (right now, GHC 7). After fixing the bug on that branch, they can then merge the branch into master. Now when Ian merges a bugfix to a GHC 7 patchlevel release he just needs to merge that bug fixing branch into the ongoing GHC 7 branch.

Naturally other workflows are possible and I'm sure other list members will chime in with their own favourites :-)

I don't think the dependencies get very deep in most cases, and my impression is that we often don't want to pull the dependencies anyway, so darcs forces us to merge the patch manually (Ian would be able to say for sure how often this happens). However, if it turned out that we had to change this workflow it wouldn't be the end of the world. Fixing bugs on the stable branch rather than HEAD would be a slight inconvenience, but is arguably the right thing anyway.

Has GHC HQ thought about these workflow issues? Are you happy with any changes that might be required to your workflows?

Ultimately I'm quite concerned with keeping GHC HQ happy (as you guys do the lions share of the work!). I feel we should only make the switch if the most frequent committers (i.e. Simon, Simon and Ian) are *totally happy* with it and any associated workflow changes that may be required.

Speaking for myself, I tend slightly towards making the switch, becuase I'm keen to make branching less painful. However, I think if it were just the three of us, there probably wouldn't be enough motivation to overcome the cost of switching, but if there is enough interest from the rest of the community that might just swing it. Cheers, Simon

On Mon, Jan 10, 2011 at 2:34 PM, Pavel Perikov wrote:

Please please consider Mercurial if migration from darcs is inevitable :)

While Mercurial is a fine choice, I think there are more Haskellers that use Git than Mercurial. Probably because GitHub is such an awesome service. Johan

On Mon, Jan 10, 2011 at 2:43 PM, Pavel Perikov wrote:

On 10.01.2011, at 16:40, Johan Tibell wrote:

...

While Mercurial is a fine choice, I think there are more Haskellers that use Git than Mercurial. Probably because GitHub is such an awesome service.

Interesting. It will be great to see any numbers (really, just curious).

No real numbers. I've just observed what other Haskellers talk about and where I usually find projects (when they are not in Darcs). We could probably pull the numbers of Hackage. Johan

On Mon, Jan 10, 2011 at 5:08 PM, Pavel Perikov wrote:

Probably most valuable are the opinions of GHC development team of course :) Git really seem to be more popular, Mercurial just seem more streamlined to me :)

Their preference if of course very important, but they partly wanted to make the change to get more contributors so in my opinion it makes sense to switch to something that the majority of Haskellers use. I'm not trying to get into a Git vs Mercurial argument here. I have more important things to do, like writing code. :) Johan

On Mon, Jan 10, 2011 at 5:34 AM, Pavel Perikov wrote:

Please please consider Mercurial if migration from darcs is inevitable :)

For what it's worth, Mercurial generally interoperates quite well with git and github, using the hg-git plugin. As a longtime Mercurial user and an occasional GHC contributor, it wouldn't be a practical problem for me if GHC were to switch to git.

On Mon, Jan 10, 2011 at 01:27:17PM +0000, Simon Marlow wrote:

On 10/01/2011 13:02, Max Bolingbroke wrote:

...

2) There was also concern that Git isn't so great on Windows. I have heard that this is less of an issue now, but I never personally suffered from any problems, so can't be sure. (FWIW I used Git on Windows industrially ~1 year ago for 3 months and didn't have problems, though the people around me occasionally had issues with e.g. case insensitivity causing obscure error messages).

Again, it would be a prerequisite that all our workflows work on Windows too. We'd have to do some research to check for problems.

The environment provided by msysgit is reasonably usable, and performance hasn't been much trouble for me personally. The binaries from it can also be used in syntevo's SmartGit product, which has a free license for non-commercial purposes, so if the gitk tools of msysgit are considered inadequate, SmartGit tends to get the job done. The only problem I've had with msysgit is that it has its own set of compilers and msys tools, which might conflict with the existing toolchains used with GHC, something which I'm unsure if it's solvable as Git has a horrible tendency to rely on having shell tools to do its ill deeds. As for whether to change or not, I'm neutral as I'm not a contributor nor have any plans to be. -- Lars Viklund | zao@acc.umu.se

On Mon, Jan 10, 2011 at 01:27:17PM +0000, Simon Marlow wrote:

It would be a prerequisite to switching that a GHC developer only has to use one VCS. So we either migrate dependencies to git, or mirror them in GHC-specific git branches.

I think it's hard to know how well it's going to work in advance (I was going to try redoing all the GHC 7 merges with git, but that won't tell us much if they would have been recorded differently in a git workflow), so perhaps we should change only the GHC repo at first, and see how it goes? That way, if we decide it's worse, we haven't done a lot of work migrating dependencies or setting up mirrors, and to roll back we only need to migrate the new git patches into the old darcs repo. Thanks Ian

On 10/01/2011, at 13:27, Simon Marlow wrote:

On 10/01/2011 13:02, Max Bolingbroke wrote:

...

However, I remember the last time this came up there were some issues that might make migration painful. From the top of my head:

1) Some people expressed concern that they would have to use two revision control systems to work on GHC, because not all GHC dependencies would be git-based.

It would be a prerequisite to switching that a GHC developer only has to use one VCS. So we either migrate dependencies to git, or mirror them in GHC-specific git branches.

I'm not sure how that is going to work. It might well be possible to build GHC using only git. But most GHC developers also contribute to various libraries which are often quite intimately linked to GHC. In particular, GHC patches are often accompanied by library patches. Unless all those libraries switch to git, too, we'll have to use both git and darcs which would be *really* annoying. Personally, I rather dislike git, mostly for the reasons that Malcolm already mentioned. Compared to darcs, it seems to get in the way much too often. It also seems to make finding buggy patches rather hard. But maybe I just don't know how to use it properly. In any case, a switch to git wouldn't deter me from contributing to GHC, but neither would a switch to any other VCS. I would certainly swear more often while developing, though. Roman

I'm not sure if your statement regarding the decoupling between contributors and VCSes holds water. The VCS is definitely a factor, but certainly not the only one. I've been demotivated by VCSes before and it has directly impacted whether I continued my involvement. Granted that the VCS was SCCS, but still... Sample size of one, not the basis for a system of government and statements regarding watery bints lying in ponds apply. -scooter Sent from my Verizon Wireless BlackBerry -----Original Message----- From: Manuel M T Chakravarty Sender: glasgow-haskell-users-bounces@haskell.org Date: Tue, 11 Jan 2011 15:01:43 To: GHC; GHC List Cc: Simon Marlow Subject: Re: RFC: migrating to git I agree with Roman's position. I would prefer to stay with darcs (it has its advantages and disadvantages, but has definitely been improving much in the past). In any case, all of GHC including all dependencies must be available and patchable with a *single* VCS. Mixing VCS' will lead to madness. Manuel PS: This talk about contributing to a project if it changes its VCS seems a bit lame to me. You contribute to a project in a serious way because you care about the project and because you need whatever improvements you are implementing, not because you like the VCS. Roman Leshchinskiy:

On 10/01/2011, at 13:27, Simon Marlow wrote:

...

On 10/01/2011 13:02, Max Bolingbroke wrote:

...

However, I remember the last time this came up there were some issues that might make migration painful. From the top of my head:

1) Some people expressed concern that they would have to use two revision control systems to work on GHC, because not all GHC dependencies would be git-based.

It would be a prerequisite to switching that a GHC developer only has to use one VCS. So we either migrate dependencies to git, or mirror them in GHC-specific git branches.

I'm not sure how that is going to work. It might well be possible to build GHC using only git. But most GHC developers also contribute to various libraries which are often quite intimately linked to GHC. In particular, GHC patches are often accompanied by library patches. Unless all those libraries switch to git, too, we'll have to use both git and darcs which would be *really* annoying.

Personally, I rather dislike git, mostly for the reasons that Malcolm already mentioned. Compared to darcs, it seems to get in the way much too often. It also seems to make finding buggy patches rather hard. But maybe I just don't know how to use it properly. In any case, a switch to git wouldn't deter me from contributing to GHC, but neither would a switch to any other VCS. I would certainly swear more often while developing, though.

Roman

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

On 11/01/2011, at 16:14, Tony Finch wrote:

On Mon, 10 Jan 2011, Roman Leshchinskiy wrote:

...

It also seems to make finding buggy patches rather hard.

Have a look at `git bisect`.

I'm aware of git bisect. It doesn't do what I want. I usually have a pretty good idea of which patch(es) might have caused a problem and I want to unpull it and its dependencies. This is easy in darcs; I have no idea how to do that in git. Roman

On 11/01/2011 19:07, Roman Leshchinskiy wrote:

On 11/01/2011, at 16:14, Tony Finch wrote:

...

On Mon, 10 Jan 2011, Roman Leshchinskiy wrote:

...

It also seems to make finding buggy patches rather hard.

Have a look at `git bisect`.

I'm aware of git bisect. It doesn't do what I want. I usually have a pretty good idea of which patch(es) might have caused a problem and I want to unpull it and its dependencies. This is easy in darcs; I have no idea how to do that in git.

We can't even do this reliably with darcs. Several times I've tried to unpull one of Simon's patches to work around a bug, and the dependencies end up being more than just the textual dependencies. Then I have to fall back to unpulling by date, which is what git would do. And then sometimes there's the separate problem that you have to retreat the library repos too, and there you have to unpull by date and some guesswork too. Cheers, Simon

Hello, On Mon, Jan 10, 2011 at 12:49 PM, Roman Leshchinskiy wrote:

On 10/01/2011, at 13:27, Simon Marlow wrote:

...

It would be a prerequisite to switching that a GHC developer only has to use one VCS. So we either migrate dependencies to git, or mirror them in GHC-specific git branches.

I'm not sure how that is going to work. It might well be possible to build GHC using only git. But most GHC developers also contribute to various libraries which are often quite intimately linked to GHC. In particular, GHC patches are often accompanied by library patches. Unless all those libraries switch to git, too, we'll have to use both git and darcs which would be *really* annoying.

If GHC and the libraries on which it depends were in git (migrated, or mirrored), then we could use git sub-modules to track the dependencies between changes to GHC and changes to the libraries. Roughly, the workflow would be like this: 1. Make a change to the library and commit it. 2. Make a change to GHC. 3. Make a GHC commit which records the change and the dependency on the commit in the library repository. This is useful because when someone gets the changes to GHC, they would know that they need to update their library as well (and there is tool support to make all updates automatically). This kind of dependency is not at all obvious with our current workflow. The same method works for going back to a previous state of the project, where one can "rewind" the libraries to their old versions too. -Iavor

On 11/01/11 21:57, Roman Leshchinskiy wrote:

On 11/01/2011, at 21:41, Iavor Diatchki wrote:

...

If GHC and the libraries on which it depends were in git (migrated, or mirrored), then we could use git sub-modules to track the dependencies between changes to GHC and changes to the libraries.

Roughly, the workflow would be like this: 1. Make a change to the library and commit it. 2. Make a change to GHC. 3. Make a GHC commit which records the change and the dependency on the commit in the library repository.

What about dependencies which go the other way? Actually, the dependency is often mutual: the GHC change won't work without the library change and the library change won't work without the GHC change. Does git support this?

As I understand it, the GHC repo would specify the required version of the library repo. Right now with darcs we don't get to do this, so if you want to back out the GHC tree to a previous state, it's impossible to back the libraries up to the right point too (I've found this quite annoying when tracking down regressions in the past). With submodules, when you make a combined GHC/library change, the relationship between the two changes would be recorded in the GHC repo, which is exactly what you want.

...

This is useful because when someone gets the changes to GHC, they would know that they need to update their library as well (and there is tool support to make all updates automatically). This kind of dependency is not at all obvious with our current workflow.

IMO, darcs-all works pretty well. I don't think I ever really had problems with missing library patches.

I often see problems where someone has done 'darcs pull' rather than './darcs-all pull' and ended up with a weird compilation error as a result. If we could eliminate this source of errors, it would be a major win. If submodules actually work for what we want to do, this would be a good reason to move to git, I think.

...

The same method works for going back to a previous state of the project, where one can "rewind" the libraries to their old versions too.

This would be useful. Unfortunately, git's rewinding seems rather crippled compared to darcs.

In what way? Cheers, Simon BTW, I just translated the GHC darcs repo into git using darcs-fastconvert (cabal install darcs-fastconvert). It took less than 10 minutes and seems to have done the right thing. I'll try to put this up tomorrow for people to play with.

On Tue, Jan 11 2011, Roman Leshchinskiy wrote:

On 11/01/2011, at 22:20, Simon Marlow wrote:

...

On 11/01/11 21:57, Roman Leshchinskiy wrote:

...

This would be useful. Unfortunately, git's rewinding seems rather crippled compared to darcs.

In what way?

Thomas says that it doesn't do automatic dependency tracking which looks like a huge weakness to me. Personally, I haven't been able to successfully unpull non-consecutive chunks of patches with git so far but I only tried 2 or 3 times before giving up.

I think the confusion might just be in terminology and model. Going back to earlier versions in git is trivial, much easier than darcs. Remember, git doesn't store patches. Git stores full snapshots of the tree, with a digraph of dependencies. The trees and history are immutable. The main advantages to darcs are that it can manipulate the sequence of patches better than git. The main advantage of git is that every version is accurately named. If two people have a commit with a given hash, they will have exactly the same files and history. Most projects seem to want most of the history to be immutable, and only do manipulation on recent stuff. "Removal" of earlier patches is an additional patch that removes it rather than removing it from the history. David

On Wed, 12 Jan 2011, Claus Reinke wrote:

In my understanding, the unorderedness of patch history in darcs is there to make distributed repos easier (fewer constraints: same set of patches, but not same order; can mix local commits and pulls from various repos, no need for a central repo),

Apart from variable patch ordering all of that is true of all DVCSs.

and because darcs has a causal rather than a temporal view of patch history (which patch depends on which other patches, instead of which patch came first).

You can emulate darcs's patch re-ordering in git if you put each independent sequence of patches on a separate branch. Then you can re-merge the branches in whatever order you want. This is a fairly common git workflow.

In other words, always keep a branch/repo that only pulls from the central repos (no other source of patches).

It is normal in git to keep a pristine branch for each remote repository that you pull from - git sets these branches up by default. There can be many remotes in a git repository. Tony. -- f.anthony.n.finch http://dotat.at/ HUMBER THAMES DOVER WIGHT PORTLAND: NORTH BACKING WEST OR NORTHWEST, 5 TO 7, DECREASING 4 OR 5, OCCASIONALLY 6 LATER IN HUMBER AND THAMES. MODERATE OR ROUGH. RAIN THEN FAIR. GOOD.

On Wed, 12 Jan 2011, Claus Reinke wrote:

What happens after the merges? Does one maintain the branches somehow, or does one lose the (in-)dependency information?

Remember that a branch in git is just a name for a point in the revision graph. When you commit to a branch the name is updated to point to the new commit. Names are local to a particular repository. When you do a merge, you do it on a particular branch which is updated to point to the merge commit. The other branches that were merged in (there's usually one but you can create octopus merges if you want) remain as they were. The merge commit contains un-named pointers to its parent commits for use by git, and conventionally records the names of the brances that were merged in the commit message for the convenience of humans. You can commit to the other branches to extend them, or delete and reconstruct them differently, without affecting the state represented by the merge. Have a look the way "topic branches" are used in the maintenance of git itself as an example of how to deal with a collection of independent patches. http://git.kernel.org/?p=git/git.git;a=blob;f=MaintNotes;hb=refs/heads/todo Tony. -- f.anthony.n.finch http://dotat.at/ HUMBER THAMES DOVER WIGHT PORTLAND: NORTH BACKING WEST OR NORTHWEST, 5 TO 7, DECREASING 4 OR 5, OCCASIONALLY 6 LATER IN HUMBER AND THAMES. MODERATE OR ROUGH. RAIN THEN FAIR. GOOD.

On 11/01/2011 23:11, Roman Leshchinskiy wrote:

On 11/01/2011, at 22:20, Simon Marlow wrote:

...

On 11/01/11 21:57, Roman Leshchinskiy wrote:

...

IMO, darcs-all works pretty well. I don't think I ever really had problems with missing library patches.

I often see problems where someone has done 'darcs pull' rather than './darcs-all pull' and ended up with a weird compilation error as a result. If we could eliminate this source of errors, it would be a major win.

A quick look at the docs seems to indicate that we'd need to do

git pull git submodule update

which doesn't look like a win over darcs-all. Also, I completely fail to understand what git submodule update does. It doesn't seem to pull all patches from the master repo. The git submodule docs are even worse than the rest of the git docs which is rather discouraging.

True, however the build system could automatically check whether you had missed this step, because it could check the hashes.

...

...

This would be useful. Unfortunately, git's rewinding seems rather crippled compared to darcs.

In what way?

Thomas says that it doesn't do automatic dependency tracking which looks like a huge weakness to me. Personally, I haven't been able to successfully unpull non-consecutive chunks of patches with git so far but I only tried 2 or 3 times before giving up.

Right, not being able to automatically commute patches is a regression compared to darcs. Git isn't universally "better" than darcs, which is why we're having this discussion - the question is, do the advantages outweigh the disadvantages. For example, you might well consider the lack of a working annotate to be a "huge weakness" in darcs, and the lack of good conflict handling is something that causes us real problems. Cheers, Simon

Hello, On Wed, Jan 12, 2011 at 11:44 AM, Roman Leshchinskiy wrote:

On 12/01/2011, at 09:22, Simon Marlow wrote:

...

On 11/01/2011 23:11, Roman Leshchinskiy wrote:

...

A quick look at the docs seems to indicate that we'd need to do

git pull git submodule update

which doesn't look like a win over darcs-all. Also, I completely fail to

understand what git submodule update does. It doesn't seem to pull all patches from the master repo. The git submodule docs are even worse than the rest of the git docs which is rather discouraging.

True, however the build system could automatically check whether you had missed this step, because it could check the hashes.

That would be an improvement. How do you pull submodule patches which the main repo doesn't depend on, though? Out of curiousity, has anyone here used submodules for something similar to what we would need?

A "submodule" is basically a "pointer" to a particular state of a remote repo. So when you do "git pull" in GHC, you get changes to the code, and also changes to this "pointer", but it won't automatically modify your local version of the sub-module repo. So at this point, if you started "git gui" you'd see that there is a mismatch between your local copy of the sub-module and the expected version. When you issue the command "git submodule update", you are telling git to advance the sub-module repo to the "expected version" (i.e., where the pointer points to). The reason this does not happen automatically is that you might have also made changes to the submodule, so you might want to do some merging there, instead of just pulling. One thing to note is that if we were to set things up with sub-modules, then every now and then we would have to advance the GHC's "expected pointer" for various libraries to the latest (or a newer) version. Of course, we could have a script do this but, at least in theory, when someone makes a commit which updates the version of a sub-module, they are asserting that they things ought to work with the newer version of the sub-module. -Iavor PS: I've only used sub-module on what project at work. At first I too was quite confused about what was going on, but I've come to think that submodules are a pretty reasonable way to deal with a situation which is inherently complex.

Excerpts from Roman Leshchinskiy's message of Wed Jan 12 18:20:25 -0500 2011:

How would we get the current functionality of darcs-all pull? Is it even possible?

Here is the rebase-y workflow. Untested, so I might have gotten one or two details wrong.

Suppose I want to hack on GHC and base (base is a submodule of GHC). For this, I want to:

- pull the latest patches to both GHC and base

# pull the latest patches for GHC, and sticks your patchset on top git pull --rebase # <resolve any conflicts that occured during rebase> # register any new submodules (if any) git submodule init # make your submodules reflect the latest version GHC has git submodule update --rebase # <resolve any conflicts that occured during rebase>

- write code - record my patches in both GHC and base

cd libraries/base git commit -asm "Base patch" cd ../.. git commit -asm "GHC patch" Note that your commit to libraries/base changed what commit it is pointing to, so your GHC commit will then pick up the changed sumbodule. If you do the commits in the opposite order, this won't happen. So commit in all submodules first.

- pull again to get whatever patches have been pushed while I was hacking

git pull --rebase git submodule update --rebase

- validate - push my patches to both GHC and base

git send-mail --to=cvs-ghc@haskell.org $PATCHES cd libraries/base git send-mail --to=cvs-ghc@haskell.org $PATCHES Cheers, Edward

On 13 January 2011 08:54, Roman Leshchinskiy wrote:

On 12 Jan 2011, at 23:31, "Edward Z. Yang" wrote:

...

Excerpts from Roman Leshchinskiy's message of Wed Jan 12 18:20:25 -0500 2011:

...

How would we get the current functionality of darcs-all pull? Is it even possible?

Here is the rebase-y workflow.

Thank you making things clearer!

...

...

# pull the latest patches for GHC, and sticks your patchset on top git pull --rebase # <resolve any conflicts that occured during rebase> # register any new submodules (if any) git submodule init # make your submodules reflect the latest version GHC has git submodule update --rebase

This doesn't pull in all base patches, though, just the ones that GHC depends on, right? How would I get all base patches?

cd libraries/base git pull [--rebase]

Hi, Just as a point of information, the following rules can help avoid some of the gotchas: - Treat submodules are read-only (i.e., don't make commits there). The reason for this is that a submodule is usually not on a branch, and so making a commit would result in a detached head. - When you pull (or change branches) use "git submodule update" to move the submodules to their correct versions (yes, it's annoying that one has to do that). - Changes to a sub-module should be done in a separate repo (not GHC's submodule). This is where you switch "hats" and become a "base" developer rather then a "GHC" developer for a bit, and use whatever workflow you normally use for development. - Every now and then you update the sub-module "pointer" of your GHC branch to a newer versions of the sub-module. You do this by setting the sub-module to the desired version (e.g., by a pull from its repo), and then committing the change to the submodule version (perhaps with other GHC changes). I agree with Simon's assessment that it is probably a good idea to start without submodules, at least until all developers are comfortable with the rest of git's model. -Iavor On Thu, Jan 13, 2011 at 12:49 AM, Simon Marlow wrote:

On 12/01/2011 22:22, Iavor Diatchki wrote:

...

Hello,

On Wed, Jan 12, 2011 at 11:44 AM, Roman Leshchinskiy mailto:rl@cse.unsw.edu.au> wrote:

On 12/01/2011, at 09:22, Simon Marlow wrote:

> On 11/01/2011 23:11, Roman Leshchinskiy wrote: >> >> A quick look at the docs seems to indicate that we'd need to do >> >> git pull >> git submodule update >> >> which doesn't look like a win over darcs-all. Also, I completely fail to understand what git submodule update does. It doesn't seem to pull all patches from the master repo. The git submodule docs are even worse than the rest of the git docs which is rather discouraging. > > True, however the build system could automatically check whether you had missed this step, because it could check the hashes.

That would be an improvement. How do you pull submodule patches which the main repo doesn't depend on, though? Out of curiousity, has anyone here used submodules for something similar to what we would need?

A "submodule" is basically a "pointer" to a particular state of a remote repo. So when you do "git pull" in GHC, you get changes to the code, and also changes to this "pointer", but it won't automatically modify your local version of the sub-module repo. So at this point, if you started "git gui" you'd see that there is a mismatch between your local copy of the sub-module and the expected version.

When you issue the command "git submodule update", you are telling git to advance the sub-module repo to the "expected version" (i.e., where the pointer points to). The reason this does not happen automatically is that you might have also made changes to the submodule, so you might want to do some merging there, instead of just pulling.

One thing to note is that if we were to set things up with sub-modules, then every now and then we would have to advance the GHC's "expected pointer" for various libraries to the latest (or a newer) version. Of course, we could have a script do this but, at least in theory, when someone makes a commit which updates the version of a sub-module, they are asserting that they things ought to work with the newer version of the sub-module.

-Iavor PS: I've only used sub-module on what project at work. At first I too was quite confused about what was going on, but I've come to think that submodules are a pretty reasonable way to deal with a situation which is inherently complex.

I spent quite some time yesterday playing with submodules to see if they would work for GHC. I'm fairly sure there are no fundamental reasons that we couldn't use them, but there are enough gotchas to put me off. I wrote down what I discovered here:

http://hackage.haskell.org/trac/ghc/wiki/DarcsConversion#Submodules

The workflow is quite involved - more steps than are required with darcs-all (understandable, because we're storing more information). However, git isn't particularly helpful if you make a mistake or forget to do something. I forsee spending a lot of time digging myself and Simon out of bizarre repository states.

I discovered that Google have this tool called "repo" which is their darcs-all for the Android source tree. That might be worth looking at as an alternative in the future:

https://sites.google.com/a/android.com/opensource/download/using-repo

If we go with git, I suggest we stick with sync-all for the time being and think about either submodules or repo as possibilities for the future.

Cheers, Simon

On 01/13/2011 12:49 AM, Simon Marlow wrote:

I spent quite some time yesterday playing with submodules to see if they would work for GHC. I'm fairly sure there are no fundamental reasons that we couldn't use them, but there are enough gotchas to put me off. I wrote down what I discovered here:

http://hackage.haskell.org/trac/ghc/wiki/DarcsConversion#Submodules

I think the "what works" section of there is already pretty compelling -- for example, it's an annoyance that "darcs-all diff" produces a diff file which mashes together all the subrepos and can't be applied at the top level. It's another annoyance that "darcs diff" doesn't produce unified diffs by default, what's the point of a diff that can't be |patch-ed? It seems from your discussion that subrepos are intended for your category "the rest of libraries (e.g. filepath, containers, bytestring, editline)" i.e. things that you expect to passively track and occasionally pick up new patches from. What's the argument against using subrepos for those? To me, the major gotcha is "git submodule update" detaching the changes, however changing the default to be a --merge would fix that for me. What about that don't you like? Would you rather want a "git submodule update --just-complain-and-exit"? The last 2 drawbacks you mention (pushing to submodules first and needing to commit to GHC for every subrepo commit) is IMHO the price to pay for a guarantee that you're always able to check out a coherent set of changes. If that's too onerous, maybe some of those libraries just belong in the main GHC repo. I'm another interested bystander who thinks that git would be a step forward, btw.

On Mon, Jan 10, 2011 at 2:02 PM, Max Bolingbroke wrote:

Naturally other workflows are possible and I'm sure other list members will chime in with their own favourites :-)

Here's the flow I use: http://nvie.com/posts/a-successful-git-branching-model/ with the exception of having the master branch be the development branch, which is what most Git users expect.

Ultimately I'm quite concerned with keeping GHC HQ happy (as you guys do the lions share of the work!). I feel we should only make the switch if the most frequent committers (i.e. Simon, Simon and Ian) are *totally happy* with it and any associated workflow changes that may be required.

I agree with this sentiment. If GHC HQ believes Git will make their job harder, I'm not in favor of a switch. (From personal experience I think it will make it easier after an initial short learning curve, but I cannot know this for certain of course.) Johan

Am 10.01.2011 14:02, schrieb Max Bolingbroke:

2) There was also concern that Git isn't so great on Windows. I have heard that this is less of an issue now, but I never personally suffered from any problems, so can't be sure. (FWIW I used Git on Windows industrially ~1 year ago for 3 months and didn't have problems, though the people around me occasionally had issues with e.g. case insensitivity causing obscure error messages).

We are using GIT for a (way smaller) C++ - project here, which is mostly Linux-based. Two of the developers (including me) were using Windows and VS for our part, while building the whole stuff (using the same sources) on Linux. The files were mounted via Samba. The main issue are the line endings, which are to be set correctly. This is a mess in TortoiseGIT, though the problem merely was that we were using the source on both systems. However, we do not exploit the full spectrum of tools and possibilities of GIT. Actually, I am unaware of many of its features... I am not contributing on GHC and am not going to. MFG Heiko

On Mon, Jan 10, 2011 at 5:25 PM, Nils Anders Danielsson wrote:

Even if GitHub is used you should probably arrange some other kind of backup solution, because GitHub reserves the right to delete your repository "for any reason at any time" (http://help.github.com/terms/).

If that would ever happen (probably less likely than someone breaking into our own machine and deleting the repo) we could take a repo from anyone's machine and put it on any old Linux machine. Go distributed version control! Johan

On Mon, Jan 10, 2011 at 12:47:43PM -0500, Norman Ramsey wrote:

My workflow has never involved much cherry-picking, and I tried revising history ('rebasing') once and didn't like it. But I use git's "cheap branching and merging" workflow *very* heavily.

Do you mean you've used this to do something similar to maintaining a GHC stable branch? Thanks Ian

On 25 January 2011 09:35, Lars Viklund wrote:

A subtree seems to be a way of getting the contents of a branch merged at a non-root location. It might be a relevant read and something to evaluate.

There is also the git-subtree project (https://github.com/apenwarr/git-subtree). They explain the difference from the subtree merge strategy as: "The main difference is that, besides merging the other project as a subdirectory, you can also extract the entire history of a subdirectory from your project and make it into a standalone project. Unlike the subtree merge strategy you can alternate back and forth between these two operations. If the standalone library gets updated, you can automatically merge the changes into your project; if you update the library inside your project, you can "split" the changes back out again and merge them back into the library project." Might be useful. A simple example of its use is shown at http://ayende.com/Blog/archive/2011/01/10/git-subtree.aspx Cheers, Max

On 11/01/2011 00:36, roconnor@theorem.ca wrote:

On Mon, 10 Jan 2011, Simon Marlow wrote:

...

It's time to consider again whether we should migrate GHC development from darcs to (probably) git.

From our perspective at GHC HQ, the biggest problem that we would hope to solve by switching is that darcs makes branching and merging very difficult for us.

I don't develop GHC so you shouldn't really listen to me, but if you think Darcs makes branching and merging very diffcult then you won't like Git et al. which make braching and merging impossible since their semantic model is broken.

Thanks for this. I distilled your example into a shell script that uses git, and demonstrates that git gets the merge wrong: http://hpaste.org/42953/git_mismerge Still, git could get this merge right, it just doesn't (I know there are more complex cases that would be very hard for git to get right). I suspect that in practice this rarely matters, because context-based merging usually does the right thing. Cheers, Simon

* Simon Marlow:

Thanks for this. I distilled your example into a shell script that uses git, and demonstrates that git gets the merge wrong:

http://hpaste.org/42953/git_mismerge

Still, git could get this merge right, it just doesn't (I know there are more complex cases that would be very hard for git to get right). I suspect that in practice this rarely matters, because context-based merging usually does the right thing.

Git will have a very hard time getting this right because it is not that history-aware. It's also unlikely that this is implemented because this mismatching of changes happens only rarely, unless you have a coding style which heavily relies on copy-and-paste. (It has happened in real-world merges, though. It is also easy to construct similar examples involving file renames, I believe.) I know only one criterion for merge correctness: developers working serially on the code base would end up with the same result. (This is based on the concept of a serializability in transaction processing systems.) It is clear that no system can satisfy this. For instance, suppose you have a LaTeX document for one-page flyer. Obviously, there is a very hard requirement that you can have only one page of text. Two parallel edits can satisfy this constraint, but their automatic merge might not. (Zooko's example is different in that there is an apparently correct solution, so it is not absolutely necessary to bail out, but of course, the authors could likely squeeze their content on a single page, too.) Inevitably, you have to make trade-offs. The Git approach seems to suit more developers and codebases than the darcs approach. Git mismerges are much rarer than non-completing darcs merges. On the other hand, speaking as a non-contributor, the requirement to deal with multiple version control systems seems awkward. But the current sub-tree approach also feels a bit clunky (same as for OpenJDK, by the way).

On Thu, Jan 13, 2011 at 4:03 PM, Simon Marlow wrote:

I've made git mirrors of the current GHC HEAD repos (all of them), so people can try out their workflows with git.

Poking around in the different repos works for me and is fast. For example: Find new files in base: $ cd libraries/base $ git status Find the definition and uses of threadWaitRead and skip git metadata $ git grep threadWaitRead See when threadWaitRead was added (and introduced in different files): $ git -SthreadWaitRead log (In 2001, by Simon M).

Note that sync-all is not executable, which is why I used "perl sync-all" rather than ./sync-all.  You can chmod it, but the chmod will be seen as a local change by git which will get in the way of future pulls, and you'll need to stash or merge or rebase the change (welcome to git :-).

This particular problem is due to darcs (which we are mirroring) does not supporting executable permissions on files. We can just set the executable bit on the file and commit it. We should set up a git daemon at some point as it's much more efficient that pulling over HTTP. Johan

On Thu, Jan 13 2011, Benedict Eastaugh wrote:

On 13 January 2011 15:30, Johan Tibell wrote:

...

We should set up a git daemon at some point as it's much more efficient that pulling over HTTP.

As of version 1.6.6, Git is much more efficient over HTTP than it used to be.

http://progit.org/2010/03/04/smart-http.html

You do have to install the git smart-http plugin in the server, or it only uses the "dumb" HTTP protocol. David

Hello Simon,

I've made git mirrors of the current GHC HEAD repos (all of them), so people can try out their workflows with git. Hopefully this should work:

git clone http://darcs.haskell.org/ghc-git/ghc.git cd ghc perl sync-all get

Thank you for this work. I cloned the git repository and tried to compile GHC. But "perl boot" does not work. % perl boot Unpacking time Failed to open stamp file: No such file or directory at boot-pkgs line 41. Running boot-pkgs failed: 512 at boot line 23. How can I create the "configure" script with GHC from this repository? --Kazu

On 17/01/2011 14:08, rl@cse.unsw.EDU.AU wrote:

On Mon, January 17, 2011 11:08 pm, Simon Marlow wrote:

...

So, we've decided to try switching to git.

That's very sad!

...

The changeover will be staged: first we'll switch the GHC repository, and if all goes well we'll switch the libraries and other sub-repositories. This means we can experiment with git for the GHC repo while we establish exactly what strategy to use for each sub-repository (change the master to git, or leave the master in darcs and mirror). It does mean for a short while GHC developers will have to use two VCS tools, but the sync-all tool will hide the difference to some extent.

The next step is to set up the git repo, probably based on the mirror I set up last week, and sort out various details (commit emails, buildbots, wiki updates etc.). I'll send out a notification once things are set up and we have a date/time for the git repo going live.

Would it be possible to also produce a definite timeline for when the other repos will switch to git? Having to work with two different VCS in one project is a bit above my pain threshold so I'd like to know when I'll be able to get back into action after the switch.

Absolutely. The reason I didn't give a definite timescale yet is that we're currently finalizing our future release plans and the two are connected, because we want to minimize the amount of merging we have to do between a git branch and a darcs branch. Anyway, I don't expect the multi-VCS situation will persist for more than a month or so, and certainly not across a release. Cheers, Simon

In my one serious attempt to use git for one of my own projects, some seemingly-innocuous operation deleted a file on me and I lost a couple hours of work. I agree with the people who have said that git's documentation and semantics are highly confusing, moreso than darcs's. For example, what does it mean to "stage" a commit? Why is there an entire GUI window for this presumably-important action, and why do things I think I've committed not appear in the change history or mysteriously reverse themselves? If ghc went to git, it wouldn't make me less likely to contribute, but I would do so by checking everything into a local darcs repo and using that to track my own changes, then letting somebody else do the work of getting them into git! Which probably would reduce the likelihood of my patch being accepted, but I consider git a complete waste of my time and have zero interest in learning to use it. Plus, while I admire everyone's willingness to consider a VCS that isn't Haskell-based, I have to admit that there's a Haskell partisan in me. And there are real advantages to being a tight-knit community. If the GHC maintainers go to the Darcs maintainers and say "We absolutely need feature X or we will have to stop using you", the Darcs maintainers are likely to say "It'll be tough but we'll find a way to do it." But we aren't by any means the biggest project using Git, so the Git maintainers would be likely to say "That's nice, keep in touch." Obligatory disclaimer - I've never written any code actually in GHC, although I have used the API (I am the author of direct-plugins). But I frequently read its code to clarify how things work, and I do expect that it's a near-certainty that I'll be hacking GHC itself at some point in the future. -- Dan Knapp "An infallible method of conciliating a tiger is to allow oneself to be devoured." (Konrad Adenauer)