On Mon, Dec 27, 2010 at 11:48:28PM +0100, Mitar wrote:

Why not have discussions in the ticket? In our projects we have been doing that for some time now (instead of the mailing list) and it showed beneficial: you keep all things in the one place, Trac also supports replies and quoting and most importantly afterwards everything is well documented in the same place.

You /can/ have discussions in trac, but IME it's not as pleasant as e-mail. Especially if you have multiple subthreads, or if you want to quote blocks of code. It also requires logging in to trac. Thanks Ian

On 28 December 2010 17:20, Yitzchak Gale wrote:

I am in favor of Ian's proposal, with the following additions:

o A proposal must be accompanied by a concrete patch, or code, as appropriate. The patch or code should preferably not be included in the proposal message; it should be posted somewhere on the Internet, with a link to it in the proposal message.

What happens if someone wants to ascertain the possible support for a proposal before they implement it? Just send an email without the "Proposal: " prefix in the subject line? -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/2/11 05:04 , Malcolm Wallace wrote:

...

A link to a patch implementing the change should be included. Patches may be hosted anywhere

This "somewhere on the Internet" thing almost guarantees that the patch will not be archived for posterity alongside the discussion. It makes future web-searches less useful than they could be.

+1 I spent last night finding out that a bunch of saved links were dead, either because the linked-to item was gone or because the hosting server was down. Granting that the latter could happen anyway, it at least is helpful for the linker and the linkee to have the same availability, especially for official proposals. - -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0gtIAACgkQIn7hlCsL25X7qQCePovUJQaeSTlH0oXB9gNj3vrZ JEkAn2LXhhM88i+qdTvXsSXUKTNTK1pu =NdWR -----END PGP SIGNATURE-----

On 03/01/2011 11:20, Ian Lynagh wrote:

On Sun, Jan 02, 2011 at 10:04:16AM +0000, Malcolm Wallace wrote:

...

...

A link to a patch implementing the change should be included. Patches may be hosted anywhere

This "somewhere on the Internet" thing almost guarantees that the patch will not be archived for posterity alongside the discussion. It makes future web-searches less useful than they could be.

How about attaching them to http://hackage.haskell.org/trac/ghc/wiki/Library_Proposals then?

(It doesn't look like you can attach files to a particular page on the haskell.org wiki, whereas I believe you can on a trac wiki).

And if we follow this discussion to its conclusion, we end up back where we started: modifying a wiki page is error prone and we need more automation, so we use the ticket system instead. I'm all for making the process more lightweight. Why not just attach patches to the proposal email? Updated proposals can be made with new patches attached. This mailing-list-only process would help address the concerns that the process is too much of a burden for the developers/maintainers themselves. Someone with commit access need not create the ticket at all: they just make a proposal on the list, and as long as it isn't controversial they commit after the discussion period. It's a small step from here to committing *before* the discussion, which is what some people want. I don't feel strongly about that, as long as we have an established protocol for discussing changes to these core APIs. Cheers, Simon

On Wed, Jan 05, 2011 at 12:16:33PM +0100, Gregory Collins wrote:

* it is better to ask forgiveness than to get permission

* it is better to avoid tedious discussion even if it means taking 2 steps forward and 1 step back sometimes. Rolling back a controversial patch is an O(1) operation.

Only if it is noticed before the next GHC major release. One of the main reasons the library process was created was to get people to look at changes before it is too late.

* pull scales better than push

What do you mean by that?

* inertia should favour those who want to make forward progress rather than those who would like to slow it down: i.e. it should be easier to get a patch through the process than it should be to slow it down

I think I am one of the more pro-change members of the community, but even I don't think that that is as obviously true as you make it sound. The phrase "forward progress" makes it sound like changes are necessarily a good thing, but some changes are rightly rejected (or improved as a result of the discussion).

* code should speak louder than email on mailing lists.

I don't really know what you mean by that, either. Just because someone submits a patch to do something, doesn't mean that that thing should be done.

* Code and API quality are ensured through a branching/release model rather than through extensive weeks-long patch review.

But shouldn't the same amount of review have to be done either way? Doing it in advance of applying the change just means that we are sure that changes to our core libraries are actually reviewed, rather than no-one getting around to it.

* Code review outside of the core team is accomplished by the fact that someone with commit access needs to apply your patch. For no-brainers discussion would not be necessary, in other cases mailing list discussion might be appropriate. The criterion for accepting changes should not be "is this perfect?" but "is this a net improvement from what we have now?"

* Code review inside the core team is accomplished post-hoc. Any reasonably modern code hosting service can be configured to spam a mailing list with patch notifications or will have an RSS feed you can subscribe to. Remember: rolling back a patch is O(1), and if a patch takes O(n) effort to make, fixing it is usually O(n/k), 2 <= k <= 10.

This is what we had before the library process, and the library process was introduced to fix its problems.

Ian had some criticism for me in our IRC discussion about this issue yesterday, saying: "Have you failed to get a change in you think should have got in?" The fact of the matter is, I would definitely like to contribute but I'm not going near our current process with a 20-foot barge pole.

As someone who uses the process quite merrily, I don't get this. What sort of a changes would you like to contribute, and what problems do you forsee? Is the problem that it will be 2 weeks (or more) before your patch is pushed to a repo? If so, why is that an issue when API-changing releases are only made once a year? Thanks Ian

Some short thoughts on this thread * Duncan makes a critical distinction between API changes Implementation changes Both changes presumably offer some benefit. Implementation changes may impose some costs, but only on other implementers (outright bugs aside). These costs are of the form "how the devil does this new code work". API changes impose costs on every user of that library, to adapt his or her code to work with the new API. That's a much larger population. Gregory advocates a "by default apply patch" policy. That's fine for implementation changes, but I think it is arguably right for API changes to have to suffer a significantly higher bar. (In fairness to Gregory, I believe that his proposals were largely directed at implementation changes. When people offer patches for GHC itself, we certainly use the "by default apply" rule, because they are internal -- they fall under the "implementation change" heading. Even then, if the patch is big it sometimes take me a while to review because I am aware that I'll be maintaining this code in 5 years time, when the author is perhaps long gone.) I think it'd be good if the library-change prcess acknowledged the above distinction explicitly, and treated the two differently, as Duncan suggests. * I fully subscribe to Johan's view that each core library needs a named, individual maintainer (or a small group of such). The libraries@ mechanism does work, but I think it'd work better if each lib had a named maintainer. * Gregory writes "The fact of the matter is, I would definitely like to contribute but I'm not going near our current process with a 20-foot barge pole. And I'm unequivocally not the only potential contributor who feels this way. From an interested observer's perspective, it seems like libraries@ is where good code goes to die, and I don't have the time, energy, or patience to endure it." Others may agree or disagree with this view, but it is a *fact* that G feels this way, and that he believes that others do too. That's alarming to me, and I think we need to take it seriously, since G has done us the courtesy of explaining his position (thank you Gregory). Quite what we might do to address these concerns isn't so clear to me. I'm agnostic about technology, my gut feel is that the core issues are not technological ones (eg github vs darcs). Simon

It's interesting to look at why we introduced the process in the first place: http://www.haskell.org/pipermail/libraries/2006-October/006005.html "The idea is to reduce the number of 'bikeshed' discussions we're having, ensure that submissions are not dropped, and generally improve productivity." that is, pretty much all the things that people are complaining that the process is not doing. We had good intentions, but I guess it didn't work. It didn't even ensure that submissions are not dropped, because we did end up with plenty of abandoned proposals. Some of us have been discussing this at length on IRC over the past couple of days. With it being a new year this is great time to take a critical look at our processes, and I agree with Simon that we should take this quite seriously. I'd like to thank Greg and Johan for sticking around to discuss this rather than giving up. The underlying problem they've raised is that this community has become a little too afraid of change. Nobody feels empowered to do anything without first consulting "the community", and that typically ends up nowhere except for very uncontentious issues. However, historically all the progress has been made by individuals or small groups Just Doing It. Hackage has enabled the community to share the fruits of each other's productivity on non-core libraries, but very little movement is happening at the core. The slow rate of change is to some extent a reaction to a period of instability we in the past (around GHC 6.8) when there was a lot of upheaval and complaints from users that their code kept breaking with new GHC releases. We took some positive steps to help with this (the base-3 compatibility package, wider use of deprecation, and generally shying away from breaking changes). There has always been a tension between improving APIs and stability, and currently we're skewed towards stability, at least for these core APIs. Compulsory review of API changes is a good thing - we get lots of eyes on the changes. However, the process would be more appropriate if we were basically happy with the APIs we have. On the contrary, we have plenty of large-scale changes that need making in base and other places, and forcing all changes through the library process is making it hard to get these cleanups done. Another thing making it hard is the lack of people actually doing it, but it's difficult to tell whether that's due to the obstacles we've put in the way, or whether there really aren't people with the time and inclination. We'll only find out by actually removing the obstacles. The process is not applied consistently either - there are libraries like bytestring which are not under the process but are arguably just as "core" as containers. I'm coming around to the view that this small-scale API change review isn't getting us where we want to go. To really improve APIs we should have a group of people looking at the whole, and making strategic decisions. Let's figure out where we're going and how to get there, rather than making a series of tiny incremental steps (slowly). Let's really make FilePath an ADT, make binary Handles a different type, move more of base out into separate packages, remove the Show superclass of Num, overhaul the containers API, finish creating the Exception hierarchy, etc. etc. We've had a period of stability, maybe now it's time for change, and lots of it. The changes being proposed by Greg and Johan, as I understand it, would amount to the following. I'm willing to give it a try; we can always go back if it doesn't work out. - maintainers are empowered to make API changes. - no formal review process for API changes, although there is an expectation that non-trivial changes would still be discussed on the list beforehand. - we would still have a clearly documented path for contributions and proposals - commit messages are copied to the list, and should detail API changes sufficiently that the community can keep an eye on what is happening and comment. - we get more maintainers (presumably they'll just appear, or something? :-) Does anyone violently disagree? Suggestions for improvements? Cheers, Simon

On 6 Jan 2011, at 11:32, Simon Marlow wrote:

the process would be more appropriate if we were basically happy with the APIs we have. On the contrary, we have plenty of large- scale changes that need making in base and other places, and forcing all changes through the library process is making it hard to get these cleanups done.

I'm coming around to the view that this small-scale API change review isn't getting us where we want to go. To really improve APIs we should have a group of people looking at the whole, and making strategic decisions. Let's figure out where we're going and how to get there, rather than making a series of tiny incremental steps (slowly).

I think I agree with this. If there is general unhappiness with the basic set of APIs, then doing a big-bang redesign of everything from scratch (yes, even as far as the Prelude) is probably the right way to go. It will be a big discussion, with plenty of arguments, but at least "the libraries committee" might only need to do it once every ten years, rather than the "death by a thousand cuts" model we have now.

Let's really make FilePath an ADT, make binary Handles a different type, move more of base out into separate packages, remove the Show superclass of Num, overhaul the containers API, finish creating the Exception hierarchy, etc. etc. We've had a period of stability, maybe now it's time for change, and lots of it.

These are all good changes: we have known for a long time they are desirable, but making them would be disruptive. This discussion has persuaded me that we are never going to get them, unless we do them wholesale in a single strike.

- maintainers are empowered to make API changes. - no formal review process for API changes, although there is an expectation that non-trivial changes would still be discussed on the list beforehand.

Actually, I think those process ideas are in danger of being understood to belong to the "incremental" model, not the "wholesale" one. My suggestion would be: * appoint (self-select?) a group of people who will commit to actively seeing this wholesale redesign through to the end. * freeze the existing set of core packages, e.g. base, containers, binary, etc. * design from scratch a new-base package. * redesign the other core packages around the new-base. * make a big-bang release of all of the new core packages at once. * encourage other packages on hackage to abandon dependencies on the old base, and move to new-base plus fresh packages that depend on it. The design discussions should be public of course, but basically, the job is to critique old code and write new code in collaboration amongst a small group of talented library designers. As a target, do you think such a group could aim to complete a draft redesign in time for Haskell 2012? Regards, Malcolm

On Jan 7, 2011, at 5:33 AM, Malcolm Wallace wrote:

I think I agree with this. If there is general unhappiness with the basic set of APIs, then doing a big-bang redesign of everything from scratch (yes, even as far as the Prelude) is probably the right way to go. It will be a big discussion, with plenty of arguments, but at least "the libraries committee" might only need to do it once every ten years, rather than the "death by a thousand cuts" model we have now.

...

* appoint (self-select?) a group of people who will commit to actively seeing this wholesale redesign through to the end. * freeze the existing set of core packages, e.g. base, containers, binary, etc. * design from scratch a new-base package. * redesign the other core packages around the new-base. * make a big-bang release of all of the new core packages at once. * encourage other packages on hackage to abandon dependencies on the old base, and move to new-base plus fresh packages that depend on it.

The design discussions should be public of course, but basically, the job is to critique old code and write new code in collaboration amongst a small group of talented library designers.

This is a very good, provocative proposal. For a long time, talk of major changes to core typeclasses, etc. was considered "Haskell Prime" territory, even if it was recognized that most initial Haskell Prime work would focus on the language and not libraries. Then the Haskell Prime committee moved models and announced that libraries would follow their own process entirely. But there wasn't any corresponding move to alter/change/disrupt the libraries process in light of this. So I agree that a library "big bang" is a very good idea, and there's a fairly standard litany of issues that could be dealt with all at once. That said, base+core is still a large amount of code. I think that there should be some sense of scope so that things don't stretch on. First, the focus should be on API-breaking changes, and not implementation details. The latter can and should be dealt with as well, but this can be dealt with much more incrementally, and to a certain degree after there's consensus on the big issues. So a timetable should recognize those different phases. Second, I think there will have to be some organized specialization with regards to different components of base and different core packages. For the most part, we can't expect the everybody to provide the same attention and care to every module -- I imagine implementors of various modules will get quite a bit of leeway, subject to code review. There should be a period of comprehensive discussion and review following the major work -- not a community approval process, but more of the beta process that ghc and the haskell-platform go through in different ways. In fact, there may need to be more than one iteration of this, depending on how smoothly things go. Finally, once the whole process has wound up, effort should be made to make the transition as easy as possible. This means comprehensive documentation, and perhaps tools for automated code transformation, which haskell-src-exts now makes it feasible to write. Cheers, Sterl

Simon Peyton-Jones schrieb:

* Gregory writes "The fact of the matter is, I would definitely like to contribute but I'm not going near our current process with a 20-foot barge pole. And I'm unequivocally not the only potential contributor who feels this way. From an interested observer's perspective, it seems like libraries@ is where good code goes to die, and I don't have the time, energy, or patience to endure it."

Others may agree or disagree with this view, but it is a *fact* that G feels this way, and that he believes that others do too. That's alarming to me, and I think we need to take it seriously, since G has done us the courtesy of explaining his position (thank you Gregory). Quite what we might do to address these concerns isn't so clear to me. I'm agnostic about technology, my gut feel is that the core issues are not technological ones (eg github vs darcs).

My main reason for not proposing something for the base libraries the official way is that I was never able to at least run a nightly build of GHC and was even more scared by building GHC + base libraries myself. I would highly appreciate a way to build base libraries without building GHC.

On 5 January 2011 11:16, Gregory Collins wrote: [SNIP]

From an interested observer's perspective, it seems like libraries@ is where good code goes to die, and I don't have the time, energy, or patience to endure it.

A standard library even if it is called "Hierarchical Libraries" or "Base" is a very different beast to a growing application or library starting ab ovo, though. Many of the changes proposed on Libraries are controversial, I haven't counted but I expect uncontroversial (and accepted) changes are the minority rather than the majority of proposals (successful recent changes I can see from my inbox seem to be adding applicative instances for ST where they were obviously missing or fleshing out some of the Control.Concurrent modules).

On Wed, Jan 05, 2011 at 01:50:26PM +0100, Johan Tibell wrote:

On Wed, Jan 5, 2011 at 1:43 PM, Stephen Tetley wrote:

...

Many of the changes proposed on Libraries are controversial, I haven't counted but I expect uncontroversial (and accepted) changes are the minority rather than the majority of proposals (successful recent changes I can see from my inbox seem to be adding applicative instances for ST where they were obviously missing or fleshing out some of the Control.Concurrent modules).

Perhaps that's because no one would go through the trouble to propose smaller patches as the work reward ratio makes it not worth it?

The implication being that the reason no-one is submitting patches for these widely used libraries is the process? Taking the first few packages from: http://donsbot.wordpress.com/2010/06/30/popular-haskell-packages-q2-2010-rep... HTTP: As I understand it, Ganesh recently grudgingly became maintainer, as no-one else stepped up. 2 or 3 bug fixes in the last year. Diff: http://urchin.earth.li/~ian/HTTP.txt parsec: 3 small tweaks, and the Cabal file altered to cope with base 4, in the last 15 months. Diff: http://urchin.earth.li/~ian/parsec.txt zlib: Not even an upload for the last 18 months. binary: Not even an upload for the last 15 months. None of those use the library process. Now here's the list of patches for the unix package in the last year, which does use the process: Sat Jan 1 20:11:50 GMT 2011 Ian Lynagh * Proper autoconf test for sem_close's library; fixes trac #4523 Fri Dec 10 09:46:45 GMT 2010 Simon Marlow * run forkprocess01 with +RTS -ls too, to test #4512 Ignore-this: 446a0f0b6057c58fc4a2f9d69fae33a1 Wed Dec 8 18:38:49 GMT 2010 Dmitry Astapov * Clarify behavior of "awaitSignal Nothing", export SignalSet that includes all signals reserved by RTS (#4504) Ignore-this: 646a2af7fbed41ee3c72a624c6032db9 Tue Nov 16 17:25:36 GMT 2010 simonpj@microsoft.com * Remove unnecessary fromIntegral calls Ignore-this: e77072c759445a64892738618ff03ea8 Fri Sep 17 16:01:14 BST 2010 Ian Lynagh * Bump version to 2.4.1.0 Thu Sep 9 13:54:59 BST 2010 Ross Paterson * avoid Foreign.unsafePerformIO Ignore-this: 890c8460db98d1ec18168d70291be9f1 Mon May 17 19:19:45 BST 2010 Eric Mertens * Add bindings for setting effective user-id and group-id Ignore-this: 49f72b40befb133cdbf63742a389dad5 Having the ability to set the effective user-id and group-id enables the application to switch between the real user-id and set-user-id multiple times. Wed Jun 2 09:25:09 BST 2010 Simon Marlow * Allow base-4.3 Ignore-this: 5d6dea210a68d3f290f69ee1feb1488a Tue May 18 09:44:22 BST 2010 Simon Marlow * add a comment about 3816 failing due to Samba Ignore-this: daf0c2087064925e67b1a5a09ce7632c Sun Apr 18 18:26:19 BST 2010 Matthias Kilian * Provide a dummy input file for queryfdoption01 Ignore-this: 4d7bfb388193ff6090f4a5fcfe798bbe Fri Apr 30 09:28:41 BST 2010 Simon Marlow * mention that forkProcess is not supported with +RTS -Nn where n > 1 Ignore-this: c7bc7e8262bc72244061a9aba2dd468d Mon Mar 29 12:27:13 BST 2010 Simon Marlow * make getAllGroupEntries work when called again; fixes #3816 Ignore-this: 8b5e3c1d8648d556b9f032a198a8475d Mon Mar 29 12:26:46 BST 2010 Simon Marlow * add test for #3816 Ignore-this: 4d07e5956a5bc6dfb7be59473ccf6b98 Mon Mar 22 13:16:15 GMT 2010 Simon Marlow * fix warnings Ignore-this: 9fd97646b47d5d8d2957c0927885041e Fri Mar 19 21:08:02 GMT 2010 Simon Marlow * handleToFd: close both sides of a DuplexHandle (#3914) Ignore-this: d1c17df79644a7ad140099d0c1474c81 Wed Jan 27 11:46:00 GMT 2010 Simon Marlow * check for EINTR in openFd Ignore-this: 28859b6a74c6d47d5c92db0eb688fa4c Wed Jan 27 11:43:29 GMT 2010 Simon Marlow * accept --with-cc to set the path to gcc (#2966) Ignore-this: 127eccfccbfc849ce05aed5904d16b75 Tue Jan 19 00:34:37 GMT 2010 gwern0@gmail.com * System.Posix.Temp: rm whitespace Ignore-this: 33b2bcf99f7b6b21461ea0aee7c12ea8 Tue Jan 19 00:30:22 GMT 2010 gwern0@gmail.com * System.Posix.Temp: pad input filenames with 6 Xs Ignore-this: 1e5a45074881f75d58e3a0f02525b264 If the argument doesn't terminate in capital Xs, the C mkstemp will simply bomb out with zero warning. This was not documented. By arbitrarily sticking a bunch of Xes at the end of all arguments, we guarantee that this exception will not be thrown, the type signature will not change, and no existing code can break (since if it was manually avoiding the exception by adding "XXX" itself, the temp files will now be simply 3 random characters longer, nothing worse). Tue Jan 19 00:27:55 GMT 2010 gwern0@gmail.com * System.Posix.Temp: improve haddocks for mkstemp to explain what that returned String actually is Ignore-this: 5f5ec7871a687a024cda18b1b0d1e044 Wed Jan 13 11:38:03 GMT 2010 Simon Marlow * fix base dependency: should be >= 4.2 (#3780), and bump verison to 2.4.0.1 Ignore-this: 121b61a9ea1ce1dcbe499285d1910d25 base's list would be more than 10 times as long. Thanks Ian

On 5 January 2011 11:16, Gregory Collins wrote:

The fact of the matter is that pushing patches to a mailing list is OK but it's still a barrier to collaboration. Those of us who have spent some time working with newer services like github or bitbucket will attest that those models are clearly superior for a variety of reasons. When someone wants to make a patch for our project, they fork our repository on github, code their patch, push it to their forked repository, and send us a pull request. Every member of our "core team" gets a message in our inboxes and any one of us can apply the patches with a couple of keystrokes.

As Stephen mentions, all these things are great for libs where there is clear ownership by a reasonably small number of people. It is how most packages in the Haskell world work. For core libraries like base etc most people feel there needs to be a more heavyweight process, if just to slow down the rate of change! That said, it's mainly the APIs that people worry about. I think it would be fine to have a much more lightweight process for maintenance or performance stuff that does not change APIs. At the moment I think we too often refer changes to the libs review process when they don't change an API and it would be fine for some suitable individual to review the patch and apply directly. So, yes I agree we should have maintainers (individuals or teams) for the core libs, but that API additions/changes should still go through a community API review process. The maintainers can help contributors with that process, indeed if the maintainer thinks the change is sensible they could lead that process themselves. Duncan

On 5 January 2011 13:01, Johan Tibell wrote:

On Wed, Jan 5, 2011 at 1:53 PM, Duncan Coutts wrote:

...

At the moment I think we too often refer changes to the libs review process when they don't change an API and it would be fine for some suitable individual to review the patch and apply directly. So, yes I agree we should have maintainers (individuals or teams) for the core libs, but that API additions/changes should still go through a community API review process. The maintainers can help contributors with that process, indeed if the maintainer thinks the change is sensible they could lead that process themselves.

I don't think anyone would be willing to be a maintainer under those conditions. Any volunteers?

Really? There are several people who quite often make direct changes to base and containers including Simon, Ian, Ross and Malcolm. There are several other libs that have maintainers but where any substantive API change would also effectively require some community review and consensus, for example I don't expect Don and I could get away with making big changes to the bytestring API. Duncan