On Thu, 2009-08-20 at 08:51 +0200, Sean Leather wrote:

The Haskell Platform steering committee have been very busy over the past two weeks drafting, discussing and redrafting.

Looks nice!

Thanks.

Our aim is to have the libraries list come to an agreement on the procedure first and then we can discuss codifying a more comprehensive set of package requirements and guidelines.

Sorry to diverge from this, but I just wanted to note that something to the effect of "following the license policy" should be mentioned in the package requirements. Yes, the "interim license policy" is not far down, but adding this would make it clearer, I think.

Technically the "interim license policy" is a sub-section of the "package requirements" section, not that this is terribly clear from the size of fonts used. So point taken. Lets clarify this when we get a proper license policy, or if we need to expand and reorganise the package requirements section in general. Duncan

On Thu, 2009-08-20 at 08:56 +0100, Magnus Therning wrote:

I found this

"Compile on all operating systems and compilers that the platform targets."

to be a little confusing. HP 2009.2.0.2 for Windows does include Win32 (providing System.Win32). Of course there's not much point in including Win32 in HP for a Unix platform. I assume the same is true for unix (providing System.Posix), but in the opposite direction, so to speak. It would be a shame to not include these two _platform-specific_ packages in HP.

You're right that we somehow need to allow these exceptions but without encouraging non-portable packages in general.

OTOH, maybe I'm not using the word "compile" in the correct sense... Would a platform-specific package qualify if building it on _other_ platforms resulted in a successful build, but with empty output?

I think "build and work" would be a better expression of the intention. Duncan

On Fri, 2009-08-21 at 02:23 +0100, Ian Lynagh wrote:

On Thu, Aug 20, 2009 at 03:04:20AM +0100, Duncan Coutts wrote:

...

There is also an example proposal:

http://trac.haskell.org/haskell-platform/wiki/Proposals/example

So please send in your comments and of course feel free to ask questions

I read the example first, so I'll give my comments on it first:

The policy is significantly less prescriptive than the example might lead you to believe. The essential requirement is given as: The proposal forms the nub of the argument for why the package should be included in the Haskell Platform. It should contain or link to enough information so that reviewers can make an informed decision on whether the package should be accepted. With further specific details given in the "Proposal content" section: http://trac.haskell.org/haskell-platform/wiki/AddingPackages#Proposalcontent I should also note that I'm happy to improve my tar proposal to pick up some of the points you make. In addition to being an example proposal I intend to submit it as a real proposal, so improvements benefit both the real proposal and the example.

My first impression is that there is an awful lot of text, and that it's mostly in paragraph form (i.e. I need to actually read it, rather than the key points being highlighted). I think this will reduce the level of review that proposals get.

Below you suggest that instead the proposal should link to existing documentation (and that such documentation be required to exist). I presume the documentation linked to would have the same amount of text. Assuming that's what you mean, I'm not sure what you're comparing to that has less text that we might expect would give us an increased level of review.

This is a proposal for the 'tar' package to be included in the next major release of the Haskell platform.

Should specify version number.

I expect that usually the exact number is not especially relevant. The exact released version now is not necessarily the same as the final one that gets included in the case that reviewers make suggestions. If there's more than one active branch then yes it's relevant so reviewers know which one to look at. Otherwise reviewers can reasonably assume the version on the hackage page. The guidance notes also mention that if reviewers are expected to have to look at a development version, that the repo should be given in the proposal.

Review comments should be sent to the libraries mailing list by ${deadline - 4 weeks}

This sounds like it is the wrong way round to me. The deadline for review comments should be "${now + 2 weeks}" (for some value of 2, and larger values for complicated/large proposals). If you make the proposal early, then it can be agreed earlier. If you make it too late for the discussion and consensus to happen by the deadline then it can wait until the next release.

Suggesting that comments be made by a time relative to the proposal time rather than the final deadline would also be consistent with the policy. The constraints are that people be given a reasonable amount of time to review things and that discussions and issues are resolved before the final deadline. I'd be happy to change the guidance to suggest now + time rather than final deadline - time.

(that's pretty much how library submission proposals work).

Sure. Though I think we need to clarify how much time is reasonable for review rather than letting the proposers pick whatever they like.

Abstract

I think this should be replaced with a link to the hackage page, which shows the abstract from the Cabal file. You might argue that having all the info in one place is convenient for reviewers, but * I am unconvinced that it is much easier * Packages that are proposed are probably widely used anyway, so this is just noise to a significant proportion of people * by only linking to it you remove the temptation to alter the text in the proposal but not apply the improvments to the Cabal file.

The policy allows the info to be inline in the proposal or linked to. It's up to the proposal author: Where appropriate, existing sources of information about the package may be copied or linked to.

Rationale

I don't think the actual rationale part of this section is useful. It mostly says "The foo library is useful because some people want to foo". If during the discussion we find that the rationale is unclear then a rationale could be added to the proposal, but mostly I expect it will be self-evident (and if not then the abstract in the Cabal file probably needs some work).

It's intended to cover these proposal requirements: The proposal forms the nub of the argument for why the package should be included in the Haskell Platform. It should explain the gap in the existing set of platform packages that the new package addresses. Alternatively if the package would duplicate or replace any existing platform packages then the reasons why the new package is better should be clearly explained. In this case the proposal should be tied to another proposal to deprecate the existing package(s) (or parts thereof) and thought should be given to how the transition should be managed. [note-7.2] The guidance notes say: The rationale is the core of your argument as to why this package should be added to the platform. Your task here is to convince people. What makes the package useful? What are the use cases? Being able to point to existing uses is always good. Similarly, existing comments or reviews from users can be helpful. Old release announcements can be a good source of inspiration here. If your package is a replacement for an existing package (or part of one) then you'll want to convince people why the new one is an improvement. I agree that what I've written for the rationale for the tar proposal is of somewhat dubious utility. In fact personally I don't like the term "rationale" for the section.

The list of users is useful (less convinced by the list of packages you think /ought/ to use it), but a bullet-pointed list would be much easier to read.

Well I'm convinced darcs ought to use it ;-). I'm citing it there as an example of an alternative to using the tar package, to use an external tar program (and the various disadvantages that has). In general I don't think it's an unreasonable argument to point out cases where existing packages could be improved by making use of the new package rather than using some existing home-grown solution. Of course this is all a judgement for the proposal author for what they think makes a convincing argument (and for the reviewers to decide if they're convinced). I'll keep it in mind that you don't find it convincing :-).

Introduction to the API

What I said about the abstract applies here, only moreso. If an intro to the API is useful, then one should already exist.

It's explicitly ok to link to one where one already exists. The policy and guidance notes say as much.

People should not be given the opportunity to /write/ an introduction to the API here; it should already be in the haddock docs on hackage, or on the project page on community, or in some similar location.

If we add that as a package requirement then it would be reasonable to change the advice to say simply to link to it. Currently the only doc requirement we've put in is: Library packages should have Haddock API documentation for all exported functions and types. I expect once we've agreed the procedure that we'll get onto more discussing more detailed requirements on things like documentation. In the tar example I had already made an introductory section to the API in the haddock docs. However on trying to present it again in the proposal (by doing a fair bit of copy and pasting) I think I made some improvements to how and the order in which concepts are presented. Writing with a particular audience in mind can help a lot.

Therefore this too should just be a link. Again, this will remove the temptation to update the proposal but not the real docs following the discussion. It will also mean people don't waste time reformatting docs in wiki markup.

The guidance notes say: Of course, if you end up making something new for the proposal then it'd be great to keep it somewhere so that new users can benefit from it too. The obvious place is in the Haddock docs, or a project website.

Design decisions

Ditto.

The same points apply. If we want to require that packages must have this kind of documentation then it ought to be spelled out explicitly as a package requirement. Currently it's just part of the proposal because it's a less onerous requirement.

Open issues

This is the interesting bit of the proposal, IMO. What I want to see for a proposal is just something like:

That's a quite different kind of thing from what the open issues section is intended for. The policy says: Open issues and objections raised by reviewers should be tracked on the proposal wiki under a separate section. [note-3.2] and An explicit checklist of the package requirements below is not required. The proposal should state however that all the requirements are met, or for any requirements that are not met, a reason why they are not met. [note-7.6] What you're asking for is a checklist. That idea was particularly controversial within the steering committee and it was not adopted. Personally I was weakly in favour if a checklist but the concerns were that it restricted the arguments and explanations that the proposal author could make. The consensus was for a much more free-form proposal rather than forcing things into a checklist.

and the goal of the discussion would then be to get a consensus on whether the yellow sections are severe enough to not accept the package (and also to give people a chance to disagree that some of the green entries really are green, e.g. "Is useful?" is subjective).

I think that would come under the complaint that that is too narrow for the argument for why the package should be included. Perhaps other members of the steering committee would like to comment on this point. On the other hand if it's not part of the proposal then I think you'd get a better reception. I know that the release team will want to use its own status board for the aspects of packages that particularly affect the release team (like building on various arches).

I've mentioned this link on IRC before, but for those who haven't seen it, this is partly inspired by: http://release.debian.org/etch/arch_qualify.html (which shows at a glance which arches are acceptable to be in a Debian release, and where the issues and potential issues are. The link in the first row gives more detail on some of the points).

My concern if this is the main focus of discussions is that it covers the "does it work" aspect reasonably well, it doesn't cover the "is it any good? how could it be better?" aspects.

...

http://trac.haskell.org/haskell-platform/wiki/AddingPackages

I've only skimmed this, due to the length, but my impression is that it is too heavy-weight. e.g. http://www.haskell.org/haskellwiki/Library_submissions has this to say about consensus:

If someone has done all this, they are entitled to expect feedback; silence during the discussion period can be interpreted as consent.

If consensus was achieved, [...]

and doesn't seem to have had problems with getting consensus, or agreeing whether we have got it.

That's the main reason we agreed to go with a consensus method. The policy states: The standard of consensus required is the same as that defined for the existing library submissions process. [note-5.4] Our early draft simply said that and nothing else. Several members of the steering committee and others who read drafts were unsure or unconvinced that consensus would be a usable method. We agreed to try it and see if it works. In addition we added the option to use this consensus protocol.

The "Achieving consensus" section of AddingPackages is 947 words according to wc (that's about double the length of the entire Library_submissions page).

You will notice that in the simple case there is essentially no formality at all: The first stage is presenting the proposal followed by open group discussion. This stage lasts long enough to give everyone the opportunity to send in their review comments. There follows a call for consensus. This is essentially the same as the libraries process (including the bit about expecting feedback and silence is consent). We only need to go further when there are remaining disagreements. So I do not agree that it is too heavy-weight. It starts out very light and gets more structured as the severity of the disagreement increases and as the final deadline approaches.

Also, the fact a credits section was deemed necessary suggests to me that too much work is involved.

It was agreed because it's a cheap and easy way to promote the work of review which is often undervalued. We also think that serious review is a great way to improve package quality and entry to the platform is an appropriate moment to do it. There is clearly a trade off between: * work for package proposers / maintainers * work for reviewers * quality of package review and quality of decisions My impression so far is that opinions are reasonably evenly split on all three. To me, that indicates we're roughly in the right place. What are you suggesting specifically that would reduce the amount of work for reviewers? Would those things increase work for the proposal authors / package maintainers? Would those things increase or decrease the level of depth/quality of review and decisions? Of course if there are any suggestions for straightforward wins in this trade-off then we should consider them seriously. Duncan

On Fri, Aug 21, 2009 at 08:58:56PM +0100, Duncan Coutts wrote:

On Fri, 2009-08-21 at 02:23 +0100, Ian Lynagh wrote:

...

On Thu, Aug 20, 2009 at 03:04:20AM +0100, Duncan Coutts wrote:

...

There is also an example proposal:

http://trac.haskell.org/haskell-platform/wiki/Proposals/example

So please send in your comments and of course feel free to ask questions

I read the example first, so I'll give my comments on it first:

The policy is significantly less prescriptive than the example might lead you to believe.

Looking again at AddingPackages, I think you are right. That said, if I were making a proposal, I would probably base it on an example (or a template, which is essentially the same thing). The AddingPackages page confuses me. If I need to read all the notes, then the page seems very hard to read, and if I don't then could the rationale be put on a separate page? Also, the [note-1.3]s could all be [rationale] or [rationale-1.3], right? That way I wouldn't think that I have to read them unless I am actually interested in the rationale. Currently the page looks daunting to me.

I should also note that I'm happy to improve my tar proposal to pick up some of the points you make. In addition to being an example proposal I intend to submit it as a real proposal, so improvements benefit both the real proposal and the example.

I've made an alternative proposal for the tar package here: http://trac.haskell.org/haskell-platform/wiki/Proposals/alternate_example I've made up some of the answers, and also just taken most of the suggested requirements to be the package requirement list. (That the trac colour scheme means that some of the hyperlinks are hard to identify, but we can sort that out somehow). I would be interested to hear people's opinions on how they compare.

...

This is a proposal for the 'tar' package to be included in the next major release of the Haskell platform.

Should specify version number.

I expect that usually the exact number is not especially relevant. The exact released version now is not necessarily the same as the final one that gets included in the case that reviewers make suggestions.

If there's more than one active branch then yes it's relevant so reviewers know which one to look at. Otherwise reviewers can reasonably assume the version on the hackage page.

I don't see the harm in giving the intended version number in the proposal. It may be that 1.18 is originall proposed and 1.20 ends up being what goes into the next release, if the changes are either following the decisions made during the discussion, or small enough that the release team would happily automatically upgrade 1.18 to 1.20 between two major releases.

The policy allows the info to be inline in the proposal or linked to.

Right, I saw that, but I think it should be /required/ to be a link.

...

Introduction to the API

People should not be given the opportunity to /write/ an introduction to the API here; it should already be in the haddock docs on hackage, or on the project page on community, or in some similar location.

If we add that as a package requirement then it would be reasonable to change the advice to say simply to link to it. Currently the only doc requirement we've put in is:

Library packages should have Haddock API documentation for all exported functions and types.

I didn't mean to give an opinion on whether or not an intro to the API should be required or not, but: * if it is required then I think it should be linked to * if it isn't then I certainly don't think the proposal should have to include one (because then there would be no reason not to require that it exists). Based on our earlier IRC discussion: <Igloo> So is the API intro in the haddock docs? <dcoutts> yes <Igloo> OK, ta <dcoutts> that's where I copied most of the stuff in the proposal from <dcoutts> but re-edited, hopefully to be better <dcoutts> I also left out detail to make it a shorter and easier intro it sounds to me like this is an example of exactly what I wanted to avoid. You've spent some time making a nice friendly introduction to the tar API, but in 6 months time when someone comes to use the tar package they won't find this documentation, as it only exists in a "package addition" proposal buried on the haskell-platform wiki. If you're going to go to all the effort of making an API intro, you ought to put it on the package's homepage.

I expect once we've agreed the procedure that we'll get onto more discussing more detailed requirements on things like documentation.

Drifting off-topic, but I think that the two discussions are orthogonal (i.e. we could have them in either order), but the "detailed requirements" discussion is the more important one. People can make proposals without a process being nailed down (and that may even help us work out what the process should be), but how can we decide if a package should be included if we don't know what the acceptance criteria are? So I think the other discussion is much closer on the critical path.

...

Therefore this too should just be a link. Again, this will remove the temptation to update the proposal but not the real docs following the discussion. It will also mean people don't waste time reformatting docs in wiki markup.

The guidance notes say:

Of course, if you end up making something new for the proposal then it'd be great to keep it somewhere so that new users can benefit from it too. The obvious place is in the Haddock docs, or a project website.

Yes, and people will honestly intend to merge the changes and new docs back while they are writing them. But 90% of the time they won't.

...

Open issues

This is the interesting bit of the proposal, IMO. What I want to see for a proposal is just something like:

That's a quite different kind of thing from what the open issues section is intended for. The policy says:

Open issues and objections raised by reviewers should be tracked on the proposal wiki under a separate section. [note-3.2]

and

An explicit checklist of the package requirements below is not required.

What you're asking for is a checklist.

Yes indeed. As in my example: http://trac.haskell.org/haskell-platform/wiki/Proposals/alternate_example

That idea was particularly controversial within the steering committee and it was not adopted. Personally I was weakly in favour if a checklist but the concerns were that it restricted the arguments and explanations that the proposal author could make. The consensus was for a much more free-form proposal rather than forcing things into a checklist.

For items that benefit from free-form text, you can link to a section beyond the checklist, as in my example. Where there would no benefit, a simple "Yes" in the table suffices, and doesn't waste any reviewer time.

...

and the goal of the discussion would then be to get a consensus on whether the yellow sections are severe enough to not accept the package (and also to give people a chance to disagree that some of the green entries really are green, e.g. "Is useful?" is subjective).

I think that would come under the complaint that that is too narrow for the argument for why the package should be included.

I don't understand what you mean by that.

...

I've mentioned this link on IRC before, but for those who haven't seen it, this is partly inspired by: http://release.debian.org/etch/arch_qualify.html (which shows at a glance which arches are acceptable to be in a Debian release, and where the issues and potential issues are. The link in the first row gives more detail on some of the points).

My concern if this is the main focus of discussions is that it covers the "does it work" aspect reasonably well, it doesn't cover the "is it any good? how could it be better?" aspects.

You can have an "Any API issues?" question, which if the answer is not "No" will link to one or more free-form text sections. But if the answer is "No" then it's clear at a glance that it is no. Much like the "Other issues" item in my example.

The standard of consensus required is the same as that defined for the existing library submissions process. [note-5.4]

Our early draft simply said that and nothing else. Several members of the steering committee and others who read drafts were unsure or unconvinced that consensus would be a usable method. We agreed to try it and see if it works. In addition we added the option to use this consensus protocol.

You will notice that in the simple case there is essentially no formality at all:

The first stage is presenting the proposal followed by open group discussion. This stage lasts long enough to give everyone the opportunity to send in their review comments. There follows a call for consensus.

In fact, I think that's the main problem I have with both the process document and the example proposal: They try to anticipate all questions that might be asked, and answer them. But the common case is that the question wouldn't be asked, so this is just more stuff for someone to write, and and more stuff for many people to read. That's why here:

I agree that what I've written for the rationale for the tar proposal is of somewhat dubious utility. In fact personally I don't like the term

you've had to write stuff "of somewhat dubious utility": because you're answering a boilerplate question that didn't need asking in this case. And in the case of consensus, I don't think we need to lay down rules now for what happens if the current system doesn't work. For one thing, it hasn't been an issue so far. For another, when we have the first case of it not working, it might be clear that the solution would be to do something else anyway.

What are you suggesting specifically that would reduce the amount of work for reviewers?

I am suggesting that there be less text written.

Would those things increase work for the proposal authors / package maintainers?

It should reduce work for the authors / package maintainers.

Would those things increase or decrease the level of depth/quality of review and decisions?

I believe it would increase the number of reviewers, giving better decisions. Thanks Ian

On Sun, Aug 23, 2009 at 07:49:23PM +0100, Duncan Coutts wrote:

The intro says:

* The rationale section gives an explanation and justification for the policy.

Is there a wording that would make it clearer that you do not have to read it unless you want an explanation for a point in the policy?

I didn't read that line. I'll expand on this (and the other questions about process description length) in a reply to your other e-mail.

...

Drifting off-topic, but I think that the two discussions are orthogonal (i.e. we could have them in either order), but the "detailed requirements" discussion is the more important one. People can make proposals without a process being nailed down (and that may even help us work out what the process should be), but how can we decide if a package should be included if we don't know what the acceptance criteria are?

The acceptance criteria is clearly specified.

Well, it says (somewhat tautologically): A package is considered as accepted if, by the discussion deadline, the libraries mailing list reach consensus to accept it. but I don't see how we can reach consensus about a package that (for example) is not -Wall clean if we haven't decided whether or not all packages in the platform must be -Wall clean.

It is not that a set of minimal requirements be met. It is that the reviewers, using their judgement come to a consensus to accept the package.

I agree that some of the requirements will be fuzzy (such as "good API"), and we may even overlook black-and-white requirements in particular cases if there is a good reason.

...

...

Would those things increase work for the proposal authors / package maintainers?

It should reduce work for the authors / package maintainers.

Even taking into account that you would expect them to write these external documents? Where is the reduction coming from?

Like you said at the start of your mail, "the 'tar' example could be improved and shortened", so to some extent at least we are in agreement, and the issue was just that the example was written for an earlier, more rigid process. I suspect that I would say that it could be shortened even more than you are thinking, with more of your paragraphs being condensed into a line in the checklist (in the common case where the package obviously satisfies the criteria).

Less text in the proposal or less text in total?

A mixture of both. Thanks Ian

On Sun, Aug 23, 2009 at 08:31:17PM +0100, Duncan Coutts wrote:

Passing all requirements would be the criteria for inclusion.

I would say "Passing all requirements would be sufficient for inclusion", but we may decide to include packages that don't meet all the requirements.

* That the policy document itself is too long and too detailed.

That it anticipates eventualities that may or may not arise in practise.

That the overall length is a problem because it is off-putting, with the danger that people simply will not read it.

Right. I was admittedly reading it at ~2am, but IIRC the way I read it was: * Hmm, the widget on the scrollbar is very small. Reading this is going to take a while. * Drag widget down a bit. See "[note-6.3]". Hmm, jumping up and down between the text and all the notes is really going to break the flow if I try and read it properly. * Drag widget down more. See sections like "Acceptance", "Proposal content" and "Package requirements", not realising I was looking at rationale. * Drag widget down more, and see the "Achieving consensus" major section. Boggle at the size as compared to the library submissions proposal. * Read a little text, probably from the early "Procedure" section. By contrast, I would probably have read this: http://trac.haskell.org/haskell-platform/wiki/AddingPackagesCore If you do want to keep the rationale in the same document then making it expandable inline with JavaScript may be better. (It would still work in non-JS browsers, but be very verbose in them).

Ian, is this a fair summary? Is there anything I've missed that I didn't address specifically in the previous email?

Yes, I think it's a fair summary and covers everything. Thanks Ian

On Tue, 2009-09-08 at 13:02 +0100, Simon Marlow wrote:

...

The main document contains: * the procedure itself, which is relatively short * a rationale, cross-linked to the procedure * a procedure to help us make decisions

http://trac.haskell.org/haskell-platform/wiki/AddingPackages

On the whole, +1.

The process is quite detailed, so we shouldn't expect reviewers to be familiar with all the rules; instead it's up to the steering committee to guide the discussion and remind people of the administrative details at each stage (e.g. "you have until X to respond, or we assume you agree", or "this is the first phase, during which we are trying to reach consensus on Z").

Right. I think that's fine. We'll all get the hang of it after the first round or so.

Specific comments:

"come to view on whether the package should be accepted" -> "help achieve a consensus ... "

Ok.

"the libraries mailing list reach consensus to accept it" -> "the reviewers reach a consensus to accept it" ?

Ok.

"Compile on all operating systems and compilers that the platform targets" -> ".. except where the package is compiler- or platform-specific" ?

Right, we want language that lets us have platform-specific stuff where it's really essential but strongly encourage portable as the default. For example we've got the win32 and unix packages because they're there to expose the native platform apis. But for example if we added file change notification I expect we'd want to provide a platform independent api rather than three packages, one for each native api (or at least we'd want to provide something portable in addition to low level native apis).

The "Interim Licesne Policy" should be one of the bullet points under Package Requirements. e.g.

"The package must be distributed under an acceptable license. The only license currently acceptable is BSD3 [rationale..]"

Yes and no :-). I'll say that it must comply with the license policy. I've been trying very hard to avoid writing the sentence "the only license acceptable is BSD3" because it's highly misleading to say that.

I'd like to see something about API style mentioned in the "Package Requirements". e.g.:

Packages admitted to the platform should follow an API style similar to those packages already in the package. While the style guidelines for package APIs have not yet been written down, we expect that to change in the future.

That's for the discussion after we agree on the procedure. There are plenty more package requirements that we'll want to discuss. We've deliberately started with a minimal set because we wanted to separate the two discussions. Ok, so I'll check the above with someone else on the steering committee and update the document. Duncan