Platform policy question: API compatability in minor releases

Hello everyone, We (the platform release team) have a policy question about the Haskell Platform. We're asking here because we think the libraries list is the right authority to decide. The Haskell Platform concept calls for major and minor releases. * The point of major releases is to update the set of common packages we all use. That is, to specify which packages and which versions (corresponding to APIs). The target is to have major releases every 6 months. * The point of minor releases is to support a major release across a range of operating systems over time and respond to bugs that are discovered in a library or tool subsequent to their release. The target is to have 2-3 minor releases after major releases at intervals on the order of 4-6 weeks. The question is this: Should we allow compatible API additions to a library in a minor release of the platform? The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs. It is clear that incompatible API changes must not be allowed in minor releases. On the other hand, bug fixes that do not change the API at all are ok. Let me put some arguments in each case: Allow compatible API additions If the change is compatible then it does not break any existing programs, so it should be allowed. Any program that works with a platform particular release will continue to work with subsequent minor versions. This is standard backwards compatibility policy. Getting minor feature improvements is sufficiently important that it is too long to wait 6 months for the next major release of the platform. Allow only bug fixes, no API additions Forwards compatibility is important too. A developer should be able to make a package and test it with platform version X.Y.2 and it should still work for a user with version X.Y.1. That way we do not force upgrades on users all the time. Users should only have to upgrade to a later minor release if earlier versions didn't build/install/work on their OS, or if they hit some particular bug which is fixed in a later minor release. There is no need for API changes in minor releases. Maintainers can make a release with just the bug fixes to go into a minor platform release and then make additional releases with new features on Hackage. Users who want the new features can get the new package version from Hackage. API additions in minor releases works against the distinction between major and minor releases. I hope that's not too biased a summary (I'll post my own opinion separately). If there is anything that I should clarify then please ask. So please post your thoughts and arguments. I hope we can come to a conclusion within a few days. This is a timely issue because we aim to make a minor release on the 1st of June and we want to finalise the list of package versions two weeks prior, by the 18th of May. http://trac.haskell.org/haskell-platform/wiki/ReleaseProcess Duncan

On May 9, 2009, at 20:50, Duncan Coutts wrote:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
Yes, I think minor additions should be allowed for two reasons: - a maintainer might not bother to release for a minor platform revision unless he has actually a bug to fix; thus a user that fails to compile a program on an earlier minor platform version likely tries to build against a buggy library anyway. Upgrading is always better and faster than debugging. - it is less of a burden for a maintainer who can keep one working copy of his/her library, rather than an additional one that corresponds to the major platform release. Furthermore, with two copies, it is always necessary to distinguish between fixes and additions. Sometimes patches are more logical if they contain both. My 2p, Axel.

On Sat, May 09, 2009 at 07:50:01PM +0100, Duncan Coutts wrote:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
IMHO, on most cases API additions are improvements, not bug fixes. Maybe there was a developer that needed that new function to be able to make better use of the library, but this kind of improvement should go to the next major release, otherwise the platform would be a moving target. However, this shouldn't be a hard restriction. Should a new addition be important enough for the package's usability, then it should be included with a big fat warning on the docs saying that you'll need to depend on a minor version of the platform. Somewhat related, ADTs should not be extended in minor releases unless they're not exported (of course). But contrary to new functions, I can't think of an ADT change that would be worhty of being added to a minor release. Another problem is that they may create partial functions on the user's code, while new functions shouldn't change the code's behaviour. Thanks, -- Felipe.

Duncan Coutts wrote:
So please post your thoughts and arguments. I hope we can come to a conclusion within a few days. This is a timely issue because we aim to make a minor release on the 1st of June and we want to finalise the list of package versions two weeks prior, by the 18th of May.
I'm in favour of allowing API 'upgrades': I don't think forwards compatiblity is that valuable, and I think that making life as easy as possible for maintainers (i.e. reduce the chances that they have to fork their package to support both the next minor release and the next major release) is a good thing. Ganesh =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================

Duncan Coutts wrote: [..]
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
How has this been handled historically by the GHC team? I lean towards only doing bug-fixing in minor releases, at least if the 6-monthly release schedule for major releases can be kept. /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus@therning.org Jabber: magnus@therning.org http://therning.org/magnus identi.ca|twitter: magthe

On Sat, 2009-05-09 at 21:26 +0100, Magnus Therning wrote:
Duncan Coutts wrote: [..]
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
How has this been handled historically by the GHC team?
I think I am correct in saying that for the most part there have been no API additions to the base/core libs in minor GHC releases. In some cases there have been exceptions in less frequently used modules like GHC.* and we have on one occasion shipped an updated Cabal library in a minor version that exposed some new API (but not one that most Setup.hs scripts were intended to). Duncan

duncan.coutts:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
I'd like to ensure that the Haskell Platform respects the minor versioning policy for its constituent parts, specified, http://haskell.org/haskellwiki/Package_versioning_policy#Version_numbers That is, if a library author follows the minor version number bump policy, then they can reasonably expect to have that library bump included in the next minor bump of the platform. If we agree on this, a lot of the decision making for library bumps becomes automated. If we don't, we have to distinguish case-by-case whether a library bump added to the API. Summary: the Platform policy should follow the policy of its consituent parts. -- Don

On Sat, May 9, 2009 at 10:32 PM, Don Stewart
duncan.coutts:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
I'd like to ensure that the Haskell Platform respects the minor versioning policy for its constituent parts, specified,
http://haskell.org/haskellwiki/Package_versioning_policy#Version_numbers
That is, if a library author follows the minor version number bump policy, then they can reasonably expect to have that library bump included in the next minor bump of the platform.
If we agree on this, a lot of the decision making for library bumps becomes automated. If we don't, we have to distinguish case-by-case whether a library bump added to the API.
Summary: the Platform policy should follow the policy of its consituent parts.
I agree with this. If the platform will eventually becomes available as a cabal-installable package I expect it to follow the PVP. That means that bumping the third version component implies (possible) API additions. Any library that wants to depend on the HP and only get bug-fix updates should specify a x.y.z.* dependency on HP. If we decide that a minor version increase in the HP package's version means bug-fix release these libraries won't benefit from that bug-fix release. Cheers, Johan

On Sun, 2009-05-10 at 00:31 +0200, Johan Tibell wrote:
I'd like to ensure that the Haskell Platform respects the minor versioning policy for its constituent parts, specified,
http://haskell.org/haskellwiki/Package_versioning_policy#Version_numbers
That is, if a library author follows the minor version number bump policy, then they can reasonably expect to have that library bump included in the next minor bump of the platform.
If we agree on this, a lot of the decision making for library bumps becomes automated. If we don't, we have to distinguish case-by-case whether a library bump added to the API.
Summary: the Platform policy should follow the policy of its consituent parts.
I agree with this.
If the platform will eventually becomes available as a cabal-installable package I expect it to follow the PVP.
That's certainly doable. If we decide that compatible API additions are ok in minor releases then we'd be using version numbers like 2009.2.0, 2009.2.1 which is compliant with the PVP. If we decide that compatible API additions must be reserved for major releases then to be PVP compatible we would need version numbers like 2009.2.0.0, 2009.2.0.1. That is, under the "no API changes in minor releases" policy then the platform meta-package would always have a 0 3rd digit. If it reduces confusion for the platform version to also follow the PVP then I'm certainly not opposed to that (even if it introduces apparent redundancy in the platform version number). Duncan

On Sat, 2009-05-09 at 13:32 -0700, Don Stewart wrote:
duncan.coutts:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
I'd like to ensure that the Haskell Platform respects the minor versioning policy for its constituent parts, specified,
http://haskell.org/haskellwiki/Package_versioning_policy#Version_numbers
That is, if a library author follows the minor version number bump policy, then they can reasonably expect to have that library bump included in the next minor bump of the platform.
I don't follow this. The package versioning policy distinguished compatible API changes from no API changes.
If we agree on this, a lot of the decision making for library bumps becomes automated. If we don't, we have to distinguish case-by-case whether a library bump added to the API.
Surely it's automatic in either case? If we choose to allow API additions then it's automatic, we just look at if the first two version numbers have not changed. If we choose to require the same API then it's automatic, we just look at if the first three version numbers have not changed.
Summary: the Platform policy should follow the policy of its consituent parts.
Yes, but we have a choice in what we pick. Duncan

duncan.coutts:
On Sat, 2009-05-09 at 13:32 -0700, Don Stewart wrote:
duncan.coutts:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
I'd like to ensure that the Haskell Platform respects the minor versioning policy for its constituent parts, specified,
http://haskell.org/haskellwiki/Package_versioning_policy#Version_numbers
That is, if a library author follows the minor version number bump policy, then they can reasonably expect to have that library bump included in the next minor bump of the platform.
I don't follow this. The package versioning policy distinguished compatible API changes from no API changes.
Right, and that's exactly what we should follow for the platform meta package.
If we agree on this, a lot of the decision making for library bumps becomes automated. If we don't, we have to distinguish case-by-case whether a library bump added to the API.
Surely it's automatic in either case?
If we choose to allow API additions then it's automatic, we just look at if the first two version numbers have not changed.
If we choose to require the same API then it's automatic, we just look at if the first three version numbers have not changed.
That's a good point! Yes, I'm advocating the "minor" (2 significant digits) policy applies both to the platform, and its constituents, identically. -- Don

duncan.coutts:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
Do we need to fixate on one or the other? That is, I think it'd be nice to allow both, distinguishing them by the magnitude of the version change (e.g. the 3rd vs the 4th magnitude, ala the PVP). It can be a burden for developers to maintain separate forks of their projects to distinguish bug fixes from API upgrades, but it seems like it would be easy for THP to maintain separate lines--- to the extent it can automate everything. My proposal is, every time a "minor update" is to be made, THP could fork x.y.z.a and provide x.y.z.(a+1) which only includes bug-fix improvements and also x.y.(z+1).(a+1) which includes both bug-fix and compatible-API updates. Every time there's a "major update", all these branches would merge back together and be obviated by the new major version. Naturally this scheme is predicated on being able to automate most of the work for updating THP. Don Stewart wrote:
I'd like to ensure that the Haskell Platform respects the minor versioning policy for its constituent parts, specified,
http://haskell.org/haskellwiki/Package_versioning_policy#Version_numbers
That is, if a library author follows the minor version number bump policy, then they can reasonably expect to have that library bump included in the next minor bump of the platform.
If we agree on this, a lot of the decision making for library bumps becomes automated. If we don't, we have to distinguish case-by-case whether a library bump added to the API.
Summary: the Platform policy should follow the policy of its consituent parts.
+1. -- Live well, ~wren

On Sat, 2009-05-09 at 23:22 -0400, wren ng thornton wrote:
duncan.coutts:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
Do we need to fixate on one or the other? That is, I think it'd be nice to allow both, distinguishing them by the magnitude of the version change (e.g. the 3rd vs the 4th magnitude, ala the PVP).
So you're suggesting we have not just major and minor releases but a third kind such that we can have major releases, bug fix releases and in between the two we also have feature releases. I would argue that that adds unnecessary complexity, not just to the release process (which can be automated) but users have to understand it. The number of versions in use simultaneously would be rather high (which detracts from the synchronisation goal).
Summary: the Platform policy should follow the policy of its consituent parts.
+1.
I hope it's clear to everyone that this point is orthogonal to the question at hand. We can ensure the platform version numbers follow whichever policy we agree on. Duncan

On Sat, May 9, 2009 at 12:50, Duncan Coutts
Allow compatible API additions
If the change is compatible then it does not break any existing programs, so it should be allowed. Any program that works with a platform particular release will continue to work with subsequent minor versions. This is standard backwards compatibility policy.
Isn't this false, though? Some existing program could easily break. Suppose a library exports functions f and g, but in the next minor revision starts exporting h. Some code that uses the library uses f and g, and defines its own (unrelated) function called h in the module that imports the library unqualified. When the library gets updated, the code now has an ambiguous reference to h. Please correct me if I'm wrong. Denis

On Sat, 2009-05-09 at 16:29 -0600, Denis Bueno wrote:
On Sat, May 9, 2009 at 12:50, Duncan Coutts
wrote: Allow compatible API additions
If the change is compatible then it does not break any existing programs, so it should be allowed. Any program that works with a platform particular release will continue to work with subsequent minor versions. This is standard backwards compatibility policy.
Isn't this false, though? Some existing program could easily break.
Suppose a library exports functions f and g, but in the next minor revision starts exporting h. Some code that uses the library uses f and g, and defines its own (unrelated) function called h in the module that imports the library unqualified. When the library gets updated, the code now has an ambiguous reference to h.
Please correct me if I'm wrong.
You're not wrong. The PVP operates under the assumption that client code that depends on a package that follows the PVP will use qualified or explicit imports. http://www.haskell.org/haskellwiki/Package_versioning_policy If anything, your point is an argument that we should not allow "compatible" changes because although we can expect people uploading packages to Hackage to understand the versioning policy of packages that their own package depends on, the same is not necessarily true of all users for all programs. The platform has a wider remit in this case as it is intended to serve as the library basis for most real world Haskell code. Duncan

On Sat, 2009-05-09 at 19:50 +0100, Duncan Coutts wrote:
The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs.
To add my personal opinion, I think we should have only bug fixes in minor releases and save new APIs for major releases. I would argue that the distinction between major and minor releases is an important one. As a developer community, major releases are what synchronise our sets of dependencies. It is that periodic synchronisation that ensures that the programs we distribute have the greatest portability. Look at it this way, currently we have a 12 month major release cycle. That's how often major versions of GHC come out, which is what has defined the platform up until now. This is what developers test against. Historically GHC has been pretty firm about not allowing new APIs the base lib and other core libs in minor GHC releases. Indeed the CPP symbol that GHC defines containing its version number deliberately does not contain the GHC minor version. So by arguing for new features in minor releases we're saying we should have a 4-6 week cycle. New library features every 6 weeks. This makes life harder for distributions and users. Instead of just picking up the first version of a major release that works ok for them, they have to track minor releases because minor releases are going to be including new features that new programs are going to start depending on. It means everything goes out of date that much quicker. Now perhaps 12 months is too long (I think it is). Perhaps 6 months is too long even (though many other groups have settled on 6 months). But surely 6 weeks is far too short. That means essentially no synchronisation effect at all. If the majority opinion is that 6 months is too long to wait for new features, then how about 4 months, or even 3? Do people see the point I'm trying to make about the value of synchronisation? It's the ability to say to platform users "this is what you need to build all the haskell code!". (Strictly speaking it's not everything you need, but you can be pretty sure most other packages will have been tested against this set of versions). But if the developers writing that code are constantly depending on the release from 6 weeks ago then we have not achieved that. Then we are not synchronising package versions between devs, users and distros for a reasonable period of time. We do actually want to compress changes into nice well-labelled, synchronised jumps where we all change versions at once. We don't want to tell users "oh! the problem is your distro packages HP 2000.4.1 but this program only works with 2000.4.2". If that's the situation we end up in then I think we've failed. There is a balance to be struck in how long the periods of stability are. Too long and we stagnate but do people also see that there is a problem with the other extreme too? We initially picked on 6 months because that's what many other projects have found to be a good balance (eg GNOME, many Linux distros). 6 months is not carved in stone though. Perhaps initially we would be better served with 4 months. Experience should tell in the end. I accept that from the point of view of package maintainers that it is less work to not have to maintain more than one branch and to make releases that mix both fixes and new features. Of course we should also consider the perspective of users of the platform. Core libraries that are have been released with GHC have of course always had stable branches. This hasn't proved to be too big of a burden on the maintainers. Darcs (and probably other systems) makes this fairly easy. Duncan

Duncan Coutts wrote:
So by arguing for new features in minor releases we're saying we should have a 4-6 week cycle. New library features every 6 weeks.
Thanks for putting some numbers on that. (and, yikes! I'm convinced. I was under the impression that there'd be fewer minor releases than that.)
Now perhaps 12 months is too long (I think it is). Perhaps 6 months is too long even (though many other groups have settled on 6 months). But surely 6 weeks is far too short. That means essentially no synchronisation effect at all.
If the majority opinion is that 6 months is too long to wait for new features, then how about 4 months, or even 3? Do people see the point I'm trying to make about the value of synchronisation?
I think that, given the rate of change in the Haskell world, 12 months is far too long. And I agree that a big part of the platform should be to provide synchronization, for which 6 weeks isn't enough. In choosing between 3, 4, or 6 months (for 4, 3, and 2 times per year) I think there's a balance that needs to be struck for the platform to be useful to users. If updates are too few and too far between (as has historically been the case), then active users will update the packages individually, thus defeating the point of the platform. On the other side of things, synchronicity. Of those three options, I think 4 months is probably ideal. Quarterly releases still don't give much time for synchronization. For stable packages (like are currently included) development is slower in terms of API changes, so 6 months may suffice, but if younger packages are added then I don't think biannual releases will allow the HP to track development closely enough. I still think that it would be good to try to have a three stage model for releases, instead of the two stage model that seems to be assumed. Adding new packages or destructive API changes strike me as being very different from compatible API changes (which are, in turn, separate from big fixes and bundling fixes). I'd like to see these two types of changes being distinguished in terms of the release schedule. -- Live well, ~wren

On Sun, 2009-05-10 at 01:18 -0400, wren ng thornton wrote:
Duncan Coutts wrote:
So by arguing for new features in minor releases we're saying we should have a 4-6 week cycle. New library features every 6 weeks.
Thanks for putting some numbers on that. (and, yikes! I'm convinced. I was under the impression that there'd be fewer minor releases than that.)
Sorry, this is a bit backwards, see the other note we just posted. If we all agreed that we should have new features in minor releases then I strongly suspect that we would not want such releases on such a rapid cycle as 4-6 weeks. That only really makes sense for small bug-fix only releases. As we say in the other email, we should first decide what we want from releases and then pick major/minor cycles to fit, and not the other way around.
Now perhaps 12 months is too long (I think it is). Perhaps 6 months is too long even (though many other groups have settled on 6 months). But surely 6 weeks is far too short. That means essentially no synchronisation effect at all.
If the majority opinion is that 6 months is too long to wait for new features, then how about 4 months, or even 3? Do people see the point I'm trying to make about the value of synchronisation?
I think that, given the rate of change in the Haskell world, 12 months is far too long. And I agree that a big part of the platform should be to provide synchronization, for which 6 weeks isn't enough.
Agreed.
In choosing between 3, 4, or 6 months (for 4, 3, and 2 times per year) I think there's a balance that needs to be struck for the platform to be useful to users. If updates are too few and too far between (as has historically been the case), then active users will update the packages individually, thus defeating the point of the platform. On the other side of things, synchronicity.
Indeed.
Of those three options, I think 4 months is probably ideal. Quarterly releases still don't give much time for synchronization. For stable packages (like are currently included) development is slower in terms of API changes, so 6 months may suffice, but if younger packages are added then I don't think biannual releases will allow the HP to track development closely enough.
Yes, one suggestion is that we start with a 4 month cycle and perhaps review it after 12 or 16 months to see if that pace is right for the longer term.
I still think that it would be good to try to have a three stage model for releases, instead of the two stage model that seems to be assumed. Adding new packages or destructive API changes strike me as being very different from compatible API changes (which are, in turn, separate from big fixes and bundling fixes). I'd like to see these two types of changes being distinguished in terms of the release schedule.
I have to say this isn't something I'd thought of. Tell us more. What would this look like? Would it be something like eg only one release per year is allowed to make breaking changes? Duncan

Duncan Coutts wrote:
On Sun, 2009-05-10 at 01:18 -0400, wren ng thornton wrote:
I still think that it would be good to try to have a three stage model for releases, instead of the two stage model that seems to be assumed. Adding new packages or destructive API changes strike me as being very different from compatible API changes (which are, in turn, separate from big fixes and bundling fixes). I'd like to see these two types of changes being distinguished in terms of the release schedule.
I have to say this isn't something I'd thought of. Tell us more. What would this look like? Would it be something like eg only one release per year is allowed to make breaking changes?
As far as time frames, I think things will need adjusting as we find out more about the development cycles of the new projects to be included in the platform, but the idea is something in the spirit of my older proposal, arguing "why choose": On a long-term arc (e.g. 6~12 months) the HP would make major releases. These releases could include new packages that have been blessed since the previous major release, remove packages that have been deprecated (heaven forfend), and destructive API changes as discussed in the PVP (moving or renaming modules/functions, irreconcilable type signatures, ADT changes,...). On a mid-term arc (e.g. 3~4 months) the HP would make semi-major releases. These releases could include compatible API changes as discussed in PVP (adding new modules/functions, generalizing type signatures (possibly),...). Potentially the addition of newly blessed packages could move here, though see the discussion below. On a short-term arc (e.g. 4~6 weeks) the HP would make minor releases. These releases could include bug fixes for the underlying projects, fixes to the packaging/cabalization of the projects, fixes to the bundling of the HP, etc. ... One of the nice things about having a three-way split like this is that it allows for clearer epoch boundaries in terms of how often different kinds of users will want to upgrade. The fast turnaround of minor releases ensures that bugs can be fixed in a prompt manner even for sites with complex installation setups (e.g. universities). The moderate turnaround of semi-major releases ensures that the HP can still track development rather closely, without sacrificing synchronization due to "hard" API changes. And the slower turnaround for major releases ensures there's still a certain level of prominence for these. It's nice for active developers to have up-to-date libraries so that their development can match with the rest of the community (hence semi-major releases). However, for an open-source community it's also important to have a certain level of ritual and celebration, in order to keep involvement vibrant and to add a sense of urgency and milestones to an otherwise calm development cycle (hence the major releases). This is one of the reasons I think it would be good to have the addition of newly blessed packages only at the major releases instead of at the semi-major releases; it adds a deadline and a celebration for people wanting to get in. Ideally this should be timed to match up with Hackathon or other community events, so folks can work hard and get the reward soon afterwards, compounding the effects of these events for building community ties. Because Hackage remains along side the HP, I'm not too worried about only having yearly major releases. Provided the right kind of automation, developers can "pre-release" major versions on Hackage and that gives enough time to see how these changes are accepted by the community, before integration into the HP. This can help to solidify destructive API changes before those changes are accepted into canon, which will improve the overall synchronization effect of the HP. But for blessing new packages, our propaganda needs to be sure to present this codevelopment as a positive thing, otherwise a long release of 12 months could inspire bitterness or resentment from those who try to make it in and narrowly miss the deadline or the quality criteria. Shortening to a 6 month major release cycle will mitigate this, though it diminishes the ritualism of the yearly release. A cycle of 8~9 months seems to be ideal in terms of grudges, though it doesn't match up with the calendar (which introduces a number of other problems). An alternative solution for the resentment problem may be to have one of the semi-major releases dubbed as a "co-major release" which allows blessing new packages, but is otherwise restricted as per semi-major releases. On the whole, I think the best approach is the simplest though: just have someone managing the media spin and working to keep people happy, rather than trying to schedule it away. -- Live well, ~wren

On Mon, 2009-05-11 at 18:31 -0400, wren ng thornton wrote:
Duncan Coutts wrote:
On Sun, 2009-05-10 at 01:18 -0400, wren ng thornton wrote:
I still think that it would be good to try to have a three stage model for releases, instead of the two stage model that seems to be assumed. Adding new packages or destructive API changes strike me as being very different from compatible API changes (which are, in turn, separate from big fixes and bundling fixes). I'd like to see these two types of changes being distinguished in terms of the release schedule.
I have to say this isn't something I'd thought of. Tell us more. What would this look like? Would it be something like eg only one release per year is allowed to make breaking changes?
As far as time frames, I think things will need adjusting as we find out more about the development cycles of the new projects to be included in the platform, but the idea is something in the spirit of my older proposal, arguing "why choose":
On a long-term arc (e.g. 6~12 months) the HP would make major releases. These releases could include new packages that have been blessed since the previous major release, remove packages that have been deprecated (heaven forfend), and destructive API changes as discussed in the PVP (moving or renaming modules/functions, irreconcilable type signatures, ADT changes,...).
On a mid-term arc (e.g. 3~4 months) the HP would make semi-major releases. These releases could include compatible API changes as discussed in PVP (adding new modules/functions, generalizing type signatures (possibly),...). Potentially the addition of newly blessed packages could move here, though see the discussion below.
On a short-term arc (e.g. 4~6 weeks) the HP would make minor releases. These releases could include bug fixes for the underlying projects, fixes to the packaging/cabalization of the projects, fixes to the bundling of the HP, etc.
It's clear that bug fix only doesn't cause major compatibility headaches (if done sanely) . It's obvious that API breakage causes upset. The argument you're making is that API compatible changes are so much less disruptive that it's ok to have them more frequently. I claim that's not really true. I claim that the synchronization is just as much about API additions as breakage. It's about stability. Users want to know which packages they should have to be able to run most Haskell code. Developers want to know what packages most users have got so they can make their code work with those versions. We can improve these periods of stability without reducing the pace of development by compressing changes into major releases rather than spreading them out. Also consider Claus's question about how long a major release remains relevant for. The more frequent our major releases (and that means new features not just breakage) the more quickly they become obsolete. I suggest we deal with the issue of users not wanting to upgrade too frequently by having a major release cycle of a sensible length and supporting old releases for a while. You bring up some good points about the pressure for maintainers to get new features into major releases. This is an important consideration in major release frequency. I suggest you take a look at Martin Michlmayr's PhD thesis in which he discusses this issue. Quality Improvement in Volunteer Free and Open Source Software Projects: Exploring the Impact of Release Management http://www.cyrius.com/publications/michlmayr-phd.html Duncan

Duncan Coutts wrote:
It's clear that bug fix only doesn't cause major compatibility headaches (if done sanely) . It's obvious that API breakage causes upset.
Indeed. (Relatedly, one detail not covered by the PVP is the issue of "strictness bug fixes". Even if previous strictness behavior is considered buggy, it's not clear that changing strictness behavior should be considered any less dangerous than consistent changes to type signature, e.g. generalizing them.)
The argument you're making is that API compatible changes are so much less disruptive that it's ok to have them more frequently.
I claim that's not really true. I claim that the synchronization is just as much about API additions as breakage. It's about stability. Users want to know which packages they should have to be able to run most Haskell code. Developers want to know what packages most users have got so they can make their code work with those versions. We can improve these periods of stability without reducing the pace of development by compressing changes into major releases rather than spreading them out.
Perhaps we should agree to disagree. Certainly additions can cause breakage (either by clashing with local definitions or by assuming they're exported), but I do think this breakage is sufficiently diminished that it's worth distinguishing it from the obvious breakage. That is, I think that many of the additions will not cause much breakage *in practice*, and so ---as per Claus' question of relevance--- allowing addition-releases between breakage-releases will serve to increase the duration of relevance and therefore overall stability. Whether this belief or your belief will be more correct depends, IMHO, on many factors that cannot be known without trying them. I, at least, do not think we have the information necessary to make *accurate* predictions about the stability/relevance of any particular approach. (Perhaps Don could provide an analysis of the frequency of API additions vs API breakages on Hackage?) I have a good deal of experience with F/OSS communities, but the pace of change in Haskell is far different than the pace in web development, applications, or OSes--- so I'm wary of how much experience from those areas can be effectively ported to this task. Another important issue that hasn't been much discussed is how long and to what extent do you plan to maintain old releases? I think the answer to this question will have major repercussions on both the frequency of non-bugfix releases and on whether it would be meaningful to distinguish enhancements from changes.
You bring up some good points about the pressure for maintainers to get new features into major releases. This is an important consideration in major release frequency. I suggest you take a look at Martin Michlmayr's PhD thesis in which he discusses this issue.
Quality Improvement in Volunteer Free and Open Source Software Projects: Exploring the Impact of Release Management http://www.cyrius.com/publications/michlmayr-phd.html
I have, it's a good read :) In Michlmayr's terms, I'm advocating for the notion of development releases for the HP as separate from development releases for the underlying projects on Hackage. Cutting edge vs bleeding edge, as it were. This is similar to GCC's three-tier model. Also it should make it easier to avoid OpenOffice.org's QA problems, while maintaining frequent (3~4 month) non-bugfix releases. -- Live well, ~wren

On Thu, 2009-05-14 at 23:09 -0400, wren ng thornton wrote:
The argument you're making is that API compatible changes are so much less disruptive that it's ok to have them more frequently.
I claim that's not really true. I claim that the synchronization is just as much about API additions as breakage. It's about stability. Users want to know which packages they should have to be able to run most Haskell code. Developers want to know what packages most users have got so they can make their code work with those versions. We can improve these periods of stability without reducing the pace of development by compressing changes into major releases rather than spreading them out.
Perhaps we should agree to disagree.
I expect so.
Certainly additions can cause breakage
But that's not what I'm saying. Even additions that cause no breakage fragment the userbase. Half the users will have version X and half will have X+compatible-extras and now developers have to decide if it's ok to take advantage of the new features and exclude half the users. The users then also feel much more pressure to upgrade which puts pressure on the distros to do the same. The point is, feature releases do turn into major releases. So we cannot pretend we're having say a 6 month cycle if we've also got feature releases in between.
Whether this belief or your belief will be more correct depends, IMHO, on many factors that cannot be known without trying them. I, at least, do not think we have the information necessary to make *accurate* predictions about the stability/relevance of any particular approach. (Perhaps Don could provide an analysis of the frequency of API additions vs API breakages on Hackage?) I have a good deal of experience with F/OSS communities, but the pace of change in Haskell is far different than the pace in web development, applications, or OSes--- so I'm wary of how much experience from those areas can be effectively ported to this task.
Right, the experience of other communities is our best guide but there are things unique to each group. For example it's fine for an application to come out every couple months. There's no particular problem in the user base using a wide range of versions. However with a programming platform we get network effect benefits when we're mostly using the same versions of most common packages most of the time. (API additions cut network connections in one direction while API breaks cut network connections in both.)
Another important issue that hasn't been much discussed is how long and to what extent do you plan to maintain old releases? I think the answer to this question will have major repercussions on both the frequency of non-bugfix releases and on whether it would be meaningful to distinguish enhancements from changes.
My guess is that it'd be feasible to aim to support three releases simultaneously. At a 6 month cycle that'd be 18 months.
You bring up some good points about the pressure for maintainers to get new features into major releases. This is an important consideration in major release frequency. I suggest you take a look at Martin Michlmayr's PhD thesis in which he discusses this issue.
Quality Improvement in Volunteer Free and Open Source Software Projects: Exploring the Impact of Release Management http://www.cyrius.com/publications/michlmayr-phd.html
I have, it's a good read :) In Michlmayr's terms, I'm advocating for the notion of development releases for the HP as separate from development releases for the underlying projects on Hackage.
Right, the HP would always be picking up existing releases from Hackage.
Cutting edge vs bleeding edge, as it were. This is similar to GCC's three-tier model. Also it should make it easier to avoid OpenOffice.org's QA problems, while maintaining frequent (3~4 month) non-bugfix releases.
So it looks like the only thing we don't quite agree on is the frequency of the release cycle, though at least we do have overlapping ranges :-) (3~4 vs 4~6) Duncan

Duncan Coutts wrote:
So by arguing for new features in minor releases we're saying we should have a 4-6 week cycle. New library features every 6 weeks.
Now perhaps 12 months is too long (I think it is). Perhaps 6 months is too long even (though many other groups have settled on 6 months). But surely 6 weeks is far too short. That means essentially no synchronisation effect at all.
If the majority opinion is that 6 months is too long to wait for new features, then how about 4 months, or even 3? Do people see the point I'm trying to make about the value of synchronisation?
Can the Haskell Platform be uninstalled / upgraded seamlessly? I think it's a key technical requirement for shorter release cycles that installing a new release does not break due to any old ones that were installed previously. Regards, apfelmus -- http://apfelmus.nfshost.com

On Sun, 2009-05-10 at 09:11 +0200, Heinrich Apfelmus wrote:
Duncan Coutts wrote:
So by arguing for new features in minor releases we're saying we should have a 4-6 week cycle. New library features every 6 weeks.
Now perhaps 12 months is too long (I think it is). Perhaps 6 months is too long even (though many other groups have settled on 6 months). But surely 6 weeks is far too short. That means essentially no synchronisation effect at all.
If the majority opinion is that 6 months is too long to wait for new features, then how about 4 months, or even 3? Do people see the point I'm trying to make about the value of synchronisation?
Can the Haskell Platform be uninstalled / upgraded seamlessly?
This depends on the packaging for each OS. The Windows installer will install a new ghc and set of packages without disturbing existing installations. The source based installer will do the same (though it has no builtin uninstall facility). Not sure yet about OSX.
I think it's a key technical requirement for shorter release cycles that installing a new release does not break due to any old ones that were installed previously.
It is possible in principle though many distro packaging systems only allow one version of a package at once. I think distros would prefer the slightly longer release cycle. They already deal with many projects that work on 6 month cycles. However they could probably cope with 4 months. They cope with the Linux kernel's 3 month cycle, though they often skip some major releases. Duncan

Duncan Coutts wrote:
Heinrich Apfelmus wrote:
Can the Haskell Platform be uninstalled / upgraded seamlessly?
This depends on the packaging for each OS.
The Windows installer will install a new ghc and set of packages without disturbing existing installations. The source based installer will do the same (though it has no builtin uninstall facility). Not sure yet about OSX.
I think it's a key technical requirement for shorter release cycles that installing a new release does not break due to any old ones that were installed previously.
It is possible in principle though many distro packaging systems only allow one version of a package at once.
Ah, what I meant is that the following scenario: 1) User installs Haskell Platform 1.0 2) New release of the Haskell Platform, version 1.1 3) User downloads the new release and installs it 4) User is no longer able to compile or run his Haskell programs because of some artifact from HP 1.0 that interferes with the new install of HP 1.1. For instance, what about his previous Hackage installs? Both HP 1.0 and HP 1.1 work fine in isolation. 5) User expresses his emotions in various ways, like breaking in tears on the #haskell channel, writing furious blog posts or divorcing Haskell entirely. So, having only one release installed is not a problem, and the Linux package managers in general are unlikely to create the mess required for 4). But I bet that 4) is going to happen for the Windows and OS X installers. I think it would be counterproductive to the goals of the HP if people don't upgrade because they don't want to risk breaking their working Haskell environment. Regards, apfelmus -- http://apfelmus.nfshost.com

it is possible to depend on bug-fixes in the Platform too, and they are even more subtle to know whether (or how much) you depend on the bug-fixes. API additions could at least face some automated testing... Also, does GHC have a warning-flag for all those imports that could have problems given API additions? And is there infrastructure work so that someday we might be able to take summaries of all HP APIs and run a test that tells you which HP versions your HP-using code could compile against? (at least for the relatively simple case of API *additions*) Crazy idea: What if we do the maintenance/bug-fix-only release every 4-6 weeks, and a release roughly 3 months cycle that allows API additions but no changes; and API changes are allowed only on the 6-month cycle? Then people who import carefully, only need to re-test their code twice a year. And forwards compatibility is not as bad as the 'every 4-6 weeks' situation, it's closer to 3 months. for example a series of releases in some year might look like: March: HP 2000.1.0.0 April: HP 2000.1.0.1 May: HP 2000.1.0.2 June: HP 2000.1.1.0 July: HP 2000.1.1.1 August: HP 2000.1.1.2 September: HP 2000.2.0.0 etc... -Isaac

Isaac Dupree wrote:
Crazy idea: What if we do the maintenance/bug-fix-only release every 4-6 weeks, and a release roughly 3 months cycle that allows API additions but no changes; and API changes are allowed only on the 6-month cycle? Then people who import carefully, only need to re-test their code twice a year. And forwards compatibility is not as bad as the 'every 4-6 weeks' situation, it's closer to 3 months.
I don't think it's so crazy. See my most recent reply to Duncan for a fuller discussion of why I think this is the best answer. -- Live well, ~wren

On Sun, May 10, 2009 at 2:20 AM, Duncan Coutts
On Sat, 2009-05-09 at 19:50 +0100, Duncan Coutts wrote: If the majority opinion is that 6 months is too long to wait for new features, then how about 4 months, or even 3? Do people see the point I'm trying to make about the value of synchronisation?
I've always pictured the HP to be for those users who don't want to live on the bleeding edge and don't want to install packages using cabal-install but rather use the distro package manager. This means that we can only get new features out to these users whenever the distro makes a new release. This means that it doesn't matter if we do several releases per distro release. The distro will pick up the latest release that is available and release with that. So I suggest we stick with a major release every 6 months and 1-2 bug-fix releases between every major release. Cheers, Johan

I think in the interests of clarity and simplicity we should avoid any API changes, even additions, in minor releases. If this means that major releases need to be more frequent than 6 months, then so be it; but we really don't want the complexity of two layers of API-changing releases. However, I'd like to argue for sticking with 6-month major releases, or at the very least 4 months. The benefit of the platform is that we get a set of compatible and well-tested libraries as a single unit, and the closer together the releases are, the harder it becomes to deliver that. The more frequent the feature releases are, the lower the overall quality: new code introduces new bugs, there needs to be enough time to stabilise. We've found in GHC that we can just about manage 12-month feature releases, and even then we often ship with some nasty bugs. Partly this is due to lack of manpower, but also it's down to the lack of testing that pre-releases get - it typically takes us about 6 months to get to a nice stable state after a major release. One disadvantage of the platform is that we might even see *less* testing of the early revisions in a new GHC release, as people stick to whatever comes with the platform, and the platform might be slightly conservative about taking new GHC major releases. In the future I'd like to aim for full ABI compatibility between minor releases. Various people have heard me rant about this before so I won't do it again (especially when the plan is still not fully formed); suffice to say that I think it should be possible to make it such that upgrading to a new minor revision of the platform doesn't break all the other packages you have built and installed using Cabal. Now, while technically speaking it's possible to allow API additions and still maintain compatible ABIs, I don't think we want to deal with that, at least not initially. For cutting-edge programmers who need more recent APIs, there is always Hackage and cabal-install, which work very nicely. Cheers, Simon

On Sat, 2009-05-09 at 19:50 +0100, Duncan Coutts wrote:
Hello everyone,
We (the platform release team) have a policy question about the Haskell Platform. We're asking here because we think the libraries list is the right authority to decide.
The Haskell Platform concept calls for major and minor releases.
I should probably point out the strategy we are trying to implement: time based releases rather than feature based. An excellent source on this issue is Martin Michlmayr's PhD thesis in which he argues for time based releases: Quality Improvement in Volunteer Free and Open Source Software Projects: Exploring the Impact of Release Management http://www.cyrius.com/publications/michlmayr-phd.html It is extremely relevant to us. He surveys many similar projects like GCC, GNOME, Debian and others.
From the abstract:
This dissertation explores why, and under which circumstances, the time based release strategy is a viable alternative to feature-driven development and discusses factors that influence a successful implementation of this release strategy. It is argued that this release strategy acts as a coordination mechanism in large volunteer projects that are geographically dispersed. The time based release strategy allows a more controlled development and release process in projects which have little control of their contributors and therefore contributes to the quality of the output. Shorter references, pertaining to GNOME: http://www.linux.com/articles/114247 http://live.gnome.org/ReleasePlanning/TimeBased http://mail.gnome.org/archives/gnome-hackers/2002-June/msg00041.html Duncan

On Sat, May 9, 2009 at 5:44 PM, Duncan Coutts
I should probably point out the strategy we are trying to implement: time based releases rather than feature based.
And a fine idea that is. I do think that aiming for a 4-6 week minor release cycle is massive overkill, though. As a distro packager, there's no way I'm going to put myself through rebuild-the-world hell once a month because the core libraries keep getting tweaked, but then I run the risk of missing genuinely important bug or security fixes because I'm not paying attention. This "be careful with update frequency" problem is probably more acute for GHC than other languages because of the way that code tends to be inlined so aggressively. I'd aim for quarterly releases, with intermediate bugfix releases iff there's a genuine serious problem that needs fixing.

On Sat, 2009-05-09 at 22:41 -0700, Bryan O'Sullivan wrote:
On Sat, May 9, 2009 at 5:44 PM, Duncan Coutts
wrote: I should probably point out the strategy we are trying to implement: time based releases rather than feature based.
And a fine idea that is. I do think that aiming for a 4-6 week minor release cycle is massive overkill, though. As a distro packager, there's no way I'm going to put myself through rebuild-the-world hell once a month because the core libraries keep getting tweaked, but then I run the risk of missing genuinely important bug or security fixes because I'm not paying attention.
It's an interesting argument, that compared to C we have a problem with upgrading too frequently because of no stable ABI etc. So on that basis you argue that we should not have too many bug fix releases that people are expected to upgrade to, but on the other hand we should have reasonably frequent (3 month) major releases that people and distros are expected to upgrade to. It's a good point though I would resolve it differently. I would suggest that we make it so that you don't actually have to upgrade to each minor bug fix release if the first one you picked worked ok for you. Then we only ask users and distros to upgrade (and thus rebuild everything) every 6 months (or perhaps 4). Note that distros do not package each minor release of GNOME (which come out on a 4-6 week basis after each major 6-month release). Most wait for x.y.2 or .3 before packaging them for a stable distro release. If we are only including bug fixes and not new features then this would be doable for the platform releases too.
This "be careful with update frequency" problem is probably more acute for GHC than other languages because of the way that code tends to be inlined so aggressively.
True. This frequency of bug-fix releases only makes sense if people don't have to upgrade to them unless they're hitting the specific bugs that have been fixed. Just as now, people often don't bother to upgrade to each new minor ghc release. For example, many distros never packaged ghc-6.8.3, sticking to .2 and that was fine. Similarly, many have skipped 6.10.2 (some even skipping 6.10.1).
I'd aim for quarterly releases, with intermediate bugfix releases iff there's a genuine serious problem that needs fixing.
In the quarterly releases would you allow breaking API changes as well as compatible API additions, or would you prefer something like only breaking changes in alternate releases? As I mentioned in my other note, I'd be happy enough with major releases every 4 months rather than 6 with only bug fixes in between. My main worry with more frequent releases is that we loose the synchronisation effect. Duncan

On Sun, May 10, 2009 at 6:05 AM, Duncan Coutts
It's a good point though I would resolve it differently. I would suggest that we make it so that you don't actually have to upgrade to each minor bug fix release if the first one you picked worked ok for you. Then we only ask users and distros to upgrade (and thus rebuild everything) every 6 months (or perhaps 4).
Okay. I do somewhat suspect that you're making rather a lot of work for yourself in that case with all those minor releases that probably won't see much use, but far be it from me to suggest how you should spend your time :-) Overall, I'm very happy with this approach, and with your readiness to engage in substantive discussion around API stability, scheduling, and the like. Thanks so much for your great work!

On Sun, 2009-05-10 at 08:15 -0700, Bryan O'Sullivan wrote:
On Sun, May 10, 2009 at 6:05 AM, Duncan Coutts
wrote: It's a good point though I would resolve it differently. I would suggest that we make it so that you don't actually have to upgrade to each minor bug fix release if the first one you picked worked ok for you. Then we only ask users and distros to upgrade (and thus rebuild everything) every 6 months (or perhaps 4).
Okay. I do somewhat suspect that you're making rather a lot of work for yourself in that case with all those minor releases that probably won't see much use, but far be it from me to suggest how you should spend your time :-)
Once we've got ~50 packages in the platform I think it's inevitable that we'll need bug fix releases. Perhaps we can get away with fewer especially once the process settles down. Time will tell. Sometimes we just need point releases to fix things for newer OS platforms that come along many months later. For example ghc-6.10.x has stopped building from source on OS X 10.5.6+ because of changes on that platform in basic toolchain programs (stricter ar/ld in this case). These are exactly the kind of cases where all other users do not need to upgrade because it already works for them. As for the amount of work, the aim is to automate everything so that producing releases is a sensible amount of work. This really is crucial or release managers will burn out.
Overall, I'm very happy with this approach, and with your readiness to engage in substantive discussion around API stability, scheduling, and the like. Thanks so much for your great work!
:-) I hope I'm not being too belligerent. Duncan

bos:
On Sat, May 9, 2009 at 5:44 PM, Duncan Coutts
wrote: I should probably point out the strategy we are trying to implement: time based releases rather than feature based.
And a fine idea that is. I do think that aiming for a 4-6 week minor release cycle is massive overkill, though. As a distro packager, there's no way I'm going to put myself through rebuild-the-world hell once a month because the core libraries keep getting tweaked, but then I run the risk of missing genuinely important bug or security fixes because I'm not paying attention. This "be careful with update frequency" problem is probably more acute for GHC than other languages because of the way that code tends to be inlined so aggressively.
I'd aim for quarterly releases, with intermediate bugfix releases iff there's a genuine serious problem that needs fixing.
Quarterly major releases? That's a good point: as far as distros are concerned, major and minor releases have about the same impact in terms of labour. -- Don

Am Samstag, 9. Mai 2009 20:50:01 schrieb Duncan Coutts:
[...] The question is this:
Should we allow compatible API additions to a library in a minor release of the platform?
The choice is between allowing only bug fixes in minor releases, or also allowing new features that add APIs but do not change any existing APIs. [...]
We should definitely allow API additions in minor platform releases. Most arguments for this have already been written, but I'd like to point out a few ones: * Consistency in numbering: The platform numbering should have the semantics as the numbering of its parts, therefore allowing API additions in minor releases. If someone is totally unwilling to risk any changes his code, he can still use bug fix releases of the platform instead of minor releases, where's the problem? * IMHO, GNOME is a not a good example, although it is cited here over and over again. GNOME is mainly supposed to be used by *end users*, while the Haskell platform is supposed to be used by *programmers*. Both target groups have very different needs. As a desktop user I probably don't care much about some feature additions, as long as my good old working environment stays the same. As a programmer I'd like to get bug fixes very quickly, get early indications of the direction where a library evolving, try out new cool features, etc. There is always a tension between stability and new features, but given *our* target group, let's not focus on the wrong thing too heavily. * A crucial point in most open source SW development is the distinction between the maintainer of a piece of SW and a packager (nomenclature my vary, but you get the point). The Haskell platform is all about packaging, nothing else. Trying to force the maintainers to handle bug fix branches just for the platform's sake is a no-go IMHO, and something I personally won't do for my libraries. Let's keep the maintainer-packager distinction as it is! * Let's assume that the Haskell platform keeps its planned interval of 6 months for major releases (something I am not totally convinced of until it has happened reliably a few times). If we allowed API additions only twice a year, this would be ridiculously slow, because the world outside doesn't stand still. The OpenGL ARB e.g. released their OpenGL 3.1 spec only 9 months after the OpenGL 3.0 spec, and the differences were by no means small. You can't compare e.g. the network package with this, because improvements or additions in network APIs are virtually non-existent compared to this, we basically still use APIs which are a few decades old. Consequently, allowing API additions only every 6 months would be more than enough for the network package, but it would be far too slow for the OpenGL package. Cheers, S.

Sven Panne wrote: [..]
* IMHO, GNOME is a not a good example, although it is cited here over and over again. GNOME is mainly supposed to be used by *end users*, while the Haskell platform is supposed to be used by *programmers*. Both target groups have very different needs. As a desktop user I probably don't care much about some feature additions, as long as my good old working environment stays the same. As a programmer I'd like to get bug fixes very quickly, get early indications of the direction where a library evolving, try out new cool features, etc. There is always a tension between stability and new features, but given *our* target group, let's not focus on the wrong thing too heavily.
GHC and the haskell platform is open source, that means that users are the future developers, please don't target _existing_ developers only! Ease of use for users should not be thrown away lightly, just look at the steps a user has to go through to install applications written in Ruby. Also, don't forget that in order to make haskell easy to access, and promote applications written in haskell it must be easy to provide packages for different distros/OSs (Linux, BSD, Mac, Win). /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus@therning.org Jabber: magnus@therning.org http://therning.org/magnus identi.ca|twitter: magthe

On Sun, 2009-05-10 at 12:05 +0200, Sven Panne wrote:
We should definitely allow API additions in minor platform releases. Most arguments for this have already been written, but I'd like to point out a few ones:
* Consistency in numbering: The platform numbering should have the semantics as the numbering of its parts, therefore allowing API additions in minor releases. If someone is totally unwilling to risk any changes his code, he can still use bug fix releases of the platform instead of minor releases, where's the problem?
We can make the numbering consistent with whichever policy we choose.
* IMHO, GNOME is a not a good example, although it is cited here over and over again. GNOME is mainly supposed to be used by *end users*, while the Haskell platform is supposed to be used by *programmers*.
GNOME releases have two target audiences. One group, as you say, are end users who just want a working desktop and don't mind too much if there are extra features so long as existing stuff doesn't break. However a major plank of the GNOME release is the "GNOME developer platform" which really is just a set of libraries for making desktop applications. Many third parties rely on these libraries and the API guarantees. It is this second aspect that is relevant to us.
* A crucial point in most open source SW development is the distinction between the maintainer of a piece of SW and a packager (nomenclature my vary, but you get the point). The Haskell platform is all about packaging, nothing else. Trying to force the maintainers to handle bug fix branches just for the platform's sake is a no-go IMHO, and something I personally won't do for my libraries. Let's keep the maintainer-packager distinction as it is!
Yes, it's about packaging. Packaging people are intermediaries between maintainers and users and they have to respond to the needs of both. Of course they cannot force anyone to do anything.
* Let's assume that the Haskell platform keeps its planned interval of 6 months for major releases (something I am not totally convinced of until it has happened reliably a few times). If we allowed API additions only twice a year, this would be ridiculously slow, because the world outside doesn't stand still. The OpenGL ARB e.g. released their OpenGL 3.1 spec only 9 months after the OpenGL 3.0 spec, and the differences were by no means small. You can't compare e.g. the network package with this, because improvements or additions in network APIs are virtually non-existent compared to this, we basically still use APIs which are a few decades old. Consequently, allowing API additions only every 6 months would be more than enough for the network package, but it would be far too slow for the OpenGL package.
Sure. But of course there is no problem with making releases on hackage and users who need OpenGL 3.1 in the few months between major HP releases could use cabal to get the latest from Hackage. The question is how often we ask all end users and distributors to update everything? For an active package the maintainer might like to put out a release every week and some subset of users will be quite happy to grab that. But how often do they want to update everything else? I bet it is more on the order of 4-6 months (and some users will prefer longer and skip updates). The platform is certainly not about arresting progress and development. It's about helping end users and developers to manage change by compressing changes into regular, well defined points -- major releases -- and in between providing a little consistency and stability. Duncan

Duncan Coutts wrote:
Sure. But of course there is no problem with making releases on hackage and users who need OpenGL 3.1 in the few months between major HP releases could use cabal to get the latest from Hackage.
I think this raises an interesting question: the newly updated (for example,)opengl package might strive to work with the existing release Haskell Platform, rather than require newer versions of other HP libraries also. Of course, cabal-install will deal with the latter too, it just leads to a few more dependency problems (& stability/testedness problems?). -Isaac

On Tue, May 12, 2009 at 10:05:55AM -0400, Isaac Dupree wrote:
I think this raises an interesting question: the newly updated (for example,)opengl package might strive to work with the existing release Haskell Platform, rather than require newer versions of other HP libraries also. Of course, cabal-install will deal with the latter too, it just leads to a few more dependency problems (& stability/testedness problems?).
I guess HP will depend on exact minor versions of the libraries. Does that mean that the new shiny OpenGL package can't depend on the old HP because it would indirectly depend on an older version of itself? Are we encouraged to depend on HP at all, or should we always target the packages themselves and leave HP on Hackage just for completeness? -- Felipe.

On Tue, 2009-05-12 at 18:21 -0300, Felipe Lessa wrote:
On Tue, May 12, 2009 at 10:05:55AM -0400, Isaac Dupree wrote:
I think this raises an interesting question: the newly updated (for example,)opengl package might strive to work with the existing release Haskell Platform, rather than require newer versions of other HP libraries also. Of course, cabal-install will deal with the latter too, it just leads to a few more dependency problems (& stability/testedness problems?).
I guess HP will depend on exact minor versions of the libraries. Does that mean that the new shiny OpenGL package can't depend on the old HP because it would indirectly depend on an older version of itself?
Are we encouraged to depend on HP at all, or should we always target the packages themselves and leave HP on Hackage just for completeness?
Nothing should depend on a meta-package like the haskell-platform package. The Linux distro community have plenty of experience with this and this is their policy. It overly constrains everything. In any case, the platform meta-package exports nothing so you'd have to specify dependencies on the things you use anyway. It'd be redundant except to constrain the versions of dependencies, which is exactly why people should not do it (we might make hackage enforce this if it becomes a problem). Duncan

Am Montag, 11. Mai 2009 10:40:27 schrieb Duncan Coutts:
On Sun, 2009-05-10 at 12:05 +0200, Sven Panne wrote:
[...] * Consistency in numbering: The platform numbering should have the semantics as the numbering of its parts, therefore allowing API additions in minor releases. If someone is totally unwilling to risk any changes his code, he can still use bug fix releases of the platform instead of minor releases, where's the problem?
We can make the numbering consistent with whichever policy we choose. [...]
I don't think so: http://www.haskell.org/haskellwiki/Package_versioning_policy clearly states our numbering policy for libraries, and if you regard the API of the HP as the union of the API of its constituent libraries, there is no choice at all if the HP wants to conform to the PvP. And I can't see a reason why the HP should be considered a special case here. Cheers, S.

On Wed, May 13, 2009 at 7:16 PM, Sven Panne
Am Montag, 11. Mai 2009 10:40:27 schrieb Duncan Coutts:
On Sun, 2009-05-10 at 12:05 +0200, Sven Panne wrote:
[...] * Consistency in numbering: The platform numbering should have the semantics as the numbering of its parts, therefore allowing API additions in minor releases. If someone is totally unwilling to risk any changes his code, he can still use bug fix releases of the platform instead of minor releases, where's the problem?
We can make the numbering consistent with whichever policy we choose. [...]
I don't think so: http://www.haskell.org/haskellwiki/Package_versioning_policy clearly states our numbering policy for libraries, and if you regard the API of the HP as the union of the API of its constituent libraries, there is no choice at all if the HP wants to conform to the PvP. And I can't see a reason why the HP should be considered a special case here.
I think Duncan meant that we first choose the policy and then adjust the version numbering scheme of HP to match. Cheers, Johan

Am Mittwoch, 13. Mai 2009 19:28:45 schrieb Johan Tibell:
I think Duncan meant that we first choose the policy and then adjust the version numbering scheme of HP to match.
Hmmm, perhaps I'm a bit slow today, but I don't understand this: What the PvP on the Wiki describes is more or less numbering folklore, with the only exception that we use two places for the major number instead of one, which is more common. Following the principle of least surprise, I therefore consider the PvP as "carved in stone". The HP is just another package, so what can be adjusted? Cheers, S.

On Wed, 2009-05-13 at 20:15 +0200, Sven Panne wrote:
Am Mittwoch, 13. Mai 2009 19:28:45 schrieb Johan Tibell:
I think Duncan meant that we first choose the policy and then adjust the version numbering scheme of HP to match.
Hmmm, perhaps I'm a bit slow today, but I don't understand this: What the PvP on the Wiki describes is more or less numbering folklore, with the only exception that we use two places for the major number instead of one, which is more common. Following the principle of least surprise, I therefore consider the PvP as "carved in stone". The HP is just another package, so what can be adjusted?
Sorry, I don't think I explained it well (and my other reply had a questionable standard of English). The thing that can be adjusted is the version scheme for the Haskell Platform. The current scheme is described here: http://trac.haskell.org/haskell-platform/wiki/VersionNumbers But this is not set in stone. If we decide that we only want to allow bug fixes in minor releases and that we do want the platform to follow the PVP then we would have to adjust the above scheme, see: http://trac.haskell.org/haskell-platform/wiki/VersionNumbers#Alternatives which says: It has been suggested that the platform version numbers follow the PVP in the sense that the platform is the union of all it's components. In this case, if we agreed that there are no API changes in minor releases then the general scheme would be $year . $major-release . 0 . $minor-version That is, the 3rd digit would always be zero. This would be because major releases typically have some slight incompatible change in at least one package while minor releases do not change any API at all, only fix bugs, so only the 4th digit changes. Duncan

On Wed, 2009-05-13 at 19:16 +0200, Sven Panne wrote:
Am Montag, 11. Mai 2009 10:40:27 schrieb Duncan Coutts:
On Sun, 2009-05-10 at 12:05 +0200, Sven Panne wrote:
[...] * Consistency in numbering: The platform numbering should have the semantics as the numbering of its parts, therefore allowing API additions in minor releases. If someone is totally unwilling to risk any changes his code, he can still use bug fix releases of the platform instead of minor releases, where's the problem?
We can make the numbering consistent with whichever policy we choose. [...]
I don't think so: http://www.haskell.org/haskellwiki/Package_versioning_policy clearly states our numbering policy for libraries, and if you regard the API of the HP as the union of the API of its constituent libraries, there is no choice at all if the HP wants to conform to the PvP. And I can't see a reason why the HP should be considered a special case here.
If we choose to allow compatible API changes in minor release then we'd use version numbers like: X.Y.0 for the first release / major release X.Y.1 for the next minor release If we choose to allow only bug-fix changes changes in minor releases (ie no API changes) then we'd use version numbers like: X.Y.0.0 for the first release / major release X.Y.0.1 for the next minor release In both cases this compiles with the PVP and in both cases the union is consistent with it components. Am I missing something? Yes, the second option is a change from the version number scheme what we've been proposing. We didn't initially think about the PVP when designing it. It is clear that everyone else thinks it is important and we have no objection to changing the scheme so that it does follow the PVP, indeed it's a sensible and consistent thing to do. The only thing causing confusion has been whether we interpret the existing platform versioning proposal in the light of the PVP and then change the platform API policy to suit, or if we decide our policy and then adjust the platform versioning scheme to follow the PVP. The latter approach is clearly more sensible. Duncan

Am Mittwoch, 13. Mai 2009 19:42:12 schrieb Duncan Coutts:
[...] If we choose to allow compatible API changes in minor release then we'd use version numbers like:
X.Y.0 for the first release / major release X.Y.1 for the next minor release
If we choose to allow only bug-fix changes changes in minor releases (ie no API changes) then we'd use version numbers like:
X.Y.0.0 for the first release / major release X.Y.0.1 for the next minor release
In both cases this compiles with the PVP and in both cases the union is consistent with it components.
Am I missing something? [...]
Nope...
The only thing causing confusion has been whether we interpret the existing platform versioning proposal in the light of the PVP and then change the platform API policy to suit, or if we decide our policy and then adjust the platform versioning scheme to follow the PVP. The latter approach is clearly more sensible.
OK, then we agree. To rephrase it: If any constituent part of the HP is upgraded to something more than a bug-fix release, the HP can't be a bug fix release, either. I guess I was confused a bit with the distinction between "platform versioning scheme" and "platform versioning policy". Cheers, S.

On Wed, 2009-05-13 at 20:21 +0200, Sven Panne wrote:
The only thing causing confusion has been whether we interpret the existing platform versioning proposal in the light of the PVP and then change the platform API policy to suit, or if we decide our policy and then adjust the platform versioning scheme to follow the PVP. The latter approach is clearly more sensible.
OK, then we agree. To rephrase it: If any constituent part of the HP is upgraded to something more than a bug-fix release, the HP can't be a bug fix release, either.
Right. Yet another way of putting it is: if we decide the policy is that HP minor releases are bug fix only then we could not include any updates in constituent parts that were not just bug fixes (such updates would have to wait for the next major HP release). Duncan

The Haskell Platform concept calls for major and minor releases.
* The point of major releases is to update the set of common packages we all use. That is, to specify which packages and which versions (corresponding to APIs). The target is to have major releases every 6 months.
* The point of minor releases is to support a major release across a range of operating systems over time and respond to bugs that are discovered in a library or tool subsequent to their release. The target is to have 2-3 minor releases after major releases at intervals on the order of 4-6 weeks.
I suspect this is too simple a picture of what people expect from the HP!-) Some examples: - stability: used to be about language stability alone, but these days, libraries are a big part of the game, language standards are a bit slow in keeping up with language extensions, and GHC releases no longer come with a "standard" kitchensink of extensions and libraries Here, the question is: can I write code/documentation wrt HP x.y.z, and will the combination remain accessible, useable, and relevant? And which stability problems does the HP not address/solve? - accessibility: there are still gaps in Haskell users being able to build Haskell infrastructure from scratch on arbitrary platforms (foreign libraries, autoconf, tool dependencies, inconsistent/late propagation of Haskell software into OS package managers, ..). Here, the question is: for the code someone writes/for the code someone wants to use, is there a HP x.y.z that will suffice to get things started? What are the remaining gaps, and what can be done to close them? - simplicity: there are currently many many different ways to set up and maintain Haskell infrastructure, and they don't necessarily work together (if you use cabal install, your native package manager doesn't know; if you use the native package manager, you may not be able to get the code or versions you need; if you build GHC head yourself, binary releases of GUI libs are no good; if you manage to get everything working and want to make a cross-platform release, you have no idea what difficulties your users may run into trying to set up the same baseline on their platforms; and so on, and so on..). Here, the question is: is the HP sufficient as the one-and-only baseline for Haskell infrastructure, ie, can everyone everywhere start from there, pushing the problems above one level further away, to the non-HP libs and tools? And what are the recommended routes for going beyond this common baseline, in light of the gaps mentioned above? There are probably other usage/intention patterns than these, and I'm not sure that different patterns lead to the same conclusions/preferences when it comes to policy decisions. So I suggest that everyone states the patterns they have in mind when suggesting particular policies, and keeps an eye open towards other patterns that these suggestions might be less suitable for. Even within the currently defined range of HP libs and tools, not everything is the same: as we know from experience with earlier GHC releases, there are core libs (both provided and used internally), and there are extralibs (provided, but not used internally). As the HP grows, one might want to add a dependentlibs category (used internally, but not quite as essential as core libs). And we've got the situation that some tools are also not quite independent (eg, GHC Api clients like Haddock are currently deeply entwined with GHC version and build process). This affects update routes: core libs and tools cannot be updated without updating the whole HP, dependentlibs can only be updated together, and extralibs can be updated. Of course, part of the rationale for the HP is that updating is not always a simple matter of doing 'cabal <DWIM>'.. For non-core libs and tools, it might even be possible to provide multiple internally consistent sets of versions within one HP version, so instead of having branches of HP versions, a single branch of HP releases could provide both stable and latest versions. I guess what I'd like to say is that it might be better to look for a map of the design and usage spaces first, as well as the remaining issues that keep cabal/hackage from being 'the last suit you'll every wear' before delving into policy decisions. Claus <DWIM>: do what I mean;-)

Hi Claus, I've been trying (unsuccessfully) to avoid replying to every single message in this thread, however you raise a number of interesting things that we've not asked ourselves yet in this discussion. On Sun, 2009-05-10 at 23:00 +0100, Claus Reinke wrote:
I suspect this is too simple a picture of what people expect from the HP!-) Some examples:
You're right. We started with much too narrow a question that was laden with too many assumptions. It's right to look at the bigger picture (so long as it helps us come to some sort of conclusion! :-) )
- stability: used to be about language stability alone, but these days, libraries are a big part of the game, language standards are a bit slow in keeping up with language extensions, and GHC releases no longer come with a "standard" kitchensink of extensions and libraries
Here, the question is: can I write code/documentation wrt HP x.y.z, and will the combination remain accessible, useable, and relevant? And which stability problems does the HP not address/solve?
'relevant' is a particularly good word in this context. What does it mean for a release to remain relevant? I'd say that a release becomes *irrelevant* shortly after developers stop testing their packages against that releases. Given that we cannot expect developers to test against more than two or three releases then irrelevance is directly related to the frequency of releases. If we have bi-annual releases then a user could expect to stick with a single release for 12-18 months. On the other hand if we do quarterly releases then the time to irrelevance is significantly reduced. Duncan
participants (15)
-
Axel Simon
-
Bryan O'Sullivan
-
Claus Reinke
-
Denis Bueno
-
Don Stewart
-
Duncan Coutts
-
Felipe Lessa
-
Heinrich Apfelmus
-
Isaac Dupree
-
Johan Tibell
-
Magnus Therning
-
Simon Marlow
-
Sittampalam, Ganesh
-
Sven Panne
-
wren ng thornton