On Wed, Aug 15, 2012 at 3:57 PM, Johan Tibell wrote:

On Wed, Aug 15, 2012 at 12:38 PM, Bryan O'Sullivan wrote:

...

I propose that the sense of the recommendation around upper bounds in the PVP be reversed: upper bounds should be specified only when there is a known problem with a new version of a depended-upon package.

This argument precisely captures my feelings on this subject. I will be removing upper bounds next time I make releases of my packages.

So we are certain that the rounds of failures that led to their being *added* will never happen again? -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

On Wed, Aug 15, 2012 at 1:02 PM, Brandon Allbery wrote:

So we are certain that the rounds of failures that led to their being *added* will never happen again?

It would be useful to have some examples of these. I'm not sure we had any when we wrote the policy (but Duncan would know more), but rather reasoned our way to the current policy by saying that things can theoretically break if we don't have upper bounds, therefore we need them. -- Johan

On Wed, Aug 15, 2012 at 1:50 PM, David Thomas wrote:

Would it make sense to have a known-to-be-stable-though soft upper bound added proactively, and a known-to-break-above hard bound added reactively, so people can loosen gracefully as appropriate?

I don't think so. It adds complexity, but more importantly it's usual for the existing upper bounds to refer to versions that don't exist at the time of writing (and hence can't be known to be stable).

As someone who recurrently is nudging a large number of maintainers every major ghc release to bump their bounds, I favor the no upper bounds approach! :) plus the whole improving ecosystem of build bot tools which play nice with cabal et al that are cropping up mean that "in principal" we could debug missing upper bounds via sort of temporal bisecting over the "event stream" of maximum available versions at a given time to sort that. (but that piece isn't that important) more pragmatically, cabal when used with hackage doesn't let you override version constraints, it just lets you add additional constraints. This makes sense if we assume that the library author is saying "things will definitely break if you violate them", but in practice upper bounds are made up guesstimation. YES, its presumably semantic versioning doesn't create a problem, but with the hackage eco system, when dealing with intelligently engineering libs that are regularly maintained, version upper bounds create more problems than than solve. just my two cents. (yes yes yes, please drop upper bounds!) cheers -Carter On Wed, Aug 15, 2012 at 5:04 PM, Michael Blume wrote:

...

it's usual for the existing upper bounds to refer to versions that don't exist at the time of writing (and hence can't be known to be stable).

Well, known to be stable given semantic versioning, then.

http://semver.org/

On Wed, Aug 15, 2012 at 1:55 PM, Bryan O'Sullivan wrote:

...

On Wed, Aug 15, 2012 at 1:50 PM, David Thomas wrote:

...

Would it make sense to have a known-to-be-stable-though soft upper bound added proactively, and a known-to-break-above hard bound added

reactively,

...

so people can loosen gracefully as appropriate?

I don't think so. It adds complexity, but more importantly it's usual for the existing upper bounds to refer to versions that don't exist at the time of writing (and hence can't be known to be stable). _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Wed, Aug 15, 2012 at 03:54:04PM -0700, Michael Sloan wrote:

Upper bounds are a bit of a catch-22 when it comes to library authors evolving their APIs:

1) If library clients aren't encouraged to specify which version of the exported API they target, then changing APIs can lead to opaque compile errors (without any information about which API is intended). This could lead the client to need to search for the appropriate version of the library.

Having the version number A.B.*, than most packages seem to mostly increase B or lower parts of the version number. If an upper bound is missing, than cabal could use any package in the range A.*.* . If an author wants to make breaking changes to his API, than he could indicate this by increasing A. I've nothing against your proposal, I just don't think that it will be done that soon. Greetings, Daniel

no one is disputing that there are conditional changes in dependencies depending on library versions. an interesting intermediate point would be have a notion of "testing with " constraints in cabal and engineering cabal to support a "--withTestedConstraints" to have a simple composable way of handling constructing build plans. at the end of the day, its an engineering problem coupled with a social factors problem. Those are hard :) On Wed, Aug 15, 2012 at 5:44 PM, Brandon Allbery wrote:

On Wed, Aug 15, 2012 at 4:44 PM, Johan Tibell wrote:

...

On Wed, Aug 15, 2012 at 1:02 PM, Brandon Allbery wrote:

...

So we are certain that the rounds of failures that led to their being *added* will never happen again?

It would be useful to have some examples of these. I'm not sure we had

Upper package versions did not originally exist, and nobody wanted them. You can see the result in at least half the packages on Hackage: upper versions came in when base got broken up, and when bytestring was merged into base --- both of which caused massive breakage that apparently even the people around at the time and involved with it no longer remember.

I'm not going to argue the point though; ignore history and remove them if you desire.

-- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On 16 August 2012 08:55, Brandon Allbery wrote:

On Wed, Aug 15, 2012 at 6:46 PM, Carter Schonwald wrote:

...

no one is disputing that there are conditional changes in dependencies depending on library versions.

Indeed. But the ghc release that split up base broke cabalised packages with no warning to users until they failed to compile. Upper bounds were put in place to avoid that kind of breakage in the future.

There's also the case where people blindly put something like "base < 10" in the .cabal file, and then it broke on the next GHC release. This happend with ghc-core-0.5: it completely failed to build with base-4 (and because cabal-install kept defaulting packages to use base-3 I think a lot of people missed cases like this and blindly thought it worked). I like having upper bounds on version numbers... right up until people abuse them. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com

On Wed, Aug 15, 2012 at 11:02 PM, MightyByte wrote:

be to add a flag to Cabal/cabal-install that would cause it to ignore upper bounds. (Frankly, I think it would also be great if

Ignore, or at least treat them as being like flags... if the versions don't converge with them, start relaxing them and retrying, then print a warning about the versions it slipped and attempt the build. That said, I'd be in favor of moving toward something based on ABI versioning instead; package versions as used by the PVP are basically a manual switch emulating that. It's not generally done because other parts of the toolchain (notably ld's shared object versioning) don't support it, but given that Cabal has greater control over the build process for Haskell programs it would be worth exploring having Cabal deal with it. (This is not something you can do with existing C or C++ toolchains; the smarts would need to be in make.) -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

On Thu, Aug 16, 2012 at 11:33:17PM -0400, wren ng thornton wrote:

However, there are certainly cases where we have hard upper bounds[1][2][3], and ignoring those is not fine. Circumventing hard upper bounds should require altering the .cabal file, given as getting things to compile will require altering the source code as well.

It's ok to have soft and hard upper bounds, but it should be always possible to ignore both - having a separate ignore option for each - without modifying the cabal file. I've the confidence, that most cabal users should be able to handle this. Nothing is more annoying if you know what you're doing, but don't have the power to do it. Greetings, Daniel

I agree with Bryan's proposal unreservedly. However, I think there might be a way to resolve the tension between: * maintaining and publishing a definite dependent-package configuration that is known to work and * having a package unencumbered with arbitrary restrictions on which future versions of its dependent packages it will work with. Can't we have both? The cabal file would only eliminate package version that are known not to work as Bryan suggests but the versions of the dependent packages that the package has been tested with -- the 'reference configuration' -- could be recorded separately. A separate build tool could take the reference configuration and direct cabal to rebuild the reference instance, specifying the fully qualified packages to use. I have a set of tools for doing this because (see http://justhub.org -- and if you have root access to an RPM-based Linux then the chances are you can try it out, otherwise the sources are available). For each project or package, separate from the cabal file recording the hard dependencies, I record the current reference configuration in a file which lists the base installation (generally a Haskell platform but it can be a bare compiler) and the list of fully-qualified dependent modules. Like the cabal file the reference configuration would get checked into the VCS and/or included in the Hackage tarball. A normal build process first builds the environment ensuring that the correct platform is selected and the exact package dependencies are installed -- this is usually just a checking step unless the package environment has been disturbed or the reference configuration has been revised. Once the environment has been checked the normal program build process proceeds as normal, where the real work generally happens. Once the project is checked out on another system (or a package is installed anew) the build step would actually build all of the dependent packages. For this to really work a sandbox mechanism is needed -- merely trying out a package/project shouldn't trash your only development environment! If a library package is to be integrated into a live project the reference environment probably won't be the one you need but I find it useful to be able to build it anyway in a clean environment and incrementally up/downgrade the packages, searching out a compatible configuration. (Being able to easily push and recover sandboxes is helpful here too.) Would this way of working resolve the tension we are seeing here? I am so used to working this way it is difficult for me to say. (As others have said here and elsewhere, functional packaging is really cool.) Chris -----Original Message----- From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of MightyByte Sent: 16 August 2012 04:02 To: Ivan Lazar Miljenovic Cc: Haskell Cafe Subject: Re: [Haskell-cafe] Platform Versioning Policy: upper bounds are not our friends On Wed, Aug 15, 2012 at 9:19 PM, Ivan Lazar Miljenovic wrote:

On 16 August 2012 08:55, Brandon Allbery wrote:

...

Indeed. But the ghc release that split up base broke cabalised packages with no warning to users until they failed to compile. Upper bounds were put in place to avoid that kind of breakage in the future.

I like having upper bounds on version numbers... right up until people abuse them.

I also tend to favor having upper bounds. Obviously they impose a cost, but it's not clear to me at all that getting rid of them is a better tradeoff. I've had projects that I put aside for awhile only to come back and discover that they would no longer build because I hadn't put upper bounds on all my package dependencies. With no upper bounds, a package might not be very likely to break for incremental version bumps, but eventually it *will* break. And when it does it's a huge pain to get it building again. If I have put effort into making a specific version of my package work properly today, I want it to always work properly in the future (assuming that everyone obeys the PVP). I don't think it's unreasonable that some activation energy be required to allow one's project to work with a new version of some upstream dependency. Is that activation energy too high right now? Almost definitely. But that's a tool problem, not a problem with the existence of upper bounds themselves. One tool-based way to help with this problem would be to add a flag to Cabal/cabal-install that would cause it to ignore upper bounds. (Frankly, I think it would also be great if Cabal/cabal-install enforced upper version bounds automatically if none were specified.) Another approach that has been discussed is detecting dependencies that are only used internally[1], and I'm sure there are many other possibilities. In short, I think we should be moving more towards purely functional builds that reduce the chance that external factors will break things, and it seems like removing upper version bounds is a step in the other direction. [1] http://cdsmith.wordpress.com/2011/01/21/a-recap-about-cabal-and-haskell-libr aries/ _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Thu, Aug 16, 2012 at 5:38 AM, Conrad Parker wrote:

...

Hi, folks -

I'm sure we are all familiar with the phrase "cabal dependency hell" at

On 16 August 2012 03:38, Bryan O'Sullivan wrote: this

...

point, as the number of projects on Hackage that are intended to hack around the problem slowly grows.

I am currently undergoing a fresh visit to that unhappy realm, as I try to rebuild some of my packages to see if they work with the GHC 7.6 release candidate.

Likewise ...

...

A substantial number of the difficulties I am encountering are related to packages specifying upper bounds on their dependencies. This is a recurrent problem, and its source lies in the recommendations of the PVP itself (problematic phrase highlighted in bold):

I think part of the problem might be that some packages (like bytestring, transformers?) have had their major version number incremented even despite being backwards-compatible. Perhaps there are incompatible changes, but most of the cabal churn I've seen recently has involved incrementing the bytestring upper bound to <0.11 without requiring any code changes to modules using Data.ByteString.

In general, I've been taking the approach recently that we have two classes of packages: some (like transformers and bytestring) have mostly-stable APIs, and most code I write only relies on those APIs. If I'm just using Data.ByteString for the ByteString type and a few functions like readFile and map, it's highly unlikely that the next version will introduce some breaking change. In those cases, I've been leaving off the upper bound entirely. For other packages that haven't yet stabilized, I've still been keeping the upper bound. In many cases, even that isn't necessary. I've tried removing the upper bounds on those as well, but I almost always end up getting someone filing a bug report that I left off some upper bound and therefore a compile failed. I agree with Bryan's argument, but I'd like to keep consistency for most packages on Hackage. If the community goes in this direction, I'll go along too. Michael

I see good arguments on both sides of the upper bounds debate, though at the current time I think the best solution is to omit upper bounds (and I have done so for most/all of my packages on hackage). But I cannot agree with this enough: On Thu, Aug 16, 2012 at 4:45 AM, Joachim Breitner wrote:

I think what we’d need is a more relaxed policy with modifying a package’s meta data on hackage. What if hackage would allow uploading a new package with the same version number, as long as it is identical up to an extended version range? Then the first person who stumbles over an upper bound that turned out to be too tight can just fix it and upload the fixed package directly, without waiting for the author to react.

I think that constraint ranges of a given package should be able to both be extended and restricted after the fact. Those in favor of the reactionary approach (as I am at the moment, or Bryan O'Sullivan) would find the ability of to restrict the version range useful, while those in favor of the proactive approach (like Joachim Breitner or Doug Beardsley) would find the ability to extend the version range useful. I suspect that attitudes towards upper bounds may well change if we can set version ranges after the fact. I know mine very well might. And the difference between reactionary and proactive approaches I think is a potential justification for the "hard" and "soft" upper bounds; perhaps we should instead call them "reactionary" and "proactive" upper bounds instead.

On Fri, Aug 17, 2012 at 12:34 PM, MigMit wrote:

What if instead of upper (and lower) bounds we just specify our interface requirements?

We already have a simple versioning scheme for which, despite it being easy to grasp, we have amply demonstrated that we cannot make it work well, because it has emergent properties that cause it to not scale well across a large community. Any vastly more complicated and detailed versioning scheme has a huge burden to prove that it won't collapse dramatically more quickly. (Frankly, I think that anything involving "specify every detail of your known dependencies" is dead on arrival from a practical standpoint: it's way too much work.) For that matter, I think that this burden applies to my own proposal to omit upper bounds unless they're needed. Fortunately, we now have several years of historical dependency data that we can go back and mine, and thereby try to model the effects of particular suggested changes.

On Fri, Aug 17, 2012 at 4:35 PM, Bryan O'Sullivan wrote:

Any vastly more complicated and detailed versioning scheme has a huge burden to prove that it won't collapse dramatically more quickly. (Frankly, I think that anything involving "specify every detail of your known dependencies" is dead on arrival from a practical standpoint: it's way too much work.)

If you do it in terms of hashed signatures, you can make the compiler and toolchain do the work for you. The problem with version numbers is that it *is* manual. It can't catch behavioral changes, though; you probably need some kind of epoch override for that in the case where it doesn't come with corresponding API changes. The reason it's never done is that you can't really do it with C or C++. We can mostly avoid that (the FFI is an issue, but it is anyway: cabal can't really check C stuff, unless you use pkg-config which was two operating modes: exact versions or no versioning). -- brandon s allbery allbery.b@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms

On 8/17/12 11:28 AM, Leon Smith wrote:

And the difference between reactionary and proactive approaches I think is a potential justification for the "hard" and "soft" upper bounds; perhaps we should instead call them "reactionary" and "proactive" upper bounds instead.

I disagree. A hard constraint says "this package *will* break if you violate me". A soft constraint says "this package *may* break if you violate me". These are vastly different notions of boundary conditions, and they have nothing to do with a proactive vs reactionary stance towards specifying constraints (of either type). The current problems of always giving (hard) upper bounds, and the previous problems of never giving (soft) upper bounds--- both stem from a failure to distinguish hard from soft! The current/proactive approach fails because the given constraints are interpreted by Cabal as hard constraints, when in truth they are almost always soft constraints. The previous/reactionary approach fails because when the future breaks noone bothered to write down when the last time things were known to work. To evade both problems, one must distinguish these vastly different notions of boundary conditions. Hard constraints are necessary for blacklisting known-bad versions; soft constraints are necessary for whitelisting known-good versions. Having a constraint at all shows where the grey areas are, but it fails to indicate whether that grey is most likely to be black or white. -- Live well, ~wren

As I see it, there are four possibilities for a given version of dependency: 1. The version DOES work. The author (or some delegate) has compiled the package against this version and the resulting code is considered good. 2. The version SHOULD work. No one has tested against this version, but the versioning policy promises not to break anything. 3. The version MIGHT NOT work. No one has tested against this version, and the versioning policy allows breaking changes. 4. The version DOES NOT work. This has been tested and the resulting code (if any) is considered not good. Obviously, cases 1 and 4 can only apply to previously released versions. The PVP requires setting upper bounds in order to distinguish cases 2 and 3 for the sake of future compatibility. Leaving off upper bounds except when incompatibility is known essentially combines cases 2 and 3. So there are two failure modes: I. A version which DOES work is outside the bounds (that is, in case 3). I think eliminating case 3 is too extreme. I like the idea of temporarily overriding upper bounds with a command-line option. The danger here is that we might actually be in case 4, in which case we don't want to override the bounds, but requiring an explicit override gives users a chance to determine if a particular version is disallowed because it is untested or because it is known to be incompatible. II. A version which DOES NOT work is inside the bounds (that is, in case 2). This happens when a package does not follow its own version policy. For example, during the base-4 transition, a version of base-3.0 was released which introduced a few breaking changes (e.g., it split the Arrow class). Alternately, a particular version might be buggy. This can already be handled by adding constraints on the command line, but it's better to release a new version of the package with more restrictive constraints. (This might not be enough, though. If I release foo-1.0.0 which depends on bar-1.0.*, and then bar-1.0.1 is released with with a bug or breaking change, I can release foo-1.0.0.1 which disallows bar-1.0.1. But we need some way of preventing cabal from using foo-1.0.0. Can Hackage deprecate specific versions?) On Wed, Aug 22, 2012 at 9:18 AM, Leon Smith wrote:

I think we actually agree more than we disagree; I do think distinguishing hard and soft upper bounds (no matter what they are called) would help, and I'm just trying to justify them to some of the more dismissive attitudes towards the idea

The only thing I think we (might) disagree on is the relative importance of distinguishing hard and soft bounds versus being able to change bounds easily after the fact (and *without* changing the version number associated with the package.)

And on that count, given the choice, I pick being able to change bounds after the fact, hands down. I believe this is more likely to significantly improve the current situation than distinguishing the two types of bound alone. However, being able to specify both (and change both) after the fact may prove to be even better.

Best, Leon

On Sat, Aug 18, 2012 at 11:52 PM, wren ng thornton wrote:

...

On 8/17/12 11:28 AM, Leon Smith wrote:

...

And the difference between reactionary and proactive approaches I think is a potential justification for the "hard" and "soft" upper bounds; perhaps we should instead call them "reactionary" and "proactive" upper bounds instead.

I disagree. A hard constraint says "this package *will* break if you violate me". A soft constraint says "this package *may* break if you violate me". These are vastly different notions of boundary conditions, and they have nothing to do with a proactive vs reactionary stance towards specifying constraints (of either type).

The current problems of always giving (hard) upper bounds, and the previous problems of never giving (soft) upper bounds--- both stem from a failure to distinguish hard from soft! The current/proactive approach fails because the given constraints are interpreted by Cabal as hard constraints, when in truth they are almost always soft constraints. The previous/reactionary approach fails because when the future breaks noone bothered to write down when the last time things were known to work.

To evade both problems, one must distinguish these vastly different notions of boundary conditions. Hard constraints are necessary for blacklisting known-bad versions; soft constraints are necessary for whitelisting known-good versions. Having a constraint at all shows where the grey areas are, but it fails to indicate whether that grey is most likely to be black or white.

-- Live well, ~wren

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Dave Menendez http://www.eyrie.org/~zednenem/

Just thinking aloud: What if we add "-current" ? pacage -current Would select the versions of the package that were current art the time the cabal file was uploaded and sucessfully compiled in hackage, if the packae is installed from hackage If the cabal file is local then current == any. This option would eliminate the need to guess bounds for package dependencies. It would also give more guaranties that the package will compile sucessfully when downloaded from hackage. Certainly,it would not guarantee it if your version of ghc differs from the one in Hackage, but it would make things more simple and would reduce the spectrum of possible failures 2012/8/24 wren ng thornton

On 8/22/12 12:35 PM, David Menendez wrote:

...

As I see it, there are four possibilities for a given version of dependency:

1. The version DOES work. The author (or some delegate) has compiled the package against this version and the resulting code is considered good. 2. The version SHOULD work. No one has tested against this version, but the versioning policy promises not to break anything. 3. The version MIGHT NOT work. No one has tested against this version, and the versioning policy allows breaking changes. 4. The version DOES NOT work. This has been tested and the resulting code (if any) is considered not good.

Obviously, cases 1 and 4 can only apply to previously released versions. The PVP requires setting upper bounds in order to distinguish cases 2 and 3 for the sake of future compatibility. Leaving off upper bounds except when incompatibility is known essentially combines cases 2 and 3.

Right-o.

So there are two failure modes:

...

I. A version which DOES work is outside the bounds (that is, in case 3). I think eliminating case 3 is too extreme. I like the idea of temporarily overriding upper bounds with a command-line option. The danger here is that we might actually be in case 4, in which case we don't want to override the bounds, but requiring an explicit override gives users a chance to determine if a particular version is disallowed because it is untested or because it is known to be incompatible.

There are two failure modes with overriding stated bounds, however. On the one hand, the code could fail to compile. Okay, we know we're in case 4; all is well. On the other hand the code could successfully compile in ways the package designer knows to be buggy/wrong; we're actually in case 4, but the user does not know this. This is why it's problematic to simply allow overriding constraints. The package developer has some special knowledge that the compiler lacks, but if all constraints are considered equal then the developer has no way to convey that knowledge to the user (i.e., in an automated machine-checkable way). Consequently, the user can end up in a bad place because they thought this second failure mode was actually the success mode.

This is why I advocate distinguishing hard constraints from soft constraints. By making this distinction, the developer has a means of conveying their knowledge to users. A soft bound defines an explicit boundary between case 1 and cases 2--4, which can be automatically (per PVP) extended to an implicit boundary between cases 1--2 and cases 3--4; a boundary which, as you say, can only be truly discovered after the code has been published. Extending soft boundaries in this way should be safe; at least it's as safe as possible with the foresight available to us. On the other hand, a hard bound defines an explicit boundary between case 4 and cases 1--3. If these are overridable, things may break silently as discussed above--- but the important thing is, in virtue of distinguishing hard and soft bounds, the user is made aware of this fact. By distinguishing hard and soft bounds, the developer can convey their special knowledge to the user. The user can ignore this information, but at least they'll do so in an informed way.

-- Live well, ~wren

______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe

-- Alberto.

On 16/08/12 14:07, Chris Smith wrote:

As a package author, when I release a new version, I know perfectly well what incompatible changes I have made to it... and those might include, for example:

1. New modules, exports or instances... low risk 2. Changes to less frequently used, advanced, or "internal" APIs... moderate risk 3. Completely revamped commonly used interfaces... high risk

Would adding a single convenience function be low or high risk? You say it is low risk, but it still risks breaking a build if a user has defined a function with the same name. I think the only meaningful distinction you can make are: 1. No change to public API at all, user code is guaranteed to "compile and work if it did so before". Perhaps new modules could also fall under this category, I'm not sure. 2. changes to exports, instances, modules, types, etc. But with the guarantee that "if it compiles, it will be correct" 3. changes to functionality, which require the user to reconsider all code. "even if it compiles, it might be wrong" For the very common case 2, the best solution is to just go ahead and try to compile it.

A. Cabal files should get a new "Compatibility" field, indicating the level of compatibility from the previous release: low, medium, high, or something like that, with definitions for what each one means.

You would need to indicate how large the change is compared to a certain previous version. "Moderate change compared to 0.10, large change compared to 0.9".

B. Version constraints should get a new syntax:

bytestring ~ 0.10.* (allow later versions that indicate low or moderate risk) bytestring ~~ 0.10.* (allow later versions with low risk; we use the dark corners of this one) bytestring == 0.10.* (depend 100% on 0.10, and allow nothing else)

Of course, this adds a good bit of complexity to the constraint solver... but not really. It's more like a pre-processing pass to replace fuzzy constraints with precise ones.

Perhaps it would be cleaner if you specified what parts of the API you depend on, instead of an arbitrary distinction between 'internal' and 'external' parts. From cabal's point of view the best solution would be to have a separate package for the internals. Then the only remaining distinction is between 'breaking' and 'non-breaking' changes. The current policy is to rely on major version numbers. But this could instead be made explicit: A cabal package should declare what API version of itself it is mostly-compatible with. To avoid forcing the creation of packages just for versioning, perhaps dependencies could be specified on parts of a package? build-depends: bytestring.internal ~< 0.11 and the bytestring package would specify what parts have changed: compatibility: bytestring.internal >= 0.11, bytestring.external >= 0.10 But these names introduce another problem: they will not be fine-grained enough until it is too late. You only know how the API is partitioned when, in the future, a part of it changes while another part does not. Twan

On Thu, Aug 16, 2012 at 10:01 AM, Chris Smith wrote:

Twan van Laarhoven wrote:

...

Would adding a single convenience function be low or high risk? You say it is low risk, but it still risks breaking a build if a user has defined a function with the same name.

Yes, it's generally low-risk, but there is *some* risk. Of course, it could be high risk if you duplicate a Prelude function or a name that you know is in use elsewhere in a related or core library... these decisions would involve knowing something about the library space, which package maintainers often do.

If you import qualified then adding functions will never break anything. Cheers, -- Felipe.

On Sat, Aug 18, 2012 at 3:57 AM, David Feuer wrote:

If the language is changed (without possibility of breakage, I believe) so that names declared in a module shadow imported names, incompatibility can only arise if two different imports offer the same name, and it is actually used.

This already happens in practice (e.g. "take", how many modules declare that?) and is one of the problems that qualified imports solve. Cheers, -- Felipe.

So that we are using concrete examples.  here is an example of a change that really shouldn't break any package: https://github.com/timthelion/threadmanager/commit/c23e19cbe78cc6964f23fdb90 b7029c5ae54dd35 The exposed functions are the same.  The behavior is changed.  But as the commiter of the change, I cannot imagine that it would break any currently working code. There is another issue though.  With this kind of change, there is no reason for a package which was written for the old version of the library, to be built with the new version.  If I am correct, that this change changes nothing for currently working code, then why should an old package be built with the newer package? The advantage in this case, is merely that we want to prevent version duplication.  We don't want to waste disk space by installing every possible iteration of a library. I personally think that disk space is so cheep, that this last consideration is not so important.  If there are packages that only build with old versions of GHC, and old libraries, why can we not just seamlessly install them?  One problem, is if we want to use those old libraries with new code.  Take the example of Python2 vs Python3.  Yes, we can seamlessly install python2 libraries, even though we use python3 normally, but we cannot MIX python2 libraries with python3 libraries. Maybe we could make Haskell linkable objects smart enough that we COULD mix old with new?  That sounds complicated. I think, Michael Sloan is onto something though with his idea of compatibility layers.  I think that if we could write simple "dictionary" packages that would translate old API calls to new ones, we could use old code without modification.  This would allow us to build old libraries which normally wouldn't be compatible with something in base using a base-old-to- new dictionary package.  Then we could use these old libraries without modification with new code. It's important that this be possible from the side of the person USING the library, and not the library author.   It's impossible to write software, if you spend all of your time waiting for someone else to update their libraries. Timothy ---------- Původní zpráva ---------- Od: Ivan Lazar Miljenovic Datum: 16. 8. 2012 Předmět: Re: [Haskell-cafe] Platform Versioning Policy: upper bounds are not our friends "On 16 August 2012 20:50, Ketil Malde wrote:

"Bryan O'Sullivan" writes:

...

I propose that the sense of the recommendation around upper bounds in the PVP be reversed: upper bounds should be specified *only when there is a known problem with a new version* of a depended-upon package.

Another advantage to this is that it's not always clear what constitutes an API change. I had to put an upper bound on binary, since 0.5 introduced laziness changes that broke my program. (I later got some help to implement a workaround, but binary-0.4.4 is still substantially faster). Understandably, the authors didn't see this as a breaking API change.

Except 0.4 -> 0.5 _is_ a major version bump according to the PVP.

So, +1.

-k -- If I haven't seen further, it is by standing in the footprints of giants

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

(http://www.haskell.org/mailman/listinfo/haskell-cafe) -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com(http://IvanMiljenovic.wordpress.com) _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe (http://www.haskell.org/mailman/listinfo/haskell-cafe)"

On Thu, Aug 16, 2012 at 05:30:07PM +0300, Yitzchak Gale wrote:

For actively maintained packages, I think the problem is that package maintainers don't find out promptly that an upper bound needs to be bumped. One way to solve that would be a simple bot that notifies the package maintainer as soon as an upper bound becomes out-of-date.

This already exists: http://packdeps.haskellers.com/ -Brent

My thoughts on the matter got a little long, so I posted them here: http://nominolo.blogspot.co.uk/2012/08/beyond-package-version-policies.html On 17 August 2012 12:48, Heinrich Apfelmus wrote:

Brent Yorgey wrote:

...

Yitzchak Gale wrote:

...

For actively maintained packages, I think the problem is that package maintainers don't find out promptly that an upper bound needs to be bumped. One way to solve that would be a simple bot that notifies the package maintainer as soon as an upper bound becomes out-of-date.

This already exists:

http://packdeps.haskellers.**com/ http://packdeps.haskellers.com/

Indeed. It even has RSS feeds, like this

http://packdeps.haskellers.**com/feed/reactive-bananahttp://packdeps.haskellers.com/feed/reactive-banana

Extremely useful!

Best regards, Heinrich Apfelmus

-- http://apfelmus.nfshost.com

______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe

-- Push the envelope. Watch it bend.

Hello, I also completely agree with Bryan's point which is why I usually don't add upper bounds on the dependencies of the packages that I maintain---I find that the large majority of updates to libraries tend to be backward compatible, so being optimistic seems like a good idea. By the way, something I encounter quite often is a situation where two packages both build on Hacakge just fine, but are not compatible with each other. Usually it goes like this: 1. Package A requires library X >= V (typically, because it needs a bug fix or a new feature). 2. Package B requires library X < V (typically, because someone added a conservative upper bound that needs to be updated). Trying to use A and B together leads to failure, which is usually resolved by having to install B manually, and remove it's upper bound by hand. This is rather unfortunate, because not only it's inconvenient but, also, now there is no released version of package B that you can explicitly depend on. -Iavor On Mon, Aug 20, 2012 at 7:11 AM, Brent Yorgey wrote:

...

On Wed, Aug 15, 2012 at 3:38 PM, Bryan O'Sullivan wrote:

...

I propose that the sense of the recommendation around upper bounds in

On Thu, Aug 16, 2012 at 06:07:06PM -0400, Joey Adams wrote: the

...

...

PVP be reversed: upper bounds should be specified only when there is a known problem with a new version of a depended-upon package.

I, too, agree. Here is my assortment of thoughts on the matter.

Here's some bad news: with cabal 1.14 (released with Haskell Platform 2012.2), cabal init defaults to bounds like these:

build-depends: base ==4.5.*, bytestring ==0.9.*, http-types ==0.6.*

I'm not sure why you think this is "bad news". I designed this to conform exactly to the current PVP. If the PVP is changed then I will update cabal init to match.

-Brent

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

I think we should encourage stable build environments to know precisely which package versions they have been using and to keep using them until told otherwise. Even when the types and constraints all work out there is a risk that upgraded packages will break. Everybody here wants cabal to just install the packages without problem, but if you want to insulate yourself from package upgrades surely sticking with proven combinations is the way to go. Chris -----Original Message----- From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Erik Hesselink Sent: 20 August 2012 08:33 To: Bryan O'Sullivan Cc: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Platform Versioning Policy: upper bounds are not our friends I am strongly against this, especially for packages in the platform. If you fail to specify an upper bound, and I depend on your package, your dependencies can break my package! For example, say I develop executable A and I depend on library B == 1.0. Library B depends on library C >= 0.5 (no upper bound). Now C 0.6 is released, which is incompatible with B. This suddenly breaks my build, even though I have not changed anything about my code or dependencies. This goes against the 'robust' aspect mentioned as one of the properties of the Haskell platform, and against the Haskell philosophy of correctness in general. This is not an imaginary problem. At my company, we've run into these problems numerous times already. Since we also have people who are not experts at Cabal and the Haskell ecosystem building our software, this can be very annoying. The fix is also not trivial: we can add a dependency on a package we don't use to all our executables or we can fork the library (B, in the example above) and add an upper bound/fix the code. Both add a lot of complexity that we don't want. Add to that the build failures and associated emails from CI systems like Jenkins. I can see the maintenance burder you have, since we have to do the same for our code. But until some Cabal feature is added to ignore upper bounds or specify soft upper bounds, please follow the PVP, also in this regard. It helps us maintain a situation where only our own actions can break our software. Erik On Wed, Aug 15, 2012 at 9:38 PM, Bryan O'Sullivan wrote:

Hi, folks -

I'm sure we are all familiar with the phrase "cabal dependency hell" at this point, as the number of projects on Hackage that are intended to hack around the problem slowly grows.

I am currently undergoing a fresh visit to that unhappy realm, as I try to rebuild some of my packages to see if they work with the GHC 7.6 release candidate.

A substantial number of the difficulties I am encountering are related to packages specifying upper bounds on their dependencies. This is a recurrent problem, and its source lies in the recommendations of the PVP itself (problematic phrase highlighted in bold):

...

When publishing a Cabal package, you should ensure that your dependencies in the build-depends field are accurate. This means specifying not only lower bounds, but also upper bounds on every dependency.

I understand that the intention behind requiring tight upper bounds was good, but in practice this has worked out terribly, leading to depsolver failures that prevent a package from being installed, when everything goes smoothly with the upper bounds relaxed. The default response has been for a flurry of small updates to packages in which the upper bounds are loosened, thus guaranteeing that the problem will recur in a year or less. This is neither sensible, fun, nor sustainable.

In practice, when an author bumps a version of a depended-upon package, the changes are almost always either benign, or will lead to compilation failure in the depending-upon package. A benign change will obviously have no visible effect, while a compilation failure is actually better than a depsolver failure, because it's more informative.

This leaves the nasty-but-in-my-experience-rare case of runtime failures caused by semantic changes. In these instances, a downstream package should reactively add an upper bound once a problem is discovered.

I propose that the sense of the recommendation around upper bounds in the PVP be reversed: upper bounds should be specified only when there is a known problem with a new version of a depended-upon package.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Of course if you wish or need to upgrade a package then you can just upgrade it -- I am not suggesting anyone should forgo upgrades! It is just that there is no need to make the stability of a build process dependent on new package releases. To upgrade a package I would fork my sandbox, hack away at the package database (removing, upgrading, installing packages) until I have a candidate combination of packages and swap in the new sandbox into my work tree and test it, reverting to the tried and tested environment if things don't work out. If the new configuration works then then I would dump the new package configuration and check it in. Subsequent updates and builds on other work trees should pick up the new environment. The key thing I was looking for was control of when your build environment gets disrupted and stability in between -- even when building from the repo. Chris -----Original Message----- From: Erik Hesselink [mailto:hesselink@gmail.com] Sent: 20 August 2012 14:35 To: Chris Dornan Cc: Bryan O'Sullivan; haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Platform Versioning Policy: upper bounds are not our friends Hub looks interesting, I'll have to try it out (though I'm not on an RPM based distro). But isn't this the goal of things like semantic versioning [0] and the PVP? To know that you can safely upgrade to a bugfix release, and relavily safely to a minor release, but on a major release, you have to take care? Haskell makes it much easier to see if you can use a new major (or minor) version of a library, since the type checker catches many (but not all!) problems for you. However, this leads to libraries breaking their API's much more easily, and that in turn causes the problems voiced in this thread. However, fixing all versions seems like a bit of a blunt instrument, as it means I'll have to do a lot of work to bring even bug fixes in. Erik [0] http://semver.org/ On Mon, Aug 20, 2012 at 3:13 PM, Chris Dornan wrote:

I think we should encourage stable build environments to know precisely which package versions they have been using and to keep using them until told otherwise. Even when the types and constraints all work out there is a risk that upgraded packages will break. Everybody here wants cabal to just install the packages without problem, but if you want to insulate yourself from package upgrades surely sticking with proven combinations is the way to go.

Chris

-----Original Message----- From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Erik Hesselink Sent: 20 August 2012 08:33 To: Bryan O'Sullivan Cc: haskell-cafe@haskell.org Subject: Re: [Haskell-cafe] Platform Versioning Policy: upper bounds are not our friends

I am strongly against this, especially for packages in the platform.

If you fail to specify an upper bound, and I depend on your package, your dependencies can break my package! For example, say I develop executable A and I depend on library B == 1.0. Library B depends on library C >= 0.5 (no upper bound). Now C 0.6 is released, which is incompatible with B. This suddenly breaks my build, even though I have not changed anything about my code or dependencies. This goes against the 'robust' aspect mentioned as one of the properties of the Haskell platform, and against the Haskell philosophy of correctness in general.

This is not an imaginary problem. At my company, we've run into these problems numerous times already. Since we also have people who are not experts at Cabal and the Haskell ecosystem building our software, this can be very annoying. The fix is also not trivial: we can add a dependency on a package we don't use to all our executables or we can fork the library (B, in the example above) and add an upper bound/fix the code. Both add a lot of complexity that we don't want. Add to that the build failures and associated emails from CI systems like Jenkins.

I can see the maintenance burder you have, since we have to do the same for our code. But until some Cabal feature is added to ignore upper bounds or specify soft upper bounds, please follow the PVP, also in this regard. It helps us maintain a situation where only our own actions can break our software.

Erik

On Wed, Aug 15, 2012 at 9:38 PM, Bryan O'Sullivan wrote:

...

Hi, folks -

I'm sure we are all familiar with the phrase "cabal dependency hell" at this point, as the number of projects on Hackage that are intended to hack around the problem slowly grows.

I am currently undergoing a fresh visit to that unhappy realm, as I try to rebuild some of my packages to see if they work with the GHC 7.6 release candidate.

A substantial number of the difficulties I am encountering are related to packages specifying upper bounds on their dependencies. This is a recurrent problem, and its source lies in the recommendations of the PVP itself (problematic phrase highlighted in bold):

...

When publishing a Cabal package, you should ensure that your dependencies in the build-depends field are accurate. This means specifying not only lower bounds, but also upper bounds on every dependency.

I understand that the intention behind requiring tight upper bounds was good, but in practice this has worked out terribly, leading to depsolver failures that prevent a package from being installed, when everything goes smoothly with the upper bounds relaxed. The default response has been for a flurry of small updates to packages in which the upper bounds are loosened, thus guaranteeing that the problem will recur in a year or less. This is neither sensible, fun, nor sustainable.

In practice, when an author bumps a version of a depended-upon package, the changes are almost always either benign, or will lead to compilation failure in the depending-upon package. A benign change will obviously have no visible effect, while a compilation failure is actually better than a depsolver failure, because it's more informative.

This leaves the nasty-but-in-my-experience-rare case of runtime failures caused by semantic changes. In these instances, a downstream package should reactively add an upper bound once a problem is discovered.

I propose that the sense of the recommendation around upper bounds in the PVP be reversed: upper bounds should be specified only when there is a known problem with a new version of a depended-upon package.

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe