`bitZero` sounds like it should equal `bit 0`, which it doesn't. zeroBits ? -Edward On Sun, Feb 16, 2014 at 11:42 AM, Herbert Valerio Riedel wrote:

Hi,

On 2014-02-16 at 15:10:42 +0100, ARJANEN Loïc Jean David wrote:

...

I'll have to come down against that proposal because, at least on amd64 for 64 bits-sized types (Int, Int64, Word & Word64), it doesn't works.

You're right, I don't know how I could have missed that :-/

Since the presumed pre-condition for the proposal (that 'bit (-1) == 0' would already hold) I hereby amend the proposal to

...

Introduce a new class method

class Bits a where ... -- | Value with all bits cleared bitZero :: a ...

modulo naming of 'bitZero'

(I'm hesitant to consume "zero" from the namespace as was suggested by Henning) _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

Am 16.02.2014 17:51, schrieb Eric Mertens:

The last time this topic came up (back when Num was split off) the discussion was derailed by the idea of a Bool class and then forgotten. I've hoped for an explicit zero element since then.

I think it was this discussion: https://www.haskell.org/pipermail/libraries/2011-October/016919.html

I like those laws insofar as they go. It'd be nice to sit down and specify the rest of the Data.Bits laws some day. (Not volunteering!) -Edward On Wed, Feb 26, 2014 at 8:30 PM, wren ng thornton wrote:

I'm +1 for the following slightly extended proposal:

* add zeroBits * with default implementation: clearBit (bit 0) 0 * and requiring the following laws for all n valid for the type: * clearBit zeroBits n == zeroBits * setBit zeroBits n == bit n * testBit zeroBits n == False * popCount zeroBits == 0

-- Live well, ~wren _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

+1 to `zeroBits`. The name fits in with `setBit` etc. I'm also okay with the name `zero`. Twan On 16/02/14 17:45, Edward Kmett wrote:

`bitZero` sounds like it should equal `bit 0`, which it doesn't.

zeroBits ?

-Edward

On Sun, Feb 16, 2014 at 11:42 AM, Herbert Valerio Riedel mailto:hvr@gnu.org> wrote:

Hi,

On 2014-02-16 at 15:10:42 +0100, ARJANEN Loïc Jean David wrote: > I'll have to come down against that proposal because, at least on amd64 for 64 > bits-sized types (Int, Int64, Word & Word64), it doesn't works.

You're right, I don't know how I could have missed that :-/

Since the presumed pre-condition for the proposal (that 'bit (-1) == 0' would already hold) I hereby amend the proposal to

> Introduce a new class method > > class Bits a where > ... > -- | Value with all bits cleared > bitZero :: a > ... > > modulo naming of 'bitZero'

(I'm hesitant to consume "zero" from the namespace as was suggested by Henning) _______________________________________________ Libraries mailing list Libraries@haskell.org mailto:Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

Le dimanche 16 février 2014, 17:49:23 Henning Thielemann a écrit :

...

...

Introduce a new class method

class Bits a where

... -- | Value with all bits cleared bitZero :: a ...

modulo naming of 'bitZero'

(I'm hesitant to consume "zero" from the namespace as was suggested by Henning) Let me defend "zero": Functions in the module that contain "Bit" in the name (clearBit, setBit, testBit) access a single bit. All functions that access many bits do not contain "Bit" (rotate, shift, xor, popCount). I

Am 16.02.2014 17:42, schrieb Herbert Valerio Riedel: propose to import the "zero" function with qualification, as well as the other functions from the module.

I also propose to provide a default implementation of "zero", like "clearBit (bit 0) 0". This way, the "zero" method can be introduced without breaking existing code.

I'm in favour of that modified proposal, so +1 for a "zero" member of Bits. And if we fear problems with collisions, we could always name the member something like zeroValue or zeroBits.

Hello *, Here's a mid-discussion summary of the proposal

...

Introduce a new class method

class Bits a where ... -- | Value with all bits cleared <0-value-method> :: a ...

modulo naming of '<0-value-method>'

from my point of view: - The idea came up already in 2011 when Num-superclass was to be removed from Bits (but discussion derailed) - So far general consensus (i.e. no "-1"s afaics) it's desired to have an all-bits-clear value introducing method in 'Bits' - Use "clearBit (bit 0) 0" as default implementation for smooth upgrade-path - Nameing for <0-value-method> boils down to two candidates: a) 'Data.Bits.zero' - based on the idea tha 'Data.Bits' ought to be imported qualified (or with explicit import-list) anyway (-> thus following PVP practice) - many existing Data.Bits.Bits methods such as 'rotate', 'complement', 'popCount', 'xor', or 'shift' don't have the name 'bit' in it (and those few that have, operate on single bits) - supporters (in no particular order): - ARJANEN Loïc Jean David - Henning Thielemann - Herbert Valerio Riedel (+0.99) - Twan van Laarhoven b) 'Data.Bits.zeroBits' - more verbose name reduces risk of namespace conflicts with unqualified imports - supporters (in no particular order): - Edward Kmett - Eric Mertens - Herbert Valerio Riedel - Twan van Laarhoven - (maybe?) ARJANEN Loïc Jean David So far there doesn't seem to be a very clear preference for 'zeroBits' over 'zero'. It might help, if those how expressed some kind of support for both variants could clarify if their preference has any bias towards 'zeroBits' or 'zero'. Cheers, hvr

Am 22.02.2014 15:56, schrieb Carter Schonwald:

Zero is a big bit of name space. Id favor zerobits over zero.

... only if you plan to add another "zero" thing to the Data.Bits module. Conflicts with other modules must be resolved with importing mechanisms, but not by finding names that are different from all other identifiers in all other modules in all other packages. It just makes no sense to have a nice module system, but use it like the #include directive of the C preprocessor and eventually maybe complain about the deficiencies of the current module system.

I am notably strongly against taking 'zero' as it is much more appropriately used by more algebraic classes, and it is an annoyingly common name to take for such an often unqualified import. -Edward On Sat, Feb 22, 2014 at 5:03 AM, Herbert Valerio Riedel wrote:

Hello *,

Here's a mid-discussion summary of the proposal

...

...

Introduce a new class method

class Bits a where ... -- | Value with all bits cleared <0-value-method> :: a ...

modulo naming of '<0-value-method>'

from my point of view:

- The idea came up already in 2011 when Num-superclass was to be removed from Bits (but discussion derailed)

- So far general consensus (i.e. no "-1"s afaics) it's desired to have an all-bits-clear value introducing method in 'Bits'

- Use "clearBit (bit 0) 0" as default implementation for smooth upgrade-path

- Nameing for <0-value-method> boils down to two candidates:

a) 'Data.Bits.zero'

- based on the idea tha 'Data.Bits' ought to be imported qualified (or with explicit import-list) anyway (-> thus following PVP practice)

- many existing Data.Bits.Bits methods such as 'rotate', 'complement', 'popCount', 'xor', or 'shift' don't have the name 'bit' in it (and those few that have, operate on single bits)

- supporters (in no particular order):

- ARJANEN Loïc Jean David - Henning Thielemann - Herbert Valerio Riedel (+0.99) - Twan van Laarhoven

b) 'Data.Bits.zeroBits'

- more verbose name reduces risk of namespace conflicts with unqualified imports

- supporters (in no particular order):

- Edward Kmett - Eric Mertens - Herbert Valerio Riedel - Twan van Laarhoven - (maybe?) ARJANEN Loïc Jean David

So far there doesn't seem to be a very clear preference for 'zeroBits' over 'zero'. It might help, if those how expressed some kind of support for both variants could clarify if their preference has any bias towards 'zeroBits' or 'zero'.

Cheers, hvr _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

Am 24.02.2014 18:34, schrieb Twan van Laarhoven:

I agree with Edward that it would be better to reserve 'zero' for something like an additive identity.

Why would you want to reserve Bits.zero for an additive zero? This makes no sense.

I'll withdraw my +1 for the name `zero`, in favor of +1 for `zeroBits`.

What is bad about Bits.zero?

Am 24.02.2014 18:57, schrieb Brandon Allbery:

On Mon, Feb 24, 2014 at 12:41 PM, Henning Thielemann mailto:schlepptop@henning-thielemann.de> wrote:

Am 24.02.2014 18 tel:24.02.2014%2018:34, schrieb Twan van Laarhoven:

I agree with Edward that it would be better to reserve 'zero' for something like an additive identity.

Why would you want to reserve Bits.zero for an additive zero? This makes no sense.

There is something vaguely smelly about specifically omitting the context

...

it is an annoyingly common name to take for such an often unqualified import.

in the original message. Yes, we're quite aware you do not consider it legitimate. Distorting someone else's meaning to press your point is also not legitimate.

The phrase "reserve 'zero'" suggests that once we choose Bits.zero, the identifier 'zero' is reserved once and for all and cannot be used for something different anymore. That is, this phrasing removes the option of qualified imports from the scope and thus generates the wrong context. Can someone please, please tell me why we must avoid qualified imports at all costs? Why is this option repeatedly ignored when just saying zeroBits (+1) or zero (-1)?

On Mon, Feb 24, 2014 at 11:56 AM, Edward Kmett wrote:

Henning,

As far as I know the only serious proponent of using qualified imports for all imports all the time is you.

Well, so am I, but I'm not the crusading sort. Unqualified import as default is definitely the dominant style, to the point where language extensions tend to assume it, e.g. record puns, or even the whole shared record names debate. Interestingly, python and java seem to lean the other way. Not sure about ocaml, but I remember qualified names from back in the day. A culture thing, I guess. Infix operators and backticks are uniquely haskelly things that probably contribute a little.

Am 24.02.2014 21:30, schrieb Henning Thielemann:

For an example let me look at your lens package. You use unqualified and implicit imports. That is according to the PVP you would need to use tight version bounds like "containers >=0.4.0 && <0.5.0.1", but you don't.

Sorry, it must be "containers >=0.4.0 && <0.5.1", but it is "containers

=0.4.0 && <0.6".

$ check-pvp . This requires that the package is configured, since only then the association of packages to modules is known. If you want to run the tool on a non-configured package you may just check all imports for addition-proof style. . $ check-pvp --include-all . It follows a detailed description of the procedure and the rationale behind it. . First the program classifies all dependencies in the Cabal file of the package. You can show all classifications with the @--classify-dependencies@

import Data.Map as Map . Additions to @Data.Map@ may clash with other identifiers,

import qualified Data.Map as Map . or . import Data.Map (Map) . The program may complain about an open list of constructors as in . import Data.Sequence (ViewL(..)) . Additions of constructors to @ViewL@ may also conflict with other identifiers. You must instead import like . import Data.Sequence (ViewL(EmptyL, (:<))) . or . import qualified Data.Sequence as Seq . The program emits an error on clashing module abbreviations like . import qualified Data.Map.Lazy as Map import qualified Data.Map.Strict as Map . This error is raised whenever multiple modules are imported with the same abbreviation, where at least one module is open for additions. Our test is overly strict in the sense that it also blames . import qualified Data.Map as Map import qualified Data.Map as Map . but I think it is good idea to avoid redundant imports anyway. . Additionally you can enable a test for hiding imports with the @--pedantic@ option. The import . import Data.Map hiding (insert) . is not bad in the sense of the PVP, but this way you depend on the existence of the identifier @insert@ although you do not need it. If it is removed in a later version of @containers@,

I got distracted by writing a tool that checks consistency of package dependencies and import styles. It's not yet perfect, but usable: https://hackage.haskell.org/package/check-pvp Description: Check whether the version ranges used in the @Build-Depends@ field matches the style of module imports according to the Package Versioning Policy (PVP). See http://www.haskell.org/haskellwiki/Package_versioning_policy. The tool essentially looks for any dependency like @containers >=0.5 && <0.6@ that allows the addition of identifiers to modules within the version range. Then it checks whether all module imports from @containers@ are protected against name clashes that could be caused by addition of identifiers. . You must run the tool in a directory containing a Cabal package. . option, otherwise only problematic dependencies are shown. . A dependency like @containers >=0.5.0.3 && <0.5.1@ does not allow changes of the API of @containers@ and thus the program does not check its imports. Clashing import abbreviations are an exception. . The dependency @containers >=0.5.1 && <0.6@ requires more care when importing modules from @containers@ and this is what the program is going to check next. This is the main purpose of the program! I warmly recommend this kind of dependency range since it greatly reduces the work to keep your package going together with its imported packages. . Dependencies like @containers >=0.5@ or @containers >=0.5 && <1@ are always problematic, since within the specified version ranges identifier can disappear. There is no import style that protects against removed identifiers. . An inclusive upper bound as in @containers >=0.5 && <=0.6@ will also cause a warning, because it is unnecessarily strict. If you know that @containers-0.6@ works for you, then @containers-0.6.0.1@ or @containers-0.6.1@ will also work, depending on your import style. A special case of inclusive upper bounds are specific versions like in @containers ==0.6@. The argument for the warning remains the same. . Please note that the check of ranges is performed entirely on the package description. The program will not inspect the imported module contents. E.g. if you depend on @containers >=0.5 && <0.6@ but import in a way that risks name clashes, then you may just extend the dependency to @containers >=0.5 && <0.6.1@ in order to let the checker fall silent. If you use the dependency @containers >=0.5 && <0.6.1@ then the checker expects that you have verified that your package works with all versions of kind @0.5.x@ and the version @0.6.0@. Other versions would then work, too, due to the constraints imposed by package versioning policy. . Let us now look at imports that must be protected against identifier additions. . The program may complain about a lax import. This means you have imported like . thus you must import either . then your import breaks although you did not use the identifier. . Finally you can control what items are checked. First of all you can select the imports that are checked. Normally the imports are checked that belong to lax dependencies like @containers >=0.5 && <0.6@. However this requires the package to be configured in order to know which import belongs to which dependency. E.g. @Data.Map@ belongs to @containers@. You can just check all imports for being addition-proof using the @--include-all@ option. Following you can write the options @--include-import@, @--exclude-import@, @--include-dependency@, @--exclude-dependency@ that allow to additionally check or ignore imports from certain modules or packages. These modifiers are applied from left to right. E.g. @--exclude-import=Prelude@ will accept any import style for @Prelude@ and @--exclude-dependency=foobar@ will ignore the package @foobar@, say, because it does not conform to the PVP. . Secondly, you may ignore certain modules or components of the package using the options @--exclude-module@, @--exclude-library@, @--exclude-executables@, @--exclude-testsuites@, @--exclude-benchmarks@. E.g. @--exclude-module=Paths_PKG@ will exclude the Paths module that is generated by Cabal. I assume that it will always be free of name clashes. . Known problems: . * The program cannot automatically filter out the @Paths@ module. . * The program cannot find and check preprocessed modules. . * The program may yield wrong results in the presence of Cabal conditions. . If this program proves to be useful it might eventually be integrated in the @check@ command of @cabal-install@. See https://github.com/haskell/cabal/issues/1703. . Alternative: If you want to allow exclusively large version ranges, i.e. @>=x.y && https://ghc.haskell.org/trac/ghc/ticket/4977. Unfortunately there is no GHC warning on clashing module abbreviations. See https://ghc.haskell.org/trac/ghc/ticket/4980.

I second that sentiment of appreciation. As much as we all disagree on some things, we all agree you make a tremendous amount of neat code available to the community. On Friday, February 28, 2014, Edward Kmett wrote:

I just want to say thank you for writing this tool.

I may not agree with your interpretation of the PVP on this particular issue, and will probably only use it on a couple of smaller packages that have almost no imports, but at least putting code out there to help those who do want to work in that style is a very helpful and valuable thing.

-Edward

On Fri, Feb 28, 2014 at 3:10 PM, Henning Thielemann < schlepptop@henning-thielemann.de> wrote:

I got distracted by writing a tool that checks consistency of package dependencies and import styles. It's not yet perfect, but usable:

https://hackage.haskell.org/package/check-pvp

...

$ check-pvp . This requires that the package is configured, since only then the association of packages to modules is known. If you want to run the tool on a non-configured package you may just check all imports for addition-proof style. . $ check-pvp --include-all . It follows a detailed description of the procedure and the rationale behind it. . First the program classifies all dependencies in the Cabal file of the package. You can show all classifications with the @--classify-dependencies@option, otherwise only problematic dependencies are shown. . A dependency like @containers >=0.5.0.3 && <0.5.1@ does not allow changes of the API of @containers@ and thus the program does not check its imports. Clashing import abbreviations are an exception. . The dependency @containers >=0.5.1 && <0.6@ requires more care when importing modules from @containers@ and this is what the program is going to check next. This is the main purpose of the program! I warmly recommend this kind of dependency range since it greatly reduces the work to keep your package going together with its imported packages. . Dependencies like @containers >=0.5@ or @containers >=0.5 && <1@ are always problematic, since within the specified version ranges identifier can disappear. There is no import style that protects against removed identifiers. . An inclusive upper bound as in @containers >=0.5 && <=0.6@ will also cause a warning, because it is unnecessarily strict. If you know that @containers-0.6@ works for you,

...

import Data.Map as Map . Additions to @Data.Map@ may clash with other identifiers,

Description: Check whether the version ranges used in the @Build-Depends@ field matches the style of module imports according to the Package Versioning Policy (PVP). See http://www.haskell.org/haskellwiki/Package_versioning_policy. The tool essentially looks for any dependency like @containers >=0.5 && <0.6@ that allows the addition of identifiers to modules within the version range. Then it checks whether all module imports from @containers@ are protected against name clashes that could be caused by addition of identifiers. . You must run the tool in a directory containing a Cabal package. . then @containers-0.6.0.1@ or @containers-0.6.1@ will also work, depending on your import style. A special case of inclusive upper bounds are specific versions like in @containers ==0.6@. The argument for the warning remains the same. . Please note that the check of ranges is performed entirely on the package description. The program will not inspect the imported module contents. E.g. if you depend on @containers >=0.5 && <0.6@ but import in a way that risks name clashes, then you may just extend the dependency to @containers >=0.5 && <0.6.1@ in order to let the checker fall silent. If you use the dependency @containers >=0.5 && <0.6.1@ then the checker expects that you have verified that your package works with all versions of kind @0.5.x@ and the version @0.6.0@. Other versions would then work, too, due to the constraints imposed by package versioning policy. . Let us now look at imports that must be protected against identifier additions. . The program may complain about a lax import. This means you have imported like . thus you must import either .

...

import qualified Data.Map as Map . or . import Data.Map (Map) . The program may complain about an open list of constructors as in . import Data.Sequence (ViewL(..)) . Additions of constructors to @ViewL@ may also conflict with other identifiers. You must instead import like . import Data.Sequence (ViewL(Empt

On Mon, Feb 24, 2014 at 10:30 PM, Henning Thielemann < schlepptop@henning-thielemann.de> wrote:

Am 24.02.2014 20:56, schrieb Edward Kmett:

As far as I know the only serious proponent of using qualified imports

...

for all imports /all/ the time is you.

First, there are some important packages like containers and bytestring that clearly are intended for qualified imports, and I really like that insertFM was replaced by Map.insert in the past. Second, your argument addresses a person not the issue.

Name conflicts don't affect you. We get that. We got that loud and clear

...

virtually every time the naming of pretty much anything has arisen on this mailing list for the last few years.

Sure, because there is no general discussion about the pros and cons of various naming styles. That's why it is discussed for every single identifier. How should a general discussion look like? What could be its outcome? That I am no longer allowed to propose the qualified import style?

That doesn't change the fact that your practice and common practice

...

diverge.

And I challenge the common practice, because it prefers convenience for few package authors over convenience for many package user (that may not even be a Haskell programmer).

For an example let me look at your lens package. You use unqualified and implicit imports. That is according to the PVP you would need to use tight version bounds like "containers >=0.4.0 && <0.5.0.1", but you don't. That is, your package does not conform to the PVP. It implies that users may have to fix your package when one of the imported packages starts to export identifiers that clash with those from "lens". Of course, there is no need to conform to the PVP. But it works best if many people adhere to it.

I have not written the PVP, that is, there must be at least one other person who cares. I understand the need for it and try to comply to it. Maybe I am in a minority. Looking at Hackage it seems I am in a minority. But I believe I am in the minority who cares whether packages work together. Shall I capitulate to the majority which does not seem to care about package interoperability?

I am also ok if sloppy common practice happens in many Hackage packages. I do not need to use them. But I need to use 'base'.

This email seems to conflate a number of different issues together. I'd like to address them separately. Firstly, regarding naming style. As I see it, there are essentially three camps on this one: 1. Short names that are intended to be imported qualified. Examples: Data.Map, Data.ByteString. 2. Longer names that can be imported unqualified. Examples: Data.IORef, Control.Concurrent.MVar. 3. Typeclass-based approaches that generalize across multiple libraries. Examples: Data.Foldable, Data.Traversable. The initial discussion came down to an argument between (1) and (2). What I disagree in your email Henning is the implication that this "sloppy practice" in base will somehow negatively affect you. As I see it, the sum total of negative effect is that you'll be required to type a few extra characters (e.g., zeroBits instead of zero). Am I missing something? Given that (a) a huge amount of base already works this way, (b) having the longer name will allow the unqualified import approach, and (c) it has a small impact on those wanting unqualified imports, I'd come down with a strong vote in favor of option (2). Next is the issue of PVP. I am someone who has stopped religiously following the PVP in the past few years. Your email seems to imply that only those following the PVP care about making sure that "packages work together." I disagree here; I don't use the PVP specifically because I care about package interoperability. The point of the PVP is to ensure that code builds. It's a purely compile-time concept. The PVP solves the problem of an update to a dependency causing a downstream package to break. And assuming everyone adheres to it[1], it ensures that cabal will never try to perform a build which isn't guaranteed to work. But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former. My point here is: please don't try to frame this argument as "sloppy people who hate compatibility" versus "PVP-adhering people who make Hackage better." Some of us who have stopped following the PVP have done so for very principled reasons, even if you disagree with them. (And Edward's comments on maintenance effort is not to be ignored either.) Three final points: * I know that back in the base 3/4 transition there was good reason for upper bounds on base. Today, it makes almost no sense: it simply prevents cabal from even *trying* to perform a compilation. Same goes with libraries like array and template-haskell, which make up most of the issue with testing of GHC 7.8 Stackage builds[3]. Can any PVP proponent explain why these upper bounds still help us on corelibs? * If you're concerned about your production code base breaking by changes on Hackage, you're doing it wrong. Basing your entire production build on the presumption that Hackage maintainers perfectly follow the PVP and never make any mistakes in new releases of their packages is a recipe for disaster. You should be pinning down the exact versions of packages you depend on. Greg Weber described a technique for this[4]. * The PVP doesn't in any way solve all problems. You can perfectly adhere to the PVP and still experience breakage. I've seen a number of examples of this in the past, mostly to do with the fact that you don't lock down the versions of transitive dependencies, which can cause re-exports to include new functions or expose (or hide) typeclass instances. Michael [1] And never makes any mistakes of course. [2] This just occurred in Stackage. [3] https://github.com/fpco/stackage/issues/128 [4] http://blog.docmunch.com/blog/2013/haskell-version-lockdown

On 2014-02-25 at 07:44:45 +0100, Michael Snoyman wrote: [...]

* I know that back in the base 3/4 transition there was good reason for upper bounds on base. Today, it makes almost no sense: it simply prevents cabal from even *trying* to perform a compilation. Same goes with libraries like array and template-haskell, which make up most of the issue with testing of GHC 7.8 Stackage builds[3]. Can any PVP proponent explain why these upper bounds still help us on corelibs?

I assume by 'corelibs' you mean the set of non-upgradeble libs, i.e. those tied to the compiler version? (E.g. `bytestring` would be upgradeable, as opposed to `base` or `template-haskell`) Well, `base` (together with the other few non-upgradeable libs) is indeed a special case; also, in `base` usually care is taken to avoid semantic changes (not visible at the type-signature level), so an upper bound doesn't gain that much in terms of protecting against semantic breakages. Otoh, the situation changes if you have a library where you have different versions, which are tied to different version ranges of base, where you want Cabal to select the matching version. Admittedly, this is a special case for when use of MIN_VERSION_base() wouldn't suffice, but I wanted to give an example exploiting upper-bounds on the `base` lib. There's one other possible minor benefit I can think of, that upper bounds give over compile-errors, which is a more user-friendly message, to point to the reason of the failure, instead of requiring you guess what the actual cause of the compile-error was. But for non-upgradeable packages such as `base`, which do big major version jumps for almost every release (mostly due to changes in GHC modules exposing internals or adding type-class instances[1]), erring on the confusing-compile-error side seems to provide more value. So, as for `base` I mostly agree, that there seems to be little benefit for upper bounds, *unless* a base3/4 situation comes up again in the future. So, I'd suggest (for those who don't want to follow PVP with `base`) to keep using at least a "super"-major upper bound, such as 'base < 5' to leave a door open for such an eventuality. Cheers, hvr [1]: I'd argue (but I'd need research this, to back this up with numbers), that we're often suffering from the PVP, because it requires us to perform major-version jumps mostly due to typeclasses, in order to protect against conflicts with possible non-hideable orphan-instances; and that (as some have suggested in past already), we might want to reconsider requiring only a minor bump on instance-additions, and discourage the orphan-instance business by requiring those packages to have tighter-than-major upper-bounds

On Tue, Feb 25, 2014 at 1:44 AM, Michael Snoyman wrote:

But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former.

I have a question for you. Is it better to save a developer some work, or is it better to force that work onto end users? Because we keep constantly seeing examples where saving the developer some upper bounds PVP work forces users to deal with unexpected errors, but since Haskell developers don't see that user pain it is considered irrelevant/nonexistent and certainly not any justification for saving developers some work. Personally, I think any ecosystem which strongly prefers pushing versioning pain points onto end users instead of developers is doing itself a severe disservice. Are there things that could be improved about versioning policy? Absolutely. But pushing problems onto end users is not an improvement. -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

Hi Brandon, On Tue, Feb 25, 2014 at 10:12:29AM -0500, Brandon Allbery wrote:

Is it better to save a developer some work, or is it better to force that work onto end users?

What is an end user? Someone installing a package containing an executable? Then the package is an end point in the dependency graph and the PVP can work pretty well for this case. But if the package contains a library, then the end user is also the developer, so you can only choose which kind of pain you prefer.

Because we keep constantly seeing examples where saving the developer some upper bounds PVP work forces users to deal with unexpected errors, but since Haskell developers don't see that user pain it is considered irrelevant/ nonexistent and certainly not any justification for saving developers some work.

I think that in most cases it doesn't really make much difference for the end user, if they're seeing a package version mismatch or if they're seeing a compile error. Sure, the package version mismatch is more telling, but in most cases he will be equally lost and has to ask for help. Greetings, Daniel

On Tue, Feb 25, 2014 at 5:12 PM, Brandon Allbery wrote:

On Tue, Feb 25, 2014 at 1:44 AM, Michael Snoyman wrote:

...

But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former.

I have a question for you.

Is it better to save a developer some work, or is it better to force that work onto end users?

Because we keep constantly seeing examples where saving the developer some upper bounds PVP work forces users to deal with unexpected errors, but since Haskell developers don't see that user pain it is considered irrelevant/nonexistent and certainly not any justification for saving developers some work.

Personally, I think any ecosystem which strongly prefers pushing versioning pain points onto end users instead of developers is doing itself a severe disservice.

Are there things that could be improved about versioning policy? Absolutely. But pushing problems onto end users is not an improvement.

I think it's a false dichotomy. I've received plenty of complaints from users about being unable to install newer versions of some dependency because a library that Yesod depends on has an unnecessary strict upper bound. Are there situations where the PVP saves a user some pain? Yes. Are there situations where the PVP causes a user some pain? Yes. It's disingenuous to frame this as a black and white "developer vs user" issue, it's far more complex than that. After a lot of experience, I believe the PVP- or at least strict adherence to it- is a net loss. And I think the *real* solution is something like Stackage, where curators have taken care of the versioning pain points instead of either developers or end users. Linux distributions have been doing this for a long time. Michael

On Tue, Feb 25, 2014 at 6:38 PM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

This thread is kinda missing an important point.

Namely that on hackage now, the admins and trustees have the power to edit the cabal files to fix broken constraints. (As do maintainers of their own packages)

Whether relaxing incorrectly conservative or over strengthening overly lax constraints, this now doesn't require a rerelease to "fix". There are valid reasons of provinence for why that might not make sense in all case

Unless I'm missing the point, doesn't that solve most of the matter? -carter

The question would still remain: who's responsible for making those changes, and what is the default position for the version bounds? We could default to leaving version bounds off, and add them after the fact as necessary. This would reduce developer and Hackage maintainer overhead, but some users may get the "scary" error messages[1]. Or we could default to the PVP approach, and then increase the work on developers/maintainers, with the flip side that (1) users will never get the "scary" error messages, and (2) until developers/maintainers make the change, users may be blocked from even attempting to compile packages together. There's also the issue that, currently, Hackage2 has turned off developer abilities to change version bounds, so all of the version tweaking onus would fall to admins and trustees. Overall, I don't see this as a big improvement over the PVP status quo. It's not any harder for me to upload version 1.0.2.1 of a package with a tweaked version bound than to go to the Hackage web interface and manually edit version 1.0.2's cabal file. What I see the editing feature as very useful for is if we want to add upper bounds after the fact. Michael [1] Which I still think have value, since they are far more informative to a package author.

This is currently disabled if you go to try it. On Tue, Feb 25, 2014 at 11:38 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

This thread is kinda missing an important point.

Namely that on hackage now, the admins and trustees have the power to edit the cabal files to fix broken constraints. (As do maintainers of their own packages)

Whether relaxing incorrectly conservative or over strengthening overly lax constraints, this now doesn't require a rerelease to "fix". There are valid reasons of provinence for why that might not make sense in all case

Unless I'm missing the point, doesn't that solve most of the matter? -carter

On Tuesday, February 25, 2014, Michael Snoyman wrote:

...

On Tue, Feb 25, 2014 at 5:12 PM, Brandon Allbery wrote:

...

On Tue, Feb 25, 2014 at 1:44 AM, Michael Snoyman wrote:

...

But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former.

I have a question for you.

Is it better to save a developer some work, or is it better to force that work onto end users?

Because we keep constantly seeing examples where saving the developer some upper bounds PVP work forces users to deal with unexpected errors, but since Haskell developers don't see that user pain it is considered irrelevant/nonexistent and certainly not any justification for saving developers some work.

Personally, I think any ecosystem which strongly prefers pushing versioning pain points onto end users instead of developers is doing itself a severe disservice.

Are there things that could be improved about versioning policy? Absolutely. But pushing problems onto end users is not an improvement.

I think it's a false dichotomy. I've received plenty of complaints from users about being unable to install newer versions of some dependency because a library that Yesod depends on has an unnecessary strict upper bound. Are there situations where the PVP saves a user some pain? Yes. Are there situations where the PVP causes a user some pain? Yes.

It's disingenuous to frame this as a black and white "developer vs user" issue, it's far more complex than that. After a lot of experience, I believe the PVP- or at least strict adherence to it- is a net loss.

And I think the *real* solution is something like Stackage, where curators have taken care of the versioning pain points instead of either developers or end users. Linux distributions have been doing this for a long time.

Michael

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

indeed. So lets think about how to add module types or some approximation thereof to GHC? (seriously, thats the only sane "best solution" i can think of, but its not something that can be done casually). Theres also the fact that any module system design will have to explicitly deal with type class instances in a more explicit fashion than we've done thus far. On Tue, Feb 25, 2014 at 12:17 PM, Vincent Hanquez wrote:

On 2014-02-25 15:12, Brandon Allbery wrote:

On Tue, Feb 25, 2014 at 1:44 AM, Michael Snoyman

...

michael@snoyman.com>> wrote:

But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former.

I have a question for you.

Is it better to save a developer some work, or is it better to force that work onto end users?

As a *user* of many libraries, I had more problems with libraries that follow the PvP religiously than the other way around. I usually like to have the latest and greatest libraries, specially text, aeson, and such, and there I have to manually bump dependencies of packages I depend on, until the developers gets to update the package on hackage (which sometimes takes many weeks).

As a *developer*, following the PvP would cost me a lot of my *free* time. This is particularly true when the surface of contact with a library is small, it's very unlikely that I will run into an API changes. When I do, I release a new package quickly that account for the API change, or I can put a upper bounds if I can't make the necessary changes quickly enough. I usually found out quite quickly with stackage nowadays, most of times, before any users get bitten.

Some other time, I'm testing some development ghc or some new unreleased libraries, and I need to remove upper bounds from packages so that I can test something.

Anyway, there's lots of reason that the PvP doesn't works fully. It solves some problems for sure, but sadly swipe all the other problems under the carpet. One problem being that a single set of numbers doesn't properly account for API complexity and stability that might differ in different modules of the same package.

-- Vincent

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On 2014-02-25 18:26, Carter Schonwald wrote:

indeed.

So lets think about how to add module types or some approximation thereof to GHC? (seriously, thats the only sane "best solution" i can think of, but its not something that can be done casually). Theres also the fact that any module system design will have to explicitly deal with type class instances in a more explicit fashion than we've done thus far.

This may be relevant: http://plv.mpi-sws.org/backpack/ Regards,

Bart, backpack is just a first design iteration. We shouldn't regard it as the final answer, but as a starting point for understanding / exploring the design space. On Wednesday, February 26, 2014, Bart Massey wrote:

Bardur: Yes! I want *this*! In addition to short-circuiting this epic argument, it looks way better. I've always wished Haskell had the ML module system, but this looks even better than that in some ways. So, yes. Let's get backpack into GHC, require it on Hackage, and get on with it. Note that backpack by itself isn't sufficient, since it only guarantees type-compatibility, not semantic compatibility. We would have to add additional rules requiring at least partial semantic compatibility of any changes to the semantics at a given name. --Bart

...

On 2014-02-25 18:26, Carter Schonwald wrote:

...

indeed.

So lets think about how to add module types or some approximation

On Tue, Feb 25, 2014 at 10:10 PM, Bardur Arantsson javascript:;> wrote: thereof

...

...

to GHC? (seriously, thats the only sane "best solution" i can think of, but its not something that can be done casually). Theres also the fact that any module system design will have to explicitly deal with type class instances in a more explicit fashion than we've done thus far.

This may be relevant:

http://plv.mpi-sws.org/backpack/

Regards,

_______________________________________________ Libraries mailing list Libraries@haskell.org javascript:; http://www.haskell.org/mailman/listinfo/libraries

---

Libraries mailing list Libraries@haskell.org javascript:; http://www.haskell.org/mailman/listinfo/libraries

On Tue, Feb 25, 2014 at 11:23:44AM -0800, Gregory Collins wrote:

Someone released a tool to attempt to do this a couple of days ago --- I haven't tried it yet but surely with a bit of group effort we can improve these tools so that they really fast and easy to use.

That's an amazing tool ... ;)

Of course, people who want to follow PVP are also going to need tooling to make sure their programs still build in the future because so many people have broken the policy in the past -- that's where proposed kludges like "cabal freeze" are going to come in.

If I understood it correctly, then cabal >1.19 supports the option '--allow-newer' to be able to ignore upper bounds, which might solve several of the issues here, so upper bounds could be set but still ignored if desired. Greetings, Daniel

On Tue, Feb 25, 2014 at 9:23 PM, Gregory Collins wrote:

On Mon, Feb 24, 2014 at 10:44 PM, Michael Snoyman wrote:

...

But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former.

That's because you maintain a lot of packages, and you're considering buildability on short time frames (i.e. you mostly care about "does all the latest stuff build right now?"). The consequences of violating the PVP are that as a piece of code ages, the probability that it still builds goes to *zero*, even if you go and dig out the old GHC version that you were using at the time. I find this really unacceptable, and believe that people who are choosing not to be compliant with the policy are BREAKING HACKAGE and making life harder for everyone by trading convenience now for guaranteed pain later. In fact, in my opinion the server ought to be machine-checking PVP compliance and refusing to accept packages that don't obey the policy.

Like Ed said, this is pretty cut and dried: we have a policy, you're choosing not to follow it, you're not in compliance, you're breaking stuff. We can have a discussion about changing the policy (and this has definitely been discussed to death before), but I don't think your side has the required consensus/votes needed to change the policy. As such, I really wish that you would reconsider your stance here.

I really don't like this appeal to authority. I don't know who the "royal we" is that you are referring to here, and I don't accept the premise that the rest of us must simply adhere to a policy because "it was decided." "My side" as you refer to it is giving concrete negative consequences to the PVP. I'd expect "your side" to respond in kind, not simply assert that we're "breaking Hackage" and other such hyperbole. Now, I think I understand what you're alluding to. Assuming I understand you correctly, I think you're advocating irresponsible development. I have codebases which I maintain and which use older versions of packages. I know others who do the same. The rule for this is simple: if your development process only works by assuming third parties to adhere to some rules you've established, you're in for a world of hurt. You're correct: if everyone rigidly followed the PVP, *and* no one every made any mistakes, *and* the PVP solved all concerns, then you could get away with the development practices you're talking about. But that's not the real world. In the real world: * The PVP itself does *not* guarantee reliable builds in all cases. If a transitive dependency introduces new exports, or provides new typeclass instances, a fully PVP-compliant stack can be broken. (If anyone doubts this claim, let me know, I can spell out the details. This has come up in practice.) * People make mistakes. I've been bitten by people making breaking changes in point releases by mistake. If the only way your build will succeed is by assuming no one will ever mess up, you're in trouble. * Just because your code *builds*, doesn't mean your code *works*. Semantics can change: bugs can be introduced, bugs that you depended upon can be resolved, performance characteristics can change in breaking ways, etc. I absolutely believe that, if you want to have code that builds reliably, you have to specify all of your deep dependencies. That's what I do for any production software, and it's what I recommend to anyone who will listen to me. Trying to push this off as a responsibility of every Hackage package author is (1) shifting the burden to the wrong place, and (2) irresponsible, since some maintainer out in the rest of the world has no obligation to make sure your code keeps working. That's your responsibility.

I've long maintained that the solution to this issue should be tooling. The dependency graph that you stipulate in your cabal file should be a *warrant* that "this package is known to be compatible with these versions of these packages". If a new major version of package "foo" comes out, a bumper tool should be able to try relaxing the dependency and seeing if your package still builds, bumping your version number accordingly based on the PVP rules. Someone released a tool to attempt to do this a couple of days ago --- I haven't tried it yet but surely with a bit of group effort we can improve these tools so that they really fast and easy to use.

Of course, people who want to follow PVP are also going to need tooling to make sure their programs still build in the future because so many people have broken the policy in the past -- that's where proposed kludges like "cabal freeze" are going to come in.

This is where we apparently fundamentally disagree. cabal freeze IMO is not at all a kludge. It's the only sane approach to reliable builds. If I ran my test suite against foo version 1.0.1, performed manual testing on 1.0.1, did my load balancing against 1.0.1, I don't want some hotfix build to automatically get upgraded to version 1.0.2, based on the assumption that foo's author didn't break anything. Michael

On Tue, Feb 25, 2014 at 12:38 PM, Michael Snoyman wrote:

On Tue, Feb 25, 2014 at 9:23 PM, Gregory Collins wrote:

...

Like Ed said, this is pretty cut and dried: we have a policy, you're choosing not to follow it, you're not in compliance, you're breaking stuff. We can have a discussion about changing the policy (and this has definitely been discussed to death before), but I don't think your side has the required consensus/votes needed to change the policy. As such, I really wish that you would reconsider your stance here.

I really don't like this appeal to authority. I don't know who the "royal we" is that you are referring to here, and I don't accept the premise that the rest of us must simply adhere to a policy because "it was decided." "My side" as you refer to it is giving concrete negative consequences to the PVP. I'd expect "your side" to respond in kind, not simply assert that we're "breaking Hackage" and other such hyperbole.

This is not an appeal to authority, it's an appeal to consensus. The community comes together to work on lots of different projects like Hackage and the platform and we have established procedures and policies (like the PVP and the Hackage platform process) to manage this. I think the following facts are uncontroversial: - a Hackage package versioning policy exists and has been published in a known location - we don't have another one - you're violating it Now you're right to argue that the PVP as currently constituted causes problems, i.e. "I can't upgrade to new-shiny-2.0 quickly enough" and "I manage 200 packages and you're driving me insane". And new major base versions cause a month of churn before everything goes green again. Everyone understands this. But the solution is either to vote to change the policy or to write tooling to make your life less insane, not just to ignore it, because the situation this creates (programs bitrot and become unbuildable over time at 100% probability) is really disappointing. Now, I think I understand what you're alluding to. Assuming I understand

you correctly, I think you're advocating irresponsible development. I have codebases which I maintain and which use older versions of packages. I know others who do the same. The rule for this is simple: if your development process only works by assuming third parties to adhere to some rules you've established, you're in for a world of hurt. You're correct: if everyone rigidly followed the PVP, *and* no one every made any mistakes, *and* the PVP solved all concerns, then you could get away with the development practices you're talking about.

There's a strawman in there -- in an ideal world PVP violations would be rare and would be considered bugs. Also, if it were up to me we'd be machine-checking PVP compliance. I don't know what you're talking about re: "irresponsible development". In the scenario I'm talking about, my program depends on "foo-1.2", "foo-1.2" depends on any version of "bar", and then when "bar-2.0" is released "foo-1.2" stops building and there's no way to fix this besides trial and error because the solver doesn't have enough information to do its work (and it's been lied to!!!). The only practical solutions right now are to: - commit to maintaining every program you've ever written on the hackage upgrade treadmill forever, or - write down the exact versions of all of the libraries you need in the transitive closure of the dependency graph. #2 is best practice for repeatable builds anyways and you're right that cabal freeze will help here, but it doesn't help much for all the programs written before "cabal freeze" comes out. But that's not the real world. In the real world:

* The PVP itself does *not* guarantee reliable builds in all cases. If a transitive dependency introduces new exports, or provides new typeclass instances, a fully PVP-compliant stack can be broken. (If anyone doubts this claim, let me know, I can spell out the details. This has come up in practice.)

Of course. But compute the probability of this occurring (rare) vs the probability of breakage given no upper bounds (100% as t -> ∞). Think about what you're saying semantically when you say you depend only on "foo > 3": "foo version 4.0 *or any later version*". You can't own up to this contract. * Just because your code *builds*, doesn't mean your code *works*.

Semantics can change: bugs can be introduced, bugs that you depended upon can be resolved, performance characteristics can change in breaking ways, etc.

I think you're making my point for me -- given that this paragraph you wrote is 100% correct, it makes sense for cabal not to try to build against the new version of a dependency until the package maintainer has checked that things still work and given the solver the go-ahead by bumping the package upper bound. This is where we apparently fundamentally disagree. cabal freeze IMO is not

at all a kludge. It's the only sane approach to reliable builds. If I ran my test suite against foo version 1.0.1, performed manual testing on 1.0.1, did my load balancing against 1.0.1, I don't want some hotfix build to automatically get upgraded to version 1.0.2, based on the assumption that foo's author didn't break anything.

This wouldn't be an assumption, Michael -- the tool should run the build and the test suites. We'd bump version on green tests. G -- Gregory Collins

On Tue, Feb 25, 2014 at 11:52 PM, Gregory Collins wrote:

On Tue, Feb 25, 2014 at 12:38 PM, Michael Snoyman wrote:

...

On Tue, Feb 25, 2014 at 9:23 PM, Gregory Collins

...

wrote:

...

Like Ed said, this is pretty cut and dried: we have a policy, you're choosing not to follow it, you're not in compliance, you're breaking stuff. We can have a discussion about changing the policy (and this has definitely been discussed to death before), but I don't think your side has the required consensus/votes needed to change the policy. As such, I really wish that you would reconsider your stance here.

I really don't like this appeal to authority. I don't know who the "royal we" is that you are referring to here, and I don't accept the premise that the rest of us must simply adhere to a policy because "it was decided." "My side" as you refer to it is giving concrete negative consequences to the PVP. I'd expect "your side" to respond in kind, not simply assert that we're "breaking Hackage" and other such hyperbole.

This is not an appeal to authority, it's an appeal to consensus. The community comes together to work on lots of different projects like Hackage and the platform and we have established procedures and policies (like the PVP and the Hackage platform process) to manage this. I think the following facts are uncontroversial:

- a Hackage package versioning policy exists and has been published in a known location - we don't have another one - you're violating it

Now you're right to argue that the PVP as currently constituted causes problems, i.e. "I can't upgrade to new-shiny-2.0 quickly enough" and "I manage 200 packages and you're driving me insane". And new major base versions cause a month of churn before everything goes green again. Everyone understands this. But the solution is either to vote to change the policy or to write tooling to make your life less insane, not just to ignore it, because the situation this creates (programs bitrot and become unbuildable over time at 100% probability) is really disappointing.

You talk about voting on the policy as if that's the natural thing to do. When did we vote to accept the policy in the first place? I don't remember ever putting my name down as "I agree, this makes sense." Talking about voting, violating, complying, etc, in a completely open system like Hackage, makes no sense, and is why your comments come off as an appeal to authority. If you want to have more rigid rules on what packages can be included, start a downstream, PVP-only Hackage, and don't allow in violating packages. If it takes off, and users have demonstrated that they care very much about PVP compliance, then us PVP naysayers will have hard evidence that our beliefs were mistaken. Right now, it's just a few people constantly accusing us of violations and insisting we spend a lot more work on a policy we believe to be flawed.

Now, I think I understand what you're alluding to. Assuming I understand

...

you correctly, I think you're advocating irresponsible development. I have codebases which I maintain and which use older versions of packages. I know others who do the same. The rule for this is simple: if your development process only works by assuming third parties to adhere to some rules you've established, you're in for a world of hurt. You're correct: if everyone rigidly followed the PVP, *and* no one every made any mistakes, *and* the PVP solved all concerns, then you could get away with the development practices you're talking about.

There's a strawman in there -- in an ideal world PVP violations would be rare and would be considered bugs.

Then you're missing my point completely. You're advocating making package management policy based on developer practices of not pinning down deep dependencies. My point is that *bugs happen*. And as I keep saying, it's not just build-time bugs: runtime bugs are possible and far worse. I see no reason that package authors should go through lots of effort to encourage bad practice.

Also, if it were up to me we'd be machine-checking PVP compliance. I don't know what you're talking about re: "irresponsible development". In the scenario I'm talking about, my program depends on "foo-1.2", "foo-1.2" depends on any version of "bar", and then when "bar-2.0" is released "foo-1.2" stops building and there's no way to fix this besides trial and error because the solver doesn't have enough information to do its work (and it's been lied to!!!). The only practical solutions right now are to:

- commit to maintaining every program you've ever written on the hackage upgrade treadmill forever, or - write down the exact versions of all of the libraries you need in the transitive closure of the dependency graph.

#2 is best practice for repeatable builds anyways and you're right that cabal freeze will help here, but it doesn't help much for all the programs written before "cabal freeze" comes out.

Playing the time machine game is silly. Older programs are broken. End of story. If we all agree to start using the PVP now, it won't fix broken programs. If we release "cabal freeze" now, it won't fix broken programs. But releasing "cabal freeze" *will* prevent this problem from happening in the future.

But that's not the real world. In the real world:

...

* The PVP itself does *not* guarantee reliable builds in all cases. If a transitive dependency introduces new exports, or provides new typeclass instances, a fully PVP-compliant stack can be broken. (If anyone doubts this claim, let me know, I can spell out the details. This has come up in practice.)

Of course. But compute the probability of this occurring (rare) vs the probability of breakage given no upper bounds (100% as t -> ∞). Think about what you're saying semantically when you say you depend only on "foo > 3": "foo version 4.0 *or any later version*". You can't own up to this contract.

That's because you're defining the build-depends to mean "I guarantee this to be the case." I could just as easily argue that `foo < 4` is also a lie: how do you know that it *won't* build? This argument has been had many times, please stop trying to make it seem like a clear-cut argument.

* Just because your code *builds*, doesn't mean your code *works*.

...

Semantics can change: bugs can be introduced, bugs that you depended upon can be resolved, performance characteristics can change in breaking ways, etc.

I think you're making my point for me -- given that this paragraph you wrote is 100% correct, it makes sense for cabal not to try to build against the new version of a dependency until the package maintainer has checked that things still work and given the solver the go-ahead by bumping the package upper bound.

Again, you're missing it. If there's a point release, PVP-based code will automatically start using that new point release. That's simply not good practice for a production system.

This is where we apparently fundamentally disagree. cabal freeze IMO is

...

not at all a kludge. It's the only sane approach to reliable builds. If I ran my test suite against foo version 1.0.1, performed manual testing on 1.0.1, did my load balancing against 1.0.1, I don't want some hotfix build to automatically get upgraded to version 1.0.2, based on the assumption that foo's author didn't break anything.

This wouldn't be an assumption, Michael -- the tool should run the build and the test suites. We'd bump version on green tests.

Maybe you write perfect code every time. But I've seen this process many times in the past: * Work on version 2 of an application. * Create a staging build of version 2. * Run automated tests on version 2. * QA manually tests version 2. * Release version 2. * Three weeks later, discover a bug. * Write a hotfix, deploy to staging, run automated tests, QA the changed code, and ship. In these circumstances, it would be terrible if my build system automatically accepted a new point release of a package on Hackage because the PVP says it's OK. Yes, we should all have 100% test coverage, with automated testing that covers all functionality of the product, and every single release would have full test coverage. But we all know that's not the real world. Letting a build system throw variables into an equation is irresponsible. Michael

On 2014-02-25 at 21:38:38 +0100, Michael Snoyman wrote: [...]

* The PVP itself does *not* guarantee reliable builds in all cases. If a transitive dependency introduces new exports, or provides new typeclass instances, a fully PVP-compliant stack can be broken. (If anyone doubts this claim, let me know, I can spell out the details. This has come up in practice.)

...are you simply referring to the fact that in order to guarantee PVP-semantics of a package version, one has to take care to restrict the version bounds of that package's build-deps in such a way, that any API entities leaking from its (direct) build-deps (e.g. typeclass instances or other re-exported entities) are not a function of the "internal" degree of freedoms the build-dep version-ranges provide? Or is there more to it?

On 2014-02-26 at 06:45:30 +0100, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 12:52 AM, Herbert Valerio Riedel wrote:

...

On 2014-02-25 at 21:38:38 +0100, Michael Snoyman wrote:

[...]

...

* The PVP itself does *not* guarantee reliable builds in all cases. If a transitive dependency introduces new exports, or provides new typeclass instances, a fully PVP-compliant stack can be broken. (If anyone doubts this claim, let me know, I can spell out the details. This has come up in practice.)

...are you simply referring to the fact that in order to guarantee PVP-semantics of a package version, one has to take care to restrict the version bounds of that package's build-deps in such a way, that any API entities leaking from its (direct) build-deps (e.g. typeclass instances or other re-exported entities) are not a function of the "internal" degree of freedoms the build-dep version-ranges provide? Or is there more to it?

That's essentially it. I'll give one of the examples I ran into. (Names omitted on purpose, if the involved party wants to identify himself, please do so, I just didn't feel comfortable doing so without your permission.) Version 0.2 of monad-logger included MonadLogger instances for IO and other base monads. For various reasons, these were removed, and the version bumped to 0.3. This is in full compliance with the PVP.

persistent depends on monad-logger. It can work with either version 0.2 or 0.3 of monad-logger, and the cabal file allows this via `monad-logger >= 0.2 && < 0.4` (or something like that). Again, full PVP compliance.

A user wrote code against persistent when monad-logger version 0.2 was available. He used a function that looked like:

runDatabase :: MonadLogger m => Persistent a -> m a

(highly simplified). In his application, he used this in the IO monad. He depended on persistent with proper lower and upper bounds. Once again, full PVP compliance.

Once I released version 0.3 of monad-logger, his next build automatically upgraded him to monad-logger 0.3, and suddenly his code broke, because there's no MonadLogger instance for IO.

Now *if* the program had been using a system like "cabal freeze" or the like, this could have never happened: cabal wouldn't be trying to automatically upgrade to monad-logger 0.3.

Will this kind of bug happen all the time? No, I doubt it. But if the point of the PVP is to guarantee that builds will work (ignoring runtime concerns), and the PVP clearly fails at that job as well, we really need to reassess putting ourselves through this pain and suffering.

From my point of view, I'd argue that

a) 'persistent' failed to live up to the "spirit" of the PVP contract, i.e. to expose a "contact-surface" which satisfies certain invariants within specific package-version ranges. b) However, the PVP can be blamed as well, as in its current form it doesn't explicitly address the issue of API-leakage from transitive build-dependencies. [1] The question for me now is whether the PVP is fixable in this respect, and at what cost. Moreover, it seems to me, it always comes down to type-class instances causing most problems with the PVP (either by requiring version-bump cascades throughout the PVP-adhering domain of Hackage, or by their hard hard to constraint leakage through package module/boundaries); maybe we need address this issue at the language-level and provide some facility for limiting the propagation of type-class instances first. [1]: An alternative to what I'm suggesting in 'a)' (i.e. that it'd be `persistent`'s obligation), could be that the package you mentioned (which broke due to monad-logger having a non-monotonic API change), might become required to include packages supplying the instances they depends upon in their build-depends, thus turning an transitive dep into a direct dependency. Cheers, hvr

On 2014-02-26 at 10:45:40 +0100, Michael Snoyman wrote: [...]

...

...

...

...are you simply referring to the fact that in order to guarantee PVP-semantics of a package version, one has to take care to restrict the version bounds of that package's build-deps in such a way, that any API entities leaking from its (direct) build-deps (e.g. typeclass instances or other re-exported entities) are not a function of the "internal" degree of freedoms the build-dep version-ranges provide? Or is there more to it?

[...]

...

From my point of view, I'd argue that

a) 'persistent' failed to live up to the "spirit" of the PVP contract, i.e. to expose a "contact-surface" which satisfies certain invariants within specific package-version ranges.

How would persistent have done better? AFAICT, the options are:

1. Do what I did: state a true version dependency on monad-logger, that it works with both version 0.2 and 0.3.

Yes, "persistent" itself does in fact work with both major versions of "monad-logger", but alas the API reachable through depending solely on "persistent" leaks details of the underlying monad-logger version used. ...but the PVP's primary statement is define how a package shall behave from the point of view of its users (where by user I mean package build-depending on `persistent`). So...

2. Constrain it to one or the other, which would be a falsehood that would restrict users' ability to use the package.

...this is would actually be, what I'd interpret the PVP to expect/require from "persistent" in order to satisfy its goal to shield the package's users from incompatible changes.

Let's try it a different way. If transformers removed a MonadIO instance between version 2 and 3 of the library, should that mean that every single package with type signatures involving MonadIO should be constrained to one specific version of transformers?

yes, that'd be what I'm suggesting here (the [1] footnote is a different suggestion for the same problem though)

...

b) However, the PVP can be blamed as well, as in its current form it doesn't explicitly address the issue of API-leakage from transitive build-dependencies. [1]

The question for me now is whether the PVP is fixable in this respect, and at what cost.

Moreover, it seems to me, it always comes down to type-class instances causing most problems with the PVP (either by requiring version-bump cascades throughout the PVP-adhering domain of Hackage, or by their hard hard to constraint leakage through package module/boundaries); maybe we need address this issue at the language-level and provide some facility for limiting the propagation of type-class instances first.

There's one other issue, which is reexports. As an extreme example, imagine:

* Version 1 of the foo package has the Foo module, and it exports foo1 and foo2. * Version 2 of the foo package has the Foo module, and it exports foo1. * Version 1 of the bar package as the Bar module, defined as:

Yes, I'm well aware of this problem, but that's easier to control, as you can use explicit import/export lists to constraint what entities you expose to direct users of your package. That's what I'm doing e.g. in http://hackage.haskell.org/package/deepseq-generics-0.1.1.1/docs/Control-Dee... where I'm explicitly naming the entities I re-export from Control.DeepSeq for convenience. (However, I'm lacking such a facility for instances)

module Bar (module Foo) where import Foo * According to the PVP, the bar package can have a version bound on foo of `foo > 1 && < 2.1`. * User code that depends on foo2 being exported from Bar will be broken by the transitive update of foo.

The example's extreme, but it's the same basic problem as typeclass instances.

[...]

...

[1]: An alternative to what I'm suggesting in 'a)' (i.e. that it'd be `persistent`'s obligation), could be that the package you mentioned (which broke due to monad-logger having a non-monotonic API change), might become required to include packages supplying the instances they depends upon in their build-depends, thus turning an transitive dep into a direct dependency.

I don't think I follow this comment, sorry.

I'm basically just saying, that the package which used "persistent", ought to add "monad-logger ==0.2.*" to its direct build-dependencies, as it depends on an instance provided by monad-logger. The huge down-side is, that you'd have to know about type-class instances leaked through persistent, in order to know that'd you have to add some of "persistent"'s transitive build-depends to your own package, in order to safe yourself from missing out on type-class instances.

Michael, Technically it'd have to do a major version bump to add the instance. Herbert, That said, transitively bumping all the dependent packages whenever anyone upstream's guts change in ways you may never know about isn't a palatable option. It requires every user of every library to know about every instance or import statement that could transitively drag along an orphan even in private modules. This just isn't a realistic model of user behavior. I'm unwilling to accept the corollary that I cannot be compatible with both the current release of a package and the current platform if there have been any instances added in between. If the user wants to lock down the instances they can fix the version of the upstream dependency that is supplying it. The way I see it, if I don't supply the data type, and I don't supply the class, then using a class is fine in my API across major versions. Otherwise nobody can ship anything that crosses more than even one base version let alone versions of other packages. The only breakages that can occur be introduced are all due to orphan instances. What you propose when carried through to its logical conclusion would basically kill all development between platform releases. -Edward On Wed, Feb 26, 2014 at 5:42 AM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 12:22 PM, Herbert Valerio Riedel wrote:

...

On 2014-02-26 at 10:45:40 +0100, Michael Snoyman wrote:

[...]

...

...

...

...

...are you simply referring to the fact that in order to guarantee PVP-semantics of a package version, one has to take care to restrict the version bounds of that package's build-deps in such a way, that any API entities leaking from its (direct) build-deps (e.g. typeclass instances or other re-exported entities) are not a function of the "internal" degree of freedoms the build-dep version-ranges provide? Or is there more to it?

[...]

...

...

From my point of view, I'd argue that

a) 'persistent' failed to live up to the "spirit" of the PVP contract, i.e. to expose a "contact-surface" which satisfies certain invariants within specific package-version ranges.

...

How would persistent have done better? AFAICT, the options are:

1. Do what I did: state a true version dependency on monad-logger, that it works with both version 0.2 and 0.3.

Yes, "persistent" itself does in fact work with both major versions of "monad-logger", but alas the API reachable through depending solely on "persistent" leaks details of the underlying monad-logger version used.

...but the PVP's primary statement is define how a package shall behave from the point of view of its users (where by user I mean package build-depending on `persistent`). So...

...

2. Constrain it to one or the other, which would be a falsehood that would restrict users' ability to use the package.

...this is would actually be, what I'd interpret the PVP to expect/require from "persistent" in order to satisfy its goal to shield the package's users from incompatible changes.

...

Let's try it a different way. If transformers removed a MonadIO instance between version 2 and 3 of the library, should that mean that every single package with type signatures involving MonadIO should be constrained to one specific version of transformers?

yes, that'd be what I'm suggesting here (the [1] footnote is a different suggestion for the same problem though)

So let's analyze how far you want to go here. Imagine if version 0.3.0 of transformers did not have a MonadIO instance for StateT, and version 0.3.1 added it. Now some library has a function:

myFunc :: MonadIO m => Int -> m String

What versions of transformers is it allowed to work with? If it allows version 0.3.0 and 0.3.1, and a user depends on the presence of the MonadIO StateT instance, the build can be broken by moving back to version 0.3.0 (which may be demanded by some other packages dependencies). This is simply the reverse of the monad-logger situation, where an instance was added instead of being removed. I don't see a reasonable solution to this situation... well, besides everyone just trusting a curator to build all of these packages for them.

And just to be clear: if persistent had bumped its lower version bound on monad-logger, then users still on the old version of monad-logger would be unable to upgrade, and for no real reason. persistent would have required a major version bump most likely[1], which would have caused all packages downstream from it to do version bumps as well.

Forgetting about my position as a library author, or as a Stackage maintainer, and speaking purely as a library *user*, this would be a terrible situation to be in.

[1] That's another hole in the PVP I think. It doesn't explicitly address the issue of an API change by eliminating compatibility with a previously accepted dependency, but I've seen huge breakages occur due to this.

...

...

...

b) However, the PVP can be blamed as well, as in its current form it doesn't explicitly address the issue of API-leakage from transitive build-dependencies. [1]

The question for me now is whether the PVP is fixable in this respect, and at what cost.

Moreover, it seems to me, it always comes down to type-class instances causing most problems with the PVP (either by requiring version-bump cascades throughout the PVP-adhering domain of Hackage, or by their hard hard to constraint leakage through package module/boundaries); maybe we need address this issue at the language-level and provide some facility for limiting the propagation of type-class instances first.

There's one other issue, which is reexports. As an extreme example, imagine:

* Version 1 of the foo package has the Foo module, and it exports foo1 and foo2. * Version 2 of the foo package has the Foo module, and it exports foo1. * Version 1 of the bar package as the Bar module, defined as:

Yes, I'm well aware of this problem, but that's easier to control, as you can use explicit import/export lists to constraint what entities you expose to direct users of your package. That's what I'm doing e.g. in

http://hackage.haskell.org/package/deepseq-generics-0.1.1.1/docs/Control-Dee...

where I'm explicitly naming the entities I re-export from Control.DeepSeq for convenience. (However, I'm lacking such a facility for instances)

Agreed, the reexport issue is something that can be dealt with, whereas typeclasses don't have such a facility right now. I just wanted to point it out to make sure we were considering all issues.

...

...

module Bar (module Foo) where import Foo * According to the PVP, the bar package can have a version bound on foo

of

...

`foo > 1 && < 2.1`. * User code that depends on foo2 being exported from Bar will be broken by the transitive update of foo.

The example's extreme, but it's the same basic problem as typeclass instances.

[...]

...

...

[1]: An alternative to what I'm suggesting in 'a)' (i.e. that it'd be `persistent`'s obligation), could be that the package you mentioned (which broke due to monad-logger having a non-monotonic API change), might become required to include packages supplying the instances they depends upon in their build-depends, thus turning an transitive dep into a direct dependency.

...

I don't think I follow this comment, sorry.

I'm basically just saying, that the package which used "persistent", ought to add "monad-logger ==0.2.*" to its direct build-dependencies, as it depends on an instance provided by monad-logger. The huge down-side is, that you'd have to know about type-class instances leaked through persistent, in order to know that'd you have to add some of "persistent"'s transitive build-depends to your own package, in order to safe yourself from missing out on type-class instances.

And my approach is that the only sane way to create repeatable builds is to *always* list the exact versions of all packages you depend upon. And in my opinion, it's far more important to ensure that the code behaves the same way than that it simply builds. The only real way to do that is to always use the same versions of all dependencies.

Michael

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

I think we should relax the PVP requirement to bump the major version number when adding an instance and instead require that the major version bump is only required when using orphan instances and otherwise only a minor version bump is required. Unless I missed some case, code that depends on a library that follows this rule should not not break. Here's my reasoning: If you add a non-orphan instance, it must be because * you define the data type or the type class in your package and * depend on a package that declares the other entity. Therefore, no package that depend on your package can declare a non-orphan instance that could collide with the instance you declare. On Wed, Feb 26, 2014 at 12:57 PM, Edward Kmett wrote:

Michael,

Technically it'd have to do a major version bump to add the instance.

Herbert,

That said, transitively bumping all the dependent packages whenever anyone upstream's guts change in ways you may never know about isn't a palatable option. It requires every user of every library to know about every instance or import statement that could transitively drag along an orphan even in private modules. This just isn't a realistic model of user behavior.

I'm unwilling to accept the corollary that I cannot be compatible with both the current release of a package and the current platform if there have been any instances added in between.

If the user wants to lock down the instances they can fix the version of the upstream dependency that is supplying it.

The way I see it, if I don't supply the data type, and I don't supply the class, then using a class is fine in my API across major versions. Otherwise nobody can ship anything that crosses more than even one base version let alone versions of other packages.

The only breakages that can occur be introduced are all due to orphan instances.

What you propose when carried through to its logical conclusion would basically kill all development between platform releases.

-Edward

On Wed, Feb 26, 2014 at 5:42 AM, Michael Snoyman wrote:

...

On Wed, Feb 26, 2014 at 12:22 PM, Herbert Valerio Riedel wrote:

...

On 2014-02-26 at 10:45:40 +0100, Michael Snoyman wrote:

[...]

...

...

...

> ...are you simply referring to the fact that in order to guarantee > PVP-semantics of a package version, one has to take care to restrict the > version bounds of that package's build-deps in such a way, that any API > entities leaking from its (direct) build-deps (e.g. typeclass instances > or other re-exported entities) are not a function of the "internal" > degree of freedoms the build-dep version-ranges provide? Or is there > more to it?

[...]

...

...

From my point of view, I'd argue that

a) 'persistent' failed to live up to the "spirit" of the PVP contract, i.e. to expose a "contact-surface" which satisfies certain invariants within specific package-version ranges.

...

How would persistent have done better? AFAICT, the options are:

1. Do what I did: state a true version dependency on monad-logger, that it works with both version 0.2 and 0.3.

Yes, "persistent" itself does in fact work with both major versions of "monad-logger", but alas the API reachable through depending solely on "persistent" leaks details of the underlying monad-logger version used.

...but the PVP's primary statement is define how a package shall behave from the point of view of its users (where by user I mean package build-depending on `persistent`). So...

...

2. Constrain it to one or the other, which would be a falsehood that would restrict users' ability to use the package.

...this is would actually be, what I'd interpret the PVP to expect/require from "persistent" in order to satisfy its goal to shield the package's users from incompatible changes.

...

Let's try it a different way. If transformers removed a MonadIO instance between version 2 and 3 of the library, should that mean that every single package with type signatures involving MonadIO should be constrained to one specific version of transformers?

yes, that'd be what I'm suggesting here (the [1] footnote is a different suggestion for the same problem though)

So let's analyze how far you want to go here. Imagine if version 0.3.0 of transformers did not have a MonadIO instance for StateT, and version 0.3.1 added it. Now some library has a function:

myFunc :: MonadIO m => Int -> m String

What versions of transformers is it allowed to work with? If it allows version 0.3.0 and 0.3.1, and a user depends on the presence of the MonadIO StateT instance, the build can be broken by moving back to version 0.3.0 (which may be demanded by some other packages dependencies). This is simply the reverse of the monad-logger situation, where an instance was added instead of being removed. I don't see a reasonable solution to this situation... well, besides everyone just trusting a curator to build all of these packages for them.

And just to be clear: if persistent had bumped its lower version bound on monad-logger, then users still on the old version of monad-logger would be unable to upgrade, and for no real reason. persistent would have required a major version bump most likely[1], which would have caused all packages downstream from it to do version bumps as well.

Forgetting about my position as a library author, or as a Stackage maintainer, and speaking purely as a library *user*, this would be a terrible situation to be in.

[1] That's another hole in the PVP I think. It doesn't explicitly address the issue of an API change by eliminating compatibility with a previously accepted dependency, but I've seen huge breakages occur due to this.

...

...

...

b) However, the PVP can be blamed as well, as in its current form it doesn't explicitly address the issue of API-leakage from transitive build-dependencies. [1]

The question for me now is whether the PVP is fixable in this respect, and at what cost.

Moreover, it seems to me, it always comes down to type-class instances causing most problems with the PVP (either by requiring version-bump cascades throughout the PVP-adhering domain of Hackage, or by their hard hard to constraint leakage through package module/boundaries); maybe we need address this issue at the language-level and provide some facility for limiting the propagation of type-class instances first.

There's one other issue, which is reexports. As an extreme example, imagine:

* Version 1 of the foo package has the Foo module, and it exports foo1 and foo2. * Version 2 of the foo package has the Foo module, and it exports foo1. * Version 1 of the bar package as the Bar module, defined as:

Yes, I'm well aware of this problem, but that's easier to control, as you can use explicit import/export lists to constraint what entities you expose to direct users of your package. That's what I'm doing e.g. in

http://hackage.haskell.org/package/deepseq-generics-0.1.1.1/docs/Control-Dee...

where I'm explicitly naming the entities I re-export from Control.DeepSeq for convenience. (However, I'm lacking such a facility for instances)

Agreed, the reexport issue is something that can be dealt with, whereas typeclasses don't have such a facility right now. I just wanted to point it out to make sure we were considering all issues.

...

...

module Bar (module Foo) where import Foo * According to the PVP, the bar package can have a version bound on

foo of

...

`foo > 1 && < 2.1`. * User code that depends on foo2 being exported from Bar will be broken by the transitive update of foo.

The example's extreme, but it's the same basic problem as typeclass instances.

[...]

...

...

[1]: An alternative to what I'm suggesting in 'a)' (i.e. that it'd be `persistent`'s obligation), could be that the package you mentioned (which broke due to monad-logger having a non-monotonic API change), might become required to include packages supplying the instances they depends upon in their build-depends, thus turning an transitive dep into a direct dependency.

...

I don't think I follow this comment, sorry.

I'm basically just saying, that the package which used "persistent", ought to add "monad-logger ==0.2.*" to its direct build-dependencies, as it depends on an instance provided by monad-logger. The huge down-side is, that you'd have to know about type-class instances leaked through persistent, in order to know that'd you have to add some of "persistent"'s transitive build-depends to your own package, in order to safe yourself from missing out on type-class instances.

And my approach is that the only sane way to create repeatable builds is to *always* list the exact versions of all packages you depend upon. And in my opinion, it's far more important to ensure that the code behaves the same way than that it simply builds. The only real way to do that is to always use the same versions of all dependencies.

Michael

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On 2014-02-26 at 13:09:37 +0100, Michael Snoyman wrote: [...]

+1. If we're discussing PVP changes, the other one I'd like to propose is:

Don't include upper bounds on base, template-haskell, or other libraries which cannot be upgraded, unless you know with certainty that your package will not compile with those other versions. Motivation:

* The bounds will never help cabal choose a better build plan.

...this assumes (as I mentioned in an earlier post) that GHC is never going to ship again with two versions of base (like in the past with base3/4). For that case, we'd want at least something like `base < 5` as upper bound in place (with the policy that `5.*` will only ever be reached if something really disruptive is done to `base`)

On Wed, Feb 26, 2014 at 2:39 PM, Ivan Lazar Miljenovic < ivan.miljenovic@gmail.com> wrote:

...

On 2014-02-26 at 13:09:37 +0100, Michael Snoyman wrote:

[...]

...

+1. If we're discussing PVP changes, the other one I'd like to propose is:

Don't include upper bounds on base, template-haskell, or other libraries which cannot be upgraded, unless you know with certainty that your

On 26 February 2014 23:25, Herbert Valerio Riedel wrote: package

...

...

will not compile with those other versions. Motivation:

* The bounds will never help cabal choose a better build plan.

...this assumes (as I mentioned in an earlier post) that GHC is never going to ship again with two versions of base (like in the past with base3/4). For that case, we'd want at least something like `base < 5` as upper bound in place (with the policy that `5.*` will only ever be reached if something really disruptive is done to `base`)

Agreed; there were a few packages that failed on Gentoo because the author stated that it worked with "base < 5", even though they'd only tested it with cabal-install and at the time it was defaulting to base-3 (though using the runhaskell Setup.hs method used the "best version").

That might be reasonable for the other such libraries though. But to be specific: are we including libraries such as bytestring, containers, etc. as those that can be upgraded or cannot be upgraded?

I'm making the more modest proposal, and just focusing on libraries which cannot at all be upgraded. Michael

On Wed, Feb 26, 2014 at 5:15 AM, Michael Snoyman wrote:

A further reason would be that Hackage demand an upper bound on base anyway

This is policy also and can be changed. -- Gregory Collins

On Wed, Feb 26, 2014 at 1:09 PM, Michael Snoyman wrote:

* The bounds will never help cabal choose a better build plan.

It won't help cabal but it might inform the user what's wrong so he/she can do something about it. Dependency errors are more high-level than compilation errors.

* The bounds make it very difficult to check and debug new versions of GHC.

I believe we added a cabal flag to skip the upper bounds check. Mikhail, do you remember?

* Including the bounds if you know the build will fail makes for more user-friendly messages.

I think the more friendly message for base is: this package doesn't work with this (new) version of GHC. We sometimes have breaking base changes (like in the upcoming 7.8 release, which changes some primops.) Having an error on the base version is better than a compilation error in that case.

On Wed, Feb 26, 2014 at 03:18:45PM +0200, Michael Snoyman wrote:

Even with that flag, we'd still have a bit of a problem. It would be nice if cabal could ignore an upper bound on template-haskell, but respect an upper bound on some other package that *can* be installed with a newer GHC. Perhaps adding that flexibility to cabal would be possible.

I think that's a call for soft bounds (known to work with) and hard bounds (known to not work with) and the flag '--allow-newer' would be only allowed to overwrite the soft bounds. Greetings, Daniel

I very much agree with the sentiment that bounds on, say, template-haskell or ghc-prim never help build plans. One of the most common complaints I get from users has to do with the garbage type errors they get out of the template-haskell that runs lens when some package somewhere has "upgraded" their `template-haskell` install to a non-compatible version. By the time they discover this they are often several packages further along in the install and only have the vaguest inkling of what caused the problem in the first place. Regarding base I still use the < 5 bound, as as Herbert pointed out if we ever needed to ship with support for two versions of base during some big cut-over, it'd be nice to have it work. That said, I'm somewhat leery of how well the community could deal with that in practice. Even the old monads-fd vs mtl split nearly rent the ecosystem asunder. ;) -Edward On Wed, Feb 26, 2014 at 7:09 AM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 2:03 PM, Johan Tibell wrote:

...

I think we should relax the PVP requirement to bump the major version number when adding an instance and instead require that the major version bump is only required when using orphan instances and otherwise only a minor version bump is required. Unless I missed some case, code that depends on a library that follows this rule should not not break.

Here's my reasoning:

If you add a non-orphan instance, it must be because

* you define the data type or the type class in your package and * depend on a package that declares the other entity.

Therefore, no package that depend on your package can declare a non-orphan instance that could collide with the instance you declare.

+1. If we're discussing PVP changes, the other one I'd like to propose is:

Don't include upper bounds on base, template-haskell, or other libraries which cannot be upgraded, unless you know with certainty that your package will not compile with those other versions. Motivation:

* The bounds will never help cabal choose a better build plan. * The bounds may cause valid builds to never be attempted. * The bounds make it very difficult to check and debug new versions of GHC. * Including the bounds if you know the build will fail makes for more user-friendly messages. * Leaving off the bounds if you're not certain will result in users getting more verbose error messages. While uglier, these error messages will be helpful for the package maintainer to adjust the package for the new version of GHC.

I'd also want to push the bounds a little bit further a make a distinction between experimental and stable packages, but that's a bigger proposal and I'd rather start with something more modest.

Michael

On Wed, Feb 26, 2014 at 11:17 AM, Johan Tibell wrote:

Hi Michael,

On Wed, Feb 26, 2014 at 6:45 AM, Michael Snoyman wrote:

...

That's essentially it. I'll give one of the examples I ran into. (Names omitted on purpose, if the involved party wants to identify himself, please do so, I just didn't feel comfortable doing so without your permission.) Version 0.2 of monad-logger included MonadLogger instances for IO and other base monads. For various reasons, these were removed, and the version bumped to 0.3. This is in full compliance with the PVP.

persistent depends on monad-logger. It can work with either version 0.2 or 0.3 of monad-logger, and the cabal file allows this via `monad-logger >= 0.2 && < 0.4` (or something like that). Again, full PVP compliance.

A user wrote code against persistent when monad-logger version 0.2 was available. He used a function that looked like:

runDatabase :: MonadLogger m => Persistent a -> m a

(highly simplified). In his application, he used this in the IO monad. He depended on persistent with proper lower and upper bounds. Once again, full PVP compliance.

Once I released version 0.3 of monad-logger, his next build automatically upgraded him to monad-logger 0.3, and suddenly his code broke, because there's no MonadLogger instance for IO.

Now *if* the program had been using a system like "cabal freeze" or the like, this could have never happened: cabal wouldn't be trying to automatically upgrade to monad-logger 0.3.

I'm trying to wrap my head around this.

* Is runDatabase above a function in persistent or related packages or a function that the user wrote?

It was in persistent. The actual function name is different, but the concept is the same.

* What was the user's dependency range for monad-logger? If he is using the IO instance of MonadLogger from monad-logger, he ought to have a monad-logger == 2.0.* dependency (since removing instances require a major version bump.)

The user didn't directly depend on monad-logger at all, as there was no need with version 0.2 of monad-logger.

* Does any involved package use orphan instances?

Nope, no orphans at all. Michael

On Wed, Feb 26, 2014 at 1:58 PM, Johan Tibell wrote:

On Wed, Feb 26, 2014 at 10:37 AM, Michael Snoyman wrote:

...

On Wed, Feb 26, 2014 at 11:17 AM, Johan Tibell wrote:

...

* What was the user's dependency range for monad-logger? If he is using the IO instance of MonadLogger from monad-logger, he ought to have a monad-logger == 2.0.* dependency (since removing instances require a major version bump.)

The user didn't directly depend on monad-logger at all, as there was no need with version 0.2 of monad-logger.

I guess the reason didn't have to depend directly on monad-logger was because he/she never mentioned any types from that package by name and thus didn't require an import of monad-logger modules.

I wonder if we should have the PVP require that if a type you export in your API loses a class instance, then you're required to bump the major version number. This would have helped in this case.

In this case, that change would not have helped at all: the type which lost an instance as IO, which was not exported by persistent (or monad-logger, for that matter). MonadLogger *also* wasn't exported by persistent. But I don't like this approach to the problem anyway. The theory seems to be that, if we just keep "improving" the PVP, and make library authors' lives more difficult, eventually we'll address all issues. Dependency freezing is a complete solution to the problem described here, I don't understand why there seems to be so much resistance to it. Michael

On Tue, Feb 25, 2014 at 4:16 PM, Vincent Hanquez wrote:

If you're going to dig an old ghc version, what's stopping you from downloading old packages manually from hackage ? I'm sure it can even be automated (more or less).

It's much more difficult because the scale is much greater. Also, if people aren't putting in version bounds, then you have no clue what versions to try. Leaving out version bounds is throwing away information.

However, I don't think we should optimise for this use case; I'ld rather use maintained packages that are regularly updated.

When I write code and get it working, I want it to work for all time. There's absolutely no reason we shouldn't be able to make that happen. If we ignore this case, then Haskell will never be suitable for use in serious production situations. Large organizations want to know that if they start using something it will continue to work. (And don't respond to this with the "avoid success at all costs" line. Haskell is now mature enough that I and a growing number of other people use Haskell on a daily basis for mission-critical applications.)

And even if I wanted to use an old package, provided it's not tied to something fairly internals like GHC's api or such, in a language like haskell, porting to recent version of libraries should be easier than in most other language.

It might be easier, but it can still require a LOT of effort...much more than is justified in some situations. And that doesn't mean that in those situations getting old code working doesn't have significant value.

Furthermore, some old libraries should not be used anymore. Consider old libraries that have security issues for example. Whilst it's not the intent, It's probably a good thing that those old libraries don't build anymore, and people are forced to move to the latest maintained version.

This argument does not hold water when getting a legacy piece of code working has significant intrinsic value. There are plenty of situations where code can have great value to a person/organization even if it doesn't touch the wild internet.

On Tue, Feb 25, 2014 at 4:51 PM, Vincent Hanquez wrote:

I'm not saying this is not painful, but i've done it in the past, and using dichotomy and educated guesses (for example not using libraries released after a certain date), you converge pretty quickly on a solution.

But the bottom line is that it's not the common use case. I rarely have to dig old unused code.

And I have code that I would like to have working today, but it's too expensive to go through this process. The code has significant value to me and other people, but not enough to justify the large cost of getting it working again.

This is moot IMHO. A large organisation would *not* rely on cabal, nor the PvP to actually download packages properly:

Sorry, let me rephrase. s/Large organizations/organizations/ Not everyone is big enough to devote the kind of resources it would take to set up their own system. I've personally worked at two such companies. Building tools that can serve the needs of these organizations will help the Haskell community as a whole.

Not only this is insecure, and as Michael mentioned, you would not get the guarantee you need anyway.

In many cases security doesn't matter because code doesn't interact with the outside world. We're not talking about guaranteeing that building with a later version is buggy. We're talking about guaranteeing that the package will work the way it always worked. It's kind of a package-level purity/immutability.

Even if the above wasn't an issue, Haskell doesn't run in a bubble. I don't expect old ghc and old packages to work with newer operating systems and newer libraries forever.

I don't expect this either. I expect old packages to work the way they always worked with the packages they always worked with.

On Tue, Feb 25, 2014 at 9:25 PM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 1:28 AM, MightyByte wrote:

...

On Tue, Feb 25, 2014 at 4:51 PM, Vincent Hanquez wrote:

...

I'm not saying this is not painful, but i've done it in the past, and

using

...

dichotomy and educated guesses (for example not using libraries released after a certain date), you converge pretty quickly on a solution.

But the bottom line is that it's not the common use case. I rarely have to dig old unused code.

And I have code that I would like to have working today, but it's too expensive to go through this process. The code has significant value to me and other people, but not enough to justify the large cost of getting it working again.

I think we need to make these cases more concrete to have a meaningful discussion. Between Doug and Gregory, I'm understanding two different use cases:

1. Existing, legacy code, built again some historical version of Hackage, without information on the exact versions of all deep dependencies. 2. Someone starting a new project who wants to use an older version of a package on Hackage.

If I've missed a use case, please describe it.

For (1), let's start with the time machine game: *if* everyone had been using the PVP, then theoretically this wouldn't have happened. And *if* the developers had followed proper practice and documented their complete build environment, then PVP compliance would be irrelevant. So if we could go back in time and twist people's arms, no problems would exist. Hurray, we've established that 20/20 hindsight is very nice :).

But what can be done today? Actually, I think the solution is a very simple tool, and I'll be happy to write it if people want: cabal-timemachine. It takes a timestamp, and then deletes all cabal files from our 00-index.tar file that represent packages uploaded after that date. Assuming you know the last date of a successful build, it should be trivial to get a build going again. And if you *don't* know the date, you can bisect until you get a working build. (For that matter, the tool could even *include* a bisecter in it.) Can anyone picture a scenario where this wouldn't solve the problem even better than PVP compliance?

This scenario is never better than PVP compliance. First of all, the user may want some packages that are newer than the timestamp, which this wouldn't support. As people have already mentioned, it's entirely possible for valid install graphs to exist that cabal will fail to find if it doesn't have upper bound information available, because it finds other *invalid* graphs. And even aside from that issue, this would push the work of making sure that a library is compatible with its dependencies onto the library *users*, instead of the developer, where it rightfully belongs (and your proposal ends up pushing even more work onto users!). Why do you think it's acceptable for users to do the testing to make sure that your code works with other packages that your code requires?

For (2), talking about older versions of a package is not relevant. I actively maintain a number of my older package releases, as I'm sure others do as well. The issue isn't about *age* of a package, but about *maintenance* of a package. And we simply shouldn't be encouraging users to start off with an unmaintained version of a package. This is a completely separate discussion from the legacy code base, where- despite the valid security and bug concerns Vincent raised- it's likely not worth updating to the latest and greatest.

Usually the case is not that somebody *wants* to use an older version of package 'foo', it's that they're using some package 'bar' which hasn't yet been updated to be compatible with the latest 'foo'. There are all sorts of reasons this may happen, including big API shifts (e.g. parsec2/parsec3, openGL), poor timing in a maintenance cycle, and the usual worldly distractions. But if packages have upper bounds, the user can 'cabal install', get a coherent package graph, and begin working. At the very worst, cabal will give them a clear lead as to what needs to be updated/who to ping. This is much better than the situation with no upper bounds, where a 'cabal install' may fail miserably or even put together code that produces garbage. And again, it's the library *user* who ends up having to deal with these problems. Upper bounds lead to a better user experience.

All of that said, I still think the only real solution is getting end users off of Hackage. We need an intermediate, stabilizing layer. That's why I started Stackage, and I believe that it's the only solution that will ultimately make library authors and end-users happy. Everything we're discussing now is window dressing.

A curated ecosystem can certainly function, but it seems like a lot more work than just following the PVP and specifying upper bounds. And upper bounds are likely to work better with packages that, for whatever reason, aren't in that curated ecosystem.

On Tue, Feb 25, 2014 at 11:11 PM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 8:03 AM, John Lato wrote:

...

On Tue, Feb 25, 2014 at 9:25 PM, Michael Snoyman wrote:

...

On Wed, Feb 26, 2014 at 1:28 AM, MightyByte wrote:

...

On Tue, Feb 25, 2014 at 4:51 PM, Vincent Hanquez wrote:

...

I'm not saying this is not painful, but i've done it in the past, and

using

...

dichotomy and educated guesses (for example not using libraries released after a certain date), you converge pretty quickly on a solution.

But the bottom line is that it's not the common use case. I rarely have to dig old unused code.

And I have code that I would like to have working today, but it's too expensive to go through this process. The code has significant value to me and other people, but not enough to justify the large cost of getting it working again.

I think we need to make these cases more concrete to have a meaningful discussion. Between Doug and Gregory, I'm understanding two different use cases:

1. Existing, legacy code, built again some historical version of Hackage, without information on the exact versions of all deep dependencies. 2. Someone starting a new project who wants to use an older version of a package on Hackage.

If I've missed a use case, please describe it.

For (1), let's start with the time machine game: *if* everyone had been using the PVP, then theoretically this wouldn't have happened. And *if* the developers had followed proper practice and documented their complete build environment, then PVP compliance would be irrelevant. So if we could go back in time and twist people's arms, no problems would exist. Hurray, we've established that 20/20 hindsight is very nice :).

But what can be done today? Actually, I think the solution is a very simple tool, and I'll be happy to write it if people want: cabal-timemachine. It takes a timestamp, and then deletes all cabal files from our 00-index.tar file that represent packages uploaded after that date. Assuming you know the last date of a successful build, it should be trivial to get a build going again. And if you *don't* know the date, you can bisect until you get a working build. (For that matter, the tool could even *include* a bisecter in it.) Can anyone picture a scenario where this wouldn't solve the problem even better than PVP compliance?

This scenario is never better than PVP compliance. First of all, the user may want some packages that are newer than the timestamp, which this wouldn't support. As people have already mentioned, it's entirely possible for valid install graphs to exist that cabal will fail to find if it doesn't have upper bound information available, because it finds other *invalid* graphs.

And even aside from that issue, this would push the work of making sure that a library is compatible with its dependencies onto the library *users*, instead of the developer, where it rightfully belongs (and your proposal ends up pushing even more work onto users!).

Why do you think it's acceptable for users to do the testing to make sure that your code works with other packages that your code requires?

You're not at all addressing the case I described. The case was a legacy project that someone is trying to rebuild. I'm not talking about any other case in this scenario. To repeat myself:

...

1. Existing, legacy code, built again some historical version of Hackage, without information on the exact versions of all deep dependencies.

In *that specific case*, why wouldn't having a tool to go back in time and build against a historical version of Hackage be *exactly* what you'd need to rebuild the project?

I had understood people talking about "legacy projects" to mean something other than how you read it. In which case, I would suggest that there is a third use case, which IMHO is more important than either of the use cases you have identified. Here's an example: 1. package foo-0.1 appears on hackage 2. package bar-0.1 appears on hackage with a dependency on foo >= 0.1 3. awesomeApp-0.1 appears on hackage, which depends on bar-0.1 and text>=1.0 4. users install awesomeApp 5. package foo-0.2 appears on hackage, with lots of breaking changes 6. awesomeApp users notice that it sometimes breaks with Hungarian characters, and the problem is traced to an error in text 6. text-1.0.0.1 is released with some bug fixes 7. awesomeApp users attempt to do cabal update; cabal install, which fails inscrutably (because it tries to mix foo-0.2 with bar-0.1) There's nothing in this situation that requires any of these packages be unmaintained. The problem is that, rather than wanting to reproduce a fixed set of package versions (which cabal already allows for if that's really desired), sometimes it's desirable that updates be held back in active code bases. Replace "foo" with "QuickCheck" for example (where for a long time users stayed with quickcheck2 because version 3 had major performance regressions in certain use cases). This sort of conflict used to happen *all the time*, and it's very frustrating to users (because something worked before, now it's not working, and they're not generally in a good position to know why). It's annoying to reproduce because the install graph cabal produces depends in part on the user's installed packages. So just because something builds on a developer's box doesn't mean that it would build on the user's box, or it would work for some users but not others (sandboxing has at least helped with that problem).

...

...

For (2), talking about older versions of a package is not relevant. I actively maintain a number of my older package releases, as I'm sure others do as well. The issue isn't about *age* of a package, but about *maintenance* of a package. And we simply shouldn't be encouraging users to start off with an unmaintained version of a package. This is a completely separate discussion from the legacy code base, where- despite the valid security and bug concerns Vincent raised- it's likely not worth updating to the latest and greatest.

Usually the case is not that somebody *wants* to use an older version of package 'foo', it's that they're using some package 'bar' which hasn't yet been updated to be compatible with the latest 'foo'. There are all sorts of reasons this may happen, including big API shifts (e.g. parsec2/parsec3, openGL), poor timing in a maintenance cycle, and the usual worldly distractions. But if packages have upper bounds, the user can 'cabal install', get a coherent package graph, and begin working. At the very worst, cabal will give them a clear lead as to what needs to be updated/who to ping. This is much better than the situation with no upper bounds, where a 'cabal install' may fail miserably or even put together code that produces garbage.

And again, it's the library *user* who ends up having to deal with these problems. Upper bounds lead to a better user experience.

I disagree with that assertion. I get plenty of complaints from users about trying to install packages and getting "confusing error messages" about cabal plan mismatches. I don't disagree that the PVP does make the user experience better in some cases. What I disagree with is the implication that it makes the user experience better in *all* cases. This is simply not a black-and-white issue.

That's a straw man, I don't think anyone has argued that they make the user experience better in *all* cases. The PVP helps significantly, it avoids especially problematic situations like the one above, and in particular it's quite easy for the developer to fix the simple cases. Unlike the 2006 status quo, when problems required manually solving the dependency graph.

On Wed, Feb 26, 2014 at 10:56 AM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 10:36 AM, John Lato wrote:

...

Upper bounds lead to a better user experience.

That's what I'm disagreeing with. I do not believe that, overall, the PVP is giving users a better experience. I've had a huge downturn in reported errors with Yesod since I stopped strictly following the PVP. It's anecdotal, but everything in this thread is really anecdotal.

But you also tell people to use stackage and yesod-platform, which fixes a lot of packages to a specific version, IIRC. That means that not having upper bounds is kind of moot. As a counter-anecdote, almost all build-related problems we've had in the past year have been (http-)conduit or tls related, due to the lack of upper bounds. Erik

On Wed, Feb 26, 2014 at 4:56 AM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 10:36 AM, John Lato wrote:

...

I had understood people talking about "legacy projects" to mean something other than how you read it. In which case, I would suggest that there is a third use case, which IMHO is more important than either of the use cases you have identified. Here's an example:

1. package foo-0.1 appears on hackage 2. package bar-0.1 appears on hackage with a dependency on foo >= 0.1 3. awesomeApp-0.1 appears on hackage, which depends on bar-0.1 and text>=1.0 4. users install awesomeApp 5. package foo-0.2 appears on hackage, with lots of breaking changes 6. awesomeApp users notice that it sometimes breaks with Hungarian characters, and the problem is traced to an error in text 6. text-1.0.0.1 is released with some bug fixes 7. awesomeApp users attempt to do cabal update; cabal install, which fails inscrutably (because it tries to mix foo-0.2 with bar-0.1)

There's another simpler example here where foo-0.2 breaks something and because bar has no upper bound, awesomeApp's build spontaneously fails. I don't think it's reasonable to require that package authors specify dependency bounds for the transitive closure of things they depend on. That's unintuitive and it doesn't scale well as the number of packages and average number of dependencies in a single package increases. The solver needs to know that foo-0.2 is not known to work with bar. We should not throw away that information. I've already encountered several cases complex enough that cabal could not find a solution even though some existed. We need to bound this problem with as many tools as we can.

IIUC, this is *exactly* the case of an unmaintained package. I'm not advocating leaving a package like bar-0.1 on Hackage without an upper bound on foo, if it's known that it breaks in that case. In order for the package to be properly maintained, the maintainer would have to (1) make bar work with foo-0.2, or (2) add an upper bound. So to me, this falls squarely into the category of unmaintained.

Calling that an unmaintained package is intellectually dishonest. Any reasonable definition of "unmaintained" will acknowledge that there will always exist some nonzero amount of time between the time foo is updated and the time bar and awesomeApp can be fixed without claiming that they are unmaintained. This is a fact of the reality we live in. I strongly believe that packages should not break spontaneously because of the actions of others.

If package maintainers are not going to be responsive to updates in the Hackage ecosystem, then I agree that they should use the PVP. I also think they should advertise their packages as not being actively maintained, and people should try to avoid using them if possible. But if an author is giving quick updates to packages, I don't see a huge benefit to the PVP for users, and instead see some downsides (inability to test against newer dependencies), not to mention the much higher maintenance burden for library authors.

I'm sorry, but this is completely ridiculous. There are life situations that happen that preclude package updates. And that doesn't mean that the package is unmaintained. Here's the other side of the story. A large majority of snap's breakages have been caused by dependencies that don't supply upper bounds such as clientsession, cipher-aes, http-conduit, etc. In fact, we switched from http-conduit to http-streams precisely because of this problem. If package maintainers are not going to be responsible and put proper upper bounds for all their dependencies, then I think they should advertise their packages as chronically unstable and likely to break anyone else that uses them at a moment's notice. People should avoid using them if possible. The downsides that you're claiming aren't the fault of the PVP. They are a deficiency in our tooling, and fully locking down the transitive closure is not a good solution. What we need is a way to differentiate known bad upper bounds from unverified upper bounds. It could be something as simple as allowing

That's what I'm disagreeing with. I do not believe that, overall, the PVP is giving users a better experience. I've had a huge downturn in reported errors with Yesod since I stopped strictly following the PVP. It's anecdotal, but everything in this thread is really anecdotal.

I suspect that the root cause for you holding tihs position is that you happen to be operating in a relatively self-contained corner of the package space and don't often depend on fast-moving packages that aren't under your control.

* Michael Snoyman [2014-02-26 21:19:25+0200]

I still have some concerns about the PVP's stance on a few issues, namely:

1. The upper bounds on non-upgradable packages, which I've raised elsewhere in this thread. 2. The lack of flexibility in interpretation. I think there's a qualitative difference between depending on the text package by simply importing Data.Text (Text), versus using some experimental features of a version 0.1 library.

So I suppose if those points were addressed, a lot of my PVP skepticism would disappear, and I'd be more inclined to coming back into the fold, so to say.

My question is: am I alone in thinking these are in fact issues with the PVP?

No, you're not alone. And here are two other issues with PVP and upper bounds: http://ro-che.info/articles/2013-10-05-why-pvp-doesnt-work.html Roman

Michael Snoyman wrote:

1. Make sure that any code that anyone ever wrote will continue to build in the future. 2. Provide guidelines to package authors to make sure that `cabal install foo` works reliably, regardless of what's happened on the rest of Hackage.

It's (1) that I take huge issue with, because (as I've been trying to demonstrate) I don't see any way that a policy like PVP could ever fully solve this problem.

As a commercial shop that must reproduce builds, we have wasted many many hours in the past on cabal hell. Almost all of it was caused by authors of packages we depend on omitting upper bounds. I am aware that some people have other needs, and have wasted their cabal hell time on the opposite problem. However, now the amount of time we waste on cabal hell, though non-zero, is much less. The reason is better ways in cabal to tweak cabal's build plan manually and reproduce winning build plans in the future, such as local cabal.config files. Whatever time we still do waste is still caused by people not being careless about upper bounds. And I believe that people with the opposite problem are wasting less and less time on cabal hell, too. As better tools continually become available, it becomes less disastrous when dependency version bounds are not exactly right. But on the other hand, those bounds are critically important semantic information about a package that make a huge contribution to the potential quality that build tools can achieve. And only the package author can easily provide that information So in my opinion, the proven, working, way forward is: 1. Continue to improve cabal's build plan tools, such as cabal freeze. And yes, cabal-timemachine would be cool :) 2. Continue to adhere to PVP. Package authors should do the best they can to guess the range of dependency versions that their package is very likely to build with. So for example: For libraries like the tagged library, or the deepseq library, which haven't changed in any essential way over any number of major version bumps, go ahead and omit the upper bound. For most vanilla packages, try to be as accurate as you can about the lower bound (without going crazy), and use a two-component upper bound. For base, if you are not using fancy new GHC features, minor version bumps are unlikely to break your package, but a major bump of base (if that will ever happen again) is likely to break quite a few packages and maybe yours. For maintainers like Edward with a huge burden - well, I'm sure Edward will figure out the most reasonable and helpful thing to do in his situation. Perhaps higher powered tools, like some variation on cabal-bounds, would be helpful for him. Etc. Thanks, Yitz

On Wed, Feb 26, 2014 at 1:56 AM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 10:36 AM, John Lato wrote:

...

On Tue, Feb 25, 2014 at 11:11 PM, Michael Snoyman wrote:

...

On Wed, Feb 26, 2014 at 8:03 AM, John Lato wrote:

...

On Tue, Feb 25, 2014 at 9:25 PM, Michael Snoyman wrote:

...

On Wed, Feb 26, 2014 at 1:28 AM, MightyByte wrote:

...

On Tue, Feb 25, 2014 at 4:51 PM, Vincent Hanquez wrote: > > I'm not saying this is not painful, but i've done it in the past, and using > dichotomy and educated guesses (for example not using libraries released > after a certain date), you converge pretty quickly on a solution. > > But the bottom line is that it's not the common use case. I rarely have to > dig old unused code.

And I have code that I would like to have working today, but it's too expensive to go through this process. The code has significant value to me and other people, but not enough to justify the large cost of getting it working again.

I think we need to make these cases more concrete to have a meaningful discussion. Between Doug and Gregory, I'm understanding two different use cases:

1. Existing, legacy code, built again some historical version of Hackage, without information on the exact versions of all deep dependencies. 2. Someone starting a new project who wants to use an older version of a package on Hackage.

If I've missed a use case, please describe it.

For (1), let's start with the time machine game: *if* everyone had been using the PVP, then theoretically this wouldn't have happened. And *if* the developers had followed proper practice and documented their complete build environment, then PVP compliance would be irrelevant. So if we could go back in time and twist people's arms, no problems would exist. Hurray, we've established that 20/20 hindsight is very nice :).

But what can be done today? Actually, I think the solution is a very simple tool, and I'll be happy to write it if people want: cabal-timemachine. It takes a timestamp, and then deletes all cabal files from our 00-index.tar file that represent packages uploaded after that date. Assuming you know the last date of a successful build, it should be trivial to get a build going again. And if you *don't* know the date, you can bisect until you get a working build. (For that matter, the tool could even *include* a bisecter in it.) Can anyone picture a scenario where this wouldn't solve the problem even better than PVP compliance?

This scenario is never better than PVP compliance. First of all, the user may want some packages that are newer than the timestamp, which this wouldn't support. As people have already mentioned, it's entirely possible for valid install graphs to exist that cabal will fail to find if it doesn't have upper bound information available, because it finds other *invalid* graphs.

And even aside from that issue, this would push the work of making sure that a library is compatible with its dependencies onto the library *users*, instead of the developer, where it rightfully belongs (and your proposal ends up pushing even more work onto users!).

Why do you think it's acceptable for users to do the testing to make sure that your code works with other packages that your code requires?

You're not at all addressing the case I described. The case was a legacy project that someone is trying to rebuild. I'm not talking about any other case in this scenario. To repeat myself:

...

1. Existing, legacy code, built again some historical version of Hackage, without information on the exact versions of all deep dependencies.

In *that specific case*, why wouldn't having a tool to go back in time and build against a historical version of Hackage be *exactly* what you'd need to rebuild the project?

I had understood people talking about "legacy projects" to mean something other than how you read it. In which case, I would suggest that there is a third use case, which IMHO is more important than either of the use cases you have identified. Here's an example:

1. package foo-0.1 appears on hackage 2. package bar-0.1 appears on hackage with a dependency on foo >= 0.1 3. awesomeApp-0.1 appears on hackage, which depends on bar-0.1 and text>=1.0 4. users install awesomeApp 5. package foo-0.2 appears on hackage, with lots of breaking changes 6. awesomeApp users notice that it sometimes breaks with Hungarian characters, and the problem is traced to an error in text 6. text-1.0.0.1 is released with some bug fixes 7. awesomeApp users attempt to do cabal update; cabal install, which fails inscrutably (because it tries to mix foo-0.2 with bar-0.1)

There's nothing in this situation that requires any of these packages be unmaintained. The problem is that, rather than wanting to reproduce a fixed set of package versions (which cabal already allows for if that's really desired), sometimes it's desirable that updates be held back in active code bases. Replace "foo" with "QuickCheck" for example (where for a long time users stayed with quickcheck2 because version 3 had major performance regressions in certain use cases).

This sort of conflict used to happen *all the time*, and it's very frustrating to users (because something worked before, now it's not working, and they're not generally in a good position to know why). It's annoying to reproduce because the install graph cabal produces depends in part on the user's installed packages. So just because something builds on a developer's box doesn't mean that it would build on the user's box, or it would work for some users but not others (sandboxing has at least helped with that problem).

IIUC, this is *exactly* the case of an unmaintained package. I'm not advocating leaving a package like bar-0.1 on Hackage without an upper bound on foo, if it's known that it breaks in that case. In order for the package to be properly maintained, the maintainer would have to (1) make bar work with foo-0.2, or (2) add an upper bound. So to me, this falls squarely into the category of unmaintained.

I disagree. I think it's unreasonable to expect that maintainers provide 24/7 availability and near-immediate maintenance releases when updated deps are released. And in the meantime (which may be days, or even a couple weeks for a single-maintainer who might be on holiday), there's plenty of time for this to bite hard. In the past, this meant that broken packages would remain available on hackage for a long time. At least now the package maintainers can do so themselves, but it still means that broken packages have escaped into the wild, which is also bad.

Let me relax my position just a bit. If package maintainers are not going to be responsive to updates in the Hackage ecosystem, then I agree that they should use the PVP. I also think they should advertise their packages as not being actively maintained, and people should try to avoid using them if possible. But if an author is giving quick updates to packages, I don't see a huge benefit to the PVP for users, and instead see some downsides (inability to test against newer dependencies), not to mention the much higher maintenance burden for library authors.

What do you consider "responsive"? 2 hours? 24? 1 week? I suspect that you have IMO unrealistic expectations of maintainers, so I think it would be good to get into specifics.

On Wed, Feb 26, 2014 at 12:36 AM, John Lato wrote:

7. awesomeApp users attempt to do cabal update; cabal install, which fails inscrutably (because it tries to mix foo-0.2 with bar-0.1)

There's nothing in this situation that requires any of these packages be unmaintained. The problem is that, rather than wanting to reproduce a fixed set of package versions (which cabal already allows for if that's really desired), sometimes it's desirable that updates be held back in active code bases

Not to mention, if I maintain "bar", I can basically never go on vacation, because the dude who maintains "foo" can push a new update and break all my users any time. G -- Gregory Collins

On Wed, Feb 26, 2014 at 1:47 PM, Alain O'Dea wrote:

On Feb 26, 2014, at 7:11, Michael Snoyman wrote:

I disagree with that assertion. I get plenty of complaints from users about trying to install packages and getting "confusing error messages" about cabal plan mismatches. I don't disagree that the PVP does make the user experience better in some cases. What I disagree with is the implication that it makes the user experience better in *all* cases. This is simply not a black-and-white issue.

Michael

This is not a new problem.

Java users faced it with Maven and it was solved by curation of Maven Central and the ability to add outside repositories as needed.

Node.js users faced it with NPM and solved it with dependency freezing.

Ruby users faced it with Gem and solved it with dependency freezing.

You've presented three examples of other languages solving the problems using the two techniques I've been advocating through this thread: curation and dependency freezing. Is there an example of a language that took an approach like the PVP and succeeded in solving it?

I imagine there are a world of different solutions to this problem. The PVP isn't a complete solution, but I consider it to be a sensible baseline (like code style conventions and warning free builds) and it appears to me to be in line with best practices from packaging systems of many other languages.

What follows is my opinion, and it comes from a position of relative inexperience with Haskell and considerable experience operating on other language communities.

I feel that the PVP should be encouraged and violations should be considered bugs. Users and concerned community members should report them to maintainers.

Please, please, please don't actually encourage this. There are many things which I consider bad practice in Haskell code. I don't open up bug reports against each package that disagrees with me. If a package on Hackage in fact does *not* build with some dependency it claims to build against, that's a perfectly reasonable thing to report (and I do so on a regular basis via Stackage). But insisting that people add upper bounds when they've clearly stated they do not want to is crossing the line. Michael

On Wed, Feb 26, 2014 at 3:47 AM, Alain O'Dea wrote:

I imagine there are a world of different solutions to this problem. The PVP isn't a complete solution, but I consider it to be a sensible baseline (like code style conventions and warning free builds) and it appears to me to be in line with best practices from packaging systems of many other languages.

Like "semantic versioning" (http://semver.org/) which is very well regarded by people I talk to. G -- Gregory Collins

On Wed, Feb 26, 2014 at 2:11 AM, Michael Snoyman wrote:

I disagree with that assertion. I get plenty of complaints from users about trying to install packages and getting "confusing error messages" about cabal plan mismatches. I don't disagree that the PVP does make the user experience better in some cases. What I disagree with is the implication that it makes the user experience better in *all* cases. This is simply not a black-and-white issue.

And how much of this is because you use your own versioning riules instead of what everyone else uses, thereby almost guaranteeing conflicts with everyone else when anything is updated? Seriously, the easiest way to get in trouble with the current system is to insist you get to not follow it, because stuff breaks. Asserting it's everyone else's fault isn't a solution. Seriously, if everyone follows your versioning system, hsenv becomes essential just as RVM is essential in the Rails community; everyone has to provide their own curated system (your Stackage isn't going to serve everyone's needs, and asking you to make it serve everyone's needs is unreasonable and quite likely an impossible task when everyone else is also doing their own thing) and users have to maintain separate ecosystems for everything they use. This isn't maintainable for much of anyone. (I see someone else later on actually advocated this as a solution. It's not. It's an ongoing maintenance disaster that leads to unmanageable systems.) -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

michael, respectfully, i've had the experience of trying to build a yesod app (git-annex) in a cabal sandbox and i had a version conflict between your own libraries. the only way I could build git-annex was by disabling the web-gui yesod piece. The key piece here is your libraries conflicted with themselves. Thats on you. likewise, 9 time out of 10, when someone tries haskell for the first and they tell me "cabal didn't work", I can guess "you tried to install yesod". They then say "yes", and then I have to explain to them that "yesod has notoriously fragile build constraints, even people who use yesod find it maddening depending on their use case". Michael, respectfully, most version constraint build failures are human friendly, yours are cthulian. I spend a wee bit of time every major GHC release helping patch all the various libs i use (including helping get pandoc ready for each new ghc). It usaually pretty easy. Its often just needing a teeny bit of CPP for an API change plus adjusting cabal range to be wider. I'm totally ok with doing that 1-2 times a year in exchange for nice things. I'd rather do a test and patch cycle every year than have surprise build failures. On Wed, Feb 26, 2014 at 11:06 AM, Michael Snoyman wrote:

On Wednesday, February 26, 2014, Brandon Allbery wrote:

...

On Wed, Feb 26, 2014 at 2:11 AM, Michael Snoyman wrote:

...

I disagree with that assertion. I get plenty of complaints from users about trying to install packages and getting "confusing error messages" about cabal plan mismatches. I don't disagree that the PVP does make the user experience better in some cases. What I disagree with is the implication that it makes the user experience better in *all* cases. This is simply not a black-and-white issue.

And how much of this is because you use your own versioning riules instead of what everyone else uses, thereby almost guaranteeing conflicts with everyone else when anything is updated? Seriously, the easiest way to get in trouble with the current system is to insist you get to not follow it, because stuff breaks. Asserting it's everyone else's fault isn't a solution.

Given that yesod followed the pvp strictly for a number of years and still faced a large number of dependency issues, I don't think the problem is in the lack of pvp support.

The most recent example of this issue was a user trying to use ghc 7.8 and getting an error message about an upper bound on base in the esqueleto package. I'm sure most of us on this mailing list would consider this an easy error message, but at least one of our mythical end users was seriously confused by this situation. It turned out that simply dropping the upper bound was sufficient.

So I come back to this being a numbers game. End users don't necessarily find cabal version error messages any better than ghc build failures. The question is how can we minimize any kind of build failure.

Michael

Seriously, if everyone follows your versioning system, hsenv becomes

...

essential just as RVM is essential in the Rails community; everyone has to provide their own curated system (your Stackage isn't going to serve everyone's needs, and asking you to make it serve everyone's needs is unreasonable and quite likely an impossible task when everyone else is also doing their own thing) and users have to maintain separate ecosystems for everything they use. This isn't maintainable for much of anyone.

Actually, why wouldn't stackage as a curated system work for everyone? Stackage already has over 10% of hackage covered, and I'd imagine if you look at hackage download numbers, that covers the vast majority of the most downloaded packages. I really do think that a curated system could make most people happy, if it gets enough buy-in.

...

(I see someone else later on actually advocated this as a solution. It's not. It's an ongoing maintenance disaster that leads to unmanageable systems.)

-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

i have no clue, i'm just there dealing with the fallout of a confused new person who is convinced that cabal just doesn't work. and no fallout shelters in sight so I have to provide quick solutions that work™ to prevent frustration radiation poisoning. why not just put all the modules in the same package? :) I jest, BUT i have a real point in saying that. why should i have to use a wrapper package which pins all the constraints? On Thu, Feb 27, 2014 at 12:58 AM, Michael Snoyman wrote:

On Wed, Feb 26, 2014 at 6:35 PM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...

michael, respectfully, i've had the experience of trying to build a yesod app (git-annex) in a cabal sandbox and i had a version conflict between your own libraries. the only way I could build git-annex was by disabling the web-gui yesod piece. The key piece here is your libraries conflicted with themselves. Thats on you.

If you have the details on when that happened, I'd be very curious to hear them. I thought my release process had ironed out those possibilities, but if not, I'd like to know about it. Which packages conflicted?

...

likewise, 9 time out of 10, when someone tries haskell for the first and they tell me "cabal didn't work", I can guess "you tried to install yesod". They then say "yes", and then I have to explain to them that "yesod has notoriously fragile build constraints, even people who use yesod find it maddening depending on their use case".

Are these people usually following the actual Yesod installation instructions? Does anyone tell them to follow the Yesod installation instructions? I created the yesod-platform because the PVP *wasn't* fixing this issue. That's why I'm so skeptical that "start using the PVP" would actually fix any of that.

...

Michael, respectfully, most version constraint build failures are human friendly, yours are cthulian.

I spend a wee bit of time every major GHC release helping patch all the various libs i use (including helping get pandoc ready for each new ghc). It usaually pretty easy. Its often just needing a teeny bit of CPP for an API change plus adjusting cabal range to be wider. I'm totally ok with doing that 1-2 times a year in exchange for nice things. I'd rather do a test and patch cycle every year than have surprise build failures.

On Wed, Feb 26, 2014 at 11:06 AM, Michael Snoyman wrote:

...

On Wednesday, February 26, 2014, Brandon Allbery wrote:

...

On Wed, Feb 26, 2014 at 2:11 AM, Michael Snoyman wrote:

...

I disagree with that assertion. I get plenty of complaints from users about trying to install packages and getting "confusing error messages" about cabal plan mismatches. I don't disagree that the PVP does make the user experience better in some cases. What I disagree with is the implication that it makes the user experience better in *all* cases. This is simply not a black-and-white issue.

And how much of this is because you use your own versioning riules instead of what everyone else uses, thereby almost guaranteeing conflicts with everyone else when anything is updated? Seriously, the easiest way to get in trouble with the current system is to insist you get to not follow it, because stuff breaks. Asserting it's everyone else's fault isn't a solution.

Given that yesod followed the pvp strictly for a number of years and still faced a large number of dependency issues, I don't think the problem is in the lack of pvp support.

The most recent example of this issue was a user trying to use ghc 7.8 and getting an error message about an upper bound on base in the esqueleto package. I'm sure most of us on this mailing list would consider this an easy error message, but at least one of our mythical end users was seriously confused by this situation. It turned out that simply dropping the upper bound was sufficient.

So I come back to this being a numbers game. End users don't necessarily find cabal version error messages any better than ghc build failures. The question is how can we minimize any kind of build failure.

Michael

Seriously, if everyone follows your versioning system, hsenv becomes

...

essential just as RVM is essential in the Rails community; everyone has to provide their own curated system (your Stackage isn't going to serve everyone's needs, and asking you to make it serve everyone's needs is unreasonable and quite likely an impossible task when everyone else is also doing their own thing) and users have to maintain separate ecosystems for everything they use. This isn't maintainable for much of anyone.

Actually, why wouldn't stackage as a curated system work for everyone? Stackage already has over 10% of hackage covered, and I'd imagine if you look at hackage download numbers, that covers the vast majority of the most downloaded packages. I really do think that a curated system could make most people happy, if it gets enough buy-in.

...

(I see someone else later on actually advocated this as a solution. It's not. It's an ongoing maintenance disaster that leads to unmanageable systems.)

-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

ah, pardon the implication (one challenge when helping people new to haskell is i don't often get the console logs that were part of their "cabal "). So we have a mix of concerns, some of which are a historical artifact of cabal having been terrible in the past, and now its not (quite) as bad, but far from perfect. things I think would help on all fronts, and I think we (as a community) actually need to figure out how to allocate manpower to the following a) improve the solver tooling. Get some sort of SMT solver ACTUALLY in cabal. The constraint plans aren't that complicated compared with SMT solver benchmarks! I know of several folks who've done their own internal hacked up cabal to do this, and i've heard rumors that *someone* (don't know ) b) proactively work on tooling to improve how constraint failures are reported (though perhaps having the SMT solver tooling would address this). It'd be very very helpful to be able to identify the *minimal* set of constraints + deps that create a conflict, and explanations in that error the exhibit how the conflict in constraints is created. I think everyone agrees these are valuable, its just it requires someone to be able to commit the time to actually doing it. (which is the tricky bit) On Thu, Feb 27, 2014 at 3:12 AM, Michael Snoyman wrote:

On Thu, Feb 27, 2014 at 10:05 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...

i have no clue, i'm just there dealing with the fallout of a confused new person who is convinced that cabal just doesn't work. and no fallout shelters in sight so I have to provide quick solutions that work™ to prevent frustration radiation poisoning.

why not just put all the modules in the same package? :) I jest, BUT i have a real point in saying that.

why should i have to use a wrapper package which pins all the constraints?

That's a great question. It seems to be the question no one's able to answer, myself included. As I keep saying, Yesod went through years of PVP compliance, and users were constantly having build problems. It may have been the cabal-install dependency solver prior to 0.14, and now that problem is resolved. I'm not sure. I know that the new dependency solver used Yesod as a stress test case.

Releasing yesod-platform was pragmatic on two fronts:

1. It guaranteed a sane build plan when cabal (for whatever reason) couldn't determine one. 2. It prevents users from getting a completely untested set of packages, hopefully insulating them from some of the turbulence of Hackage.

I think you need to dial back your assumptions here. Your initial email made it sound like I'd broken the Yesod stack single-handedly a bunch of times recently. That worries me. If that actually happened, please explain how it happened. If users are getting build errors when they don't use yesod-platform, well, that's something we've known about in the Yesod community for a long time. The PVP didn't solve it, yesod-platform is a hack which fixes most of the end-user issue, and I'd love to get to a real solution.

Michael

I don't care. I want a better solver. And I'm willing to help make it happen. On Thursday, February 27, 2014, Greg Weber wrote:

I actually think work on the a) cabal solver has been a distraction from more pressing issues: the need for sandboxes (that is done now) and reproducible builds (frozen dependencies). If you look at Ruby's Bundler, which has been extremely successful, it has historically (maybe they have a better solver now) been a dumb tool in terms of its solver that works extremely well. I think 90+% of this conversation is pretty wasteful, because once we have reproducible builds everything is going to change. If the energy could be re-directed to being able to create reproducible builds in Haskell, then we could figure out what the next most important priority is.

Of course, I agree that better error messages like b) are always valuable.

yesod-platform is essentially a reproducible build and it has been able to fix dependency issues that previously seemed unfixable. At my work we add a cabal.config to our projects to create reproducible builds, and everyone in industry does something similar for their applications.

On Thu, Feb 27, 2014 at 12:49 AM, Carter Schonwald < carter.schonwald@gmail.comjavascript:_e(%7B%7D,'cvml','carter.schonwald@gmail.com');

...

wrote:

...

ah, pardon the implication (one challenge when helping people new to haskell is i don't often get the console logs that were part of their "cabal ").

So we have a mix of concerns, some of which are a historical artifact of cabal having been terrible in the past, and now its not (quite) as bad, but far from perfect.

things I think would help on all fronts, and I think we (as a community) actually need to figure out how to allocate manpower to the following

a) improve the solver tooling. Get some sort of SMT solver ACTUALLY in cabal.

The constraint plans aren't that complicated compared with SMT solver benchmarks! I know of several folks who've done their own internal hacked up cabal to do this, and i've heard rumors that *someone* (don't know )

b) proactively work on tooling to improve how constraint failures are reported (though perhaps having the SMT solver tooling would address this).

It'd be very very helpful to be able to identify the *minimal* set of constraints + deps that create a conflict, and explanations in that error the exhibit how the conflict in constraints is created.

I think everyone agrees these are valuable, its just it requires someone to be able to commit the time to actually doing it. (which is the tricky bit)

On Thu, Feb 27, 2014 at 3:12 AM, Michael Snoyman javascript:_e(%7B%7D,'cvml','michael@snoyman.com');

...

wrote:

...

On Thu, Feb 27, 2014 at 10:05 AM, Carter Schonwald < carter.schonwald@gmail.comjavascript:_e(%7B%7D,'cvml','carter.schonwald@gmail.com');

...

wrote:

...

i have no clue, i'm just there dealing with the fallout of a confused new person who is convinced that cabal just doesn't work. and no fallout shelters in sight so I have to provide quick solutions that work™ to prevent frustration radiation poisoning.

why not just put all the modules in the same package? :) I jest, BUT i have a real point in saying that.

why should i have to use a wrapper package which pins all the constraints?

That's a great question. It seems to be the question no one's able to answer, myself included. As I keep saying, Yesod went through years of PVP compliance, and users were constantly having build problems. It may have been the cabal-install dependency solver prior to 0.14, and now that problem is resolved. I'm not sure. I know that the new dependency solver used Yesod as a stress test case.

Releasing yesod-platform was pragmatic on two fronts:

1. It guaranteed a sane build plan when cabal (for whatever reason) couldn't determine one. 2. It prevents users from getting a completely untested set of packages, hopefully insulating them from some of the turbulence of Hackage.

I think you need to dial back your assumptions here. Your initial email made it sound like I'd broken the Yesod stack single-handedly a bunch of times recently. That worries me. If that actually happened, please explain how it happened. If users are getting build errors when they don't use yesod-platform, well, that's something we've known about in the Yesod community for a long time. The PVP didn't solve it, yesod-platform is a hack which fixes most of the end-user issue, and I'd love to get to a real solution.

Michael

_______________________________________________ Libraries mailing list Libraries@haskell.orgjavascript:_e(%7B%7D,'cvml','Libraries@haskell.org'); http://www.haskell.org/mailman/listinfo/libraries

On Thu, Feb 27, 2014 at 9:35 AM, Austin Seipp wrote:

Hi Greg,

...

I actually think work on the a) cabal solver has been a distraction from more pressing issues: the need for sandboxes (that is done now) and reproducible builds (frozen dependencies). If you look at Ruby's Bundler, which has been extremely successful, it has historically (maybe they have a better solver now) been a dumb tool in terms of its solver that works extremely well. I think 90+% of this conversation is pretty wasteful, because once we have reproducible builds everything is going to change. If the energy could be re-directed to being able to create reproducible builds in Haskell, then we could figure out what the next most important

On Thu, Feb 27, 2014 at 10:30 AM, Greg Weber wrote: priority

...

is.

I'd like to carefully point out however, that it is not a zero-sum game - work dedicated to improving the constraint solver is not work which is implicitly taken away any other set of tools - like a 'freeze' command. There is no 'distraction' IMO - it is a set of individuals (or companies, even) each with their own priorities. I think this is the sign of a healthy community, actually - one that places importance on its tools and seeks to find optimal ways to improve them in a variety of ways. A freeze command and an improved solver are both excellent (and worthy) improvements.

I agree that it is not zero sum, but I do think that at some point the wrong priorities must have been chosen since I have to go to special effort to produce a consistent build. Also this is all getting mixed up with a lot of talk about PVP and other things whose relevance changes if the underlying installation machinery supports what every application developer should be doing.

In reality, bundler works precisely for the reason you said it did: it avoids all the actually difficult problems. But that comes at a cost, because Bundler for example can't actually tell me when things *are* going to break. If I bump my dependencies, create a new Gemfile lock, and test - it could all simply explode later on at runtime, even if it could have been concluded from the constraints that it was all invalid in the first place. The only thing bundler buys me is that this explosion won't potentially extend to the rest of my global environment when it happens. Which is a good thing, truth be told, and why it is so popular - otherwise this happens constantly.

This wasn't my experience using bundler. Bundler supports conservative upgrades that create consistent packages. So if you want to upgrade something you place a range on it and ask Bundler to upgrade it. I don't doubt though that it may let you manually subvert the system.

These two concerns are, as far as I can see, in no way opposed in spirit or practice, and suggesting one is essentially wasted effort that distracts people - when I see no evidence of that - strikes me as odd.

I think the industrial Haskell group supported work on a better solver, which was definitely helpful, but I just think it would have been wiser to support work on consistent builds first. I agree that they can be worked on independently.

That was the plan. I spend a large amount of company time contributing back to open source. Originally I was going to spend some of it on helping implement cabal freeze, but I left it in the hands of others that were more capable and I haven't checked back in a long time. On Thu, Feb 27, 2014 at 1:00 PM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

cool, so is docmunch going to allocate some money or manpower to help out? :)

On Thu, Feb 27, 2014 at 3:36 PM, Greg Weber wrote:

...

On Thu, Feb 27, 2014 at 9:35 AM, Austin Seipp wrote:

...

Hi Greg,

...

I actually think work on the a) cabal solver has been a distraction from more pressing issues: the need for sandboxes (that is done now) and reproducible builds (frozen dependencies). If you look at Ruby's Bundler, which has been extremely successful, it has historically (maybe they have a better solver now) been a dumb tool in terms of its solver that works extremely well. I think 90+% of this conversation is pretty wasteful, because once we have reproducible builds everything is going to change. If the energy could be re-directed to being able to create reproducible builds in Haskell, then we could figure out what the next most important

On Thu, Feb 27, 2014 at 10:30 AM, Greg Weber wrote: priority

...

is.

I'd like to carefully point out however, that it is not a zero-sum game - work dedicated to improving the constraint solver is not work which is implicitly taken away any other set of tools - like a 'freeze' command. There is no 'distraction' IMO - it is a set of individuals (or companies, even) each with their own priorities. I think this is the sign of a healthy community, actually - one that places importance on its tools and seeks to find optimal ways to improve them in a variety of ways. A freeze command and an improved solver are both excellent (and worthy) improvements.

I agree that it is not zero sum, but I do think that at some point the wrong priorities must have been chosen since I have to go to special effort to produce a consistent build. Also this is all getting mixed up with a lot of talk about PVP and other things whose relevance changes if the underlying installation machinery supports what every application developer should be doing.

...

In reality, bundler works precisely for the reason you said it did: it avoids all the actually difficult problems. But that comes at a cost, because Bundler for example can't actually tell me when things *are* going to break. If I bump my dependencies, create a new Gemfile lock, and test - it could all simply explode later on at runtime, even if it could have been concluded from the constraints that it was all invalid in the first place. The only thing bundler buys me is that this explosion won't potentially extend to the rest of my global environment when it happens. Which is a good thing, truth be told, and why it is so popular - otherwise this happens constantly.

This wasn't my experience using bundler. Bundler supports conservative upgrades that create consistent packages. So if you want to upgrade something you place a range on it and ask Bundler to upgrade it. I don't doubt though that it may let you manually subvert the system.

...

These two concerns are, as far as I can see, in no way opposed in spirit or practice, and suggesting one is essentially wasted effort that distracts people - when I see no evidence of that - strikes me as odd.

I think the industrial Haskell group supported work on a better solver, which was definitely helpful, but I just think it would have been wiser to support work on consistent builds first. I agree that they can be worked on independently.

On Wed, Feb 26, 2014 at 11:06 AM, Michael Snoyman wrote:

On Wednesday, February 26, 2014, Brandon Allbery wrote:

...

Seriously, if everyone follows your versioning system, hsenv becomes essential just as RVM is essential in the Rails community; everyone has to provide their own curated system (your Stackage isn't going to serve everyone's needs, and asking you to make it serve everyone's needs is unreasonable and quite likely an impossible task when everyone else is also doing their own thing) and users have to maintain separate ecosystems for everything they use. This isn't maintainable for much of anyone.

...

Actually, why wouldn't stackage as a curated system work for everyone? Stackage already has over 10% of hackage covered, and I'd imagine if you look at hackage download numbers, that covers the vast majority of the most downloaded packages. I really do think that a curated system could make most people happy, if it gets enough buy-in.

Because you don't have any particular reason to curate packages you are not using yourself, and as Hackage grows you also won't have *time* to do so, even if you dropped Yesod and devoted your full time to Stackage curation. -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

On 25/02/2014 06:44, Michael Snoyman wrote:

Next is the issue of PVP. I am someone who has stopped religiously following the PVP in the past few years. Your email seems to imply that only those following the PVP care about making sure that "packages work together." I disagree here; I don't use the PVP specifically because I care about package interoperability.

The point of the PVP is to ensure that code builds. It's a purely compile-time concept. The PVP solves the problem of an update to a dependency causing a downstream package to break. And assuming everyone adheres to it[1], it ensures that cabal will never try to perform a build which isn't guaranteed to work.

But that's only one half of the "package interoperability" issue. I face this first hand on a daily basis with my Stackage maintenance. I spend far more time reporting issues of restrictive upper bounds than I do with broken builds from upstream changes. So I look at this as purely a game of statistics: are you more likely to have code break because version 1.2 of text changes the type of the map function and you didn't have an upper bound, or because two dependencies of yours have *conflicting* versions bounds on a package like aeson[2]? In my experience, the latter occurs far more often than the former.

It's worth mentioning that cabal failing to find a solution is far less costly for me to discover than cabal finding a solution and then having a build of a large graph of packages fail, because by that point I've wasted a lot of time waiting for the build and I now have a thoroughly confused package database to recover from (whether using a sandbox or not). Ganesh

On Mon, Feb 24, 2014 at 11:21 PM, Sven Panne wrote:

Regarding upper bounds: I never understood what their advantage should be, IMHO they only lead to a version hell where you can't compile your stuff anymore *only* because of the bounds, not because of anything else.

IMHO this is a bad enough outcome, but it can also allow you to compile code in a way that it behaves incorrectly (if a function's behavior has changed but the type has not). It also leads to a situation where cabal generates what it thinks is an acceptable dependency solution, but that solution fails, making the user need to solve the dependency tree themselves and specify constraints on the cabal command line. This is reason the PVP specifies upper bounds on versions: it makes that work the responsibility of the developer rather than the user. At the time the PVP was introduced, users often experienced serious hardships when installing various combinations of packages, and IIRC it was widely perceived that developers should shoulder the load of making sure their packages would work together as specified. However, I think the PVP may have been a victim of its own success; user complaints about botched installs and invalid install plans seem quite rare these days, and some developers are pushing back against this extra workload. (or maybe there are no Haskell users?) John L.

On Tue, Feb 25, 2014 at 2:51 AM, Daniel Trstenjak < daniel.trstenjak@gmail.com> wrote:

Hi Sven,

...

o_O Dumb question: Can somebody please explain why this doesn't conform to the PVP? I have a very hard time reading that out of http://www.haskell.org/haskellwiki/Package_versioning_policy. Perhaps I'm looking at the wrong document or this interpretation is just wishful thinking...

If I'm getting it right, you don't have to increase a major version number if you're e.g. just adding another function.

But if the user of your library imports unqualified or implicit, then he will also get your added function and this function might conflict with functions in your code base.

Note: this particular concern would be much lessened at least for local definitions, had we done anything with Lennart's perfectly reasonable suggestion to change the scoping rules to let local definitions win over imports. When I mentioned above that I had had half a dozen problems in five years, four of them would have been resolved successfully by that proposal. -Edward

On Tue, Feb 25, 2014 at 10:16 AM, Brandon Allbery wrote:

because the affected users are just users and apparently not important enough to consider when setting versioning policy.

Users are important enough to consider, but their needs should not trump all others. In particular, (nearly?) all software on Hackage is given to users at no charge. Developers invest their time. Their needs are important too. If policies make it too troublesome for developers to maintain software or publicly post it on Hackage, they will just stop posting it. Obviously there is a balance to be struck, as if you make things too hard for users then there will be no users. The problem is that the PVP is putting a considerable maintenance burden on developers but it's not even clear there is commensurate benefit to users. Often it's hard to get different packages to work together because upper bounds are too tight.

On Mon, Feb 24, 2014 at 1:09 PM, Henning Thielemann wrote:

Am 24.02.2014 18:57, schrieb Brandon Allbery:

...

There is something vaguely smelly about specifically omitting the context

...

it is an annoyingly common name to take for

such an often unqualified import.

in the original message. Yes, we're quite aware you do not consider it legitimate. Distorting someone else's meaning to press your point is also not legitimate.

The phrase "reserve 'zero'" suggests that once we choose Bits.zero, the identifier 'zero' is reserved once and for all and cannot be used for something different anymore. That is, this phrasing removes the option of qualified imports from the scope and thus generates the wrong context.

Can someone please, please tell me why we must avoid qualified imports at all costs? Why is this option repeatedly ignored when just saying zeroBits (+1) or zero (-1)?