The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

newer
digamma function

older
whenJust / forM_ (was Re:...

Johan Tibell

7 Dec 2012 7 Dec '12

11:20 a.m.

On Thu, Dec 6, 2012 at 11:52 PM, Michael Snoyman wrote:

...

I thought the natural response to this would be to put a lower bound on text. After all, following the PVP, there should be no problem with updating a minor version number. However, when I did so, many users of the Haskell Platform were no longer able to compile due to conflicting dependencies. The result: I had to put in some dirty conditional compilation tricks in persistent to avoid inlining `pack` when using older versions of text.

I've meant to write about this but haven't yet found time to do so. Assuming that the goal of the PVP is to make sure that a packages that previously compiled continues to compile, dependencies must be taken into account when computing package version number bumps. For example, given these packages: A-1.0 B-1.0 depends on A-1.0.0.* C-1.0 depends on A-1.0.0.* and B-1.0.0.* Now assume that A-2.0 is released and B starts depending on the new version of A (without changing its own API). He now have this situation: A-1.0 A-2.0 B-1.?.?.? depends on A-2.0.0.* C-1.0 depends on A-1.0.0.* and B-1.0.0.* If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1 even though C's dependency on B (i.e. B-1.0.0.*) suggests that it would. If B updates it's lower bound on a dependency it must bump its own major version number. We can summarize this under the slogan "versions are APIs"; the versions you depend on are part of your API. Aside: Even if cabal install would backtrack and *not* use the new version of B (i.e. B-1.0.0.1) and thus be able to compile C-1.0 using the old version of B things would still be bad as C's dependency constraints would be a lie. I think the PVP policy must be changed to require that increasing lower bounds on dependencies requires an increase in the major version number. -- Johan

Show replies by date

Ian Lynagh

7 Dec 7 Dec

11:28 a.m.

New subject: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...

If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work? Thanks Ian

Johan Tibell

11:30 a.m.

On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh wrote:

...

On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...
If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work?

See my aside at the end. Yes, it will backtrack and use the older version, but the users might get confused why the new version isn't used as it's version number suggests that it could be.

Ian Lynagh

11:41 a.m.

New subject: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Fri, Dec 07, 2012 at 08:30:45AM -0800, Johan Tibell wrote:

...

On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh wrote:

...
On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...
If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work?

See my aside at the end. Yes, it will backtrack and use the older version, but the users might get confused why the new version isn't used as it's version number suggests that it could be.

I don't think most users would notice, and for those that do the "-v" flag to cabal-install ought to tell them why. I don't think this (theoretical?) confusion issue is worth the effort of bumping major versions all the way up the dependency tree. Thanks Ian

Conrad Parker

9 Dec 9 Dec

11:15 p.m.

On 8 December 2012 00:41, Ian Lynagh wrote:

...

On Fri, Dec 07, 2012 at 08:30:45AM -0800, Johan Tibell wrote:

...
On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh wrote:

...
On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...
If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work?

See my aside at the end. Yes, it will backtrack and use the older version, but the users might get confused why the new version isn't used as it's version number suggests that it could be.

I don't think most users would notice, and for those that do the "-v" flag to cabal-install ought to tell them why.

I don't think this (theoretical?) confusion issue is worth the effort of bumping major versions all the way up the dependency tree.

What if the minor version bump in B is for fixing a security issue? Conrad.

Ian Lynagh

10 Dec 10 Dec

5:56 a.m.

New subject: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Mon, Dec 10, 2012 at 12:15:47PM +0800, Conrad Parker wrote:

...

On 8 December 2012 00:41, Ian Lynagh wrote:

...
On Fri, Dec 07, 2012 at 08:30:45AM -0800, Johan Tibell wrote:

...
On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh wrote:

...
On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...
If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work?

See my aside at the end. Yes, it will backtrack and use the older version, but the users might get confused why the new version isn't used as it's version number suggests that it could be.

I don't think most users would notice, and for those that do the "-v" flag to cabal-install ought to tell them why.

I don't think this (theoretical?) confusion issue is worth the effort of bumping major versions all the way up the dependency tree.

What if the minor version bump in B is for fixing a security issue?

Then the proposal doesn't help: It doesn't matter whether you give the new version of B a version number of 1.0.0.1 or 2.0.0.0, cabal-install will still install B 1.0 and A 1.0 when asked to install C. Thanks Ian

Duncan Coutts

8 Dec 8 Dec

3:11 p.m.

On 7 December 2012 16:30, Johan Tibell wrote:

...

On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh wrote:

...
On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...
If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work?

See my aside at the end. Yes, it will backtrack and use the older version, but the users might get confused why the new version isn't used as it's version number suggests that it could be.

I agree with Ian here, we can find the solution ok and we don't need tighter constraints. Indeed it actually over-tightens, since if B can work with A-1.0 or 2.0 then we've eliminated a possible solution. Yes, the solver could and should tell us when it's not selecting the "best" (usually latest) version (it already does this) and why (it doesn't do that yet). Also, I don't think it scales, if we followed it strictly and said every major bump in any dependent package forces a bump on all things that depend on it, it'd just be far too much, especially when it's actually redundant. But yes it is true that the "real" API of a package is in the version number of the package itself, and the versions of all the packages which are "exposed" by that package (meaning packages that define types that are mentioned in the api of the exposing package). But it that has the property that I only notice changes in the types in those dependencies if I also depend on them directly myself. So we can get away with specifying local info, and not have to encode the full transative info (I think). Duncan

Simon Marlow

10 Dec 10 Dec

10:49 a.m.

On 08/12/12 20:11, Duncan Coutts wrote:

...

On 7 December 2012 16:30, Johan Tibell wrote:

...
On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh wrote:

...
On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:

...
If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will still work?

See my aside at the end. Yes, it will backtrack and use the older version, but the users might get confused why the new version isn't used as it's version number suggests that it could be.

I agree with Ian here, we can find the solution ok and we don't need tighter constraints. Indeed it actually over-tightens, since if B can work with A-1.0 or 2.0 then we've eliminated a possible solution.

Yes, the solver could and should tell us when it's not selecting the "best" (usually latest) version (it already does this) and why (it doesn't do that yet).

Also, I don't think it scales, if we followed it strictly and said every major bump in any dependent package forces a bump on all things that depend on it, it'd just be far too much, especially when it's actually redundant.

I agree with all that.

...

But yes it is true that the "real" API of a package is in the version number of the package itself, and the versions of all the packages which are "exposed" by that package (meaning packages that define types that are mentioned in the api of the exposing package). But it that has the property that I only notice changes in the types in those dependencies if I also depend on them directly myself. So we can get away with specifying local info, and not have to encode the full transative info (I think).

This has always bugged me. I'd be surprised if your hypothesis here were true - it seems likely that if the types from a dependency leak into the API of a package, then the PVP applies to those "re-exposed" parts of the API too. One consequence of this is that if you have a non-private dependency and depend on a range of major versions, you better not re-expose any of the APIs that change between major versions. Otherwise your package has a random API, and that's not allowed. I didn't say anything in the PVP about this because I wasn't sure exactly what to say, but it would be good to nail it down. Cheers, Simon

Johan Tibell

11:02 a.m.

On Sat, Dec 8, 2012 at 12:11 PM, Duncan Coutts wrote:

...

But yes it is true that the "real" API of a package is in the version number of the package itself, and the versions of all the packages which are "exposed" by that package (meaning packages that define types that are mentioned in the api of the exposing package). But it that has the property that I only notice changes in the types in those dependencies if I also depend on them directly myself. So we can get away with specifying local info, and not have to encode the full transative info (I think).

It's worse than that. The failure to build in my example above is a failure to find a valid install plan due to conflicting dependency ranges, not due to clashing symbols from different package versions. It happens even with private dependencies! -- Johan

Simon Marlow

11 Dec 11 Dec

4:45 a.m.

On 10/12/12 16:02, Johan Tibell wrote:

...

On Sat, Dec 8, 2012 at 12:11 PM, Duncan Coutts wrote:

...
But yes it is true that the "real" API of a package is in the version number of the package itself, and the versions of all the packages which are "exposed" by that package (meaning packages that define types that are mentioned in the api of the exposing package). But it that has the property that I only notice changes in the types in those dependencies if I also depend on them directly myself. So we can get away with specifying local info, and not have to encode the full transative info (I think).

It's worse than that. The failure to build in my example above is a failure to find a valid install plan due to conflicting dependency ranges, not due to clashing symbols from different package versions. It happens even with private dependencies!

I'm a bit lost. The example from your mail that started this thread *does* compile, because Cabal finds a valid install plan by avoiding the new version of B. It's working exactly as intended, isn't it? Then you said that the user might get confused about why the new B wasn't being used, to which the response is that we need better diagnostics. What's the problem you're getting at above? Cheers, Simon

Johan Tibell

11:14 a.m.

On Tue, Dec 11, 2012 at 1:45 AM, Simon Marlow wrote:

...

I'm a bit lost. The example from your mail that started this thread *does* compile, because Cabal finds a valid install plan by avoiding the new version of B. It's working exactly as intended, isn't it? Then you said that the user might get confused about why the new B wasn't being used, to which the response is that we need better diagnostics.

What's the problem you're getting at above?

The problem is one of two things, depending on how you look at it. On the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with any version of A-1.0.0, that is a lie (as per the scenario in my original email) and the PVP needs to be adjusted. On the other hand, if you take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with the A-1.0.0 API then users and developers might get confused when newer releases of libraries don't get picked up even though the version bounds suggests that they would. We could try to solve this by cabal telling the user about when this situation appears, but understanding why package version conflicts appear is already hard to understand for users. I guess one reason I brought this up is that it hasn't really been discussed before to my knowledge. -- Johan

Simon Marlow

11:34 a.m.

On 11/12/12 16:14, Johan Tibell wrote:

...

On Tue, Dec 11, 2012 at 1:45 AM, Simon Marlow wrote:

...
I'm a bit lost. The example from your mail that started this thread *does* compile, because Cabal finds a valid install plan by avoiding the new version of B. It's working exactly as intended, isn't it? Then you said that the user might get confused about why the new B wasn't being used, to which the response is that we need better diagnostics.

What's the problem you're getting at above?

The problem is one of two things, depending on how you look at it. On the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with any version of A-1.0.0, that is a lie (as per the scenario in my original email) and the PVP needs to be adjusted.

If users expect this interpretation, then I think we should tackle that with better documentation. It's important that the dependencies on a package are a *local* statement about the APIs that *this* package requires, and don't need to take into account constraints imposed elsewhere. The consequences of the alternative are pretty severe: every package author basically has to do Cabal's job and perform a backtracking search, solving the constraints so they can put the results in their build-depends. Furthermore they would have to re-do this every time a new constraint is added to the system (somebody uploads a new package that might be a dependency of one of yours).

...

On the other hand, if you take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with the A-1.0.0 API then users and developers might get confused when newer releases of libraries don't get picked up even though the version bounds suggests that they would. We could try to solve this by cabal telling the user about when this situation appears, but understanding why package version conflicts appear is already hard to understand for users.

Right, better diagnostics is the solution. I couldn't use B-1.0.0.1 because it depends on A-2.0 and I couldn't use A-2.0 because C requires A-1.0.0.* Although, listing all the solutions that *don't* work will quickly get very verbose. This is a tricky UI problem. Perhaps there should be a way to ask questions - "why didn't you use B-1.0.0.1?". Cheers, Simon

Johan Tibell

11:58 a.m.

On Tue, Dec 11, 2012 at 8:34 AM, Simon Marlow wrote:

...

On 11/12/12 16:14, Johan Tibell wrote:

...
The problem is one of two things, depending on how you look at it. On the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with any version of A-1.0.0, that is a lie (as per the scenario in my original email) and the PVP needs to be adjusted.

If users expect this interpretation, then I think we should tackle that with better documentation. It's important that the dependencies on a package are a *local* statement about the APIs that *this* package requires, and don't need to take into account constraints imposed elsewhere.

The consequences of the alternative are pretty severe: every package author basically has to do Cabal's job and perform a backtracking search, solving the constraints so they can put the results in their build-depends. Furthermore they would have to re-do this every time a new constraint is added to the system (somebody uploads a new package that might be a dependency of one of yours).

It's still a local statement: if we changed the PVP (which I don't suggest we do at this point), it would be to require that every time you bump a lower version bound on a dependency you have to bump your own major version number. You don't have to recursively look at all your dependencies as we assume they also follow the PVP.

...

Right, better diagnostics is the solution.

I couldn't use B-1.0.0.1 because it depends on A-2.0 and I couldn't use A-2.0 because C requires A-1.0.0.*

Although, listing all the solutions that *don't* work will quickly get very verbose. This is a tricky UI problem. Perhaps there should be a way to ask questions - "why didn't you use B-1.0.0.1?".

Better diagnostic is probably they way forward for now. I feel that cabal used to be better at telling me why it made certain dependency choices. Now it tells me very little (i.e. basically a list of packages it tried, without information about why they were tried). -- Johan

Andres Löh

12:19 p.m.

...

Better diagnostic is probably they way forward for now. I feel that cabal used to be better at telling me why it made certain dependency choices. Now it tells me very little (i.e. basically a list of packages it tried, without information about why they were tried).

There's a bug in cabal-install-1.16.0 which I fixed in HEAD (I hope) that caused cabal-install to no longer print the reason why a goal is tried when -v3 is selected. One of the problems in printing diagnostics after the fact is that there's a lot of information during solving, but most of it is discarded. Currently, most of the info about the actual install plan is recomputed after the plan has been communicated back by the solver. This has the advantage of making it easy to still support both solvers, but it's making it unnecessarily difficult to include additional info in the final output. I have quite a number of ideas on how to improve the diagnostics in principle, but I can currently give no estimate on how long it will take me in practice to implement any of this. Cheers, Andres

Joachim Breitner

5:34 p.m.

New subject: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

Hi, Am Dienstag, den 11.12.2012, 08:14 -0800 schrieb Johan Tibell:

...

On Tue, Dec 11, 2012 at 1:45 AM, Simon Marlow wrote:

...
I'm a bit lost. The example from your mail that started this thread *does* compile, because Cabal finds a valid install plan by avoiding the new version of B. It's working exactly as intended, isn't it? Then you said that the user might get confused about why the new B wasn't being used, to which the response is that we need better diagnostics.

What's the problem you're getting at above?

The problem is one of two things, depending on how you look at it. On the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with any version of A-1.0.0, that is a lie (as per the scenario in my original email) and the PVP needs to be adjusted.

I don’t think this is a good semantics for the dependency. I always understand package dependencies as: If you find a set of packaging such that all constraints are fulfilled, then things will build correctly. What if A-1.0.0.1 would fail for other reasons, besides a bumped dependency? E.g. A decides to implement an internal parser with happy, and adds that to the list of tools required. Now people who could build C before, but don’t have happy installed, cannot build C. But should that really require A to bump a major version?

...

On the other hand, if you take "C-1.0 depends on A-1.0.0.*" to mean that C states that it can work with the A-1.0.0 API then users and developers might get confused when newer releases of libraries don't get picked up even though the version bounds suggests that they would.

A user should not worry about that. What he expects is to do cabal install and get some working installation. Or better, a User would use stackage or a distribution, where such issues are taken care of. A developer can be expected to know about these issue and be able to cope with it. And for her

...

We could try to solve this by cabal telling the user about when this situation appears, but understanding why package version conflicts appear is already hard to understand for users.

would indeed be useful. And, as always, with my Debian maintainer hat on: I’d prefer not to see unnecessary meta data changes (as long as the promise above is preserved), as they cause work for us. Greetings, Joachim -- Joachim "nomeata" Breitner mail@joachim-breitner.de | nomeata@debian.org | GPG: 0x4743206C xmpp: nomeata@joachim-breitner.de | http://www.joachim-breitner.de/

Michael Snoyman

8 Dec 8 Dec

2:48 p.m.

On Fri, Dec 7, 2012 at 6:20 PM, Johan Tibell wrote:

...

On Thu, Dec 6, 2012 at 11:52 PM, Michael Snoyman wrote:

...
I thought the natural response to this would be to put a lower bound on text. After all, following the PVP, there should be no problem with updating a minor version number. However, when I did so, many users of the Haskell Platform were no longer able to compile due to conflicting dependencies. The result: I had to put in some dirty conditional compilation tricks in persistent to avoid inlining `pack` when using older versions of text.

I've meant to write about this but haven't yet found time to do so.

Assuming that the goal of the PVP is to make sure that a packages that previously compiled continues to compile, dependencies must be taken into account when computing package version number bumps. For example, given these packages:

A-1.0 B-1.0 depends on A-1.0.0.* C-1.0 depends on A-1.0.0.* and B-1.0.0.*

Now assume that A-2.0 is released and B starts depending on the new version of A (without changing its own API). He now have this situation:

A-1.0 A-2.0 B-1.?.?.? depends on A-2.0.0.* C-1.0 depends on A-1.0.0.* and B-1.0.0.*

If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no longer compiles (due to a version constraint failure) with B-1.0.0.1 even though C's dependency on B (i.e. B-1.0.0.*) suggests that it would. If B updates it's lower bound on a dependency it must bump its own major version number.

We can summarize this under the slogan "versions are APIs"; the versions you depend on are part of your API.

Aside: Even if cabal install would backtrack and *not* use the new version of B (i.e. B-1.0.0.1) and thus be able to compile C-1.0 using the old version of B things would still be bad as C's dependency constraints would be a lie.

I think the PVP policy must be changed to require that increasing lower bounds on dependencies requires an increase in the major version number.

-- Johan

+1. I already try and follow this policy myself, I've seen the lack of clarity on it lead to versioning problems in the past. If anyone is interested, I can try to dredge up the actual incidents. Michael

4544

Age (days ago)

4548

Last active (days ago)

List overview

Download

15 comments

8 participants

participants (8)

Andres Löh
Conrad Parker
Duncan Coutts
Ian Lynagh
Joachim Breitner
Johan Tibell
Michael Snoyman
Simon Marlow