On Thu, Sep 27, 2012 at 7:50 PM, Brent Yorgey wrote:

Ought we to include the commit

"Make cabal-install build with Cabal-1.16." https://github.com/haskell/cabal/commit/782a48717140df7bd695f7d16441dc9b0db5...

? Also, shouldn't we try to have the new cabal-install release work with GHC 7.6? In that case I suppose we'll want

https://github.com/haskell/cabal/commit/d9d33822fcb8addb55a1493e38d2aad2ae34... https://github.com/haskell/cabal/commit/c59cf4991a3518bce05e64d298361819e56b...

?

Definitely. Perhaps I've should have added that there are some commits that I will definitely cherry-pick. Perhaps I should have said, are there any commits that you care about in particular that you want to make sure make it into the release. -- Johan

On Fri, Sep 28, 2012 at 5:55 PM, Duncan Coutts

wrote:

Being able to do this relies on both editing on the server, and for the client to actually use the updated .cabal file. So this patch provides the client side part of the feature. The server side already works (we've actually made quite a few edits to .cabal files on hackage post-release) but the UI for it will improve significantly in the new server which we expect to go live during the lifetime of this next cabal-install release. So getting the feature in now would be a real help to everyone (and ideally, most users will never notice).

This feels like a big piece of work to be pushing in right before a release. It worries me because now there's an invisible piece of metadata floating around that neither OS packagers nor library maintainers can see. In a way, this feels strictly worse than having a package just plain break, because it's not hard to imagine a cycle of "package breaks; magic invisible dependency update silently fixes it; new version is released; package re-breaks because maintainer never found out about previous breakage". I could be convinced that in fact everything is going to be awesome and it will all somehow work, but in the short term I'd prefer to delay sprinkling invisible dependency pixie dust on packages, and let this wait until the next release, 3 months down the line.

Hi, I don't want to hold up the release on this issue if it needs more discussion. I'd like to make another Cabal/cabal-install release before end of the year (probably around GHC's release timeline), to include the sandbox stuff once done. Would it be OK to wait until then with these two patches? -- Johan

On Mon, Oct 1, 2012 at 1:26 AM, Duncan Coutts wrote:

There's plenty of time for more discussion of the feature in general before we start actually using it.

Okay, then I'm fine with the patches going in. Thanks, Duncan!

On 01/10/2012 09:26, Duncan Coutts wrote:

On 1 October 2012 04:21, Johan Tibell wrote:

...

Hi,

I don't want to hold up the release on this issue if it needs more discussion. I'd like to make another Cabal/cabal-install release before end of the year (probably around GHC's release timeline), to include the sandbox stuff once done. Would it be OK to wait until then with these two patches?

I think these specific patches do not need discussion now, here's why: the feature comes in two parts, server and client, the bulk of the feature is the server side part which is not implemented yet. There's plenty of time for more discussion of the feature in general before we start actually using it. But if we decide it's ok, then we have to wait even longer to start using it. We can deploy new server side code quickly, but not client side code. If we decide it's not ok, then we don't make use of the feature server-side and then we rip out the client code. So that's why I want to get the client code in now, in advance of the server side implementation.

Also note, I have discussed this before with several people (and I think on this list too, a couple years back). So Bryan didn't see it, but it's not like I'm inventing this crazy idea from nowhere and forcing it on people. I'm very happy to discuss the details again, but that's independent of these two patches.

Forgive me if I'm being stupid, but why can't the feature be implemented entirely on the server? That is, when the dependencies are updated by some as-yet-unspecified mechanism on the server, a complete new version of the package is generated and added to the index. A new tarball would be generated on the server, by replacing the .cabal file in the previous tarball. The behaviour would be exactly as if a new version had been uploaded, but it is all done by Hackage, using some pre-agreed version numbering policy. This would eliminate the problems of having invisible metadata floating around, and having 'cabal unpack' doing clever magic behind your back. The index would remain a pure function of the set of tarballs. Cheers, Simon

Duncan Coutts writes: [...]

I mentioned I'd written a feature to update the .cabal file we use when building with the same one as is in the package archive index. I've now pushed that to the head branch. I think it would be good to have on the 1.16 branch and included in the cabal-install release.

For context, as you know there's been a lot of discussion about dependency version bounds and if we should use conservative or optimistic bounds. Something that we've been thinking about for a while that would help here is if we could adjust the dependencies after a release.

For example if you used conservative bounds on a dependency and later on a new version of that dependency is released, and it so happens that you were lucky this time and your package does still build and work with the new version. It'd be nice if you (or some other helper monkey) could just adjust the version constraints to reflect reality.

Or the other way around, if you were using optimistic bounds and a new version of a dep is released and you were unlucky and it now fails, then again we can tighten the version bounds.

IMO, being able to modify .cabal server-side after the fact helps the situation with (overly) conservative version bounds, but it isn't practical for optimistic bounds, and here's why: As everybody knows, with conservative version bounds, the Hackage database is always in a "consistent" state by definition at all times, meaning that any combination of packages satisfying the respective inter-package version constraints is guaranteed to build properly[1]. The worst thing that can happen is that the solver can't find a set of compatible packages. When a new package is added to the Hackage database, this property/invariant is preserved (which is essential for having reproducible builds). With optimistic bounds on the other hand, not every combination of packages for which the inter-package constraint are satisfied is also guaranteed to build properly. In fact, it only takes one fundamental package to be uploaded with an incompatible API change to break almost every packages' compilation (until those are fixed by adding version bounds (which I claim doesn't work well) or adapting to the new incompatible API (which could lead to avalanche effects)). But let's assume a simpler scenario involving only two packages: 1) Hackage contains the package text-0.10 2) there's a package text-foo which depends on 'text >= 0.10' (i.e. with no upper bound), for which N versions exist on Hackage (all with the same 'text >= 0.10' constraint): 0.1.0, 0.1.0.1, 0.1.0.2, 0.1.1, 0.1.2, 0.2, ...some more... , 0.7.2.2, 0.7.2.3 3) now text-0.11 gets uploaded, which has an incompatible API change (for instance the function 'breakOn' which text-foo uses internally gets renamed to 'breakBy') 4) text-foo-0.7.2.3's .cabal file is updated server-side to have the version constraint set to 'text >= 0.10 && text < 0.11' Alas, 4) isn't enough, as it leaves N-1 versions of text-foo with an invalid version constraint, and if the cabal solver tries to find a compatible package of 'text-foo' it will simply backtrack to text-foo-0.7.2.2 which still has the optimistic 'text >= 0.10' constraint. In order to really fix 'text-foo' one would have to edit all its N version's .cabal files server-side. So what I'm trying to say is that the current design of the CABAL solver and the proposed server-side-.cabal edit feature lends itself better to relaxing version constraints than to restrict version constraints. Does this make any sense? Cheers, hvr [1] By "building properly" I mean that the build doesn't fail due to inter-package API incompatibilities.

Malcolm Wallace writes:

On 3 Dec 2012, at 11:35, Herbert Valerio Riedel wrote:

...

2) there's a package text-foo which depends on 'text >= 0.10' (i.e. with no upper bound), for which N versions exist on Hackage (all with the same 'text >= 0.10' constraint):

0.1.0, 0.1.0.1, 0.1.0.2, 0.1.1, 0.1.2, 0.2, ...some more... , 0.7.2.2, 0.7.2.3

4) text-foo-0.7.2.3's .cabal file is updated server-side to have the version constraint set to 'text >= 0.10 && text < 0.11'

Alas, 4) isn't enough, as it leaves N-1 versions of text-foo with an invalid version constraint

I don't understand why you suggest that only the most recently uploaded version of text-foo would have its cabal file updated on the server. If the scheme is going to work at all, it must update _every_ version of text-foo that could be affected.

(...in contrast to conservative-bounds, for which the integrity wouldn't break if one doesn't update *every* version of text-foo with relaxed bounds -- it would just artificially reduce the reachable space of compatible package/version combinations)

Is that an unrealistic expectation?

I don't know... I guess it depends on how much the process of making the version bounds stricter can be automatized, as otherwise it's a potentially tedious task that would make it easy to miss something, and thus not fully re-establish the inter-package version-constraint integrity. Some kind of automatization would also be important in order to keep the time window as small as possible when Hackage (in its current implementation) loses temporarily its inter-package integrity due to uploads of new packages with breaking API changes. cheers, hvr