On Sat, Sep 11, 2010 at 21:17, Jason Dagit wrote:

On Sat, Sep 11, 2010 at 11:56 AM, Thomas DuBuisson wrote:

...

...

- is there a "specification" of which are the "core" packages?

From the FAQ linked by Paolo:

http://www.haskell.org/cabal/FAQ.html#dependencies-conflict

"To avoid this problem in the future, avoid upgrading core packages. The latest version of cabal-install has disabled the upgrade command to make it a bit harder for people to break their systems in this way."

I think that's what Paolo meant by "core" package.  Sadly the FAQ doesn't say what core means.  Nor is that page user editable.  I think "core" here must refer to packages that ghc is linked to.  For example, the process package in the example on the FAQ.

You are mostly correct, thanks for understanding. As said my case, "core package" extends to packages cabal is linked to. Still, cabal should know and protect them. "core package" is also used in the output of "cabal upgrade" (which in my release, 0.8.2 with Cabal 1.8.0.6, is disabled): Below is the list of packages that it would have tried to upgrade. You can use the 'install' command to install the ones you want. Note that it is generally not recommended to upgrade core packages. It is noteworthy that "cabal install --help" has no such warning. Anyway, package manager Gentoo and *BSDs can upgrade everything while the system is running. It would be nice to do the same here, including "cabal install ghc", but "if you want, you can fix it" would be an appropriate response - I kind-of know I won't get to do that, likely, at least not while starting my PhD.

I actually had this problem last weekend and I make a habit of never running 'cabal upgrade' and never installing things globally.  Yet some how on my system a package that ghc was built with did get upgraded and installed in my user package db.  It was causing various things to fail to configure.  If I recall correctly it was the directory package, but I'll use FOO as a place holder.  I used the suggested command line: ghc-pkg unregister --user FOO-X

ghc-pkg said it was ignoring the command because it would break packages. So then I tried adding --force.   At that point, ghc-pkg still said it was ignoring me and that I should use --force.  This was on ghc-6.12.1.  I tried it one more time with the --force option then ran ghc-pkg list FOO, and all instances of the FOO package were gone. At that point I could no longer configure any packages needing FOO. In the end I had to reinstall ghc so I took it as a chance to upgrade to 6.12.3.

Hmm... how comes that your global DB was user-writable? Luckily, it's owned by root here. Anyway, I consider most of the debugging I did quite challenging for non-Unix-guys, and reinstalling from scratch would have been surely faster. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

Hi Paolo, Paolo Giarrusso wrote:

$ cabal install --dry cabal-install leksah-0.8.0.6 [... does not work ...]

However, trying to install cabal-install and leksah separately works quite well.

So do install them separately. cabal install p1 p2 is supposed to find a single consistent install plan for p1 and p2 and the transitive dependencies of either of them. This is useful if you plan to use p1 and p2 in a single project. Tillmann

Hi Paolo, Paolo Giarrusso wrote:

...

cabal install p1 p2 is supposed to find a single consistent install plan for p1 and p2 and the transitive dependencies of either of them. This is useful if you plan to use p1 and p2 in a single project.

Ahah! Then it's a feature. The need for consistency stems from a bug: in a tracker entry you linked to, http://hackage.haskell.org/trac/hackage/ticket/704, duncan argues that "we also want to be able to do things like linking multiple versions of a Haskell package into a single application".

I think this is a slightly different matter. Consider a package pair, which defines an abstract datatype of pairs in version 1: module Pair (Pair, fst, snd, pair) where data Pair a b = Pair a b fst (Pair a b) = a snd (Pair a b) = b pair a b = Pair a b In version 2 of pair, the internal representation of the datatype is changed: module Pair (Pair, fst, snd, pair) where data Pair a b = Pair b a fst (Pair b a) = a snd (Pair b a) = b pair a b = Pair b a Now we have a package foo which depends on pair-1: module Foo (foo) where import Pair foo = pair 42 '?' And a package bar which depends on pair-2: module Bar (bar) where import Pair bar = fst Now, we write a program which uses both foo and bar: module Program where import Foo import Bar main = print $ bar $ foo Even with the technical ability to link all of foo, bar, pair-1 and pair-2 together, I don't see how this program could be reasonably compiled. Therefore, I think that the notion of consistent install plans is relevant semantically, not just to work around some deficiency in the linking system. Tillmann

On 13 September 2010 20:54, Paolo Giarrusso wrote:

On Sun, Sep 12, 2010 at 20:46, Tillmann Rendel wrote:

...

Paolo Giarrusso wrote:

...

in a tracker entry you linked to, http://hackage.haskell.org/trac/hackage/ticket/704, duncan argues that "we also want to be able to do things like linking multiple versions of a Haskell package into a single application". [snip] Even with the technical ability to link all of foo, bar, pair-1 and pair-2 together, I don't see how this program could be reasonably compiled. Therefore, I think that the notion of consistent install plans is relevant semantically, not just to work around some deficiency in the linking system.

Your case is valid, but OTOH there other cases to support: if I link together two programs which use _internally_ different versions of regex packages, cabal should support that - and here I guess we agree.

Paolo, If I've understood correctly, in this series of emails you're pointing out two problems: 1. upgrading packages can break dependencies (and Cabal does not do a lot to prevent/avoid this) 2. cabal ought to allow using multiple versions of a single package in more circumstances than it does now Both of these issues are known to the Cabal hackers (i.e. me and a few other people). I'll share my views on the problem and the solution. 1. This is certainly a problem. The current situation is not awful but it is a bit annoying sometimes. We do now accurately track when packages get broken by upgrading dependencies so it should not be possible to get segfaults by linking incompatible ABIs. My preferred solution is to follow the example of Nix and use a persistent package store. Then installing new packages (which includes what people think of as upgrading) become non-destructive operations: no existing packages would be broken by an upgrade. It would be necessary to allow installing multiple instances of the same version of a package. If we do not allow multiple instances of a package then breaking things during an upgrade will always remain a possibility. We could work harder to avoid breaking things, or to try rebuilding things that would become broken but it could never be a 100% solution. 2. This is a problem of information and optimisitic or pesimistic assumptions. Technically there is no problem with typechecking or linking in the presense of multiple versions of a package. If we have a type Foo from package foo-1.0 then that is a different type to Foo from package foo-1.1. GHC knows this. So if for example a package uses regex or QC privately then other parts of the same program (e.g. different libs) can also use different versions of the same packages. There are other examples of course where types from some common package get used in interfaces (e.g. ByteString or Text). In these cases it is essential that the same version of the package be used on both sides of the interface otherwise we will get a type error because text-0.7:Data.Text.Text does not unify with text-0.8:Data.Text.Text. The problem for the package manager (i.e. cabal) is knowing which of the two above scenarios apply for each dependency and thus whether multiple versions of that dependency should be allowed or not. Currently cabal does not have any information whatsoever to make that distinction so we have to make the conservative assumption. If for example we knew that particular dependencies were "private" dependencies then we would have enough information to do a better job in very many of the common examples. My preference here is for adding a new field, build-depends-private (or some such similar name) and to encourage packages to distinguish between their public/visible dependencies and their private/invisible deps. Duncan

On the topic of cabal odisseys: I think it would help to document (prominently) what Cabal fundamentally doesn't (try to) do, to avoid optimistic expectations (and hence the surprises when Cabal doesn't meet those expectations), and to point out the design choices behind many bug tickets (even if the choices are temporary and driven by limited manpower). Such as - cabal doesn't keep track of what it installs, hence - no uninstall - no application tracking - library tracking and information about installed library configurations only via ghc-pkg .. - cabal's view of package interfaces is limited to explicitly provided package names and version numbers, hence - no guarantee that interface changes are reflected in version number differences - no detailed view of interfaces (types, functions, ..) - no reflection of build/configure options in versions - no reflection of dependency versions/configurations in dependent versions - no knowledge of whether dependencies get exposed in dependent package interfaces .. This is just to demonstrate the kind of information I'd like to see - Duncan&co know the real list, though they don't seem to have it spelled out anywhere? So users see that it works to say "cabal install" and build up their own often optimistic picture of what (current) Cabal is supposed to be able to do. It might even be useful to have a category of "core-tickets" or such on the trac, to identify these as work-package opportunities for hackathons, GSoC and the like, positively affecting many tickets at once. On to the specific issue at hand:

2. cabal ought to allow using multiple versions of a single package in more circumstances than it does now .. 2. This is a problem of information and optimisitic or pesimistic assumptions. Technically there is no problem with typechecking or linking in the presense of multiple versions of a package. If we have a type Foo from package foo-1.0 then that is a different type to Foo from package foo-1.1. GHC knows this.

So if for example a package uses regex or QC privately then other parts of the same program (e.g. different libs) can also use different versions of the same packages. There are other examples of course where types from some common package get used in interfaces (e.g. ByteString or Text). In these cases it is essential that the same version of the package be used on both sides of the interface otherwise we will get a type error because text-0.7:Data.Text.Text does not unify with text-0.8:Data.Text.Text.

The problem for the package manager (i.e. cabal) is knowing which of the two above scenarios apply for each dependency and thus whether multiple versions of that dependency should be allowed or not. Currently cabal does not have any information whatsoever to make that distinction so we have to make the conservative assumption. If for example we knew that particular dependencies were "private" dependencies then we would have enough information to do a better job in very many of the common examples.

My preference here is for adding a new field, build-depends-private (or some such similar name) and to encourage packages to distinguish between their public/visible dependencies and their private/invisible deps.

This private/public distinction should be inferred, or at least be checked, not just stated. I don't know how GHC computes its ABI hashes - are they monolithic, or modular (so that the influence of dependencies, current package, current compiler, current option settings, could be extracted)? Even for monolithic ABI hashes, it might be possible to compute the package ABI hash a second time, setting the versions of all dependencies declared to be private to 0.0.0 and seeing if that makes a difference (if it does, the supposedly private dependency leaks into the package ABI, right?). And secondly, how about making it possible for cabal files to express sharing constraints for versions of dependencies? To begin with, there is currently no way to distinguish packages with different flag settings or dependency versions. If I write a package extended-base, adding the all-important functions car and cdr to an otherwise unchanged Data.List, that package will work with just about any version of base (until car and cdr appear in base:-), but the resulting packages may not be compatible, in spite of identical version numbers. If it was possible to refer to those package parameters (build options and dependency versions), one could then add constraints specifying which package parameters ought to match for a build configuration to be acceptable. Let us annotate package identifiers with their dependencies where the current lack of dependency and sharing annotations means "I don't care how this was built". Then build-depends: a, regex means "I need a and regex, but I don't care whether a also uses some version of regex", while build-depends: a, regex sharing: a(regex)==regex would mean "I need any version of a and regex, as long as a depends on same the version of regex I depend on" (choose any syntax that works). And build-depends: a, b sharing: a(text)==b(text) would mean "I need any version of a and b, as long as they both depend on the same version of text". I can already think of situations where one might want more complex constraints (e.g. "I want any version of base and mtl, but either both have to be before the Either-move, or both have to be after the move" - this part could be expressed already, but there is currently no way to enforce this transitively, for dependencies, so if I depend on c, which depends on a new base, and on d, which depends on an old mtl, I'm out of luck, I think). Or: "never mix mtl and its alternatives". Of course, this scheme depends on whether it is possible to reverse engineer the information about dependencies of installed packages from ghc-pkg info or ABI hashes.. but if the general idea is sound, perhaps that could be made possible? Just a suggestion, Claus

On Sep 13, 1:52 pm, Ross Paterson wrote:

On Sat, Sep 11, 2010 at 12:17:27PM -0700, Jason Dagit wrote:

...

"To avoid this problem in the future, avoid upgrading core packages. The latest version of cabal-install has disabled the upgrade command to make it a bit harder for people to break their systems in this way."

It's not always possible.  In particular, random-1.0.0.2 (shipped with GHC 6.12.*) depends on the time package, of which more recent versions have been released.  That can trigger rebuilding of random-1.0.0.2, and thus haskell98-1.0.1.1.

My case was similar, old-locale had been upgraded. And currently, haskell98 would not necessarily be rebuilt automatically - it would just break. The point is that time should _not_ be upgradeable with the current system, for the same reason as "cabal upgrade" is disabled. The alternative is to allow Cabal to rebuild itself _and_ GHC (and everything) safely. BTW, I just realized that thanks to static linking, the ghc and cabal binaries should never stop working - only using those libraries for building packages could break. If you always upgrade locally, you can always remove packages from the user DB as I did.

It might help if the release of random with GHC 7.0 had a tight dependency on the version of the time package shipped with it.  Maybe all the core packages need tight dependencies.

If you mean it as a hack for cabal's brokenness, it could be OK, but otherwise than that, I believe it's a bad idea. GHC itself does not depend on any specific package; it's the compiled GHC package which has a tight dependency. So I don't like the concept. Furthermore, if a GHC release (whichever it is) had a tight dependency, that would be annoying for when you want to compile that release against different libraries - if that is supposed to work. But actually, on my system I can't see ghc in cabal's DB; I can see that in the ghc-pkg database for binaries, but there all dependencies are tight, probably because they are dependencies among installed packages. Have a look: $ cabal info QuickCheck [...] Dependencies: mtl -any, base >=4 && <5, random -any, base >=3 && <4, random -any, base <3, ghc -any, extensible- exceptions -any $ ghc-pkg describe QuickCheck | grep depends depends: base-3.0.3.1 random-1.0.0.1 Best regards -- Paolo Giarrusso, PhD student http://www.informatik.uni-marburg.de/~pgiarrusso/

On Sat, Sep 11, 2010 at 22:29, Paolo Giarrusso wrote:

On Sat, Sep 11, 2010 at 21:43, Daniel Fischer wrote:

...

...

I had a broken dependency which I could see with "grep" but not otherwise.

...

ghc-pkg check?

I tried that, but it didn't notice any breakage.

As far as I understand, a package won't depend on packages which were hidden during its build, and all available packages will be recorded as dependencies in the ghc-pkg DB. Yet, for some reason this did not work. Sorry for my phrasing - I was explaining my understanding of how dependencies are discovered, but it was unclear.

I wanted to add is that I discovered this with an error similar to: <command line>: unknown package old-locale-1.0.0.2 where the mentioned module had just been unregistered. BTW, there's no obvious way to re-register a module because the file to pass to ghc-pkg register is not saved on disk. To debug it, I used cabal install -v, then I had to manually invoke ghc -v (because cabal install didn't pass -v to ghc), then finally used grep because the command line did not mention the offending module. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

On Sep 12, 2:51 am, wren ng thornton wrote:

On 9/11/10 3:43 PM, Daniel Fischer wrote:

...

...

- is there a "specification" of which are the "core" packages?

...

"core" as in *do not update*? Basically, what comes with GHC shouldn't be updated. Though I heard updating Cabal was okay.

I tried updating Cabal once (recently) and it broke things in the same way. FWIW.

My Cabal was upgraded and is alive, but the bugs we see don't make the upgrade reliable. Most variations of the trap I hit would have killed cabal, period. Some could have killed GHC. And even for my case, resurrecting my old friend Cabal after his attempted suicide was not for the faint-heart, and took much more time than letting him die and replacing him with some other friend, i.e. reinstalling from scratch the user packages - I just never allow my programmed friends to die, if I can avoid that. I value their liveness :-D. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

Hi all, I would like to share a new very cool result given by the current Cabal: if packages FOO, BAR and FOOBAR exist, such that FOO depends on BAR which depends on FOOBAR, and all three are installed, you just can't safely upgrade FOOBAR. When the same version of BAR is ever recompiled against the new FOOBAR, its ABI might change and FOO _will_ thus break if you're not very lucky. Call it the No Upgrade Theorem, if you want. Note that upgrades of FOO and BAR can still be safe: the key hypothesis is a dependency chain of 3 packages, and the thesis is that the deepest of the three ones in the chain (FOOBAR in the above example) can't be upgraded safely. Not that it's necessarily Cabal's "fault", IMHO. Given this kind of cross-module inlining, proper dependency handling (which includes what we discussed) seems insanely complicated to get right. Of course, Cabal can be fixed, but I would call that a major achievement. On Sep 13, 12:53 am, Brandon S Allbery KF8NH wrote:

On 9/11/10 15:43 , Daniel Fischer wrote:

...

On Saturday 11 September 2010 20:38:21, Paolo Giarrusso wrote:

...

- is there a "specification" of which are the "core" packages?

...

"core" as in *do not update*? Basically, what comes with GHC shouldn't be updated. Though I heard updating Cabal was okay.

Some consistency would be nice; IIRC GHC refers to them as "boot libraries".

"Boot libraries" clearly refers to GHC bootstrapping. But with a statically linked GHC (like on my system at least, and maybe everywhere), GHC dependencies are irrelevant, only "transitive dependency closure of (those used by Cabal + those used by any programs (base, ghc-prim, rts, integer...))" are relevant. Which includes an additional package, "pretty", and excludes many other. To verify this theory, I upgraded "bytestring" (on which GHC but not Cabal depend), and I can still configure+build+copy packages. But of course, bytestring fulfills the hypothesis of the above theorem, so installing a package with enough dependencies might trigger actual breakage. That's however not because ghc depends on bytestring. Only Cabal is special, because if you break that you can't recompile anything any more, and because recompiling Cabal dependencies and Cabal takes special effort (as said elsewhere). That's why "core packages" should include just cabal (recursive) deps. Fixing the "No Upgrade Theorem" is easier, just recompile more stuff and/or give more warnings. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

Hi! First, sorry for some confusion - I wanted to post further details needed for the analysis in the mail I lost, and I had excluded them from my summary to keep it short. On Sun, Sep 12, 2010 at 15:26, Tillmann Rendel wrote:

Hi Paolo,

Paolo Giarrusso wrote:

...

- when recompiling a package with ABI changes, does cabal always update dependent packages?

It never recompiles them. Recompilation should not be needed, because different versions of packages exports different symbols, so a package can never be linked against the wrong version of its dependency.

= the one they needed. An "Hello World program", using no recent functions, could thus be backportable even to an earlier version of

My problem was with reinstalling the same version of a package. Again, sorry for the confusion (see above). cabal should then recompile (not "update", strictly speaking) dependent packages: Recompiling the same version of a package does not always yield the same result, ABI-wise. The problem was an ABI difference between two compilation results for old-time-1.0.0.2, one linked against old-locale-1.0.0.1, the other against version 1.0.0.2 of the same package. The ABI difference was _probably_ due to different cross-module inlining decisions: the disappeared symbols were: oldzmtimezm1zi0zi0zi2_SystemziTime_a149_{closure,info}, i.e. (I assume) old-time-1.0.0.2-System.Time.a149. That name is generated by GHC during optimization, and my educated guess is that it is exported to allow cross-module inlining. In particular, a149 is mentioned in the .hi interface of old-time's System.Time - <libs dir>/old-time-1.0.0.2/System/Time.hi It thus _seems_ that the ABI of a module is a (pure) function of the versions of all its (transitive) dependencies, and their compilation options - in particular, the optimization options and the used compiler version. More formally, my conjecture is: - let us represent "package a depends on b" as "a =>> b", where a, b are package names tagged with a version - let =>>* be the transitive-reflexive closure of =>> we need to compute: DEPS(p) = {q | p =>>* q } TAGGED_DEPS(p) = { (q, compilation_opts(q)) | q \in DEPS(p) } where compilation_opts(q) are the compilation options used to compile package q. Then, the ABI of a module is a pure function of TAGGED_DEPS(p), not just of p. My experience proves at least that the ABI does not depend just on p. And since, after the discussion up-to-now, it turns out that this is unexpected, I reported this as a bug: http://hackage.haskell.org/trac/hackage/ticket/738 Another result of the above is that a mere "cabal install --reinstall -O2 FooPackage", recompiling the same version of FooPackage with -O2 instead of the default (-O1), requires recompiling all packages which depends on (i.e. use symbols from) FooPackage; recompiling them implies recompiling packages depending on them, and so on, in a "transitively closed" way. A final result is that shipping Haskell libraries in binary form (i.e. closed source libraries) is not supported without major changes. Not that I care directly, but somebody might, especially since you choose the BSD license over the GPL one. And fixing that might also make package upgrades more "sustainable", i.e. allowing to upgrade a library while rebuilding less dependent modules. To ship a binary library, one would need: - disabling cross-module inlining when building the closed source library, but that's not enough: one still needs to link to a specific version of a package, because the version number is mangled into the name, up to the 4th version level. That's inconvenient, because a library can't benefit from a change affecting only the implementation, and not the interface, of a library (e.g., a bugfix or a security fix). == On .NET, changes to the last version number must be binary-compatible, so 1.0.0.1 and 1.0.0.2 are equivalent at link time, and 1.0.0 and 1.0.1 are incompatible but can coexist. == On ELF, for libraries which care (say glibc), each symbol is available (with the same ABI) from a given library version onwards; client binaries depend on at dynamic linking time on a library version the C library. But I don't propose that, it requires enormous effort. Most other libraries use a Scheme which is similar to the .NET one, barring some big but superficial differences. - resorting to full static linking, if that's supported; even then, one can't statically link to the runtime system, and typechecking needs to use versioned typenames.

However, see the following tickets:

 http://hackage.haskell.org/trac/hackage/ticket/701  http://hackage.haskell.org/trac/hackage/ticket/704

Had a look, thanks - but they do not apply here, the problem is with Haskell symbols.

...

Interestingly, HTTP, directory, process, zip-archive were not reinstalled, which confirms that Cabal had reinstalled them before just because of an upgrade to the dependencies.

I think you are misinterpreting this.

When you asked cabal-install to install pandoc, it tried to make a consistent install plan for all its transitive dependencies. cabal-install will not touch a package which is not a transitive dependency of the package you request to be installed. Therefore, cabal-install will not touch Cabal if you ask it to install pandoc.

Thank you very much for the explanation. Note that the policy you describe is not necessarily "the right one", it is just a choice which can work, with the fixes you propose (read on for alternatives).

To make a consistent install plan, cabal-install has to pick exactly one version number for each of the transitive dependencies, so that all version constraints of all transitive dependencies are fullfilled. For some reason, cabal-install picked old-locale-1.0.0.2 instead of the already installed old-locale-1.0.0.1, and newer versions of HTTP, directory etc. too.

My case was slightly different, old-locale-1.0.0.2 (user-level) and .1 (system level) were already present. old-time-1.0.0.2 was installed at the system level, and linked against the older, system-level old-locale. So, while no dependency was broken, old-locale had been upgraded, by me or by cabal on its own. Even if it was my fault, only "cabal upgrade" gives a warning, while "cabal install --help" doesn't mention the issue, which is "suboptimal" :-D. And of course, cabal should prevent a user from shooting himself in his foot, but that's a more advanced feature. A consistent plan for installing pandoc would have been to either compile it using the older old-locale, or to recompile old-time against the new version. Sadly, cabal chose the latter alternative, since it is not obviously wrong, and this broke everything, except packages on which pandoc depended: they were recompiled because of the changed dep. I later forced the other install plan, with some pain, but it works. So cabal behavior is inconsistent: it knows that recompiling a package means recompiling its dependencies, but it should either do so also for packages unrelated to the one being installed, or it should not recompile any package (as you suggest).

I think this is the bug: cabal-install should not be allowed to install old-locale, because doing so apparantly causes havoc.

Lacking a proper dependency handling (which _is_ not trivial), yes. And it's the best current choice. But one can recompile and upgrade on a running system, without downtime nor glitches, both the C standard library and libraries without stable ABIs (see Gentoo Linux). Thus, upgrading old-locale and automatically rebuilding its dependencies would be a sensible feature. It just should not happen unless the user requests a full upgrade. Finally, while old-locale is a core package and one can restrict its upgrades, if I recompile non-core package A, and non-core package B breaks, that's still "havoc" - the only difference is that B is not cabal, and the bug is thus less severe and you can recompile manually B. OTOH, fixing this less severe and more complicated bug, via transitively closed dependency tracking as described above, helps supporting complete upgrades.

Looking at the inter-dependencies of pandoc's transitive dependencies, I do not see a reason to install a new version of a package instead of keeping the old.

Maybe it's somehow related to the transition from base-3 to base-4? The newer old-locale was already installed - see above my description of the two possible install plans.

For the transition, I had added: preference: base >= 4 based on a suggestion from somebody on haskell-cafe and the resulting discussion. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/