Best approach to avoid dependency hells
Hello everyone, I've spent the last couple of days fighting my way around a dependency hell with my own libraries and packages. If I install them package by package (by hand), I'm very likely to hit one of these conflicts that I'm talking about. A simple example of something that did happen: - Package A depends on bytestring, and is compatible with both 0.9.* and 0.10.* - Package B depends on ghc, which I installed using the package manager and which depends on bytestring 0.9.2 - Package B depends on package A. I do not want to add to the cabal files that B depends on bytestring 0.9.*, because it's not true, or that A depends on bytstring 0.9, because it's also not true. I would like to the best approach to 1) avoid these situations and 2) fix them when they happen. Regarding (2), Is there any way to automatically rebuild all the broken packages? Regarding (1), I thought about using cabal-dev, but I think I'll just have the same problem, won't I? After all, I started from a clean fresh ghc installation and I just installed what I needed to make my program run. I also thought about adding process, bytestring and other global packages to the list of constraints in ~/.cabal/config. Would this be a good approach, or will it lead to bigger headaches in the future (for instance, in the form of programs that just cannot be installed at all)? Also, is there any way to detect potential dependency hells without having to install a package? (Some tool that will analyse the dependency information (statically) and determine more strict or incorrect upper and lower bounds for the dependencies of a package.) Cheers, Ivan
On Wed, Dec 5, 2012 at 10:52 AM, Ivan Perez
Hello everyone,
I've spent the last couple of days fighting my way around a dependency hell with my own libraries and packages.
If I install them package by package (by hand), I'm very likely to hit one of these conflicts that I'm talking about. A simple example of something that did happen: - Package A depends on bytestring, and is compatible with both 0.9.* and 0.10.* - Package B depends on ghc, which I installed using the package manager and which depends on bytestring 0.9.2 - Package B depends on package A.
I do not want to add to the cabal files that B depends on bytestring 0.9.*, because it's not true, or that A depends on bytstring 0.9, because it's also not true.
I would like to the best approach to 1) avoid these situations and 2) fix them when they happen.
Regarding (2), Is there any way to automatically rebuild all the broken packages? Regarding (1), I thought about using cabal-dev, but I think I'll just have the same problem, won't I? After all, I started from a clean fresh ghc installation and I just installed what I needed to make my program run.
Let's say you are working in a directory foo that contains a cabal file for package foo. When you type 'cabal-dev install', cabal-dev looks at foo.cabal in the current directory, it uses it to calculate constraints and then installs the dependencies locally that satisfy those constraints. It also has an install-deps command if you just want to install the deps but not build/install foo. Additionally, if you want to use locally modified versions of the dependencies use 'cabal add-source <path to dep>' from the foo directory to add your locally modified package to foo's index so that subsequent cabal-dev operations can find it and install it. Basically, don't install the deps by hand. Let cabal-dev figure out the mutually compatible versions and install that *locally* to where you're building foo. As for the problem of recursively rebuilding the deps, I think 'cab' [1] solves this but I haven't tried it. [1] http://hackage.haskell.org/package/cab
I also thought about adding process, bytestring and other global packages to the list of constraints in ~/.cabal/config. Would this be a good approach, or will it lead to bigger headaches in the future (for instance, in the form of programs that just cannot be installed at all)?
Adding these constraints will come back to bite you. That's been my experience.
Also, is there any way to detect potential dependency hells without having to install a package? (Some tool that will analyse the dependency information (statically) and determine more strict or incorrect upper and lower bounds for the dependencies of a package.)
--dry-run? Jason
On Wed, Dec 5, 2012 at 11:13 AM, Jason Dagit
Let's say you are working in a directory foo that contains a cabal file for package foo. When you type 'cabal-dev install', cabal-dev looks at foo.cabal in the current directory, it uses it to calculate constraints and then installs the dependencies locally that satisfy those constraints. It also has an install-deps command if you just want to install the deps but not build/install foo. Additionally, if you want to use locally modified versions of the dependencies use 'cabal add-source <path to dep>' from the foo directory to add your locally modified package to foo's index so that subsequent cabal-dev operations can find it and install it.
(minor clarity note: I don't believe there is a cabal add-source, but there is a cabal-dev add-source, which I think Jason meant above) cabal-dev add-source can cause problems, and it is not always clear what it does. Since ~cabal-install-0.10.2, it's also not (usually) necessary. A simpler workflow is to 'cabal-dev install' a list of packages, eg: $ cabal-dev install <path to dep> <path to dep> <path to foo> That should run the solver once, finding a global solution, and installing the required dependencies (and foo) into the local sandbox. You should only need this for packages that are not on hackage, or that you've modified and want to use local versions. --Rogan
Basically, don't install the deps by hand. Let cabal-dev figure out the mutually compatible versions and install that *locally* to where you're building foo.
As for the problem of recursively rebuilding the deps, I think 'cab' [1] solves this but I haven't tried it.
[1] http://hackage.haskell.org/package/cab
I also thought about adding process, bytestring and other global packages to the list of constraints in ~/.cabal/config. Would this be a good approach, or will it lead to bigger headaches in the future (for instance, in the form of programs that just cannot be installed at all)?
Adding these constraints will come back to bite you. That's been my experience.
Also, is there any way to detect potential dependency hells without having to install a package? (Some tool that will analyse the dependency information (statically) and determine more strict or incorrect upper and lower bounds for the dependencies of a package.)
--dry-run?
Jason
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
After using Haskell in production for awhile this is my go-to method for
minimizing pain when dealing with dependencies. As long as you follow these
steps you should be able to deal with most dependency issues.
1. NEVER use cabal, always use cabal-dev. Preferably with latest version
that supports parallelized builds with -j<cores>. It's just not worth
having your builds break all the time. Rebuilding the world isn't too bad
with -j12 and cabal won't trip up as much trying to resolve version
conflicts. Sometimes you will get a "this will break existing packages"
message when using cabal-dev but I think that's just because cabal is
confused, --force-reinstalls is your friend in this situation and shouldn't
really break anything because it's a sandboxed build.
2. Make a directory called lib in your project and drop any of your
dependencies in there, then your build rule will be the same for all your
projects:
cabal-dev install . ./lib*
This is handy because now you can drop any misbehaving packages in your lib
folder if you need to fix any version conflicts (although I don't find
myself doing this much)
3. Dealing with version conflicts:
In my experience most version conflicts are caused by unnecessary upper
bound constraits (4.*, && < 5.0). I'd recommend anyone writing packages to
only include these if you know the newer version actually breaks something
or causes unacceptable performance changes. All my projects only have lower
bounds (>= v) and I've never had any problems with dependency issues
between my packages.
If you get to this point and you still have a misbehaving package, simply
download the tarball from hackage and drop it in your lib folder, remove
upper bounds, and voila, it will work most of the time.
Cheers,
Bill
On Wed, Dec 5, 2012 at 1:52 PM, Ivan Perez
Hello everyone,
I've spent the last couple of days fighting my way around a dependency hell with my own libraries and packages.
If I install them package by package (by hand), I'm very likely to hit one of these conflicts that I'm talking about. A simple example of something that did happen: - Package A depends on bytestring, and is compatible with both 0.9.* and 0.10.* - Package B depends on ghc, which I installed using the package manager and which depends on bytestring 0.9.2 - Package B depends on package A.
I do not want to add to the cabal files that B depends on bytestring 0.9.*, because it's not true, or that A depends on bytstring 0.9, because it's also not true.
I would like to the best approach to 1) avoid these situations and 2) fix them when they happen.
Regarding (2), Is there any way to automatically rebuild all the broken packages? Regarding (1), I thought about using cabal-dev, but I think I'll just have the same problem, won't I? After all, I started from a clean fresh ghc installation and I just installed what I needed to make my program run. I also thought about adding process, bytestring and other global packages to the list of constraints in ~/.cabal/config. Would this be a good approach, or will it lead to bigger headaches in the future (for instance, in the form of programs that just cannot be installed at all)?
Also, is there any way to detect potential dependency hells without having to install a package? (Some tool that will analyse the dependency information (statically) and determine more strict or incorrect upper and lower bounds for the dependencies of a package.)
Cheers, Ivan
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
On 12-12-05 01:52 PM, Ivan Perez wrote:
I've spent the last couple of days fighting my way around a dependency hell with my own libraries and packages.
If I install them package by package (by hand), I'm very likely to hit one of these conflicts that I'm talking about. A simple example of something that did happen: - Package A depends on bytestring, and is compatible with both 0.9.* and 0.10.* - Package B depends on ghc, which I installed using the package manager and which depends on bytestring 0.9.2 - Package B depends on package A.
I do not understand the conflict. First you have GHC, and it comes with bytestring-0.9.2. Then one of two things happen, but I can't see a conflict either way: * You request A, but A should be fine with the existing bytestring-0.9.2. Then you request B, but B should be fine with the existing A you have, and indifferent about bytestring. OR * You request B. So cabal-install bring in A and then B. A should be fine with the existing bytestring-0.9.2, and then B should be fine with that A and indifferent about bytestring. If there is a conflict, it cannot be deduced from your data.
I would like to the best approach to 1) avoid these situations and 2) fix them when they happen.
To prevent or foresee problems, add --dry-run to your "cabal install" commands to see what packages are to be brought in, and check that list. If the list contains packages you already have, same version or different version, then it is a potential problem. It is fine with the pros of cabal matters, they know how to live with it, and they tell others it's fine. But it is not fine for non-pros, it causes confusion. I am fine with adding constraints such as "bytestring == the version that comes with GHC" to your "cabal install" commands and/or your ~/.cabal/config, contrary to what someone else says. The problem with setting it in ~/.cabal/config is that you will forget to change it when you change to another GHC version. But adding it manually as --constraint flags to "cabal install" may help with specific cases. How do you know whether it helps or hurts one case? Use --dry-run to find out. Play with adding and omitting constraints to see which list you like most. Your different projects can have different needs. This is why cabal-dev or other sandboxing methods are a good idea. This does not deny that some packages benefit all of your projects and can be installed outside sandboxes. You have to choose carefully. To clean up a mess or bad state, see my http://www.vex.net/~trebla/haskell/sicp.xhtml#remove Actually, see the whole thing. It does not suffice to erase ~/.cabal (and mostly unnecessary). It does not suffice to erase GHC and then install the same GHC version again. You are still missing an important piece of metadata.
On 8 December 2012 03:12, Albert Y. C. Lai
I do not understand the conflict. First you have GHC, and it comes with bytestring-0.9.2. Then one of two things happen, but I can't see a conflict either way:
* You request A, but A should be fine with the existing bytestring-0.9.2. Then you request B, but B should be fine with the existing A you have, and indifferent about bytestring.
When you install A, you may not know that you'll need to depend on a lower version of bytestring later on. Cabal will pick the highest version available (0.10 if present). If a program you install later on depends on A (needs bytestring-0.10) and ghc (needs bytestring-0.9), you'll have a conflict. Using cabal-dev seems like a good approach. Just yesterday I was able to install hoodle without problems (even though installing it with cabal would have broken many packages). The fact that you can unpack any package, fix the problems, add its path as source and install when other packages depend on it saved me a lot of time already.
On 12-12-08 07:39 AM, Ivan Perez wrote:
When you install A, you may not know that you'll need to depend on a lower version of bytestring later on. Cabal will pick the highest version available (0.10 if present). If a program you install later on depends on A (needs bytestring-0.10) and ghc (needs bytestring-0.9), you'll have a conflict.
You are saying, before "cabal install A", you already have both bytestring 0.9.2 and 0.10 installed. This is new data not revealed last time. And it is important, this is exactly the cause, you have too many bytestring's installed. (You cannot be saying, you start without bytestring 0.10, and "cabal install A" brings in bytestring 0.10. You start with GHC, and since it depends on bytestring 0.9.2, you start with bytestring 0.9.2 too. And you don't already have another bytestring. Then "cabal install A" will stick with bytestring 0.9.2.)
I started with a clean empty .cabal dir. Then I installed some program
that 1) depended on bytestring 0.10 and 2) I needed to compile another
library or program. That's when the problem started. Without using
cabal-dev, I see no easy way to avoid installing bytestring 0.10 when
I need a program that depends on it.
On 8 December 2012 16:32, Albert Y. C. Lai
On 12-12-08 07:39 AM, Ivan Perez wrote:
When you install A, you may not know that you'll need to depend on a lower version of bytestring later on. Cabal will pick the highest version available (0.10 if present). If a program you install later on depends on A (needs bytestring-0.10) and ghc (needs bytestring-0.9), you'll have a conflict.
You are saying, before "cabal install A", you already have both bytestring 0.9.2 and 0.10 installed. This is new data not revealed last time. And it is important, this is exactly the cause, you have too many bytestring's installed.
(You cannot be saying, you start without bytestring 0.10, and "cabal install A" brings in bytestring 0.10. You start with GHC, and since it depends on bytestring 0.9.2, you start with bytestring 0.9.2 too. And you don't already have another bytestring. Then "cabal install A" will stick with bytestring 0.9.2.)
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
participants (5)
-
Albert Y. C. Lai -
Bill Casarin -
Ivan Perez -
Jason Dagit -
Rogan Creswick