Hi Simon, On Thu, Sep 5, 2013 at 9:15 AM, Simon Peyton-Jones wrote:

Can I ask what the Cabal team's position is with respect to the question of allowing the same package to be installed several times, each compiled against a different collection of dependencies?

I think that we all agree that in the long term a Nix-like package database is the ideal solution to the "Cabal hell" problem (I even mentioned this in the "Future Work" section of the post you linked). However, it seems to be much harder to implement than sandboxes. There was already a GSoC project on this topic last year, with mixed results [1][2] (there was some progress, but the patches never made it to GHC/Cabal). Personally, I'm myself interested in working on this in the future. [1] https://www.youtube.com/watch?v=h4QmkyN28Qs [2] http://ghc.haskell.org/trac/ghc/wiki/Commentary/GSoCMultipleInstances -- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

fyi, below you can find an expanded list of related links. I do appreciate the new sandboxing feature in 1.18 and I think it is a great feature, but it is not addressing the core issue here: adding support for multiple instances of the same version. It also seems to me that supporting this would be a relatively low hanging fruit that could have a big impact to the whole ecosystem once addressed. So why not address it? I don't have enough exposure to Ghc or Cabal to make a difference here, now (what is needed is an elaboration of the issues together with a minimal solution), but I would be willing to participate in a supporting role (testing, reviewing, and minor bug fixes). GSoC 2012 - Enable GHC to use multiple instances of a package for compilation - Philipp Schuster http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/phischu/1 http://www.google-melange.com/gsoc/project/google/gsoc2012/phischu/19001 https://github.com/phischu/cabal GHC Commentary http://ghc.haskell.org/trac/ghc/wiki/Commentary/Packages/MultiInstances http://ghc.haskell.org/trac/ghc/wiki/Commentary/Packages http://ghc.haskell.org/trac/ghc/wiki/Commentary/GSoCMultipleInstances Mailing list http://comments.gmane.org/gmane.comp.lang.haskell.ghc.devel/443 http://markmail.org/message/4qvegvx32lhlo66g#query:+page:1+mid:bwdgykv4g2hzq... On Thu, Sep 5, 2013 at 4:33 AM, Simon Peyton-Jones wrote:

| > Can I ask what the Cabal team's position is with respect to the | question of allowing the same package to be installed several times, | each compiled against a different collection of dependencies? | | I think that we all agree that in the long term a Nix-like package | database is the ideal solution to the "Cabal hell" problem (I even | mentioned this in the "Future Work" section of the post you linked). | However, it seems to be much harder to implement than sandboxes.

I have not read the full GSoC page (thanks for that link), but are you all convinced that it *needs* to be that hard?

The fundamental idea is, after all, so simple! (I could elaborate.) Perhaps the GSoc project was trying to be more ambitious.

Or maybe I'm simply under-estimating. But fundamentally it seems simple, so I'm suggesting we should perhaps try a bit harder to ferret out the underlying simplicity rather than assuming it must be hard.

Simon

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

On 2013-09-06 01:04, Johan Tibell wrote:

You raise a good point that sandboxing only addresses the issue of conflicts between two projects (that want to use different dependencies), but not the issue of actually using two different versions of a package (at different versions) in a single project*. The latter is valuable but also a bit tricky. We'd need to make sure that the two versions of the dependencies are private dependencies of some other dependencies. For example, it's OK to use two different versions of Parsec as long as you don't pass values from one version to the other.

Can't this be determined (conservatively) from the Exposed-modules? If you allow declaration of dependencies to be split into two -- one list of dependencies for the exposed-modules and one list of the internal modules of a package -- it seems it should be a lot easier to handle this. The package hash would be made up only of its Expose-modules and their dependencies. It would then be up to the author of the .cabal file to ensure that the set of dependencies of the Exposed-modules is minimal. IMO the *validation-only* portion of this would also be an excellent standalone feature -- it would let package authors ensure "hygiene" wrt. dependencies. (That is, you're only exposing what you're expecting and there's no way to acccidentally expose an implementation detail via accidental reuse of a dependency's types.). I know very little of GHC/Cabal internals, so I don't know how much work it would take to implement just the validation portion of this, but I think it might make a good self-contained first step towards a full implementation of private packages. Anyone with more knowledge of internals care to chime in? Regards,

| Let me start by saying that I'm in principle for moving to a Nix-style | package DB (i.e. multiple instances of the same package and version). ... | The problem is at least harder to solve than sandboxing. The reason I | had Mikhail implement sandboxes during GSoC is that I was skeptical | that we could make the Nix-solution work during a GSoC project, not | because the student didn't do a good job but because it's hard and | requires lots of expertise. This is the bit I still find it hard to get my head round. The basic idea is so simple, that I think if someone (not me, not Johan) had the time to think about it carefully, I bet there is a good path. But I could be wrong, and I have thought about it less than Johan. Simon | If we magically could find for Duncan and | Simon M to work on it, I think we could do it. I personally don't have | time to do any cabal work except for managing the project. | | The main downside with sandboxes compared to Nix-style package DBs is | that every new sandbox will induce one long build when all | dependencies are built. This takes a couple of minutes. After that | build, sandboxes and Nix packages should behave about the same | (correct me if I'm wrong). At that point incremental build times | matter more, as they make the edit-compile-test cycle longer. Avoiding | extra work (unnecessary linking) and doing things faster (more | parallelism) help here. | | Cheers, | Johan | | On Thu, Sep 5, 2013 at 12:15 AM, Simon Peyton-Jones | wrote: | > Can I ask what the Cabal team's position is with respect to the | question of allowing the same package to be installed several times, | each compiled against a different collection of dependencies? | > | > This is the problem that you built sandboxes to work around; the "What | are sandboxes and why are they needed" section of | http://coldwa.st/e/blog/2013-08-20-Cabal-sandbox.html describes the | problem well. | > | > It would be possible for Cabal to continue to work around deficiencies | in GHC, but wouldn't it be better for us to work together to fix the | underlying problem once and for all? | > | > Simon wrote a wiki page about this. I think this is it | > | http://ghc.haskell.org/trac/ghc/wiki/Commentary/Packages/MultiInstances | > though it's in a bit less detail than I expected, so there may be | something else. | > | > In my limited understanding, this single change would do more to | alleviate "cabal hell" so extensively written about than anything else | we can do. It would require changes in both Cabal and GHC. It can't be | that hard... shall we just do it? Isn't it more important, for our | users, than the other things Johan lists below? | > | > Simon | > | > | > | > | -----Original Message----- | > | From: cabal-devel [mailto:cabal-devel-bounces@haskell.org] On Behalf | Of | > | Johan Tibell | > | Sent: 05 September 2013 05:14 | > | To: cabal-devel@haskell.org | > | Subject: What's next? | > | | > | Hi all, | > | | > | With 1.18 out the door it's time to look towards the future. Here | are | > | the major themes I'd like to see us work on next: | > | | > | ## Faster builds | > | | > | There are several interesting things we could do and are doing here. | > | | > | * Avoid relinking if possible. This reduces incremental build times | > | (i.e. when you run cabal build after making some change) by avoiding | > | relinking e.g. all the test suites and/or benchmarks. See | > | https://github.com/haskell/cabal/pull/1410 for some in-progress | work. | > | | > | * Build components and different ways (e.g. profiling) in parallel. | > | We could build both profiling and non-profiling versions in | parallel. | > | We could also build e.g. all test suites in parallel. The key | > | challenge here is to coordinate all parallel jobs so we don't spawn | > | too many. See https://github.com/haskell/cabal/pull/1413 | > | | > | * Build modules in parallel. This fine granularity would let us | > | making building a single package faster, which is the most common | case | > | after all. There has been some GSoC work here e.g. | > | http://hackage.haskell.org/trac/ghc/ticket/910 and some work by | > | Mikhail. | > | | > | ## Do the right thing automatically | > | | > | The focus here should be on avoiding manual steps the cabal could do | > | for the user. | > | | > | * Automatically install dependencies when needed. When `cabal | build` | > | would fail due to a missing dependency, just install this dependency | > | instead of bugging the user to do it. This will probably have to be | > | limited to sandboxes where we can't break the user's system | > | | > | * GHCi support could be improved by rebinding :reload to rerun e.g. | > | preprocessors automatically. This would enable the users to develop | > | completely from within ghci (i.e. faster edit-save-type-error | cycle). | > | We have most of what we need here (i.e. GHC macro support) but | someone | > | needs to make the final change to generate a .ghci file to pass in | the | > | ghci invocation. | > | | > | ## Support large projects | > | | > | We need to better support projects with tens or hundreds of | packages. | > | As projects grow it's naturally to split them up into a bunch of | > | packages (libraries and executables). Some changes to the project | > | might need changes across a couple of packages (e.g. to a supporting | > | library and to an executable) in a single change and developers need | > | to be able to | > | | > | * build several packages that have local changes, and | > | * share changes to packages with other developers without making a | > | release to some local Hackage server every time. | > | | > | Both can be done by having a single source repo with all the | packages | > | and using `cabal sandbox add-source` to make sure they get built | when | > | needed. However, that method only scales up to a handful of | packages. | > | | > | I think we want to think about moving away from a world where all | > | cabal commands are run in the context of some "current" package | (i.e. | > | in the current working directory). I think we want to be able to | stand | > | in the root directory of some source repo and build things from | there. | > | Example: | > | | > | $ git clone git://my-company/repo my-project | > | $ cd my-project | > | $ ls | > | lib-pkg1 lib-pkg2 exe-pkg1 exe-pkg2 ... | > | $ cabal sandbox init # or something similar | > | $ edit lib-pkg1/Lib.hs | > | $ edit exe-pkg1/Exe.hs | > | $ cabal build exe-pkg1 # picks up changes to lib-pkg1 | > | | > | This has implication for many things e.g. where the .cabal-sandbo | and | > | the dist directories are kept. Perhaps dist would have a | subdirectory | > | per package (right now we do something similar for sandbox | > | dependencies). | > | | > | I imagine that the syntax for e.g. cabal build would have to | extended to | > | | > | cabal build [[DIR':']COMPONENT] | > | | > | Example: | > | | > | cabal build lib-pkg1:some-component | > | | > | Similar for `cabal test` and `cabal bench`. | > | | > | Cheers, | > | Johan | > | | > | _______________________________________________ | > | cabal-devel mailing list | > | cabal-devel@haskell.org | > | http://www.haskell.org/mailman/listinfo/cabal-devel

On Thu, Sep 5, 2013 at 12:06 PM, Iustin Pop wrote:

On Wed, Sep 04, 2013 at 09:14:03PM -0700, Johan Tibell wrote:

...

## Do the right thing automatically

The focus here should be on avoiding manual steps the cabal could do for the user.

* Automatically install dependencies when needed. When `cabal build` would fail due to a missing dependency, just install this dependency instead of bugging the user to do it. This will probably have to be limited to sandboxes where we can't break the user's system

I'm not sure if here by sandbox and break you mean break the cabal/package installation, or protect against malicious code.

If it's not the latter (and even if it is, how safe are the sandboxes?), I would argue that until cabal can verify authenticity of downloaded archives, it would be better to not do this automatically. Maybe add a new command, cabal fetch-deps or something like that, that can do it, but leave 'cabal build' as a "safe" command.

By break I mean break the package DB by forcefully re-installing a package. In a sandbox this is safe, as we have a single install plan for the whole sandbox and it's always safe to reinstall everything if need be. As for security I don't think this is much less secure than telling the user to type 'cabal install' manually. We better focus our security efforts on making sure we speak HTTPS to Hackage, validate uploads there, etc. For the extra security conscious we can add a `no-automatic-downloads` setting to ~/.cabal/config. -- Johan

On Sat, Sep 7, 2013 at 1:29 PM, Herbert Valerio Riedel wrote: sounds like https://github.com/haskell/cabal/pull/1455

That‘s great if they’re adding a warning for modules, but it'd be even better if it would warn about any missing files not just modules. For example, extra-source-files that are used for tests. That's probably hard to make a warning for without duplicating builds, but perhaps instead of a warning build should actually do sdist && build dist/*.tar.gz. Might be slow though.