- cabal-devel - Haskell.org

patch applied (hackage-server): "Add 'resetState' to StateComponent" and 2 others
by devnull＠community.haskell.org 27 Nov '12

27 Nov '12

Tue Nov 27 10:11:58 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Add 'resetState' to StateComponent Ignore-this: a4dcd475cc01e6593f8fe735cfed171 resetState has type StateComponent -> BlobStorage -> FilePath -> IO (StateComponent st) and basically creates a new state component with the initial state. This is now used consistently throughout the initFooFeature functions to initialize the state, so that many initFeature methods now simply look like initFooFeature <some params> = do st <- fooStateComponent .. return (fooFeature <params> st) fooStateComponent = do st <- openLocalStateFrom ... return StateComponent { ... , resetState = fooStateComponent } This is a nicer separation of concerns, but more importantly, this is a stepping stone towards fixing 'testRoundtrip'. M ./Distribution/Server/Features/BuildReports.hs -19 +15 M ./Distribution/Server/Features/Check.hs -18 +14 M ./Distribution/Server/Features/Core.hs -18 +17 M ./Distribution/Server/Features/Distro.hs -18 +14 M ./Distribution/Server/Features/Documentation.hs -24 +19 M ./Distribution/Server/Features/DownloadCount.hs -19 +16 M ./Distribution/Server/Features/HaskellPlatform.hs -18 +15 M ./Distribution/Server/Features/Mirror.hs -13 +14 M ./Distribution/Server/Features/PreferredVersions.hs -21 +16 M ./Distribution/Server/Features/Tags.hs -19 +16 M ./Distribution/Server/Features/Upload.hs -41 +43 M ./Distribution/Server/Features/Users.hs -26 +28 M ./Distribution/Server/Framework/Feature.hs -1 +24 Tue Nov 27 15:32:33 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Remove testRoundtrip Ignore-this: 64252fffcbd74c025d4eedb862003847 StateComponent now provides enough informaiton to define testRoundtrip generically for all features (just equality on states). This is possible because the check for *blobs* will not be necessary anymore, as we will take care of this generically, too (we don't yet, though). This now throws an error in test-backup in the import/export for Core; presumably this error was always there, we just didn't notice it becauset testRoundtrip was set up incorrectly. (I think the error is not so important, rounding error for dates.) M ./Distribution/Server.hs -48 +11 M ./Distribution/Server/Features.hs -2 +2 M ./Distribution/Server/Features/BuildReports.hs -2 +1 M ./Distribution/Server/Features/BuildReports/Backup.hs -7 +1 M ./Distribution/Server/Features/Check.hs -2 +1 M ./Distribution/Server/Features/Check/State.hs -1 +1 M ./Distribution/Server/Features/Check/Types.hs -1 +1 M ./Distribution/Server/Features/Core.hs -12 +1 M ./Distribution/Server/Features/Distro.hs -3 +1 M ./Distribution/Server/Features/Documentation.hs -10 +2 M ./Distribution/Server/Features/DownloadCount.hs -2 +1 M ./Distribution/Server/Features/HaskellPlatform.hs -2 +1 M ./Distribution/Server/Features/HaskellPlatform/State.hs -1 +1 M ./Distribution/Server/Features/Mirror.hs -3 +1 M ./Distribution/Server/Features/Packages.hs -1 M ./Distribution/Server/Features/PreferredVersions.hs -2 +1 M ./Distribution/Server/Features/ServerApiDoc.hs -1 +1 M ./Distribution/Server/Features/Tags.hs -2 +1 M ./Distribution/Server/Features/Tags/Backup.hs -1 M ./Distribution/Server/Features/Upload.hs -6 +3 M ./Distribution/Server/Features/Upload/State.hs -2 +2 M ./Distribution/Server/Features/Users.hs -4 +2 M ./Distribution/Server/Framework/BackupDump.hs -38 +2 M ./Distribution/Server/Framework/BackupRestore.hs -8 M ./Distribution/Server/Framework/Feature.hs -16 +84 M ./Distribution/Server/Users/State.hs -2 +2 M ./Main.hs -19 +32 Tue Nov 27 15:49:02 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Include fractions of seconds during export Ignore-this: cce3a160b0605067d9319256a9f7df5f This fixes the roundtrip test for Core M ./Distribution/Server/Framework/BackupRestore.hs -1 +1

1 0

patch applied (hackage-server): "Make import pure for Tags"
by devnull＠community.haskell.org 26 Nov '12

26 Nov '12

Mon Nov 26 14:29:41 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Make import pure for Tags Ignore-this: 3f973d3f9708e04bbb9289fc426325fa M ./Distribution/Server/Features/Tags/Backup.hs -20 +21 M ./Distribution/Server/Framework/BackupRestore.hs -5 +38

1 0

patch applied (hackage-server): "Fix the 'restore' process"
by devnull＠community.haskell.org 26 Nov '12

26 Nov '12

Mon Nov 26 14:30:48 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Fix the 'restore' process Ignore-this: 5199ad4a1f430bd4c338f42054ecd0b2 - Not all features implemented 'restore' after the refactoring, causing restore to fail - 'importTar' assumed a single RestoreBackup per feature, while we now have multiple (though, thankfully, they implemented Monoid) Note that 'testRoundtrip' is still broken, but it turns out it was already broken even before the refactoring. The problem is that testRoundtripByQuery returns a check which compares the state of the database using the AcidState handle which corresponds to the *initial database* (before the export and reimport), and hence is comparing a database to itself and always returns success. It should instead use the handle created after the re-import, but that handle is not yet available when 'testRoundtrip' is called, so this requires some re-engineering of how tests are set up. Will fix this separately. M ./Distribution/Server.hs -6 +11 M ./Distribution/Server/Features/Check.hs +1 M ./Distribution/Server/Features/HaskellPlatform.hs -5 +6 M ./Distribution/Server/Features/PreferredVersions.hs +1 M ./Distribution/Server/Framework/BackupRestore.hs +1 M ./Main.hs -2 +4

1 0

patch applied (hackage-server): "Remove obsolete code" and 2 others
by devnull＠community.haskell.org 23 Nov '12

23 Nov '12

Thu Nov 22 16:25:53 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Remove obsolete code Ignore-this: 80f0a57579216f3bad55e8159b30a3e1 M ./Distribution/Server/Features/ServerApiDoc.hs -8 Fri Nov 23 12:48:44 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Make export pure Ignore-this: d7404f43bbb30bf13791b6fb75657ffb The only reason export lived in IO was because it needed access to the blob store. Now instead we define data BackupEntry = BackupByteString [FilePath] ByteString | BackupBlob [FilePath] BlobId This has various other advantages: we now have the option of storing blob entries differently (or indeed not at all) when we generate the tar file. For now, the export format is still the same, but at least we have the option now of treating blobs differently. Note that there was already an ExportEntry (as opposed to BackupEntry) type which was isomorphic to the (new) BackupEntry type. I have removed that type completely, and removed some of the conversions between them. M! ./Distribution/Server.hs -2 +4 M! ./Distribution/Server/Features/BuildReports.hs -1 +2 M! ./Distribution/Server/Features/BuildReports/Backup.hs -12 +8 M! ./Distribution/Server/Features/Check.hs -2 +3 M! ./Distribution/Server/Features/Core.hs -9 +6 M! ./Distribution/Server/Features/Core/Backup.hs -20 +20 M! ./Distribution/Server/Features/Distro.hs -1 +2 M! ./Distribution/Server/Features/Distro/Backup.hs -5 +4 M! ./Distribution/Server/Features/Documentation.hs -5 +5 M! ./Distribution/Server/Features/DownloadCount.hs -2 +2 M! ./Distribution/Server/Features/DownloadCount/Backup.hs -1 +1 M! ./Distribution/Server/Features/HaskellPlatform.hs -1 +2 M! ./Distribution/Server/Features/Mirror.hs -5 +5 M! ./Distribution/Server/Features/PreferredVersions.hs -2 +3 M! ./Distribution/Server/Features/Tags.hs -2 +2 M! ./Distribution/Server/Features/Tags/Backup.hs -1 +1 M! ./Distribution/Server/Features/Upload.hs -12 +12 M! ./Distribution/Server/Features/Upload/Backup.hs -1 +1 M! ./Distribution/Server/Features/Upload/State.hs -4 +12 M! ./Distribution/Server/Features/Users.hs -13 +13 M! ./Distribution/Server/Framework/BackupDump.hs -31 +22 M! ./Distribution/Server/Framework/BackupRestore.hs -8 +12 M! ./Distribution/Server/Framework/Feature.hs -1 +2 M! ./Distribution/Server/Users/Backup.hs -15 +16 M! ./Distribution/Server/Users/State.hs -4 +12 Fri Nov 23 12:52:53 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Resolve merge conflicts Ignore-this: 7f0f37fcf807173ca8e571ac92e12f5b M ./Distribution/Server/Features/PreferredVersions.hs -1 +1

1 0

Re: Cabal and GHC
by Duncan Coutts 23 Nov '12

23 Nov '12

On 22 November 2012 10:39, Michael Snoyman <michael(a)snoyman.com> wrote: > > On Thu, Nov 22, 2012 at 10:32 AM, Simon Peyton-Jones <simonpj(a)microsoft.com> > wrote: >> >> Michael Snoyman wrote a blog post about Solving Cabal Hall, which came >> across via the peerless Haskell Weekly News. >> http://www.yesodweb.com/blog/2012/11/solving-cabal-hell >> The solution is obvious: we should make it possible to instally >> yesod-platform-2.7 twice, > Having GHC support multiple copies of the same package/version would > certainly be a step in the right direction. It still wouldn't completely > address the goals I'm getting at in my blog post. What I'm really aiming for > overall is to provide users with a stable, vetted set of packages that are > known to work well together. But for those of us who will still be using > plain Hackage, this improvement would be very helpful. Right, and we need both. Your suggestion is very sensible. It's similar to the suggestion people have made before about having "testing" and "stable" sets of packages (these suggestions often come from people using debian terminology), and managing things like a distribution. In both cases the point is to provide something to end users where we've set up the available packages such that we know most things work together. (This is why debian users don't run into problems most of the time, because volunteers have done a huge amount of work behind the scenes to eliminate conflicting versions). So yes we need that, and it will require a fair bit of work, and we need to distribute that amongst many people. I think eventually the best mechanism to do it will be to use hackage and use tagging, but we can certainly experiment now by using separate hackage instances. But we also need to address the problems of developers in the thick of things, who must work with the latest and greatest unsable versions, before people have got round to smoothing out all the conflicting dependencies and patching packages so they can all use the same versions of common dependencies. That's where the nix-style approach that we've been advocating for years comes in. That's what Philipp Schuster's GSoC this summer was all about. That's aiming to do exactly what Simon is talking about here. It's about allowing sets of packages to be installed simultaneously that have inconsistent sets of dependencies. There's a slight overlap with sandboxing, but the way I see it, the nix approach is the right underlying implementation mechanism and then sandboxing is more of a UI issue. Duncan

2 1

Re: Cabal and GHC
by Johan Tibell 23 Nov '12

23 Nov '12

On Thursday, November 22, 2012, Manuel M T Chakravarty wrote: > Simon Peyton-Jones <simonpj(a)microsoft.com <javascript:;>>: > > So the invariant I suggest is > > * If it'd work in an empty database, it should work in any non-empty one > > * Installing X should never break the existing installation of Y > > This makes a lot of sense to me. We've been working on this for some time and this property is sometimes known as hermetic builds. The first approximation will be sandbox in (as it is relatively easy to implement). Long term we want a write only Nix like package store.

1 0

Re: Cabal and GHC
by Henning Thielemann 22 Nov '12

22 Nov '12

On Thu, 22 Nov 2012, Brandon Allbery wrote: > I am wondering if we're trying to solve the problem in the wrong way. > The core of the problem is that various things get baked into libraries > in the name of optimization, such that a given binary library has > dependencies on precise versions of other libraries; it's not like C > where anything supporting the ABI can use the same .a/.lib or > .so/.dll/.dylib. C supports inlining. This should cause the same kind of problems, shouldn't it?

1 0

patch applied (hackage-server): "Fix a stray use of update' which should be updateState"
by devnull＠community.haskell.org 22 Nov '12

22 Nov '12

Thu Nov 22 18:50:41 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Fix a stray use of update' which should be updateState Ignore-this: 9858c081438a8f8349ce6c2b6f5cca53 M ./Distribution/Server/Features/PreferredVersions.hs -1 +1

1 0

patch applied (hackage-server): "Refactor: reify state components"
by devnull＠community.haskell.org 22 Nov '12

22 Nov '12

Thu Nov 22 16:12:04 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Refactor: reify state components Ignore-this: 4835f105d2a522c05d3158cb73aa0c02 This should make it harder to forget to provide backup methods (and easier to check that they are provided, and which are provided). It also simplifies the HackageFeature data type. Next step is to see if we can change the definition of StateComponent so that we can express better what the backup should do. M ./Distribution/Server.hs -23 +28 M ./Distribution/Server/Features.hs +6 M ./Distribution/Server/Features/BuildReports.hs -15 +18 M ./Distribution/Server/Features/Check.hs -10 +17 M ./Distribution/Server/Features/Core.hs -27 +33 M ./Distribution/Server/Features/Distro.hs -29 +32 M ./Distribution/Server/Features/Documentation.hs -45 +48 M ./Distribution/Server/Features/DownloadCount.hs -23 +20 M ./Distribution/Server/Features/HaskellPlatform.hs -12 +21 M ./Distribution/Server/Features/Html.hs +1 M ./Distribution/Server/Features/LegacyRedirects.hs -2 +8 M ./Distribution/Server/Features/Mirror.hs -21 +16 M ./Distribution/Server/Features/NameSearch.hs -2 +8 M ./Distribution/Server/Features/PackageContents.hs -10 +26 M ./Distribution/Server/Features/PackageList.hs +1 M ./Distribution/Server/Features/Packages.hs -2 +9 M ./Distribution/Server/Features/PreferredVersions.hs -28 +34 M ./Distribution/Server/Features/ServerApiDoc.hs -7 +19 M ./Distribution/Server/Features/Tags.hs -22 +20 M ./Distribution/Server/Features/Upload.hs -48 +56 M ./Distribution/Server/Features/Users.hs -57 +59 M ./Distribution/Server/Framework/Feature.hs -33 +46 M ./hackage-server.cabal -1 +2

1 0

RE: Cabal and GHC
by Simon Peyton-Jones 22 Nov '12

22 Nov '12

Having GHC support multiple copies of the same package/version would certainly be a step in the right direction. It still wouldn't completely address the goals I'm getting at in my blog post. What I'm really aiming for overall is to provide users with a stable, vetted set of packages that are known to work well together. But for those of us who will still be using plain Hackage, this improvement would be very helpful. Indeed. I've always thought of the Haskell Platform as doing the "stable set" part, though there must be plenty of space between HP at one end and Hackage at the other, and that's what you are exploring I think. Regardless, being able to install multiple instantiations of the same package/version is desirable for the "stable set" story, so that more than one stable set is possible; and so that trying to use a package outside the stable set doesn't mess up your stable-set installation. The question is: who will do the work? Simon From: michael.snoyman(a)gmail.com [mailto:michael.snoyman@gmail.com] On Behalf Of Michael Snoyman Sent: 22 November 2012 10:40 To: Simon Peyton-Jones Cc: Haskell Libraries (libraries(a)haskell.org) cabal-devel(a)haskell.org Subject: Re: Cabal and GHC On Thu, Nov 22, 2012 at 10:32 AM, Simon Peyton-Jones <simonpj(a)microsoft.com<mailto:simonpj@microsoft.com>> wrote: Michael Snoyman wrote a blog post about Solving Cabal Hall, which came across via the peerless Haskell Weekly News. http://www.yesodweb.com/blog/2012/11/solving-cabal-hell But I'd be unlikely to return there so I'm posting this to the libraries and cabal-devel lists. I'm not deep in Cabal lore, but it seems to me that there is a fairly easy way to do a Lot Better than we are now. Let's go back to "Identifying the Problem". Michael doesn't say, but I'm guessing that Fay depends on yesod-platform. Very well, so Cabal sees I'm installing Fay, which depends on yesod-platform, which dpends on data-default and depends directly on data-default If the package database was empty, Cabal would try to figure out a version of data-default that is acceptable to both yesod-platform and to Fay. And that's what we want! Suppose that Cabal figures out that yesod-platform-2.7 and data-default-0,4 would work. Then, in an empty package database, it could just go ahead and install those. But in your example, the database isn't empty; we have already isntalled yesod-platform-2.7, dependin gon data-default-0.5. And (here's the rub) you can only have yesod-platform-2.7 installed once. One solution might be 1. *un* install yesod-platform-2.7 (depending on data-default-0.5) and 2. *re* install it (depending on data-default-0.4). But that would break any programs that use both yesod-platform-2.7 and data-default-0.5. The solution is obvious: we should make it possible to instally yesod-platform-2.7 twice, once version depending on data-default-0.4 once version depending on data-default-0.5. Now, if Cabal can figure out a plan based on an empty database, it can deliver on that plan even in a non-empty database, without messing up any existing installations. Of course, if Fay depends exclusively on data-default-0.4, and yesod-platform depends exclusively on data-default-0.5, then they genuinely are incompatible, and Cabal can and should say so. That's not Cabal Hell. That's just saying that the package authors have *specified* that they are incompatible. Summary. I may be way off beam here, but I think we can easily make things way better than they are. By "easily" I mean with just design and implementation work. That's not free, and no one has any time.... but it's well within reach. I'd even been wondering about trying to crowd-fund this development so that Well Typed can do it. But let's first see if it'd solve the problem. There must be a wiki page somewhere that articulates the challenge, points to blog posts about it, and discusses solutions. Simon _______________________________________________ Libraries mailing list Libraries(a)haskell.org<mailto:Libraries@haskell.org> http://www.haskell.org/mailman/listinfo/libraries Hi Simon, Having GHC support multiple copies of the same package/version would certainly be a step in the right direction. It still wouldn't completely address the goals I'm getting at in my blog post. What I'm really aiming for overall is to provide users with a stable, vetted set of packages that are known to work well together. But for those of us who will still be using plain Hackage, this improvement would be very helpful. MIchael

2 1