November 2012 - cabal-devel

patch applied (hackage-server): "Fix comparison on tarballs"
by devnull＠community.haskell.org 29 Nov '12

29 Nov '12

Thu Nov 29 10:36:19 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Fix comparison on tarballs Ignore-this: 648378d5c1cdccfb8db791fe524f64ce If the tarballs didn't contain the exact same entries, in the exact same order, then `equalTarBall` would report lots of superfluous errors due to a call to `mergeBy` on unsorted lists. M ./Distribution/Server/Framework/BackupRestore.hs -1 +3

1 0

Re: Cabal and GHC
by Simon Marlow 28 Nov '12

28 Nov '12

On 22/11/2012 19:54, Gábor Lehel wrote: > On Thu, Nov 22, 2012 at 8:41 PM, Henning Thielemann > <lemming(a)henning-thielemann.de> wrote: >> >> On Thu, 22 Nov 2012, Brandon Allbery wrote: >> >>> I am wondering if we're trying to solve the problem in the wrong way. The >>> core of the problem is that various things get baked into libraries in the >>> name of optimization, such that a given binary library has dependencies on >>> precise versions of other libraries; it's not like C where anything >>> supporting the ABI can use the same .a/.lib or .so/.dll/.dylib. >> >> >> C supports inlining. This should cause the same kind of problems, shouldn't >> it? > > Indeed, and in Qt for example which has a strict binary compatibility > guarantee, making a public function inline means you effectively > cannot change it until the next major release. (In this case "making > it inline" and "including the definition in the header file" are > effectively synonymous, one requires the other.) > > The problem is that in Haskell inlining is a lot more important for performance. I would like to see GHC support fixed ABIs, and the work I did with ABI hashing in GHC was aiming towards exactly that. For fixed ABIs you would need to have the user explicitly declare every inline function, and then hash the definitions as part of the ABI. (also do something about strictness and arity, and other cross-module optimisation hints). It might be painful, but it could be optional, and the gains are quite nice: the ability to upgrade a library in-place without recompiling everything that depends on it. Especially now that we're moving towards shared libraries, this would become more useful. I'm sure it's not going to happen soon, but it would be an interesting project for someone (probably larger than a GSoC project though). Cheers, Simon

3 2

patch applied (hackage-server): "Remove two old unused utils modules" and 5 others
by devnull＠community.haskell.org 27 Nov '12

27 Nov '12

Fri Nov 23 14:25:40 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Remove two old unused utils modules Ignore-this: 15dcb6047f16fd05610359fac8ef0114 R ./Distribution/Server/Util/ActionLog.hs R ./Distribution/Server/Util/TimeLogger.hs Tue Nov 27 17:01:52 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Simplify import client command line validation a bit Ignore-this: 2b30fc71f1896987368dcd68c667c028 M ./ImportClient.hs -13 +14 Tue Nov 27 17:05:47 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Allow concurrent upload for package metadata in the import client Ignore-this: e9656155c4f6973914347da92be4a1b0 M ./ImportClient.hs -16 +33 Tue Nov 27 16:57:28 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Grrr, the periodic download refresh is unconditional Ignore-this: 5698c5c4f327723da89d8263e2a2c6d9 In the meantime, make it wait a bit longer. M ./Distribution/Server/Features/PackageList.hs -1 +2 Tue Nov 27 17:30:59 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Overhaul all the caching Ignore-this: e82f108830a5b42576bd4b0bba677ed4 Separate the previous uses of caches into two kinds: 1. simple in-memory vars and 2. proper caches. For the former we just add a simple strict wrapper around MVar, with read/write/modify operations, while for latter kind we use asynchronous caches that are constructed with their update action. Then update the features to use the appropriate kind of cache. Additionally, the old async cache infrastructure had a feature to disable updating of caches for the benefit of bulk import operations. The corresponding thing in the new infrastructure is a tunable knob to delay cache updates so that more changes can accumulate before doing the expensive cache update operation. M ./Distribution/Server.hs -6 +9 M ./Distribution/Server/Features.hs -5 +5 M ./Distribution/Server/Features/Core.hs -29 +50 M ./Distribution/Server/Features/DownloadCount.hs -7 +6 M ./Distribution/Server/Features/Html.hs -29 +40 M ./Distribution/Server/Features/NameSearch.hs -30 +63 M ./Distribution/Server/Features/PackageList.hs -13 +12 M ./Distribution/Server/Features/Packages.hs -14 +18 M ./Distribution/Server/Features/Tags.hs -5 +4 M ./Distribution/Server/Features/Users.hs -12 +11 M ./Distribution/Server/Framework.hs -2 +6 M ./Distribution/Server/Framework/Cache.hs -79 +129 M ./Distribution/Server/Framework/Instances.hs -1 +8 A ./Distribution/Server/Framework/MemState.hs M ./Distribution/Server/Framework/Types.hs -1 +12 R ./Distribution/Server/Util/AsyncVar.hs M ./Distribution/Server/Util/TextSearch.hs -2 +5 M ./Main.hs -18 +26 M ./hackage-server.cabal -1 +1 Tue Nov 27 17:41:01 GMT 2012 Duncan Coutts <duncan(a)well-typed.com> * Move a use of runHook into the feature that owns the hook Ignore-this: ffaef756e8b9c49027a4cf867c64d0be Other features should not really prod hooks. M ./Distribution/Server/Features/Core.hs +1 M ./Distribution/Server/Features/PreferredVersions.hs -1

1 0

patch applied (hackage-server): "Add 'resetState' to StateComponent" and 2 others
by devnull＠community.haskell.org 27 Nov '12

27 Nov '12

Tue Nov 27 10:11:58 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Add 'resetState' to StateComponent Ignore-this: a4dcd475cc01e6593f8fe735cfed171 resetState has type StateComponent -> BlobStorage -> FilePath -> IO (StateComponent st) and basically creates a new state component with the initial state. This is now used consistently throughout the initFooFeature functions to initialize the state, so that many initFeature methods now simply look like initFooFeature <some params> = do st <- fooStateComponent .. return (fooFeature <params> st) fooStateComponent = do st <- openLocalStateFrom ... return StateComponent { ... , resetState = fooStateComponent } This is a nicer separation of concerns, but more importantly, this is a stepping stone towards fixing 'testRoundtrip'. M ./Distribution/Server/Features/BuildReports.hs -19 +15 M ./Distribution/Server/Features/Check.hs -18 +14 M ./Distribution/Server/Features/Core.hs -18 +17 M ./Distribution/Server/Features/Distro.hs -18 +14 M ./Distribution/Server/Features/Documentation.hs -24 +19 M ./Distribution/Server/Features/DownloadCount.hs -19 +16 M ./Distribution/Server/Features/HaskellPlatform.hs -18 +15 M ./Distribution/Server/Features/Mirror.hs -13 +14 M ./Distribution/Server/Features/PreferredVersions.hs -21 +16 M ./Distribution/Server/Features/Tags.hs -19 +16 M ./Distribution/Server/Features/Upload.hs -41 +43 M ./Distribution/Server/Features/Users.hs -26 +28 M ./Distribution/Server/Framework/Feature.hs -1 +24 Tue Nov 27 15:32:33 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Remove testRoundtrip Ignore-this: 64252fffcbd74c025d4eedb862003847 StateComponent now provides enough informaiton to define testRoundtrip generically for all features (just equality on states). This is possible because the check for *blobs* will not be necessary anymore, as we will take care of this generically, too (we don't yet, though). This now throws an error in test-backup in the import/export for Core; presumably this error was always there, we just didn't notice it becauset testRoundtrip was set up incorrectly. (I think the error is not so important, rounding error for dates.) M ./Distribution/Server.hs -48 +11 M ./Distribution/Server/Features.hs -2 +2 M ./Distribution/Server/Features/BuildReports.hs -2 +1 M ./Distribution/Server/Features/BuildReports/Backup.hs -7 +1 M ./Distribution/Server/Features/Check.hs -2 +1 M ./Distribution/Server/Features/Check/State.hs -1 +1 M ./Distribution/Server/Features/Check/Types.hs -1 +1 M ./Distribution/Server/Features/Core.hs -12 +1 M ./Distribution/Server/Features/Distro.hs -3 +1 M ./Distribution/Server/Features/Documentation.hs -10 +2 M ./Distribution/Server/Features/DownloadCount.hs -2 +1 M ./Distribution/Server/Features/HaskellPlatform.hs -2 +1 M ./Distribution/Server/Features/HaskellPlatform/State.hs -1 +1 M ./Distribution/Server/Features/Mirror.hs -3 +1 M ./Distribution/Server/Features/Packages.hs -1 M ./Distribution/Server/Features/PreferredVersions.hs -2 +1 M ./Distribution/Server/Features/ServerApiDoc.hs -1 +1 M ./Distribution/Server/Features/Tags.hs -2 +1 M ./Distribution/Server/Features/Tags/Backup.hs -1 M ./Distribution/Server/Features/Upload.hs -6 +3 M ./Distribution/Server/Features/Upload/State.hs -2 +2 M ./Distribution/Server/Features/Users.hs -4 +2 M ./Distribution/Server/Framework/BackupDump.hs -38 +2 M ./Distribution/Server/Framework/BackupRestore.hs -8 M ./Distribution/Server/Framework/Feature.hs -16 +84 M ./Distribution/Server/Users/State.hs -2 +2 M ./Main.hs -19 +32 Tue Nov 27 15:49:02 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Include fractions of seconds during export Ignore-this: cce3a160b0605067d9319256a9f7df5f This fixes the roundtrip test for Core M ./Distribution/Server/Framework/BackupRestore.hs -1 +1

1 0

patch applied (hackage-server): "Make import pure for Tags"
by devnull＠community.haskell.org 26 Nov '12

26 Nov '12

Mon Nov 26 14:29:41 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Make import pure for Tags Ignore-this: 3f973d3f9708e04bbb9289fc426325fa M ./Distribution/Server/Features/Tags/Backup.hs -20 +21 M ./Distribution/Server/Framework/BackupRestore.hs -5 +38

1 0

patch applied (hackage-server): "Fix the 'restore' process"
by devnull＠community.haskell.org 26 Nov '12

26 Nov '12

Mon Nov 26 14:30:48 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Fix the 'restore' process Ignore-this: 5199ad4a1f430bd4c338f42054ecd0b2 - Not all features implemented 'restore' after the refactoring, causing restore to fail - 'importTar' assumed a single RestoreBackup per feature, while we now have multiple (though, thankfully, they implemented Monoid) Note that 'testRoundtrip' is still broken, but it turns out it was already broken even before the refactoring. The problem is that testRoundtripByQuery returns a check which compares the state of the database using the AcidState handle which corresponds to the *initial database* (before the export and reimport), and hence is comparing a database to itself and always returns success. It should instead use the handle created after the re-import, but that handle is not yet available when 'testRoundtrip' is called, so this requires some re-engineering of how tests are set up. Will fix this separately. M ./Distribution/Server.hs -6 +11 M ./Distribution/Server/Features/Check.hs +1 M ./Distribution/Server/Features/HaskellPlatform.hs -5 +6 M ./Distribution/Server/Features/PreferredVersions.hs +1 M ./Distribution/Server/Framework/BackupRestore.hs +1 M ./Main.hs -2 +4

1 0

patch applied (hackage-server): "Remove obsolete code" and 2 others
by devnull＠community.haskell.org 23 Nov '12

23 Nov '12

Thu Nov 22 16:25:53 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Remove obsolete code Ignore-this: 80f0a57579216f3bad55e8159b30a3e1 M ./Distribution/Server/Features/ServerApiDoc.hs -8 Fri Nov 23 12:48:44 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Make export pure Ignore-this: d7404f43bbb30bf13791b6fb75657ffb The only reason export lived in IO was because it needed access to the blob store. Now instead we define data BackupEntry = BackupByteString [FilePath] ByteString | BackupBlob [FilePath] BlobId This has various other advantages: we now have the option of storing blob entries differently (or indeed not at all) when we generate the tar file. For now, the export format is still the same, but at least we have the option now of treating blobs differently. Note that there was already an ExportEntry (as opposed to BackupEntry) type which was isomorphic to the (new) BackupEntry type. I have removed that type completely, and removed some of the conversions between them. M! ./Distribution/Server.hs -2 +4 M! ./Distribution/Server/Features/BuildReports.hs -1 +2 M! ./Distribution/Server/Features/BuildReports/Backup.hs -12 +8 M! ./Distribution/Server/Features/Check.hs -2 +3 M! ./Distribution/Server/Features/Core.hs -9 +6 M! ./Distribution/Server/Features/Core/Backup.hs -20 +20 M! ./Distribution/Server/Features/Distro.hs -1 +2 M! ./Distribution/Server/Features/Distro/Backup.hs -5 +4 M! ./Distribution/Server/Features/Documentation.hs -5 +5 M! ./Distribution/Server/Features/DownloadCount.hs -2 +2 M! ./Distribution/Server/Features/DownloadCount/Backup.hs -1 +1 M! ./Distribution/Server/Features/HaskellPlatform.hs -1 +2 M! ./Distribution/Server/Features/Mirror.hs -5 +5 M! ./Distribution/Server/Features/PreferredVersions.hs -2 +3 M! ./Distribution/Server/Features/Tags.hs -2 +2 M! ./Distribution/Server/Features/Tags/Backup.hs -1 +1 M! ./Distribution/Server/Features/Upload.hs -12 +12 M! ./Distribution/Server/Features/Upload/Backup.hs -1 +1 M! ./Distribution/Server/Features/Upload/State.hs -4 +12 M! ./Distribution/Server/Features/Users.hs -13 +13 M! ./Distribution/Server/Framework/BackupDump.hs -31 +22 M! ./Distribution/Server/Framework/BackupRestore.hs -8 +12 M! ./Distribution/Server/Framework/Feature.hs -1 +2 M! ./Distribution/Server/Users/Backup.hs -15 +16 M! ./Distribution/Server/Users/State.hs -4 +12 Fri Nov 23 12:52:53 GMT 2012 Edsko de Vries <edsko(a)well-typed.com> * Resolve merge conflicts Ignore-this: 7f0f37fcf807173ca8e571ac92e12f5b M ./Distribution/Server/Features/PreferredVersions.hs -1 +1

1 0

Re: Cabal and GHC
by Duncan Coutts 23 Nov '12

23 Nov '12

On 22 November 2012 10:39, Michael Snoyman <michael(a)snoyman.com> wrote: > > On Thu, Nov 22, 2012 at 10:32 AM, Simon Peyton-Jones <simonpj(a)microsoft.com> > wrote: >> >> Michael Snoyman wrote a blog post about Solving Cabal Hall, which came >> across via the peerless Haskell Weekly News. >> http://www.yesodweb.com/blog/2012/11/solving-cabal-hell >> The solution is obvious: we should make it possible to instally >> yesod-platform-2.7 twice, > Having GHC support multiple copies of the same package/version would > certainly be a step in the right direction. It still wouldn't completely > address the goals I'm getting at in my blog post. What I'm really aiming for > overall is to provide users with a stable, vetted set of packages that are > known to work well together. But for those of us who will still be using > plain Hackage, this improvement would be very helpful. Right, and we need both. Your suggestion is very sensible. It's similar to the suggestion people have made before about having "testing" and "stable" sets of packages (these suggestions often come from people using debian terminology), and managing things like a distribution. In both cases the point is to provide something to end users where we've set up the available packages such that we know most things work together. (This is why debian users don't run into problems most of the time, because volunteers have done a huge amount of work behind the scenes to eliminate conflicting versions). So yes we need that, and it will require a fair bit of work, and we need to distribute that amongst many people. I think eventually the best mechanism to do it will be to use hackage and use tagging, but we can certainly experiment now by using separate hackage instances. But we also need to address the problems of developers in the thick of things, who must work with the latest and greatest unsable versions, before people have got round to smoothing out all the conflicting dependencies and patching packages so they can all use the same versions of common dependencies. That's where the nix-style approach that we've been advocating for years comes in. That's what Philipp Schuster's GSoC this summer was all about. That's aiming to do exactly what Simon is talking about here. It's about allowing sets of packages to be installed simultaneously that have inconsistent sets of dependencies. There's a slight overlap with sandboxing, but the way I see it, the nix approach is the right underlying implementation mechanism and then sandboxing is more of a UI issue. Duncan

2 1

Re: Cabal and GHC
by Johan Tibell 23 Nov '12

23 Nov '12

On Thursday, November 22, 2012, Manuel M T Chakravarty wrote: > Simon Peyton-Jones <simonpj(a)microsoft.com <javascript:;>>: > > So the invariant I suggest is > > * If it'd work in an empty database, it should work in any non-empty one > > * Installing X should never break the existing installation of Y > > This makes a lot of sense to me. We've been working on this for some time and this property is sometimes known as hermetic builds. The first approximation will be sandbox in (as it is relatively easy to implement). Long term we want a write only Nix like package store.

1 0

Re: Cabal and GHC
by Henning Thielemann 22 Nov '12

22 Nov '12

On Thu, 22 Nov 2012, Brandon Allbery wrote: > I am wondering if we're trying to solve the problem in the wrong way. > The core of the problem is that various things get baked into libraries > in the name of optimization, such that a given binary library has > dependencies on precise versions of other libraries; it's not like C > where anything supporting the ABI can use the same .a/.lib or > .so/.dll/.dylib. C supports inlining. This should cause the same kind of problems, shouldn't it?

1 0