October 2011 - cabal-devel

patch applied (/home/srv/code/hackage-server): "Remove most of the package checks for the mirroring case"
by duncan＠lun.haskell.org 10 Oct '11

10 Oct '11

Mon Oct 10 09:17:13 BST 2011 Duncan Coutts <duncan(a)community.haskell.org> * Remove most of the package checks for the mirroring case Ignore-this: 6d2f4928f44edccad3cd34cc7898de93 For mirroring, we have to be able to upload packages that would fail the current QA checks. M ./Distribution/Server/Features/Mirror.hs -1 +1 M ./Distribution/Server/Packages/Unpack.hs -14 +16

1 0

patch applied (/home/srv/code/hackage-server): "Implement robust continuous/live mirroring"
by duncan＠lun.haskell.org 09 Oct '11

09 Oct '11

Sun Oct 9 21:30:32 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Implement robust continuous/live mirroring Ignore-this: f8efccf223a32f39a7d7fc1cfa599413 The HTTP error handling infrastructure is substantially changed. Unfortunately this means it's not shared with the build bot client anymore. That can be re-shared later. M ./Distribution/Client.hs -3 +4 M ./MirrorClient.hs -59 +726 M ./hackage-server.cabal -1 +1

1 0

patch applied (/home/srv/code/hackage-server): "Rework the async var to batch updates and amortise the work" and 3 others
by duncan＠lun.haskell.org 09 Oct '11

09 Oct '11

Sun Oct 9 17:29:55 BST 2011 Duncan Coutts <duncan(a)community.haskell.org> * Rework the async var to batch updates and amortise the work Ignore-this: 7541312d71e65f8effa0c4ba33d0b480 Previously, every single write to the var was laboriously evaluated and since jobs keep comming in faster than they can be evaluated, then the input queue just grows without bound. But if we're just writing then if we have several new inputs in the queue then we can just skip to the final update and evaluate that one. That's now what we do, with a slight complication to do with incremental updates. M ./Distribution/Server/Util/AsyncVar.hs -16 +34 M ./hackage-server.cabal +1 Sun Oct 9 17:38:01 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Use version in hackage-mirror user agent string Ignore-this: 1412e6389f024ba53bfc563dc6687dc4 M ./Distribution/Client.hs -1 +3 Sun Oct 9 17:44:24 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Keep mirror client files in per-server dirs Ignore-this: 4b102dbb3490bff0f4726221d00a1f05 And delete package tarballs when they've been successfully uploaded. M ./Distribution/Client.hs -5 +5 M ./MirrorClient.hs -6 +10 Sun Oct 9 17:48:56 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Depend on the latest HTTP lib, for bug fixes Ignore-this: 987fbac8094bcd99370b753eeae60fac M ./hackage-server.cabal -1 +1

1 0

patch applied (/home/srv/code/hackage-server): "Have the MirrorClient use the system proxy" and 6 others
by duncan＠lun.haskell.org 09 Oct '11

09 Oct '11

Tue Sep 27 09:59:51 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Have the MirrorClient use the system proxy Ignore-this: c2f1dc9d7ae36a9acc200a4f527f432 M ./MirrorClient.hs +1 Tue Sep 27 10:35:04 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Split out client functionality unrelated to mirroring from MirrorClient Ignore-this: bfc13126ed4e974c460a916fab313c33 A ./Distribution/Client.hs M ./MirrorClient.hs -262 +11 M ./hackage-server.cabal +1 Tue Sep 27 10:37:56 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Turn on some warnings for hackage-mirror and fix them Ignore-this: e03f60dfe9e774da86704e45ed4c125e M ./Distribution/Client.hs -1 +5 M ./MirrorClient.hs -1 +1 M ./hackage-server.cabal +5 Tue Sep 27 11:22:17 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Rough template of Hackage build client Ignore-this: 7476e699115135d3bef3390e0fef8b8e A ./BuildClient.hs M ./Distribution/Client.hs -3 +30 M ./MirrorClient.hs -25 +5 M ./hackage-server.cabal +23 Wed Sep 28 13:28:14 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Bump acid-state and happstack versions to avoid bug Ignore-this: 67da02f641e393571db92286e06fd23a There is a bug in happstack-server before v6.2.2 when you try to read the cookies/request params of a POST/PUT request with a content-type different from multipart/form-data or application/x-www-form-urlencoded The bug is described in http://comments.gmane.org/gmane.comp.lang.haskell.web/2033 As a result I changed the version bounds to v6.2.* M ./hackage-server.cabal -2 +2 Wed Sep 28 22:13:52 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * A mostly functional build client Ignore-this: d65dd5e7e53015635d97ab3f9a38b298 M ./BuildClient.hs -27 +165 M ./Distribution/Client.hs -6 +20 M ./MirrorClient.hs -5 +2 M ./hackage-server.cabal -2 +2 Wed Sep 28 22:17:54 BST 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Comments/debug code only Ignore-this: 997aeb8b78d431180c90dfd7f47e42af M ./BuildClient.hs -3 +4

1 0

patch applied (cabal): "Use the configured proxy even for uploading build reports"
by Duncan Coutts 09 Oct '11

09 Oct '11

Wed Sep 28 14:08:59 PDT 2011 Max Bolingbroke <batterseapower(a)hotmail.com> * Use the configured proxy even for uploading build reports M ./cabal-install/Distribution/Client/BuildReports/Upload.hs -3 +1 M ./cabal-install/Distribution/Client/HttpUtils.hs -12 +20 M ./cabal-install/Distribution/Client/Upload.hs -13 +5 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=…

1 0

patch applied (cabal): "GHC 7.2+ no longer generates _stub.o files"
by Duncan Coutts 09 Oct '11

09 Oct '11

Sat Sep 10 12:53:29 PDT 2011 Duncan Coutts <duncan(a)community.haskell.org> * GHC 7.2+ no longer generates _stub.o files So stop looking for them. This could otherwise cause problems if one switches ghc version without cleaning the build dir since we'll pick up the old _stub.o files and end up with duplicate linker symbols. M ./cabal/Distribution/Simple/GHC.hs -3 +7 View patch online: http://darcs.haskell.org/cgi-bin/darcsweb.cgi?r=cabal;a=darcs_commitdiff;h=…

1 0

patch applied (/home/srv/code/hackage-server): "First go at adding continuous/live mirroring" and 8 others
by duncan＠lun.haskell.org 09 Oct '11

09 Oct '11

Thu Oct 6 11:29:45 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * First go at adding continuous/live mirroring Ignore-this: 69423437dcf02091da86ae3480bc128d M ./MirrorClient.hs -32 +105 M ./hackage-server.cabal -1 +1 Thu Oct 6 12:09:20 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Improve the mirror client help messages Ignore-this: 30d7c7730b11b88871fa6a4df3d79014 M ./MirrorClient.hs -5 +35 Sat Oct 8 15:30:24 BST 2011 Duncan Coutts <duncan(a)community.haskell.org> * Rework the async var to batch updates and amortise the work Ignore-this: 3f53d68d36f1847b061dcb7ddcb3e23d Previously, every single write to the var was laboriously evaluated and since jobs keep comming in faster than they can be evaluated, then the input queue just grows without bound. But if we're just writing then if we have several new inputs in the queue then we can just skip to the final update and evaluate that one. That's now what we do, with a slight complication to do with incremental updates. M ./Distribution/Server/Util/AsyncVar.hs -16 +34 M ./hackage-server.cabal +1 Sat Oct 8 16:55:51 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Overhaughl the mirror client error handling infrastructure and strategy Ignore-this: 59347decddab335ba0f9f75fd66d3d0f M ./MirrorClient.hs -127 +443 Sat Oct 8 17:01:55 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Only rewrite the mirror client error files if they've changed Ignore-this: b82b608bfcd9a6d2e4473bb616ab563c M ./MirrorClient.hs -2 +4 Sat Oct 8 19:20:12 BST 2011 Duncan Coutts <duncan(a)community.haskell.org> * Disable idle GC for the server Ignore-this: 970125139572aee660c2dae06f904464 It doesn't correctly detect when we're really idle and ends up using ~15% CPU when the server is otherwise idle. M ./hackage-server.cabal +2 Sat Oct 8 19:22:16 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Depend on the latest HTTP lib, for bug fixes Ignore-this: a5f411822c140575974829df3e5eacec M ./MirrorClient.hs -3 M ./hackage-server.cabal -1 +1 Sun Oct 9 00:43:15 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Report when a checkpoint is finished writing. Ignore-this: 2f451e601c3d8fb9c1bb45e34d55f0eb It's just that little bit more reassuring. M ./Main.hs +1 Sun Oct 9 01:12:09 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Handle IO errors in mirroring without losing package error info Ignore-this: 718a385e3718d7cc12a6312109648dfc Previously if we got an IO exception, rather than an HTTP level error, then we had no chance to trap it locally so we would lose all the accumulated info about the packages with errors. Now we cheat. Instead of using a proper state monad, we use a reader monad with an IORef that way, we can catch the IO exception in the monad run function, and still read out the latest package error state. Also handle user interrupt ^C similarly. M ./MirrorClient.hs -15 +33

2 1

Hackers hacking on hackage: introductions
by Duncan Coutts 07 Oct '11

07 Oct '11

Hi all, I'm very pleased to say that recently we've had a number of developers express an interest in helping with the new hackage server, some at the recent hackathons (HacPDX-II, CamHac) and some after my plea for help at the recent Haskell Symposium. I've tried to encourage everyone who's contacted me with some interest to get onto this mailing list [1] and to talk to each other (not just to me). By having everyone aware of each other and talking to each other we can give each other encouragement and advice, code review etc. [1]: http://www.haskell.org/mailman/listinfo/cabal-devel I'll quickly outline some resources for getting started and then do some quick introductions to let everyone know who's indicated they're interested in looking at hackage. Hacker resources ---------------- Firstly, the wiki is getting into reasonable shape: http://hackage.haskell.org/trac/hackage/wiki/HackageDB/2.0 It has a quickstart hacking guide with links to the darcs repo. The first thing is to try getting it working on your machine then check the TODO file and the todo lists on the wiki. Our initial priority is things needed to get a public server running in parallel with live mirroring. I can give more details on this. People working on hackage ------------------------- Since I'm introducing people, I'll start with myself... I'm not really the maintainer of the new hackage-server, but since nobody else is yet then I sort-of get stuck with that role. The current design of the new codebase is partly my fault (but in a good way, I think it's a reasonable design). I supervised Matt Gruen last year on his GSoC project who did most of the actual hard work of coding. I have actually been hacking on the codebase myself in recent months, partly at the various hackathons. My involvement is almost entirely in a volunteer capacity, though currently I'm doing a little work on the mirroring side of things for one of the members of the IHG. Matthew Gruen as I mentioned above did his GSoC project on the hackage-server last summer, so the much of the existing code is due to him, in particular the implementation of the resource and dynamic path stuff, plus many of the Features.* modules. Matt has occasionally been able to help out in the last year, in particular he gave us lots of help during the HacPDX-II hackathon a couple months ago. Max Bolingbroke accosted me after my brief talk at the Haskell Symposium and asked what needed to be done to get the new hackage stuff going. As you can see he's already sent in patches for a new documentation build bot. The new server design involves the server being mostly a passive store of data and calls for specialised clients to do things like building documentation and uploading it. Max doesn't know it yet, but he's already got a commit bit on the source repo. I guess he knows now. David Terei also grabbed me at the Haskell Symposium and is keen to help. David Laing contacted me today and is looking for a bigger project than his previous personal side projects. So please help answer his questions and point him in the right direction. And if you want a helper, he might be the person to ask Erik Hesselink is running a hackage server instance internally at his company and is keen to get it more feature complete. Ben Millwood was at CamHac and wrote a number of patches. Stefan Wehr was also at CamHac and did some work on providing access to package changelog files within the package tarballs (via serving the contents of the package tarball). I don't know if Stefan has time to finish this off, but it's a great feature to have. Thomas DuBuisson organised the HacPDX-II hackathon and was keen to have the new hackage server be a focal point of the hackathon. In fact this is what really got the ball rolling again. Thomas also wrote some patches to clean up the package tags feature. David Lazar was also at HacPDX-II and made a number of cleanup patches and fixed some things to do with package deprecation. Jeremy Shaw took part in HacPDX-II and sent us a couple big patches to bring us up to the latest happstack version and switch us from happstack-state to the newer nicer acid-state. Jeremy is a good person to ask about happstack issues. Antoine Latter has made several contributions over the last couple years, including various tricky issues with HTTP. Conrad Parker is keen to see the new server get going. He also wants to deploy it at his company. He's got lots of design ideas for how it should work, and the relation between the package repositories and the cabal clients. Last, but not least, is of course Ross Paterson. Ross wrote the current hackage-scripts which is behind the current hackage.haskell.org site and he has maintained and improved it over the last five years. While people like me have been fussing over new server design ideas, Ross' site has actually been working and has coped with growth from a handful to thousands of packages and hoards of users. No doubt I'm forgetting some people, apologies. Whether I mentioned you, forgot, or didn't even know, do please send a follow-up to introduce yourself and let everyone else know what you're interested in and/or how you're willing to help. BTW, while I've suggested to people that they darcs send their patches to this list for review, I don't want to stand in the way of patches getting in. I'm certainly not the only person who can review. I will be handing out commit bits to ensure that we don't have a bottleneck. Of course reviewing other people's patches and commenting on the approach is very valuable. It makes it much easier for someone with a commit bit to push it in if it's been positively reviewed by one or two people. Duncan

1 0

patch applied (/home/srv/code/hackage-server): "Use version in hackage-mirror user agent string" and 2 others
by duncan＠lun.haskell.org 07 Oct '11

07 Oct '11

Wed Oct 5 12:56:03 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Use version in hackage-mirror user agent string Ignore-this: c1d9136810b593bf84503f436d3bab8 M ./MirrorClient.hs -1 +3 Wed Oct 5 12:56:35 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Keep mirror client files in per-server dirs Ignore-this: c4938a438fad1af321323bcd0b1139d1 And delete package tarballs when they've been successfully uploaded. M ./MirrorClient.hs -10 +13 Wed Oct 5 12:58:22 BST 2011 Duncan Coutts <duncan(a)well-typed.com> * Install the various new haddock data files Ignore-this: 8bb6837cd5853a44ac9f59258221a8f2 M ./hackage-server.cabal +4

1 0

Proposal: Make it possibly specify the component to build form the command line
by Johan Tibell 06 Oct '11

06 Oct '11

Hi, Here's a another proposal. Now when we start having more components in one package (library, executables, test suites, and soon benchmarks) it would be convient to specify what to build directly from the command line without having to reconfigure e.g. cabal build my-test This would also allow one to enable all components e.g. cabal configure --enable-tests --enable-benchmarks but only build one of them (for faster build times) cabal build my-benchmark Would that work well with how Cabal keeps track of what to build? Cheers, Johan

2 1