On Mon, 2012-07-02 at 12:25 +0100, Ian Lynagh wrote:
Hi all,
I'm planning to spend some time, on behalf of the Industrial Haskell Group, working on Hackage 2 in the coming weeks.
[..]
Now #913 I assume is not a blocker. #919 I assume is also not a blocker. And #914 and #915 are improvements to the internals, so presumably also not blockers. #426 is not supported by Hackage 1 as far as I can see, so is not a blocker as it is not a regression.
I agree with this analysis.
So that leaves 3 tickets as blockers:
#911: We need to do something here. With Hackage 1, it takes manual approval before you can upload packages, and at the very least Hackage 2 should match that. I have the impression that that is already possible (by restricting package upload to a group, and requiring accounts to be added to that group by an admin), but I haven't confirmed that yet.
Right, I don't think we need to do any more than make sure uploaders are in the appropriate group. It *should* currently be the case that only accounts in the package group can upload, and the first time you upload a new named package then you get added as the initial member of the new package group. Currently for testing purposes anyone can register an account and can then upload new packages. We have two options here: restrict account creation to be manual like in hackage 1, or add a new system-wide "uploaders" group for accounts that are authorised to upload new packages and have a manual admin step to add people to the uploaders group. The latter will allow for registered users who are not uploaders which would be useful later to allow things like non-anonymous commenting etc.
#916: At the very least, Hackage 2 needs to support URLs that Hackage 1 supported (unless a conscious decision has been made not to). Ideally we would get the URLs right on the initial release, so that people don't start using the wrong ones. Doesn't sound hard, so may as well do before the switchover.
Yes, barring mistakes this should work already. There's a "legacy" feature module that provides a bunch of redirects.
#918: This is the main missing functionality currently missing from the user's point of view.
Right. You'll see there's some code for a doc builder. This needs to be improved. Perhaps it can share code with the mirror client which is reasonably robust.
Conclusion ----------
I think the following are the blockers for deploying Hackage 2:
* #911 upload perms; may be good enough already * #916 check URLs are OK * #918 build haddock (and HsColour) docs
Right.
* Show source respository on package pages
Should be easy to port that from the old code.
* Support the existing "Distributions" files, and show info on package pages
I advocated at the time the feature was added that it should be done differently so that the hackage server does not poll some url, but people in charge of distros push instead. I think it would not be a blocker to not implement the distribution info system as it is now and when eventually spending the time to implement it, switch to doing it in a more sensible way.
(plus enough testing to give us confidence in it, of course).
One of the main things here is adding tests that the database dump/restore mechanism round trips correctly.
Does that match other people's opinions? Did I miss anything?
Looks good. Something to keep in mind is memory usage. I know Jeremy is looking at this from the infrastructure side, but I think from the app side there's also some likely culprits. Cabal's GenericPackageDescription type is very large in memory. Having 10's of 1000's of these means lots of memory. One hopefully easy way to save memory here without going to the hassle of redoing Cabal's type definitions is simply to increase sharing. There's a huge amount of repeated information. Start by sharing all the package names and versions. Then there's other meta-data that rarely changes between versions of the same package. This kind of thing should be easy to evaluate, just write a test prog that reads the index file and look at peak memory use. Then try sharing stuff and see how much it drops. This sharing optimisation would still be useful even if later we go and redo GenericPackageDescription to be more compact. Duncan