September 2012 - cabal-devel

patch applied (hackage-server): "De-orphan some more instances"
by devnull＠community.haskell.org 07 Sep '12

07 Sep '12

Fri Sep 7 17:56:18 BST 2012 Ian Lynagh <ian(a)well-typed.com> * De-orphan some more instances Ignore-this: 228a7ee88760855406af87ae37e340c3 M ./Distribution/Server/Framework/BlobStorage.hs -1 +6 M ./Distribution/Server/Packages/State.hs -13 +1 M ./Distribution/Server/Packages/Types.hs -1 +8

1 0

patch applied (hackage-server): "De-orphan the SafeCopy PackageIndex instance"
by devnull＠community.haskell.org 07 Sep '12

07 Sep '12

Fri Sep 7 17:42:46 BST 2012 Ian Lynagh <ian(a)well-typed.com> * De-orphan the SafeCopy PackageIndex instance Ignore-this: 7307cd707f2862aa6ae5e9c2e4e4264c M ./Distribution/Server/Packages/PackageIndex.hs +9 M ./Distribution/Server/Packages/State.hs -8

1 0

patch applied (hackage-server): "Bump base64-bytestring dependency" and 10 others
by devnull＠community.haskell.org 07 Sep '12

07 Sep '12

Thu Aug 23 13:15:31 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Bump base64-bytestring dependency Ignore-this: 27b56b44488ca6d13ec612c62ab8aa7f M ./hackage-server.cabal -1 +1 Thu Aug 23 13:30:04 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Remove redundant warning suppression flags Ignore-this: d45d30c13b2751e8583c43704ffd156d M ./hackage-server.cabal -2 Thu Aug 23 13:32:32 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Remove -fno-warn-warnings-deprecations flags Ignore-this: 87dde3829cc35154cb09e8d7b8fbc588 M ./hackage-server.cabal -3 Thu Aug 23 13:33:48 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Remove -fno-warn-unused-do-bind flags Ignore-this: faa2118e4c81641fb6b98c812e495256 M ./hackage-server.cabal -3 +3 Thu Aug 23 13:35:12 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Remove -fno-warn-orphans flags Ignore-this: 64c60f02f9a4a163b275c7110e6a692e M ./hackage-server.cabal -3 Thu Aug 23 13:58:46 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Use serveDirectory rather than the deprecated fileServe Ignore-this: e70ddbad0142ee975282813ea03ba0fd I haven't included defaultIxFiles in the list of index files, as I don't think we use them. M ./Distribution/Server.hs -1 +1 Thu Aug 23 14:00:55 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Fix another deprecated warning Ignore-this: 997f4664cb083925ab9c6c55ea748de9 M ./Distribution/Server.hs -3 +1 Thu Aug 23 14:14:02 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Fix warnings and whitespace in HaddockLex.x Ignore-this: f033ec0d03858ef4baedc3f96174b6af M ./Distribution/Server/Pages/Package/HaddockLex.x -32 +38 Thu Aug 23 14:24:06 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Fix warnings and whitespace in HaddockParse.y Ignore-this: 6d2dd9b6f79736ddf5e1a6ae2cbcff61 M ./Distribution/Server/Pages/Package/HaddockParse.y -54 +62 Thu Aug 23 14:24:45 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Remove unused import Ignore-this: 654b35b89e9155f9915b76025747f6c4 M ./Distribution/Server/Pages/Package/HaddockParse.y -2 Thu Aug 23 19:07:27 BST 2012 Ian Lynagh <ian(a)well-typed.com> * Remove comments re eliminating some deps from hackage-{mirror,build} Ignore-this: 9e6e17ae4547f97dff88c2dc6225ad26 I don't think the sort of refactoring that would be necessary is worthwhile. M ./hackage-server.cabal -2

1 0

Hackage 2 blocking tickets
by Ian Lynagh 07 Sep '12

07 Sep '12

Hi all, FYI, I've been making tickets labelled 'hackage2' and 'important' for issues that we need to look at before migrating to Hackage2. The list is here: https://github.com/haskell/cabal/issues?labels=hackage2%2Cimportant&page=1&… Currently we have: * https://github.com/haskell/cabal/issues/1027 Migrate deprecations from Hackage1 * https://github.com/haskell/cabal/issues/1029 Improve user signup etc * https://github.com/haskell/cabal/issues/1030 Import user accounts from Hackage 1 * https://github.com/haskell/cabal/issues/1031 Rewrite to use SQL backend I don't think we're all agreed to do the SQL rewrite at the moment, but we should at least make a conscious decision about it. Thanks Ian

3 5

Streamlining Account Requests
by Leon Smith 07 Sep '12

07 Sep '12

On Wed, Sep 5, 2012 at 9:57 AM, Duncan Coutts <duncan.coutts(a)googlemail.com> wrote: > In the new server the security is a little better, but we still want > to have a manual step to grant uploader rights. > Well, my impression is that the new server has a number of incomplete features that we are ignoring for now, and that we are adopting essentially the same model as before. In particular the per-package upload groups have been disabled as we aren't importing the old accounts and have no way to initially populate the groups. And personally, I'm ok with leaving this functionality disabled for the indefinite future. However, I do think we need to streamline the account request process. I have a pretty good idea of what I would like to have for LtU, which might also be a good starting point for hackage. Basically: 1. An account request form, that would require a username, a private email address, a private comment to the administrators, and some public profile information. The profile information is there mostly because spammers love to put stuff in there that would identify themselves as spammers. (But this might not be a huge problem on the new hackage? The web form might still attract spam, however.) Even so, I do think some kind of minimalistic profile would be nice to provide for accounts. 2. An email confirmation system. The private email address would generally need to be confirmed before an adminstrator would review the request. 3. A administrator page that lists outstanding requests with links to sub-pages to review each individual request. 4. Each sub-page would include all relevant information with respect to the request on one page: A. Username, private email, private comment, and public profile information B. IP address and User-Agent header associated with both the request and confirmation C. Link to a google search on the email address D. Comments and evaluations by other administrators 5. Administrators would have the ability to classify the request, as well as give a level of certainty of that classification. This would only be for communication with other administrators as well as to collect data that we might someday apply some machine learning techniques to. Approval or rejection would be a separate process. 6. That the system would automatically send an email once an request is approved. This would be a great system for LtU, I don't know how well it would also fit hackage. However the phrase "beggars can't be choosers" comes to mind; I'm not very good at web dev, and I have plenty of other things on my plate. I did get a PostgreSQL schema put together for my system this weekend, and a very minimal start on a Snap web app as a proof of concept. My idea for integration, if this basic approach eventually gets to production, would be to use Data.Acid.Remote to create accounts inside the existing server. Best, Leon

3 3

Hackage 2 and acid-state vs traditional databases
by Ian Lynagh 07 Sep '12

07 Sep '12

Hi all, I've had a bit of experience with Hackage 2 and acid-state now, and I'm not convinced that it's the best fit for us: * It's slow. It takes about 5 minutes for me to stop and then start the server. It's actually surprising just how slow it is, so it might be possible/easy to get this down to seconds, but it still won't be instantaneous. * Memory usage is high. It's currently in the 700M-1G range, and to get it that low I had to stop the parsed .cabal files from being held in memory (which presumably has an impact on performance, although I don't know how significant that is), and disable the reverse dependencies feature. It will grow at least linearly with the number of package/versions in Hackage. * Only a single process can use the database at once. For example, if the admins want a tool that will make it easier for them to approve user requests, then that tool needs to be integrated into the Hackage server (or talk to it over HTTP), rather than being standalone. * The database is relatively opaque. While in principle tools could be written for browsing, modifying or querying it, currently none exist (as far as I know). * The above 2 points mean that, for example, there was no easy way for me to find out how many packages use each top-level module hierarchy (Data, Control, etc). This would have been a simple SQL query if the data had been in a traditional database, but as it was I had to write a Haskell program to process all the package .tar.gz's and parse the .cabal files manually. * acid-state forces us to use a server-process model, rather than having processes for individual requests run by apache. I don't know if we would have made this choice anyway, so this may or may not be an issue. But the current model does mean that adding a feature or fixing a bug means restarting the process, rather than just installing the new program in-place. Someone pointed out that one disadvantage of traditional databases is that they discourage you from writing as if everything was Haskell datastructures in memory. For example, if you have things of type data Foo = Foo { str :: String, bool :: Bool, ints :: [Int] } stored in a database then you could write either: foo <- getFoo 23 print $ bool foo or b <- getFooBool 23 print b The former is what you would more naturally write, but would require constructing the whole Foo from the database (including reading an arbitrary number of Ints). The latter is thus more efficient with the database backend, but emphasises that you aren't working with regular Haskell datastructures. This is even more notable with the Cabal types (like PackageDescription) as the types and various utility functions already exist - although it's currently somewhat moot as the current acid-state backend doesn't keep the Cabal datastructures in memory anyway. The other issue raised is performance. I'd want to see (full-size) benchmarks before commenting on that. Has anyone else got any thoughts? On a related note, I think it would be a little nicer to store blobs as e.g. 54/54fb24083b14b5916df11f1ffcd03b26/foo-1.0.tar.gz rather than 54/54fb24083b14b5916df11f1ffcd03b26 I don't think that this breaks anything, so it should be noncontentious. Thanks Ian

5 7

Hackage2 server ready for testing
by Ian Lynagh 07 Sep '12

07 Sep '12

Hi all, All the bugs identified in the Hackage 2 server as blockers have now been fixed, and I have a mirror of Hackage running here: http://new-hackage.haskell.org/ I suggest that we proceed with: * A 3 week testing period, to be announced on haskell@ and libraries@ * Assuming no new blocking bugs are found which justify additional testing time, disable uploads to the Hackage 1 server * Throw the testing Hackage 2 instance away, and make a new mirror from the Hackage 1 server * Point hackage.haskell.org at the new Hackage 2 instance I think that a 3 week testing period will be sufficient as I expect most testing will happen in the first week or two, or the last few days, so any extra time is essentially wasted. I was actually initially going to propose 2 weeks, but given the timing of ICFP et al I think an extra week is justified. The reason I propose throwing away this test instance is that I think it's important that people are able to test uploading, and therefore uploading dummy packages and dummy versions should be encouraged. Does that plan sound reasonable to everyone? Is anyone willing to volunteer to be an admin for the new Hackage server please? (people need to have an admin add their account to the uploaders group before they can upload). A couple more notes: The mirror is a week or two old, so the latest package uploads won't be on it. I don't plan to set up continuous mirroring, as I think it may be more useful for people to be able to upload new versions to both servers themselves, for testing. The documentation builder has built (or failed to build) docs for the latest versions of all packages, but it's still working through older versions. It will therefore take up to 35mins before it notices that a new package has been uploaded, and builds docs for it. Finally, I would like to thank the Industrial Haskell Group (http://industry.haskell.org/) for funding this work. Thanks Ian -- Ian Lynagh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

6 14

Re: Hackage 2
by Ian Lynagh 06 Sep '12

06 Sep '12

[moving to cabal-dev@] On Thu, Aug 30, 2012 at 08:13:01PM -0400, Leon Smith wrote: > Ok, I tried to upload the most recent version of postgresql-simple, and I > couldn't because I'm not the maintainer for that package. > > Has anybody ever uploaded a package that they shouldn't have on the old > Hackage? Is this security really necessary at this stage in the game? > > I'm very much of the philosophy that, given that we must approve > accounts, that we rely on social processes instead of technical solutions > for these types of access control issues until (and unless) experience > proves that we do need some kind of technical solution. But by then, > hopefully we'll have a better idea of what we need. I wasn't involved in the design of that, so I wonder if someone who was could comment? The most analogous large system I'm familiar with is Debian, which AFAIK has no technical measures to stop any developer uploading any package, but is careful about who it allows to become a developer. Perhaps it is redundant to have both the "uploaders" group and the "per-package uploaders" functionality enabled, and even if the software continues to support both, we should only enable one or the other on the central Hackage site? In which case, which do we want? If we do stay with the "per-package uploaders" feature, then I wonder whether we should have a special case for packages with an empty access list (i.e. all currently existing packages), such that anyone can upload a new version of them, and anyone doing so becomes the sole uploader for that package? That would avoid the admins having to get involved for every package, as well as every person. Thanks Ian

8 18

Hackage Accounts
by Leon Smith 06 Sep '12

06 Sep '12

I have a question for Ross Paterson, namely do you decline an account request, and if so, what reasons do you have for declining an account request? I mean, I would guess the current process doesn't attract spam account requests, and it's hard to see how you would know that a given account has malicious intent... Best, Leon

5 7

Hackage 2
by Ross Paterson 06 Sep '12

06 Sep '12

A few minor issues: In Hackage 1 packages can be deprecated, optionally indicating a recommended replacement (the "deprecated" and "superseded by" fields of the per-package "tags" file), e.g. partial-lens and binary-strict-0.3.1. These don't seem to be in Hackage 2. Several packages lack documentation. They don't say why (an issue in itself), but it appears to be libraries that interface to C libraries, and libraries included with GHC. There's little to be done about the former, but it's really useful to have the documentation of the GHC libraries there too. The base package can't be built outside the GHC tree, but I think it warrants special treatment. The package her-lexer-0.1 shows links to documentation, but they lead to an error message.

3 5