- cabal-devel - Haskell.org

RE: Advance notice that I'd like to make Cabal depend on parsec
by Simon Peyton-Jones 21 Mar '13

21 Mar '13

Is it essential, or even sensical, that the serialization format GHC needs for storing package info bear any relation to the human authored form? If not, the split out of the package types could be accomplished in a way where GHC uses simple show/read(P) style serialization for storage of package info, where as cabal-lib would use a lovely parsec parser for humans. I'd like this approach. Good idea -- esp if it makes the packaging story simpler. GHC already uses a binary format for interface files, so there’s no good reason to use a human-readable format for package data base stuff. For interface files you can read them with ghc --show-iface, and as Ian remarks something similar is already true for the package data base. Simon From: ghc-devs-bounces(a)haskell.org [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Mark Lentczner Sent: 17 March 2013 16:57 To: dag.odenhall(a)gmail.com Cc: Haskell Libraries; cabal-devel; Duncan Coutts; ghc-devs(a)haskell.org; Antoine Latter Subject: Re: Advance notice that I'd like to make Cabal depend on parsec This thread is raising all sorts of questions for me: Is it essential, or even sensical, that the serialization format GHC needs for storing package info bear any relation to the human authored form? If not, the split out of the package types could be accomplished in a way where GHC uses simple show/read(P) style serialization for storage of package info, where as cabal-lib would use a lovely parsec parser for humans. I'd like this approach. The issue of putting the yet one more HP package into GHC's core packages is increasing the exposure of the difficulty of the current GHC/HP relationship. See also threads in HP's mailing list for why can't we bump some packages in GHC's core set for the next HP release. The split arrangement is strange because we have two groups making up what is in the HP, but they have different processes and aims. The complex technical relationship between the moving parts only heightens the difficulty. Perhaps the major cause is that because GHC is shipped as a library itself, it exposes all it's package dependencies. And as it is a large, and growing, piece of software, the list only wants to grow. But I wonder how often GHC is used as a library itself? If not often, then perhaps GHC should be shipped as two parts: Just a compiler (plus the small number of packages that the compiler forces), and ghc-lib as an optional, even separate, package - perhaps one with even a traditional way of depending on other packages. In otherwords, users that wanted to incorporate the ghc-lib into their programs would depend, and download, and configure, and build, ghc-lib indpenendant of the GHC binaries installed on their system. Perhaps then, GHC, the compiler, built from ghc-lib, would be bootstrapped not from the past compiler, but from the past HP..... Okay, perhaps that is all just fantasy. But, no other programming system operates the way we do. They all fall into one of two camps: * The dominant implementation is maintained, built, and shipped along with a large collection of "common packages". Examples: Python, Ruby, PHP, Java. * The dominant implementation is shipped as a bare tool, and large common libraries are maintained and shipped independently. Examples: C++ (think g++ and boost), JavaScript (think browsers, and jQuery). We are in the middle and, I think, experiencing growing pains because of it. - Mark On Sat, Mar 16, 2013 at 3:42 PM, dag.odenhall(a)gmail.com<mailto:dag.odenhall@gmail.com> <dag.odenhall(a)gmail.com<mailto:dag.odenhall@gmail.com>> wrote: I'd love to have a proper parser and source-location-aware AST for sake of editor/IDE tools, so +1 from me. If you don't end up doing this after all, I'd still like to see your parser in a separate package, although I understand if you don't feel like maintaining two parsers especially given the tedious process for verifying they work similarly. I guess it could still be useful in the same way we find haskell-src-exts useful despite some incompatibilities with GHC. On Thu, Mar 14, 2013 at 3:53 PM, Duncan Coutts <duncan.coutts(a)googlemail.com<mailto:duncan.coutts@googlemail.com>> wrote: Hi folks, I want to give you advance notice that I would like to make Cabal depend on parsec. The implication is that GHC would therefore depend on parsec and thus it would become a core package, rather than just a HP package. So this would affect both GHC and the HP, though I hope not too much. The rationale is that Cabal needs to parse things, like .cabal files and currently we do not have a decent parser in the core libraries. By decent I mean one that can produce error messages with source locations and that doesn't have unpredictable memory use. The only parser in the core libraries at the moment is Text.ParserCombinators.ReadP from the base package and that fails my "decent" criteria on both counts. Its idea of an error message is (), and on some largish .cabal files we take 100s of MB to parse (I realise that the ReadP in the base package is a cutdown version so I don't mean to malign all ReadP-style libs out there). Partly due to the performance problem, the terrible .cabal file error messages, and partly because Doaitse Swierstra keeps asking me if .cabal files have a grammar, I've been writing a new .cabal parser. It uses an alex lexer and a parsec parser. It's fast and the error messages are pretty good. I have reverse engineered a grammar that closely matches the existing parser and .cabal files in the wild, though I'm not sure Doaitse will be satisfied with the approach I've taken to handling layout. Why did I choose parsec? Practicality dictates that I can only use things in the core libraries, and the nearest thing we have to that is the parser lib that is in the HP. I tried to use happy but I could not construct a grammar/lexer combo to handle the layout (also, happy is not exactly known for its great error messages). I've been doing regression testing against hackage and I'm satisfied that the new parser matches close enough. I've uncovered all kinds of horrors with .cabal files in the wild relying on quirks of the old parser. I've made adjustments for most of them but I will be breaking a half dozen old packages (most of those don't actually build correctly because though their syntax errors are not picked up by the parser, they do cause failure eventually). So far I've just done the outline parser, not the individual field parsers. I'll be doing those next and then integrate. So this change is still a bit of a ways off, but I thought it'd be useful to warn people now. Duncan _______________________________________________ cabal-devel mailing list cabal-devel(a)haskell.org<mailto:cabal-devel@haskell.org> http://www.haskell.org/mailman/listinfo/cabal-devel

2 1

Re: Advance notice that I'd like to make Cabal depend on parsec
by Duncan Coutts 18 Mar '13

18 Mar '13

On Sun, 2013-03-17 at 09:57 -0700, Mark Lentczner wrote: > This thread is raising all sorts of questions for me: [..] > The issue of putting the yet one more HP package into GHC's core packages > is increasing the exposure of the difficulty of the current GHC/HP > relationship. See also threads in HP's mailing list for why can't we bump > some packages in GHC's core set for the next HP release. The split > arrangement is strange because we have two groups making up what is in the > HP, but they have different processes and aims. The complex technical > relationship between the moving parts only heightens the difficulty. This is certainly worth thinking about. Perhaps parsec is the straw that broke the camel's back. It's not qualitatively different from the other core libs: for all of them we have the issue about version pinning and the effects on the release cycle. Duncan

1 0

Re: Advance notice that I'd like to make Cabal depend on parsec
by Ian Lynagh 18 Mar '13

18 Mar '13

On Sun, Mar 17, 2013 at 09:57:25AM -0700, Mark Lentczner wrote: > > Is it essential, or even sensical, that the serialization format GHC needs > for storing package info bear any relation to the human authored form? If > not, the split out of the package types could be accomplished in a way > where GHC uses simple show/read(P) style serialization for storage of > package info, where as cabal-lib would use a lovely parsec parser for > humans. I'd like this approach. I think it would be feasible to stop GHC itself from using the human readable format. The only place I can think of it being used is in the package database, but we could use either Read/Show for that, or just exclusively use the binary format. It would be a little less user-friendly, but maybe worth it to remove the ghc library dependencies on most-of-Cabal, mtl and parsec. > Perhaps the major cause is that because GHC is shipped as a library itself, > it exposes all it's package dependencies. Yes. > In otherwords, users that wanted to > incorporate the ghc-lib into their programs would depend, and download, and > configure, and build, ghc-lib indpenendant of the GHC binaries I think this would create more problems than it solves. > Okay, perhaps that is all just fantasy. But, no other programming system > operates the way we do. They all fall into one of two camps: > > - The dominant implementation is maintained, built, and shipped along > with a large collection of "common packages". Examples: Python, Ruby, PHP, > Java. > - The dominant implementation is shipped as a bare tool, and large > common libraries are maintained and shipped independently. Examples: C++ > (think g++ and boost), JavaScript (think browsers, and jQuery). > > We are in the middle and, I think, experiencing growing pains because of it. I would say that we are doing the first option, in the form of the HP. It's just that the core gets frozen (i.e., ghc + libs gets released) earlier than the higher level libraries. I don't think that moving (back) to trying to freeze/release everything all at once would be an improvement. You just need to remain strong, and keep saying "no" :-) (you're doing a great job, BTW!) Thanks Ian

3 7

Advance notice that I'd like to make Cabal depend on parsec
by Duncan Coutts 16 Mar '13

16 Mar '13

Hi folks, I want to give you advance notice that I would like to make Cabal depend on parsec. The implication is that GHC would therefore depend on parsec and thus it would become a core package, rather than just a HP package. So this would affect both GHC and the HP, though I hope not too much. The rationale is that Cabal needs to parse things, like .cabal files and currently we do not have a decent parser in the core libraries. By decent I mean one that can produce error messages with source locations and that doesn't have unpredictable memory use. The only parser in the core libraries at the moment is Text.ParserCombinators.ReadP from the base package and that fails my "decent" criteria on both counts. Its idea of an error message is (), and on some largish .cabal files we take 100s of MB to parse (I realise that the ReadP in the base package is a cutdown version so I don't mean to malign all ReadP-style libs out there). Partly due to the performance problem, the terrible .cabal file error messages, and partly because Doaitse Swierstra keeps asking me if .cabal files have a grammar, I've been writing a new .cabal parser. It uses an alex lexer and a parsec parser. It's fast and the error messages are pretty good. I have reverse engineered a grammar that closely matches the existing parser and .cabal files in the wild, though I'm not sure Doaitse will be satisfied with the approach I've taken to handling layout. Why did I choose parsec? Practicality dictates that I can only use things in the core libraries, and the nearest thing we have to that is the parser lib that is in the HP. I tried to use happy but I could not construct a grammar/lexer combo to handle the layout (also, happy is not exactly known for its great error messages). I've been doing regression testing against hackage and I'm satisfied that the new parser matches close enough. I've uncovered all kinds of horrors with .cabal files in the wild relying on quirks of the old parser. I've made adjustments for most of them but I will be breaking a half dozen old packages (most of those don't actually build correctly because though their syntax errors are not picked up by the parser, they do cause failure eventually). So far I've just done the outline parser, not the individual field parsers. I'll be doing those next and then integrate. So this change is still a bit of a ways off, but I thought it'd be useful to warn people now. Duncan

12 24

cabal install --only
by Mikhail Glushenkov 15 Mar '13

15 Mar '13

Hi all, What is the purpose of the "install" command's "--only" flag? The documentation says "Only installs the package in the current directory", but this is already supported by running "cabal install" in the current directory without arguments. Would it be fine if I removed "--only"? -- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

5 6

Re: Advance notice that I'd like to make Cabal depend on parsec
by Duncan Coutts 15 Mar '13

15 Mar '13

On Fri, 2013-03-15 at 10:10 +0100, Herbert Valerio Riedel wrote: > Duncan Coutts <duncan.coutts(a)googlemail.com> writes: > > > [...] > > > I've been doing regression testing against hackage and I'm satisfied > > that the new parser matches close enough. I've uncovered all kinds of > > horrors with .cabal files in the wild relying on quirks of the old > > parser. I've made adjustments for most of them but I will be breaking a > > half dozen old packages (most of those don't actually build correctly > > because though their syntax errors are not picked up by the parser, they > > do cause failure eventually). > > btw, why not just keep the current parser as a "legacy" parser in the > code, for older .cabal files (or as a fallback parser, in case the new > stricter parsec-parser fails)? I'm satisfied at this point that the number of packages affected by the change is so low that it's not worth the extra maintenance. As I mentioned, most of the ones that break in the new parser are actually already broken in the sense that they will not build (because of mistakes in the .cabal file that just happen not to be caught by the old parser). So the amount of "real" breakage is trivial. Also, with the new hackage server we will be able to fix .cabal files post-release so if we do care about those few older packages we can actually fix them. Duncan

1 0

Re: Advance notice that I'd like to make Cabal depend on parsec
by Roman Cheplyaka 15 Mar '13

15 Mar '13

* Ivan Lazar Miljenovic <ivan.miljenovic(a)gmail.com> [2013-03-15 22:12:47+1100] > On 15 March 2013 22:05, Roman Cheplyaka <roma(a)ro-che.info> wrote: > > * Heinrich Apfelmus <apfelmus(a)quantentunnel.de> [2013-03-15 10:38:37+0100] > >> Duncan Coutts wrote: > >> >Hi folks, > >> > > >> >I want to give you advance notice that I would like to make Cabal depend > >> >on parsec. The implication is that GHC would therefore depend on parsec > >> >and thus it would become a core package, rather than just a HP package. > >> >So this would affect both GHC and the HP, though I hope not too much. > >> > > >> >[..] > >> > > >> >Why did I choose parsec? Practicality dictates that I can only use > >> >things in the core libraries, and the nearest thing we have to that is > >> >the parser lib that is in the HP. I tried to use happy but I could not > >> >construct a grammar/lexer combo to handle the layout (also, happy is not > >> >exactly known for its great error messages). > >> > >> Reuse is good, but the implication I'm worried about is this: Can I > >> upgrade the parsec package installed on my system by doing a user > >> install from hackage ? Without an implementation of more flexible > >> package installations (multiple versions installed simultaneously), > >> any dependency of GHC has its version number essentially set into > >> stone. > > > > We've had that working for a long time. Right now I even have multiple > > installed versions of Cabal-the-library itself. > > > > It's not that Parsec would be automatically linked into each executable. > > It's just that ghc-the-program would have Parsec linked into it. > > And ghc-the-library, which means that anything that uses > ghc-as-a-library (and indeed even Cabal-as-a-library) no longer has a > choice of which version of parsec they use. Right. But in this regard, GHC API and Cabal are no different from any other libraries that suffer from the same issue. (Except that it's hard to recompile GHC to use an alternative Parsec version.) And these are not exactly the most popular libraries either — so I doubt this change will have a large impact. Roman

1 0

patch applied (hackage-server): "Fix data files wildcard" and 1 others
by devnull＠community.haskell.org 13 Mar '13

13 Mar '13

Tue Mar 12 22:52:33 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Fix data files wildcard Ignore-this: 1b975f7ba945befd15a16526a469af4d M ./hackage-server.cabal -1 +1 Wed Mar 13 00:41:42 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Change the host-URI handling so it is independent of the port we serve on Ignore-this: 7c9cfd79a000cf9f0cb13f2a253b439c The public URI for the server may involve proxying etc, so the public URI might be http://new-hackage.haskell.org/ while we're actually serving on port 18512, which is reverse proxied to port 80. So we need to be able to set the public URI independently of our serving port, otherwise we would make links to http://new-hackage.haskell.org:18512/ which isn't accessable. (Most links are relative, but some have to be absolute, e.g. in emails and in rss feeds) M ./Distribution/Server.hs -11 +9 M ./Distribution/Server/Features/LegacyPasswds.hs -6 +3 M ./Distribution/Server/Features/NameSearch.hs -7 +4 M ./Distribution/Server/Features/RecentPackages.hs -1 +1 M ./Distribution/Server/Features/UserSignup.hs -15 +16 M ./Distribution/Server/Framework/ServerEnv.hs -1 +1 M ./Distribution/Server/Pages/Recent.hs -12 +8 M ./Main.hs -19 +47

1 0

patch applied (hackage-server): "Revert to --static-dir and --state-dir flag names"
by devnull＠community.haskell.org 12 Mar '13

12 Mar '13

Tue Mar 12 22:48:52 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Revert to --static-dir and --state-dir flag names Ignore-this: a232d4ef02db77555703650d8bb930bc M ./Main.hs -2 +2

1 0

patch applied (hackage-server): "Tidy up the command line flags code, make sure all commands have a -v flag" and 7 others
by devnull＠community.haskell.org 12 Mar '13

12 Mar '13

Thu Mar 7 22:43:10 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Tidy up the command line flags code, make sure all commands have a -v flag Ignore-this: 817fc3128b7aa928a376b366b6fba58b The backup and restore commands in particular really need that verbosity control, if only for testing/debugging. M ./Main.hs -98 +126 Fri Mar 8 17:15:33 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Change the backup/restore to keep blob files outside of the tarball Ignore-this: 17eb98bb6956b8c998caac622f2f8d23 Previously a full backup would cost about 17Gb, now the incremental cost can be around 13Mb. M ./Distribution/Server/Features/Core/Backup.hs -1 +11 M ./Distribution/Server/Features/Upload.hs -1 +1 M ./Distribution/Server/Framework/BackupDump.hs -68 +144 M ./Distribution/Server/Framework/BackupRestore.hs -94 +109 M ./Distribution/Server/Framework/BlobStorage.hs -16 +21 M ./Distribution/Server/Framework/Feature.hs -6 +6 M ./Main.hs -73 +150 M ./hackage-server.cabal +1 Tue Mar 12 14:47:46 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Fixes for backup restore in "consume blobs" mode Ignore-this: 52b4c67aa754625e256ceccd988d163b M ./Distribution/Server/Framework/BackupRestore.hs -21 +46 Tue Mar 12 14:49:03 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Make sure all the server commands that need it have the --static flag Ignore-this: bc565314bbc9532bddc8e9cf52aa9421 It's needed by lots of them now, as several features now load templates. M ./Main.hs -48 +75 Tue Mar 12 14:50:10 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Fix the importing of maintainer groups Ignore-this: 21bbd2389d1a90e13c83ca8d044af31e Involves rewriting how the group lookup works. For groupResourcesAt we have not a single group but an indexed collection of groups, and so we need a way to lookup a group / check if a group exists. The previous solution involved the lookup never failing and having a method on the group to check if it is actually exists / if it's backed by something. Not only is that annoying and backwards but it meant we were doing the wrong thing in some cases. For maintainer groups we were checking if the maintainer group existed in the db, which meant we could not add members to the group if it was initially empty (because it was counted as not existing). That showed up as the import client failing to add users to the per-package maintainer groups (which all start out empty). So we get rid of the groupExists checks and instead allow the group lookup function to fail. M ./Distribution/Server/Features/BuildReports.hs -3 +3 M ./Distribution/Server/Features/Distro.hs -1 M ./Distribution/Server/Features/Html.hs -14 +15 M ./Distribution/Server/Features/Mirror.hs -1 M ./Distribution/Server/Features/PackageCandidates.hs -5 +4 M ./Distribution/Server/Features/Upload.hs -59 +48 M ./Distribution/Server/Features/Users.hs -41 +41 M ./Distribution/Server/Users/Group.hs -4 M ./ImportClient.hs -1 +1 Tue Mar 12 21:58:45 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Extend the ErrorResponse type with headers and use it in auth errors Ignore-this: 3b7118227e7373a2c90e7053fb8fb7b5 So we can now use throwError for auth errors. Previously we had to use finishWith which would bypass all the custom error page stuff. So this will give us pretty pages for auth errors. M ./Distribution/Server.hs -3 +4 M ./Distribution/Server/Features/LegacyPasswds.hs -6 +4 M ./Distribution/Server/Features/PackageCandidates.hs -2 +2 M ./Distribution/Server/Features/StaticFiles.hs -10 +22 M ./Distribution/Server/Features/Upload.hs -2 +2 M ./Distribution/Server/Features/Users.hs -13 +17 M ./Distribution/Server/Framework/Auth.hs -18 +23 M ./Distribution/Server/Framework/Error.hs -12 +24 Tue Mar 12 22:40:05 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Improve the pages used in the account upgrade process Ignore-this: 83cb10ecd4d779c59b1eaec0b3f7ac03 ./datafiles/templates/account-upgrade.html.st -> ./datafiles/templates/LegacyPasswds/htpasswd-upgrade.html.st M ./Distribution/Server/Features/LegacyPasswds.hs -11 +34 M ./Distribution/Server/Framework/Templating.hs -1 +2 A ./datafiles/templates/LegacyPasswds/ A ./datafiles/templates/LegacyPasswds/htpasswd-upgrade-success.html.st M ./datafiles/templates/LegacyPasswds/htpasswd-upgrade.html.st -1 +1 M ./hackage-server.cabal +1 Tue Mar 12 22:41:30 GMT 2013 Duncan Coutts <duncan(a)well-typed.com> * Revert to the previous time format in backup tarballs Ignore-this: c0890eafee32c44f99742ace139227c6 M ./Distribution/Server/Framework/BackupRestore.hs -2 +5

1 0