May 2009 - cabal-devel

patch applied (cabal): Register XmlSyntax and RegularPatterns as known extensions in Language.Haskell.Extension
by Duncan Coutts 30 May '09

30 May '09

Fri May 29 03:28:48 PDT 2009 Niklas Broberg <d00nibro(a)chalmers.se> * Register XmlSyntax and RegularPatterns as known extensions in Language.Haskell.Extension Ignore-this: 32aacd8aeef9402a1fdf3966a213db7d Concrete XML syntax is used in the Haskell Server Pages extension language, and a description can be found in the paper "Haskell Server Pages through Dynamic Loading" by Niklas Broberg, published in Haskell Workshop '05. Regular expression pattern matching is described in the paper "Regular Expression Patterns" by Niklas Broberg, Andreas Farre and Josef Svenningsson, published in ICFP '04. M ./Language/Haskell/Extension.hs +14 View patch online: http://darcs.haskell.org/cabal/_darcs/patches/20090529102848-fef0b-68434794…

1 0

draft proposal for relative paths in installed package descriptions
by Duncan Coutts 29 May '09

29 May '09

All, This is a draft proposal for a common mechanism for implementing relative paths in installed package descriptions. Comments and feedback are welcome. I'm cc'ing this to the cabal and ghc-users lists but let's keep the discussion on the libraries list. There has been some discussion of this issue on the cabal and ghc-users list, but it's a relatively long thread and the idea has evolved somewhat so this is an attempt to present the proposal clearly. Proposal ======== This proposal is an extension to the Cabal spec section 3 http://haskell.org/cabal/proposal/x272.html Motivation ---------- Being able to have relative paths in the installed package description is useful as it makes it possible to construct relocatable (prefix-independent) package installations. This is useful when distributing compilers along with packages and there may be other uses. This proposal does not require that all paths are relative. It is still perfectly ok to use absolute paths where appropriate. It just adds an option to use relative paths. The aim is for a single simple specification that any compiler can implement and have the tools work uniformly, rather than ad-hoc implementations for each compiler. Details ------- In the installed package description, we will allow paths that begin with the variables ${pkgroot} or ${pkgrooturl}. For example: library-dirs: ${pkgroot}/foo-1.0 haddock-html: ${pkgrooturl}/doc/foo-1.0 They may only appear at the beginning of the paths and must be followed by a directory separator (platform-dependent '/' or '\'). This is because the vars refer to a directory so it does not make sense to construct extensions like ${pkgroot}blah. The use of '{}' is required and is to avoid any ambiguity (especially since the string "$pkgrooturl" is otherwise an extension of "$pkgroot"). Directly relative file paths like "blah" or "./blah" are not allowed. Fields containing paths must be absolute or begin with ${pkgroot}. Field containing urls must be absolute or begin with ${pkgrooturl}. The var ${pkgroot} is to be interpreted as the directory containing the installed package description. For ghc this will be the dir containing the package.conf db, for hugs/nhc for each package there is a specific installed package description file, and ${pkgroot} is thus the directory containing that file. The syntax of the string representing this directory is the usual system-dependent filepath syntax, e.g. windows: c:\ghc\ghc-6.12.1 unix: /opt/ghc-6.12.1 The var ${pkgrooturl} is to be interpreted as a url representation of the directory containing the installed package description. For ghc this will be the dir containing the package.conf db, for hugs/nhc for each package there is a specific installed package description file, and ${pkgroot} is thus the directory containing that file. The syntax of the string representing this directory is as a valid file url including the file:// prefix e.g. windows: file:///c:/ghc/ghc-6.12.1 unix: file:///opt/ghc-6.12.1 This is similar to how relative paths in .cabal files are interpreted relative to the directory containing the .cabal file, however in this case we mark relative paths more explicitly using a variable. So in the original example library-dirs: ${pkgroot}/foo-1.0 haddock-html: ${pkgrooturl}/doc/foo-1.0 If we assume this installed package description is at c:\ghc\ghc-6.12.1 \package.conf on windows or /opt/ghc-6.12.1/package.conf on unix then the compiler and other tools would interpret these as library-dirs: c:\ghc\ghc-6.12.1/foo-1.0 haddock-html: file:///c:/ghc/ghc-6.12.1/doc/foo-1.0 or on unix: library-dirs: /opt/ghc-6.12.1/foo-1.0 haddock-html: file:///opt/ghc-6.12.1/doc/foo-1.0 Tools ----- Tools which process the installed package description format will be expected to interpret these relative paths. This requires that they can discover the path for the ${pkgroot}. How they discover this is dependent on the Haskell implementation. Some implementations use separate files for each installed package description, some embed them in library package files, some use databases of installed package descriptions. Haskell implementations should provide a mechanism to discover the path for an installed package description. Duncan

1 0

Re: haddock-2.3.0 literate comments discarded from .lhs input
by Duncan Coutts 29 May '09

29 May '09

On Thu, 2009-05-28 at 23:40 +0100, Claus Reinke wrote: > >> If you don't want to move from absolute paths for non-core packages, > >> the current system should just work, right? > > > > Yes. > > The current system being the $topdir one. Yep. It works, it's just not nice, it's ghc-specific and only make sense when ghc is installed in a prefix-independent way. > > Though it also allows for the possibility of relocatable sets of > > packages that are not installed relative to the compiler. But more > > importantly it's more general and simpler than the current '$topdir' > > that ghc uses. > > 'it' now being the new system evolving in this thread, or have I missed > anything? The new system I've been proposing. > >> (a) making ghc-pkg (optionally) instantiate any variables in its > >> database in (all of) its command-line output and > > > > Yes, though I'm only asking for two vars (previously one), not an ad-hoc > > set of vars. > > > >> (b) allowing non-core packages to be relocated without having to > >> update ghc-pkg's database. > > > > In my suggested system this is possible if that set of packages use > > their own package db (containing relative paths). > > That is news to me - was that specified before this thread moved > to ghc-users? It was in the first email that was cc'ed to ghc-users: How about this: a way to specify paths in the package registration info that are relative to the location of the package db they are in. That makes sense beyond just ghc and even with would allow other sets of relocatable packages, not just those installed with ghc. > > In your system it's possible by updating some var in a central registry > > and having that set of packages use paths relative to that var. > > So, essentially, your system would have to keep a file listing the > various package.conf locations (currently, GHC only knows about > two: system/user, everything else would have to be passed on the > commandline..). While my system would have to keep a file listing > the variable bindings, so that tools processing the package db can > instantiate the variables. If you want multiple relocatable sets of packages that are immediately "available" in the environment. > I could see both approaches being useful, even together. > > > So ghc's current system uses two vars, $topdir and $httptopdir. > > This is GHC's view of its database. It should be useable independently, > via ghc-pkg and ghc api clients (such as GHC, GHCi, Haddock, ..) - > all of which should be able to resolve the variable bindings, in the > same way. It's not usable independently, ghc does not always have a topdir. This makes life hard for tools. It's also not clear what topdir would mean in the context of other compilers. > Btw, it would really be nice if the package handling code > was shared rather than duplicated. It would be nice, yes. > > I'm proposing to replace those with a standardised ${pkgroot} and > > ${pkgrooturl} vars which are usable by all compilers and in more > > situations. > > Now you are talking about Cabal's view of its database. Cabal does not own the package databases, however it does expect that they are in the format describe by the Cabal spec, which places obligations on Haskell implementations to be somewhat package-aware. > It doesn't have to expose the underlying implementation's view, > especially since the other implementations organise their package > handling differently. All compilers use the same information (it's in the Cabal spec). They do store it differently but they all identify the location of the information using a file path. That seems pretty universal, compared to $topdir. > And why just two variables? Is $pkgroot about .hi files, .a/.so./.dll > files, or about include files, or haddock indices, or ..? You only need one variable to identify the location of the installed package description. All relative paths can be constructed from that. The second variable is to allow for two representations of the same location, one as a native system path, the other as a URL. We do not need different variables for different kinds of files (except in as much as some fields use paths and some urls). > In windows, these tend to end in a common sub-hierarchy, but you're > aiming for something general, right? If you're making a relocatable package then these files will be in a common sub-hierarchy and you would use relative paths. If you're not making a relocatable package (eg following the Linux FSH) then you would not use relative paths. So that should be general. It does not remove any existing capability and it adds the ability to have relative paths for relocatable packages. Perhaps what you're saying is that we should be able to take any package whether it lives in a common sub-hierarchy or not and relocate it. In general this is problematic since packages can embed paths and if those paths are not relative to a common root then you have to specify them all (Cabal enables this by setting environment variables). Assuming that's ok, then even this rare use case is still possible, just by editing the package registration information. It doesn't need to be "simplified" by having one var per package entry. > > You're proposing a central registry of vars and to have ghc-pkg > > (optionally) expand these vars which could be used anywhere in the > > installed package descriptions. Presumably you're also suggesting some > > mechanism to query and update this registry of variables. > > > > Is that a fair summary? > > I think so. And you're proposing several separate registries (hasn't > that been a Cabal problem in the past, even with just user and system > to choose from?). Cabal can be instructed to use a specific package db, not just global and user. > Presumably you're also suggesting some mechanism > to query and update the meta-registry of package database locations. I wasn't actually proposing a meta-registry (ghc already supports this via an environment variable) but it is a possible extension which brings it closer to the capabilities of your proposal. My proposal is just to replace the use of $topdir with something that every compiler can implement and which tools can understand. The fact that it could be extended without having to modify the installed package description format or the tools which understand that format is a bonus. I would not argue particularly for or against such an extension, my main concern is for a simple clear spec and the sanity of tool authors. Duncan

1 0

Re: Patch to register XmlSyntax and RegularPatterns as known extensions
by Duncan Coutts 29 May '09

29 May '09

On Fri, 2009-05-29 at 09:16 +0100, Magnus Therning wrote: > >> Are these papers available publicly anywhere? If so perhaps we should > >> link to those instead. The ACM links do not give full text access except > >> to registered users/institutions. > > > > Well, I have them on my own webpage, the reason I didn't link to that > > was that our department is going through a system reorganisation and > > those links won't be valid for more than a few months longer. I could > > of course submit a new patch later, but I'd really prefer to link to a > > central location that wouldn't require another patch from me each time > > I move somewhere. ACM seemed to me to be the most stable place that > > also provides the full text, if only to registered people. > > > > It seems I can upload the papers to Chalmers' central publication > > library, but it is a bit of a hassle... > > Interestingly the full papers can be downloaded from citeseer Then perhaps just refer to the papers by name, if they turn up so easily on google. Duncan

1 0

Re: Patch to register XmlSyntax and RegularPatterns as known extensions
by Isaac Dupree 28 May '09

28 May '09

Does HaRP/RegularPatterns have any relation to XHaskell (e.g. see http://code.google.com/p/xhaskell/wiki/homepage )? -Isaac Niklas Broberg wrote: > Hi all, > > I'd like to have concrete XML syntax (HSP) and regular expression > pattern syntax (HaRP) registered in Language.Haskell.Extension. The > attached patch registers XmlSyntax and RegularPatterns as constructors > in the Language.Haskell.Extension.Extension data type, along with > haddock documentation. > > Cheers, > > /Niklas > > > ------------------------------------------------------------------------ > > _______________________________________________ > Libraries mailing list > Libraries(a)haskell.org > http://www.haskell.org/mailman/listinfo/libraries

1 0

Re: Patch to register XmlSyntax and RegularPatterns as known extensions
by Duncan Coutts 28 May '09

28 May '09

On Thu, 2009-05-28 at 23:37 +0200, Niklas Broberg wrote: > > I'd like to have concrete XML syntax (HSP) and regular expression > > pattern syntax (HaRP) registered in Language.Haskell.Extension. The > > attached patch registers XmlSyntax and RegularPatterns as constructors > > in the Language.Haskell.Extension.Extension data type, along with > > haddock documentation. + -- | Allow concrete XML syntax to be used in expressions and patterns, + -- as per the Haskell Server Pages extension language: + -- <http://www.haskell.org/haskellwiki/HSP>. The ideas behind it are + -- discussed in this paper: + -- <http://portal.acm.org/citation.cfm?id=1088353> + | XmlSyntax + + -- | Allow regular pattern matching over lists, as discussed in the + -- following paper: <http://portal.acm.org/citation.cfm?id=1016863> + | RegularPatterns Are these papers available publicly anywhere? If so perhaps we should link to those instead. The ACM links do not give full text access except to registered users/institutions. > Eh, and to those reading this on the libraries list, the patch was > intended for cabal of course. :-) And the reason Niklas is cc'ing it here is to give people a chance to comment, eg to make sure the names are sensible and that the docs are ok. Of course these docs are far better than for all the other listed extensions! (patches gratefully accepted) Niklas is one of the first victims of the new hard-line policy on providing at least some description along with registered extension names. Duncan

1 0

Re: haddock-2.3.0 literate comments discarded from .lhs input
by Duncan Coutts 28 May '09

28 May '09

On Thu, 2009-05-28 at 14:12 +0100, Claus Reinke wrote: > > But if you're registering global packages that are installed outside of > > the GHC tree then you wouldn't register them using relative paths. I'm > > not saying everything must use relative paths. > > Please don't move your windmills while I'm fighting them!-) > > If you don't want to move from absolute paths for non-core packages, > the current system should just work, right? Yes. Though it also allows for the possibility of relocatable sets of packages that are not installed relative to the compiler. But more importantly it's more general and simpler than the current '$topdir' that ghc uses. > I thought we were talking about > (a) making ghc-pkg (optionally) instantiate any variables in its > database in (all of) its command-line output and Yes, though I'm only asking for two vars (previously one), not an ad-hoc set of vars. > (b) allowing non-core packages to be relocated without having to > update ghc-pkg's database. In my suggested system this is possible if that set of packages use their own package db (containing relative paths). In your system it's possible by updating some var in a central registry and having that set of packages use paths relative to that var. > > For some reason I really dislike the idea that we make up specific vars > > like $cabaltopdir for specific purposes. Perhaps that's just me. I want > > a general solution, not something that forces everyone to adopt > > conventions like installing everything in ~/.cabal/. That's just a > > sensible default, but the user rightly has full control over --prefix, > > --libdir etc etc. > > Personally, I only dislike the idea of hardcoding specific variable names > in ghc-pkg, which is why I suggested a name-independent approach > (I also dislike the current duplication of code in ghc-pkg/ghc api/..). > > $cabaltopdir would just improve the handling of the default cabal > install locations, without dictating where users say those default locations > should be - and if users move specific packages/package parts to > different absolute locations, those absolute locations would still have > to appear in the package database, but I'd expect that to be an > exception. So ghc's current system uses two vars, $topdir and $httptopdir. I'm proposing to replace those with a standardised ${pkgroot} and ${pkgrooturl} vars which are usable by all compilers and in more situations. You're proposing a central registry of vars and to have ghc-pkg (optionally) expand these vars which could be used anywhere in the installed package descriptions. Presumably you're also suggesting some mechanism to query and update this registry of variables. Is that a fair summary? > Let's say I wanted to move a GHC/Cabal/HP installation to a > USB drive: moving GHC/corelibs is straightforward (it doesn't > care under what drive name the USB drive gets mounted on the > lecture theatre computer), but how would I move Cabal-installed > non-core packages (not to mention Cabal itself?)? Is that use case > documented in some faq? Ok, so you want to construct a set of relocatable packages. This needs to be decided from the beginning when you compile said packages because otherwise packages can have paths baked into them. There are some restrictions on making relocatable packages, eg you can't set --libdir to an absolute path, it has to be relative to the --prefix. In addition to making the package relocatable, we would have to register the package into a package db that lives relative to the packages in question. This db would contain relative paths (using ${pkgroot}). Once this is done then the whole lot would be relocatable onto a USB drive or whatever. To use this set of packages you would need to specify --package-conf= to ghc, or --package-db= to cabal. > If the extra package paths are absolute, it would involve something > like search&replace on the concrete representation of the supposedly > abstract package database, but as long as that representation is a > simple text file, that might not be too troublesome; Aye, so if you want to be able to move then then it's better if they're relative. > if the extra package paths are relative to a $cabaltopdir, it would > involve telling GHC about the new location prefix whenever calling > it directly (or telling Cabal about its new location, and Cabal passing > that on when calling GHC). So that's the bit in your suggestion that corresponds to using --package-conf= in my suggestion. And it assumes that you don't need to set $cabaltopdir to two values simultaniously, eg if the machine you've moved it to on the USB stick also has cabal packages that it needs to use. > > It's true that the GHC build system cannot work in a directory > > containing spaces, and that's probably too hard to fix. However using > > tools (eg happy, alex) that are in a dir containing spaces should not be > > nearly so hard to fix. > > Maybe so, but last time (end of January) I asked about the GHC build > (in a space-free path) using tools where cabal installs them by default > (with spaces in path), Simon M answered: "It's not practical in general > to cope with spaces in paths in the build system. IIRC we tried to get > this right once and gave up.". So if there is a tool path specific subset > of the problem that could be solved more easily, it doesn't seem to help. Spaces in paths in general is indeed hard. The case where the build tree is in a path with no spaces but some of the external tools are is much easier. Simon was talking about the more general, harder case. > >> Is that even possible on unix systems, with their various packaging and > >> location traditions? > > I'm not sure what you're referring to. > > Some unix branches seem to distinguish themselves merely by different > package management/location. But apart from Mac frameworks, I'm > not aware of any unix that would not expect libraries/binaries/docs to > be installed in different locations (instead of common, language-specific > roots, the common roots are purpose-specific). > > Relocation might not be as typical there, though (don't know how > mounting external drives affects paths there, or whether users might > want to relocate packages between different prefixes like /opt, /etc, > /usr, .., or just from user-local to system-wide locations). It's not especially common after installation. The advantage of a prefix-independent binary install is that it makes installation easy, just a copy. Duncan

1 0

Re: haddock-2.3.0 literate comments discarded from .lhs input
by Duncan Coutts 28 May '09

28 May '09

On Wed, 2009-05-27 at 19:47 +0100, Claus Reinke wrote: > > We need a clear spec on what variables tools are expected to handle and > > how they are to be interpreted. > > Currently, there seem to be $topdir and $httptopdir. And I can't see a justification for there being two. > Given the split between GHC and HP, it might be useful to have an > additional $hptopdir, or just a general mechanism for variables in > ghc-pkg's database (I recall being disappointed when what looked like > environment variables were unaffected by environment settings..). I'd rather not create ad-hoc vars which everyone needs to know about for things like the platform. > > Supporting relocatable sets of packages is a good idea. We should aim to > > have something that is usable by each compiler, not just ghc, so > > interpreting paths relative to ghc's libdir doesn't seem ideal. > > GHC makes no reference to libdir, it simply talks about a $topdir > (where it would like to store things it needs) and $httptopdir (where > haddocks might be found). Yes ok, on windows the topdir is the parent of dir containing ghc.exe. > > How about this: a way to specify paths in the package registration info that > > are relative to the location of the package db they are in. > > ahem. That sounds like a backwards step, being dependent on two > locations instead of one. I don't follow this. Which two? > Before the HP, windows GHCs could be relocated without needing to > update the ghc-pkg database, even if some packages were installed > outside GHCs $topdir. I don't see how this is related to what the Windows installer for the HP is doing. Sure, since it's installing packages relative to ghc and we'd like the whole thing to be relocatable then it should use relative paths. I don't think anyone disputes that, the question is how to implement relative paths. > With your variant, just about any change would need updating. I must be missing something. If you move package.conf and the packages in one go, then nothing needs changing as far as I can see. > Assuming that the parts are independently located by whatever the OS > packaging conventions say, and can be independently relocated > otherwise, it seems simpler to continue with the variable scheme, but > with improved support and documentation for it. My suggestion seems very simple! I'm clearly missing some problem which you can see. To be clear, here's what I'm imagining: blah/package.conf blah/lib/foo-1.0/libfoo-1.0.a and package.conf would contain foo-1.0 with paths looking like "$dbdir/lib/foo-1.0". That is, we interpret $dbdir (or whatever var name we agree on) as being "blah/" because that's the dir containing the db. So crucially, it doesn't really matter where ghc.exe is. Assuming ghc can find the package conf then it can find all the files. So it'd let you create multiple relocatable package collections. If the primary package db is kept relative to ghc (eg in ghc's topdir) then the whole ghc installation including libs is relocatable Duncan

1 1

patch applied (cabal): Resolve merge conflict with dynlibPref patch
by Duncan Coutts 28 May '09

28 May '09

Thu May 28 04:52:49 PDT 2009 Duncan Coutts <duncan(a)haskell.org> * Resolve merge conflict with dynlibPref patch The dynlibPref patch accidentally was only pushed to ghc's branch. M ./Distribution/Simple/Install.hs -2 +6 View patch online: http://darcs.haskell.org/cabal/_darcs/patches/20090528115249-adfee-c2bb6cd7…

1 0

patch applied (cabal): Use dynlibdir = libdir for the moment
by Duncan Coutts 28 May '09

28 May '09

Tue May 19 06:41:15 PDT 2009 Duncan Coutts <duncan(a)well-typed.com> * Use dynlibdir = libdir for the moment It will need more thought about how much control the user needs and what the default shared libs management scheme should be. M! ./Distribution/Simple/Install.hs -2 +3 View patch online: http://darcs.haskell.org/cabal/_darcs/patches/20090519134115-b0bbe-c6684b71…

1 0