
Dear library folk Dominic has been writing a crypto library: | I'm not clear what happens at the moment. I had naively assumed that having | created my library that it would be put into the ghc cvs repository and | would then be available in the same way other libraries are when you install | ghc. | | Do libraries get put into the ghc cvs repository? | If they do, do they get put into the ghc release? | If they don't then how does someone who wants to use one get hold of it? George wants Koen's Chunk library: | > Take a look at: | > | > http://www.cs.chalmers.se/Cs/Grundutb/Kurser/afp/Lab2/Chunk.hs | > | > Is that what you want? | Yes, that's much more like it. Any chance of this module getting put into | the standard distribution? These both raise a common question: how can someone distribute a library? (This is a different question to the library-infrastructure question about how to build a library, which we've been discussing recently.) What Dominic and George would like is for their desired library to just "come with GHC" (and Hugs, NHC). But this model is unsustainable for us (the GHC team), and I expect it is unsustainable for Hugs and NHC. It's just too much work to be responsible for all the libraries that anyone might contribute. Adding just one more module isn't a problem, but iterating that process certainly is. Simon and I are beginning to sink --- partly a welcome side effect of GHC's widespread use --- so we need to find ways to get ourselves off the critical path for libraries, to avoid impeding their development. (As I remarked in an earlier message, I think that ways to make libraries flourish is the top challenge facing the Haskell community.) The main alternative to "packaged by the GHC/Hugs/NHC team" route is this: the person writing the library makes it into a separate GHC package (and something similar for Hugs, NHC), and distributes it separately This is the model followed by Sven's GLUT package, for example. GLUT is physically kept in the GHC/Hugs CVS library repository, but Sven maintains and distributes it. He can fix bugs, and distribute new versions whenever he likes, independent of GHC releases. This is quite a high-cost model, though. It's a fair amount of work to build binary distributions of libraries for different platforms and for different Haskell implementations. For example, I can't see Koen doing that just for his one-module Chunk library; and Dominic might find it a bit daunting for his crypto libraries too. What we want, I think, is an aggregation mechanism. In particular there could be one or more "contrib" packages, optionally hosted at fptools/libraries, containing code contributed and maintained by different folk, but packaged and distributed by one person, a Contrib Builder (better titles welcome) Note that a) A Contrib Builder may, but does not have to, use the library infrastructure we are discussing in another thread. But s/he will need to provide some infrastructure so that people contributing libraries can build and test their library. The Contrib Builder will probably want to do automated nightly build/test runs of the package. b) A given package (= group of library modules, distributed as a single bundle) can have a Contrib Builder for GHC, another for Hugs and another for NHC. That's the way the core libraries work right now: there's a single shared code base in fptools/libraries, and the GHC, Hugs and NHC teams each build distributions from that common code base. When a new version of GHC/Hugs/NHC comes out, the appropriate Conrib Builder(s) should make a new version of their package for the new version of the compiler, if necessary. But of course, the package can be released more frequently than that. c) There can be more than one contrib package. A package is just a unit of distribution, to amortise the cost of building distributions for multiple modules. Ideally, a package would contain thematically related modules, but not necessarily. d) Every module in a package will occupy a slot in the common hierarchical module name space; so library authors should negotiate for their name, using the libraries mailing list, as they do now. OK so that's the idea. 1. Does it seem like a reasonable idea? 2. There is a pressing need for the first Contrib Builder for libraries like Dominic's crypto and Koen's Chunk library. Any volunteers? Even for one implementation target? Simon

Simon PJ writes:
What we want, I think, is an aggregation mechanism. In particular
there could be one or more "contrib" packages, optionally hosted at fptools/libraries, containing code contributed and maintained by different folk, but packaged and distributed by one person, a Contrib Builder (better titles welcome)
Actually, I think we need multiple contrib builders just as there are for GHC at the moment. Specifically: - we need someone who knows about and cares about FreeBSD to produce freebsd ports of libraries - we need someone who knows about and cares about Windows to produce .msi packages - we need someone who knows about and cares about Debian to produce debian packages and so on for each format we want releases in. It's unlikely that one person will have experience of all the different systems we want releases for so it has to be multiple people. For any one format, it may be easier to have one person do all packages for all compilers and all libraries or it may be easier to split the task by compiler, by software license, by repository, or whatever. (Obviously, using a common infrastructure makes it easier for one person to handle more libraries and more compilers.) Simon also says:
The Contrib Builder will probably want to do automated nightly build/test runs of the package.
I think this suggestion blurs the distinction between library writers and contrib builders. It is a library writer's responsibility to deal with bugs - they would probably benefit hugely from this sort of infrastructure. Contrib builders should only be dealing with bugs they have introduced in a particular package or, arguably, with portability issues. The way I see it, there should be a very clearly defined interface between library writers and contrib builders. Library writers should provide version numbered releases (absolutely _not_ daily CVS snapshots) together with certain metadata (documentation, which version number this is, which particular versions of other libraries the library works with, etc.). Contrib builders apply some patches to customize it for a particular system (debian, freebsd, etc.), compiles it, runs the testsuite, builds the documentation and packages it up. Contrib builders might also act as a conduit for bug reports. -- Alastair Reid ps The above assumes that we want binary packages (I think there's plenty of evidence (Redhat, Debian, etc.) that this is what people want to use). It assumes that we want to work with the 'native' package mechanism on a system rather than providing our own parallel mechanism. It assumes that authors aren't always able to update their library whenever there is a significant change in one of the things they depend on: the nightly tests fail every night for a month before they get dealt with. Finally, it assumes that most people want official releases of code that the author feels are in a usable state rather than random snaphots of an evolving system.

* Alastair Reid
- we need someone who knows about and cares about FreeBSD to produce freebsd ports of libraries
That's what I am already doing: http://haskell.org/freebsd/
Regards,
Olli
--
Oliver Braun ::

Alastair Reid
- we need someone who knows about and cares about Debian to produce debian packages
I wouldn't mind being the Debian person for this task, and there are several others who are probably willing and qualified, but I have been hesitant to package Haskell libraries for Debian until some of the issues raised in the Library Infrastructure thread are resolved. (This is probably not a good attitude on my part, but at least I'm interested in helping to resolve them.) As a reminder, there seme to be two ways to package Haskell libraries on Debian, 1) have a large number of packages (hunit-ghc4, hunit-ghc5, hunit-ghc4-prof, hunit-ghc5-prof, hunit-hugs, hunit-nhc, etc) or 2) build libraries at install time a la elisp. In particular, I can immagine a Haskell Library Infrastructure which makes it easy to automatically generate debian binary packages for all of the compilers that are installed on the build machine. Not to bring it up again, but a distutils type system like Python's (with some modification) seems to solve both the "Library Infrastructure" problem, and the "Contrib packages" problem. Also, it should be noted that there's a place in sorceforge for Haskell Libraries that might be useful in the interim. You can read more about all of this on the wiki: http://www.haskell.org/hawiki/LibrariesAndTools peace, isaac

On Tue, May 06, 2003 at 03:01:44PM +0100, Alastair Reid wrote:
Contrib builders apply some patches to customize it for a particular system (debian, freebsd, etc.), compiles it, runs the testsuite, builds the documentation and packages it up. Contrib builders might also act as a conduit for bug reports.
Perhaps unsurprisingly this is a pretty good description of a Debian maintainer. For Debian, I think a nice solution might be to have a package of the library building software which includes a way for implementation and library packages to register with it. Then, when a new library is installed it would be rebuilt for any implementations installed and registered, and when GHC is upgraded it would rebuild all the libraries installed and registered. This nicely solves the problem of changing binary formats. The only subtlety I can see is making sure things get built in the right order when installing multiple packages at once. To do this I think we would want to be able to tell the library building system to "compile" the library foo in /source/path (which would live in /source/path/foo) and to put the result in /dest/path (would go in /dest/path/foo) for ghc5|nhc98|hugs. It would then automatically also compile bar if foo depends on bar (via library meta-information). It should be easy to generate Debian dependencies from this library meta-information. As I've described it the source would stay around, so "compiling" for hugs could just be making a symlink. For other ways of doing it they may need to be copied. This could be an option in the package building system. Thanks Ian

On Wednesday 07 May 2003 10:00 am, Ian Lynagh wrote:
For Debian, I think a nice solution might be to have a package of the library building software which includes a way for implementation and library packages to register with it. Then, when a new library is installed it would be rebuilt for any implementations installed and registered, and when GHC is upgraded it would rebuild all the libraries installed and registered. This nicely solves the problem of changing binary formats. The only subtlety I can see is making sure things get built in the right order when installing multiple packages at once.
This sounds a lot like the Common Lisp compiler system. I don't know much about CL, but I do know that when I had SBCL installed for a brief while, if I updated a CL library, it would recompile the library for SBCL. This is obviously a bit different than Haskell, since SBCL uses an Image file that must be updated periodically to use the new functionality, but perhaps we could make a similar mechanism for Haskell. -Brent

* "Brent A. Fulgham"
On Wednesday 07 May 2003 10:00 am, Ian Lynagh wrote:
Then, when a new library is installed it would be rebuilt for any implementations installed and registered, and when GHC is upgraded it would rebuild all the libraries installed and registered. This nicely solves the problem of changing binary formats. The only subtlety I can see is making sure things get built in the right order when installing multiple packages at once.
This sounds a lot like the Common Lisp compiler system.
This sounds a lot like what I proposed aeons ago on one of the ghc mailing lists - I think. It's long ago and I don't have a reference handy now, sorry. It got shot down for some technical reasons, or complexity or so, IIRC. SimonM might remember. BTW: Brent, Ian - maybe something along the lines of what Debian does with kernel module packages? Cheers, Michael, in a hurry. :( -- /~\ ASCII ribbon | "You could wire up a dead rat to a DIMM socket and the PC \ / campaign | BIOS memory test would pass it just fine." X against | -- Ethan Benson / \ HTML mail |

On Thu, May 08, 2003 at 08:36:01AM +0200, Michael Weber wrote:
* "Brent A. Fulgham"
[2003-05-07T22:54-0700]: On Wednesday 07 May 2003 10:00 am, Ian Lynagh wrote:
Then, when a new library is installed it would be rebuilt for any implementations installed and registered, and when GHC is upgraded it would rebuild all the libraries installed and registered. This nicely solves the problem of changing binary formats. The only subtlety I can see is making sure things get built in the right order when installing multiple packages at once.
This sounds a lot like the Common Lisp compiler system.
This sounds a lot like what I proposed aeons ago on one of the ghc mailing lists - I think. It's long ago and I don't have a reference handy now, sorry. It got shot down for some technical reasons, or complexity or so, IIRC. SimonM might remember.
It is slightly more complex than other approaches, but I think it is significantly nicer from the sysadmin and user's point of view.
BTW: Brent, Ian - maybe something along the lines of what Debian does with kernel module packages?
For the the benefit of those not in the know, this would mean installing a library package would put the source in /source/path/mylibrary. The sysadmin would then have to come along and do "make-library ghc mylibrary" which would create mylibrary.deb which he would then install. When he upgraded GHC he would have to repeat the process (as any library debs would have to depend on the exact version of GHC for binary compatibility reasons) (although probably a single command to build all the debs) and similarly when he installed nhc98. This makes a lot more work for the sysadmin. I much prefer what I suggested before. Oh, one thing that would be nice is to have conflicts on implementation versions, so you can say "Conflicts: ghc < 5.04, hugs". Thanks Ian

tor 2003-05-08 klockan 14.27 skrev Ian Lynagh:
On Thu, May 08, 2003 at 08:36:01AM +0200, Michael Weber wrote:
* "Brent A. Fulgham"
[2003-05-07T22:54-0700]: On Wednesday 07 May 2003 10:00 am, Ian Lynagh wrote:
Then, when a new library is installed it would be rebuilt for any implementations installed and registered, and when GHC is upgraded it would rebuild all the libraries installed and registered. This nicely solves the problem of changing binary formats. The only subtlety I can see is making sure things get built in the right order when installing multiple packages at once.
This sounds a lot like the Common Lisp compiler system.
This sounds a lot like what I proposed aeons ago on one of the ghc mailing lists - I think. It's long ago and I don't have a reference handy now, sorry. It got shot down for some technical reasons, or complexity or so, IIRC. SimonM might remember.
It is slightly more complex than other approaches, but I think it is significantly nicer from the sysadmin and user's point of view.
On the other hand, when something breaks, they break quite badly. Using exact-version dependencies on ghc means that some packages will hold others back, but at least *things won't break*. One big problem with recompile-on-the-fly is that you need all the build dependencies installed, which are usually a lot different from the binary dependencies. The whole idea with binary distribution is that you don't need a build environment (granted, you normally have a haskell compiler installed if you're using haskell libraries) and build dependencies can be very very extensive. /Martin -- Martin Sjögren sjogren@debian.org -- marvin@dum.chalmers.se GPG key: http://www.mdstud.chalmers.se/~md9ms/gpg.html let hello = "hello" : hello in putStr (unlines hello)

"Simon Peyton-Jones"
These both raise a common question: how can someone distribute a library? (This is a different question to the library-infrastructure question about how to build a library, which we've been discussing recently.) [..] OK so that's the idea.
1. Does it seem like a reasonable idea?
2. There is a pressing need for the first Contrib Builder for libraries like Dominic's crypto and Koen's Chunk library. Any volunteers? Even for one implementation target?
Languages like Perl and Python face the similar problems as we do, although they have less language implementations to worry about (at least for Python, there is more than one implementation). So, the question is whether we can't more or less follow their lead. Perl is certainly extremely successful at establishing itself by way of a flourishing libraries community. I don't know many details of how the Perl community works, but here are what seem to be core points: * Well-known central repository: CPAN http://www.cpan.org/ * Binary distributions of library packages can be obtained independent of the main system. CPAN has something called Perl Ports. On my RedHat 8.0 system, grepping for Perl packages leads to the appended result. To facilitate this, it would probably be worthwhile to have some infrastructure that makes it easy for a library author to make a new library portable across operating systems and Haskell implementations (and their associated build and package systems). If this were really simple, we might not need to agglomerate unrelated libraries into random contrib packages. I don't really have much of a concrete idea of how such an infrastructure should look like, but maybe somebody else has. Cheers, Manuel -=- AttitudeAdjuster chak 608 (~): rpm -qa |grep perl perl-Storable-1.0.14-15 perl-DateManip-5.40-27 perl-XML-Dumper-0.4-22 perl-Inline-0.43-7 perl-5.8.0-55 perl-SGMLSpm-1.03ii-6 perl-libwww-perl-5.65-2 groff-perl-1.18-6 perl-suidperl-5.8.0-55 openssl-perl-0.9.6b-33 perl-Filter-1.28-9 perl-HTML-Parser-3.26-14 perl-XML-Parser-2.31-12 perl-libxml-enno-1.02-25 gimp-perl-1.2.3-9 perl-Parse-Yapp-1.05-26 perl-libxml-perl-0.07-25 perl-XML-Grove-0.46alpha-21 perl-PDL-2.3.2-10 perl-URI-1.21-3 perl-XML-Twig-3.05-3 perl-CGI-2.81-55 perl-CPAN-1.61-55 perl-HTML-Tagset-3.03-25 perl-XML-Encoding-1.01-20 perl-DB_File-1.804-55
participants (9)
-
Alastair Reid
-
Brent A. Fulgham
-
Ian Lynagh
-
Isaac Jones
-
Manuel M T Chakravarty
-
Martin Sjögren
-
Michael Weber
-
Oliver Braun
-
Simon Peyton-Jones