Peter Simons writes:

Any thoughts?

Yes: perfect! A lot of the libraries discussion seems to be about how to provide binary packages and how to integrate builds for various compilers etc etc. Give me a way to download the source hierarchy in full or in part, and a way to tell the compiler where to look for it, and I'll be quite happy. I think it makes sense to have the source around -- preferably a CVS checkout -- for all but the most basic and fundamental of functions, that way we can browse, learn, improve and enhance, and submit the modifications easily. (Do builds get so complex that compiling from source is unfeasible?) It seems to me that binary packages should be mostly optional, and can be deferred to the compiler developers to snarf from the "hslibs" (stable) hierarchy and include at their leisure? -kzm -- If I haven't seen further, it is by standing in the footprints of giants

Alastair Reid writes:

So, sure, let's have people build their own systems from cvs checkouts if they want but let's not make this the standard way of doing things. Let's make version numbered releases the standard.

Yes, I absolutely agree, of course. The important part is distributing source, which is simple and will work well in most cases. I imagine the shipped-with-the-compiler binary libraries will be more or less cast in rock, the "stable" hierarchy will be fairly stable, and anything will go for the "unstable" hierarchy -- which is where CVS will be more appropriate. There should probably be test suites to cover dependencies to ensure that all libraries are adequately in sync -- I'm not sure how easy this is to do in practice. BTW, Debian is unusual, in that updates happen all the time. (There are numbered releases, but they aren't commonly used, I think. They manage to make it work pretty well -- better than most others do for third party stuff, IMHO. Perhaps that model can be used? -kzm -- If I haven't seen further, it is by standing in the footprints of giants

Peter Simons writes:

Ketil Z Malde writes:

...

Alastair Reid writes:

...

Let's make version numbered releases the standard.

...

Yes, I absolutely agree, of course. The important part is distributing source, which is simple and will work well in most cases.

I agree as well. Following the CVS repository is for library developers. Regular releases should be provided and, if possible, should come with of the compilers.

Whenever people mention altering or shipping something with the compilers, red flags go up in my brain. I think that one of the reasons for this project is to decouple more stuff from the compiler distributions. In general, I'd like to steer the discussion about distribution toward things that integrate with packaging systems like dpkg & FreeBSD's ports. I would like to see a system flexible enough that: - Authors can distribute libraries & tools from their web pages in a way that makes it very easy for users to download and use them. - Packagers for various OSs have a sane way to create packages. - Compiler distributions can hook into a set of libraries that consensus says most people will want by default (but may not be a part of the Haskell standard). We might call this "hslibs". In debian, ghc5 might depend on hslibs-ghc5, which will in turn depend on a bunch of small packages. And more optionally: - A central repository of useful Haskell libraries can be maintained in a way that blends the best of cathedral & bazaar. How do you feel about that way of putting it? peace, isaac

G'day all. On Thu, May 29, 2003 at 12:12:18AM +0200, Peter Simons wrote:

Again, I refer to boost.org. The project is doing exactly what I proposed, and it has lifted the C++ language to another level. They have practically taken over the language and library development for the new C++ Standard there! So the idea can't be that bad after all.

Boost is a little more structured than what we want, I think. They emphasise good software engineering practices, such as code reviews. For libraries that we want part of the standard, we may want to follow that. For others, we probably won't want to go that far. Cheers, Andrew Bromage

Isaac Jones writes:

Yes, the packages will be left to the distributors. However, I think that the library infrastructure should be able to provide some metadata and tools for assisting packagers [1].

I just realize we really _are_ approaching the problem from different ends! :-) For the sake of avoiding misunderstandings, which I believe we had, let me re-state my approach in other words. I'll address (hopefully) all points you made, even though I don't quote your text: A Haskell module is the most re-usable software component I have ever seen. It is virtually impossible to write something that is _not_ re-usable by someone else. The problem is, though, that many modules are really small, and thus do not warrant the effort of "releasing" them. Plus, not everybody has the infrastructure to do it. So if we could provide a platform where it takes you five minutes to put your source code into an, say, Unstable.YourName directory, hopefully many people would do just that. And then the source code would be _there_. It could be re-used by others -- whereas otherwise, nobody would ever have seen it. Now, if the module is interesting and people are sending you e-mail, asking for features or saying they like it, you'll be writing a short manual and clean-up the API before you know it. So in effect, the Haskell community would have one more good library to re-use. And if the library is nice, we can "publish" it in the Stable area and users of our distribution would just have it on their hard disk. This is already an incredibly useful service, even if it doesn't do anything but distribute the sources -- no build system and nothing. BUT, once you have two or three libraries in this repository, you could go ahead and create a build system for them. The more code comes flowing in, the more complex will your build system get, until some day, when it can - build all the Stable libraries on virtually any platform and with virtually any compiler, - run automated regression tests and produce result reports, - generate all the library documentation (maybe based on Haddock) and install them, either as HTML, PDF, or man pages(!), - generate a sophisticated web site, on which all the contents of the archive can be browsed, searched, etc., - create several different binary package formats and provide means for users to run the regression tests on them. Assuming this system exists, then everybody can check his code into the Unstable area and provide build information for the it. (And this has to be _really_ simple.) So everybody else can just check-out parts of the Unstable branch and integrate them seamlessly into the rest, albeit in the Unstable namespace. Or programmers can chose to re-use the build system for their own projects, if they want to release their package themselves.

Isn't some kind of common build infrastructure [...] necessary so that [different compilers] don't have to repackage it?

I doubt there is such a thing as a common build infrastructure on which everybody could agree. Just look at the reality today: We have BSD Make, GNU Make, Jam, Boost.Jam, Odin, Cook, Automake and only god knows what else -- all being actively used for all kinds of projects. Even within the Haskell community, not everybody is using the fptools, even though it clearly can build pretty much everything. In my opinion, the reason for this lies in the nature of the task, not in the lack of a better system. Given this, I don't have a problem with the fact that people will use a different build infrastructure than the one I designed.

...

Again, I refer to boost.org.

[...] I'm rather convinced that duplicating any one system will leave us with 1) the flaws of that system and 2) the flaws that Haskell adds to the system by not being a perfect fit.

Uh, I was referring to Boost's model of running the source repository and their success with it -- not to their build infrastructure. Which, by the way, is entirely inappropriate for Haskell, unless we're interested in several hundred kilobytes worth of C++-compiler properties. :-) Peter

Isaac Jones writes:

This is the beauty of a layered tools approach like the one I describe. [...] there's a lot of work to be done in both on the distribution end and on the "building" end.

I guess I have to apologize. I didn't realize you intend to separate the two stages so clearly, until now. :-( In that case, forget everything I even argued about. I'm completely with you.

Do you mind helping to convert the build system for the libraries you put together once the "building" side is ready?

Not at all ... Like you suggested, my next step will be to come up with a description of "my" project's goal, a repository layout, and a prototype build system. This document can then be discussed, revised, thrown away, and be re-written. :-)

- Meta information that the libraries should provide (you probably don't have to or want to worry about this yet) will be important to methods of download / depends / install

I am very fond XML for this purpose, because it is so re-usable (and can be verified for correctness). If we'd store this meta information in a simple document, for which we provide a DTD, then I don't see a reason why this information couldn't be re-used by a build infrastructure wherever possible. In this case, there would be no overlap between the two efforts at all. Of course the same can be achieved with any format, admittedly, but for XML there might be really good editors available, like psgml for Emacs, etc.

- If not a standard make system, at least a standard method for invoking the make system ala distutils and Debian. You probably don't want to worry about this yet either, but keep it in mind.

In fact, I favor an Autoconf- and Make-based system over all alternatives, just because everybody knows how to use those tools. That's a huge advantage for the users and for the developers as well. If there is a "better" solution, I'd like it to be able to generate Makefiles and configure scripts, as a fallback. :-)

- collect the libraries that are already included with various compilers and see what the differences are both in what libraries are included and behaviors of the libraries

- find orphaned libraries (ala Numeric Quest) and try to find the authors to give them licenses.

If anybody is aware of candidate libraries, please let me know -- preferably including an URL. :-) Also, it would help immensely if everybody would look through his source codes and just make useful modules available for release. Then we'd have more original material, too. I'm sure that licensing will not be a show-stopper issue. Peter P. S.: Sorry for the long delay in answering your e-mail, Isaac. For various reasons, I had to re-install my work station from the scratch lately, and it will still take a while until I'm fully operational again.

Alastair Reid writes:

...

There should probably be test suites to cover dependencies to ensure that all libraries are adequately in sync -- I'm not sure how easy this is to do in practice.

It's best done as the library is being written because it could actually help development and because the author likely has a better idea of what a function is supposed to do.

(I was thinking of cross-module dependencies, quite possibly involving multiple authors)

But test suites are usually ignored or built long after the fact.

Like documentation? I suppose this is a programmers' mindset thing.

Since quickcheck is quite easy to use, I wonder what's missing? Is it infrastructure (a handy driver program and some make targets) or some well-worked examples to cut and paste from? Where do I find examples of how to quickly add quickcheck to my library?

Examples to cut and paste from, would be my vote. Which is why I like to stress installing libraries as source.

Debian does use numbered releases though.

Of individual components, but not of the whole repository as such. Anyway, I'm not sure dependencies will be so crucial for Haskell source, most incompatibilities should be caught at compile time by type checking, a luxury Linux distributions don't have to the same extent. Some dependencies on specific behaviour is caused by the need to work around bugs, this could be alleviated by having source and an obvious path for updates -- i.e. instead of working around a bug, I can fix it, and send the fix upstream. -kzm -- If I haven't seen further, it is by standing in the footprints of giants

Isaac Jones writes:

* Standard library search paths might be necessary, and standard flags would be useful. hmake could go some of the way (maybe it already does?) in hiding different compiler behavior where possible.

Some of the issues I've found in adapting the fptools infrastructure are: 1) It's possible to install packages in global filespace (e.g., /usr/lib/greencard) or in local filespace (e.g., $HOME/lib/greencard). If you're putting it in global filespace, it is appropriate to register it in the global package list (e.g., /usr/lib/ghc-5.04.2/package.conf) but where do you register it if installing it locally? The fix seems to be to have ghc-pkg default to recording packages in the local fielspace (e.g., $HOME/lib/ghc-5.04.2/package.conf). The same will go for any other compiler and for anything that maintains a list of installed packages. There has to be a standard place in local filespace ($HOME/...) for the list to be maintained. An easy way of doing this seems to be to copy and extend the ghc flag --print-libdir which tells you where the global copy is stored. That is, we should add an additional flag to tell you where the local copy is supposed to be. 2) When compiling and installing libraries with hugs, ghc, nhc, it's useful to store it in a directory which includes both the compiler name and the compiler version. For example: /usr/lib/greencard/$compiler.$version This makes it easy to have multiple versions of your compiler installed. It's also useful to be able to find the HsFFI.h file required by the ffi extension. For this, it would be nice if all major tools provided a way to extract things like version number in an easily parsed way. For example, it'd be nice if compilers could act like this: $ hc --print-install-info compiler = GHC major-version = 5 minor-version = 04 patch-level = 2 include-dir = /usr/lib/ghc-5.04.2/include global-libdir = /usr/lib/ghc-5.04.2 local-libdir = $HOME/lib ... Note that I want the tool itself to print this information not to have a separate program which prints the info. This seems to work better if I ask for the library to be compiled and built with a particular compiler (e.g., $HOME/cvs-dirs/nhc98) because there's no risk that I'll run the wrong program to get the install info. 3) When installing libraries, it initially seems fine to install the greencard library in /usr/lib/greencard. But then we move onto the X11 library and try to install into /usr/lib/X11. Hmmmm, that doesn't work so well. It'd be good to have some standard suggestions for people to use. Some which spring to mind are: /usr/lib/HS$package /usr/lib/haskell-libraries/$package /usr/lib/haskell-libraries/$compiler.$version/$package /usr/lib/$compiler.$version/$package Note that with the last, we're using the directory that the compiler itself uses so there's a possibility of conflicts.

* The make system should be wrapped with a python-style distutils system which provides a ./setup.hs with standard targets so that every library gets built and installed the same way

I think it is a great idea to have this extra level of indirection. It means that people can use makefiles, configure scripts, etc. suited to their application, preferences, choice of license, etc. and still play with everyone else. Obviously this needs a different version of setup.hs for each infrastructure but this hopefulyl isn't too hard. I think it's a mistake to make it a Haskell script though because that's bound to lead to bootstrapping issues. I'd use a shell script instead. -- Alastair Reid

Alastair Reid wrote:

...

I think it's a mistake to make it a Haskell script though because that's bound to lead to bootstrapping issues. I'd use a shell script instead.

On Wednesday 28 May 2003 10:38 pm, David Roundy wrote:

The shell script would lead to issues on windows, though. [...]

Remember, we're talking about a script for _building_ packages not for installing them. For windows, people will use .msi packages to install. People who want to build packages generally need to install a bunch of other things too. Obviously, GHC users will have installed cygwin (whcih includes shells) but even Hugs users need to install a C compiler if they're using a package that uses the foreign function interface (and quite a lot of libraries involve the ffi). Also, my experience from Hugs is that people do not want to build packages from source and, though I think binary packages are a flawed concept, I choose to use binary packages for my Linux system even though I could theoretically build my entire system from source. -- Alastair Reid

On Wed, 28 May 2003 17:04:07 +0100 Alastair Reid wrote:

...

Simon Marlow and I would like to propose Isaac Jones for this coordination role.

I've exchanged a bunch of email with him about how libraries ought to work and I think Isaac would be great as coordinator.

-- Alastair Reid

He seems enthusiastic, capable and takes the initiative. A smaller example being some Control.Monad.State documentation (http://www.syntaxpolice.org/~ijones/tmp/Control.Monad.State.html), and prodding me (implicitly/explicitly) to write up what I could for other Control.Monad libraries (http://haskell.org/hawiki/MonadTemplateLibrary) so that there is at least basic documentation. So consider this my 'Me too' post to Alastair's.