- Libraries - Haskell.org

Ideas for library infrastructure
by Andres Loeh 29 Apr '03

29 Apr '03

This is an attempt to summarize and separate the problems, in the way I see them: There are two things that are too complicated right now: [1] A library writer needs clear instructions what he can do with his library to package it in a standardized way usable with as many Haskell systems as possible. [2] A library user needs a normalized way to install additional libraries into his Haskell system(s) and keep them up-to-date. The following factors make both problems hard to solve: [A] There are multiple Haskell "compilers" out there, with very different properties. They support a different number of extensions, have different command line flags, one isn't even a compiler ;-) Furthemore, they have their own package systems which are incompatible. [B] Different versions of the same compiler are binary-incompatible. So on a compiler upgrade, the user will either need to reinstall libraries completely or at least do something to recompile the libraries. The current approach, as mentioned before, is to make the set of standard libraries increasingly large. I agree that this is not a really good solution, because compile times increase, the users may get huge sets of libraries they don't really need, and they may get them multiple times. A project that works that way, for example, is KDE. It is quite hard to upgrade single packages without upgrading the entire Desktop environment plus libraries. There, I don't find that situation ideal, and still, KDE is at least centrally organized and has global releases. The Haskell compilers are all separate and should stay that way. The situation as it is almost completely rules out binary-only versions of libraries. I will thus try to focus on source based libraries from now on. I vote for a two-level approach: I. Define a clear and precise way how and where a Haskell library has to be installed on a system. This installation should be as compiler-independent as possible. The files should go in a central place, and there should be additional information at that location of the kind that is now for example stored in GHC's package configuration file. As far as a library does not depend on a particular Haskell compiler (for example, it needs C libraries as well), these C libraries may well aready be built and installed. But the Haskell code should be installed in source form. Authors of libraries should find these guidlines in a clear and easy to understand form somewhere on the web, and there should be tool support to facilitate conforming to the standard. I'd go for some well-documented Makefile includes, but if nicely written, a Haskell based system may also work. But I think, the system and language here is secondary, as long as it is clear _what_ needs to be done ... II. Each Haskell compiler should be extended with a tool that can take a library from the standardized source based install described in I. This tool could be similar in interface to ghc-pkg, but should not only collect package configuration information, but also _build_ the Haskell code for the compiler and create and install libraries to the right positions as necessary. The tool should provide a way to _add_, to _remove_, and to _completely regenerate_ libraries. Additionaly, there needs to be one single tool, that levels the potential differences between the package tools of the Haskell compilers and keeps track of all the compilers installed in a system. Let's call it "global package manager". The last thing that would be called in the Makefile (or whatever) that is supplied with a library is the global package manager with the option "add" and the library name. The global package manager would look in the standardized central place for the library, find out for which of the installed Haskell compilers it works, and call their respective package managers that will the build and add the library individually. The last thing that would be called in the Makefile (or whatever) of a Haskell compiler would be its own package manager with the option to regenerate all libraries. Because they are installed in a place independent of the compiler itself, they will survive a compiler upgrade and can automatically be rebuilt for the new version. For hugs rebuilding would be nearly trivial, for GHC is might take a long time, but that's life. Summary of the proposal: * Define a clear standard to install Haskell libraries on a system in a way that is independent of a specific Haskell compiler (both type and version). * Support library writers by tools to make sure they can conform easily to that standard. This includes a global package manager that can be called in the end to pass responsibility over the library to the compilers installed on the system. * For each Haskell compiler, write a/modify the package manager so that it can handle the addition, removal, and complete regeneration of such source-based installed libraries. [1] This will make the situation easier for package writers, because they do not have to care about actually _building_ the library for a specific compiler or compiler version. They will only have to conform to a specific installation format, and that will be supported by tools. [2] This makes life easier for the users, because they can just install additional libraries and "never" have to worry about them again. If they install additional Haskell compilers or upgrade to newer compiler versions, the libraries will always automatically become available. Of course I realize that I probably have forgotten many things and that I may not have made myself clear at all points. So you're all invited to tear this proposal apart! Best, Andres -- Andres Loeh, Universiteit Utrecht mailto:andres@cs.uu.nl mailto:mail@andres-loeh.de http://www.andres-loeh.de

3 2

RE: bug in hGetPS?
by Simon Marlow 28 Apr '03

28 Apr '03

> I don't see that hGetPS should be consistent with hGetLine. > If hGetLine > returns and empty string, you assume the file has two > newlines in a row, > while if hGetPS returns an empty string, since you asked for > a given number > of chars, you know that you've reached the end of the file. I mean that hGetPS is inconsistent with hGetLine in the sense that hGetPS never throws the EOF exception. The consistent interpretation would be for hGetPS to throw an EOF exception if the Handle is positioned at the end of the file, unless the requested byte count is zero. It wouldn't make sense to have hGetLine behave in the same way as hGetPS does now, because as you say there would be two possible meanings for an empty string returned. Cheers, Simon

1 0

RE: bug in hGetPS?
by Simon Marlow 28 Apr '03

28 Apr '03

> I'm not sure if this counts as a bug or a feature... > > If hGetPS h len encounters an EOF before len, it returns a > PackedString > containing the remainder of the file. However, if h is > already at the end > of the file, an exception is raised rather than returning an > empty string. > To me this is rather counter-intuitive. It matches the behaviour of System.IO.hGetLine. Data.PackedString.hGetPS behaves in the way you're suggesting: it just returns short reads instead of flagging EOF. Unfortuantely this is inconsistent with hGetLine, so I'm not sure if the behaviour should stay that way. Cheers, Simon

2 1

possible bug in Eq of packedStrings
by David Roundy 28 Apr '03

28 Apr '03

I think I've come across a bug in the (==) function of PackedString, but it's hard to reproduce. I'm attaching a set of files that hopefully will reproduce the problem. My assumption is that if two packed strings are unequal, then when their unpacked versions must also be equal. This is, of course, assuming one starts with the packed versions, since packing of strings may be lossy... Anyhow, the guts of the problem is in the following (where c1 and c2 are packed strings): if c1 == c2 then return True else do putStr $ "First version:\n" putStr $ unpackPS c1 ++ "\n" putStr $ "second version:\n" putStr $ unpackPS c2 ++ "\n" if unpackPS c1 == unpackPS c2 then putStr "But unpacked, they are the same!!!\n" else putStr "And unpacked they are different!\n" return False I think that it is a bug if the line "But unpacked, they are the same!!!" ever gets printed (which it does--but it's hard to reproduce). This bug is rather troubling to me, since one of the reasons I'm using packed strings is for the assumed efficiency of comparison, the other being of course storage space, which would be sufficient in itself to mandate packed strings. Anyhow, I'm attaching a test program with a test file (which needs to be named both test1 and test2) which is able to reproduce an error in the above. Sorry I haven't made it any simpler. -- David Roundy http://www.abridgegame.org

2 2

RE: Ideas for library infrastructure
by Simon Marlow 28 Apr '03

28 Apr '03

Ralf Hinze writes: > I like Andres' proposal. It puts some burden on the > compiler writers (but this work has to be invested > only once) and considerably simplifies things for > the library writers. Putting the burden on the compiler writers is exactly what we want to avoid - us compiler writers tend to be too busy writing compilers! Instead, what I would like to see is a separate library infrastructure built by non-compiler writers that smooths over the differences between

1 0

bug in hGetPS?
by David Roundy 26 Apr '03

26 Apr '03

I'm not sure if this counts as a bug or a feature... If hGetPS h len encounters an EOF before len, it returns a PackedString containing the remainder of the file. However, if h is already at the end of the file, an exception is raised rather than returning an empty string. To me this is rather counter-intuitive. Since hGetPS already has one way of flagging the EOF condition (returning a string shorter than len), I would prefer to be able (for example) to simply copy a file by a loop such as: cp h1 h2 = do ps <- hGetPS h1 buflen hPutPS h2 ps if lengthPS ps == buflen then cp h1 h2 else return () I know I could just as easily write the same code as: cp h1 h2 = do ps <- hGetPS h1 buflen hPutPS h2 ps eof <- hIsEOF h1 if not eof then cp h1 h2 else return () but since the first should be as fast as the second, I don't see why it shouldn't be made safe. -- David Roundy http://www.abridgegame.org

1 0

FW: First Attempt at Crypto Library
by Simon Marlow 25 Apr '03

25 Apr '03

Folks, Domenic Steinitz has contributed some crypto code (see attachment and msg below). According to our draft hierarchy, there is a place in the hierarchy reserved for crypto libraries: FileFormat.Encryption (see http://www.haskell.org/~simonmar/lib-hierarchy.html) Question 1: there's a comment next to 'FileFormat' which mentions that 'Codec' might be a more accurate name. It seems to me that 'Codec' would indeed be a better choice: many of the libraries under FileFormat are more like pure stream-transformers than file formats, and Codec encompasses both. What do people think about changing this? (there aren't any other implementations of libraries under FileFormat that I'm aware of). If FileFormat became Codec, then the existing FileFormat.Encoding looks a bit odd. But moving FileFormat.Encoding.Base64 up to Codec.Base64 (similarly for FileFormat.Encoding.Yenc) would seem to make sense. Question 2: what should the insides of the FileFormat.Encryption (or Codec.Encryption) hierarchy look like? Cheers, Simon -----Original Message----- From: Dominic Steinitz [mailto:dominic.steinitz@blueyonder.co.uk] Sent: 21 April 2003 18:47 To: Simon Peyton-Jones; Simon Marlow Cc: libraries-request(a)haskell.org Simon, Simon, Here's my first attempt at a crypto library. It compiles and I can run a test using ghc-inplace. The test checks with the example in http://www.itl.nist.gov/fipspubs/fip81.htm (except I couldn't find an example with PKCS#5 padding). I'm not sure what the next steps are. Dominic Steinitz

8 13

RE: FW: First Attempt at Crypto Library
by Simon Marlow 25 Apr '03

25 Apr '03

> > I initially thought along these lines, but then concluded > that it's not > > that simple (or at least, that's not the only interpretation of the > > classification). > > Is a deep hierarchy really necessary? Well, it seems useful to group codecs by purpose: Audio, Video, Compression, etc. so I believe we're not into diminishing returns at this level. Matt Harden said: > I would agree with Codec.Binary.Base64. Codec.Text should be > reserved for codecs that deal with text data; i.e. translating > text between various encodings like unicode, ISO 8859-1, ascii, > ebcdic, etc. That's a good point. Unless anyone objects, let's reserve Codec.Text for the text encodings (UTF-8 and so on). The options for Base64 and friends are therefore: Codec.Binary.Base64, Codec.Base64, or Codec.General.Base64. Or even Codec.BinaryToText.Base64? None stands out as the obvious choice to me, but we could defer the decision until the library actually exists ;-) Cheers, Simon

8 10

RE: FW: Library infrastructure
by Simon Marlow 24 Apr '03

24 Apr '03

Some people have mentioned other systems that might be relevant: the BSD ports tree, CPAN, etc. I'd just like to clarify what (IMHO) we're not aiming for here: in particular, we don't need a system that - downloads source - manages dependencies (beyond what the compiler needs to know) And I'd argue that we don't even need a system that builds binary distributions. The reason is that these tasks are all handled by the various OS package systems: BSD ports, RPM, apt-get, emerge, and so on. We don't want to duplicate or worse, bypass, the local package management system. We do need a system that helps you generate compliant package skeletons in the local native format: an RPM spec, a BSD ports Makefile, or a Gentoo ebuild. Cheers, Simon

7 11

FW: Library infrastructure
by Dominic Steinitz 23 Apr '03

23 Apr '03

I'm not clear what happens at the moment. I had naively assumed that having created my library that it would be put into the ghc cvs repository and would then be available in the same way other libraries are when you install ghc. Do libraries get put into the ghc cvs repository? If they do, do they get put into the ghc release? If they don't then how does someone who wants to use one get hold of it? My thinking with the crypto library was that there were quite a few crypto functions available but they were spread out all over the place and had different interfaces and wouldn't it be useful if there were one library with a uniform interface. I'd be happy to help in building a library infrastructure but I think we should do the minimum possible to deliver the absolutely essential elements only. How do we capture the requirements? Do we have anything at all that does some of the job that we can build on? >From my reading of this thread, I thought Malcolm Wallace had something. Dominic Steinitz

2 1