April 2003 - Libraries

Crytpo for nhc
by Dominic Steinitz 18 May '03

18 May '03

I have been trying to write a Crypto library and so far it works with ghc and hugs. Am I right in thinking I am going to struggle with nhc? It uses Word8 and Word64 heavily. For example, cipher block chaining relies on xoring Word64. I attach the library and a test program. I'd be grateful for any suggestions on how to make it work with nhc. Dominic Steinitz

2 2

RE: X11 libraries
by Simon Marlow 15 May '03

15 May '03

> > According to the current hierarchy document[1] we have these > > libraries under Graphics.X11.Xlib. Perhaps these names would be > > correct: > > > Graphics.X11.Xlib > > Graphics.X11.Xlib.Types > > Graphics.X11.X1ib.StdDIS > > > The reason for separating Xlib into its own hierarchy is that there > > are other APIs under X11 (Xt, Xaw, Xmu). > > That makes sense except that I didn't tell you enough about Types and > StdDIS. Types contains generic X11 types (which would be used by Xt, > Xaw, etc). StdDIS contains generic GreenCard definitions and would > also be used by Xt, etc. So I see things developing as follows: > > Graphics.X11 -- reexports all children except StdDIS > Graphics.X11.StdDIS -- should be shared by all GC'd files > Graphics.X11.Types -- shared by all X11 impls > Graphics.X11.Xlib > Graphics.X11.Xt > Graphics.X11.Xaw > Graphics.X11.Xmu > > [StdDIS is just a copy of the file included in GreenCard distributions > and would go away if we were to do the sensible thing and have a > single StdDIS somewhere in the libraries tree.] Ok for Graphics.X11.Types. For StdDIS, if it is shared by all greencard-generated source, then it probably should be something like Greencard.StdDIS and kept in a separate greencard package. For now, putting it in Graphics.X11.StdDIS (and not advertising its existence) should be fine. > > Win32 is currently listed as being under System.Win32. However, > > since we've just put Dotnet at the top of the hierarchy, and Win32 > > contains a range of both systems and graphics stuff, I'd be happy > > for it to be at the top of the hierarchy too. > > I think I'd prefer both to be under System. For Dotnet at least, there's a whole hierarchy that lives under it that mirrors the .NET class hierarchy. System.Dotnet.System.Xml would be a bit of a mouthful, for example. But also these libraries don't fall into any single category in the Haskell hierarchy, so there's no reason we should place them under System any more than Graphics, for example. System.Posix is different: it only deals with system interaction (although the network bits of System.Posix really live under Network, so it's not a clean break). > > HGL is assigned Graphics.HGL, which seems like the right thing. > > Ok, I'll put it there. Or Graphics.Rendering.HGL as Malcolm suggested (actually this is probably more accurate). > > The NHC folks seem to be using their own Makefiles, called > > Makefile.nhc98. > > I'll probably do the same. > > Hmmm, I wonder if the existing scriptMakefile used by Hugs to convert > and compile the libraries should be moved into libraries/base ? By all means... > I gather that the Win32 library is needed for windows version of GHC > so it's a bit moer of an issue there... It's distibuted with the Win32 GHC, but isn't needed by it (AFAIK). Cheers, Simon

4 6

RE: Ideas for library infrastructure
by Simon Marlow 29 Apr '03

29 Apr '03

[oops, accidentally sent the message before I'd finished it...] Ralf Hinze writes: > I like Andres' proposal. It puts some burden on the > compiler writers (but this work has to be invested > only once) and considerably simplifies things for > the library writers. Putting the burden on the compiler writers is exactly what we want to avoid - us compiler writers tend to be too busy writing compilers! Instead, what I would like to see is a separate library infrastructure that smooths over the differences between systems and compilers, and provides something that library writers can just plug their library sources into. I don't think we need to (or indeed, should) go as far as specifying installation locations, or extra tools that compilers should provide. All this should be embodied in the library infrastructure, which is written once and maintained by a single group of individuals. Cheers, Simon

4 4

Ideas for library infrastructure
by Andres Loeh 29 Apr '03

29 Apr '03

This is an attempt to summarize and separate the problems, in the way I see them: There are two things that are too complicated right now: [1] A library writer needs clear instructions what he can do with his library to package it in a standardized way usable with as many Haskell systems as possible. [2] A library user needs a normalized way to install additional libraries into his Haskell system(s) and keep them up-to-date. The following factors make both problems hard to solve: [A] There are multiple Haskell "compilers" out there, with very different properties. They support a different number of extensions, have different command line flags, one isn't even a compiler ;-) Furthemore, they have their own package systems which are incompatible. [B] Different versions of the same compiler are binary-incompatible. So on a compiler upgrade, the user will either need to reinstall libraries completely or at least do something to recompile the libraries. The current approach, as mentioned before, is to make the set of standard libraries increasingly large. I agree that this is not a really good solution, because compile times increase, the users may get huge sets of libraries they don't really need, and they may get them multiple times. A project that works that way, for example, is KDE. It is quite hard to upgrade single packages without upgrading the entire Desktop environment plus libraries. There, I don't find that situation ideal, and still, KDE is at least centrally organized and has global releases. The Haskell compilers are all separate and should stay that way. The situation as it is almost completely rules out binary-only versions of libraries. I will thus try to focus on source based libraries from now on. I vote for a two-level approach: I. Define a clear and precise way how and where a Haskell library has to be installed on a system. This installation should be as compiler-independent as possible. The files should go in a central place, and there should be additional information at that location of the kind that is now for example stored in GHC's package configuration file. As far as a library does not depend on a particular Haskell compiler (for example, it needs C libraries as well), these C libraries may well aready be built and installed. But the Haskell code should be installed in source form. Authors of libraries should find these guidlines in a clear and easy to understand form somewhere on the web, and there should be tool support to facilitate conforming to the standard. I'd go for some well-documented Makefile includes, but if nicely written, a Haskell based system may also work. But I think, the system and language here is secondary, as long as it is clear _what_ needs to be done ... II. Each Haskell compiler should be extended with a tool that can take a library from the standardized source based install described in I. This tool could be similar in interface to ghc-pkg, but should not only collect package configuration information, but also _build_ the Haskell code for the compiler and create and install libraries to the right positions as necessary. The tool should provide a way to _add_, to _remove_, and to _completely regenerate_ libraries. Additionaly, there needs to be one single tool, that levels the potential differences between the package tools of the Haskell compilers and keeps track of all the compilers installed in a system. Let's call it "global package manager". The last thing that would be called in the Makefile (or whatever) that is supplied with a library is the global package manager with the option "add" and the library name. The global package manager would look in the standardized central place for the library, find out for which of the installed Haskell compilers it works, and call their respective package managers that will the build and add the library individually. The last thing that would be called in the Makefile (or whatever) of a Haskell compiler would be its own package manager with the option to regenerate all libraries. Because they are installed in a place independent of the compiler itself, they will survive a compiler upgrade and can automatically be rebuilt for the new version. For hugs rebuilding would be nearly trivial, for GHC is might take a long time, but that's life. Summary of the proposal: * Define a clear standard to install Haskell libraries on a system in a way that is independent of a specific Haskell compiler (both type and version). * Support library writers by tools to make sure they can conform easily to that standard. This includes a global package manager that can be called in the end to pass responsibility over the library to the compilers installed on the system. * For each Haskell compiler, write a/modify the package manager so that it can handle the addition, removal, and complete regeneration of such source-based installed libraries. [1] This will make the situation easier for package writers, because they do not have to care about actually _building_ the library for a specific compiler or compiler version. They will only have to conform to a specific installation format, and that will be supported by tools. [2] This makes life easier for the users, because they can just install additional libraries and "never" have to worry about them again. If they install additional Haskell compilers or upgrade to newer compiler versions, the libraries will always automatically become available. Of course I realize that I probably have forgotten many things and that I may not have made myself clear at all points. So you're all invited to tear this proposal apart! Best, Andres -- Andres Loeh, Universiteit Utrecht mailto:andres@cs.uu.nl mailto:mail@andres-loeh.de http://www.andres-loeh.de

3 2

RE: bug in hGetPS?
by Simon Marlow 28 Apr '03

28 Apr '03

> I don't see that hGetPS should be consistent with hGetLine. > If hGetLine > returns and empty string, you assume the file has two > newlines in a row, > while if hGetPS returns an empty string, since you asked for > a given number > of chars, you know that you've reached the end of the file. I mean that hGetPS is inconsistent with hGetLine in the sense that hGetPS never throws the EOF exception. The consistent interpretation would be for hGetPS to throw an EOF exception if the Handle is positioned at the end of the file, unless the requested byte count is zero. It wouldn't make sense to have hGetLine behave in the same way as hGetPS does now, because as you say there would be two possible meanings for an empty string returned. Cheers, Simon

1 0

RE: bug in hGetPS?
by Simon Marlow 28 Apr '03

28 Apr '03

> I'm not sure if this counts as a bug or a feature... > > If hGetPS h len encounters an EOF before len, it returns a > PackedString > containing the remainder of the file. However, if h is > already at the end > of the file, an exception is raised rather than returning an > empty string. > To me this is rather counter-intuitive. It matches the behaviour of System.IO.hGetLine. Data.PackedString.hGetPS behaves in the way you're suggesting: it just returns short reads instead of flagging EOF. Unfortuantely this is inconsistent with hGetLine, so I'm not sure if the behaviour should stay that way. Cheers, Simon

2 1

possible bug in Eq of packedStrings
by David Roundy 28 Apr '03

28 Apr '03

I think I've come across a bug in the (==) function of PackedString, but it's hard to reproduce. I'm attaching a set of files that hopefully will reproduce the problem. My assumption is that if two packed strings are unequal, then when their unpacked versions must also be equal. This is, of course, assuming one starts with the packed versions, since packing of strings may be lossy... Anyhow, the guts of the problem is in the following (where c1 and c2 are packed strings): if c1 == c2 then return True else do putStr $ "First version:\n" putStr $ unpackPS c1 ++ "\n" putStr $ "second version:\n" putStr $ unpackPS c2 ++ "\n" if unpackPS c1 == unpackPS c2 then putStr "But unpacked, they are the same!!!\n" else putStr "And unpacked they are different!\n" return False I think that it is a bug if the line "But unpacked, they are the same!!!" ever gets printed (which it does--but it's hard to reproduce). This bug is rather troubling to me, since one of the reasons I'm using packed strings is for the assumed efficiency of comparison, the other being of course storage space, which would be sufficient in itself to mandate packed strings. Anyhow, I'm attaching a test program with a test file (which needs to be named both test1 and test2) which is able to reproduce an error in the above. Sorry I haven't made it any simpler. -- David Roundy http://www.abridgegame.org

2 2

RE: Ideas for library infrastructure
by Simon Marlow 28 Apr '03

28 Apr '03

Ralf Hinze writes: > I like Andres' proposal. It puts some burden on the > compiler writers (but this work has to be invested > only once) and considerably simplifies things for > the library writers. Putting the burden on the compiler writers is exactly what we want to avoid - us compiler writers tend to be too busy writing compilers! Instead, what I would like to see is a separate library infrastructure built by non-compiler writers that smooths over the differences between

1 0

bug in hGetPS?
by David Roundy 26 Apr '03

26 Apr '03

I'm not sure if this counts as a bug or a feature... If hGetPS h len encounters an EOF before len, it returns a PackedString containing the remainder of the file. However, if h is already at the end of the file, an exception is raised rather than returning an empty string. To me this is rather counter-intuitive. Since hGetPS already has one way of flagging the EOF condition (returning a string shorter than len), I would prefer to be able (for example) to simply copy a file by a loop such as: cp h1 h2 = do ps <- hGetPS h1 buflen hPutPS h2 ps if lengthPS ps == buflen then cp h1 h2 else return () I know I could just as easily write the same code as: cp h1 h2 = do ps <- hGetPS h1 buflen hPutPS h2 ps eof <- hIsEOF h1 if not eof then cp h1 h2 else return () but since the first should be as fast as the second, I don't see why it shouldn't be made safe. -- David Roundy http://www.abridgegame.org

1 0

FW: First Attempt at Crypto Library
by Simon Marlow 25 Apr '03

25 Apr '03

Folks, Domenic Steinitz has contributed some crypto code (see attachment and msg below). According to our draft hierarchy, there is a place in the hierarchy reserved for crypto libraries: FileFormat.Encryption (see http://www.haskell.org/~simonmar/lib-hierarchy.html) Question 1: there's a comment next to 'FileFormat' which mentions that 'Codec' might be a more accurate name. It seems to me that 'Codec' would indeed be a better choice: many of the libraries under FileFormat are more like pure stream-transformers than file formats, and Codec encompasses both. What do people think about changing this? (there aren't any other implementations of libraries under FileFormat that I'm aware of). If FileFormat became Codec, then the existing FileFormat.Encoding looks a bit odd. But moving FileFormat.Encoding.Base64 up to Codec.Base64 (similarly for FileFormat.Encoding.Yenc) would seem to make sense. Question 2: what should the insides of the FileFormat.Encryption (or Codec.Encryption) hierarchy look like? Cheers, Simon -----Original Message----- From: Dominic Steinitz [mailto:dominic.steinitz@blueyonder.co.uk] Sent: 21 April 2003 18:47 To: Simon Peyton-Jones; Simon Marlow Cc: libraries-request(a)haskell.org Simon, Simon, Here's my first attempt at a crypto library. It compiles and I can run a test using ghc-inplace. The test checks with the example in http://www.itl.nist.gov/fipspubs/fip81.htm (except I couldn't find an example with PKCS#5 padding). I'm not sure what the next steps are. Dominic Steinitz

8 13