August 2003 - Libraries

RE: Libraries and hierarchies
by Simon Marlow 08 Aug '03

08 Aug '03

> Yes. I would have thought that library authors would want to avoid > this. I guess you don't mind: the library user just grafts one of the > libraries in somewhere else. So the mapping supplied with > the package is > presumably just a default that can be overridden. And to read someone > else's source code I also need their personalized mappings for the > various packages they're using. > > So you have extended the syntax of the language: you need a syntax for > these mappings, and the static semantics operates on ordinary source > plus this mapping language. Thinking about it, I'm not sure that the language definition needs to be extended at all, to allow the proposed scheme. The language definition (with hierarchical modules, of course) continues to specify the meaning of a collection of modules, each with an absolute module name. What we are proposing is two mechanisms that the compilation system can provide so that the user has more flexibility in constructing a set of modules with absolute names. 1. A method by which a module can change its module name after being compiled. 2. A method by which a module can be exposed as several absolute module names to future compilations. I think these could be expressed as source-to-source transformations, leaving the Haskell 98 + hierarchical modules spec as it is. What about the mappings between packages and sites? Don't we need to specify a syntax for these, and include it in the language spec? I don't think so. This is all just a facility provided by the compilation system for populating the module namespace with modules - we don't need to add it to the language, in the same way that we don't need to include search paths and filenames in the language. This is something of a technicality I suppose, but it's nice to know that the language spec can remain unchanged. It's also useful to cleanly separate the meaning of a program as a set of modules, from the means by which we arrive at that set of modules. Cheers, Simon

3 2

RE: Libraries and hierarchies
by Simon Marlow 08 Aug '03

08 Aug '03

> My question was: how do I know if the following is legal: > > module Foo (f) where > import A.B.C (f) > import X.C (f) > > I have to look at the graftings, which are additional to the text. No, you just need the text of the modules A.B.C and X.C, same as before. > > Well, how do you currently understand some program > distributed without > > the full source for all the libraries it uses? You need > instructions > > for where to find those libraries - the only new thing is > that now you > > also need grafting info. It's just an extension to the > dependency info > > that comes with some source code. > > All I need is the documentation of the package, which is an > abstraction > of the source text of its modules. I see an import in the program, > and I look up the module name in the documentation. Documentation is an important point. We could have dynamic documentation that changes with the grafting (Haddock.cgi, anyone?), or we could use relative module names only in the documentation. Or we could just continue to use the default module names. In practice, I don't think using the default module names in the documentation will lead to many difficulties - most of the time, people aren't going to use any weird local graftings. The Haddock documentation will probably including information about where the root of the package is for each module (it already contains the package name). > Currently, the legality and meaning of a program is > determined by applying > the static semantics to the text of the modules comprising > the program. > True, this ignores how those modules were found, but it is coherent. > But what is the input to the static semantics under the new scheme? > It seems that it's modules grouped into packages, plus graftings. > Therefore those things are part of the language. Think of it like this: the static semantics still takes source text as input, and the package grafter transforms source text into source text. It's like CPP: there's a trasnformation which is applied to the source code before it is compiled. We don't include the specification of CPP as part of the language, and we loosely describe CPP source code as "CPP'd Haskell 98", or sometimes just "Haskell 98" when we're being lazy. Cheers, Simon

2 1

RE: Libraries and hierarchies
by Simon Marlow 08 Aug '03

08 Aug '03

> I may be misunderstanding the transformations you have in > mind, but how > do I tell whether A.B.C.f and X.C.f are the same thing? Like this: module X.C (module A.B.C) where import A.B.C > Also, suppose I have the documentation for someone's package, and I'm > looking at a program that uses that package, but grafted in at some > unknown point in the hierarchy. How do I understand that program? Well, how do you currently understand some program distributed without the full source for all the libraries it uses? You need instructions for where to find those libraries - the only new thing is that now you also need grafting info. It's just an extension to the dependency info that comes with some source code. Cheers, Simon

2 1

RE: Raw I/O library proposal, second (more pragmatic) draft
by Simon Marlow 08 Aug '03

08 Aug '03

> > fileRead :: File -> FileOffset -> Integer -> Buffer -> IO () > > fileGet :: File -> FileOffset -> Integer -> IO ImmutableBuffer > > Should fileRead return the number of bytes read, in case of > asking for > more bytes than are in the file? With fileGet you can simply > look at the size of the return buffer. Good point. > > -- | Flushes the buffer to the operating system for an output > > -- buffer, or discards buffered data for an input buffer. > > flush :: s -> IO () > > Those are two very different things, aren't they? The first > pushes data > along the pipe (rather than discarding it). The second discards data, > equivalent to calling streamGetAvailable and ignoring the result. It's not quite the same as calling streamGetAvailable. flush would discard data that is buffered on the Haskell side, whereas streamGetAvailable might grab any data that is buffered in the OS. I'm not sure whether this is useful or not, but it's exactly what hFlush does currently. > > -- | Flushes the buffered data as far as possible, even to the > > -- physical media if it can. It returns 'True' if the data > > -- has definitely been flushed as far as it can go: to the > > -- disk for a disk file, to the screen for a terminal, > and so on. > > sync :: s -> IO Bool > > This one seems to be output only. Would it ever be meaningful to call > sync on an input-stream? You're right. If flush was moved to OutputStream, then it would make sense to move sync there too. > > class InputStream s where > > class OutputStream s where > > Presumably you wanted Stream superclasses for these? Not necessarily - there's no need for it, other than to reduce the size of contexts on types. > > streamGet :: s -> IO Word8 > > What does this return after the last byte has been read? What > does this return for an empty file? Exactly as hGetChar, that is it throws an exception at the end of the file. > > Parameterise InputStream over the element type too, so we can > > combine InputStream and TextInputStream? NO: uses multiparam type > > classes. > > Unless of course your stream type is a type constructor: > > class InputStream s where > streamGet :: s m e -> m e > ... I can see the advantage in abstracting over the element type (in fact, I think I'll explore that). What would be the motivation for abstracting over the monad type? IMHO, it has to be done consistently or not at all. That is, *all* our IO code should be abstracted over the monad, and I don't think that's a good idea. Cheers, Simon

4 3

RE: Raw I/O library proposal, second (more pragmatic) draft
by Simon Marlow 08 Aug '03

08 Aug '03

> On Wed, 6 Aug 2003, Simon Marlow wrote: > > > -- TODO: what if a file refers to a FIFO? > > fileInputStream :: File -> FileOffset -> IO FileInputStream > > fileOutputStream :: File -> FileOffset -> IO FileOutputStream > > Part of my original design was that File values always refer to > random-access files. Most of the operations on Files make no > sense with > anything else. openFile might succeed on something like /dev/kmem > (although I'm not convinced that this is type-safe :-), but > to open a FIFO > you had to use openXputStream instead. This avoids the > questions raised by > using the functions above on non-seekable handles (What would the > FileOffset argument mean? and What happens if you put more > than one input > or output stream on the same FIFO?). I agree. Should we provide openInputStream :: FilePath -> IO InputStream openOutputStream :: FilePath -> IO OutputStream what if you want to open both streams? You have a race condition, so we must provide openInputOutputStream :: FilePath -> IO (InputStream,OutputStream) Furthermore, we have to discourage the use of openFile + fileInputStream in favour of openInputSream, because the former combination doesn't work with FIFOs and special files (this seems a bit fragile, but I'm not sure what to do about it). What mechanism should underly these functions? Should it be data FilesystemObject = FSFile File | FSFIFO FIFO | FSTerminal Terminal | FSDevice Device openFilesytemObject :: FilePath -> IO FilesystemObject Now perhaps we're getting too Unix specific. I don't have a clear idea for how to proceed. Suggestions? Cheers, Simon

1 0

RE: Raw I/O library proposal, second (more pragmatic) draft
by Simon Marlow 08 Aug '03

08 Aug '03

> Here's the code for hFlush from the GHC 6.0 sources: > > > -------------------------------------------------------------- > ------------- > -- hFlush > > -- The action `hFlush hdl' causes any items buffered for output > -- in handle `hdl' to be sent immediately to the operating > -- system. > > hFlush :: Handle -> IO () > hFlush handle = > wantWritableHandle "hFlush" handle $ \ handle_ -> do > buf <- readIORef (haBuffer handle_) > if bufferIsWritable buf && not (bufferEmpty buf) > then do flushed_buf <- flushWriteBuffer (haFD handle_) > (haIsStream > handle_) buf > writeIORef (haBuffer handle_) flushed_buf > else return () I stand corrected :-) I was thinking about what happens when a read/write Handle with a full read buffer is written to. This is one of the really awkward cases in the current IO library - you have to discard the input buffer, but also move the file pointer back to where it should be before you start writing. Cheers, Simon

1 0

RE: Libraries and hierarchies
by Simon Peyton-Jones 07 Aug '03

07 Aug '03

| Wouldn't one still want the module prefix + relative module name | combination to produce unique module names? So presumably the prefix | should include the package name (though without a version number). I don't think so. Under our proposal, "original names" are globally unique and stable, but the name for a module in a compilation depends on what graftings you have done. Maybe you are saying "what happens if you graft two modules from different packages so that both modules get the same name?". For example, suppose we have two packages dataOps-1.0 has a module with (relative) name A.B foo-2.7 has a module with (relative) name X.A.B Suppose you graft in dataOps-1.0 at site DataOps.X and foo-2.7 at site DataOps Then "import DataOps.X.A.B" is ambiguous: it could mean either module. That is plainly an error. To be consistent with name clashes within module, the complaint should probably only come if you actually try to import it. Was that what you meant? Simon

3 2

Re: Libraries and hierarchies
by Graham Klyne 07 Aug '03

07 Aug '03

At 11:10 01/08/03 +0100, Simon Marlow wrote: >You could even use this mechanism, in a per-user configuration file say, >to site the GTK+HS library directly in Gtk.*, if you're too lazy to type >Graphics.UI all the time. This wouldn't be recommended though: any >source code you write won't compile on someone else's system that >doesn't have the same convention. It's not clear to me how this final problem is solved in the general case of the scheme you describe, unless everybody uses the same defaults, which practically speaking I think is pretty much the same as having a fixed global hierarchy. Later, you say: [[ Modules in other packages can be imported only by uttering their full path names in the global hierarchy (of the compiler that is compiling the package). ]] The "of the compiler..." bit looks problematic to me. Another thing that worries me a little is the apparent complexity of the scheme you proposed. I think a clear and primary goal should be to make it easy to install, compile and use any software library from any source. Anything less would, I think, negatively impact take-up of Haskell as a serious programming tool. #g --------------------------------- Graham Klyne <GK(a)NineByNine.net> Nine by Nine http://www.ninebynine.net/

1 0

RE: Libraries and hierarchies
by Simon Marlow 06 Aug '03

06 Aug '03

> | am i right in assuming that "current directory" refers to the > directory > | in which the file being compiled is located? e.g. > | if module Main has a declartion "import M" > | ghc A/B/Main.hs > | will look for "A/B/M.hs" > > No. Our current idea is that "current directory" means "directory in > which you started the compiler. Generalising slightly - "current directory" means each of the directories in the search path, which by default just contains the directory in which you started the compiler. But you can change the search path using the -i flag. Cheers, Simon

1 0

RE: Libraries and hierarchies
by Simon Peyton-Jones 06 Aug '03

06 Aug '03

[Nothing new in this message: just a summary, with some useful terminology.] | Interestingly, this is another way that source code can unambiguously | refer to libraries: by package name, site, and module name (with the | former two properties being specified out-of-band in some additional | matter that comes with the source code - perhaps a package specification | or similar). This is rather less ugly than using GUIDs. We can also | provide GUIDs, of course, because doing so is cheap. Suppose someone makes a package called glut-6.0 with modules Graphics, Graphics.GLUT, and so on. Suppose Graphics.GLUT includes a function foo. Then: The "original name" of function foo is glut6.0:Graphics.GLUT.foo That is <package-name>:<relative-path-within-package> Grafting the package glut6.0 into the module hierarchy does not change the original name of anything in the package. This is important, because the binaries generated for the package use original names for cross-module references, and we don't want those to change when we graft. All this does is change the need for *module names* to be globally unique into the requirement for *package names* to be unique. This problem is much easier because a) There are fewer packages, so some central help-yourself global registry is feasible, as Simon M suggests. b) Package names can be longish and clunky (e.g. including a version number), because we provide a way to avoid mentioning them in source code. I personally think that using GUIDs for package names would be a mistake. They are essentially opaque to people, so there needs to be some separate infrastructure to describe what a GUID names, and the extra layer of indirection seems to buy little. It's not hard to have unique package names (I claim) and that'll do the job nicely. Indeed a package name, as suggested here, then *is* a globally unique ID, or GUID, only with a comprehensible name. The "graft package into tree" mechanism can be seen as a method to implement (b). Installing a package means you can name modules in that package using A.B.C module notation, without explicitly mentioning the package itself. As Simon's message above said, source code therefore only makes sense given some set of graftings (= package -> module prefix mapping), and one might want to make that part of the package (source-code) description. Ganesh asks | Would the following be possible under your proposal: | | module M1.A imports Collections.Foo and only works with version 1 | module M2.B imports Collections.Foo and only works with version 2 | module C imports M1.A and M2.B Yes, this is ok, as Simon indicated earlier. Presumably there are two packages coll-1.0 and coll-2.0. We graft them in at (say) Collections, and Old.Collections and away we go. Or, we invoke the compiler for M1.A with a command line flag to graft in coll-1.0 at Collections, and then compile M2.B grafting coll-2.0 at Collections. Either way, a value constructed by package coll-1.0 will be type-incompatible with functions in coll-2.0. The former will have original names "coll-1.0:Foo.T" while the latter will have "coll-2.0:Foo.T". Trying to provide transparent type upgrade is too hard. Manuel didn't like the fact that an import could mean "relative or absolute". But presumably we want to continue to write little programs with three modules A, B, Main, and have Main just say 'import A'. So presumably the current directory is always implicitly grafted into the module hierarchy at the root -- and that is all we need for making internal references within a package work out. Simon

5 4