- Libraries - Haskell.org

Re: Text I/O library proposal, first draft
by Ashley Yakeley 06 Aug '03

06 Aug '03

At 2003-08-05 20:24, Ben Rudiak-Gould wrote: >1. We don't really need to work out a single recoder interface that's best >for every purpose. Well, something has to be in the library. Certainly I'd like to see pure (non-IO) encodings: if these are fast enough, and we have all the available ones, it might be appropriate to define all others in terms of that. Otherwise I suppose there could be a base interface for pure encodings and a base interface for IO-based encodings. This of course would be less desirable from a purist viewpoint. >2. There are regional encodings in use which are not stateless (JIS is the >only one I know of personally), so I think that a currentLocaleEncoding >function will have to return a stateful interface. OK >We have little choice but to use the C library's tables, >and the only interface to them is the C conversion functions. ... >4. Existing C conversion libraries (whose interfaces we can't change) >store their state in opaque data structures and provide no way to copy >that state. Which conversion library were you thinking of? This is true for iconv, which has a very simple interface, but apparently not for libc: see <http://www.gnu.org/manual/glibc-2.2.5/html_node/Keeping-the-state.html> -- Ashley Yakeley, Seattle WA

1 0

Re: Text I/O library proposal, first draft
by Ashley Yakeley 05 Aug '03

05 Aug '03

In article <Pine.LNX.4.21.0307311506290.31823-100000(a)dark.darkweb.com>, Ben Rudiak-Gould <benrg(a)dark.darkweb.com> wrote: > [Crossposted to Haskell and Libraries. Replies to Libraries.] There's a Haskell Internationalistion mailing list too. Also check out the project on SF: <http://sourceforge.net/projects/haskell-i18n/> There's a bunch of my code for Unicode properties, plus a couple of UTF8 implementations. > module System.TextIOFirstDraft (...) where could be put in Text.* hierarchy > type BlockRecoder from to = > Ptr from -> BlockLength -> Ptr to -> BlockLength > -> IO (BlockLength,BlockLength) UArray and MArray would be slightly cleaner if you're doing the IO thing. But actually my biggest problem is that this is in the IO monad. Given your code, I should be able to write these without resorting to unsafePerformIO: encodeUTF8 :: String -> [Word8] decodeUTF8 :: [Word8] -> Maybe String -- Nothing if not valid Actually, if one makes certain assumptions about encodings, you could get away with something like this: type Encoder base t = t -> [base] type Decoder base t = forall m. (Monad m) => m base -> m t Is this any less efficient? Probably not if you're writing your BlockRecoders in Haskell. > type TextEncoder = BlockRecoder Word32 Octet > type TextDecoder = BlockRecoder Octet Word32 On GHC, Char has exactly the range 0 to 0x10FFFF, as per Unicode codepoints. If this becomes standardised as part of an internationalisation effort, you might want to use Char rather than Word32. -- Ashley Yakeley, Seattle WA

2 2

Test
by Ashley Yakeley 05 Aug '03

05 Aug '03

I've been having trouble posting to this list (but not the other Haskell-related lists) from gmane (NNTP gateway for mailing lists, see <http://gmane.org/> or connect to <nntp://news.gmane.org>). The two messages from me that turned up on the list out of order were messages I had to resend manually to the list address because gmane apparently dropped them. But if you're reading this, the problem is fixed. -- Ashley Yakeley, Seattle WA

1 0

RE: Libraries and hierarchies
by Simon Marlow 05 Aug '03

05 Aug '03

Hi Iavor, > i like this, as i've wanted something like this, and it also seems > simple, although i may be missing something. > if i download a package, i simply put it on my system somewhere, > and add this place to the compiler search path (via a flag, or a > compiler configuration flag). then we can have relative and absolute > imports: absolute imports would be searched for in the > compiler search > path, while absolute imports will be searched for in the same > directory > as the importing module. alternatively one could only have > one kind of > import, that is always searched for in the compiler path. if > i want to > use one version of a library or another, i simply need to adjust the > path when i compile my program. thus "import A.B" would mean i want > module B that is in a subdirectory A of one of the directories in the > path. ambiguous imports should be reported as errors. Since this whole topic is rather hard to think about (I constantly find myself thinking in circles, anyway), could I ask you to elaborate on the scheme you had in mind? - what do you write in the source? are any syntax extensions required? - module header - import declarations - what is the compiler's strategy for finding imports? Is it the same for source files and interface/object files? - what is distributed with a library (a) in source form, (b) in binary form? - can a compiler produce objects/interfaces that are not fixed to a particular absolute module name in the hierarchy? (i.e. so that we can do binary distributions of libraries that aren't tied to particular absolute module names). - is it possible to have modules that reside at several places in the hierarchy simultaneously? Cheers, Simon

2 1

RE: Text I/O library proposal, first draft
by Hal Daume 05 Aug '03

05 Aug '03

this is true; however, it is likely that it would be much faster for me to read in word8s and parse them to Ints myself than to read in Chars (using the old libraries) and parse those to ints (I've done some experiments and the overhead for reading haskell Chars is very non-negligible). i'd like to take advantage of this new infrastructure, if possible... -- Hal Daume III | hdaume(a)isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume > -----Original Message----- > From: libraries-admin(a)haskell.org > [mailto:libraries-admin@haskell.org] On Behalf Of John Meacham > Sent: Thursday, July 31, 2003 4:36 PM > To: libraries(a)haskell.org; haskell(a)haskell.org > Subject: Re: Text I/O library proposal, first draft > > > presumably if you are doing random access on the file, it is > in a known > nonarbitrary text encoding (like utf8). in which case you can > read/access the file with the binary routines and just use the > appropriate text conversions to get data out. > John > > On Thu, Jul 31, 2003 at 03:55:44PM -0700, Hal Daume wrote: > > Hi Ben, > > > > > Bad things: > > > > > > * There's no way to implement fgetpos/fsetpos type > functionality, > > > because coders don't expose their internal state. (In > fact, there > > > would need to be a way to explicitly copy the state, > since it may > > > well include IORefs, Ptrs, etc.) Is this a serious problem? > > > > Yes! This is an enormously serious problem. At least for me. > > > > It's not a problem for writing files, but I really really > really need > > this functionality when reading files. Reason: I'm often > tooling around > > in very large (1gb or greater) files which happen to be > sorted on some > > sort of index and I need to do binary search in them. To > load all the > > file into Haskell or to do linear search is impossible. > > > > Other than that, I rather like the design. > > > > - Hal > > -- > > Haskell mailing list > > Haskell(a)haskell.org > > http://www.haskell.org/mailman/listinfo/haskell > > > > -- > -------------------------------------------------------------- > ------------- > John Meacham - California Institute of Technology, Alum. - > john(a)foo.net > -------------------------------------------------------------- > ------------- > _______________________________________________ > Libraries mailing list > Libraries(a)haskell.org > http://www.haskell.org/mailman/listinfo/libraries >

1 0

RE: Libraries and hierarchies
by Simon Marlow 05 Aug '03

05 Aug '03

Hi Alastair, Thanks for the comments. > I wonder if the extra complication is really that high. > Suppose that every module somehow has a unique name > (128 bit hash, global registry, whatever) and that we have > setup two mappings to the same thing: > > Foo.Bar -> Reid.Consulting.Bar.version27 > Baz.Bar -> Reid.Consulting.Bar.version27 > > to implement this in Hugs, it is _almost_ enough to simply > run all source code through a preprocessor which replaces > occurences of Foo.Bar and Baz.Bar with Reid.Consulting.Bar.version27. Yes, this is a simpler implementation. > >(b) needing unique package names, > > The original proposal tied this feature strongly to packages. > I wonder if this is really necessary. Perhaps not - we only use packages as a convenient notion for a collection of modules within which relative module names can be used, and as a way to identify the collection. > Putting that aside, it seems that unique package names are > fairly easy to come by. Some possibilities are: > > 1) Use the URI of the primary download site. > From choice, the URI would be for a tarfile or whatever > but all that really matters is that it is unique. > This will typically include a version number but, if not, > the user could add one if they care. > > 2) In the likely event that we have a few big repositories > and many small ones, simply prefix the name of the package > by the repository name or, if not available, name of author. > This is effectively the same as (1) but avoids using URIs. > > 3) Directory name on your system. We can't use (3) in GHC, because the package name has to be fixed at compile time, and can't be changed once the binaries have been generated. It's an advantage to have short package names, because (at least in GHC) the package name will be included in every symbol in a compiled library. URIs are attractive, but long. It will make debugging really painful. Another scheme would be to use short package names (eg. "gtk-0.15", "hgl-3.00"), and have a web page (eg. on the Wiki) where everyone can register package names, with links to the appropriate download sites. This works well for the Python folks, and I think having short, readable package names will make the whole scheme more accessible. --- Some notes on dependencies, which occurred to Simon & I: An interesting property of this scheme is that if you distribute a binary library, then you only have to list the package names on which it depends: it doesn't matter where in the hierarchy the packages are sited on the target machine, it will still work. If you distribute source code, then the dependency list must include both package names and sites; this is something that can easily be incorporated into Isaac's distribution toolkit. We need an easy way to set up the system with the correct packages & sites for compiling some code, but that shouldn't be too hard. Interestingly, this is another way that source code can unambiguously refer to libraries: by package name, site, and module name (with the former two properties being specified out-of-band in some additional matter that comes with the source code - perhaps a package specification or similar). This is rather less ugly than using GUIDs. We can also provide GUIDs, of course, because doing so is cheap. Cheers, Simon

1 0

RE: Libraries and hierarchies
by Simon Marlow 04 Aug '03

04 Aug '03

> > Modules in other packages can be imported only by uttering > their full > > path names in the global hierarchy (of the compiler that is > compiling > > the package). > i don't quite understand this part. how is a module going to import > modules from another package, if it does not know where these > packages > are going to be installed? if i am writing a module M, which imports > A.B.X.N which is module X.N from a package P that was > installed at A.B, > doesn't that force everyone who is trying to use my package > to install P at the same place: A.B? Well, this is the key issue. Standing back a moment, I see three alternatives to organising the hierarchy: 1. We have a global registry for module names. The hierarchy is always populated with the same modules on every Haskell system. We have a mechanism for naming libraries after email addresses or domain names, or GUIDs, to ensure that everyone can get a non-clashing module name when they want one. This suffers from (a) needing a global registry for ordinary names (i.e. the non-GUID ones) and (b) overly long module names, (c) no library versioning. 2. We have no global registry for module names. Everyone is free to name library modules whatever they choose. In practice, groups of Haskell programmers will get together and maintain collections of non-clashing libraries, such as the fptools/libraries collection. This suffers from (a) possible name clashes between libraries, (b) no way to unambiguously refer to a specifiy library in Haskell source, (c) still no way to abbreviate long module names, and (d) still no GUIDs or versioning. 3. As (2), but the full name of a module is not fixed in the source code, and can be mapped into the hierarchy at multiple places. This is the method proposed in the message that started this thread. Because modules can be placed in the hierarchy in multiple places, it gives a good way to add library versioning and GUIDs to libraries. This suffers from (a) extra complication in the implementation and specification of the language, (b) needing unique package names, (c) referring unambiguously to a specific library must be done by GUID. So I guess what we're saying is that we're proposing (3) as a solution to the problems of (2). It probably isn't the best solution - indeed, I'm still worried about the implementation difficulties and the fact that we need unique package names, but perhaps solutions to these can be found. To answer the question, to unambiguously refer to another library under this story, you must import it by GUID. Cheers, Simon

4 3

RE: Libraries and hierarchies
by Simon Marlow 04 Aug '03

04 Aug '03

> It also gives the advantage that modules can be moved in the > hierarchy (by a > command-line flag or whatever) specifically to support > whatever a particular > program happens to require. > > Would the following be possible under your proposal: > > module M1.A imports Collections.Foo and only works with version 1 > module M2.B imports Collections.Foo and only works with version 2 > module C imports M1.A and M2.B > > Could we compile/use M1.A with Graphics.UI mapped to > Graphics.UI.v1 and M2.B > with Graphics.UI mapped to Graphics.UI.v2 and then correctly > compile module C? Yes, absolutely. There are issues about exactly what the programmer would have to do to make this happen: under the story I had in mind you would have to fiddle with a user-local package.conf file to change the mapping. Perhaps command-line flags could be added to make this more convenient, though. Cheers, Simon

2 1

RE: Libraries and hierarchies
by Simon Marlow 04 Aug '03

04 Aug '03

> Simon Marlow writes: > : > | After installing the library, the tree of modules it contains will > | be grafted into the global hierarchy at possibly many places, and > | the modules can then only be imported by uttering their full path > | names in the global hierarchy. > > Hi. > > If that makes a module (module M where data T) appear twice in the > global hierarchy (at A and B), do the types A.M.T and B.M.T become > distinct from each other? > > Sorry if that's an obtuse question. I hope it has a short answer. That's a good question. The short answer is "no". Unfortunately that's made me realise that the simple implementation schemes we proposed won't work. The compiler needs to internally use module names which consist of a pair of (package name, relative module name), so that it can identify modules from the same package but different parts of the hierarchy. This won't be too difficult for GHC, but I fear it will be more of an upheaval for Hugs. Cheers, Simon

1 0

Re: Raw I/O library proposal, second (more pragmatic) draft
by Ashley Yakeley 04 Aug '03

04 Aug '03

In article <3429668D0E777A499EE74A7952C382D19D1238(a)EUR-MSG-01.europe.corp.microsoft .com>, "Simon Marlow" <simonmar(a)microsoft.com> wrote: > I wanted to float a generalisation of this scheme, though. I'm > wondering whether it might be a good idea to make InputStream and > OutputStream into type classes, the advantage being that this makes > streams more extensible - one example is that memory-mapped files fit > neatly into this framework. I already have 6 examples of things that > can have streams layered on top (or *are* streams), and there are almost > certainly more. > > Here's some signatures for you to peruse: > > class Stream s where > closeStream :: s -> IO () > streamSetBuffering :: s -> BufferMode -> IO () > streamGetBuffering :: s -> IO BufferMode > streamFlush :: s -> IO () > isEOS :: s -> IO Bool > > class InputStream s where > streamGet :: s -> IO Word8 > streamReadBuffer :: s -> Integer -> Buffer -> IO () > streamGetBuffer :: s -> Integer -> IO ImmutableBuffer > streamGetContents :: s -> IO [Word8] > > class OutputStream s where > streamPut :: s -> Word8 -> IO () > streamPuts :: s -> [Word8] -> IO () > streamWriteBuffer :: s -> Integer -> Buffer -> IO () I have some issues/queries with this. 1. The monad (and maybe also the "byte" type) should be parameterised. There are two ways of doing this. General way: class InputStream s m | s -> m where streamGet :: s -> m Word8 ... Haskell 98-compatible way: class InputStream s where streamGet :: s m -> m Word8 ... Of course you'll need to be able to parameterise your buffer types, too. 2. streamWriteBuffer probably wants an ImmutableBuffer. The idea is that a "Buffer" is a special kind of "ImmutableBuffer" that also supports being set. Probably you should rename them. They're just flavours of "Array" anyway, aren't they? 3. I note all the class members have types of the form "s -> a" each for some "a" not dependent on "s". This means streams might be a candidate for data structures: data InputStream m = { isStream :: Stream m, streamGet :: m Word8, streamReadBuffer :: Integer -> Buffer -> IO () ... } I'm not sure which is preferable however. Data-structure inheritance has to be done by hand (except see point 9., it might only be a close function), and they don't allow default implementations (yet). 4. Shouldn't "streamFlush" belong to OutputStream? 5. It might be useful to have a "streamAvailable" function in InputStream that gets all bytes immediately available without blocking. 6. Shouldn't "isEOS" belong to InputStream? Come to think of it, perhaps streamGet should return a (Maybe Word8) instead of a Word8. 7. OutputStream should have a streamSendEOS separate from streamClose. This is used in TCP, where one can say "I'm done sending" on the outgoing channel while still receiving octets on the incoming channel. I would argue that for file streams, it would be appropriate for streamSendEOS to set the end of file at that point. 8. Buffering is a special thing, isn't it? Should input-buffering and output-buffering be separated? Should there be separate classes for buffering? 9. If you accept all this, all you have left in Stream is closeStream, which I would argue is all that "sources" and "sinks" really have in common. And perhaps some don't even need that. The class might be better named "Closable" or somesuch. > -- Files in the filesystem, with access rights > data File > data FileInputStream -- instance Stream, InputStream > data FileOutputStream -- instance Stream, OutputStream Would it be worth also providing a random-access-based interface to files in addition to a stream-based? I know either one can be built on the other. For instance: accessFile :: File -> IO FileAccess closeAccess :: FileAccess -> IO () getFileLength :: FileAccess -> IO Integer; setFileLength :: FileAccess -> Integer -> IO (); readFile :: FileAccess -> Integer -> Integer -> IO ImmutableBuffer writeFile :: FileAccess -> Integer -> ImmutableBuffer -> IO () Of course, this might be better generalised as a class. > -- URIs: > data URIStream > getURI :: URI -> IO URIStream > instance InputStream URIStream Eeesh. URIs hide a lot of semantics, which this rather glosses over. And isn't writing an FTP client hard? -- Ashley Yakeley, Seattle WA

1 0